Commit Graph

819 Commits

Author SHA1 Message Date
yihaoDeng f4bde33dc3 avoid mem leak 2023-06-02 06:06:05 +00:00
Haojun Liao cce87a036f other: merge 3.0. 2023-06-02 13:34:20 +08:00
Haojun Liao d45aa7ef84
Merge pull request #21555 from taosdata/fix/liaohj_main
other: merge main into 3.0.
2023-06-02 13:11:12 +08:00
Haojun Liao f1bd829977 refactor: do some internal refactor. 2023-06-02 13:10:54 +08:00
Haojun Liao 27b7d1ec88 refactor: do some internal refactor. 2023-06-02 11:17:44 +08:00
yihaoDeng 4d0ce625c4 Merge branch '3.0' of https://github.com/taosdata/TDengine into feature/TD-19042 2023-06-01 22:55:30 +08:00
wangmm0220 a7418d439b Merge branch '3.0' of https://github.com/taosdata/TDengine into feature/TD-19042 2023-06-01 22:50:51 +08:00
wangmm0220 56782a5d41 fix:open task case & modify mqRebVgReq encode/decode style 2023-06-01 18:29:16 +08:00
Haojun Liao 5d98a319c7
Merge pull request #21515 from taosdata/feature/TD-19042
feat: support create topic as stable with conditions
2023-06-01 09:56:55 +08:00
Haojun Liao 5301c5756a other: merge main into 3.0. 2023-06-01 09:29:18 +08:00
Haojun Liao 6d04c4e2fd refactor: do some internal refactor. 2023-05-31 19:05:40 +08:00
Haojun Liao 9e021087a9 refactor: do some internal refactor. 2023-05-31 19:04:53 +08:00
Haojun Liao a3abe25539 refactor: do some internal refactor. 2023-05-31 19:03:37 +08:00
Haojun Liao e237b4ac39 refactor: do some internal refactor. 2023-05-31 09:19:05 +08:00
wangmm0220 689ee05527 Merge branch '3.0' of https://github.com/taosdata/TDengine into feature/TD-19042 2023-05-30 15:45:46 +08:00
wangmm0220 41ed0138b4 fix:compile error 2023-05-29 17:25:11 +08:00
wangmm0220 07bdb95b1a fix:compile error 2023-05-29 17:10:11 +08:00
wangmm0220 5eb1c559e5 fix:save offset if latest 2023-05-29 15:51:00 +08:00
liuyao b541a0a053 set sched status 2023-05-29 15:42:47 +08:00
wangmm0220 78db3520a6 fix:conflict 2023-05-29 14:57:03 +08:00
wangmm0220 a71878d07f feat: support create topic as stable with conditions 2023-05-29 11:46:59 +08:00
Haojun Liao 1e786624b2
Merge pull request #21501 from taosdata/refact/fillhistory
fix(stream): set the skip offset when resume stream task.
2023-05-27 22:45:54 +08:00
Haojun Liao b38e253019 fix(stream): set the skip offset when resume stream task. 2023-05-26 22:51:58 +08:00
Haojun Liao 87957e2bb0
Merge pull request #21478 from taosdata/fix/TD-23423
fix tsdb read error version
2023-05-26 16:56:09 +08:00
Haojun Liao f61f37c936 fix(tmq): set the correct offset rsp when no poll occuring. 2023-05-26 14:20:01 +08:00
Haojun Liao 1e845acabe fix(tmq): allow seek before get assignment 2023-05-26 11:34:12 +08:00
Haojun Liao cea8334427
Merge pull request #21449 from taosdata/refact/fillhistory
refactor: remove the module dependency.  TD-24279
2023-05-26 09:01:45 +08:00
liuyao 75eb724c37 fix tsdb read error version 2023-05-26 08:55:23 +08:00
Haojun Liao da9f33a483 fix: fix error in set function ptr. 2023-05-25 09:56:34 +08:00
Haojun Liao 4a42f6b983 fix: fix error in api call. 2023-05-25 00:50:03 +08:00
wangmm0220 8537e4e80e feat:add tag filter for stable subscribe 2023-05-24 18:26:55 +08:00
Haojun Liao 1f7f326bed refactor: do some internal refactor and set the api function ptr. 2023-05-24 13:22:05 +08:00
liuyao 180abf2229 set agg task status 2023-05-24 09:34:40 +08:00
liuyao b5d01f9bab set agg pause status 2023-05-24 09:19:28 +08:00
Haojun Liao 39a1fa8f78 refactor: do some internal refactor. 2023-05-23 19:20:25 +08:00
wangmm0220 ed612088c0 fix:lost data because tsdbreader or task is killed 2023-05-23 10:32:13 +08:00
wangmm0220 d8387b95ff fix:set task status to 0 in inti 2023-05-22 20:14:43 +08:00
wangmm0220 591b51f624 fix:set task code success after kill 2023-05-22 12:42:21 +08:00
Haojun Liao 899e4d3350 refactor: do some internal refactor. 2023-05-20 18:13:58 +08:00
Haojun Liao de44c9160c refactor: do some internal refactor. 2023-05-20 18:05:39 +08:00
wangmm0220 7b735c67d0 fix:move sleep logic from rpc thread to tmq thread & fix some error 2023-05-20 16:36:25 +08:00
wangmm0220 8183be4aff fix:move sleep logic from rpc thread to tmq thread & fix some error 2023-05-20 16:13:52 +08:00
wangmm0220 04858fae22 fix:tsdbreader is free by mistake 2023-05-20 12:20:54 +08:00
wangmm0220 1ddbdad33c fix:set task status killed when vnode receive subscribe msg from mnode 2023-05-19 21:40:33 +08:00
wangmm0220 33d6df8717 fix:set task status killed when vnode receive subscribe msg from mnode 2023-05-19 19:11:49 +08:00
wangmm0220 c7c255e967 fix:set task status killed when vnode receive subscribe msg from mnode 2023-05-19 18:06:05 +08:00
wangmm0220 ff2b545b27 fix:set task status killed when vnode receive subscribe msg from mnode 2023-05-19 17:16:47 +08:00
wangmm0220 f7ab8dabf0 fix:[TD-23972] push subscribe msg to vnode even though consumer not change 2023-05-19 15:14:39 +08:00
wangmm0220 238254e49e fix:add task status check in doQueueScan & checkout consumer in tqProcessPollReq 2023-05-19 14:06:43 +08:00
Haojun Liao d8525123a2 enh(stream): add API to retrieve last ts for multi-tables. 2023-05-19 11:03:41 +08:00
Haojun Liao c4c5bbb65c other: merge 3.0 2023-05-18 17:28:40 +08:00
liuyao da3fc5111e fix pause agg task 2023-05-18 15:29:41 +08:00
Haojun Liao 81f821a6fc refactor: do some internal refactor. 2023-05-18 10:51:43 +08:00
wangmm0220 54f4860139 fix:rollback [TD-23972] push subscribe msg to vnode even though consumer not change 2023-05-17 19:44:56 +08:00
Haojun Liao b9bda9b30f fix(stream): set the correct start offset for resumed stream task. 2023-05-17 19:36:42 +08:00
wangmm0220 405cdfa7bc fix:error in pHandle lock 2023-05-17 19:02:16 +08:00
wangmm0220 1d9dd153ea fix:error in pHandle lock 2023-05-17 18:57:45 +08:00
Haojun Liao a4d9210b07 fix(tmq): add filter for taosX poll rsp. 2023-05-17 16:40:44 +08:00
wangmm0220 4bb25c280f fix:add log 2023-05-17 15:54:36 +08:00
Haojun Liao 041ffa91bd other: merge main. 2023-05-17 15:42:15 +08:00
Haojun Liao f55f8d245c fix(tmq): avoid the offset is revised by pool rsp which is issued before invoking tmq_offset_seek function. 2023-05-17 15:19:51 +08:00
Haojun Liao 366cd1871a
Merge pull request #21314 from taosdata/fix/liaohj_main
fix(stream):extract delete msg from wal.
2023-05-17 14:52:49 +08:00
Haojun Liao 3896e148fd log(tmq): add some logs. 2023-05-17 11:37:38 +08:00
Haojun Liao f3f52c0232 fix(stream): add some logs and set the wal offset to be the ended version when finishing the step2 recover stage. 2023-05-17 10:43:23 +08:00
wangmm0220 c52a4ea4f0 fix:merge from 3.0 2023-05-17 01:11:55 +08:00
wangmm0220 90099ea1c0 fix:compile error in windows 2023-05-16 18:41:35 +08:00
wangmm0220 e6860cada8 fix:compile error in windows 2023-05-16 18:35:10 +08:00
wangmm0220 a52939127f fix:dead lock 2023-05-16 17:58:31 +08:00
wangmm0220 e6e0ac7603 fix:alwalys in exec if consumer Id mismatch & check result for wal seek ver 2023-05-16 14:01:38 +08:00
Haojun Liao e35d145734 fix(stream): fix memory leak. 2023-05-16 13:59:13 +08:00
Haojun Liao 30e7cb5860 fix(tmq): avoid return delete msg for table subscription. 2023-05-16 10:42:02 +08:00
Haojun Liao 5e16db4e19 fix(stream):extract delete msg from wal. 2023-05-16 08:54:03 +08:00
wangmm0220 62520af2e5 fix:core dump 2023-05-16 00:32:21 +08:00
wangmm0220 36454c73ed fix:add lock to pHandle for safety 2023-05-15 18:38:14 +08:00
Haojun Liao 170c603995 other: merge 3.0 2023-05-13 20:16:55 +08:00
Haojun Liao 99148d6719 other: merge 3.0 2023-05-13 20:02:48 +08:00
Haojun Liao 236926b9ee other: merge enh/rocksdbstate 2023-05-13 19:49:28 +08:00
Haojun Liao 40ffe0cd83 fix(tmq): check handle status before close sub. 2023-05-13 16:45:21 +08:00
Haojun Liao b383651d8b fix(tmq): fix the invalid write and set the flag when closing taosx sub. 2023-05-13 16:17:26 +08:00
Haojun Liao 99422c87c3 fix(tq): set the offset to be the consume offset, instead of the committed offset. And update the table sim. 2023-05-12 16:14:56 +08:00
Haojun Liao 7ef16e1922 other: merge main. 2023-05-12 11:49:53 +08:00
liuyao 8b7b804606 opt partition by tbname 2023-05-11 18:47:27 +08:00
Haojun Liao 26bea33ccc fix(tmq): add null check for print log. 2023-05-11 17:56:19 +08:00
Haojun Liao 2b736ffd60 fix(stream): avoid duplicated task id in task id list. 2023-05-11 11:47:57 +08:00
Haojun Liao 8a1b0ed052 refactor: do some internal refactor. 2023-05-11 10:35:09 +08:00
yihaoDeng 5708fcb5cb Merge branch 'enh/3.0' into enh/rocksdbSstateMerge 2023-05-10 08:02:16 +00:00
Yihao Deng c6e5879f44
Merge branch 'main' into enh/rocksdbSstate 2023-05-10 10:04:28 +08:00
yihaoDeng 8356533eb1 Merge branch 'enh/standVer' into enh/rocksdbSstateMerge 2023-05-09 12:13:16 +00:00
dapan1121 1f973123c4
Merge pull request #21177 from taosdata/fix/TD-23972
fix:[TD-23972] push subscribe msg to vnode even though consumer not c…
2023-05-08 20:29:31 +08:00
Yihao Deng cb223acf82
Merge branch 'main' into enh/rocksdbSstate 2023-05-08 09:45:27 +08:00
Haojun Liao 9392c03b7f fix(stream): add some logs. 2023-05-06 19:05:55 +08:00
wangmm0220 279fe0803f fix:[TD-23972] push subscribe msg to vnode even though consumer not change 2023-05-06 18:35:24 +08:00
Haojun Liao da6212bab0 other: merge main. 2023-05-06 12:51:38 +08:00
wangmm0220 55eddbfb5e fix:[TD-23972] push subscribe msg to vnode even though consumer not change 2023-05-06 10:54:05 +08:00
Haojun Liao 1fa4327f6c Merge branch 'main' into fix/liaohj_main 2023-05-06 10:15:23 +08:00
Haojun Liao 29fcd1b83a refactor: do some internal refactor. 2023-05-06 10:08:05 +08:00
Haojun Liao 8958aabe4b fix(stream): opt task scan performance. 2023-05-05 19:33:30 +08:00
Haojun Liao 8b12d4d3da fix(stream): secure the delete task operation. TD-1198 2023-05-05 15:02:03 +08:00
Haojun Liao 11dccf12de
Merge pull request #21131 from taosdata/fix/liaohj_main
refactor: do some internal refactor.
2023-05-04 11:26:00 +08:00
Haojun Liao 851193a033
Merge pull request #21060 from taosdata/fix/TS-3222
opti:change push mgr to consume msg for subscribe
2023-05-04 10:16:41 +08:00
liuyao d8a1f15d10 fix resum bug 2023-05-04 09:28:06 +08:00
Haojun Liao 3c61932ecc refactor: do some internal refactor. 2023-05-01 19:28:54 +08:00
Haojun Liao 777ed1761f refactor: update logs. 2023-04-29 18:24:46 +08:00
Haojun Liao bbb8aaa0dc refactor: add some logs. 2023-04-29 02:37:10 +08:00
Haojun Liao b69137df44 refactor(log): add some logs. 2023-04-29 00:00:25 +08:00
Haojun Liao c647ded0da fix(stream): add some logs. 2023-04-28 19:47:49 +08:00
Haojun Liao b7c8346b44 other: merge other branch. 2023-04-28 13:06:47 +08:00
Haojun Liao f9a64cbcd6 other: add some logs. 2023-04-28 12:58:03 +08:00
liuyao 2984c5a758 merge main 2023-04-28 10:34:46 +08:00
wangmm0220 cce868d140 fix:offset encode assert error 2023-04-28 09:45:16 +08:00
wangmm0220 1ee1b0422c fix:change push mgr from SArray to Hash 2023-04-27 23:44:27 +08:00
wangmm0220 db5b5c828e fix:add log for msg push 2023-04-27 18:35:10 +08:00
Haojun Liao 91f9b58f9a refactor: do some internal refactor. 2023-04-27 16:06:06 +08:00
wangmm0220 8677b56a4f fix:assert error in tqProcessSubmitReqForSubscribe if put pHandle to array twice 2023-04-27 15:44:44 +08:00
Haojun Liao b7040ed15e enh(stream): support resume from the latest data, and do some internal refactor. 2023-04-27 15:07:54 +08:00
wangmm0220 224d87b131 fix:cosume null if rebalance 2023-04-26 17:58:14 +08:00
54liuyao b79313ecd7 pause&resume 2023-04-25 16:59:09 +08:00
wangmm0220 d1e5d6e0f9 fix:pHandle->msg is not null if rebalance 2023-04-25 16:23:58 +08:00
wangmm0220 41bec8560a opti:change push mgr to consume msg for subscribe 2023-04-25 09:58:28 +08:00
wangmm0220 8572f4a32f opti:change push mgr to consume msg for subscribe 2023-04-24 19:41:30 +08:00
wangmm0220 77e03bfd78 opti:change push mgr to consume msg for subscribe 2023-04-23 20:14:49 +08:00
liuyao 641c707b37 pause&resume 2023-04-23 17:37:10 +08:00
liuyao 5bd8fe5441 feat:add pause&resume 2023-04-23 15:09:01 +08:00
54liuyao 5089528a02 feat:stream pause&resume 2023-04-23 15:08:51 +08:00
Haojun Liao 9cf89c6b6f enh(tmq): add check for consumer Id; 2023-04-23 11:51:43 +08:00
yihaoDeng 917b07348c Merge branch 'fix/rocksdbState' into enh/rocksRevert 2023-04-23 02:37:26 +00:00
Haojun Liao 3dbe208712 enh(tmq): fix memory error and add retrieve wal info as required. 2023-04-22 15:34:46 +08:00
wangmm0220 6efbcfec4b Merge branch 'main' of https://github.com/taosdata/TDengine into fix/TS-3222 2023-04-21 17:46:35 +08:00
Haojun Liao 6c641cff39 fix(stream): fix error in start stream tasks. 2023-04-21 09:37:16 +08:00
Haojun Liao 471abd9160 log: update the log. 2023-04-20 23:39:13 +08:00
Haojun Liao 5e6c06e253 fix(stream): fix the race condition during create new stream tasks. 2023-04-20 23:21:18 +08:00
Haojun Liao f9801ba9c5 enh(stream): stop stream asap. 2023-04-20 18:58:07 +08:00
Haojun Liao 742b5ee08c other: update wal logs level. 2023-04-20 13:42:16 +08:00
Haojun Liao c654f1145d enh(stream): add new msg for seek, and do some internal refactor. 2023-04-20 10:59:16 +08:00
wangmm0220 594d68b8a4 fix:remove lock for consume handler 2023-04-19 11:46:59 +08:00
wangmm0220 14fd2e7904 fix:remove lock for consume handler 2023-04-19 11:44:52 +08:00
wangmm0220 25542c26c7 fix:remove tmPushMsg in consumer 2023-04-19 10:43:13 +08:00
Haojun Liao 7a8dd0db45 refactor: update the log. 2023-04-18 16:08:51 +08:00
Haojun Liao 76becf59b7
Merge pull request #20947 from taosdata/feature/3_liaohj
fix(stream): set the correct initial checkpoint versions.
2023-04-18 11:26:48 +08:00
Haojun Liao 2ae05b8cf6 fix(stream): set the correct initial offset value. 2023-04-18 08:56:31 +08:00
liuyao 27d14f742d fix:stream memory leak 2023-04-17 17:27:02 +08:00
Haojun Liao be90d2c511 fix(stream): disable stream task when no tasks exist. 2023-04-16 23:07:54 +08:00
Haojun Liao 1350af5267 fix(stream): set the correct initial checkpoint version to restore the operators state and add check for the initial destination tables. 2023-04-16 22:48:22 +08:00
Haojun Liao ab4b08a4b5 Merge branch 'feature/3_liaohj' into enh/rocksRevert 2023-04-14 17:44:50 +08:00
Haojun Liao e0cb8aa534 fix(stream): don't the initial task status and do some internal refactor. 2023-04-14 17:43:37 +08:00
Haojun Liao eb0e1f848a fix(stream): remove unused tqreader, do some internal refactor, set the meta value for streamtask. 2023-04-14 12:00:09 +08:00
Haojun Liao b57c813681 other: merge feature/3_liaohj 2023-04-14 10:36:54 +08:00
Haojun Liao f8b672f09f other: merge main. 2023-04-14 09:07:18 +08:00
Haojun Liao 4d83118ff7 fix(stream): fix memory leak. 2023-04-13 23:49:14 +08:00
Haojun Liao 70b45a4cf0 fix(stream): all data should be extracted from wal. 2023-04-13 23:22:09 +08:00
Haojun Liao 5ab54481f0 refactor: do some internal refactor. 2023-04-11 19:24:34 +08:00
Haojun Liao f083697dba refactor(tq): do some internal refactor. 2023-04-11 19:04:33 +08:00
yihaoDeng b24b91886a Merge branch 'dev3' into enh/rocksRevert 2023-04-11 10:59:00 +00:00
Xiaoyu Wang 4c04b155d5 merge main 2023-04-11 15:56:28 +08:00
Haojun Liao 18479d8115 refactor: do some internall refactor. 2023-04-11 15:42:24 +08:00
Haojun Liao 15cceb5a5a refactor: do some internal refactors. 2023-04-11 11:53:10 +08:00
Haojun Liao a4ba5401f9 enh(stream): set the start version of all operators. 2023-04-11 10:01:36 +08:00
Haojun Liao 7419c0bfe9 enh(stream): the stream will start after vnode restore being completed. 2023-04-10 16:56:14 +08:00
Haojun Liao f5b52749c2 refactor: do some internal refactor. 2023-04-09 01:39:09 +08:00
Haojun Liao 871cfb4760
Merge pull request #20775 from taosdata/fix/TD-23450
fix:tq memory leak
2023-04-06 11:48:45 +08:00
Haojun Liao 2b2059ef18
Merge pull request #20534 from taosdata/fix/TD-22671
fix:error in TD-23218 & remove useless logic
2023-04-06 11:47:24 +08:00
yihaoDeng 5aff5aa93b add backpressure 2023-04-04 12:38:56 +00:00
54liuyao 519e2f1cfd fix:tq memory leak 2023-04-04 18:21:37 +08:00
yihaoDeng 628bb62c8a add backpressure 2023-04-04 07:20:05 +00:00
Xiaoyu Wang e9d7a099cc merge main 2023-04-04 14:50:58 +08:00
Haojun Liao d48be5c0ce refactor: do some internal refactor. 2023-04-03 18:23:35 +08:00
Haojun Liao a3ac8b647c refactor: do some internal refactor. 2023-04-03 18:20:35 +08:00
Haojun Liao 482c319bee refactor: do some internal refactor. 2023-04-03 18:07:28 +08:00
Haojun Liao 3ef7d43dae refactor: do some internal refactor. 2023-04-03 17:47:48 +08:00
wangmm0220 2d6436c57d fix:modify log & sleep 500ms if data is inserting while consume 2023-04-03 11:24:22 +08:00
wangmm0220 97b5c7abee fix:merge from main 2023-03-31 16:20:21 +08:00
Haojun Liao 5f4cb41e28 fix(tmq): set correct start offset value. 2023-03-29 23:45:41 +08:00
Haojun Liao 56c98d7768 refactor: do some internal refactor. 2023-03-29 19:35:04 +08:00
wangmm0220 2243184a4d fix:conflicts 2023-03-29 17:53:05 +08:00
wangmm0220 9d9ae749d1 opti:tmq logic & fix CI cases 2023-03-29 17:27:04 +08:00
wangmm0220 d0c55ef572 fix:conflicts 2023-03-28 18:12:13 +08:00
Haojun Liao a883e57cf2 Merge branch 'main' into fix/liaohj 2023-03-28 15:07:32 +08:00
Haojun Liao 68f310dd48 refactor: do some internal refactor. 2023-03-28 15:05:10 +08:00
Haojun Liao 4fd3064fd9 refactor: do some internal refactor. 2023-03-28 14:43:37 +08:00
Haojun Liao fca79dbc53 Merge branch 'fix/liaohj' into main 2023-03-28 11:20:01 +08:00
Haojun Liao c158087d66 fix(tmq): kill the ongoing tsdb scans while transferring the ownership of vnode to other consumers. 2023-03-28 09:53:49 +08:00
Haojun Liao 5c128e2292 refactor: do some internal refactor. 2023-03-28 09:27:48 +08:00
wangmm0220 f1d67b6059 fix:conflict 2023-03-25 19:03:23 +08:00
wangmm0220 750cd8c5a4 fix:ci error 2023-03-25 16:33:47 +08:00
wangmm0220 9f5e435140 fix:conflict from main 2023-03-25 11:49:37 +08:00
Haojun Liao b22cc82270 others: merge main. 2023-03-24 15:43:13 +08:00
wangmm0220 5d06474f2e fix:send data batch if consume wal where subscribe db 2023-03-24 14:43:29 +08:00
wangmm0220 bf78f63e22 fix:send data batch if consume wal where subscribe db 2023-03-24 10:10:52 +08:00
wangmm0220 5f9c7e9a7d fix:[TS-2961] consume error change from snapshot to wal for taosx 2023-03-22 20:14:19 +08:00
wangmm0220 6fae38619d fix:error in TD-23218 & remove useless logic 2023-03-22 19:36:12 +08:00
wangmm0220 dd4cf0597b fix:error in optimize consume logic 2023-03-22 17:04:02 +08:00
dapan1121 5b02af6a91 fix: merge main to 3.0 2023-03-22 16:08:16 +08:00
Haojun Liao 69ca2b2f7b fix(tmq): do some internal refactor. 2023-03-21 19:40:36 +08:00
Haojun Liao c3acab2070 fix(tmq): do some internal refactor. 2023-03-21 19:38:34 +08:00
Haojun Liao ca29dd344f other: merge main. 2023-03-21 14:04:52 +08:00
Haojun Liao 18b121c69c fix(tmq): fix the invalid write and do some internal refactor. 2023-03-21 13:45:43 +08:00
Haojun Liao ea0cc2c584
Merge pull request #20554 from taosdata/fix/TS-2828
fix: add retry if insert error with meta change
2023-03-21 09:01:22 +08:00
wangmm0220 f7b7320f5f fix:set offset if read wal none 2023-03-20 16:57:45 +08:00
wangmm0220 f80078030c fix:error in TD-23218 & remove useless logic 2023-03-20 15:40:01 +08:00
Haojun Liao 2517f9bb83 fix(tmq): add some logs. 2023-03-20 15:27:16 +08:00