Commit Graph

605 Commits

Author SHA1 Message Date
Haojun Liao d7101f7109 enh(stream): support dispatch checkpoint msg. 2023-07-06 18:26:22 +08:00
Haojun Liao 5a9461a69a other: merge 3.0 2023-07-06 11:52:21 +08:00
Haojun Liao f4a3567935 other: merge 3.0 2023-07-06 11:24:30 +08:00
Haojun Liao 03dbcbf055 refactor: do some internal refactor. 2023-07-05 15:55:55 +08:00
yihaoDeng 7d30a6e27a vnode snapshot read 2023-06-30 11:48:46 +00:00
Haojun Liao 57fcd553f8 refactor: do some internal refactor. 2023-06-28 13:13:13 +08:00
Haojun Liao a19e63fd23 fix(stream): fix error in handling fill history. 2023-06-27 23:00:14 +08:00
Haojun Liao b73444b291 fix(stream): start stream task in case of scan history completing. 2023-06-27 09:44:51 +08:00
Haojun Liao 402c091df0 fix(stream): fix sma error. 2023-06-25 19:41:44 +08:00
liuyao 41c193945f fill history pause&resume 2023-06-19 17:28:19 +08:00
yihaoDeng 0eeaab9cd3 Merge branch 'refact/fillhistory1' into refact/fillhistory 2023-06-19 13:53:37 +08:00
yihaoDeng b6fe5d917b refactor checkpoint 2023-06-17 14:05:49 +08:00
Haojun Liao 64365666b7 fix(stream): update the info, and do some internal refactor. 2023-06-16 16:04:43 +08:00
yihaoDeng 9f9171719a support fill history 2023-06-15 21:29:52 +08:00
liuyao 693942b821 trans state 2023-06-15 14:13:17 +08:00
Haojun Liao 53377c2c1f fix(stream): wait for stream task completed. 2023-06-15 01:14:41 +08:00
liuyao 9a3708e17b pause&resume fill history 2023-06-14 15:54:07 +08:00
Haojun Liao 7c6fbd7772 enh(stream): do some internal refactor and support secondary scan for history data. 2023-06-14 14:10:09 +08:00
Haojun Liao 11f0c3b336 refactor: do some internal refactor. 2023-06-14 10:46:46 +08:00
Haojun Liao b44447e6f3 enh(stream): support restore from disk. 2023-06-14 10:19:03 +08:00
Haojun Liao 97da2a8fac fix(stream): set the fill history status. 2023-06-13 16:10:55 +08:00
Haojun Liao fb24ed161d enh(stream): set correct task status. 2023-06-12 22:48:23 +08:00
Haojun Liao 78a240be8f enh(stream): add dispatch msg. 2023-06-12 13:12:45 +08:00
yihaoDeng a2030dc378 add trigger checkpoint 2023-06-09 12:12:51 +00:00
Haojun Liao 69c9eda7af fix(stream): fix race condition. 2023-06-09 11:37:38 +08:00
Haojun Liao 53246ed6bb other: merge other branch. 2023-06-09 09:41:10 +08:00
Haojun Liao 6c13491428 refactor: do some internal refactor. 2023-06-08 16:55:41 +08:00
yihaoDeng 8b91617145 Merge branch 'enh/dv3' into enh/triggerCheckpoint 2023-06-08 07:20:41 +00:00
yihaoDeng 77f287d9be add trigger checkpoint 2023-06-08 07:19:59 +00:00
Haojun Liao 30eba472c3
Merge pull request #21602 from taosdata/fix/TD-24600-dv3
fix/td-24600-dv3
2023-06-08 13:42:02 +08:00
Haojun Liao cb26dd9fa2 refactor(stream): do some internal refactor. 2023-06-08 10:17:21 +08:00
Haojun Liao 06cf3588da refactor: do some internal refactor. 2023-06-07 15:03:04 +08:00
liuyao 1e9c4d5fac opt stream block dispatch 2023-06-07 10:45:27 +08:00
Haojun Liao 6a3c935b4a fix(stream): fix error in fill history scan. 2023-06-07 09:15:55 +08:00
Haojun Liao 0dd933013c enh(stream): make history task for stream running. 2023-06-05 17:58:00 +08:00
Haojun Liao fcc706c45c enh(stream): refactor the fill history task. 2023-06-05 15:59:01 +08:00
yihaoDeng 74d5f02969 fix invalid free 2023-06-05 04:56:07 +00:00
Haojun Liao 077e1a1ed4 Merge branch '3.0' into refact/fillhistory 2023-06-05 11:41:13 +08:00
yihaoDeng 32c3cfd51a Avoid creating the same ID task multiple times 2023-06-01 11:13:27 +00:00
Haojun Liao eda06081ff enh(stream): refactor and serialize the attributes of history tasks. 2023-05-30 16:55:37 +08:00
Haojun Liao be8fd9e48b enh(stream): create additional task for history data processing. 2023-05-30 14:47:10 +08:00
Haojun Liao 3628888ecc refactor: do some internal refactor. 2023-05-29 18:02:45 +08:00
Haojun Liao f2887278e2 refactor: do some internal refactor. 2023-05-29 15:44:45 +08:00
Haojun Liao de44c9160c refactor: do some internal refactor. 2023-05-20 18:05:39 +08:00
Haojun Liao 6ded3284da refactor: do some internal refactor. 2023-05-20 11:37:17 +08:00
yihaoDeng 9380b22394 merge 3.0 2023-05-18 05:46:38 +00:00
Haojun Liao ffee5ebc61 Merge branch '3.0' into feature/3_liaohj 2023-05-17 17:47:44 +08:00
Haojun Liao 041ffa91bd other: merge main. 2023-05-17 15:42:15 +08:00
Haojun Liao e35d145734 fix(stream): fix memory leak. 2023-05-16 13:59:13 +08:00
yihaoDeng 9c0d2c87c6 Merge branch 'enh/3.0' into enh/rocksdbSstateMerge 2023-05-15 01:52:42 +00:00
Haojun Liao 8e15c64499 refactor: do some internal refactor. 2023-05-13 23:22:36 +08:00
Haojun Liao 236926b9ee other: merge enh/rocksdbstate 2023-05-13 19:49:28 +08:00
liuyao 8b7b804606 opt partition by tbname 2023-05-11 18:47:27 +08:00
yihaoDeng 4e8f5f5d32 factor code 2023-05-11 09:58:53 +00:00
yihaoDeng afaee9c386 factor code 2023-05-11 09:51:01 +00:00
yihaoDeng 4715aacd41 factor code 2023-05-11 09:44:40 +00:00
yihaoDeng f652f71d68 factor code 2023-05-11 06:37:57 +00:00
Haojun Liao dbd79cc80c refactor: do some internal refactor. 2023-05-10 22:22:01 +08:00
yihaoDeng 55014aa003 Merge branch 'enh/rocksdbSstate' into enh/rocksdbSstateMerge 2023-05-09 08:58:51 +00:00
yihaoDeng 08c3a63341 Merge branch 'enh/standVer' into enh/rocksdbSstateMerge 2023-05-06 12:37:54 +00:00
Haojun Liao da6212bab0 other: merge main. 2023-05-06 12:51:38 +08:00
Haojun Liao 8958aabe4b fix(stream): opt task scan performance. 2023-05-05 19:33:30 +08:00
yihaoDeng f7a7743af7 fix case failure 2023-05-05 08:32:33 +00:00
Haojun Liao 23c49594b7 other: merge main. 2023-05-04 16:41:53 +08:00
liuyao d8a1f15d10 fix resum bug 2023-05-04 09:28:06 +08:00
Haojun Liao 3c61932ecc refactor: do some internal refactor. 2023-05-01 19:28:54 +08:00
yihaoDeng dda1c8803c Merge branch 'enh/rocksdbSstate' into enh/rocksdbSstateMerge 2023-04-28 09:17:22 +00:00
yihaoDeng 36a4413bf1 merge rocksdb inst 2023-04-27 14:38:33 +00:00
Haojun Liao b7040ed15e enh(stream): support resume from the latest data, and do some internal refactor. 2023-04-27 15:07:54 +08:00
liuyao 641c707b37 pause&resume 2023-04-23 17:37:10 +08:00
54liuyao 5089528a02 feat:stream pause&resume 2023-04-23 15:08:51 +08:00
yihaoDeng 917b07348c Merge branch 'fix/rocksdbState' into enh/rocksRevert 2023-04-23 02:37:26 +00:00
Haojun Liao 5e6c06e253 fix(stream): fix the race condition during create new stream tasks. 2023-04-20 23:21:18 +08:00
Haojun Liao fb32ffd960 fix(stream): update the reference count value to be int32, insead of int8 2023-04-18 18:06:03 +08:00
Haojun Liao 1f3c8c3d80 fix(stream): update the ref type. 2023-04-18 17:15:31 +08:00
Haojun Liao ac137b4b33 refactor: do some internal refactor. 2023-04-17 09:46:58 +08:00
Haojun Liao be90d2c511 fix(stream): disable stream task when no tasks exist. 2023-04-16 23:07:54 +08:00
Haojun Liao ab4b08a4b5 Merge branch 'feature/3_liaohj' into enh/rocksRevert 2023-04-14 17:44:50 +08:00
Haojun Liao e0cb8aa534 fix(stream): don't the initial task status and do some internal refactor. 2023-04-14 17:43:37 +08:00
Haojun Liao eb0e1f848a fix(stream): remove unused tqreader, do some internal refactor, set the meta value for streamtask. 2023-04-14 12:00:09 +08:00
Haojun Liao b57c813681 other: merge feature/3_liaohj 2023-04-14 10:36:54 +08:00
Haojun Liao 70b45a4cf0 fix(stream): all data should be extracted from wal. 2023-04-13 23:22:09 +08:00
Haojun Liao 5ab54481f0 refactor: do some internal refactor. 2023-04-11 19:24:34 +08:00
yihaoDeng b24b91886a Merge branch 'dev3' into enh/rocksRevert 2023-04-11 10:59:00 +00:00
Haojun Liao 18479d8115 refactor: do some internall refactor. 2023-04-11 15:42:24 +08:00
Haojun Liao 15cceb5a5a refactor: do some internal refactors. 2023-04-11 11:53:10 +08:00
Haojun Liao a4ba5401f9 enh(stream): set the start version of all operators. 2023-04-11 10:01:36 +08:00
Haojun Liao 7419c0bfe9 enh(stream): the stream will start after vnode restore being completed. 2023-04-10 16:56:14 +08:00
Haojun Liao f5b52749c2 refactor: do some internal refactor. 2023-04-09 01:39:09 +08:00
yihaoDeng 628bb62c8a add backpressure 2023-04-04 07:20:05 +00:00
Haojun Liao a3ac8b647c refactor: do some internal refactor. 2023-04-03 18:20:35 +08:00
Haojun Liao 482c319bee refactor: do some internal refactor. 2023-04-03 18:07:28 +08:00
Haojun Liao 3ef7d43dae refactor: do some internal refactor. 2023-04-03 17:47:48 +08:00
Haojun Liao 9992942a7a
Merge pull request #20658 from taosdata/feat/TD-22418
feat:coverity scan
2023-03-29 10:44:53 +08:00
Haojun Liao 68f310dd48 refactor: do some internal refactor. 2023-03-28 15:05:10 +08:00
54liuyao cf2d169f75 feat:coverity scan 2023-03-27 16:17:49 +08:00
54liuyao 1d58f85a54 feat:coverity scan 2023-02-22 10:10:04 +08:00
Xiaoyu Wang c597de099f merge main 2023-02-20 13:41:27 +08:00
Liu Jicong 2a4708d82a fix: save fill history state 2023-02-16 17:01:09 +08:00
Xiaoyu Wang eb86f5ae71 merge main 2023-01-28 18:10:08 +08:00
Xiaoyu Wang 24157e78d6 merge main 2023-01-28 17:52:58 +08:00
Liu Jicong 9913ed1dd1 fix: concurrency issue 2023-01-12 14:28:34 +08:00
Liu Jicong bfb92457fa refactor: stream dispatch 2022-12-29 16:24:30 +08:00
Shengliang Guan d342ebcd77
Merge pull request #18720 from taosdata/refact/submit_req
Refact/submit req
2022-12-28 10:29:00 +08:00
Liu Jicong dcb71e4af1 refactor: stream dispatch 2022-12-27 15:17:43 +08:00
Liu Jicong 74a253eac4 refactor 2022-12-08 13:34:42 +08:00
Shengliang Guan 6d94afe48f
Revert "refact: adjust some assert cases" 2022-12-07 21:51:17 +08:00
Shengliang Guan d5126d469a refact: replcase ASSERT with tAssert 2022-12-07 18:42:48 +08:00
Liu Jicong b133a56219 refactor: support submitreq2 2022-12-07 15:19:34 +08:00
Minglei Jin 6cb65f336d Merge branch 'v3.0' into feat/async_commit 2022-12-02 14:49:58 +08:00
Liu Jicong 3424b184d3 refactor(stream): distributed checkpoint 2022-12-02 03:48:45 +08:00
Minglei Jin 506554b5d2 Merge branch 'v3.0' into fix/tdb-async-commit 2022-11-28 16:20:48 +08:00
Liu Jicong 48894c8bdb enh(stream): new api for stream queue 2022-11-25 18:03:40 +08:00
Minglei Jin 78335f83d2 tdb/api: migrate txn related api changes of meta, sma, tq, and stream meta/state 2022-11-17 20:02:13 +08:00
Liu Jicong a9bf05bc01 refactor: stream meta ref count 2022-11-15 19:11:42 +08:00
Liu Jicong bc469f7f7b fix(stream): delete tb should be checked in write thread 2022-11-08 11:30:29 +08:00
Liu Jicong 2fbc8e6306 refactor(stream): stream deploy and state transfer 2022-11-03 13:22:23 +08:00
Liu Jicong 2e640c38a2 refactor: stream and tmq message 2022-10-31 18:37:28 +08:00
Liu Jicong 8d8fd2b2bc refactor(stream): recover and fill history 2022-10-26 10:20:33 +08:00
Liu Jicong 75e5e490fd refactor(stream): recover and fill history 2022-10-26 00:31:00 +08:00
Liu Jicong e0b5b9263e refactor: remove unused code 2022-10-21 10:42:24 +08:00
Liu Jicong 035e6b13c8 enh(stream): internal optimize 2022-10-21 10:29:05 +08:00
Liu Jicong 72fba5a8ce fix(stream): check delete uid 2022-09-21 11:54:44 +08:00
Liu Jicong 4c6a8bbc95 feat(stream): stream state support tuple 2022-09-14 16:50:42 +08:00
54liuyao 1f5d9a38e7 feat(stream):change single interval disc buff and interval delete 2022-09-07 14:48:14 +08:00
Liu Jicong 24261cc90b refactor(stream): refine stream backend interface 2022-08-24 15:53:29 +08:00
Liu Jicong 2d628491b2 feat(stream): support tdb state backend 2022-08-23 19:30:04 +08:00
Liu Jicong 963287798d refactor(mnode): drop stream task 2022-08-17 23:18:39 +08:00
Liu Jicong 853e6e2988 refactor(mnode): drop stream task 2022-08-17 19:19:58 +08:00
Liu Jicong 357b21cfeb refactor(tmq): use tdb to store check info 2022-08-17 13:58:56 +08:00
Liu Jicong 6f75e03a55 refactor(stream): remove unneccessary param in converting data 2022-08-15 11:06:53 +08:00
Liu Jicong b8a3654cd6 enh(stream): show stream 2022-08-11 13:37:32 +08:00
Liu Jicong 18f6fb6123 enh(tmq): speed up consumer recover 2022-08-10 16:04:03 +08:00
Liu Jicong fc225a580d enh(stream): recover process 2022-08-09 21:37:21 +08:00
Liu Jicong d9a4ea305d fix(stream): memory error 2022-08-09 14:11:41 +08:00
Liu Jicong ad032a0a90 enh(stream): stream recover 2022-08-09 10:36:27 +08:00
Liu Jicong c411b9aebb fix(stream): memory leak 2022-08-08 18:09:36 +08:00
Liu Jicong bb0ce39511 refactor(stream): add stream meta 2022-08-08 15:05:54 +08:00
Liu Jicong 43026ac31d refactor(stream) 2022-08-08 11:29:50 +08:00
Liu Jicong 7c9d57bd89 refactor(stream): unify sink and dispatch 2022-08-05 20:16:47 +08:00
Liu Jicong f0f4748320 fix(util): concurrently hash iterate 2022-08-05 16:14:19 +08:00
Liu Jicong 6f8683de7e enh(tmq): retry consumer close 2022-08-04 20:37:02 +08:00
Liu Jicong e64e41679e fix schedule 2022-08-04 17:00:45 +08:00
Liu Jicong d6cc88ed4b refactor(stream): exec scheduler 2022-08-04 16:19:53 +08:00
Liu Jicong d43eec5d16 enh(stream): directly dispatch 2022-08-02 22:23:33 +08:00
Liu Jicong a9d11b58ad enh(stream): recover 2022-07-28 16:19:50 +08:00
Liu Jicong 00acf4520c refactor(stream): remove option 2022-07-27 16:49:03 +08:00
Liu Jicong 3fa5791889 fix(stream): dispatch 2022-07-27 10:52:25 +08:00
Liu Jicong 81d8d2b937 feat(stream): add version in stream block 2022-07-21 17:27:44 +08:00
Liu Jicong 414673db57
Merge branch '3.0' into feature/stream 2022-07-19 19:46:36 +08:00
54liuyao 8e41fc912b feat(stream): add num of children 2022-07-19 14:17:53 +08:00
Liu Jicong b6f55fa09e refactor(stream): batch optimization for submit msg 2022-07-18 21:32:40 +08:00
Liu Jicong 500553d581 refactor: add debug log 2022-07-18 11:41:56 +08:00
Liu Jicong 280ed4f65d feat(stream): pipelined exec for recovering 2022-07-15 21:29:53 +08:00
Liu Jicong 9394e3380f feat(stream): recover from failure 2022-07-15 17:58:53 +08:00
Haojun Liao 75aca0b031 fix(query): set correct fill output column index, fix some memory leak, and do some internal refactor, 2022-07-14 13:53:23 +08:00
Liu Jicong a717042acf refactor(stream): simple batch 2022-07-13 17:24:36 +08:00
Liu Jicong 8a6e4b877a refactor(stream): simple batch 2022-07-13 16:37:33 +08:00
Liu Jicong d637ffe22c refactor(tmq): prepare only needed 2022-07-12 14:10:22 +08:00
Liu Jicong 866e4c4be0 refactor(stream): internal refactor 2022-07-07 17:56:52 +08:00
Liu Jicong 8c5dbde918 refactor(stream): stream reader created in scanner 2022-07-06 14:20:07 +08:00
Liu Jicong c2955a807b feat(tmq): offset support snapshot 2022-06-29 21:04:51 +08:00
Liu Jicong 383e7a495b fix(query): enable output check 2022-06-28 13:45:50 +08:00
54liuyao 10e90ce973 feat(stream): auto pull data 2022-06-25 16:44:14 +08:00
Liu Jicong 81ed902d7f fix(stream): stream rsp to retrieve msg if data is empty 2022-06-25 14:03:06 +08:00
Liu Jicong f2f5d68f3b fix(stream): memory leak 2022-06-23 22:05:00 +08:00
Liu Jicong 14a488c92c feat(stream): stream support multiple type input 2022-06-22 21:21:39 +08:00
Liu Jicong fb596625d3 feat(stream): support retrieve 2022-06-22 19:12:31 +08:00
Liu Jicong c409760f75 feat(stream): drop stream 2022-06-21 11:32:36 +08:00
Liu Jicong 450675f672 feat(stream): support snode 2022-06-20 20:18:34 +08:00
Liu Jicong 53ef66961a feat(stream): support snode 2022-06-20 14:29:18 +08:00
Liu Jicong 8eb08fa0d4 feat(stream): trigger mode add max delay 2022-06-15 16:11:20 +08:00
Liu Jicong 73b8f02ac7 fix(stream): set child id 2022-06-10 12:00:38 +08:00
Liu Jicong 7ab3a1e42c feat(wal): support restore from snapshot 2022-06-09 23:08:06 +08:00
Liu Jicong 9e4b3adc7c fix(stream): msg dispatch 2022-06-09 11:45:06 +08:00
Liu Jicong cbbf5f5d4d refactor(stream) 2022-06-07 21:35:49 +08:00
Liu Jicong 46bea47b62 enh(stream): direct sink if not dispatch 2022-06-07 19:58:06 +08:00
Liu Jicong 65900c32c3 enh(stream): use same name rule with schemaless 2022-06-07 15:40:11 +08:00
Liu Jicong 858868d76a refactor(stream): distributed execution 2022-06-07 10:54:18 +08:00
Liu Jicong 1b26da1bae refactor(stream): distributed execution 2022-06-06 22:18:50 +08:00
Liu Jicong 2677a3c8d9 refactor(stream) 2022-06-06 20:15:36 +08:00
Liu Jicong f3b680f630 refactor(tmq): push mode 2022-06-01 22:22:14 +08:00
Liu Jicong e1a5000e50 refactor(tmq): tq sink and push 2022-06-01 17:26:15 +08:00
Liu Jicong d3a2e98d60 fix(stream): double free 2022-05-26 20:52:27 +08:00
Liu Jicong a0f3138888 fix(stream): tb name error 2022-05-26 11:16:35 +08:00
Liu Jicong 79e97271ca fix(stream): memory error 2022-05-25 18:43:25 +08:00
Liu Jicong 5c5d9ed7a2 fix: windows compile 2022-05-23 17:08:07 +08:00
Liu Jicong a393dee9d2 refactor(stream) 2022-05-23 16:51:12 +08:00
Liu Jicong 7391a756f6 refactor(stream) 2022-05-20 18:53:21 +08:00
Liu Jicong 71984b4268 refactor(stream): exec 2022-05-20 05:43:36 +08:00
Liu Jicong f77f910264 refactor(stream) 2022-05-16 02:11:47 +08:00
Liu Jicong a6015d2f39 refactor(stream) 2022-05-15 08:14:23 +08:00
Liu Jicong a6b258a45f feat(tmq): change default config 2022-05-10 17:12:11 +08:00
Liu Jicong d17b478e96 feat(stream): auto create ctb 2022-05-08 02:37:23 +08:00
Liu Jicong 3ed1ab6acb enh(stream): auto create ctb 2022-05-07 23:19:05 +08:00
Hongze Cheng 323ce214b6 refact encode APIs 2022-05-07 10:03:06 +00:00
Liu Jicong a5706ea7e5 feat(stream): insert data 2022-05-05 18:57:09 +08:00
Liu Jicong 3ae3fd65e8 fix: memory error 2022-04-28 16:31:35 +08:00
Liu Jicong 16f99d9916 stream support fix ep sink 2022-03-29 10:40:13 +08:00
Liu Jicong b27fa6c8e3 stream add shuffle dispatcher 2022-03-28 17:22:18 +08:00
Liu Jicong a3b4de6631 stream support sma 2022-03-28 14:40:22 +08:00
Liu Jicong e0f192046c multiple level stream schedule 2022-03-28 11:47:39 +08:00
Liu Jicong 564236f230 refactor stream 2022-03-26 14:50:26 +08:00
Liu Jicong 03b2162309 refactor stream 2022-03-25 21:03:12 +08:00
Liu Jicong 59a4eec186 stream extract to module 2022-03-25 20:19:09 +08:00