Commit Graph

703 Commits

Author SHA1 Message Date
Haojun Liao 8277d608f4 other: update logs. 2023-05-11 16:53:19 +08:00
yihaoDeng e6474caddf factor code 2023-05-11 08:51:41 +00:00
Haojun Liao b2d141fe40 fix(stream): limit the batch size. 2023-05-11 15:07:07 +08:00
yihaoDeng f652f71d68 factor code 2023-05-11 06:37:57 +00:00
Haojun Liao 2b736ffd60 fix(stream): avoid duplicated task id in task id list. 2023-05-11 11:47:57 +08:00
Haojun Liao b02b3117b5 refactor: do some internal refactor. 2023-05-11 10:14:29 +08:00
yihaoDeng d473fcf6b0 refactor code 2023-05-11 01:19:54 +00:00
yihaoDeng c7602a4c1a refactor code 2023-05-11 01:19:40 +00:00
Haojun Liao dbd79cc80c refactor: do some internal refactor. 2023-05-10 22:22:01 +08:00
yihaoDeng b4bd6a4f1c refactor code 2023-05-10 10:49:10 +00:00
Yihao Deng c6e5879f44
Merge branch 'main' into enh/rocksdbSstate 2023-05-10 10:04:28 +08:00
liuyao 62ccd4bf16 avoid delete data 2023-05-10 09:11:04 +08:00
yihaoDeng 8356533eb1 Merge branch 'enh/standVer' into enh/rocksdbSstateMerge 2023-05-09 12:13:16 +00:00
liuyao 8224d49702 fix:after restarting taosd, stream does not work. 2023-05-09 17:24:11 +08:00
yihaoDeng 55014aa003 Merge branch 'enh/rocksdbSstate' into enh/rocksdbSstateMerge 2023-05-09 08:58:51 +00:00
yihaoDeng b5fdaa327d avoid deadlock 2023-05-09 07:00:29 +00:00
Yihao Deng cb223acf82
Merge branch 'main' into enh/rocksdbSstate 2023-05-08 09:45:27 +08:00
Haojun Liao 31ac1e3eaa fix(stream): revise the input queue capacity. 2023-05-08 00:00:55 +08:00
Haojun Liao d026e7ef16 refactor: update some logs. 2023-05-07 22:55:49 +08:00
yihaoDeng 163a862f5f fix mem leak 2023-05-06 13:57:33 +00:00
yihaoDeng 08c3a63341 Merge branch 'enh/standVer' into enh/rocksdbSstateMerge 2023-05-06 12:37:54 +00:00
Haojun Liao c141abfbb8 fix(stream): set correct msg size; 2023-05-06 20:03:07 +08:00
Haojun Liao 13f3ca4226 fix(stream): add input queue size limitation. 2023-05-06 18:47:29 +08:00
yihaoDeng 8be29dc98e failed to open rocksdb cf 2023-05-06 10:35:23 +00:00
Haojun Liao aca5760ceb fix(query): fix syntax error. 2023-05-06 13:17:31 +08:00
Haojun Liao da6212bab0 other: merge main. 2023-05-06 12:51:38 +08:00
Haojun Liao 1fa4327f6c Merge branch 'main' into fix/liaohj_main 2023-05-06 10:15:23 +08:00
liuyao 76bf0aea56 opt stream input queue 2023-05-06 10:14:27 +08:00
Haojun Liao 3afc5bdfcf refactor: add some logs. 2023-05-06 10:09:51 +08:00
Haojun Liao 29fcd1b83a refactor: do some internal refactor. 2023-05-06 10:08:05 +08:00
Haojun Liao 5d4a3116d5
Merge pull request #21139 from taosdata/fix/liaohj_main
refactor: do some internal refactor.
2023-05-06 09:49:26 +08:00
yihaoDeng d7e86aca05 fix case failure 2023-05-05 15:14:16 +00:00
Haojun Liao 8958aabe4b fix(stream): opt task scan performance. 2023-05-05 19:33:30 +08:00
yihaoDeng f7a7743af7 fix case failure 2023-05-05 08:32:33 +00:00
Haojun Liao 8b12d4d3da fix(stream): secure the delete task operation. TD-1198 2023-05-05 15:02:03 +08:00
yihaoDeng a1b9dcf595 fix mem leak 2023-05-05 02:20:11 +00:00
liuyao 74ab689719 opt batch num 2023-05-05 08:59:07 +08:00
yihaoDeng f20b28bc9f fix mem leak 2023-05-04 14:29:56 +00:00
yihaoDeng 6b0bda6708 fix mem leak 2023-05-04 13:38:19 +00:00
yihaoDeng c2de8c91dd fix mem leak 2023-05-04 11:52:29 +00:00
yihaoDeng f9c49f4938 add compile opt on windows 2023-05-04 10:22:13 +00:00
Haojun Liao 2b7a816b86 fix(tmq): fix syntax error. 2023-05-04 16:59:26 +08:00
Haojun Liao 23c49594b7 other: merge main. 2023-05-04 16:41:53 +08:00
yihaoDeng c27d01a510 Merge branch 'enh/addTtlToStream' into enh/rocksdbSstateMerge 2023-05-04 08:22:31 +00:00
Haojun Liao 09a04052ec refactor: do some internal refactor. 2023-05-04 16:15:14 +08:00
liuyao d8a1f15d10 fix resum bug 2023-05-04 09:28:06 +08:00
Haojun Liao 3b814f2478 fix(stream): do some internal refactor. 2023-05-01 23:21:29 +08:00
Haojun Liao a1263b8b04 fix(query):fix syntax error. 2023-05-01 19:40:26 +08:00
Haojun Liao 3c61932ecc refactor: do some internal refactor. 2023-05-01 19:28:54 +08:00
Haojun Liao 6c86847b12 refactor: do some internal refactor. 2023-05-01 14:58:59 +08:00
Haojun Liao 777ed1761f refactor: update logs. 2023-04-29 18:24:46 +08:00
Haojun Liao bbb8aaa0dc refactor: add some logs. 2023-04-29 02:37:10 +08:00
Haojun Liao b69137df44 refactor(log): add some logs. 2023-04-29 00:00:25 +08:00
Haojun Liao c647ded0da fix(stream): add some logs. 2023-04-28 19:47:49 +08:00
yihaoDeng dda1c8803c Merge branch 'enh/rocksdbSstate' into enh/rocksdbSstateMerge 2023-04-28 09:17:22 +00:00
yihaoDeng afc6023a8a merge rocksdb inst 2023-04-28 06:32:33 +00:00
Haojun Liao b7c8346b44 other: merge other branch. 2023-04-28 13:06:47 +08:00
Haojun Liao f9a64cbcd6 other: add some logs. 2023-04-28 12:58:03 +08:00
yihaoDeng c4f36d32ae merge rocksdb inst 2023-04-28 01:31:37 +00:00
yihaoDeng 984a69671e merge rocksdb inst 2023-04-27 14:46:33 +00:00
yihaoDeng 36a4413bf1 merge rocksdb inst 2023-04-27 14:38:33 +00:00
jiajingbin ebf9a13fe7 inc batch num 2023-04-27 18:15:17 +08:00
Haojun Liao 6780833cd2 Merge remote-tracking branch 'origin/enh/rocksdbSstate' into enh/rocksdbSstate 2023-04-27 15:08:17 +08:00
Haojun Liao b7040ed15e enh(stream): support resume from the latest data, and do some internal refactor. 2023-04-27 15:07:54 +08:00
liuyao 8b5251004f inc queue cap 2023-04-27 14:04:46 +08:00
liuyao dbdc31539c inc queue cap 2023-04-27 10:46:08 +08:00
liuyao 4c4a4775de opt stream 2023-04-26 19:13:58 +08:00
liuyao 863d30e5a9 opt tq read 2023-04-26 18:58:19 +08:00
liuyao 2533b5c5c2 op stream 2023-04-26 17:00:33 +08:00
yihaoDeng 10582fb5e0 opt write opt 2023-04-25 09:49:28 +00:00
yihaoDeng 84cae91f9f rocksdb_iter_value 2023-04-25 09:39:57 +00:00
yihaoDeng c64a7d3e5f add ttl to stream state key 2023-04-25 08:54:03 +00:00
yihaoDeng 05df0f6ed7 add ttl to stream state key 2023-04-25 02:36:44 +00:00
yihaoDeng 81bd7661bd add ttl to stream state key 2023-04-24 14:20:01 +00:00
yihaoDeng f15b806e15 opt write opt 2023-04-24 09:08:48 +00:00
yihaoDeng e1535124d1 opt write opt 2023-04-24 09:08:07 +00:00
yihaoDeng 23803c04ed opt write opt 2023-04-24 08:49:56 +00:00
yihaoDeng 2d5218246d add ttl to stream state key 2023-04-24 08:18:02 +00:00
yihaoDeng 3abdbec813 add ttl to stream state key 2023-04-24 08:17:57 +00:00
liuyao eca7b68fa0 ignore update 2023-04-24 15:01:56 +08:00
Haojun Liao 48c36a8c35 Merge branch 'main' into enh/rocksdbSstate 2023-04-24 14:44:13 +08:00
yihaoDeng ae3730c318 add ttl to stream state key 2023-04-23 13:29:18 +00:00
Haojun Liao 623ea6df5f fix(stream): add task status check. 2023-04-23 17:42:56 +08:00
yihaoDeng 917b07348c Merge branch 'fix/rocksdbState' into enh/rocksRevert 2023-04-23 02:37:26 +00:00
yihaoDeng 54f3bc2543 change write opt 2023-04-21 10:06:37 +00:00
liuyao 426a794e7e add ini 2023-04-21 17:47:00 +08:00
yihaoDeng c71bbb0109 change write opt 2023-04-21 06:24:46 +00:00
yihaoDeng 98d6ff5a5d change write opt 2023-04-21 05:55:10 +00:00
yihaoDeng 57c4544bf0 change write opt 2023-04-21 05:54:52 +00:00
Haojun Liao b6f4cac619 fix(stream): set the correct offset version. 2023-04-21 13:35:24 +08:00
Haojun Liao 6c641cff39 fix(stream): fix error in start stream tasks. 2023-04-21 09:37:16 +08:00
Haojun Liao 5e6c06e253 fix(stream): fix the race condition during create new stream tasks. 2023-04-20 23:21:18 +08:00
Haojun Liao dc733352db enh(stream): set the max input queue size to be 3000. 2023-04-20 17:38:10 +08:00
Haojun Liao 8323ad8670 enh(stream): add more check to stop stream asap. 2023-04-20 17:29:08 +08:00
Haojun Liao b60b1796f7 fix(stream): add lock during check wal to create new stream task. 2023-04-20 15:53:41 +08:00
yihaoDeng 3272d14e33 change write opt 2023-04-20 04:05:35 +00:00
yihaoDeng aa613285e7 change write opt 2023-04-20 00:21:16 +00:00
yihaoDeng 9d6e8b3fa7 change write opt 2023-04-20 00:18:57 +00:00
yihaoDeng b640a0330d change write opt 2023-04-19 23:55:09 +00:00
yihaoDeng 6ff0b09e5e change write opt 2023-04-19 23:46:08 +00:00
yihaoDeng 22afdaa6a4 change write opt 2023-04-19 14:15:17 +00:00
yihaoDeng 10e44c63e1 change write opt 2023-04-19 13:12:53 +00:00
yihaoDeng 5efa5f6b9f change write opt 2023-04-19 13:01:13 +00:00
54liuyao 5b7ccc42ca inc sleep time 2023-04-19 13:38:30 +08:00
liuyao ae117bd7c3 fill state 2023-04-19 09:15:12 +08:00
Haojun Liao 9f16802451 fix(stream): set the initial value. 2023-04-18 16:42:40 +08:00
Haojun Liao 76becf59b7
Merge pull request #20947 from taosdata/feature/3_liaohj
fix(stream): set the correct initial checkpoint versions.
2023-04-18 11:26:48 +08:00
liuyao 5460eaf29a memory leak 2023-04-18 09:50:13 +08:00
yihaoDeng 47bf2374db fix compile error 2023-04-17 15:19:33 +00:00
yihaoDeng fc6d9afa19 fix compile error 2023-04-17 15:01:47 +00:00
yihaoDeng 6f6fd3e147 fix compile error 2023-04-17 15:01:41 +00:00
Haojun Liao 930b267a75 fix(stream): set the correct start offset for stream task. 2023-04-17 22:59:00 +08:00
liuyao 27d14f742d fix:stream memory leak 2023-04-17 17:27:02 +08:00
liuyao afc8e5cea7 fix asan bug 2023-04-17 11:48:08 +08:00
Haojun Liao ac137b4b33 refactor: do some internal refactor. 2023-04-17 09:46:58 +08:00
Haojun Liao 10b3fd9426 other: merge main. 2023-04-17 09:19:48 +08:00
Haojun Liao dc78233f86 Merge branch 'enh/rocksRevert' of github.com:taosdata/tdengine into enh/rocksRevert 2023-04-15 09:36:15 +08:00
Haojun Liao 04e042f21d Merge branch 'feature/3_liaohj' into enh/rocksRevert 2023-04-15 09:35:43 +08:00
Haojun Liao a3b02a80c0 fix(stream): update the version when open stream tasks. 2023-04-15 00:10:08 +08:00
Haojun Liao 3aa6897738 refactor: increase the buffer size 2023-04-14 22:56:11 +08:00
Haojun Liao e05d61456a fix(stream): disable the status check. 2023-04-14 22:45:22 +08:00
yihaoDeng 964db517dc Merge branch 'enh/rocksRevert' of https://github.com/taosdata/TDengine into enh/rocksRevert 2023-04-14 14:06:13 +00:00
yihaoDeng 80f348de1d fix recover 2023-04-14 14:05:59 +00:00
Haojun Liao d420734971 Merge remote-tracking branch 'origin/enh/rocksRevert' into enh/rocksRevert 2023-04-14 18:03:19 +08:00
Haojun Liao d0c706ac7c ehn(stream): increase the capacity of input queue to be 2000 items. 2023-04-14 18:02:41 +08:00
Haojun Liao ab4b08a4b5 Merge branch 'feature/3_liaohj' into enh/rocksRevert 2023-04-14 17:44:50 +08:00
Haojun Liao e0cb8aa534 fix(stream): don't the initial task status and do some internal refactor. 2023-04-14 17:43:37 +08:00
yihaoDeng efa7b48c1b Merge branch 'enh/rocksRevert' of https://github.com/taosdata/TDengine into enh/rocksRevert 2023-04-14 09:40:25 +00:00
yihaoDeng 4c70cfdfad startGroupTableMe 2023-04-14 09:36:08 +00:00
liuyao b8ac811f20 fix bug 2023-04-14 17:35:17 +08:00
Haojun Liao eb0e1f848a fix(stream): remove unused tqreader, do some internal refactor, set the meta value for streamtask. 2023-04-14 12:00:09 +08:00
Haojun Liao 5850a3ab01 Merge remote-tracking branch 'origin/enh/rocksRevert' into enh/rocksRevert 2023-04-14 10:37:17 +08:00
Haojun Liao b57c813681 other: merge feature/3_liaohj 2023-04-14 10:36:54 +08:00
yihaoDeng 1b25523153 Merge branch 'enh/rocksRevert' of https://github.com/taosdata/TDengine into enh/rocksRevert 2023-04-14 02:11:07 +00:00
yihaoDeng c0ebdb92d3 startGroupTableMe 2023-04-14 02:10:33 +00:00
Haojun Liao f8b672f09f other: merge main. 2023-04-14 09:07:18 +08:00
Haojun Liao 4d83118ff7 fix(stream): fix memory leak. 2023-04-13 23:49:14 +08:00
Haojun Liao 70b45a4cf0 fix(stream): all data should be extracted from wal. 2023-04-13 23:22:09 +08:00
yihaoDeng 36eef2fdef refactor code 2023-04-13 09:57:04 +00:00
liuyao 34bf638f19 fix bug 2023-04-13 16:21:57 +08:00
liuyao 9be89ce00a delete expired check point 2023-04-12 17:33:58 +08:00
yihaoDeng 4625dd98f0 fix write crash 2023-04-12 07:39:54 +00:00
yihaoDeng d0246a80cf Merge branch 'enh/rocksRevert' of https://github.com/taosdata/TDengine into enh/rocksRevert 2023-04-12 05:50:22 +00:00
yihaoDeng 7e4eefcde1 fix write crash 2023-04-12 05:50:01 +00:00
liuyao 819154e1fc use delete_mark to drop check point 2023-04-12 13:47:54 +08:00
yihaoDeng ed4aaade8c add backend 2023-04-12 03:03:51 +00:00
yihaoDeng 7db12bb317 fix invalid free 2023-04-11 14:21:16 +00:00
yihaoDeng a8ffb82370 fix invalid free 2023-04-11 14:13:35 +00:00
yihaoDeng 29d80b1c76 fix invalid free 2023-04-11 14:04:39 +00:00
yihaoDeng a3836b2363 fix invalid free 2023-04-11 12:52:03 +00:00
yihaoDeng 743ed82daa fix invalid free 2023-04-11 11:55:47 +00:00
Haojun Liao 5ab54481f0 refactor: do some internal refactor. 2023-04-11 19:24:34 +08:00
yihaoDeng b24b91886a Merge branch 'dev3' into enh/rocksRevert 2023-04-11 10:59:00 +00:00
yihaoDeng ff8cb09f63 refactor backend 2023-04-11 10:47:15 +00:00
Xiaoyu Wang 4c04b155d5 merge main 2023-04-11 15:56:28 +08:00
Haojun Liao 18479d8115 refactor: do some internall refactor. 2023-04-11 15:42:24 +08:00
liuyao 95d81604c7 feat:encode&&decode stream scan op 2023-04-11 14:34:31 +08:00
Haojun Liao 15cceb5a5a refactor: do some internal refactors. 2023-04-11 11:53:10 +08:00
liuyao 74da3c05bc feat:set check point id 2023-04-11 10:33:10 +08:00
Haojun Liao a4ba5401f9 enh(stream): set the start version of all operators. 2023-04-11 10:01:36 +08:00
Haojun Liao 7419c0bfe9 enh(stream): the stream will start after vnode restore being completed. 2023-04-10 16:56:14 +08:00
yihaoDeng eab6f84b9b Merge branch 'enh/rocksRevert' of https://github.com/taosdata/TDengine into enh/rocksRevert 2023-04-10 04:52:15 +00:00
yihaoDeng 1416da1967 opt write snapshot 2023-04-10 04:51:00 +00:00
Haojun Liao f5b52749c2 refactor: do some internal refactor. 2023-04-09 01:39:09 +08:00
liuyao fe9ca49d3f feat:optimize get patitionby name 2023-04-07 18:47:57 +08:00
liuyao 955f8f9f07 feat:optimize get patitionby name 2023-04-07 18:45:09 +08:00
yihaoDeng 1e3569367a opt write snapshot 2023-04-07 07:38:02 +00:00
yihaoDeng c082536242 opt write snapshot 2023-04-07 07:26:15 +00:00
liuyao 1f63859e21 feat:add buff swap 2023-04-07 12:50:59 +08:00
liuyao ad16487180 feat:add checkpoint 2023-04-06 18:26:47 +08:00
liuyao 0716a4aee8 feat:add check point 2023-04-06 16:45:22 +08:00
54liuyao 1892b806b5 feat:add stream state buff 2023-04-04 17:09:35 +08:00
54liuyao 429b5cd646 feat:add stream file state 2023-04-04 17:09:35 +08:00
Haojun Liao bd4c33d1c2 fix(tmq):fix the invalid read. 2023-04-04 15:57:50 +08:00
yihaoDeng 628bb62c8a add backpressure 2023-04-04 07:20:05 +00:00
Xiaoyu Wang e9d7a099cc merge main 2023-04-04 14:50:58 +08:00
Haojun Liao d48be5c0ce refactor: do some internal refactor. 2023-04-03 18:23:35 +08:00
Haojun Liao a3ac8b647c refactor: do some internal refactor. 2023-04-03 18:20:35 +08:00
Haojun Liao 482c319bee refactor: do some internal refactor. 2023-04-03 18:07:28 +08:00
Haojun Liao 3ef7d43dae refactor: do some internal refactor. 2023-04-03 17:47:48 +08:00
yihaoDeng 6ee0180217 fix invalid read/write error 2023-04-03 03:32:01 +00:00
yihaoDeng 7f4d0a17e6 add backend 2023-03-30 06:09:40 +00:00
yihaoDeng c3a8ecd9c2 reset version 2023-03-30 03:56:01 +00:00
yihaoDeng 009d5099b4 reset version 2023-03-30 01:53:26 +00:00
yihaoDeng 8387f04476 reset version 2023-03-30 01:53:13 +00:00
yihaoDeng 03f80fbe36 add backend 2023-03-29 07:28:34 +00:00
yihaoDeng 50b094ab5d add backend 2023-03-29 07:26:11 +00:00
yihaoDeng 2ca072344e add backend 2023-03-29 06:56:14 +00:00
yihaoDeng e3002f1872 add backend 2023-03-29 06:17:38 +00:00
yihaoDeng 1cf96cdbf1 add backend 2023-03-29 04:18:16 +00:00
yihaoDeng e6e9828d5a add backend 2023-03-29 03:26:59 +00:00
yihaoDeng ae54e994ca add backend 2023-03-29 01:27:39 +00:00
yihaoDeng 1e597503bc add backend 2023-03-28 15:16:06 +00:00
yihaoDeng 0e4445b3b5 add backend 2023-03-28 15:09:58 +00:00
yihaoDeng 0d11430be1 tqadd backend 2023-03-28 14:31:11 +00:00
yihaoDeng 86fe216921 add backend 2023-03-28 11:23:24 +00:00
yihaoDeng ca5f813c5e add backend 2023-03-28 09:27:42 +00:00
yihaoDeng 1c21d67570 add backend 2023-03-28 06:24:16 +00:00
yihaoDeng 47cb4339f0 add backend 2023-03-28 02:27:20 +00:00
yihaoDeng e1046015ae add backend 2023-03-28 02:25:11 +00:00