Commit Graph

8135 Commits

Author SHA1 Message Date
Hongze Cheng ec3685db94
Merge pull request #23600 from taosdata/fix/TD-27206
fix(vnode/cos): fix error printing to avoid buffer overflow
2023-11-09 06:06:54 -06:00
kailixu 572eb691fb enh: support delete msg for rsma 2023-11-09 19:51:01 +08:00
dapan1121 4f67629ea1
Merge pull request #23621 from taosdata/fix/TS-4122
correct colid in blockinfo
2023-11-09 18:09:09 +08:00
dmchen 2fbf0a532d fix/TD-27243 2023-11-09 18:02:06 +08:00
yihaoDeng 757425bff8 Merge branch '3.0' into enh/refactorBackend 2023-11-09 17:58:49 +08:00
dmchen 65bebb0912 fix/TD-27243 2023-11-09 09:51:35 +00:00
Minglei Jin d3cf6a4340 tsdb/file: new nlevel field for remove 2023-11-09 17:44:24 +08:00
Hongze Cheng b1fbdfcdc9
Merge pull request #23617 from taosdata/fix/little_typo
fix: code typo
2023-11-09 03:35:31 -06:00
wade zhang c0c665d785
Merge pull request #23614 from taosdata/FIX/TD-27230-3.0
fix: close vnode in the failed mode properly in vmCloseVnode
2023-11-09 16:32:20 +08:00
wangjiaming0909 a6600ab23a correct colid in blockinfo 2023-11-09 15:41:01 +08:00
Minglei Jin 386c25a99a tsdb/retention: remove file when ref's clear 2023-11-09 15:24:19 +08:00
kailixu 9cf154543f chore: merge 3.0 2023-11-09 15:09:30 +08:00
kailixu aa3ba74f6e chore: code optimization 2023-11-09 15:03:14 +08:00
dapan1121 8314cb6013
Merge pull request #23494 from taosdata/enh/TD-23769-3.0x
enh: checkpoint for rsma
2023-11-09 14:51:37 +08:00
kailixu 74cddda24b Merge branch '3.0' into enh/TD-25601-3.0 2023-11-09 14:37:13 +08:00
Hongze Cheng e60f69a7ed fix: code typo 2023-11-09 14:20:27 +08:00
yihaoDeng f74edc7618 Merge branch '3.0' into enh/refactorBackend 2023-11-09 06:14:08 +00:00
Minglei Jin 398771da5b Merge branch '3.0' into fix/TD-27206 2023-11-09 14:06:15 +08:00
liuyao e56a3e8a1e Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898 2023-11-09 13:59:39 +08:00
Benguang Zhao b80770dea8 fix: close vnode in the failed mode properly in vmCloseVnode 2023-11-09 13:05:52 +08:00
kailixu 1a950afdb8 enh: test case for rsma snapshot 2023-11-09 12:09:27 +08:00
Haojun Liao 257ab4f270
Merge pull request #23526 from taosdata/feat/TD-26060
feat:checkpoint backup function
2023-11-09 11:46:28 +08:00
kailixu 349e190120 enh: rsma result 2023-11-09 11:23:50 +08:00
kailixu 3495efaac7 enh: rsma exception process 2023-11-09 10:49:40 +08:00
Haojun Liao 62f34c757e fix(stream): remove invalid free. 2023-11-09 10:10:37 +08:00
Minglei Jin 1bb10bb862 vnode/cos: check get object block size optionally 2023-11-09 09:32:13 +08:00
kailixu 501d5a7acc Merge branch '3.0' into enh/TD-23769-3.0x 2023-11-08 20:47:15 +08:00
kailixu 72ecb0431c enh: rsma checkpoint 2023-11-08 20:45:44 +08:00
Minglei Jin 5ccdde4495 vnode/cos: error on incomplete fetching 2023-11-08 20:43:52 +08:00
yihaoDeng 3abf0ae4b2 Merge branch 'feat/TD-26060' into enh/refactorBackend 2023-11-08 17:31:16 +08:00
liuyao a8b415ed35 Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898 2023-11-08 17:26:00 +08:00
Minglei Jin 128353a861 vnode/cos: fix get object handler mem 2023-11-08 16:04:24 +08:00
Minglei Jin e5bbcf76f1 vnode/cos: fix get object block callback 2023-11-08 15:57:56 +08:00
Haojun Liao 99dbbe61d7 refactor(stream): do some internal refactor. 2023-11-08 15:20:47 +08:00
Minglei Jin a1e692a796 fix(vnode/cos): fix error printing to avoid buffer overflow 2023-11-08 15:18:55 +08:00
wangjiaming0909 3e850df3c2 fix: last(ts2), ts2, ts caused data err 2023-11-08 14:43:11 +08:00
Minglei Jin 3f125bc6bf vnode/cos: make err msg buffer big enough for detailed msg 2023-11-08 14:30:30 +08:00
liuyao 340fb3e727 Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898 2023-11-08 14:16:59 +08:00
Haojun Liao 658fafd03c refactor: do some internal refactor. 2023-11-08 14:02:08 +08:00
wangjiaming0909 fc6c4fba90 feat: cache scan for select last(c), ts/c 2023-11-08 13:39:28 +08:00
Haojun Liao 809dd064a7 refactor(stream): create sim env for stream processing. 2023-11-08 11:04:16 +08:00
Haojun Liao 17bd6badec refactor(stream): do checkpoint for each task. 2023-11-08 10:47:54 +08:00
yihaoDeng 7ada7f34ac fix:merge from 3.0 2023-11-08 10:45:15 +08:00
kailixu d1106c51dc enh: rsma code optimization 2023-11-08 09:54:03 +08:00
Minglei Jin 764f8d2e45 fix(vnode/s3): move init & cleanup to dnode 2023-11-08 09:03:19 +08:00
kailixu b9f389a49f Merge branch '3.0' into enh/TD-23769-3.0x 2023-11-08 08:36:35 +08:00
kailixu f2d24306b1 enh: rsma tasks share one checkpoint and fix memory leak 2023-11-08 08:35:52 +08:00
Haojun Liao 666b8c531b
Merge pull request #23548 from taosdata/fix/liaohj
refactor: do some internal refactor.
2023-11-08 08:08:36 +08:00
Haojun Liao e052d3cd62 fix(stream): check null. 2023-11-08 00:10:08 +08:00
Haojun Liao 9ba854d49d Merge branch '3.0' into fix/3_liaohj 2023-11-07 23:51:07 +08:00
Haojun Liao 7bfea5a82e fix(stream): fix syntax error. 2023-11-07 22:31:35 +08:00
kailixu 6b738884a4 chore: rsma checkpoint 2023-11-07 19:59:05 +08:00
wangmm0220 c0068abc9a fix:merge from 3.0 2023-11-07 18:55:46 +08:00
Haojun Liao 2674698b36 fix(stream): restart tasks in stream threads, instead of write thread. 2023-11-07 18:51:46 +08:00
yihaoDeng 95b14ee39b merge 3.0 2023-11-07 17:15:49 +08:00
Haojun Liao 1301b92844 fix(stream): fix errors in scan-history, introduced by refactor 2023-11-07 16:41:15 +08:00
Haojun Liao 8c2f6bab0e fix(stream): exec scan-history in timer. 2023-11-07 16:40:03 +08:00
Haojun Liao 7370daa6e0 fix(stream): record the start failure tasks. 2023-11-07 16:39:56 +08:00
Haojun Liao e38e526dda fix(stream):reset the starting flag for the followers. 2023-11-07 16:39:12 +08:00
Haojun Liao 207b53f3f6 fix(stream): fix errors in scan-history, introduced by refactor 2023-11-07 16:19:09 +08:00
kailixu ac62714492 Merge branch '3.0' into enh/TD-23769-3.0x 2023-11-07 14:20:38 +08:00
Haojun Liao 4d8548e938 refactor: do some internal refactor. 2023-11-06 23:42:16 +08:00
kailixu f5d796a081 chore: print stream state 2023-11-06 19:53:09 +08:00
Haojun Liao d9474016fc fix(stream): exec scan-history in timer. 2023-11-06 18:55:53 +08:00
kailixu dc5284a19c chore: add debug info 2023-11-06 17:27:36 +08:00
Haojun Liao 6b1889284b fix(stream): record the start failure tasks. 2023-11-06 15:45:44 +08:00
Minglei Jin ec908fa4d0 fix(cache/reader): use int64 max instead of uint 2023-11-06 15:34:28 +08:00
wangmm0220 dffaac55ea fix:move cos from util to common 2023-11-06 14:57:32 +08:00
Minglei Jin 31ce6453c2 fix(cmake/s3): make cmake config portable 2023-11-06 14:17:27 +08:00
Haojun Liao 69863febe6 fix(stream):reset the starting flag for the followers. 2023-11-06 13:17:13 +08:00
Haojun Liao bfd14076d7 fix(stream): fix dead-lock. 2023-11-06 12:05:47 +08:00
Haojun Liao d41fd27189 fix(stream): restart tasks in stream threads, instead of write thread. 2023-11-06 11:57:21 +08:00
Haojun Liao 0e44950a37 enh(stream): execute scan history by using time slice 1s. 2023-11-06 09:21:33 +08:00
Haojun Liao 680eb5674e refactor: do some internal refactor. 2023-11-06 09:15:49 +08:00
kailixu 7e8c123fca chore: rsma code optimization 2023-11-06 08:46:04 +08:00
Haojun Liao d440da0d8c enh(stream): execute scan history by using time slice 1s. 2023-11-05 01:26:53 +08:00
Haojun Liao e13f003b90 refactor: do some internal refactor. 2023-11-05 01:23:23 +08:00
kailixu 411151d671 fix: buffer overflow/buffer use after free/memory leak 2023-11-04 08:44:26 +08:00
yihaoDeng f038f88c07 Merge branch 'feat/TD-26060' into enh/refactorBackend 2023-11-03 18:29:18 +08:00
Haojun Liao 3415ba5195
Merge pull request #23528 from taosdata/fix/liaohj
enh(tsdb): on-demand open stt file reader.
2023-11-03 17:29:55 +08:00
kailixu 77eadc59dc Merge branch '3.0' into enh/TD-23769-3.0x 2023-11-03 16:26:07 +08:00
Haojun Liao d3ddaaa9f3 other(tsdb): adjust some log info. 2023-11-03 15:08:25 +08:00
Haojun Liao de5c4d8d5d other(tsdb): adjust some log info. 2023-11-03 15:07:40 +08:00
kailixu c95fc014a8 enh: rsma checkpoint 2023-11-03 14:57:36 +08:00
kailixu a48968e5e8 enh: rsma checkpoint 2023-11-03 14:33:45 +08:00
Haojun Liao 34403534ed Merge branch '3.0' into fix/3_liaohj 2023-11-03 14:06:58 +08:00
liuyao b1d1e2c778 Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898 2023-11-03 14:02:59 +08:00
Haojun Liao b4d14d3847 fix(tsdb): fix error during add on-demand opening stt file reader. 2023-11-03 13:51:18 +08:00
Haojun Liao 675c9ca613 enh(tsdb): on-demand open stt file reader. 2023-11-03 13:51:18 +08:00
wangmm0220 0a1f1085fc Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-26060 2023-11-03 11:33:50 +08:00
Haojun Liao c980e2030c fix(tsdb): fix error during add on-demand opening stt file reader. 2023-11-03 11:04:32 +08:00
Haojun Liao 0acf4f1795 enh(tsdb): on-demand open stt file reader. 2023-11-03 10:35:05 +08:00
Hongze Cheng 1ff3f96b08
Merge pull request #23465 from taosdata/fix/TD-26189-last-tier-write
enh(tsdb/retention): enable last tier write
2023-11-02 21:21:38 -05:00
yihaoDeng 24e961976f Merge branch 'enh/new3.0' into enh/refactorBackend 2023-11-03 09:31:57 +08:00
liuyao 31ad73f6d1 Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898 2023-11-03 09:12:17 +08:00
Haojun Liao ff5d69352f
Merge pull request #23493 from taosdata/fix/3_liaohj
fix(stream): add unsupport trans.
2023-11-03 09:07:33 +08:00
Haojun Liao 70388bafd2
Merge pull request #23523 from taosdata/fix/liaohj
fix(stream): fix deadlock in transfer state.
2023-11-03 09:07:23 +08:00
kailixu d9ed63473c Merge branch '3.0' into enh/TD-23769-3.0x 2023-11-03 08:37:41 +08:00
yihaoDeng 6ce878948f Merge branch 'enh/new3.0' into enh/refactorBackend 2023-11-02 20:44:36 +08:00
kailixu 6c944bb192 enh: skip rsma_task during load stream tasks 2023-11-02 20:32:54 +08:00
wangmm0220 159873692c feat:checkpoint bakeup using rsync 2023-11-02 20:13:29 +08:00
liuyao 389117b806 Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898 2023-11-02 20:07:12 +08:00
Haojun Liao b3d57e4a2d fix(stream): not transfer state if state is not appropriate. 2023-11-02 19:31:35 +08:00
Haojun Liao 7a32b3a209 fix(stream): not transfer state if state is not appropriate. 2023-11-02 19:12:36 +08:00
kailixu fa5d896787 enh: rsma checkpoint 2023-11-02 18:56:36 +08:00
Haojun Liao c4079332ff fix(stream):update the log level. 2023-11-02 18:18:04 +08:00
Haojun Liao 0a227e807f refactor: set different tq level. 2023-11-02 18:18:04 +08:00
Haojun Liao 48aff868b6 fix(stream):update the log level. 2023-11-02 18:16:47 +08:00
Haojun Liao 3aed8c29ae refactor: set different tq level. 2023-11-02 17:12:20 +08:00
Hongze Cheng faea18c38f fix: invalid read when file is deleted 2023-11-02 16:59:11 +08:00
kailixu 2d597659bc enh: rsma checkpoint 2023-11-02 15:20:54 +08:00
liuyao 330b04a8bc Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898 2023-11-02 14:36:13 +08:00
Haojun Liao f5fa3edeba
Merge pull request #23503 from taosdata/fix/liaohj
fix(stream): fix dead lock.
2023-11-02 13:52:50 +08:00
kailixu 722777f8c9 enh: rsma checkpoint 2023-11-02 13:15:31 +08:00
kailixu 96b5024347 enh: rsma checkpoint 2023-11-02 13:05:36 +08:00
kailixu 698fb804f9 enh: disable update tbUidList during reboot 2023-11-02 12:54:23 +08:00
kailixu 76536e1c82 enh: rsma logic 2023-11-02 11:03:18 +08:00
Haojun Liao 95efa07e92 fix(stream): add full unsupported event filtering. 2023-11-02 11:02:39 +08:00
Minglei Jin 46d3be1f08 config/page-cache-size: new parameter for pcache size 2023-11-02 10:10:03 +08:00
kailixu 9649e87cab fix: rsma checkpoint 2023-11-02 09:44:43 +08:00
Haojun Liao 351a31302f fix(stream): add full unsupported event filtering. 2023-11-02 09:43:48 +08:00
Minglei Jin d0327e07b8 tsdb/pg-cache: use pgno for cache key instead of blkno 2023-11-02 09:42:19 +08:00
Hongze Cheng a6ee7c4685 fix: possible coredump 2023-11-01 20:34:47 +08:00
kailixu 3803f952f9 chore: rsma checkpoint verify 2023-11-01 20:05:00 +08:00
Haojun Liao 7cf90dde5c refactor: do some internal refactor. 2023-11-01 19:38:13 +08:00
Haojun Liao 00f029e44f refactor: add some logs. 2023-11-01 19:38:12 +08:00
Haojun Liao 9a9c1d689d fix(stream): add lock log. 2023-11-01 19:38:12 +08:00
Haojun Liao 246ed4e022 refactor: do some internal refactor. 2023-11-01 17:19:21 +08:00
liuyao 505c44d9aa Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898 2023-11-01 16:50:46 +08:00
kailixu c32e60d199 chore: more code for rsma checkpoint 2023-11-01 16:27:41 +08:00
Haojun Liao 12bc4f32cf refactor: add some logs. 2023-11-01 16:12:15 +08:00
yihaoDeng 4c8edb5545 Merge branch 'enh/new3.0' into enh/refactorBackend 2023-11-01 16:07:49 +08:00
kailixu 59be62c96e chore: set stream input checkpoint 2023-11-01 15:34:17 +08:00
Minglei Jin 6f0fa5d47e tsdb/retention: remove incorrect right brace 2023-11-01 14:56:57 +08:00
Haojun Liao 5ff89bc098 fix(stream): add lock log. 2023-11-01 13:19:53 +08:00
kailixu ab266c712f chore: checkpoint for rsma stream state 2023-11-01 12:29:54 +08:00
Haojun Liao 0cf0ff3bbf
Merge pull request #23455 from taosdata/fix/liaohj
refactor: do some internal refactor.
2023-11-01 12:01:54 +08:00
Haojun Liao 3e6c89d8eb fix(stream): add unsupport trans. 2023-11-01 11:43:02 +08:00
Minglei Jin 221afff9ee vnode/write: remove keep modify code block 2023-11-01 11:00:47 +08:00
Minglei Jin 64644d673b
Merge branch '3.0' into fix/TD-26189-last-tier-write 2023-11-01 10:53:55 +08:00
Minglei Jin a61502411d tsdb/pg-cache: new page cache for tsdb s3 read file page 2023-11-01 10:49:05 +08:00
kailixu 1530fe0c49 Merge branch '3.0' into enh/TD-23769-3.0x 2023-11-01 09:52:16 +08:00
kailixu ad1e6accd4 chore: build checkpoint for rsma 2023-11-01 09:45:58 +08:00
Haojun Liao 7359f98eca
Merge pull request #23428 from taosdata/fix/3_liaohj
other: merge refactor into 3.0
2023-11-01 09:44:31 +08:00
Haojun Liao 639d5cde47 fix(stream): add unsupport trans. 2023-10-31 23:14:02 +08:00
yihaoDeng bb8252df16 fix mem leak 2023-10-31 20:59:24 +08:00
kailixu e6684fa5de enh: rsma retetion and stream state 2023-10-31 20:26:42 +08:00
dapan1121 d40cc27abc
Merge pull request #23464 from taosdata/enh/TD-26638-3.0
enh: use placeholder '-' for interval of 1st retention level
2023-10-31 18:36:43 +08:00
Haojun Liao a1aa2380ae other: merge 3.0 2023-10-31 16:42:51 +08:00
Haojun Liao 0b18192b23 fix(stream): fix error in finishing event handling. 2023-10-31 16:40:16 +08:00