Commit Graph

248 Commits

Author SHA1 Message Date
yihaoDeng f5ef2e727f fix stream backend convert 2023-11-16 14:41:04 +08:00
yihaoDeng 4476e46a5f rm dup data 2023-11-14 17:43:13 +08:00
yihaoDeng 4104840cea Merge branch '3.0' into enh/refactorBackend 2023-11-13 18:07:38 +08:00
liuyao 37c102a115 Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898 2023-11-13 16:06:17 +08:00
yihaoDeng de7f2a2d1a Merge branch '3.0' into enh/refactorBackend 2023-11-13 11:11:28 +08:00
Haojun Liao 9ee9174608 other: merge 3.0 2023-11-11 15:48:31 +08:00
Haojun Liao af08a189c1 fix(stream): disable concurrently restart stream tasks. 2023-11-11 15:40:10 +08:00
yihaoDeng a3fdb6dd59 Merge branch '3.0' into enh/refactorBackend 2023-11-10 16:06:08 +08:00
liuyao 6d2d5823dc Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898 2023-11-10 09:24:05 +08:00
yihaoDeng 5f9b922a04 support s3 2023-11-09 20:15:08 +08:00
yihaoDeng 757425bff8 Merge branch '3.0' into enh/refactorBackend 2023-11-09 17:58:49 +08:00
yihaoDeng dd2252598e upload by rsync 2023-11-09 17:39:34 +08:00
dapan1121 8314cb6013
Merge pull request #23494 from taosdata/enh/TD-23769-3.0x
enh: checkpoint for rsma
2023-11-09 14:51:37 +08:00
liuyao e56a3e8a1e Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898 2023-11-09 13:59:39 +08:00
yihaoDeng 3abf0ae4b2 Merge branch 'feat/TD-26060' into enh/refactorBackend 2023-11-08 17:31:16 +08:00
yihaoDeng 92fa1043b8 Merge branch 'enh/refactorBackend' of https://github.com/taosdata/TDengine into enh/refactorBackend 2023-11-08 17:12:32 +08:00
yihaoDeng 76fcc49c6d fix stream transfer error 2023-11-08 17:10:54 +08:00
Haojun Liao 470f244032 Merge remote-tracking branch 'origin/enh/refactorBackend' into enh/refactorBackend
# Conflicts:
#	include/libs/stream/tstream.h
2023-11-08 15:21:21 +08:00
Haojun Liao 99dbbe61d7 refactor(stream): do some internal refactor. 2023-11-08 15:20:47 +08:00
wangmm0220 c4c1423243 fix:error 2023-11-08 15:01:04 +08:00
liuyao 340fb3e727 Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898 2023-11-08 14:16:59 +08:00
yihaoDeng fb77d811f9 Merge branch 'enh/refactorBackend' of https://github.com/taosdata/TDengine into enh/refactorBackend 2023-11-08 10:48:36 +08:00
Haojun Liao 17bd6badec refactor(stream): do checkpoint for each task. 2023-11-08 10:47:54 +08:00
yihaoDeng 7ada7f34ac fix:merge from 3.0 2023-11-08 10:45:15 +08:00
yihaoDeng 594da2d603 merge 3.0 2023-11-08 10:04:46 +08:00
kailixu b9f389a49f Merge branch '3.0' into enh/TD-23769-3.0x 2023-11-08 08:36:35 +08:00
wangmm0220 f0b2ed567c fix:upload checkpoint to s3 2023-11-07 18:49:12 +08:00
Haojun Liao ed0c580d11 refactor: do some internal refactor. 2023-11-07 17:04:55 +08:00
Haojun Liao 8c2f6bab0e fix(stream): exec scan-history in timer. 2023-11-07 16:40:03 +08:00
Haojun Liao e38e526dda fix(stream):reset the starting flag for the followers. 2023-11-07 16:39:12 +08:00
yihaoDeng f038f88c07 Merge branch 'feat/TD-26060' into enh/refactorBackend 2023-11-03 18:29:18 +08:00
wangmm0220 0a1f1085fc Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-26060 2023-11-03 11:33:50 +08:00
wangmm0220 3ac11eba0e fix:modify upload checkpoint interface 2023-11-03 11:33:18 +08:00
wangmm0220 159873692c feat:checkpoint bakeup using rsync 2023-11-02 20:13:29 +08:00
liuyao 505c44d9aa Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898 2023-11-01 16:50:46 +08:00
yihaoDeng 4c8edb5545 Merge branch 'enh/new3.0' into enh/refactorBackend 2023-11-01 16:07:49 +08:00
kailixu 1530fe0c49 Merge branch '3.0' into enh/TD-23769-3.0x 2023-11-01 09:52:16 +08:00
kailixu ad1e6accd4 chore: build checkpoint for rsma 2023-11-01 09:45:58 +08:00
Haojun Liao 1ccbb77981 fix(stream): fix dead lock when shutdown all tasks. 2023-10-27 09:59:37 +08:00
Haojun Liao e359da7b19 fix(stream): fix bugs caused by refactor. 2023-10-27 09:54:57 +08:00
Haojun Liao 8729bdac1e fix(stream): fix bugs caused by refactor. 2023-10-27 09:54:57 +08:00
Haojun Liao 579dd8e6b2 refactor(stream): do some internal refactor. 2023-10-27 09:54:56 +08:00
Haojun Liao 6f3c0d17cc refactor(stream): add state machine to manage the state of stream tasks. 2023-10-27 09:54:49 +08:00
Haojun Liao 596eb073b7 refactor(stream): do some internal refactor. 2023-10-27 09:54:48 +08:00
Haojun Liao 91e3fd66e6 enh(stream): sm for stream tasks. 2023-10-27 09:54:44 +08:00
Haojun Liao 2409a17517 refactor(stream): do some internal refactor. 2023-10-27 09:54:44 +08:00
Haojun Liao 164159f293 enh(stream): add parameter to limit the stream sink task. 2023-10-27 09:54:07 +08:00
liuyao f51c985914 stream event window 2023-10-25 17:50:16 +08:00
yihaoDeng 1a4cfd11a5 Merge branch 'enh/new3.0' into enh/refactorBackend 2023-10-20 15:28:10 +08:00
yihaoDeng 2004c1a346 fix transfer crash 2023-10-17 18:19:52 +08:00
yihaoDeng 77542cc461 fix stream snap deadlock 2023-10-17 11:23:39 +08:00
Haojun Liao 84c5359d5b other: merge 3.0 2023-10-13 08:58:26 +08:00
yihaoDeng a56d831a14 refactor backend 2023-10-12 20:16:41 +08:00
Haojun Liao eb4449999f other: merge 3.0 2023-10-12 10:33:48 +08:00
yihaoDeng 4c94fbb44f refactor backend 2023-10-11 20:26:19 +08:00
yihaoDeng f264ffdcd6 refactor backend 2023-10-11 20:12:07 +08:00
yihaoDeng 28421cd35e refactor backend 2023-10-10 20:51:41 +08:00
yihaoDeng b36163a197 add session iter 2023-10-10 15:12:21 +08:00
yihaoDeng 825341d90a add sess iter 2023-10-10 14:35:02 +08:00
Haojun Liao 59e284c332 fix(stream): quit from loop when input queue is full. 2023-10-09 20:10:21 +08:00
Haojun Liao 28008f8263 enh(stream): add more info for sink task. 2023-10-09 20:09:22 +08:00
Haojun Liao cf14772dd9 enh(stream): optimize the scan wal perf 2023-10-09 20:09:21 +08:00
Haojun Liao 583ac1fc45 refactor: do some internal refactor. 2023-10-09 20:09:20 +08:00
Haojun Liao d10b07d23d refactor: do some internal refactor. 2023-10-09 20:08:53 +08:00
Haojun Liao dfe8641c0c refactor: optimize the fill-history task launch policy, and do some other internal refactor. 2023-10-09 20:08:52 +08:00
Haojun Liao 7f8da725bc enh(stream): add quota limitation for sink task. 2023-10-09 20:08:13 +08:00
Haojun Liao 7167c3c5de fix(stream): add new timer for each task. 2023-10-09 20:08:08 +08:00
Haojun Liao 52763ca2a3 refactor: do some internal refactor. 2023-10-09 20:08:07 +08:00
Haojun Liao 2b16976902 refactor(stream): refactor send msg. 2023-10-09 20:08:03 +08:00
Haojun Liao ef9efc8a8e refactor: do refactor the sink task. 2023-10-09 20:07:53 +08:00
Haojun Liao 1f45bd82d2 other: merge stream fix in main. 2023-10-09 20:07:01 +08:00
Haojun Liao 7ccc273206 other: merge stream fix in main. 2023-10-09 20:05:29 +08:00
Haojun Liao e84eeee6b8 refactor: do some internal refactor. 2023-10-09 20:04:17 +08:00
yihaoDeng 2aa88dfe9b fix mem leak 2023-10-09 15:53:42 +08:00
yihaoDeng c146327da8 refact task backend 2023-10-07 16:48:36 +08:00
Haojun Liao 7ce3d2c200 fix(stream): quit from loop when input queue is full. 2023-10-07 14:54:11 +08:00
yihaoDeng fa70221238 refact task backend 2023-10-07 12:17:55 +08:00
Haojun Liao c20dd002fb enh(stream): add more info for sink task. 2023-10-07 01:14:00 +08:00
Haojun Liao 41ef1a45c1 enh(stream): optimize the scan wal perf 2023-10-04 22:06:46 +08:00
Haojun Liao 5718158fc0 refactor: do some internal refactor. 2023-10-01 22:27:29 +08:00
yihaoDeng 4f66dc538f add cvt state 2023-09-28 11:52:18 +00:00
yihaoDeng 54e3ac2c1e refact task backend 2023-09-28 15:46:12 +08:00
Haojun Liao 11641e1115 refactor: do some internal refactor. 2023-09-27 14:18:01 +08:00
Haojun Liao e13efd5b04 refactor: optimize the fill-history task launch policy, and do some other internal refactor. 2023-09-27 14:00:48 +08:00
Haojun Liao aec15ae39f enh(stream): add quota limitation for sink task. 2023-09-26 23:29:54 +08:00
Haojun Liao a74b462ffb fix(stream): add new timer for each task. 2023-09-24 02:25:53 +08:00
Haojun Liao 4ea737571d refactor: do some internal refactor. 2023-09-24 01:26:51 +08:00
yihaoDeng 19e52c364e add cvt state 2023-09-22 08:26:03 +00:00
Haojun Liao 898aea5cfb refactor(stream): refactor send msg. 2023-09-22 14:52:18 +08:00
yihaoDeng 470849982e add cvt func 2023-09-21 09:41:30 +00:00
Haojun Liao 17586af0ce refactor: do refactor the sink task. 2023-09-21 13:32:57 +08:00
Haojun Liao 6633fb971b fix(stream): update logs. 2023-09-20 18:18:31 +08:00
Haojun Liao 4846194bbb refactor: do some internal refactor, remove the related fill-history task by sending mesg to replay it in follower node. 2023-09-16 21:37:30 +08:00
Haojun Liao 2bfd6e3355 refactor: do some internal refactor. 2023-09-15 09:25:22 +08:00
Haojun Liao b240e91c91 enh(stream): add token bucket to limit the rate of sink operation. 2023-09-07 09:58:48 +08:00
Haojun Liao 3fc3dafca6 refactor: do some internal refactor. 2023-08-31 13:46:48 +08:00
Haojun Liao e53b5d4392 refactor: do some internal refactor. 2023-08-31 11:01:53 +08:00
yihaoDeng 8ed2836a2b refactor code 2023-08-29 09:52:03 +08:00
yihaoDeng a2901411b0 refactor code 2023-08-28 18:52:10 +08:00
yihaoDeng 0959758bd5 fix transfer error 2023-08-23 21:43:10 +08:00