Haojun Liao
|
6898eba6ed
|
fix(stream): expand stream tasks are divided into two phase, the first is to build stream task and then expand stream task before start stream tasks.
|
2024-05-14 14:34:31 +08:00 |
|
Haojun Liao
|
fb248b2682
|
fix(stream): not save the stream task for rsma.
|
2024-05-13 17:26:39 +08:00 |
|
Haojun Liao
|
3dfffe9113
|
fix(stream): async load and start stream tasks.
|
2024-05-13 15:31:26 +08:00 |
|
Haojun Liao
|
370c255128
|
refactor: do some internal refactor.
|
2024-05-11 10:27:55 +08:00 |
|
Haojun Liao
|
11ed1f54b3
|
refactor: do some internal refactor.
|
2024-05-11 10:09:23 +08:00 |
|
Haojun Liao
|
7f93ec2c53
|
fix(stream): update the checkpoint info for followers, and some internal refactor.
|
2024-05-11 09:37:02 +08:00 |
|
Haojun Liao
|
6693efaa85
|
refactor: do some internal refactor.
|
2024-05-09 10:11:37 +08:00 |
|
Haojun Liao
|
9ce8f6953d
|
refactor: do some internal refactor.
|
2024-05-08 13:52:20 +08:00 |
|
Haojun Liao
|
18c66a5491
|
refactor: do some internal refactor.
|
2024-05-06 18:17:38 +08:00 |
|
Haojun Liao
|
7ad7ef7cfc
|
refactor: do some internal refactor.
|
2024-05-06 17:03:41 +08:00 |
|
Haojun Liao
|
94a0f9ab8f
|
other: merge 3.0
|
2024-04-30 17:12:46 +08:00 |
|
Haojun Liao
|
629502e12e
|
refactor: do some internal refactor.
|
2024-04-30 17:00:51 +08:00 |
Yihao Deng
|
3fd6770c81
|
avoid invalid read
|
2024-04-29 05:52:54 +00:00 |
|
Yihao Deng
|
b54198ba4a
|
avoid invalid read
|
2024-04-29 03:18:38 +00:00 |
|
Yihao Deng
|
97aadd3aa7
|
avoid invalid read
|
2024-04-29 03:17:33 +00:00 |
|
Yihao Deng
|
2beb46c9df
|
Merge branch '3.0' into fix/TD-29844
|
2024-04-29 01:27:20 +00:00 |
|
Yihao Deng
|
9e19c52924
|
Remove unnecessary memory free in uploadCheckpointData function
|
2024-04-28 06:50:54 +00:00 |
|
Yihao Deng
|
fac8530580
|
fix invalid read
|
2024-04-28 06:50:33 +00:00 |
|
Yihao Deng
|
af567ea2cb
|
fix invalid read
|
2024-04-28 06:47:31 +00:00 |
|
Haojun Liao
|
b2d8260f14
|
refactor: do some internal refactor.
|
2024-04-28 10:05:22 +08:00 |
|
Haojun Liao
|
8adaeb59a3
|
refactor: do some internal refactor.
|
2024-04-28 09:50:21 +08:00 |
|
Haojun Liao
|
8b53f76691
|
fix(stream): discard the repeatly recv checkpoint-source msg.
|
2024-04-26 15:56:34 +08:00 |
|
Haojun Liao
|
d1ecfe5cf3
|
refactor: do some internal refactor.
|
2024-04-25 14:24:20 +08:00 |
|
Haojun Liao
|
82cde46614
|
fix(stream): fix memory leak.
|
2024-04-23 19:48:50 +08:00 |
|
Haojun Liao
|
e7dee4ac63
|
refactor(stream): refactor check msg retry procedure and be tolerant to the dnode disconnection during check downstream tasks.
|
2024-04-18 15:55:21 +08:00 |
|
Haojun Liao
|
665107ad3a
|
enh(stream): add more info in meta table.
|
2024-04-15 14:23:37 +08:00 |
|
Haojun Liao
|
9ca84091df
|
fix(stream): reset the ready upstream counter after task-reset.
|
2024-04-01 16:31:10 +08:00 |
|
Haojun Liao
|
cc0b32b1f8
|
fix(stream): transfer state before do checkpoint, to avoid the retrieve and deadlock by using waiting .
|
2024-03-14 16:13:41 +08:00 |
54liuyao
|
0e8170de49
|
remove invalid code
|
2024-02-20 10:17:47 +08:00 |
|
Haojun Liao
|
2bba011e4e
|
Merge remote-tracking branch 'origin/feat/TD-22023' into feat/TD-22023
|
2024-02-05 17:51:23 +08:00 |
|
Haojun Liao
|
7a18e5910c
|
enh(stream): halt the initial stream tasks when fill-history is set for count window agg
|
2024-02-05 17:51:09 +08:00 |
|
Haojun Liao
|
3236ef7bae
|
feat(stream): return the stream source task exec delay, and do some internal refactor.
|
2024-02-02 11:26:47 +08:00 |
|
Haojun Liao
|
87e543c824
|
fix(stream): pass down the transId of checkpoint, to make sure the downstream task can report the error transId successfully.
|
2024-02-01 17:14:36 +08:00 |
|
Haojun Liao
|
3751e11394
|
fix(stream): fix dead lock.
|
2024-01-25 17:11:16 +08:00 |
|
Haojun Liao
|
8a7e38ad15
|
fix(stream): fix checkpoint failure check.
|
2024-01-24 13:10:53 +08:00 |
|
Haojun Liao
|
d973f66cea
|
fix(stream): do checkpoint after fill-history task completed.
|
2024-01-19 16:18:02 +08:00 |
|
Haojun Liao
|
76e6c453e7
|
enh(stream): send msg to mnode when the transferring state completed.
|
2024-01-18 15:43:36 +08:00 |
|
Haojun Liao
|
8ede6d87ce
|
refactor: do some internal refactor.
|
2024-01-16 15:40:40 +08:00 |
|
Haojun Liao
|
63494de7f9
|
refactor: do some internal refactor.
|
2024-01-16 10:28:07 +08:00 |
|
Haojun Liao
|
cacbb4e369
|
fix(stream): save the checkpoint info for sink tasks.
|
2024-01-15 17:40:35 +08:00 |
|
Haojun Liao
|
d46b5e6c9a
|
fix(stream): enable sink tasks handle the checkpoint.
|
2024-01-15 09:53:15 +08:00 |
|
Haojun Liao
|
ce855bc493
|
refactor: do some internal refactor.
|
2024-01-15 09:21:31 +08:00 |
|
Haojun Liao
|
9703018f56
|
fix(stream): add checkpoint status check.
|
2024-01-11 16:47:51 +08:00 |
|
Haojun Liao
|
6d37e596ec
|
refactor: do some internal refactor.
|
2024-01-04 18:47:49 +08:00 |
|
Haojun Liao
|
cd85dae3fd
|
fix(stream): fix the dead lock.
|
2023-12-24 23:30:00 +08:00 |
|
Haojun Liao
|
84583580e2
|
fix(stream): transfer the checkpoint trans id.
|
2023-12-21 19:12:49 +08:00 |
|
Haojun Liao
|
8a8ddf5699
|
fix(stream): record the checkpoint failure when nodeEp changed.
|
2023-12-21 10:30:44 +08:00 |
yihaoDeng
|
d8093d9f21
|
not do chkp when pbackend not inited
|
2023-12-15 11:47:44 +08:00 |
|
yihaoDeng
|
5189f3d15b
|
not do chkp when pbackend not inited
|
2023-12-15 09:18:37 +08:00 |
Minglei Jin
|
49ecb1093f
|
cos/put: new arg withcp to control resumable uploading
|
2023-11-30 17:20:21 +08:00 |
|
yihaoDeng
|
a4d475b582
|
fix rsma
|
2023-11-27 20:27:53 +08:00 |
|
yihaoDeng
|
6129ebc294
|
Merge branch '3.0' into enh/refactorBackend
|
2023-11-23 17:41:37 +08:00 |
|
yihaoDeng
|
6a0501967d
|
Merge branch '3.0' into fix/fixChkpRespMemleak
|
2023-11-23 14:06:04 +08:00 |
|
Haojun Liao
|
4ea49fe285
|
fix(stream): fix the race condition in dispatching data.
|
2023-11-22 23:49:16 +08:00 |
|
yihaoDeng
|
947775155b
|
fix transfer crash
|
2023-11-21 20:51:11 +08:00 |
|
yihaoDeng
|
a92602b667
|
fix mem leak
|
2023-11-21 15:23:28 +08:00 |
|
yihaoDeng
|
939aa7a3cc
|
Merge branch '3.0' into enh/refactorBackend
|
2023-11-20 17:33:30 +08:00 |
|
yihaoDeng
|
0d724244dc
|
Merge branch '3.0' into enh/refactorBackend
|
2023-11-20 16:55:14 +08:00 |
|
Haojun Liao
|
7cfa45e95e
|
enh(stream): add sink_quota/scan-history-idle-duration column for stream tasks.
|
2023-11-20 11:59:49 +08:00 |
|
yihaoDeng
|
aa2c09ef5b
|
rm dup data
|
2023-11-14 18:09:00 +08:00 |
|
yihaoDeng
|
4476e46a5f
|
rm dup data
|
2023-11-14 17:43:13 +08:00 |
|
yihaoDeng
|
4104840cea
|
Merge branch '3.0' into enh/refactorBackend
|
2023-11-13 18:07:38 +08:00 |
|
yihaoDeng
|
de7f2a2d1a
|
Merge branch '3.0' into enh/refactorBackend
|
2023-11-13 11:11:28 +08:00 |
|
Haojun Liao
|
9ee9174608
|
other: merge 3.0
|
2023-11-11 15:48:31 +08:00 |
|
Haojun Liao
|
97772e9aab
|
fix(stream): the checkpoint version can only be updated when generating checkpoint.
|
2023-11-11 15:40:05 +08:00 |
|
yihaoDeng
|
1e8d46c405
|
Merge branch '3.0' into enh/refactorBackend
|
2023-11-10 18:37:25 +08:00 |
|
yihaoDeng
|
a3fdb6dd59
|
Merge branch '3.0' into enh/refactorBackend
|
2023-11-10 16:06:08 +08:00 |
|
yihaoDeng
|
5f9b922a04
|
support s3
|
2023-11-09 20:15:08 +08:00 |
|
yihaoDeng
|
dd2252598e
|
upload by rsync
|
2023-11-09 17:39:34 +08:00 |
|
yihaoDeng
|
3abf0ae4b2
|
Merge branch 'feat/TD-26060' into enh/refactorBackend
|
2023-11-08 17:31:16 +08:00 |
|
yihaoDeng
|
92fa1043b8
|
Merge branch 'enh/refactorBackend' of https://github.com/taosdata/TDengine into enh/refactorBackend
|
2023-11-08 17:12:32 +08:00 |
|
yihaoDeng
|
2de5c4a97d
|
add bg thread
|
2023-11-08 17:11:11 +08:00 |
|
yihaoDeng
|
76fcc49c6d
|
fix stream transfer error
|
2023-11-08 17:10:54 +08:00 |
|
Haojun Liao
|
f949333e8b
|
refactor:do some internal refactor.
|
2023-11-08 16:57:55 +08:00 |
|
Haojun Liao
|
99dbbe61d7
|
refactor(stream): do some internal refactor.
|
2023-11-08 15:20:47 +08:00 |
wangmm0220
|
c4c1423243
|
fix:error
|
2023-11-08 15:01:04 +08:00 |
|
wangmm0220
|
047eb14f67
|
fix:error
|
2023-11-08 11:57:49 +08:00 |
|
Haojun Liao
|
17bd6badec
|
refactor(stream): do checkpoint for each task.
|
2023-11-08 10:47:54 +08:00 |
|
wangmm0220
|
07aa749fcd
|
fix:adapt to windows
|
2023-11-08 10:28:45 +08:00 |
|
wangmm0220
|
f0b2ed567c
|
fix:upload checkpoint to s3
|
2023-11-07 18:49:12 +08:00 |
|
yihaoDeng
|
f038f88c07
|
Merge branch 'feat/TD-26060' into enh/refactorBackend
|
2023-11-03 18:29:18 +08:00 |
|
wangmm0220
|
5a6c3f2b04
|
fix:add windows support
|
2023-11-03 17:45:37 +08:00 |
|
wangmm0220
|
0a1f1085fc
|
Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-26060
|
2023-11-03 11:33:50 +08:00 |
|
wangmm0220
|
3ac11eba0e
|
fix:modify upload checkpoint interface
|
2023-11-03 11:33:18 +08:00 |
|
yihaoDeng
|
24e961976f
|
Merge branch 'enh/new3.0' into enh/refactorBackend
|
2023-11-03 09:31:57 +08:00 |
|
wangmm0220
|
159873692c
|
feat:checkpoint bakeup using rsync
|
2023-11-02 20:13:29 +08:00 |
|
Haojun Liao
|
7cf90dde5c
|
refactor: do some internal refactor.
|
2023-11-01 19:38:13 +08:00 |
|
Haojun Liao
|
00f029e44f
|
refactor: add some logs.
|
2023-11-01 19:38:12 +08:00 |
|
yihaoDeng
|
4c8edb5545
|
Merge branch 'enh/new3.0' into enh/refactorBackend
|
2023-11-01 16:07:49 +08:00 |
|
Haojun Liao
|
88469568b8
|
fix(stream): handle failure during checkpoint
|
2023-10-31 10:45:14 +08:00 |
|
Haojun Liao
|
0d4892f25b
|
fix(stream): fix bugs caused by refactor sm.
|
2023-10-27 09:55:00 +08:00 |
|
Haojun Liao
|
6f3c0d17cc
|
refactor(stream): add state machine to manage the state of stream tasks.
|
2023-10-27 09:54:49 +08:00 |
|
yihaoDeng
|
1a4cfd11a5
|
Merge branch 'enh/new3.0' into enh/refactorBackend
|
2023-10-20 15:28:10 +08:00 |
|
Haojun Liao
|
af4618f958
|
fix(stream): remove invalid clear of rspmsg list.
|
2023-10-09 20:10:27 +08:00 |
|
Haojun Liao
|
21fb502690
|
enh(stream): add trans to reset task status to avoid being frozen in checkpoint status due to doing checkpoint failure of partial tasks.
|
2023-10-09 20:10:26 +08:00 |
|
Haojun Liao
|
af2c0c95e9
|
refactor: do some internal refactor.
|
2023-10-09 20:10:22 +08:00 |
|
Haojun Liao
|
8477271240
|
fix(stream): fix dead lock.
|
2023-10-09 20:09:00 +08:00 |
|
Haojun Liao
|
52763ca2a3
|
refactor: do some internal refactor.
|
2023-10-09 20:08:07 +08:00 |
|
Haojun Liao
|
3baddbe67a
|
fix(stream): update the checkpoint record start time.
|
2023-10-09 20:08:07 +08:00 |
|
Haojun Liao
|
a6ffb845c8
|
fix(stream): fix invalid checkpoint start time.
|
2023-10-09 20:08:07 +08:00 |
|
Haojun Liao
|
e5535fa1f5
|
fix(stream): handle dispatch checkpoint-trigger failure.
|
2023-10-09 20:08:01 +08:00 |
|
Haojun Liao
|
65cc4040f3
|
fix(stream): update logs.
|
2023-10-09 20:08:00 +08:00 |
|
Haojun Liao
|
6d3a134077
|
fix(stream): ignore the dispatch failure, and set the correct rsp counter.
|
2023-10-09 20:07:54 +08:00 |
|
Haojun Liao
|
e4ac835554
|
fix(stream): disable token bucket.
|
2023-10-09 20:07:53 +08:00 |
|
Haojun Liao
|
a3d13d6634
|
fix(stream): remove invalid cached task info in buffer.
|
2023-10-09 20:07:08 +08:00 |
|
Haojun Liao
|
1f45bd82d2
|
other: merge stream fix in main.
|
2023-10-09 20:07:01 +08:00 |
|
Haojun Liao
|
d76e3ac718
|
fix(stream): add task status check.
|
2023-10-09 20:06:10 +08:00 |
|
Haojun Liao
|
053643be19
|
fix(stream): add timestamp.
|
2023-10-09 20:06:05 +08:00 |
|
Haojun Liao
|
5c0b8ea804
|
enh(stream): log the checkpoint time.
|
2023-10-09 20:06:04 +08:00 |
|
Haojun Liao
|
0459a4628c
|
fix(stream): fix stream task id error.
|
2023-10-09 20:05:36 +08:00 |
|
Haojun Liao
|
7ccc273206
|
other: merge stream fix in main.
|
2023-10-09 20:05:29 +08:00 |
|
Haojun Liao
|
e84eeee6b8
|
refactor: do some internal refactor.
|
2023-10-09 20:04:17 +08:00 |
|
Haojun Liao
|
8266c4ff77
|
fix(stream): remove invalid clear of rspmsg list.
|
2023-10-09 15:26:42 +08:00 |
|
Haojun Liao
|
ee6516f3c4
|
enh(stream): add trans to reset task status to avoid being frozen in checkpoint status due to doing checkpoint failure of partial tasks.
|
2023-10-09 10:13:53 +08:00 |
|
Haojun Liao
|
ab17bfdf09
|
refactor: do some internal refactor.
|
2023-10-07 19:16:55 +08:00 |
|
yihaoDeng
|
e41da13d5b
|
refact task backend
|
2023-10-07 17:07:26 +08:00 |
|
Haojun Liao
|
8c998673ad
|
fix(stream): fix dead lock.
|
2023-09-27 17:15:49 +08:00 |
|
Haojun Liao
|
4ea737571d
|
refactor: do some internal refactor.
|
2023-09-24 01:26:51 +08:00 |
|
Haojun Liao
|
7f265e181d
|
fix(stream): update the checkpoint record start time.
|
2023-09-23 18:05:14 +08:00 |
|
Haojun Liao
|
6ef68fdf06
|
fix(stream): fix invalid checkpoint start time.
|
2023-09-23 17:57:09 +08:00 |
|
Haojun Liao
|
b075b4438b
|
fix(stream): handle dispatch checkpoint-trigger failure.
|
2023-09-21 23:15:18 +08:00 |
|
Haojun Liao
|
50d5fb6641
|
fix(stream): update logs.
|
2023-09-21 19:14:34 +08:00 |
|
Haojun Liao
|
4925b67d4a
|
fix(stream): ignore the dispatch failure, and set the correct rsp counter.
|
2023-09-21 17:22:09 +08:00 |
|
Haojun Liao
|
9f9b712332
|
fix(stream): disable token bucket.
|
2023-09-21 01:19:58 +08:00 |
|
Haojun Liao
|
92f18989df
|
fix(stream): remove invalid cached task info in buffer.
|
2023-09-20 19:19:04 +08:00 |
|
Haojun Liao
|
6633fb971b
|
fix(stream): update logs.
|
2023-09-20 18:18:31 +08:00 |
|
Haojun Liao
|
2a781c509f
|
fix(stream): add task status check.
|
2023-09-20 15:32:15 +08:00 |
|
Haojun Liao
|
309630eb11
|
fix(stream): add timestamp.
|
2023-09-19 14:22:05 +08:00 |
|
Haojun Liao
|
b7efedf4aa
|
enh(stream): log the checkpoint time.
|
2023-09-19 09:53:33 +08:00 |
|
Haojun Liao
|
01144c58a7
|
fix(stream): fix stream task id error.
|
2023-09-17 01:19:59 +08:00 |
|
Haojun Liao
|
4846194bbb
|
refactor: do some internal refactor, remove the related fill-history task by sending mesg to replay it in follower node.
|
2023-09-16 21:37:30 +08:00 |
|
Haojun Liao
|
2bfd6e3355
|
refactor: do some internal refactor.
|
2023-09-15 09:25:22 +08:00 |
|
Haojun Liao
|
f602aa965f
|
fix(stream): fix coverity scan issues.
|
2023-09-12 09:16:18 +08:00 |
|
Haojun Liao
|
85a5c45098
|
fix(stream): limit the already processed data.
|
2023-09-10 00:01:25 +08:00 |
|
Haojun Liao
|
2285b834b0
|
refactor: do some internal refactor.
|
2023-08-31 13:53:12 +08:00 |
|
Haojun Liao
|
3fc3dafca6
|
refactor: do some internal refactor.
|
2023-08-31 13:46:48 +08:00 |
|
Haojun Liao
|
e53b5d4392
|
refactor: do some internal refactor.
|
2023-08-31 11:01:53 +08:00 |
|
Haojun Liao
|
3a7a220d43
|
fix(stream): fix error found by ci.
|
2023-08-30 10:14:01 +08:00 |
|
Haojun Liao
|
391103bc5a
|
fix(stream): do dome internal refactor.
|
2023-08-28 15:54:01 +08:00 |
|
Haojun Liao
|
46529d383c
|
fix(stream): build checkpoint for sink tasks.
|
2023-08-27 00:18:41 +08:00 |
|
Haojun Liao
|
b7c36bbfe4
|
fix(stream): fix the status error.
|
2023-08-26 00:49:23 +08:00 |
|
Haojun Liao
|
ce1a3a379c
|
fix(stream): handle the case when the task not in ready state, and the checkpoint msg arrived.
|
2023-08-25 10:12:12 +08:00 |
|
Haojun Liao
|
f7ea875af1
|
fix(stream): add some logs.
|
2023-08-24 23:27:48 +08:00 |
|
Haojun Liao
|
4afe603c6e
|
fix(stream): fix error.
|
2023-08-11 14:17:09 +08:00 |
|
Haojun Liao
|
8dfef8768a
|
other: merge 3.0
|
2023-07-29 22:54:16 +08:00 |
|
Haojun Liao
|
9f943213ed
|
enh(stream): update the hb info.
|
2023-07-28 14:23:53 +08:00 |
|
Haojun Liao
|
1e34f84f78
|
other: merge 3.0
|
2023-07-24 15:23:03 +08:00 |
|
Haojun Liao
|
902f9d1f94
|
fix(stream): set not-ready initial value.
|
2023-07-18 18:29:35 +08:00 |
|
Haojun Liao
|
901b7d8ecc
|
fix(stream): if a source task is set to be non-normal, no data should be put into inputQ anymore.
|
2023-07-18 15:18:57 +08:00 |
|
Haojun Liao
|
2f24f776fb
|
fix(stream): fix error in generating checkpoint for state_window.
|
2023-07-18 09:04:28 +08:00 |