2410 Commits

Author	SHA1	Message	Date
yihaoDeng	a215332e2f	change chkpid gen way	2023-11-09 11:24:18 +08:00
yihaoDeng	50666987f0	change chkpid gen way	2023-11-09 11:23:17 +08:00
Haojun Liao	ef646873ef	Merge pull request #23596 from taosdata/fix/streamTransferError ... fix stream transfer err	2023-11-09 10:51:22 +08:00
kailixu	edef4de7b2	chore: revert the code	2023-11-08 20:57:57 +08:00
kailixu	72ecb0431c	enh: rsma checkpoint	2023-11-08 20:45:44 +08:00
kailixu	e72582a1fa	Merge branch 'fix/streamTransferError' into enh/TD-23769-3.0x	2023-11-08 20:45:21 +08:00
yihaoDeng	3abf0ae4b2	Merge branch 'feat/TD-26060' into enh/refactorBackend	2023-11-08 17:31:16 +08:00
yihaoDeng	92fa1043b8	Merge branch 'enh/refactorBackend' of https://github.com/taosdata/TDengine into enh/refactorBackend	2023-11-08 17:12:32 +08:00
yihaoDeng	2de5c4a97d	add bg thread	2023-11-08 17:11:11 +08:00
yihaoDeng	76fcc49c6d	fix stream transfer error	2023-11-08 17:10:54 +08:00
Haojun Liao	f949333e8b	refactor:do some internal refactor.	2023-11-08 16:57:55 +08:00
Haojun Liao	470f244032	Merge remote-tracking branch 'origin/enh/refactorBackend' into enh/refactorBackend ... # Conflicts: # include/libs/stream/tstream.h	2023-11-08 15:21:21 +08:00
Haojun Liao	99dbbe61d7	refactor(stream): do some internal refactor.	2023-11-08 15:20:47 +08:00
yihaoDeng	6a82cd7fbf	Merge branch 'main' of https://github.com/taosdata/TDengine into fix/fixStreamTransferErrMain	2023-11-08 15:14:31 +08:00
yihaoDeng	b3c056e6cb	fix stream transfer err	2023-11-08 15:12:47 +08:00
wangmm0220	c4c1423243	fix:error	2023-11-08 15:01:04 +08:00
yihaoDeng	7e03fa6664	fix stream transfer err	2023-11-08 14:57:20 +08:00
yihaoDeng	2e7d041e0a	fix stream transfer err	2023-11-08 14:48:28 +08:00
liuyao	340fb3e727	Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898	2023-11-08 14:16:59 +08:00
yihaoDeng	7b0891981e	merge 3.0	2023-11-08 14:14:01 +08:00
wangmm0220	7d19fcf19f	Merge branch 'feat/TD-26060' of https://github.com/taosdata/TDengine into feat/TD-26060	2023-11-08 11:59:03 +08:00
wangmm0220	047eb14f67	fix:error	2023-11-08 11:57:49 +08:00
yihaoDeng	fb77d811f9	Merge branch 'enh/refactorBackend' of https://github.com/taosdata/TDengine into enh/refactorBackend	2023-11-08 10:48:36 +08:00
Haojun Liao	17bd6badec	refactor(stream): do checkpoint for each task.	2023-11-08 10:47:54 +08:00
yihaoDeng	7ada7f34ac	fix:merge from 3.0	2023-11-08 10:45:15 +08:00
wangmm0220	07aa749fcd	fix:adapt to windows	2023-11-08 10:28:45 +08:00
yihaoDeng	594da2d603	merge 3.0	2023-11-08 10:04:46 +08:00
kailixu	b9f389a49f	Merge branch '3.0' into enh/TD-23769-3.0x	2023-11-08 08:36:35 +08:00
kailixu	f2d24306b1	enh: rsma tasks share one checkpoint and fix memory leak	2023-11-08 08:35:52 +08:00
Haojun Liao	e052d3cd62	fix(stream): check null.	2023-11-08 00:10:08 +08:00
kailixu	6b738884a4	chore: rsma checkpoint	2023-11-07 19:59:05 +08:00
wangmm0220	f0b2ed567c	fix:upload checkpoint to s3	2023-11-07 18:49:12 +08:00
Haojun Liao	73d75aac25	fix(stream): add ref for task.	2023-11-07 18:43:35 +08:00
Haojun Liao	26b96b149d	refactor: wait for a while when no quota available.	2023-11-07 18:43:35 +08:00
Haojun Liao	0979ff944b	fix(stream): update the fill-time for quota limitation.	2023-11-07 18:43:34 +08:00
Haojun Liao	2bdde122fc	refactor: wait a while when no token available.	2023-11-07 18:43:34 +08:00
Haojun Liao	a0d81734f1	fix(stream): add ref for task.	2023-11-07 18:18:58 +08:00
Haojun Liao	43b14ba8b4	refactor: wait for a while when no quota available.	2023-11-07 17:49:49 +08:00
Haojun Liao	06d02b862e	fix(stream): update the fill-time for quota limitation.	2023-11-07 17:33:30 +08:00
Haojun Liao	0b59681158	refactor: wait a while when no token available.	2023-11-07 17:18:58 +08:00
Haojun Liao	884770dab1	fix(stream): fix syntax error.	2023-11-07 17:12:06 +08:00
Haojun Liao	344c3c5b13	fix(stream): fix syntax error.	2023-11-07 17:11:36 +08:00
Haojun Liao	9c49de69f8	refactor: do some internal refactor.	2023-11-07 17:05:20 +08:00
Haojun Liao	ed0c580d11	refactor: do some internal refactor.	2023-11-07 17:04:55 +08:00
Haojun Liao	1301b92844	fix(stream): fix errors in scan-history, introduced by refactor	2023-11-07 16:41:15 +08:00
Haojun Liao	c9a1cc50fb	refactor: do some internal refactor.	2023-11-07 16:40:04 +08:00
Haojun Liao	9125fe44c6	fix(stream): fix error.	2023-11-07 16:40:04 +08:00
Haojun Liao	8c2f6bab0e	fix(stream): exec scan-history in timer.	2023-11-07 16:40:03 +08:00
Haojun Liao	7370daa6e0	fix(stream): record the start failure tasks.	2023-11-07 16:39:56 +08:00
Haojun Liao	e38e526dda	fix(stream):reset the starting flag for the followers.	2023-11-07 16:39:12 +08:00
Haojun Liao	207b53f3f6	fix(stream): fix errors in scan-history, introduced by refactor	2023-11-07 16:19:09 +08:00
yihaoDeng	4f53ffd76f	refactor backend	2023-11-07 16:01:57 +08:00
yihaoDeng	1ee82f5e2c	refactor backend	2023-11-07 15:50:42 +08:00
liuyao	467c27c758	recover flush mark	2023-11-07 15:45:38 +08:00
yihaoDeng	8f7ad2bd90	refactor backend	2023-11-07 15:20:58 +08:00
yihaoDeng	042522ac7a	refactor backend	2023-11-07 15:18:24 +08:00
yihaoDeng	e015134d6d	refactor backend	2023-11-07 14:46:52 +08:00
liuyao	e972ab16fe	recover flush mark	2023-11-07 14:11:08 +08:00
yihaoDeng	e5449bf161	refactor backend	2023-11-07 11:49:56 +08:00
yihaoDeng	95cfc5eb46	refactor checkpoint	2023-11-07 09:55:55 +08:00
yihaoDeng	021859361f	refactor checkpoint	2023-11-07 09:22:57 +08:00
Haojun Liao	96f9264906	fix(stream): add null ptr check.	2023-11-07 00:38:02 +08:00
Haojun Liao	1312b9ea36	fix(stream): fix error.	2023-11-06 23:44:01 +08:00
Haojun Liao	4d8548e938	refactor: do some internal refactor.	2023-11-06 23:42:16 +08:00
yihaoDeng	c9bf4f6507	refactor checkpoint	2023-11-06 20:19:11 +08:00
yihaoDeng	819be65db0	refactor checkpoint	2023-11-06 19:59:11 +08:00
kailixu	f5d796a081	chore: print stream state	2023-11-06 19:53:09 +08:00
Haojun Liao	312dbc1caa	refactor: do some internal refactor.	2023-11-06 19:49:25 +08:00
yihaoDeng	9ca1151b67	refactor checkpoint	2023-11-06 19:47:47 +08:00
Haojun Liao	1f09d3a46b	fix(stream): fix error.	2023-11-06 19:46:47 +08:00
Haojun Liao	d9474016fc	fix(stream): exec scan-history in timer.	2023-11-06 18:55:53 +08:00
yihaoDeng	aee1b700a6	Merge branch 'enh/new3.0' into enh/refactorBackend	2023-11-06 18:08:03 +08:00
yihaoDeng	ba52769867	Merge branch 'enh/new3.0' into enh/refactorBackend	2023-11-06 17:56:09 +08:00
Haojun Liao	6b1889284b	fix(stream): record the start failure tasks.	2023-11-06 15:45:44 +08:00
yihaoDeng	feeead50c5	Merge branch 'enh/new3.0' into enh/refactorBackend	2023-11-06 13:26:38 +08:00
Haojun Liao	69863febe6	fix(stream):reset the starting flag for the followers.	2023-11-06 13:17:13 +08:00
Haojun Liao	d41fd27189	fix(stream): restart tasks in stream threads, instead of write thread.	2023-11-06 11:57:21 +08:00
Haojun Liao	0e44950a37	enh(stream): execute scan history by using time slice 1s.	2023-11-06 09:21:33 +08:00
Haojun Liao	1f756f58f6	fix(stream): add need update node into update node list.	2023-11-06 09:21:33 +08:00
Haojun Liao	680eb5674e	refactor: do some internal refactor.	2023-11-06 09:15:49 +08:00
Haojun Liao	d440da0d8c	enh(stream): execute scan history by using time slice 1s.	2023-11-05 01:26:53 +08:00
Haojun Liao	c71f6713a7	fix(stream): add need update node into update node list.	2023-11-05 01:24:23 +08:00
Haojun Liao	e13f003b90	refactor: do some internal refactor.	2023-11-05 01:23:23 +08:00
kailixu	411151d671	fix: buffer overflow/buffer use after free/memory leak	2023-11-04 08:44:26 +08:00
yihaoDeng	f038f88c07	Merge branch 'feat/TD-26060' into enh/refactorBackend	2023-11-03 18:29:18 +08:00
wangmm0220	5a6c3f2b04	fix:add windows support	2023-11-03 17:45:37 +08:00
kailixu	77eadc59dc	Merge branch '3.0' into enh/TD-23769-3.0x	2023-11-03 16:26:07 +08:00
kailixu	a48968e5e8	enh: rsma checkpoint	2023-11-03 14:33:45 +08:00
Haojun Liao	53b2158c54	refactor: do some internal refactor.	2023-11-03 13:51:17 +08:00
wangmm0220	0a1f1085fc	Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-26060	2023-11-03 11:33:50 +08:00
wangmm0220	3ac11eba0e	fix:modify upload checkpoint interface	2023-11-03 11:33:18 +08:00
yihaoDeng	d82d51cf8f	Merge branch 'enh/new3.0' into enh/refactorBackend	2023-11-03 10:40:00 +08:00
yihaoDeng	24e961976f	Merge branch 'enh/new3.0' into enh/refactorBackend	2023-11-03 09:31:57 +08:00
Haojun Liao	9d77351bb2	refactor: do some internal refactor.	2023-11-03 09:30:50 +08:00
yihaoDeng	069dd8de84	Merge branch 'enh/new3.0' into enh/refactorBackend	2023-11-03 09:28:30 +08:00
liuyao	31ad73f6d1	Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898	2023-11-03 09:12:17 +08:00
yihaoDeng	0a73832fd2	add checkpoint delta	2023-11-02 20:39:55 +08:00
kailixu	6c944bb192	enh: skip rsma_task during load stream tasks	2023-11-02 20:32:54 +08:00
wangmm0220	159873692c	feat:checkpoint bakeup using rsync	2023-11-02 20:13:29 +08:00
Haojun Liao	b3d57e4a2d	fix(stream): not transfer state if state is not appropriate.	2023-11-02 19:31:35 +08:00
Haojun Liao	7a32b3a209	fix(stream): not transfer state if state is not appropriate.	2023-11-02 19:12:36 +08:00
Haojun Liao	706f1e4744	fix(stream): fix deadlock in transfer state.	2023-11-02 18:18:04 +08:00
Haojun Liao	39175dc03b	fix(stream): fix deadlock in transfer state.	2023-11-02 18:07:36 +08:00
Haojun Liao	c48c801e19	fix(stream): return error code.	2023-11-02 11:02:40 +08:00
Haojun Liao	95efa07e92	fix(stream): add full unsupported event filtering.	2023-11-02 11:02:39 +08:00
Haojun Liao	05b3d51803	fix(stream): add qualified status in assert	2023-11-02 11:02:39 +08:00
Haojun Liao	420825c62f	fix(stream): return error code.	2023-11-02 10:22:03 +08:00
yihaoDeng	7e7a39f077	Merge branch 'enh/new3.0' into enh/refactorBackend	2023-11-02 10:15:34 +08:00
yihaoDeng	b911492270	Merge branch 'enh/new3.0' into enh/refactorBackend	2023-11-02 09:46:36 +08:00
Haojun Liao	351a31302f	fix(stream): add full unsupported event filtering.	2023-11-02 09:43:48 +08:00
Haojun Liao	e14d42c01e	fix(stream): add qualified status in assert	2023-11-02 08:46:18 +08:00
Haojun Liao	dc8a90c864	fix(stream): fix memory leak.	2023-11-02 08:13:11 +08:00
Haojun Liao	79e971385f	fix(stream): fix memory leak.	2023-11-02 08:11:51 +08:00
Haojun Liao	df7633650e	fix(stream): fix memory leak.	2023-11-01 20:35:45 +08:00
Haojun Liao	e1b3e00b3d	fix(stream): fix memory leak.	2023-11-01 20:10:20 +08:00
Haojun Liao	6dc468c55f	fix(stream): fix dead-lock	2023-11-01 19:38:13 +08:00
Haojun Liao	fa7ebf63ea	refactor: do some internal refactor.	2023-11-01 19:38:13 +08:00
Haojun Liao	9156e8449d	fix(stream): fix deference null ptr.	2023-11-01 19:38:13 +08:00
Haojun Liao	7cf90dde5c	refactor: do some internal refactor.	2023-11-01 19:38:13 +08:00
Haojun Liao	00f029e44f	refactor: add some logs.	2023-11-01 19:38:12 +08:00
Haojun Liao	385e699cc2	fix(stream): do some internal refactor.	2023-11-01 19:38:12 +08:00
Haojun Liao	bd12ae88e1	fix(stream):fix the bug when event in waiting list not fulfilled.	2023-11-01 19:38:12 +08:00
Haojun Liao	9a9c1d689d	fix(stream): add lock log.	2023-11-01 19:38:12 +08:00
Haojun Liao	e7609d8e56	fix(stream): fix dead-lock	2023-11-01 19:09:21 +08:00
Haojun Liao	ae25f09c45	refactor: do some internal refactor.	2023-11-01 18:03:44 +08:00
Haojun Liao	899d13b4ae	fix(stream): fix deference null ptr.	2023-11-01 17:25:59 +08:00
Haojun Liao	246ed4e022	refactor: do some internal refactor.	2023-11-01 17:19:21 +08:00
liuyao	505c44d9aa	Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-21898	2023-11-01 16:50:46 +08:00
Haojun Liao	12bc4f32cf	refactor: add some logs.	2023-11-01 16:12:15 +08:00
yihaoDeng	4c8edb5545	Merge branch 'enh/new3.0' into enh/refactorBackend	2023-11-01 16:07:49 +08:00
Haojun Liao	778c29b808	fix(stream): do some internal refactor.	2023-11-01 14:07:18 +08:00
Haojun Liao	e4aeea176b	fix(stream):fix the bug when event in waiting list not fulfilled.	2023-11-01 13:59:51 +08:00
Haojun Liao	5ff89bc098	fix(stream): add lock log.	2023-11-01 13:19:53 +08:00
Haojun Liao	3f561fb8e4	fix(stream): remove invalid assert.	2023-11-01 11:43:02 +08:00
Haojun Liao	3e6c89d8eb	fix(stream): add unsupport trans.	2023-11-01 11:43:02 +08:00
kailixu	1530fe0c49	Merge branch '3.0' into enh/TD-23769-3.0x	2023-11-01 09:52:16 +08:00
kailixu	ad1e6accd4	chore: build checkpoint for rsma	2023-11-01 09:45:58 +08:00
Haojun Liao	0fb4cfd94f	fix(stream): remove invalid assert.	2023-11-01 00:33:17 +08:00
Haojun Liao	639d5cde47	fix(stream): add unsupport trans.	2023-10-31 23:14:02 +08:00
yihaoDeng	bb8252df16	fix mem leak	2023-10-31 20:59:24 +08:00
Haojun Liao	294c59150c	fix(stream): add jump out loop by checking task status.	2023-10-31 19:31:28 +08:00
Haojun Liao	4694859325	fix(stream): add jump out loop by checking task status.	2023-10-31 19:31:00 +08:00
Haojun Liao	0b18192b23	fix(stream): fix error in finishing event handling.	2023-10-31 16:40:16 +08:00
Haojun Liao	762cfef498	fix(stream): adjust the stop condition for stream tasks.	2023-10-31 16:40:16 +08:00
Haojun Liao	4c0242a58d	fix(stream): fix error in finishing event handling.	2023-10-31 16:39:14 +08:00
Haojun Liao	e01c7f3cfd	fix(stream): adjust the stop condition for stream tasks.	2023-10-31 11:51:47 +08:00
Haojun Liao	88469568b8	fix(stream): handle failure during checkpoint	2023-10-31 10:45:14 +08:00
Haojun Liao	635243f01a	refactor: do some internal refactor.	2023-10-31 10:45:12 +08:00
Haojun Liao	0c982093cd	fix(stream): enable the set of fill-history task to be un-init.	2023-10-31 10:45:12 +08:00
Haojun Liao	86dfc52a6e	fix(stream): add event check when handling the end of event.	2023-10-31 10:45:12 +08:00
Haojun Liao	72f3e4446d	fix(stream): handle failure during checkpoint	2023-10-31 10:38:17 +08:00
yihaoDeng	7fad855247	fix mem leak	2023-10-30 18:16:12 +08:00
yihaoDeng	ee6b620597	fix mem leak	2023-10-30 17:29:32 +08:00
Haojun Liao	fdf3f210ac	refactor: do some internal refactor.	2023-10-30 14:39:24 +08:00
Haojun Liao	93232a0e14	fix(stream): enable the set of fill-history task to be un-init.	2023-10-30 11:48:51 +08:00
Haojun Liao	19c5882cbe	fix(stream): add event check when handling the end of event.	2023-10-30 10:22:24 +08:00
Haojun Liao	980f6fc46e	fix(stream): pause the task when they are still in uninit status.	2023-10-30 09:32:40 +08:00
Haojun Liao	1bbad532a7	fix(stream): add more check.	2023-10-30 09:32:40 +08:00
Haojun Liao	076b533ba3	fix(stream): fix race condition when state transferring.	2023-10-30 09:32:40 +08:00
Haojun Liao	85befa10b9	fix(stream): avoid clear the restart flag .	2023-10-30 09:32:39 +08:00
Haojun Liao	f708ddb792	fix(stream): pause the task when they are still in uninit status.	2023-10-30 09:09:21 +08:00
Haojun Liao	790d869494	fix(stream): add more check.	2023-10-29 22:57:33 +08:00
Haojun Liao	3c07c3ec2f	fix(stream): fix race condition when state transferring.	2023-10-29 02:27:27 +08:00
Haojun Liao	c7285c0ede	fix(stream): add expired epset node list for stream tasks in hb to mnode.	2023-10-27 23:41:57 +08:00
Haojun Liao	4c267b538f	fix(stream): add expired epset node list for stream tasks in hb to mnode.	2023-10-27 23:05:41 +08:00
Haojun Liao	62cc33715c	fix(stream): fix deadlock.	2023-10-27 15:05:36 +08:00
Haojun Liao	0c2079a963	fix(stream): fix deadlock.	2023-10-27 15:03:21 +08:00
yihaoDeng	ebdf6d1cc6	fix mem leak	2023-10-27 14:18:56 +08:00
Haojun Liao	afe7fdf672	fix(stream): avoid clear the restart flag .	2023-10-27 14:12:06 +08:00
Haojun Liao	dcca6b0868	other: merge 3.0.	2023-10-27 11:41:28 +08:00
Haojun Liao	29b3a7a5e9	refactor: do some internal refactor.	2023-10-27 09:59:53 +08:00
Haojun Liao	9c9d02b181	fix(stream): disable follower nodes exec reset.	2023-10-27 09:59:52 +08:00
Haojun Liao	52d8e05c0c	fix(stream): resolve the nodeUpdate msg compatible issue.	2023-10-27 09:59:52 +08:00
Haojun Liao	454624516c	fix(stream): handle the repeatly issued nodeUpdate msg from dnode.	2023-10-27 09:59:51 +08:00
Haojun Liao	9c9cd08e2e	fix(stream): not clear role flag.	2023-10-27 09:59:44 +08:00
Haojun Liao	33dc51cc86	fix(stream): add some logs.	2023-10-27 09:59:44 +08:00
Haojun Liao	1ccbb77981	fix(stream): fix dead lock when shutdown all tasks.	2023-10-27 09:59:37 +08:00
Haojun Liao	9a1a8feec2	fix(stream): do send check info if downstream is not leader.	2023-10-27 09:58:53 +08:00
Haojun Liao	2493c21ed3	fix(stream): fix the invalid write in sma	2023-10-27 09:58:52 +08:00
Haojun Liao	e8520997b0	fix(stream): disable continue check for downstream tasks.	2023-10-27 09:58:52 +08:00
Haojun Liao	3bcffcf1da	other: merge stream refactor from main.	2023-10-27 09:56:02 +08:00
Haojun Liao	b0ef072556	fix(stream): clear flag for checkpoint.	2023-10-27 09:55:07 +08:00
Haojun Liao	f682ad0479	refactor: do some internal refactor.	2023-10-27 09:55:06 +08:00
Haojun Liao	efede8669f	refactor: do some internal refactor.	2023-10-27 09:55:06 +08:00
Haojun Liao	9ebf806789	fix(stream): clear the flag.	2023-10-27 09:55:05 +08:00
Haojun Liao	48ff521968	fix(stream): update the nodeEp even during checkpoint trans.	2023-10-27 09:55:02 +08:00
Haojun Liao	0d4892f25b	fix(stream): fix bugs caused by refactor sm.	2023-10-27 09:55:00 +08:00
Haojun Liao	a31613ae95	refactor: add ref in log.	2023-10-27 09:54:59 +08:00
Haojun Liao	226b6596d6	fix(stream): fix bugs caused by refactor.	2023-10-27 09:54:58 +08:00
Haojun Liao	091238e7be	fix(stream): fix bugs caused by refactor sm.	2023-10-27 09:54:58 +08:00
Haojun Liao	e1f6043eed	fix(stream): fix bugs caused by refactor sm.	2023-10-27 09:54:57 +08:00
Haojun Liao	e359da7b19	fix(stream): fix bugs caused by refactor.	2023-10-27 09:54:57 +08:00
Haojun Liao	8729bdac1e	fix(stream): fix bugs caused by refactor.	2023-10-27 09:54:57 +08:00
Haojun Liao	1eac7f44c0	fix(stream): fix error in stream.	2023-10-27 09:54:56 +08:00
Haojun Liao	579dd8e6b2	refactor(stream): do some internal refactor.	2023-10-27 09:54:56 +08:00
Haojun Liao	6f3c0d17cc	refactor(stream): add state machine to manage the state of stream tasks.	2023-10-27 09:54:49 +08:00
Haojun Liao	596eb073b7	refactor(stream): do some internal refactor.	2023-10-27 09:54:48 +08:00
Haojun Liao	91e3fd66e6	enh(stream): sm for stream tasks.	2023-10-27 09:54:44 +08:00
Haojun Liao	2409a17517	refactor(stream): do some internal refactor.	2023-10-27 09:54:44 +08:00
Haojun Liao	c9bdb56d79	refactor: do some internal refactor.	2023-10-27 09:54:09 +08:00
Haojun Liao	164159f293	enh(stream): add parameter to limit the stream sink task.	2023-10-27 09:54:07 +08:00
Haojun Liao	62c58ecd8b	fix(stream):add more check for test cases.	2023-10-27 09:54:06 +08:00
Haojun Liao	a4f6042966	fix(stream): fix error in creating update node trans.	2023-10-27 09:52:14 +08:00
Haojun Liao	ffa798c4fa	enh(stream): add some logs.	2023-10-27 09:52:14 +08:00
yihaoDeng	7e1e68d8f5	fix stream case error	2023-10-26 18:27:09 +08:00
yihaoDeng	87db93e72e	fix stream case error	2023-10-26 11:36:29 +08:00
liuyao	d0e4a5d0f7	opt max delay	2023-10-25 20:29:25 +08:00
yihaoDeng	db0f6258fe	fix stream case error	2023-10-25 19:38:22 +08:00
liuyao	f51c985914	stream event window	2023-10-25 17:50:16 +08:00
Haojun Liao	f36b24ed24	fix(stream): disable follower nodes exec reset.	2023-10-25 14:16:36 +08:00
Haojun Liao	985371e841	fix(stream): resolve the nodeUpdate msg compatible issue.	2023-10-25 10:01:20 +08:00
Haojun Liao	03165d5327	fix(stream): handle the repeatly issued nodeUpdate msg from dnode.	2023-10-25 09:55:02 +08:00
Haojun Liao	26aaa4a83c	other: merge main.	2023-10-25 00:39:14 +08:00
Haojun Liao	5b42204dba	fix(stream): not clear role flag.	2023-10-25 00:32:46 +08:00
Haojun Liao	13d979a2ee	fix(stream): add some logs.	2023-10-25 00:03:46 +08:00
Haojun Liao	8cde39eebd	fix(stream): fix dead lock when shutdown all tasks.	2023-10-24 19:11:24 +08:00
yihaoDeng	d43b3b4a32	refactor stream backend	2023-10-24 16:17:54 +08:00
yihaoDeng	b3c54a0f06	refactor stream backend	2023-10-24 16:04:43 +08:00
Haojun Liao	6ffb5ee476	fix(stream): do send check info if downstream is not leader.	2023-10-24 15:49:50 +08:00
Haojun Liao	f1498f8929	fix(stream): fix the invalid write in sma	2023-10-24 15:24:34 +08:00
Haojun Liao	0f9328330b	fix(stream): disable continue check for downstream tasks.	2023-10-24 11:56:36 +08:00
yihaoDeng	d46cf878eb	refactor stream backend	2023-10-24 11:53:52 +08:00
Haojun Liao	7a8b046a35	fix(sma): fix bugs in sma and do some internal refactor.	2023-10-24 09:40:38 +08:00
yihaoDeng	09192acf96	fix mem leak	2023-10-23 10:20:57 +08:00
Haojun Liao	059e94e428	fix(stream): clear flag for checkpoint.	2023-10-22 01:25:00 +08:00
Haojun Liao	4826b34a2b	refactor: do some internal refactor.	2023-10-21 16:32:44 +08:00
Haojun Liao	06cd67e26e	refactor: do some internal refactor.	2023-10-21 16:20:03 +08:00
Haojun Liao	21e98e358e	fix(stream): clear the flag.	2023-10-21 02:41:03 +08:00
Haojun Liao	d1095e2c4c	Merge remote-tracking branch 'origin/refact/streamsm' into refact/streamsm	2023-10-21 01:42:59 +08:00
Haojun Liao	49014d4b68	fix(stream): update the nodeEp even during checkpoint trans.	2023-10-21 01:42:43 +08:00
yihaoDeng	aaf7e070ee	Merge branch 'refact/streamsm' of https://github.com/taosdata/TDengine into refact/streamsm	2023-10-20 20:49:17 +08:00
yihaoDeng	e95735eea4	fix mem leak	2023-10-20 20:49:01 +08:00
Haojun Liao	c68c0eda79	fix(stream): fix bugs caused by refactor sm.	2023-10-20 16:27:39 +08:00
Haojun Liao	4feebffb05	refactor: add ref in log.	2023-10-20 15:59:43 +08:00
Haojun Liao	c4268ca395	fix(stream): fix bugs caused by refactor.	2023-10-20 15:54:30 +08:00
yihaoDeng	1c3c58062d	Merge branch 'enh/new3.0' into enh/refactorBackend	2023-10-20 15:40:47 +08:00
yihaoDeng	1a4cfd11a5	Merge branch 'enh/new3.0' into enh/refactorBackend	2023-10-20 15:28:10 +08:00
Haojun Liao	3b8c85f632	fix(stream): fix bugs caused by refactor sm.	2023-10-20 14:19:57 +08:00
Haojun Liao	270a847809	fix(stream): fix bugs caused by refactor sm.	2023-10-20 10:23:48 +08:00
wade zhang	628667f22c	Merge pull request #23231 from taosdata/FEAT/TS-3816-3.0 ... feat: support restoring vnodes on replaced disks as needed	2023-10-20 09:02:25 +08:00
Haojun Liao	4d5d078fbe	fix(stream): fix bugs caused by refactor.	2023-10-19 18:00:54 +08:00
Haojun Liao	d5059826a8	fix(stream): fix bugs caused by refactor.	2023-10-19 17:59:04 +08:00
Haojun Liao	c1cebae6ba	fix(stream): fix error in stream.	2023-10-19 16:20:27 +08:00
Haojun Liao	614bdd9fb9	Merge pull request #23220 from taosdata/feat/TD-26644 ... opt bloom filter	2023-10-19 15:34:43 +08:00
Haojun Liao	da8f6a4c2f	Merge pull request #23248 from taosdata/feat/TD-26647 ... recover session buff state	2023-10-19 15:34:03 +08:00
Haojun Liao	402aefd95d	refactor(stream): do some internal refactor.	2023-10-19 15:09:47 +08:00
Benguang Zhao	c4e9069a66	fix: set nextProcessedVer properly in tqProcessTaskScanHistory	2023-10-19 11:18:57 +08:00
Haojun Liao	f4caeca24a	refactor(stream): add state machine to manage the state of stream tasks.	2023-10-19 10:34:20 +08:00
Haojun Liao	54ec679b58	refactor(stream): do some internal refactor.	2023-10-18 15:34:53 +08:00
yihaoDeng	c6ba2ca205	fix transfer crash	2023-10-18 14:49:12 +08:00
yihaoDeng	d6840ae07f	fix transfer crash	2023-10-18 14:22:52 +08:00
yihaoDeng	1b755dd4e7	fix transfer crash	2023-10-18 14:22:33 +08:00
yihaoDeng	e49b9b2276	fix transfer crash	2023-10-18 11:34:11 +08:00
Haojun Liao	2904cfd4d5	Merge branch 'main' into refact/streamsm	2023-10-18 10:29:09 +08:00
Haojun Liao	5115a1ac88	Merge pull request #23304 from taosdata/fix/TD-26771 ... double free	2023-10-18 10:09:17 +08:00
yihaoDeng	d67fdb5e46	fix transfer crash	2023-10-17 21:08:31 +08:00
yihaoDeng	2004c1a346	fix transfer crash	2023-10-17 18:19:52 +08:00
Haojun Liao	64ed7e49a2	enh(stream): sm for stream tasks.	2023-10-17 13:47:55 +08:00
yihaoDeng	77542cc461	fix stream snap deadlock	2023-10-17 11:23:39 +08:00
Haojun Liao	7b7ce10475	refactor(stream): do some internal refactor.	2023-10-17 11:23:04 +08:00
Haojun Liao	14035c763c	refactor: do some internal refactor.	2023-10-17 09:57:13 +08:00
Haojun Liao	fdb6ec1fa3	enh(stream): add parameter to limit the stream sink task.	2023-10-16 22:52:39 +08:00
liuyao	990d36654d	release buff	2023-10-16 19:49:45 +08:00
Haojun Liao	a1aa2c9e0c	fix(stream):add more check for test cases.	2023-10-16 18:47:01 +08:00
Haojun Liao	67bee17e5e	other: merge main.	2023-10-16 09:55:41 +08:00
Haojun Liao	f90cf54714	fix(stream): fix error in creating update node trans.	2023-10-15 21:56:49 +08:00
Haojun Liao	6611f216ac	enh(stream): add some logs.	2023-10-13 19:45:29 +08:00
liuyao	7269dedb39	Merge branch '3.0' of https://github.com/taosdata/TDengine into 3.0	2023-10-13 11:35:01 +08:00
liuyao	429125be93	session state recover	2023-10-13 11:32:35 +08:00
Haojun Liao	33d9f772be	other: merge 3.0	2023-10-13 09:06:13 +08:00
Haojun Liao	84c5359d5b	other: merge 3.0	2023-10-13 08:58:26 +08:00
yihaoDeng	d6ff282006	refactor backend	2023-10-12 20:42:36 +08:00
yihaoDeng	8c1a9f7c58	refactor backend	2023-10-12 20:28:40 +08:00
yihaoDeng	0ba7e2278a	refactor backend	2023-10-12 20:21:11 +08:00
yihaoDeng	a56d831a14	refactor backend	2023-10-12 20:16:41 +08:00
yihaoDeng	861d26e356	refactor backend	2023-10-12 15:24:50 +08:00
yihaoDeng	f704d4b82e	refactor backend	2023-10-12 11:45:18 +08:00
yihaoDeng	bfd205b3f7	refactor backend	2023-10-12 11:31:39 +08:00
Haojun Liao	eb4449999f	other: merge 3.0	2023-10-12 10:33:48 +08:00
yihaoDeng	4c94fbb44f	refactor backend	2023-10-11 20:26:19 +08:00
yihaoDeng	f264ffdcd6	refactor backend	2023-10-11 20:12:07 +08:00
yihaoDeng	83b52a4c1d	refactor backend	2023-10-11 20:02:40 +08:00
yihaoDeng	031506aef8	refactor backend	2023-10-11 19:46:35 +08:00
yihaoDeng	4f47162512	refactor backend	2023-10-11 16:36:43 +08:00
yihaoDeng	722d72659b	refactor backend	2023-10-11 16:10:16 +08:00
yihaoDeng	0baef335d7	refactor backend	2023-10-11 16:03:04 +08:00
yihaoDeng	4cff121a38	refactor backend	2023-10-11 14:46:41 +08:00
liuyao	e9fc079d26	opt bloom filter	2023-10-11 10:14:11 +08:00
Haojun Liao	1d4f0c8599	Merge pull request #23201 from taosdata/enh/TD-26646 ... Enh/td 26646	2023-10-10 22:59:37 +08:00
yihaoDeng	28421cd35e	refactor backend	2023-10-10 20:51:41 +08:00
Haojun Liao	05dfadd8f8	fix(stream): set initial version for fill-history task. disable reentry of scan-history task function.	2023-10-10 15:52:48 +08:00
Haojun Liao	5f9b858fc1	other: merge main.	2023-10-10 15:38:46 +08:00
yihaoDeng	b36163a197	add session iter	2023-10-10 15:12:21 +08:00
yihaoDeng	825341d90a	add sess iter	2023-10-10 14:35:02 +08:00
Haojun Liao	adf6f65d09	fix(stream): set initial version for fill-history task. disable reentry of scan-history task function.	2023-10-10 14:00:30 +08:00
Alex Duan	9da96ebf63	Merge pull request #23017 from taosdata/case/TD-26408-MAIN ... fix: add case for lost data after split vgroup (main)	2023-10-10 00:54:34 -05:00
Alex Duan	e566027cef	Merge pull request #23019 from taosdata/case/TD-26408-3.0 ... fix: add case for lost data after split 3.0	2023-10-10 00:53:56 -05:00
yihaoDeng	6adf7532a8	add sess iter	2023-10-10 11:35:18 +08:00
yihaoDeng	7ac4ef5beb	add sess iter	2023-10-10 10:41:46 +08:00
yihaoDeng	0fc6ffe6b6	change test case	2023-10-09 20:51:26 +08:00
Haojun Liao	a25d94ab6f	fix(stream): retry when error occurs during timer.	2023-10-09 20:10:28 +08:00
Haojun Liao	7d39164133	fix(stream): disable retrieving results during checkpoint procedure	2023-10-09 20:10:27 +08:00
Haojun Liao	af4618f958	fix(stream): remove invalid clear of rspmsg list.	2023-10-09 20:10:27 +08:00
Haojun Liao	847baa8b71	fix(stream): fix the pause failure bug.	2023-10-09 20:10:27 +08:00
Haojun Liao	21fb502690	enh(stream): add trans to reset task status to avoid being frozen in checkpoint status due to doing checkpoint failure of partial tasks.	2023-10-09 20:10:26 +08:00
Haojun Liao	64bde93f85	fix(stream): not idle for sink task when no tokens are available.	2023-10-09 20:10:25 +08:00
Haojun Liao	9a7167aaba	fix(stream): sleep for 10ms	2023-10-09 20:10:24 +08:00
Haojun Liao	9af8ce21ed	fix(stream): wait for a while when no available token.	2023-10-09 20:10:23 +08:00
Haojun Liao	4780474877	fix(stream): continue	2023-10-09 20:10:23 +08:00
Haojun Liao	da833ccb30	refactor: inc the dump batch.	2023-10-09 20:10:23 +08:00
Haojun Liao	2936ac9b89	fix(stream): add missing refactor.	2023-10-09 20:10:22 +08:00
Haojun Liao	af2c0c95e9	refactor: do some internal refactor.	2023-10-09 20:10:22 +08:00
Haojun Liao	940d6a3cb4	fix(stream): check items in queue and qall, stead of only the queue items.	2023-10-09 20:10:21 +08:00
Haojun Liao	e95aea84ae	fix(stream): fix syntax error.	2023-10-09 20:10:21 +08:00
Haojun Liao	59e284c332	fix(stream): quit from loop when input queue is full.	2023-10-09 20:10:21 +08:00
Haojun Liao	9d5a3b8d78	other: merge stream fix.	2023-10-09 20:10:13 +08:00
Haojun Liao	c5997bee5b	fix(stream): update the current processing version.	2023-10-09 20:09:23 +08:00
Haojun Liao	28008f8263	enh(stream): add more info for sink task.	2023-10-09 20:09:22 +08:00
Haojun Liao	b3b9e89d30	fix(stream): fix the invalid check.	2023-10-09 20:09:22 +08:00
Haojun Liao	cf14772dd9	enh(stream): optimize the scan wal perf	2023-10-09 20:09:21 +08:00
Haojun Liao	d2160efa05	refactor: wait for 100ms before scan wal again.	2023-10-09 20:09:21 +08:00
Haojun Liao	932d93ab35	refactor: wait for more time for future data	2023-10-09 20:09:21 +08:00
Haojun Liao	7682c033ad	fix(stream): add null check.	2023-10-09 20:09:21 +08:00
Haojun Liao	0ab5d4b342	refactor: do some internal refactor.	2023-10-09 20:09:20 +08:00
Haojun Liao	c295ec322a	refactor: do some internal refactor.	2023-10-09 20:09:20 +08:00
Haojun Liao	d5abb21335	fix(stream): fix invalid check.	2023-10-09 20:09:20 +08:00
Haojun Liao	583ac1fc45	refactor: do some internal refactor.	2023-10-09 20:09:20 +08:00
Haojun Liao	83f84d92bc	refactor: do some internal refactor (discard the checkpoint source msg during restoring).	2023-10-09 20:09:10 +08:00
Haojun Liao	bb4ba54f28	fix(stream): enable new time range for stream task.	2023-10-09 20:09:09 +08:00
Haojun Liao	c759d90bf6	enh(stream): optimize the stream status when creating without fill-history option.	2023-10-09 20:09:09 +08:00
Haojun Liao	db897fb03a	fix(stream): opt perf.	2023-10-09 20:09:09 +08:00
Haojun Liao	0a2b70fd7f	fix(stream): fix syntax error.	2023-10-09 20:09:06 +08:00
Haojun Liao	8477271240	fix(stream): fix dead lock.	2023-10-09 20:09:00 +08:00
Haojun Liao	12eabb0c2c	log(stream): update logs.	2023-10-09 20:09:00 +08:00
Haojun Liao	b716f0d1fd	fix(stream): fix dead lock.	2023-10-09 20:08:57 +08:00
Haojun Liao	60a720efbb	log: update logs.	2023-10-09 20:08:53 +08:00
Haojun Liao	d10b07d23d	refactor: do some internal refactor.	2023-10-09 20:08:53 +08:00
Haojun Liao	6d2dbd549b	fix(stream): avoid repeatly free obj.	2023-10-09 20:08:53 +08:00
Haojun Liao	6b5770aab7	log(stream): update the log.	2023-10-09 20:08:52 +08:00
Haojun Liao	dfe8641c0c	refactor: optimize the fill-history task launch policy, and do some other internal refactor.	2023-10-09 20:08:52 +08:00
Haojun Liao	38164435f9	refactor: do some internal refactor.	2023-10-09 20:08:52 +08:00
Haojun Liao	f2848de12a	fix(stream): reduce the counter when remove stream tasks.	2023-10-09 20:08:52 +08:00
Haojun Liao	ecb3b44026	fix(stream): limit sink quota.	2023-10-09 20:08:52 +08:00
Haojun Liao	2c0e54f4f7	fix(stream): update the retry interval.	2023-10-09 20:08:13 +08:00
Haojun Liao	7f8da725bc	enh(stream): add quota limitation for sink task.	2023-10-09 20:08:13 +08:00
Haojun Liao	a248f3d088	refactor: do some internal refactor.	2023-10-09 20:08:12 +08:00
Haojun Liao	0db357401d	fix(stream): add new status for meta.	2023-10-09 20:08:12 +08:00
Haojun Liao	3912d80c02	fix(stream): do some internal refactor.	2023-10-09 20:08:11 +08:00
Haojun Liao	feb09c8361	fix(stream): fix error in pause scan-history data.	2023-10-09 20:08:11 +08:00
Haojun Liao	6a98b11bef	fix(stream): fix invalid read and memory leak.	2023-10-09 20:08:11 +08:00
Haojun Liao	65f66f0986	fix(stream): transfer state when all downstream rsp already.	2023-10-09 20:08:11 +08:00
Haojun Liao	a32b56f381	fix(stream): fix memory leak.	2023-10-09 20:08:10 +08:00
Haojun Liao	4af7812f0b	fix(stream): fix the invalid counter for ready tasks in vnode, fix an invalid read.	2023-10-09 20:08:10 +08:00
Haojun Liao	4fe1382cd9	fix(stream): adjust nodeUpdate check interval.	2023-10-09 20:08:09 +08:00
Haojun Liao	f4f55fcbce	fix(stream): record the tasks started info, and do some internal refactor.	2023-10-09 20:08:09 +08:00
Haojun Liao	68b003c804	fix(stream): fix race condition.	2023-10-09 20:08:09 +08:00
Haojun Liao	fc96ec6bae	fix(stream): do not wait for the checkpoint finish before stop tasks.	2023-10-09 20:08:09 +08:00
Haojun Liao	7167c3c5de	fix(stream): add new timer for each task.	2023-10-09 20:08:08 +08:00
Haojun Liao	7336228c16	refactor: do some internal refactor.	2023-10-09 20:08:08 +08:00
Haojun Liao	477f57aa0e	refactor: do some internal refactor.	2023-10-09 20:08:08 +08:00
Haojun Liao	52763ca2a3	refactor: do some internal refactor.	2023-10-09 20:08:07 +08:00
Haojun Liao	3baddbe67a	fix(stream): update the checkpoint record start time.	2023-10-09 20:08:07 +08:00
Haojun Liao	a6ffb845c8	fix(stream): fix invalid checkpoint start time.	2023-10-09 20:08:07 +08:00
Haojun Liao	c05a8de3e8	fix(stream): rsp when checkpoint source failed.	2023-10-09 20:08:06 +08:00
Haojun Liao	b4bc4fac98	fix(stream): disable auto rsp for checkpoint source.	2023-10-09 20:08:06 +08:00
Haojun Liao	e671e87234	fix(stream): fix invalid assert.	2023-10-09 20:08:06 +08:00
Haojun Liao	d1a6b8bd22	refactor: do some internal refactor.	2023-10-09 20:08:06 +08:00
Haojun Liao	46b17aa9f1	fix(stream): fix race condition.	2023-10-09 20:08:06 +08:00
Haojun Liao	e49409a11e	fix(stream): add some logs.	2023-10-09 20:08:05 +08:00
Haojun Liao	5eee76df65	fix(stream): add msg identifier check.	2023-10-09 20:08:05 +08:00
Haojun Liao	192c47a7c2	fix(stream): add some logs.	2023-10-09 20:08:05 +08:00
Haojun Liao	693254f602	fix(stream): remove invalid free.	2023-10-09 20:08:04 +08:00
Haojun Liao	f3bd5c411d	fix(stream): remove the blocking flag after successfully retry sending msg.	2023-10-09 20:08:04 +08:00
Haojun Liao	ed9e035095	fix(stream): clear the failure vnode id list before send msg.	2023-10-09 20:08:04 +08:00
Haojun Liao	3726aa723e	fix(stream): fix invalid read.	2023-10-09 20:08:04 +08:00
Haojun Liao	e0d19af000	fix(stream): refactor re-try dispatch msg for stream tasks.	2023-10-09 20:08:03 +08:00
Haojun Liao	2b16976902	refactor(stream): refactor send msg.	2023-10-09 20:08:03 +08:00
Haojun Liao	27e00b1f19	fix(stream): add some logs, and remove invalid assert.	2023-10-09 20:08:02 +08:00
Haojun Liao	12687eb287	fix(stream): add check info detailed information.	2023-10-09 20:08:02 +08:00
Haojun Liao	ab7c18f604	refactor: do some internal refactor.	2023-10-09 20:08:01 +08:00
Haojun Liao	e5535fa1f5	fix(stream): handle dispatch checkpoint-trigger failure.	2023-10-09 20:08:01 +08:00
Haojun Liao	65cc4040f3	fix(stream): update logs.	2023-10-09 20:08:00 +08:00
Haojun Liao	6d3a134077	fix(stream): ignore the dispatch failure, and set the correct rsp counter.	2023-10-09 20:07:54 +08:00
Haojun Liao	7f4d9b6e41	fix(stream): wait for the task checkpoint before stop.	2023-10-09 20:07:53 +08:00
Haojun Liao	ef9efc8a8e	refactor: do refactor the sink task.	2023-10-09 20:07:53 +08:00
Haojun Liao	e4ac835554	fix(stream): disable token bucket.	2023-10-09 20:07:53 +08:00
Haojun Liao	a3d13d6634	fix(stream): remove invalid cached task info in buffer.	2023-10-09 20:07:08 +08:00
Haojun Liao	1f45bd82d2	other: merge stream fix in main.	2023-10-09 20:07:01 +08:00
Haojun Liao	d752fb1ed6	fix(stream): exec task in task queuee threads.	2023-10-09 20:06:11 +08:00
Haojun Liao	1ebafe4a1f	fix(stream): add more check.	2023-10-09 20:06:11 +08:00
Haojun Liao	c403a15f36	fix(stream): check the status before exec stream task when handling the dispatch requests.	2023-10-09 20:06:11 +08:00
Haojun Liao	d76e3ac718	fix(stream): add task status check.	2023-10-09 20:06:10 +08:00
Haojun Liao	e27111ecd1	enh(stream): add node stage check.	2023-10-09 20:06:09 +08:00
Haojun Liao	ce85945583	fix(stream): set correct size for results generated by scan history stream tasks.	2023-10-09 20:06:05 +08:00
Haojun Liao	053643be19	fix(stream): add timestamp.	2023-10-09 20:06:05 +08:00
Haojun Liao	b38f9e0288	fix(stream): update logs.	2023-10-09 20:06:05 +08:00
Haojun Liao	5c0b8ea804	enh(stream): log the checkpoint time.	2023-10-09 20:06:04 +08:00
Haojun Liao	b95ad74c7f	fix(stream): not handle the check msg for follower tasks.	2023-10-09 20:06:04 +08:00
Haojun Liao	e6fb9ee46c	fix(stream): reset task counter.	2023-10-09 20:06:04 +08:00
Haojun Liao	c3a3cf21fb	fix(stream): add some logs.	2023-10-09 20:05:39 +08:00
Haojun Liao	08b37dfc13	fix(stream): fix memory leak.	2023-10-09 20:05:38 +08:00
Haojun Liao	734f648891	fix(stream): set the output normal for stream dispatch when encountering the transfer state.	2023-10-09 20:05:38 +08:00
Haojun Liao	636eccc160	fix(stream): add null check	2023-10-09 20:05:37 +08:00
Haojun Liao	57bf1cadc4	fix(stream): fix error in decode stream task.	2023-10-09 20:05:37 +08:00
Haojun Liao	9d6199adf7	fix(stream): fix invalid read.	2023-10-09 20:05:37 +08:00
Haojun Liao	0459a4628c	fix(stream): fix stream task id error.	2023-10-09 20:05:36 +08:00
Haojun Liao	7ccc273206	other: merge stream fix in main.	2023-10-09 20:05:29 +08:00
Haojun Liao	98ef566eb4	fix(stream): remove fill-history sink task.	2023-10-09 20:04:17 +08:00
Haojun Liao	e84eeee6b8	refactor: do some internal refactor.	2023-10-09 20:04:17 +08:00
Haojun Liao	fe7864b976	fix(stream): fix error in drop task.	2023-10-09 20:04:17 +08:00
Haojun Liao	c33e521aa1	fix(stream): update some logs.	2023-10-09 20:04:17 +08:00
Haojun Liao	39e0c57323	fix(stream): set the correct guard flag.	2023-10-09 20:04:16 +08:00
Haojun Liao	3d1f29ff21	fix(stream): fix memory leak.	2023-10-09 20:04:16 +08:00
yihaoDeng	c6e9134cea	add sess iter	2023-10-09 20:02:23 +08:00
Haojun Liao	6fa54789cb	fix(stream): disable follower send hb to mnode. and do some internal refactor.	2023-10-09 19:59:31 +08:00
Haojun Liao	1256eafddb	other: merge stream fix.	2023-10-09 19:59:24 +08:00
Haojun Liao	283118bf68	fix(stream):fix missing unlock.	2023-10-09 19:56:17 +08:00
Haojun Liao	aaa12ed4f8	refactor: do some internal refactor about the sched status set	2023-10-09 19:56:17 +08:00
Haojun Liao	01d104706e	refactor(stream): remove unnecessary streamschedTask function to improve the performance.	2023-10-09 19:56:17 +08:00
Haojun Liao	a678e722ce	fix(stream): limit the token bucket capacity	2023-10-09 19:56:17 +08:00
Haojun Liao	b0ef091430	fix(stream): retry when error occurs during timer.	2023-10-09 16:40:05 +08:00
Haojun Liao	be0a2e4573	fix(stream): disable retrieving results during checkpoint procedure	2023-10-09 16:33:28 +08:00
yihaoDeng	2aa88dfe9b	fix mem leak	2023-10-09 15:53:42 +08:00
Haojun Liao	8266c4ff77	fix(stream): remove invalid clear of rspmsg list.	2023-10-09 15:26:42 +08:00
Haojun Liao	d87db2f829	fix(stream): fix the pause failure bug.	2023-10-09 14:48:11 +08:00
yihaoDeng	28b546a4ad	fix mem leak	2023-10-09 14:25:57 +08:00
yihaoDeng	be42143362	Merge branch '3.0' into case/TD-26408-3.0	2023-10-09 14:24:31 +08:00
yihaoDeng	3f9784f475	fix mem leak	2023-10-09 14:07:22 +08:00
yihaoDeng	0b92a6b1ea	fix mem leak	2023-10-09 11:16:16 +08:00
Haojun Liao	74ff0c5b50	Merge branch 'main' into fix/liaohj	2023-10-09 10:14:53 +08:00
Haojun Liao	ee6516f3c4	enh(stream): add trans to reset task status to avoid being frozen in checkpoint status due to doing checkpoint failure of partial tasks.	2023-10-09 10:13:53 +08:00
Haojun Liao	4b8cd3bccd	fix(stream): not idle for sink task when no tokens are available.	2023-10-09 10:07:19 +08:00
yihaoDeng	f49a2cbeb0	refact task backend	2023-10-08 18:14:17 +08:00
liuyao	1cbb8ef444	init bloomfilter array	2023-10-08 14:43:44 +08:00
yihaoDeng	b5994b31e2	fix mem leak	2023-10-08 13:33:02 +08:00
yihaoDeng	e3f18495e0	fix mem leak	2023-10-08 11:22:26 +08:00
yihaoDeng	2fa91341a7	refact task backend	2023-10-08 10:35:49 +08:00
Haojun Liao	23fb26798f	fix(stream): sleep for 10ms	2023-10-07 19:47:49 +08:00
Haojun Liao	e4e4c5a36e	fix(stream): wait for a while when no available token.	2023-10-07 19:41:49 +08:00
Haojun Liao	45bb3f50d7	fix(stream): continue	2023-10-07 19:25:40 +08:00
liuyao	95ffe6fa23	stream recover	2023-10-07 19:24:21 +08:00
Haojun Liao	62cdcbc7e3	refactor: inc the dump batch.	2023-10-07 19:20:48 +08:00
Haojun Liao	4d6f143146	fix(stream): add missing refactor.	2023-10-07 19:18:26 +08:00
Haojun Liao	ab17bfdf09	refactor: do some internal refactor.	2023-10-07 19:16:55 +08:00
Haojun Liao	0e8a8aabc1	fix(stream): check items in queue and qall, stead of only the queue items.	2023-10-07 19:16:25 +08:00
liuyao	1e323c0408	fix issue	2023-10-07 19:11:46 +08:00
yihaoDeng	98099ebbae	fix mem leak	2023-10-07 18:41:27 +08:00
liuyao	19bbf10579	fix issue	2023-10-07 17:59:53 +08:00
yihaoDeng	97fdcbc60b	fix mem leak	2023-10-07 17:48:24 +08:00
liuyao	1aaf873134	fix issue	2023-10-07 17:43:50 +08:00
yihaoDeng	bdc4afec83	refact task backend	2023-10-07 17:17:30 +08:00
yihaoDeng	e41da13d5b	refact task backend	2023-10-07 17:07:26 +08:00
yihaoDeng	3ebc7eef5f	refact task backend	2023-10-07 17:03:12 +08:00
yihaoDeng	c146327da8	refact task backend	2023-10-07 16:48:36 +08:00
Haojun Liao	d11c0a113c	fix(stream): fix syntax error.	2023-10-07 14:56:09 +08:00
Haojun Liao	7ce3d2c200	fix(stream): quit from loop when input queue is full.	2023-10-07 14:54:11 +08:00
yihaoDeng	66315fe405	refact task backend	2023-10-07 13:53:44 +08:00
yihaoDeng	82e0242889	refact task backend	2023-10-07 13:43:52 +08:00
yihaoDeng	fa70221238	refact task backend	2023-10-07 12:17:55 +08:00
liuyao	460ee86fdf	fix issue	2023-10-07 11:55:40 +08:00
liuyao	59d47e1e1f	fix issuse	2023-10-07 10:34:35 +08:00
Haojun Liao	115a284ac0	enh(stream): update the checkpoint interval	2023-10-07 09:56:03 +08:00
Haojun Liao	011627d50a	fix(stream): update the current processing version.	2023-10-07 09:07:44 +08:00
Haojun Liao	c20dd002fb	enh(stream): add more info for sink task.	2023-10-07 01:14:00 +08:00
Haojun Liao	98659ad323	fix(stream): fix the invalid check.	2023-10-05 00:09:47 +08:00
Haojun Liao	41ef1a45c1	enh(stream): optimize the scan wal perf	2023-10-04 22:06:46 +08:00
Haojun Liao	45e413f8d5	refactor: wait for 100ms before scan wal again.	2023-10-03 20:57:19 +08:00
Haojun Liao	132fb8e82e	refactor: wait for more time for future data	2023-10-03 15:42:00 +08:00
Haojun Liao	38aff641ee	fix(stream): add null check.	2023-10-03 02:39:57 +08:00
Haojun Liao	58f410a9df	refactor: do some internal refactor.	2023-10-03 02:37:52 +08:00
Haojun Liao	a543f4ca97	refactor: do some internal refactor.	2023-10-03 01:52:03 +08:00
Haojun Liao	5564eb215c	fix(stream): fix invalid check.	2023-10-03 00:38:24 +08:00
Haojun Liao	5718158fc0	refactor: do some internal refactor.	2023-10-01 22:27:29 +08:00
yihaoDeng	b8e265e502	add cvt state	2023-09-30 09:05:28 +00:00
Haojun Liao	31acbc6fe2	refactor: do some internal refactor (discard the checkpoint source msg during restoring).	2023-09-28 22:44:13 +08:00
yihaoDeng	eac16b9d1b	add cvt state	2023-09-28 11:59:53 +00:00
yihaoDeng	4f66dc538f	add cvt state	2023-09-28 11:52:18 +00:00
Haojun Liao	f2d8431619	fix(stream): enable new time range for stream task.	2023-09-28 18:26:52 +08:00
yihaoDeng	c5bb37a5cb	refact task backend	2023-09-28 17:30:49 +08:00
yihaoDeng	bb265887e6	refact task backend	2023-09-28 17:07:19 +08:00
yihaoDeng	54e3ac2c1e	refact task backend	2023-09-28 15:46:12 +08:00
Haojun Liao	80ec77cbb6	enh(stream): optimize the stream status when creating without fill-history option.	2023-09-28 11:23:03 +08:00
Haojun Liao	22e485d132	fix(stream): opt perf.	2023-09-28 09:25:23 +08:00
Haojun Liao	f525778aef	fix(stream): fix syntax error.	2023-09-28 00:53:36 +08:00
Haojun Liao	8c998673ad	fix(stream): fix dead lock.	2023-09-27 17:15:49 +08:00
Haojun Liao	2dade996bb	log(stream): update logs.	2023-09-27 15:37:12 +08:00
Haojun Liao	8f2d217834	log: update logs.	2023-09-27 14:48:53 +08:00
yihaoDeng	d4f87378fd	avoid first tag index conflict	2023-09-27 14:44:09 +08:00
Haojun Liao	11641e1115	refactor: do some internal refactor.	2023-09-27 14:18:01 +08:00
Haojun Liao	34d547bfff	fix(stream): avoid repeatly free obj.	2023-09-27 14:05:14 +08:00
Haojun Liao	ae020fcd2d	log(stream): update the log.	2023-09-27 14:03:57 +08:00
Haojun Liao	e13efd5b04	refactor: optimize the fill-history task launch policy, and do some other internal refactor.	2023-09-27 14:00:48 +08:00
Haojun Liao	a7e6933ce7	refactor: do some internal refactor.	2023-09-27 09:46:15 +08:00
Haojun Liao	37bf387453	fix(stream): reduce the counter when remove stream tasks.	2023-09-27 09:36:20 +08:00
Haojun Liao	9ab9de0ad3	fix(stream): limit sink quota.	2023-09-27 00:30:38 +08:00
Haojun Liao	27dcbcb96f	fix(stream): update the retry interval.	2023-09-26 23:42:02 +08:00
Haojun Liao	aec15ae39f	enh(stream): add quota limitation for sink task.	2023-09-26 23:29:54 +08:00
liuyao	c95dc167b7	add ci	2023-09-26 17:19:21 +08:00
Haojun Liao	890d658aea	refactor: do some internal refactor.	2023-09-26 14:11:45 +08:00
Haojun Liao	f3f18b0fa3	fix(stream): add new status for meta.	2023-09-26 12:15:12 +08:00
liuyao	82e18e32b0	add ci	2023-09-26 10:41:12 +08:00
Haojun Liao	d214a2f6bd	fix(stream): do some internal refactor.	2023-09-26 09:32:57 +08:00
Haojun Liao	6cc3618ed9	fix(stream): fix error in pause scan-history data.	2023-09-26 01:09:55 +08:00
Haojun Liao	5a4a0aa0e3	fix(stream): fix invalid read and memory leak.	2023-09-25 23:10:33 +08:00
yihaoDeng	ed172a939c	rename cfg default name	2023-09-25 18:09:35 +08:00
Haojun Liao	c9576d7a85	fix(stream): transfer state when all downstream rsp already.	2023-09-25 17:31:38 +08:00
Haojun Liao	c615c513e8	fix(stream): fix memory leak.	2023-09-25 16:01:17 +08:00
Haojun Liao	fcf4cc7169	fix(stream): fix the invalid counter for ready tasks in vnode, fix an invalid read.	2023-09-25 15:58:24 +08:00
Haojun Liao	1af95969b2	fix(stream): adjust nodeUpdate check interval.	2023-09-25 10:40:30 +08:00
Haojun Liao	eb4078b7f3	fix(stream): record the tasks started info, and do some internal refactor.	2023-09-25 09:59:05 +08:00
Haojun Liao	c987f61300	fix(stream): fix race condition.	2023-09-25 01:37:27 +08:00
Haojun Liao	da82a5f19e	fix(stream): do not wait for the checkpoint finish before stop tasks.	2023-09-24 22:03:07 +08:00
Haojun Liao	a74b462ffb	fix(stream): add new timer for each task.	2023-09-24 02:25:53 +08:00
Haojun Liao	b7dcd94c64	refactor: do some internal refactor.	2023-09-24 01:56:52 +08:00
Haojun Liao	3a75c21e66	refactor: do some internal refactor.	2023-09-24 01:55:59 +08:00
Haojun Liao	4ea737571d	refactor: do some internal refactor.	2023-09-24 01:26:51 +08:00
Haojun Liao	7f265e181d	fix(stream): update the checkpoint record start time.	2023-09-23 18:05:14 +08:00
Haojun Liao	6ef68fdf06	fix(stream): fix invalid checkpoint start time.	2023-09-23 17:57:09 +08:00
Haojun Liao	acd3321494	fix(stream): rsp when checkpoint source failed.	2023-09-23 14:13:33 +08:00
Haojun Liao	7d5f76c6f1	fix(stream): disable auto rsp for checkpoint source.	2023-09-23 11:31:14 +08:00
Haojun Liao	5155519559	fix(stream): fix invalid assert.	2023-09-22 23:06:48 +08:00
Haojun Liao	cf6c255f22	refactor: do some internal refactor.	2023-09-22 22:50:21 +08:00
yihaoDeng	8890f5c950	add cvt state	2023-09-22 12:51:29 +00:00
Haojun Liao	fcbd747afc	fix(stream): fix race condition.	2023-09-22 19:35:12 +08:00
liuyao	5cc1ae1266	fix issue	2023-09-22 19:25:19 +08:00
Haojun Liao	1ca05fb3af	fix(stream): add some logs.	2023-09-22 18:56:31 +08:00
Haojun Liao	5b12e276cf	fix(stream): add msg identifier check.	2023-09-22 18:13:18 +08:00
Haojun Liao	bb1a2322ca	fix(stream): add some logs.	2023-09-22 17:27:10 +08:00
yihaoDeng	11da631bb5	add cvt state	2023-09-22 09:08:42 +00:00
Haojun Liao	07ec6f38b8	fix(stream): remove invalid free.	2023-09-22 16:53:22 +08:00
Haojun Liao	5a695f632c	fix(stream): remove the blocking flag after successfully retry sending msg.	2023-09-22 16:44:55 +08:00
yihaoDeng	970b1cb840	add cvt state	2023-09-22 08:38:41 +00:00
yihaoDeng	173604ee1a	add cvt state	2023-09-22 08:32:14 +00:00
yihaoDeng	19e52c364e	add cvt state	2023-09-22 08:26:03 +00:00
Haojun Liao	6c357104a5	fix(stream): clear the failure vnode id list before send msg.	2023-09-22 16:19:25 +08:00
Haojun Liao	0f5f21ba09	fix(stream): fix invalid read.	2023-09-22 16:05:12 +08:00
Haojun Liao	619a9f6d08	fix(stream): refactor re-try dispatch msg for stream tasks.	2023-09-22 15:57:55 +08:00
liuyao	4f8e6d7795	Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TD-26174	2023-09-22 15:00:52 +08:00
liuyao	410a177637	fix issue	2023-09-22 15:00:26 +08:00
Haojun Liao	898aea5cfb	refactor(stream): refactor send msg.	2023-09-22 14:52:18 +08:00
Haojun Liao	87ea745861	fix(stream): add some logs, and remove invalid assert.	2023-09-22 10:53:03 +08:00
Haojun Liao	f7bcff862d	fix(stream): add check info detailed information.	2023-09-22 10:23:56 +08:00
wangmm0220	5555371fbc	fix:memory leak	2023-09-22 10:04:53 +08:00
liuyao	4979fc5283	fix issue	2023-09-22 09:40:08 +08:00
wangmm0220	803b17a2db	fix:memory leak	2023-09-22 09:31:58 +08:00
Haojun Liao	b1ba716299	refactor: do some internal refactor.	2023-09-22 09:24:36 +08:00
Haojun Liao	b075b4438b	fix(stream): handle dispatch checkpoint-trigger failure.	2023-09-21 23:15:18 +08:00
Haojun Liao	50d5fb6641	fix(stream): update logs.	2023-09-21 19:14:34 +08:00
yihaoDeng	470849982e	add cvt func	2023-09-21 09:41:30 +00:00
Haojun Liao	4925b67d4a	fix(stream): ignore the dispatch failure, and set the correct rsp counter.	2023-09-21 17:22:09 +08:00