Commit Graph

348 Commits

Author SHA1 Message Date
medcl a58d15721b handle license and apply locally 2023-06-08 22:54:39 +08:00
medcl 9f90e49ae4 add license apply proxy api 2023-06-08 13:22:25 +08:00
liugq df4de53d29 auto detect time agg location 2023-06-08 12:53:14 +08:00
liugq 282ff24669 fix agg key convert to string error 2023-06-08 11:01:51 +08:00
liugq ce14d82406 get metric data with root agg timestamp 2023-06-08 10:43:30 +08:00
liugq 4a3961ab1f fix adding cluster_uuid 2023-06-07 21:05:59 +08:00
liugq be2787db97 fix formula notwork when there is only one metric items 2023-06-07 16:06:44 +08:00
Kassian Sun 523a6eeb37 [task_manager] rename from migration, do final cleanups 2023-06-06 10:25:34 +08:00
Kassian Sun 4d60f7c174 [migration] info api returns repeat in labels 2023-06-06 10:02:36 +08:00
Kassian Sun 68b054783b [migration] don't send notification for repeating child tasks 2023-06-05 17:45:19 +08:00
Kassian Sun 5af5b2142e [migration] fix task doc stats 2023-06-05 17:09:42 +08:00
sunjiacheng 0f0794aca7 [migration] support repeating task & incremental index (#108)
[migration] return error info for comparison/migration task

[migration] list api returns running_children

[migration] add pause/resume and repeating info

mark invalid repeat as done

[migration] fix doc count

revert unneeded changes

cleanup marshals & unmarshals

update apis

v2, unify repeat & incremental configuration

[migration] support repeating task & incremental index

Co-authored-by: Kassian Sun <kassiansun@outlook.com>
Co-authored-by: silenceqi <silenceqi@infini.ltd>
Co-authored-by: liugq <silenceqi@hotmail.com>
Co-authored-by: hardy <luohf@infinilabs.com>
2023-06-05 16:21:00 +08:00
Kassian Sun 155cb4cfbe [migration] clear child tasks before split task 2023-06-01 06:10:19 +08:00
Kassian Sun 09ba88c411 [migration] get instance from scheduler, initialize instance automatically 2023-05-30 11:41:53 +08:00
Kassian Sun dd98ba7dff [migration] don't filter out non-running pipeline task 2023-05-26 16:03:07 +08:00
medcl 241c9f57fb sleep when no message was found in queue 2023-05-26 12:02:39 +08:00
Kassian Sun d98ea04037 [comparison] info API update progress calculation 2023-05-25 10:07:43 +08:00
liugq 8d5e7e8a6b fix empty cluster_uuid 2023-05-23 15:15:56 +08:00
Kassian Sun 1789d74ece [migration] add option to skip scroll/bulk count check 2023-05-19 19:15:11 +08:00
Kassian Sun 5000deefa9 [migration] fix missing partition for running sub tasks 2023-05-19 19:12:37 +08:00
Kassian Sun 86c8615537 [comparison] always clean queue after task ended 2023-05-19 18:35:04 +08:00
Kassian Sun a00bc59d6b [migration] pipeline_task use execution_instance_id from labels 2023-05-19 15:31:42 +08:00
Kassian Sun be09c69212 [comparison] fill execution_instance_id for pipeline tasks 2023-05-19 15:25:32 +08:00
Kassian Sun 02243264e8 [comparison] merge source/target dump partitions 2023-05-19 14:42:23 +08:00
Kassian Sun d4202897e1 [migration][util] handle nil slice without error log 2023-05-19 14:15:55 +08:00
Kassian Sun 6cceb8af26 [migration] add back module.go 2023-05-19 09:46:52 +08:00
Kassian Sun 7620a76ef7 [migration] cleanup unused code 2023-05-18 22:27:20 +08:00
Kassian Sun b95b1ff703 [comparison] add info APIs
- add comparison task info & by index info API
- update comparison task permission control
- unify migration task progress calculation
- cluster migration/comparison set start time
- add split task utilities
- clear index migration/comparison state before running
2023-05-18 17:06:34 +08:00
Kassian Sun 5825bd0f71 [migration] split pipeline code, clear labels for each type 2023-05-17 19:11:03 +08:00
Kassian Sun d0706f460d [migration] handle dump_hash/index_diff pending_stop 2023-05-17 17:43:44 +08:00
Kassian Sun 8d27425fc0 [migration] fix GetChildTasks 2023-05-17 16:58:38 +08:00
Kassian Sun 556a3d49d8 [migration] extract UpdateStoppedChildTasksToReady 2023-05-17 16:44:27 +08:00
Kassian Sun 7e79c509b2 [comparison] initial implementation
- handle new task types: cluster_comparison, index_comparison
- handle new pipeline tasks: dump_hash, index_diff
- split common task APIs
- add cluster_comparison task creation API
- optimize & clean migration codes
2023-05-17 15:43:34 +08:00
liugq d1ef70d8d1 fix no index,node metadata after console setup 2023-05-17 12:03:31 +08:00
Kassian Sun e8a0b43fba [migration][pipeline] filter duplicated logs 2023-05-16 14:34:47 +08:00
Kassian Sun 9f48efe45c [migration] split cluster_migration 2023-05-16 12:13:39 +08:00
Kassian Sun ba409072c8 [migration] split scheduler/index_migration 2023-05-16 11:09:00 +08:00
silenceqi 38d2a746f2 Merge pull request '[migration] mark task as stopped if pipeline not found' (#83) from fix/migration into master 2023-05-15 14:42:35 +08:00
Kassian Sun 205744cd44 [migration] mark task as stopped if pipeline not found 2023-05-15 14:37:55 +08:00
silenceqi 4a269acf22 move topn api from framework to console (#82)
move topn api from framework to console

Co-authored-by: liugq <silenceqi@hotmail.com>
2023-05-10 23:33:35 +08:00
liugq c37380c009 move insight model from framework to console 2023-05-08 16:23:36 +08:00
liugq 75b912426b add websocket proxy 2023-05-06 14:53:07 +08:00
Kassian Sun 7c375a8d73 [migration] get es configs dynamically before creating pipeline 2023-04-28 16:13:30 +08:00
liugq 9b3005e005 add refresh index api 2023-04-24 19:41:43 +08:00
Kassian Sun 17a5ad6daa [migration] scroll pipeline only check last run pipeline log 2023-04-24 16:57:16 +08:00
Kassian Sun d1e947f826 [migration] check finished pipeline logs by updated time 2023-04-24 16:52:51 +08:00
Kassian Sun 99cf8b769f [migration] check STOPPED logs after pending_stop 2023-04-24 16:31:44 +08:00
Kassian Sun cbfe890982 [migration] scroll/bulk support manually retry 2023-04-24 15:58:51 +08:00
liugq 00c48bf9d5 Merge branch 'master' of ssh://git.infini.ltd:64221/infini/console 2023-04-24 14:38:04 +08:00
liugq 0ac40751cd return pipeline task info 2023-04-24 14:37:57 +08:00