Kassian Sun
7e79c509b2
[comparison] initial implementation
...
- handle new task types: cluster_comparison, index_comparison
- handle new pipeline tasks: dump_hash, index_diff
- split common task APIs
- add cluster_comparison task creation API
- optimize & clean migration codes
2023-05-17 15:43:34 +08:00
Kassian Sun
9f48efe45c
[migration] split cluster_migration
2023-05-16 12:13:39 +08:00
Kassian Sun
ba409072c8
[migration] split scheduler/index_migration
2023-05-16 11:09:00 +08:00
Kassian Sun
7c375a8d73
[migration] get es configs dynamically before creating pipeline
2023-04-28 16:13:30 +08:00
Kassian Sun
cbfe890982
[migration] scroll/bulk support manually retry
2023-04-24 15:58:51 +08:00
Kassian Sun
b1d3e67954
[migration] index migration skip retrying scroll if possible
2023-04-24 14:22:40 +08:00
Kassian Sun
06554054af
[migration] remove default sleep, schedule by task type & status
2023-04-24 14:19:39 +08:00
Kassian Sun
20498e2e76
[migration] tune cleaning gateway error logs
2023-04-22 15:36:47 +08:00
Kassian Sun
da908cebfb
[migration] fix missing return
2023-04-22 15:23:55 +08:00
Kassian Sun
b263c480c7
[migration] add back instance init error check
2023-04-22 15:02:05 +08:00
Kassian Sun
28a6c55671
[migration] refresh instance state after major task stopped
2023-04-22 14:48:53 +08:00
Kassian Sun
c1ed375813
[migration] print instance task limit in debug level
2023-04-21 18:28:41 +08:00
Kassian Sun
651c86feff
[migration] batch task by task type
2023-04-21 18:21:01 +08:00
Kassian Sun
52ff99af0c
[migration] use different error message for old-version tasks
2023-04-20 17:44:45 +08:00
Kassian Sun
e27a51f884
[migration] fix wrong pending_stop notification
2023-04-20 17:44:45 +08:00
Kassian Sun
c691bbc607
[migration] move disk queue clean up to index_migration level
2023-04-20 17:44:45 +08:00
Kassian Sun
ae9134afd5
[migration] fail cluster migration if docs count unmatch
2023-04-20 17:44:45 +08:00
Kassian Sun
57e1fb0844
[migration] optimize major task progress calculation
2023-04-20 17:44:45 +08:00
Kassian Sun
1f95d4af48
[migration] fix major task state handling
2023-04-20 17:44:45 +08:00
Kassian Sun
1bca4335f1
[migration] fix progress when index task is running
2023-04-20 17:44:45 +08:00
Kassian Sun
7d675ba3d1
[migration] optimize scroll/bulk error handling and status resetting
2023-04-20 17:44:45 +08:00
Kassian Sun
c89c8616b1
[migration] clean all business_id checks
2023-04-20 17:44:45 +08:00
Kassian Sun
527c3c9e46
[migration] delete ready1 state
2023-04-20 17:44:45 +08:00
Kassian Sun
7472052d90
[migration] optimze bulk error message
2023-04-20 17:44:45 +08:00
Kassian Sun
cd85d21ef0
[migration] split es_scroll/bulk_indexing pipeline task
2023-04-20 17:44:44 +08:00
Kassian Sun
1634640134
[migration] fix partion query nil check
2023-04-19 11:34:01 +08:00
Kassian Sun
eced674817
[migration] cleanup unused target config
2023-04-17 12:41:34 +08:00
Kassian Sun
81c9df8ef4
[migration] fix index source modification
2023-04-17 12:33:23 +08:00
Kassian Sun
c73c44724d
[migration] index_migration & pipeline define struct
2023-04-14 11:30:16 +08:00
Kassian Sun
db6d33f22b
[migration] always try to clean disk queues
2023-04-12 10:05:51 +08:00
sunjiacheng
dc606e457d
[migration] fix bulk_indexing counting logic ( #60 )
...
[migration] fix bulk_indexing counting logic
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-04-11 15:58:24 +08:00
Kassian Sun
4b12d372f1
[migration] task log copy task labels
2023-04-11 08:30:59 +08:00
liugq
4eb742d47e
format log message of migration task
2023-04-10 18:21:27 +08:00
liugq
b5218b8516
add compress settings of migration
2023-04-10 14:17:18 +08:00
liugq
db1c7322f1
update migration notifaction link url
2023-04-06 12:04:07 +08:00
Kassian Sun
b16ff4e888
[migration] track user actions in task logging
2023-04-04 16:18:13 +08:00
Kassian Sun
87d28c9cfc
[migration] fix pending_stop checking logic
2023-04-04 16:02:31 +08:00
silenceqi
bae99585c1
Merge pull request '[notification] update enum & list API' ( #52 ) from feature/notification into master
2023-04-04 10:24:42 +08:00
Kassian Sun
c3a1d418d5
[notification] update enum & list API
2023-04-04 10:18:45 +08:00
Kassian Sun
d5d46de854
[migration] improve pending_stop & es error handling
2023-04-03 20:25:45 +08:00
Kassian Sun
b31bb6c090
[migration] update logging logic
2023-04-03 20:25:07 +08:00
sunjiacheng
8fddcdaf09
[notification] add /notification/read, add message_type ( #47 )
...
[notification] add /notification/read, add message_type
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-04-02 15:02:30 +08:00
sunjiacheng
55ac534b02
[api] add list notification api ( #35 )
...
[api] add list notification api
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
Co-authored-by: liugq <silenceqi@hotmail.com>
Co-authored-by: medcl <m@medcl.net>
Co-authored-by: silenceqi <silenceqi@infini.ltd>
2023-04-01 18:52:02 +08:00
sunjiacheng
e75a566f03
[plugin][migration] add more logs ( #46 )
...
[plugin][migration] add more logs
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-03-30 18:24:26 +08:00
Kassian Sun
6de2e14235
[plugin][migration] add error loggings
2023-03-30 17:18:24 +08:00
liugq
6d537bb798
set es_scroll partition_size default value to 1
2023-03-29 18:41:20 +08:00
liugq
4871fbcb12
checking instance is avaiable when there is no pipeline logs
2023-03-29 16:17:19 +08:00
sunjiacheng
0f5626c41d
[migration] fix source filter dsl ( #43 )
...
[migration] fix source filter dsl
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-03-29 13:03:15 +08:00
sunjiacheng
633d0333f2
[plugins][migration] status_log -> logging ( #40 )
...
[plugins][migration] status_log -> logging
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-03-28 20:12:23 +08:00
sunjiacheng
69cf1d67a9
migration_v2 ( #38 )
...
bulk write after docs was scrolled complete
update default config with migration dispatcher
add bulk parameter idle_timeout_in_seconds, slice_size
clear queue before creating pipeline
check instance available when task state is running
add check_instance_available config for migration dispatcher
get migration task progress info by index_name_unique(index_name+doctype)
get instance list only by gateway
get migration task progress info from es and instance pipeline context
rewrite logic of handling running migration major task
calc complete time in api getDataMigrationTaskOfIndex
init
Co-authored-by: liugq <silenceqi@hotmail.com>
2023-03-28 17:54:35 +08:00