7e79c509b2
[comparison] initial implementation
...
- handle new task types: cluster_comparison, index_comparison
- handle new pipeline tasks: dump_hash, index_diff
- split common task APIs
- add cluster_comparison task creation API
- optimize & clean migration codes
2023-05-17 15:43:34 +08:00
9f48efe45c
[migration] split cluster_migration
2023-05-16 12:13:39 +08:00
ba409072c8
[migration] split scheduler/index_migration
2023-05-16 11:09:00 +08:00
7c375a8d73
[migration] get es configs dynamically before creating pipeline
2023-04-28 16:13:30 +08:00
cbfe890982
[migration] scroll/bulk support manually retry
2023-04-24 15:58:51 +08:00
b1d3e67954
[migration] index migration skip retrying scroll if possible
2023-04-24 14:22:40 +08:00
06554054af
[migration] remove default sleep, schedule by task type & status
2023-04-24 14:19:39 +08:00
20498e2e76
[migration] tune cleaning gateway error logs
2023-04-22 15:36:47 +08:00
da908cebfb
[migration] fix missing return
2023-04-22 15:23:55 +08:00
b263c480c7
[migration] add back instance init error check
2023-04-22 15:02:05 +08:00
28a6c55671
[migration] refresh instance state after major task stopped
2023-04-22 14:48:53 +08:00
c1ed375813
[migration] print instance task limit in debug level
2023-04-21 18:28:41 +08:00
651c86feff
[migration] batch task by task type
2023-04-21 18:21:01 +08:00
52ff99af0c
[migration] use different error message for old-version tasks
2023-04-20 17:44:45 +08:00
e27a51f884
[migration] fix wrong pending_stop notification
2023-04-20 17:44:45 +08:00
c691bbc607
[migration] move disk queue clean up to index_migration level
2023-04-20 17:44:45 +08:00
ae9134afd5
[migration] fail cluster migration if docs count unmatch
2023-04-20 17:44:45 +08:00
57e1fb0844
[migration] optimize major task progress calculation
2023-04-20 17:44:45 +08:00
1f95d4af48
[migration] fix major task state handling
2023-04-20 17:44:45 +08:00
1bca4335f1
[migration] fix progress when index task is running
2023-04-20 17:44:45 +08:00
7d675ba3d1
[migration] optimize scroll/bulk error handling and status resetting
2023-04-20 17:44:45 +08:00
c89c8616b1
[migration] clean all business_id checks
2023-04-20 17:44:45 +08:00
527c3c9e46
[migration] delete ready1 state
2023-04-20 17:44:45 +08:00
7472052d90
[migration] optimze bulk error message
2023-04-20 17:44:45 +08:00
cd85d21ef0
[migration] split es_scroll/bulk_indexing pipeline task
2023-04-20 17:44:44 +08:00
1634640134
[migration] fix partion query nil check
2023-04-19 11:34:01 +08:00
eced674817
[migration] cleanup unused target config
2023-04-17 12:41:34 +08:00
81c9df8ef4
[migration] fix index source modification
2023-04-17 12:33:23 +08:00
c73c44724d
[migration] index_migration & pipeline define struct
2023-04-14 11:30:16 +08:00
db6d33f22b
[migration] always try to clean disk queues
2023-04-12 10:05:51 +08:00
dc606e457d
[migration] fix bulk_indexing counting logic ( #60 )
...
[migration] fix bulk_indexing counting logic
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-04-11 15:58:24 +08:00
4b12d372f1
[migration] task log copy task labels
2023-04-11 08:30:59 +08:00
4eb742d47e
format log message of migration task
2023-04-10 18:21:27 +08:00
b5218b8516
add compress settings of migration
2023-04-10 14:17:18 +08:00
db1c7322f1
update migration notifaction link url
2023-04-06 12:04:07 +08:00
b16ff4e888
[migration] track user actions in task logging
2023-04-04 16:18:13 +08:00
87d28c9cfc
[migration] fix pending_stop checking logic
2023-04-04 16:02:31 +08:00
bae99585c1
Merge pull request '[notification] update enum & list API' ( #52 ) from feature/notification into master
2023-04-04 10:24:42 +08:00
c3a1d418d5
[notification] update enum & list API
2023-04-04 10:18:45 +08:00
d5d46de854
[migration] improve pending_stop & es error handling
2023-04-03 20:25:45 +08:00
b31bb6c090
[migration] update logging logic
2023-04-03 20:25:07 +08:00
8fddcdaf09
[notification] add /notification/read, add message_type ( #47 )
...
[notification] add /notification/read, add message_type
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-04-02 15:02:30 +08:00
55ac534b02
[api] add list notification api ( #35 )
...
[api] add list notification api
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
Co-authored-by: liugq <silenceqi@hotmail.com>
Co-authored-by: medcl <m@medcl.net>
Co-authored-by: silenceqi <silenceqi@infini.ltd>
2023-04-01 18:52:02 +08:00
e75a566f03
[plugin][migration] add more logs ( #46 )
...
[plugin][migration] add more logs
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-03-30 18:24:26 +08:00
6de2e14235
[plugin][migration] add error loggings
2023-03-30 17:18:24 +08:00
6d537bb798
set es_scroll partition_size default value to 1
2023-03-29 18:41:20 +08:00
4871fbcb12
checking instance is avaiable when there is no pipeline logs
2023-03-29 16:17:19 +08:00
0f5626c41d
[migration] fix source filter dsl ( #43 )
...
[migration] fix source filter dsl
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-03-29 13:03:15 +08:00
633d0333f2
[plugins][migration] status_log -> logging ( #40 )
...
[plugins][migration] status_log -> logging
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-03-28 20:12:23 +08:00
69cf1d67a9
migration_v2 ( #38 )
...
bulk write after docs was scrolled complete
update default config with migration dispatcher
add bulk parameter idle_timeout_in_seconds, slice_size
clear queue before creating pipeline
check instance available when task state is running
add check_instance_available config for migration dispatcher
get migration task progress info by index_name_unique(index_name+doctype)
get instance list only by gateway
get migration task progress info from es and instance pipeline context
rewrite logic of handling running migration major task
calc complete time in api getDataMigrationTaskOfIndex
init
Co-authored-by: liugq <silenceqi@hotmail.com>
2023-03-28 17:54:35 +08:00
Kassian Sun
sunjiacheng
liugq
silenceqi