7e79c509b2
[comparison] initial implementation
...
- handle new task types: cluster_comparison, index_comparison
- handle new pipeline tasks: dump_hash, index_diff
- split common task APIs
- add cluster_comparison task creation API
- optimize & clean migration codes
2023-05-17 15:43:34 +08:00
d1ef70d8d1
fix no index,node metadata after console setup
2023-05-17 12:03:31 +08:00
e8a0b43fba
[migration][pipeline] filter duplicated logs
2023-05-16 14:34:47 +08:00
9f48efe45c
[migration] split cluster_migration
2023-05-16 12:13:39 +08:00
ba409072c8
[migration] split scheduler/index_migration
2023-05-16 11:09:00 +08:00
38d2a746f2
Merge pull request '[migration] mark task as stopped if pipeline not found' ( #83 ) from fix/migration into master
2023-05-15 14:42:35 +08:00
205744cd44
[migration] mark task as stopped if pipeline not found
2023-05-15 14:37:55 +08:00
4a269acf22
move topn api from framework to console ( #82 )
...
move topn api from framework to console
Co-authored-by: liugq <silenceqi@hotmail.com>
2023-05-10 23:33:35 +08:00
c37380c009
move insight model from framework to console
2023-05-08 16:23:36 +08:00
75b912426b
add websocket proxy
2023-05-06 14:53:07 +08:00
7c375a8d73
[migration] get es configs dynamically before creating pipeline
2023-04-28 16:13:30 +08:00
9b3005e005
add refresh index api
2023-04-24 19:41:43 +08:00
17a5ad6daa
[migration] scroll pipeline only check last run pipeline log
2023-04-24 16:57:16 +08:00
d1e947f826
[migration] check finished pipeline logs by updated time
2023-04-24 16:52:51 +08:00
99cf8b769f
[migration] check STOPPED logs after pending_stop
2023-04-24 16:31:44 +08:00
cbfe890982
[migration] scroll/bulk support manually retry
2023-04-24 15:58:51 +08:00
00c48bf9d5
Merge branch 'master' of ssh://git.infini.ltd:64221/infini/console
2023-04-24 14:38:04 +08:00
0ac40751cd
return pipeline task info
2023-04-24 14:37:57 +08:00
b1d3e67954
[migration] index migration skip retrying scroll if possible
2023-04-24 14:22:40 +08:00
06554054af
[migration] remove default sleep, schedule by task type & status
2023-04-24 14:19:39 +08:00
20498e2e76
[migration] tune cleaning gateway error logs
2023-04-22 15:36:47 +08:00
da908cebfb
[migration] fix missing return
2023-04-22 15:23:55 +08:00
b263c480c7
[migration] add back instance init error check
2023-04-22 15:02:05 +08:00
28a6c55671
[migration] refresh instance state after major task stopped
2023-04-22 14:48:53 +08:00
c1ed375813
[migration] print instance task limit in debug level
2023-04-21 18:28:41 +08:00
651c86feff
[migration] batch task by task type
2023-04-21 18:21:01 +08:00
5ad2e042a9
[migration] remove unused log ( #71 )
...
[migration] remove unused log
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-04-21 16:02:13 +08:00
52ff99af0c
[migration] use different error message for old-version tasks
2023-04-20 17:44:45 +08:00
e27a51f884
[migration] fix wrong pending_stop notification
2023-04-20 17:44:45 +08:00
c691bbc607
[migration] move disk queue clean up to index_migration level
2023-04-20 17:44:45 +08:00
ae9134afd5
[migration] fail cluster migration if docs count unmatch
2023-04-20 17:44:45 +08:00
57e1fb0844
[migration] optimize major task progress calculation
2023-04-20 17:44:45 +08:00
1f95d4af48
[migration] fix major task state handling
2023-04-20 17:44:45 +08:00
1bca4335f1
[migration] fix progress when index task is running
2023-04-20 17:44:45 +08:00
7d675ba3d1
[migration] optimize scroll/bulk error handling and status resetting
2023-04-20 17:44:45 +08:00
c89c8616b1
[migration] clean all business_id checks
2023-04-20 17:44:45 +08:00
d37649faa2
[migration] update code review issues
2023-04-20 17:44:45 +08:00
527c3c9e46
[migration] delete ready1 state
2023-04-20 17:44:45 +08:00
7472052d90
[migration] optimze bulk error message
2023-04-20 17:44:45 +08:00
cd85d21ef0
[migration] split es_scroll/bulk_indexing pipeline task
2023-04-20 17:44:44 +08:00
dd68300ab2
layout search support filtering with param is_fixed
2023-04-20 10:49:24 +08:00
1634640134
[migration] fix partion query nil check
2023-04-19 11:34:01 +08:00
ff219a211b
[migration] fix query_string json field name
2023-04-19 10:33:16 +08:00
c6b9f30277
update mapping with specify type
2023-04-18 18:16:27 +08:00
c32d6b76a1
adding api of delete data migration task
2023-04-18 15:53:07 +08:00
f18a2bc054
Merge branch 'master' into init_indices
2023-04-17 13:57:49 +08:00
eced674817
[migration] cleanup unused target config
2023-04-17 12:41:34 +08:00
81c9df8ef4
[migration] fix index source modification
2023-04-17 12:33:23 +08:00
896adad58b
fix conflicts
2023-04-17 10:27:17 +08:00
b61d0b1593
add index init api
2023-04-17 10:15:44 +08:00
c73c44724d
[migration] index_migration & pipeline define struct
2023-04-14 11:30:16 +08:00
db6d33f22b
[migration] always try to clean disk queues
2023-04-12 10:05:51 +08:00
dc606e457d
[migration] fix bulk_indexing counting logic ( #60 )
...
[migration] fix bulk_indexing counting logic
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-04-11 15:58:24 +08:00
4b12d372f1
[migration] task log copy task labels
2023-04-11 08:30:59 +08:00
4eb742d47e
format log message of migration task
2023-04-10 18:21:27 +08:00
b5218b8516
add compress settings of migration
2023-04-10 14:17:18 +08:00
e8f9c83245
optimize notification api
2023-04-06 21:07:29 +08:00
3fafe39b0d
add alerting notification
2023-04-06 16:16:44 +08:00
c8cd0b80a4
Merge branch 'master' of ssh://git.infini.ltd:64221/infini/console
2023-04-06 12:04:12 +08:00
db1c7322f1
update migration notifaction link url
2023-04-06 12:04:07 +08:00
8c623870e3
[notification] support search by keyword
2023-04-06 11:02:34 +08:00
b16ff4e888
[migration] track user actions in task logging
2023-04-04 16:18:13 +08:00
87d28c9cfc
[migration] fix pending_stop checking logic
2023-04-04 16:02:31 +08:00
400bd2d002
[notification] update _search defaults, fix /read ( #53 )
...
[notification] update _search defaults, fix /read
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-04-04 14:49:56 +08:00
061b528309
Merge branch 'master' of ssh://git.infini.ltd:64221/infini/console
2023-04-04 11:57:38 +08:00
be533cd609
add resource_id
2023-04-04 11:57:30 +08:00
bae99585c1
Merge pull request '[notification] update enum & list API' ( #52 ) from feature/notification into master
2023-04-04 10:24:42 +08:00
c3a1d418d5
[notification] update enum & list API
2023-04-04 10:18:45 +08:00
d5d46de854
[migration] improve pending_stop & es error handling
2023-04-03 20:25:45 +08:00
b31bb6c090
[migration] update logging logic
2023-04-03 20:25:07 +08:00
97f5b958b3
[notification] /read add types param
2023-04-03 18:45:34 +08:00
272f29403d
fix getting alerting rule 404
2023-04-03 18:05:27 +08:00
abe7acf177
add privilege control of migration
2023-04-03 11:57:06 +08:00
b29dc9ca0f
[migration] check status before start/stop task
2023-04-03 10:04:16 +08:00
f9f60b23e7
[notification] update API endpoint
2023-04-02 15:27:51 +08:00
8fddcdaf09
[notification] add /notification/read, add message_type ( #47 )
...
[notification] add /notification/read, add message_type
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-04-02 15:02:30 +08:00
55ac534b02
[api] add list notification api ( #35 )
...
[api] add list notification api
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
Co-authored-by: liugq <silenceqi@hotmail.com>
Co-authored-by: medcl <m@medcl.net>
Co-authored-by: silenceqi <silenceqi@infini.ltd>
2023-04-01 18:52:02 +08:00
1a645a655a
support cardinality agg
2023-04-01 12:25:06 +08:00
9bd5682122
save cluster distribution in major magration task
2023-04-01 11:35:20 +08:00
3987c5db37
Merge branch 'master' of ssh://git.infini.ltd:64221/infini/console
2023-03-31 14:26:50 +08:00
f716e2acf5
filter overview cluster
2023-03-31 14:26:11 +08:00
e75a566f03
[plugin][migration] add more logs ( #46 )
...
[plugin][migration] add more logs
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-03-30 18:24:26 +08:00
6de2e14235
[plugin][migration] add error loggings
2023-03-30 17:18:24 +08:00
6d537bb798
set es_scroll partition_size default value to 1
2023-03-29 18:41:20 +08:00
4871fbcb12
checking instance is avaiable when there is no pipeline logs
2023-03-29 16:17:19 +08:00
0f5626c41d
[migration] fix source filter dsl ( #43 )
...
[migration] fix source filter dsl
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-03-29 13:03:15 +08:00
08a1673c1a
[plugin] fix naming ( #42 )
...
[plugin] fix naming
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-03-29 13:02:55 +08:00
f187bbe499
add field type and reserved
2023-03-29 11:37:16 +08:00
305a11ec40
update api url of getting task info with index level
2023-03-29 10:26:05 +08:00
633d0333f2
[plugins][migration] status_log -> logging ( #40 )
...
[plugins][migration] status_log -> logging
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-03-28 20:12:23 +08:00
05cc6fd9d2
add context to pipeline
2023-03-28 19:01:58 +08:00
69cf1d67a9
migration_v2 ( #38 )
...
bulk write after docs was scrolled complete
update default config with migration dispatcher
add bulk parameter idle_timeout_in_seconds, slice_size
clear queue before creating pipeline
check instance available when task state is running
add check_instance_available config for migration dispatcher
get migration task progress info by index_name_unique(index_name+doctype)
get instance list only by gateway
get migration task progress info from es and instance pipeline context
rewrite logic of handling running migration major task
calc complete time in api getDataMigrationTaskOfIndex
init
Co-authored-by: liugq <silenceqi@hotmail.com>
2023-03-28 17:54:35 +08:00
cd56a4ff24
Merge pull request '[plugin][api] visualization/data supports latest statistics' ( #36 ) from feature/top_hits into master
2023-03-28 10:28:25 +08:00
845d77508b
return null if _source is empty
2023-03-23 09:00:07 +08:00
17abd00e20
[plugin][api] visualization/data supports top_hits
2023-03-22 19:04:34 +08:00
bc024bcfce
filter empty group of alerting rule
2023-03-22 10:46:26 +08:00
99f03fc574
add elastic status overview api
2023-03-16 10:55:21 +08:00
6d2f02af9c
Merge pull request 'Elasticsearch, easysearch, opensearch compatibility update' ( #32 ) from es_distribution into master
2023-03-11 19:11:06 +08:00
eba379eade
test update
2023-03-11 19:09:25 +08:00
f27c7a2951
elasticsearch distribution judgement logic update
2023-03-11 13:30:59 +08:00
Kassian Sun
liugq
silenceqi
sunjiacheng
medcl