155cb4cfbe
[migration] clear child tasks before split task
2023-06-01 06:10:19 +08:00
09ba88c411
[migration] get instance from scheduler, initialize instance automatically
2023-05-30 11:41:53 +08:00
dd98ba7dff
[migration] don't filter out non-running pipeline task
2023-05-26 16:03:07 +08:00
241c9f57fb
sleep when no message was found in queue
2023-05-26 12:02:39 +08:00
d98ea04037
[comparison] info API update progress calculation
2023-05-25 10:07:43 +08:00
8d5e7e8a6b
fix empty cluster_uuid
2023-05-23 15:15:56 +08:00
1789d74ece
[migration] add option to skip scroll/bulk count check
2023-05-19 19:15:11 +08:00
5000deefa9
[migration] fix missing partition for running sub tasks
2023-05-19 19:12:37 +08:00
86c8615537
[comparison] always clean queue after task ended
2023-05-19 18:35:04 +08:00
a00bc59d6b
[migration] pipeline_task use execution_instance_id from labels
2023-05-19 15:31:42 +08:00
be09c69212
[comparison] fill execution_instance_id for pipeline tasks
2023-05-19 15:25:32 +08:00
02243264e8
[comparison] merge source/target dump partitions
2023-05-19 14:42:23 +08:00
d4202897e1
[migration][util] handle nil slice without error log
2023-05-19 14:15:55 +08:00
6cceb8af26
[migration] add back module.go
2023-05-19 09:46:52 +08:00
7620a76ef7
[migration] cleanup unused code
2023-05-18 22:27:20 +08:00
b95b1ff703
[comparison] add info APIs
...
- add comparison task info & by index info API
- update comparison task permission control
- unify migration task progress calculation
- cluster migration/comparison set start time
- add split task utilities
- clear index migration/comparison state before running
2023-05-18 17:06:34 +08:00
5825bd0f71
[migration] split pipeline code, clear labels for each type
2023-05-17 19:11:03 +08:00
d0706f460d
[migration] handle dump_hash/index_diff pending_stop
2023-05-17 17:43:44 +08:00
8d27425fc0
[migration] fix GetChildTasks
2023-05-17 16:58:38 +08:00
556a3d49d8
[migration] extract UpdateStoppedChildTasksToReady
2023-05-17 16:44:27 +08:00
7e79c509b2
[comparison] initial implementation
...
- handle new task types: cluster_comparison, index_comparison
- handle new pipeline tasks: dump_hash, index_diff
- split common task APIs
- add cluster_comparison task creation API
- optimize & clean migration codes
2023-05-17 15:43:34 +08:00
d1ef70d8d1
fix no index,node metadata after console setup
2023-05-17 12:03:31 +08:00
e8a0b43fba
[migration][pipeline] filter duplicated logs
2023-05-16 14:34:47 +08:00
9f48efe45c
[migration] split cluster_migration
2023-05-16 12:13:39 +08:00
ba409072c8
[migration] split scheduler/index_migration
2023-05-16 11:09:00 +08:00
38d2a746f2
Merge pull request '[migration] mark task as stopped if pipeline not found' ( #83 ) from fix/migration into master
2023-05-15 14:42:35 +08:00
205744cd44
[migration] mark task as stopped if pipeline not found
2023-05-15 14:37:55 +08:00
4a269acf22
move topn api from framework to console ( #82 )
...
move topn api from framework to console
Co-authored-by: liugq <silenceqi@hotmail.com>
2023-05-10 23:33:35 +08:00
c37380c009
move insight model from framework to console
2023-05-08 16:23:36 +08:00
75b912426b
add websocket proxy
2023-05-06 14:53:07 +08:00
7c375a8d73
[migration] get es configs dynamically before creating pipeline
2023-04-28 16:13:30 +08:00
9b3005e005
add refresh index api
2023-04-24 19:41:43 +08:00
17a5ad6daa
[migration] scroll pipeline only check last run pipeline log
2023-04-24 16:57:16 +08:00
d1e947f826
[migration] check finished pipeline logs by updated time
2023-04-24 16:52:51 +08:00
99cf8b769f
[migration] check STOPPED logs after pending_stop
2023-04-24 16:31:44 +08:00
cbfe890982
[migration] scroll/bulk support manually retry
2023-04-24 15:58:51 +08:00
00c48bf9d5
Merge branch 'master' of ssh://git.infini.ltd:64221/infini/console
2023-04-24 14:38:04 +08:00
0ac40751cd
return pipeline task info
2023-04-24 14:37:57 +08:00
b1d3e67954
[migration] index migration skip retrying scroll if possible
2023-04-24 14:22:40 +08:00
06554054af
[migration] remove default sleep, schedule by task type & status
2023-04-24 14:19:39 +08:00
20498e2e76
[migration] tune cleaning gateway error logs
2023-04-22 15:36:47 +08:00
da908cebfb
[migration] fix missing return
2023-04-22 15:23:55 +08:00
b263c480c7
[migration] add back instance init error check
2023-04-22 15:02:05 +08:00
28a6c55671
[migration] refresh instance state after major task stopped
2023-04-22 14:48:53 +08:00
c1ed375813
[migration] print instance task limit in debug level
2023-04-21 18:28:41 +08:00
651c86feff
[migration] batch task by task type
2023-04-21 18:21:01 +08:00
5ad2e042a9
[migration] remove unused log ( #71 )
...
[migration] remove unused log
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-04-21 16:02:13 +08:00
52ff99af0c
[migration] use different error message for old-version tasks
2023-04-20 17:44:45 +08:00
e27a51f884
[migration] fix wrong pending_stop notification
2023-04-20 17:44:45 +08:00
c691bbc607
[migration] move disk queue clean up to index_migration level
2023-04-20 17:44:45 +08:00
ae9134afd5
[migration] fail cluster migration if docs count unmatch
2023-04-20 17:44:45 +08:00
57e1fb0844
[migration] optimize major task progress calculation
2023-04-20 17:44:45 +08:00
1f95d4af48
[migration] fix major task state handling
2023-04-20 17:44:45 +08:00
1bca4335f1
[migration] fix progress when index task is running
2023-04-20 17:44:45 +08:00
7d675ba3d1
[migration] optimize scroll/bulk error handling and status resetting
2023-04-20 17:44:45 +08:00
c89c8616b1
[migration] clean all business_id checks
2023-04-20 17:44:45 +08:00
d37649faa2
[migration] update code review issues
2023-04-20 17:44:45 +08:00
527c3c9e46
[migration] delete ready1 state
2023-04-20 17:44:45 +08:00
7472052d90
[migration] optimze bulk error message
2023-04-20 17:44:45 +08:00
cd85d21ef0
[migration] split es_scroll/bulk_indexing pipeline task
2023-04-20 17:44:44 +08:00
dd68300ab2
layout search support filtering with param is_fixed
2023-04-20 10:49:24 +08:00
1634640134
[migration] fix partion query nil check
2023-04-19 11:34:01 +08:00
ff219a211b
[migration] fix query_string json field name
2023-04-19 10:33:16 +08:00
c6b9f30277
update mapping with specify type
2023-04-18 18:16:27 +08:00
c32d6b76a1
adding api of delete data migration task
2023-04-18 15:53:07 +08:00
f18a2bc054
Merge branch 'master' into init_indices
2023-04-17 13:57:49 +08:00
eced674817
[migration] cleanup unused target config
2023-04-17 12:41:34 +08:00
81c9df8ef4
[migration] fix index source modification
2023-04-17 12:33:23 +08:00
896adad58b
fix conflicts
2023-04-17 10:27:17 +08:00
b61d0b1593
add index init api
2023-04-17 10:15:44 +08:00
c73c44724d
[migration] index_migration & pipeline define struct
2023-04-14 11:30:16 +08:00
db6d33f22b
[migration] always try to clean disk queues
2023-04-12 10:05:51 +08:00
dc606e457d
[migration] fix bulk_indexing counting logic ( #60 )
...
[migration] fix bulk_indexing counting logic
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-04-11 15:58:24 +08:00
4b12d372f1
[migration] task log copy task labels
2023-04-11 08:30:59 +08:00
4eb742d47e
format log message of migration task
2023-04-10 18:21:27 +08:00
b5218b8516
add compress settings of migration
2023-04-10 14:17:18 +08:00
e8f9c83245
optimize notification api
2023-04-06 21:07:29 +08:00
3fafe39b0d
add alerting notification
2023-04-06 16:16:44 +08:00
c8cd0b80a4
Merge branch 'master' of ssh://git.infini.ltd:64221/infini/console
2023-04-06 12:04:12 +08:00
db1c7322f1
update migration notifaction link url
2023-04-06 12:04:07 +08:00
8c623870e3
[notification] support search by keyword
2023-04-06 11:02:34 +08:00
b16ff4e888
[migration] track user actions in task logging
2023-04-04 16:18:13 +08:00
87d28c9cfc
[migration] fix pending_stop checking logic
2023-04-04 16:02:31 +08:00
400bd2d002
[notification] update _search defaults, fix /read ( #53 )
...
[notification] update _search defaults, fix /read
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-04-04 14:49:56 +08:00
061b528309
Merge branch 'master' of ssh://git.infini.ltd:64221/infini/console
2023-04-04 11:57:38 +08:00
be533cd609
add resource_id
2023-04-04 11:57:30 +08:00
bae99585c1
Merge pull request '[notification] update enum & list API' ( #52 ) from feature/notification into master
2023-04-04 10:24:42 +08:00
c3a1d418d5
[notification] update enum & list API
2023-04-04 10:18:45 +08:00
d5d46de854
[migration] improve pending_stop & es error handling
2023-04-03 20:25:45 +08:00
b31bb6c090
[migration] update logging logic
2023-04-03 20:25:07 +08:00
97f5b958b3
[notification] /read add types param
2023-04-03 18:45:34 +08:00
272f29403d
fix getting alerting rule 404
2023-04-03 18:05:27 +08:00
abe7acf177
add privilege control of migration
2023-04-03 11:57:06 +08:00
b29dc9ca0f
[migration] check status before start/stop task
2023-04-03 10:04:16 +08:00
f9f60b23e7
[notification] update API endpoint
2023-04-02 15:27:51 +08:00
8fddcdaf09
[notification] add /notification/read, add message_type ( #47 )
...
[notification] add /notification/read, add message_type
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-04-02 15:02:30 +08:00
55ac534b02
[api] add list notification api ( #35 )
...
[api] add list notification api
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
Co-authored-by: liugq <silenceqi@hotmail.com>
Co-authored-by: medcl <m@medcl.net>
Co-authored-by: silenceqi <silenceqi@infini.ltd>
2023-04-01 18:52:02 +08:00
1a645a655a
support cardinality agg
2023-04-01 12:25:06 +08:00
9bd5682122
save cluster distribution in major magration task
2023-04-01 11:35:20 +08:00
3987c5db37
Merge branch 'master' of ssh://git.infini.ltd:64221/infini/console
2023-03-31 14:26:50 +08:00
f716e2acf5
filter overview cluster
2023-03-31 14:26:11 +08:00
e75a566f03
[plugin][migration] add more logs ( #46 )
...
[plugin][migration] add more logs
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-03-30 18:24:26 +08:00
6de2e14235
[plugin][migration] add error loggings
2023-03-30 17:18:24 +08:00
6d537bb798
set es_scroll partition_size default value to 1
2023-03-29 18:41:20 +08:00
4871fbcb12
checking instance is avaiable when there is no pipeline logs
2023-03-29 16:17:19 +08:00
0f5626c41d
[migration] fix source filter dsl ( #43 )
...
[migration] fix source filter dsl
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-03-29 13:03:15 +08:00
08a1673c1a
[plugin] fix naming ( #42 )
...
[plugin] fix naming
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-03-29 13:02:55 +08:00
f187bbe499
add field type and reserved
2023-03-29 11:37:16 +08:00
305a11ec40
update api url of getting task info with index level
2023-03-29 10:26:05 +08:00
633d0333f2
[plugins][migration] status_log -> logging ( #40 )
...
[plugins][migration] status_log -> logging
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-03-28 20:12:23 +08:00
05cc6fd9d2
add context to pipeline
2023-03-28 19:01:58 +08:00
69cf1d67a9
migration_v2 ( #38 )
...
bulk write after docs was scrolled complete
update default config with migration dispatcher
add bulk parameter idle_timeout_in_seconds, slice_size
clear queue before creating pipeline
check instance available when task state is running
add check_instance_available config for migration dispatcher
get migration task progress info by index_name_unique(index_name+doctype)
get instance list only by gateway
get migration task progress info from es and instance pipeline context
rewrite logic of handling running migration major task
calc complete time in api getDataMigrationTaskOfIndex
init
Co-authored-by: liugq <silenceqi@hotmail.com>
2023-03-28 17:54:35 +08:00
cd56a4ff24
Merge pull request '[plugin][api] visualization/data supports latest statistics' ( #36 ) from feature/top_hits into master
2023-03-28 10:28:25 +08:00
845d77508b
return null if _source is empty
2023-03-23 09:00:07 +08:00
17abd00e20
[plugin][api] visualization/data supports top_hits
2023-03-22 19:04:34 +08:00
bc024bcfce
filter empty group of alerting rule
2023-03-22 10:46:26 +08:00
99f03fc574
add elastic status overview api
2023-03-16 10:55:21 +08:00
6d2f02af9c
Merge pull request 'Elasticsearch, easysearch, opensearch compatibility update' ( #32 ) from es_distribution into master
2023-03-11 19:11:06 +08:00
eba379eade
test update
2023-03-11 19:09:25 +08:00
f27c7a2951
elasticsearch distribution judgement logic update
2023-03-11 13:30:59 +08:00
c8f7927821
use default distribution with Elasticsarch
2023-03-08 18:01:53 +08:00
50e8f7fc79
sync cluster distribution to config file while setup
2023-03-08 12:18:08 +08:00
d37da4679e
use total value instead of doc_value agg as doc count
2023-03-06 15:00:14 +08:00
d57e9cf476
change es cluster version of system cluster support from 7.3 to 5.3
2023-03-06 10:59:52 +08:00
2c8ee843f1
refactoring elastic.GetDateHistogramIntervalField
2023-03-02 20:54:21 +08:00
5d0e745012
support setup with easysearch and opensearch
2023-03-02 17:28:51 +08:00
662205c540
add layout api permission
2023-02-28 12:27:26 +08:00
f1c18797b0
add query arg view_id for layout search api
2023-02-27 18:04:07 +08:00
5637c65977
add view layout api
2023-02-24 19:32:10 +08:00
ce4a056efa
clear code of builtin user
2023-02-24 12:31:17 +08:00
8431144dca
Merge pull request 'add credential management capabilities' ( #28 ) from credential into master
2023-02-23 11:30:05 +08:00
0cd198b805
rename tempID to GlobalSystemElasticsearchID
2023-02-23 11:10:38 +08:00
67382361a1
use orm.Create replace orm.Save method
2023-02-23 09:15:05 +08:00
fc8a47373a
[build] fix src/* imports ( #27 )
...
[build] fix src/* imports
Co-authored-by: Kassian Sun <kassiansun@outlook.com>
2023-02-21 21:48:20 +08:00
5d002145b9
add credential change callback for updating password of system cluster
2023-02-21 15:39:26 +08:00
a820e0d52f
handle err
2023-02-15 11:55:49 +08:00
46432da7aa
Merge branch 'master' into credential
2023-02-13 11:35:49 +08:00
4495de8ae9
initialize bug fixed
2023-02-10 18:16:05 +08:00
31e314ec87
initialize bug fixed
2023-02-10 18:13:56 +08:00
e11207dbc0
init credential secret
2023-02-10 09:51:42 +08:00
844969c5c1
Merge branch 'master' of ssh://git.infini.ltd:64221/infini/console
2023-02-09 14:14:29 +08:00
c1739df13b
remove unused parameters
2023-02-09 14:14:24 +08:00
6bcca588cb
fixed error output of parsing migration config
2023-02-08 11:37:33 +08:00
aaf0822593
remove src from import path
2023-02-08 10:55:16 +08:00
c22bf646f3
fixed compile error of migration code
2023-02-08 10:51:12 +08:00
fa9ae1c465
code optimize after code review
2023-02-06 16:51:30 +08:00
02adb4d121
refactoring pipelines
2023-01-27 12:09:09 +08:00
e50f273cda
fixed error of getting detail info of alert history
2023-01-06 17:49:52 +08:00
b93fb2f3c2
refactoring orm.Delete
2022-12-13 14:26:40 +08:00
c17359adb3
add orm.Context
2022-12-08 20:48:59 +08:00
Kassian Sun
medcl
liugq
silenceqi
sunjiacheng