e45367f924
fix: resolve coverity scan issues in sync and wal
2023-02-23 14:16:21 +08:00
b28c0ad952
fix:remove useless logic in tmq
2023-02-21 18:49:39 +08:00
0c7a4bfad9
enh: refactor syncBeginSnapshot and walBeginSnapshot for logRetention
2023-02-20 19:19:49 +08:00
c597de099f
merge main
2023-02-20 13:41:27 +08:00
bcf6d5b641
fix: clear some asserts in walRead.c
2023-02-16 16:06:12 +08:00
fd379310f6
Merge pull request #19979 from taosdata/fix/tq_oom
...
fix: handle insufficient resource
2023-02-16 09:17:52 +08:00
e4a365c265
remove assert
2023-02-14 16:36:55 +08:00
c597a1e4ce
refactor: do some internal refactor.
2023-02-13 15:46:47 +08:00
3382a24a69
merge main
2023-02-10 18:03:11 +08:00
5ce5dded75
enh: reset wal reader after walReadVer
2023-02-08 14:41:15 +08:00
0f00862d8b
Merge branch 'fix/TD-22070' of https://github.com/taosdata/TDengine into fix/3.0_merge_main
2023-01-30 10:17:59 +08:00
98e174606c
Merge pull request #19578 from taosdata/FIX/TD-21942-main
...
enh: trim the last WAL idx file which is not fsynced
2023-01-30 09:31:28 +08:00
5e7b6602c5
Merge branch 'main' of https://github.com/taosdata/TDengine into fix/3.0_merge_main
2023-01-29 14:05:15 +08:00
f5c4ca3380
fix: wal ref
2023-01-28 17:17:55 +08:00
2a71d47668
fix: wal ref
2023-01-28 16:11:17 +08:00
95493432ed
enh: trim the last WAL idx file which is not fsynced
2023-01-16 14:12:26 +08:00
9a0c2805f4
merge main
2023-01-13 13:54:04 +08:00
97ce96a2fa
enh: not fsync idx file in walFsync
2023-01-11 19:24:51 +08:00
df6e9631e1
fix: synchronize access within walFsync
2023-01-10 11:24:05 +08:00
398161eb5f
Merge branch 'main' into merge/mainto3.0_0108
2023-01-09 10:00:53 +08:00
2890a8cb96
fix: return error on failing to truncate raft log
2022-12-28 20:34:40 +08:00
0c4ade9373
fix: update sync node commitIndex only if matchTerm equals currentTerm upon heartbeat
2022-12-28 20:25:52 +08:00
c4fad84c7c
enh: reset commitVer in WAL on restore
2022-12-28 20:25:52 +08:00
fbc9e77004
Merge branch '3.0' into refact/submit_req
2022-12-26 10:26:17 +08:00
ed3a0c1dc1
chore: vgId format uniform
2022-12-24 23:44:01 +08:00
0c269c317b
merge 3.0
2022-12-23 17:50:05 +08:00
6cbcbd1402
remove assert
2022-12-21 15:37:50 +08:00
50729f0cc4
refactor: remove assert
2022-12-21 14:16:28 +08:00
2a7de0cd46
refactor: remove assert
2022-12-21 13:54:14 +08:00
41dcd32b8e
refact: adjust some logs
2022-12-20 15:01:03 +08:00
48f7fc5c3d
Merge branch '3.0' into refact/submit_req
2022-12-12 18:11:17 +08:00
a6fff95795
fix: close wal ref
2022-12-09 17:18:21 +08:00
271822ebc4
refactor: support submitreq2
2022-12-08 01:55:47 +08:00
6d94afe48f
Revert "refact: adjust some assert cases"
2022-12-07 21:51:17 +08:00
d5126d469a
refact: replcase ASSERT with tAssert
2022-12-07 18:42:48 +08:00
f69a188f7e
refact: replcase ASSERT with tAssert
2022-12-07 18:37:40 +08:00
3f1a0cefbb
fix: walSaveMeta ahead of deleting WAL files as gc in walEndSnapshot
...
delete WAL files in descending order during walRollback
2022-12-06 00:51:45 +08:00
d9c261cd13
refact: remove some error codes
2022-12-03 10:17:02 +08:00
68f8ebac9d
Merge branch 3.0 to FIX/TD-19334-3.0
2022-11-25 13:10:06 +08:00
f68e41a40e
enh: initialize log buffer again after receiving a complete snapshot
2022-11-23 15:57:40 +08:00
95f8e96eb5
fix: align firstVer with lastVer too if no WAL logs exist
2022-11-22 23:42:08 +08:00
1344671869
fix(wal): add debug log
2022-11-22 17:24:43 +08:00
6f5d90658e
fix(wal): add debug log
2022-11-22 16:40:18 +08:00
b600b4c8bc
refactor(sync): modify wal pInfo++, to find which case can not pass
2022-11-21 16:18:50 +08:00
051b73c1ed
Merge branch 3.0 to FIX/TD-19334-3.0
2022-11-17 17:28:11 +08:00
2b5257f905
fix: memory lead found in asan
2022-11-16 20:45:15 +08:00
c8f9707d5a
Merge pull request #18203 from taosdata/feature/stream
...
enh(stream): add tbname map cache
2022-11-16 17:15:30 +08:00
201ce10fbc
fix: restore wal info changes
2022-11-16 15:45:40 +08:00
ab57d069d4
enh(stream): add tbname map cache
2022-11-16 15:18:31 +08:00
b62a85dfdc
Merge pull request #18192 from taosdata/feature/stream
...
fix(stream): timer refer stream task
2022-11-16 12:40:50 +08:00
015c00aaca
fix(stream): timer refer stream task
2022-11-16 11:16:32 +08:00
555bf7d4cb
Merge pull request #18181 from taosdata/feature/stream
...
fix(stream): check task
2022-11-16 08:11:30 +08:00
6478fcb2c1
fix(stream): check task
2022-11-15 22:32:52 +08:00
ede50530f3
Merge pull request #18178 from taosdata/feature/wal
...
fix(wal): set commit ver and snapshot ver after restore
2022-11-15 21:26:03 +08:00
20e31aacca
fix(wal): set commit ver and snapshot ver after restore
2022-11-15 19:48:16 +08:00
a9bf05bc01
refactor: stream meta ref count
2022-11-15 19:11:42 +08:00
efde22f081
fix: memory leak
2022-11-15 11:59:29 +08:00
7c8b2867ce
enh(stream): add stream task info into information schema
2022-11-14 19:23:42 +08:00
aa9e935854
enh(stream): set path max length to 1024
2022-11-14 14:46:50 +08:00
bc469f7f7b
fix(stream): delete tb should be checked in write thread
2022-11-08 11:30:29 +08:00
5878c9a31a
fix: impl ring log buffer
2022-11-07 14:27:22 +08:00
77a6cf503e
Merge pull request #17824 from taosdata/FIX/TS-1984-3.0
...
fix: set firstTrial as false if error encountered in loop of walScanLogGetLastVer
2022-11-02 15:49:48 +08:00
8a5130a88d
fix: set firstTrial as false if error encountered in loop of walScanLogGetLastVer
2022-11-02 10:21:40 +08:00
cec6642700
Merge pull request #17811 from taosdata/feature/stream
...
fix(tmq): handle adding subtable for subscribing stb
2022-11-01 21:09:43 +08:00
f0a66602da
fix(taosx): set version
2022-11-01 19:28:42 +08:00
871757240a
Merge pull request #17777 from taosdata/FIX/TS-1984-3.0
...
fix: fsync wal files and meta if data size not synced yet beyond a threshold
2022-11-01 18:53:10 +08:00
94e7c79ca0
fix(tmq): handle adding subtable for subscribing stb
2022-11-01 18:18:26 +08:00
4e46b78aa0
fix: update info of WAL entry found within loop in walScanLogGetLastVer
2022-11-01 17:56:24 +08:00
3c1c95c5e6
enh: turn error on WAL data size to recover exceeding a limit as warn
2022-11-01 17:26:31 +08:00
4237cafb21
enh: rename walRecoverSizeLimit to walFsyncDataSizeLimit
2022-11-01 17:19:33 +08:00
ffa6906a9f
Merge branch 'FIX/TS-1984-3.0' of github.com:taosdata/TDengine into FIX/TS-1984-3.0
2022-11-01 15:52:25 +08:00
6907e3dfe3
enh: scan forward step by step in walScanLogGetLastVer
2022-11-01 15:25:18 +08:00
a031010a9e
Merge pull request #17721 from taosdata/FIX/TD-19673-3.0
...
enh: check completeness of WAL log entries after repairing
2022-11-01 13:59:34 +08:00
26778f29da
enh: add config for persist wal at the beginning of vnodeCommit
2022-11-01 10:42:13 +08:00
07db554fab
enh: persist wal at the beginning of vnodeCommit
2022-11-01 00:00:18 +08:00
cccfa17027
fix: fsync wal files and their meta if data size unsynced exceeds a limit
2022-10-31 23:58:49 +08:00
9652029121
fix: update testcases of walRollback in walMetaTest.cpp
2022-10-31 21:08:54 +08:00
1d0eae197d
fix: not allow to rollback WAL across snapshot point
2022-10-31 21:06:23 +08:00
fd3a1de904
fix: use the relative pos during searching in walScanLogGetLastVer
2022-10-28 11:47:44 +08:00
31a0421878
enh: check completeness of WAL log entries after repairing
2022-10-27 15:55:22 +08:00
2a3448552e
test(wal): rollback multiple file
2022-10-26 18:03:25 +08:00
a16b010e9f
fix: memory leak
2022-10-26 11:35:15 +08:00
75e5e490fd
refactor(stream): recover and fill history
2022-10-26 00:31:00 +08:00
25bb914ac1
merge 3.0
2022-10-21 16:30:43 +08:00
1c26f1b53e
fix(wal): reference
2022-10-21 16:13:55 +08:00
05d04dcd2d
Merge remote-tracking branch 'origin/3.0' into enh/TD-19461
2022-10-21 11:07:48 +08:00
6c3cd3e19e
test: adjust log
2022-10-21 10:39:29 +08:00
c4e568d41e
Merge pull request #17523 from taosdata/FIX/TD-19673-3.0
...
enh: provide more info on misalignment detection of idx entries after…
2022-10-21 09:06:57 +08:00
ce1b6147a0
enh(client): add debug log
2022-10-20 15:34:27 +08:00
06f5d91d21
enh: provide more info on misalignment detection of idx entries after appending
2022-10-20 15:17:52 +08:00
3b0968d7f9
enh: enable the -Wformat=2 compiler option
2022-10-19 09:24:15 +08:00
3cbfd2d160
Merge pull request #17392 from taosdata/FIX/TD-19183-3.0
...
enh: not allowed to start on disk space unavailable
2022-10-18 17:46:14 +08:00
cf2b6ccbfb
enh: write idx ahead of writing its log entry in WAL
2022-10-18 11:23:24 +08:00
af9aa9d783
enh: assert on alignment of idx entries in walWriteIndex
2022-10-18 11:23:13 +08:00
f0427c9322
fix: ftrunc on appending WAL log failure for recovery in walWriteImpl
2022-10-18 11:22:21 +08:00
3190ef65a3
fix: deal with misaligned idx entries in walCheckAndRepairIdxFile
2022-10-18 11:21:48 +08:00
a4fd2def98
fix: resolve a typo in errormsg printing in walScanLogGetLastVer
2022-10-17 14:22:20 +08:00
9a3ac0ca57
Merge branch '3.0' into feature/stream
2022-10-17 10:17:45 +08:00
6f9153c6e3
fix: error handling
2022-10-17 10:13:10 +08:00
e3f50c0d48
fix: align vers.firstVer with vers.snapshotVer+1 in WAL meta info
2022-10-16 13:10:57 +08:00
616eb5f903
Merge branch '3.0' into FIX/TD-19239-3.0
2022-10-16 00:29:01 +08:00
b6c31ee67d
fix: fsync the current wal log and idx files at first to ensure validity of meta in walSaveMeta
2022-10-15 23:57:37 +08:00
7fb827debd
fix: refactor walScanLogGetLastVer, walCheckAndRepairMeta, and walCheckAndRepairIdx.
...
* search for the last entry in the contiguous range of valid WAL from the last pos fsynced as firstTrial
* find the last entry before the last pos fsynced backwards as second trial
* reserve sufficient space for computing CRC32 checksum, esp. of WAL record body
* rebuild meta info to resolve potential misalignment between lists of meta and actual log files
* retract commitIndex and appliedIndex to lastLogIndex if needed
* put an upper size limit on possibly corrupted WAL range to be recovered
2022-10-15 23:56:31 +08:00
c693384ec4
enh: replace asserts with error logs in walReadVer
2022-10-14 22:57:28 +08:00
948fa60ef0
fix: check the length of remaining content for computing CRC32 chksum of WAL record body
2022-10-14 22:57:28 +08:00
63b13db10f
fix: eliminate asserts on getting and committing log entries
2022-10-14 22:57:27 +08:00
df18ddbbb5
enh: remove WAL log file corrupted completely
2022-10-14 22:57:27 +08:00
9055f63742
fix: ftruncate WAL log after the last valid WAL record properly in walScanLogGetLastVer
2022-10-14 22:43:41 +08:00
76865258cf
enh: protect WAL meta by flushing to a tmpfile at first and then renaming
2022-10-14 22:32:15 +08:00
36bf800e15
fix: improve fault tolerance on disk space full with error handling enhanced
2022-10-14 22:02:27 +08:00
6f8d4d6955
fix(tmq): push msg
2022-10-14 13:31:05 +08:00
db6b0503a1
refactor(stream): change state table name
2022-10-12 14:34:04 +08:00
69352e6e89
fix tbanme length check
2022-10-12 10:50:54 +08:00
41474e40bf
enh(wal): auto fix
2022-10-12 02:08:58 +08:00
26fae13c61
refactor(stream): state/session map id to tbname
2022-10-11 15:27:56 +08:00
fbfae317b3
refactor
2022-10-11 02:09:15 +08:00
b1594a6857
fix coverity scan
2022-10-10 11:44:17 +08:00
78b4d10249
optimize fix condition
2022-09-22 00:54:23 +08:00
a6733dbbd1
feat(wal): auto fix corrupt file
2022-09-21 23:51:24 +08:00
756a2d3d5d
merge 3.0
2022-09-19 19:29:20 +08:00
1604d72967
add log for wal
2022-09-19 17:40:38 +08:00
86c0fb56e0
enhance stream backend for sma
2022-09-19 17:27:25 +08:00
b6ddf5bfd7
fix: update wal reader's curFileFirstVer
2022-09-08 18:30:15 +08:00
c5874ca892
fix: add filter logic for tmq in stable wal
2022-08-31 18:32:28 +08:00
d7b804e02e
fix stream load task
2022-08-24 10:42:34 +08:00
2d628491b2
feat(stream): support tdb state backend
2022-08-23 19:30:04 +08:00
18f6fb6123
enh(tmq): speed up consumer recover
2022-08-10 16:04:03 +08:00
f0f4748320
fix(util): concurrently hash iterate
2022-08-05 16:14:19 +08:00
56f2e49764
refactor: adjust logs
2022-08-02 17:12:07 +08:00
bd31c9da5a
Merge pull request #15657 from taosdata/fix/dnode
...
fix: write snapshot after apply queue is empty
2022-08-01 20:13:08 +08:00
b084dd8ee6
feat(tq): transport snapshot
2022-08-01 19:12:37 +08:00
99f29f6455
refactor: adjust log
2022-08-01 17:23:52 +08:00
bb6fabcc11
refactor(stream): use vnode meta
2022-07-30 20:20:12 +08:00
6d67b17113
enh(wal): add lock to guarantee read behaviour
2022-07-30 16:35:49 +08:00
0c942e622a
enh(wal): remove file failure handling
2022-07-28 13:37:57 +08:00
00acf4520c
refactor(stream): remove option
2022-07-27 16:49:03 +08:00
3fa5791889
fix(stream): dispatch
2022-07-27 10:52:25 +08:00
ecb7a9c133
fix(wal): snapshot end with no deleting
2022-07-26 14:53:39 +08:00
c2164dd8aa
fix(wal): snapshot end with no deleting
2022-07-26 14:13:03 +08:00
0b8e9af8ec
refactor(wal)
2022-07-26 13:33:26 +08:00
222e925644
feat(wal): ref
2022-07-25 16:40:19 +08:00
5b654779ba
Merge pull request #15372 from taosdata/fix/tsim
...
test: regression case
2022-07-25 15:23:41 +08:00
095f6aa4e0
fix(tmq): correctly set reader status
2022-07-23 20:07:16 +08:00
5d20804e76
fix(wal): use after free
2022-07-20 17:10:41 +08:00
b74560537c
fix(stream): check task exist
2022-07-20 16:25:20 +08:00
bf37f3fa68
feat(wal): remove wal log size limitation
2022-07-20 16:15:53 +08:00
a3133b9f6d
feat(wal): log applied ver
2022-07-20 10:55:28 +08:00
bc1af6deaf
refactor(stream): batch optimization for submit msg
2022-07-19 15:52:08 +08:00
053786e6b4
fix(stream): data exec
2022-07-18 15:50:36 +08:00
d637ffe22c
refactor(tmq): prepare only needed
2022-07-12 14:10:22 +08:00
Benguang Zhao
wangmm0220
Xiaoyu Wang
Haojun Liao
Liu Jicong
Haojun Liao
Hongze Cheng
Shengliang Guan
kailixu
Shengliang Guan
Minghao Li
benguang.zhao
Minglei Jin