xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
76,779 Commits 3 Branches 0 Tags 479 MiB
Commit Graph

184 Commits

Author SHA1 Message Date
liuyao 2de6736160
feat(stream): add new trigger continuous_window_close (#30125) 
* opt stream build twa result

* opt force window close memory

* feat(stream):optimize new interval and scan operator

* adj log

* opt code

* opt code

* fill history

* fix issue for fill history

* add ci

* feat(stream): add new stream nonblock interval operator

* adjust code

* use new scan operator

* use new scan operator

* add log

* fix issue

* recover stream scan next function

* fix issue

* fix issue

* fix issue

* ignore disorder data

* fix issue

* fix issue for interval sliding

* fix issue

* fix ci issue

* fix ci issue

* fix ci issue

* add semi && final nonblock interval operator

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* refactor(stream): track the msgId for each upstream tasks.

* fix(stream): fix race condition.

* fix(stream): update the task last msgId when putting into input queue succ.

* fix issue

* fix issue

* put recalculate data to rocksdb

* fix issue

* fix issue

* enh(query)[TD-33071]: add support for saving and restoring tsdbReader scan progress

- Implement functionality to save scan progress during tsdbReader operations
- Enable resuming scans from the last saved position

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* add rec interval check

* enh(stream):add recalculate tasks.

* enh(stream): support the re-calculate the tasks.

* fix issue && do refactor

* do refactor

* fix issue

* fix issue

* update backend opt

* add new interface

* add new session operator

* support blob

* add new session operator

* fix issue

* add rec state for rec task

* fix invalid read

* add new session window

* enh(stream): update the stream tasks backend.

* new session operator

* add pull data

* fix(stream): fix error in expand stream backend.

* fix issue

* fix issue

* fix issue

* merge code

* fix issue

* fix(stream): check for null ptr.

* fix(stream): add more check.

* fix issue

* fix issue

* fix issue

* add debug code

* fix issue

* fix issue

* fix issue

* set rec end flag

* fix(stream): opt re-calculate stream tasks.

* fix issue

* fix issue

* add new operator

* enh(stream): dispatch recalculate block to agg tasks.

* fix issue

* fix issue

* fix(stream): adjust filter.

* fix issue

* refactor

* refactor(stream): adjust the recalculate end block.

* fix issue

* fix(stream): set correct create trigger block.

* fix issue

* fix(stream): fix error in build recalculate end block.

* fix(stream): check null ptr.

* add stream client && fix issue

* fix mem leak

* fix(stream): free msg.

* add stream client

* fix(stream): fix error.

* add stream client && fix issue

* add stream client

* refactor(stream): set the recalculate task complete.

* add wend and group_id for session window dest table

* feat(stream): refactor and set the recalcul agg task complete.

* add cfg for adapter

* fix issue

* add state && event operator

* feat(stream): support fill-history task.

* add new fill operator

* fix(stream): set correct backend when updating fill-history task to recalculate task.

* add new fill operator

* fix(stream): s2025-03-06 11:10:31.272
et ops always open in scan tsdb

* fix(stream):set the correct taskType for sink task.

* new fill operator

* adj stream fill operator

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* adj test

* fix issue

* fix(stream): fix issue

* fix(steam): fix issue

* fix(steam): fix issue

* fix(steam): fix issue

* fix(steam): fix issue

* fix(stream): fix issue

* fix(stream): fix issue

* fix(stream): fix issue

* fix: ut com error

* fix(stream): fix mem leak and adjust operator type check rule

* fix(stream): fix mem leak and adjust test case

* refactor code

* fix(stream): free items.

* fix(stream): free fix memory leak.

* fix(stream): fix syntax error.

* fix: ignore unexpect block

* fix: adjust op type

---------

Co-authored-by: Haojun Liao <hjliao@taosdata.com>
Co-authored-by: Jinqing Kuang <kuangjinqingcn@gmail.com>
Co-authored-by: yihaoDeng <luomoxyz@126.com>
 2025-03-14 20:14:01 +08:00
Kaili Xu 13f9bddf3d
feat: support customized taos/taosd (#29736) 
* feat: support TDAcoreOS

* chore: cmake options for TD_ACORE

* chore: disable lemon for TD_ACORE

* chore: add lzma2 and msvcregex

* chore: cmake for lzma2

* chore: adapt for TD_ACORE

* chore: adapt strcasecmp for TD_ACORE

* chore: adapt for geos/threadName

* chore: build adapt for TD_ACORE

* chore: build adapt for TD_ACORE

* chore: build adapt for TD_ACORE

* chore: build adapt for TD_ACORE

* chore: build adapt for TD_ACORE termio

* chore: refact transComm.h for TD_ACORE

* chore: refact transportInt.h for TD_ACORE

* chore: refact trans.c for TD_ACORE

* chore: refact trpc.h for TD_ACORE

* chore: refact transCli.c/transComm.c/transSvr.c for TD_ACORE

* chore: refact uv.h for TD_ACORE

* chore: refact geosWrapper.h for TD_ACORE

* chore: refact token/builtins/udf for TD_ACORE

* chore: refact rocks for TD_ACORE

* chore: refact tsdbCache.c for TD_ACORE, use LRU cache for last/last_row, not use rocksdb

* chore: refact FAIL to _ERR to solve conflicts for TD_ACORE

* chore: restore lemon.c/lempar.c

* chore: support build lemon for TD_ACORE

* chore: refact trpc and siginfo_t for TD_ACORE

* chore: refact timezone for TD_ACORE

* chore: refact lz4 for TD_ACORE

* chore: refact TD_ACORE to make compile pass

* chore: code optimization for TD_ASTRA

* feat: support run taos with taosd integrated

* feat: support invoke taos shell

* feat: support invoke taos shell

* feat: support invoke taos shell

* chore: code optimization

* chore: fix undefined reference problem os TD_ASTRA

* chore: resolve compile problem for TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix getpid

* chore: fix typo

* chore: set stack size and ajust min pack size for TD_ASTRA

* chore: fix pthread create parameters

* chore: chmod adapt for TD_ASTRA

* chore: fix trans compile problem

* chore: adapt chmod for TD_ASTRA

* chore: byte alignment for TD_ASTRA

* chore: more code for adaption of TD_ASTRA

* chore: more code for adaption of TD_ASTRA

* chore: more code for adaption of TD_ASTRA

* chore: byte alignment for TD_ASTRA

* chore: conditional compile option

* chore: adapt for TD_ASTRA

* chore: adjust taosPId and msvcregex for TD_ASTRA

* chore: log dir separator for wal build name

* chore: fix type of pointer parameter

* chore: fix compile problem of tsdbGetS3Size

* enh: get last ver from wal log for TD_ASTRA

* enh: refact wal meta ver

* enh: refact wal meta ver

* fix: typo of taosUcs4Compare

* enh: process return value of CI

* chore: more code for TD_ASTRA adaption

* chore: return value of taosCloseFile in walMeta.c

* chore: fix compile problem

* chore: fix compile problem of TD_ASTRA

* fix: update macro for tq and stream task

* chore: code optimization for TD_ASTRA

* chore: restore create log and init cfg interface

* chore: restore strncasecmp and strcasecmp

* fix: adjust the field position of SDataBlockInfo

* fix: pragma pack min size

* fix: pragma pack min size

* chore: more code for TD_ASTRA adaption

* fix: type of parameters

* chore: adapt strncasecmp and strcasecmp for TD_ASTRA

* chore: restore interface of init log

* enh: pack push optimization

* fix: taos init cfg

* add astra support

* fix: fetch the value of suid

* chore: switch of  build with udf

* add temp code

* chore: more code for TD_ASTRA adaption

* chore: add macro ERRNO to replace errno

* chore: bytes align for TD_ASTRA

* fix: remove obsolete codes

* enh: support USE_UDF macro

* fix compile error

* fix: resolve redefinition problem

* fix: compile problem of log.cpp

* fix: compile problem of osTimezone

* fix: resolve compile problem of udf

* fix: pragma definition on windows

* fix: ucs4 and stpncpy for TD_ASTRA

* fix: memory align problem for TD_ASTRA

* enh: solve memory leak for TD_ASTRA_RPC

* fix: compile problem of taosSetInt64Aligned

* fix: restore mndSubscribe.c

* fix: scalar for udf

* chore: code adaption for TD_ASTRA

* chore: code optimization for TD_ASTRA

* fix: typo of add definition

* fix: typo of macro in tudf.h

* chore: remove void to make CI pass

* enh: move macro from cmake.platform to cmake.options

* enh: byte align for hash node and error code

* chore: restore the size for lru cache

* enh: restore some code about pack push

* chore: restore the pack push in tmsg.h

* fix: add macro of pack pop for windows

---------

Co-authored-by: yihaoDeng <luomoxyz@126.com>
 2025-03-14 13:32:13 +08:00
Simon Guan 1a9f75e663 Merge branch '3.0' into fix/internal 2025-03-04 13:47:03 +08:00
Haojun Liao 67853b4a54 enh(stream): add checkpoint queue for source tasks. 2025-02-28 13:46:40 +08:00
Simon Guan b91dd10846 enh: add log level to each log entry 2025-02-27 09:55:11 +08:00
Haojun Liao 78e976caff refactor(stream): limit stream input q size. 2025-02-11 22:57:28 +08:00
Haojun Liao 093233203d
Revert "refactor(stream): do some internal refactor ." 2025-01-24 17:11:15 +08:00
Shengliang Guan 3089e3d904
Merge pull request #29544 from taosdata/fix/tag 
refactor(stream): do some internal refactor .
 2025-01-24 16:27:22 +08:00
Haojun Liao d344228242 enh(stream): reduce the in-queue size to be 10MiB. 2025-01-08 19:11:41 +08:00
54liuyao f6500b3551 ci(test):delete invalid code&& add ci 2024-12-26 11:16:02 +08:00
54liuyao 4adf24b466 ci(stream):add stream unit test 2024-12-23 18:25:25 +08:00
54liuyao 9866b9437d try fix window com error 2024-12-20 11:27:29 +08:00
54liuyao 979a669ecd try fix window com error 2024-12-20 09:37:14 +08:00
54liuyao 5b10753ac0 add ut 2024-12-19 18:23:57 +08:00
Shengliang Guan f9124dad6c
Merge pull request #28623 from taosdata/fix/reset 
fix(stream): record the failed checkpointId, after receive reset task info.
 2024-12-17 18:16:36 +08:00
Haojun Liao daecc4a290 other: merge main 2024-12-17 17:31:56 +08:00
Haojun Liao d2146a2efb other: merge main 2024-12-17 09:23:13 +08:00
Haojun Liao aa12038fc4 fix(stream): record the failure of dispatch msg, and set the update node id. 2024-12-10 17:58:53 +08:00
Shengliang Guan e99b888654 Merge branch 'main' into merge/mainto3.0 2024-12-09 19:06:06 +08:00
Haojun Liao cea647daf6 refactor: limit the number of items in inputq to be 5120 2024-12-07 12:21:28 +08:00
Haojun Liao 187997d5f0 fix(stream): record the failed checkpointId, after receive reset task info. 2024-11-14 17:40:37 +08:00
Haojun Liao b2b5a14d9d fix(stream): reset task add the failed chkptId info. 2024-11-14 17:38:40 +08:00
Haojun Liao 06c02c14c2 other: merge main 2024-11-11 15:27:53 +08:00
54liuyao ba31bf2c52 Merge branch '3.0' of https://github.com/taosdata/TDengine into fix/TD-30837 2024-11-06 09:42:50 +08:00
Haojun Liao bb05b9b912 fix(stream): record the failed checkpointId, after receive reset task info. 2024-11-04 10:11:22 +08:00
Haojun Liao e7b3a3aec8 fix(stream): reset task add the failed chkptId info. 2024-11-01 18:58:22 +08:00
Haojun Liao cacc7db61c fix(stream): set the correct start ts. 2024-10-28 19:33:11 +08:00
Haojun Liao c0932d419d fix(stream): fix error in pause/resume for force_window_close 2024-10-28 18:07:11 +08:00
Haojun Liao 380f433499 fix(stream): use the refId in stream meta list, in order to avoid access already freed stream tasks. 2024-10-27 15:40:17 +08:00
54liuyao d13819bbf4 Merge branch '3.0' of https://github.com/taosdata/TDengine into fix/TD-30837 2024-09-30 09:05:46 +08:00
Haojun Liao 4c98786352 fix(stream): use meta id instead of ptr. 2024-09-29 16:12:46 +08:00
Haojun Liao c5eca0486e Merge remote-tracking branch 'origin/fix/TD-30837' into fix/TD-30837 
# Conflicts:
#	source/dnode/vnode/src/tq/tq.c
#	source/libs/stream/inc/streamInt.h
#	source/libs/stream/src/streamSched.c
 2024-09-19 16:39:54 +08:00
Haojun Liao e1d8e98b11 feat(stream): support the force_window_close trigger. 2024-09-19 16:23:51 +08:00
Haojun Liao b494163f28 refactor: check return value. 2024-09-14 13:39:23 +08:00
Haojun Liao 6ba428fbf8 other: merge 3.0. 2024-08-01 11:15:49 +08:00
Haojun Liao 7201526674 fix(stream): check return value. 2024-07-22 09:13:39 +08:00
Haojun Liao 2ca6726922 refactor: check return value for each function. 2024-07-18 22:52:32 +08:00
Haojun Liao fb3fe03c1f fix(stream): to avoid repeatly start checkpoint timer if previous timer is not started yet. 2024-07-18 14:57:39 +08:00
Haojun Liao b35d107bf9 refactor: check return value for each function. 2024-07-17 18:55:04 +08:00
Haojun Liao 2b00d6549d refactor: check return value for each function. 2024-07-17 17:57:08 +08:00
Haojun Liao 2aa4d6028b refactor: check return value for each function. 2024-07-17 16:56:36 +08:00
Haojun Liao c4cde6f268 fix(stream): mark the timer launched by which checkpoint procedure. 2024-07-15 14:23:33 +08:00
Haojun Liao 6d64d0e081 fix(stream): fix memory leak. 2024-07-01 15:34:04 +08:00
Haojun Liao 025437df0c fix(stream): fix error in checkpoint consensus. 2024-06-27 19:25:18 +08:00
Haojun Liao 48bf5eb4d8 other: merge 3.0 2024-06-27 19:23:06 +08:00
Hongze Cheng b761bd2e38 enh: port main change to 3.0 2024-06-26 10:32:15 +08:00
Haojun Liao 1818edcb2b fix(stream): consensus the start checkpoint id, and extract the streamhb related functions into a new file. 2024-06-26 10:18:32 +08:00
Haojun Liao 21e1763ff4 fix(stream): set correct flag for checkpoint. 2024-06-22 11:16:20 +08:00
Haojun Liao 4490f40c96 fix(stream): do some internal refactor. 2024-06-14 14:43:57 +08:00
Haojun Liao 0f8c0fa8cb fix(stream): add dispatch data monitor to handle the network broken problem that may cause the stream process frozen. 2024-06-13 14:56:20 +08:00