Commit Graph

681 Commits

Author SHA1 Message Date
Ganlin Zhao 728e42f723 fix: add forbid system table functions 2023-06-14 11:10:51 +08:00
Dingle Zhang 984211f3b4
Feature/3.0 geometry (#21037)
* Add GEOMETRY data type and make sql.c able to parse it. The GEMETRY works like BINARY so far.

* add GEOMETRY type into gConvertTypes to fix some issues like DELETE calling

* change some test cases to make sure no same timestamp is inserted, and add my smoketest.sh

* Add a function MakePoint() and introduce a lib geometry

* implement sql functions GeomFromText() and AsText()

* Use GEOS *_r funcions instead for thread safety

* Handle with TSDB_DATA_TYPE_GEOMETRY when INSERT geometry data by converting WKT.
Add geosWrapper to wrap the basic GEOS functions for TDEngine.

* refactor AsText and MakePoint functions to be like GeomFromText

* Show WKT when print geometry data in screen
Dump hex data when dump geometry data in a file

* define TYPE_BYTES item for TSDB_DATA_TYPE_GEOMETRY, which casued some strange issues.

* set number of decimals of WKT to 6

* Implement SQL function Intersects()

* refactor geometry sql functions

* Add geosErrMsgeHandler() to get the GEOS error detail

* use threadlocal to instantiate SGeosContext
call destroyGeosContext() only if the thread exists

* remove SGeosContext *context param for all geometry functions since we use thread local one,
so that all caller do not need to know the context.

* Modify Intersects() to call PreparedIntersects() when one of param is a constant, which has higher performance.

* rename prepareFn() to initCtxFn() to avoid confusion with PreparedFn

* Add prefix "ST_" for all geometry functions

* move getThreadLocalGeosCtx() and destroyThreadLocalGeosCtx() into util,
so that all unit test tools can compile

* Add unit test for geometry lib, only test MakePoint so far

* refactor and enhance existing cases in geomFuncTest

* implement NULL type and NULL value test for geomFuncTest

* add test on geomFromText()

* add unit test on AsText() in geomFuncTest

* combine some makePointFunction test items

* add intersectsFunctionTwoColumns test
refactor on callGeomFromTextWrapper functions

* enhance intersectsFunction test to add cases like input constant , NULL type, NULL value, or wrong content

* add more cases into intersectsFunction test

* Add basic test on geometry in system test

* Add ST_GeomFromText and ST_AsText function test in system test on geometry

* add ST_Intersects function test in system test on geometry

* support to check expectedErrno in system test on geometry

* adjust geomTest unit test and geometry system test

* add geometry data type and functions in doc english version

* implement touchesFunction() in geometry lib
refactor geometry relation functions model

* separate gemFuncTest into several src files

* add unit test on touchesFunction

* support sql function ST_Touches()
add system test on ST_Touches

* add docs for ST_Touches()

* Add ST_Contains()

* Add ST_Covers()

* Add ST_Equals()

* add swapAllowed param for geomRelationFunction()
read geom2 earlier intead of at doGeosRelation()

* Add ST_ContainsProperly()

* build on windows

* Merge from 3.0 to 3.0_geometry

* change macro definition TSDB_DATA_TYPE_GEOMETRY as the last one for compatibility

* change '\\NULL' to 'NULL' back in shellDumpFieldToFile()

* add /usr/local/include into include directory

* add /usr/local/inlcude and /usr/local/lib in cmake.platform for DARWIN
2023-05-24 15:36:46 +08:00
Ganlin Zhao a3ca23b688 forbid interp ignoring null value used for multiple cols 2023-05-19 16:43:39 +08:00
Ganlin Zhao 6879a784ae reserve value node 2023-05-16 15:07:07 +08:00
Ganlin Zhao ace27326b0 enable ignore_null value param in client 2023-05-16 14:13:40 +08:00
Ganlin Zhao 9581dcae7f enh: add param for to_unixtimestamp to return timestamp type 2023-04-18 14:17:22 +08:00
Ganlin Zhao d1544e8859 fix: fix illegal usage of _isfilled/_irowts 2023-04-13 14:05:04 +08:00
Ganlin Zhao 3078e2d487 feat(query): INTERP support boolean type 2023-04-10 18:35:35 +08:00
Xiaoyu Wang c134432c45 merge main 2023-04-06 09:31:45 +08:00
Ganlin Zhao 4819b22a21 fix: fix tsim crash on windows due to invalid input to strftime 2023-04-04 13:40:53 +08:00
Ganlin Zhao 23ec9287c9 fix: fix tsim crash on windows due to invalid input to strftime 2023-04-04 10:33:41 +08:00
Xiaoyu Wang 16394862a6 enh: last_row is keep order function 2023-04-03 18:11:46 +08:00
Ganlin Zhao 6c64b760a0 fix: fix tsim crash on windows due to invalid input to strftime 2023-04-03 17:58:43 +08:00
dapan1121 a8da9f31e7 fix: join query error 2023-03-30 18:04:32 +08:00
Haojun Liao 1093c64586
Merge pull request #20649 from taosdata/fix/TD-23272
fix:add combine function for last row
2023-03-29 10:43:49 +08:00
Ganlin Zhao 2198ae2060 fix(query): spread/elapsed using sma result for computation 2023-03-28 10:49:49 +08:00
54liuyao 3a12e36b4b fix:add combine function for last row 2023-03-27 14:13:17 +08:00
54liuyao ffa59f69aa fix:add combine function for groupKey 2023-03-23 15:20:45 +08:00
dapan1121 05c7bc170e
Merge pull request #20098 from taosdata/feat/TS-2671
feat(query): optimize percentile function performance
2023-02-23 10:24:06 +08:00
Ganlin Zhao 39d38c15f1 fix param check 2023-02-22 14:08:03 +08:00
Ganlin Zhao 61140d3143 fix parameter check 2023-02-22 11:09:23 +08:00
Ganlin Zhao 84765ff670 refactor code 2023-02-22 10:47:49 +08:00
Ganlin Zhao 7e01d092a2 optimize repeat scan 2023-02-22 09:25:18 +08:00
Ganlin Zhao 28693ce285 feat(query): allow percentile function to take multiple params
to enhance performance
2023-02-20 14:07:24 +08:00
Shengliang Guan 3016f35e9f Merge branch '3.0' into merge/mainto3.0_0105 2023-01-05 16:08:23 +08:00
Ganlin Zhao 763f702fff fix assert in builtims.c 2022-12-30 10:50:40 +08:00
Ganlin Zhao 5da2fe9bc5 feat(query): add _isfilled pseudocolumn to indicate data is origin
or filled.
2022-12-14 16:56:10 +08:00
Shengliang Guan 6d94afe48f
Revert "refact: adjust some assert cases" 2022-12-07 21:51:17 +08:00
Shengliang Guan d5126d469a refact: replcase ASSERT with tAssert 2022-12-07 18:42:48 +08:00
Shengliang Guan f69a188f7e refact: replcase ASSERT with tAssert 2022-12-07 18:37:40 +08:00
dapan1121 0ecb0d26f2
Merge pull request #18694 from taosdata/fix/TS-2144
enh(query): add timetruncate function ignore timezone option for 1d
2022-12-05 17:18:18 +08:00
Ganlin Zhao e18da31a5d enh(query): add timetruncate function ignore timezone option for 1d 2022-12-05 12:00:12 +08:00
Xiaoyu Wang 37b65c16c8 enh: ins_tables count optimize 2022-11-29 14:59:48 +08:00
Xiaoyu Wang d00f65a8fe Merge remote-tracking branch 'origin/3.0' into enh/3.0_planner_optimize 2022-11-18 15:05:19 +08:00
Ganlin Zhao a115ff1299 fix(query): fix time precision convert overflow
TD-20456
2022-11-16 17:25:57 +08:00
Xiaoyu Wang 6a50e9e271 Merge remote-tracking branch 'origin/3.0' into enh/3.0_planner_optimize 2022-11-16 10:58:06 +08:00
dapan1121 49a3623f1a
Merge pull request #18112 from taosdata/fix/TD-19892
fix(query): report error if certain function query stable has duplicate timestamps
2022-11-15 17:24:08 +08:00
Xiaoyu Wang 40c7562470 enh: optimize statement for querying the number of sub tables of the super table 2022-11-15 16:38:51 +08:00
Xiaoyu Wang 901c9b8f77 fix: some problems of parser 2022-11-14 16:53:01 +08:00
Ganlin Zhao 2b53f7be84 fix(query): report error if certain function query stable has duplicate
timestamps

TD-19892
2022-11-11 16:43:45 +08:00
Ganlin Zhao eec1b0bfb0 fix(query): report error if certain function query stable has duplicate
timestamps

TD-19892
2022-11-11 16:22:31 +08:00
Ganlin Zhao 1cba568604 fix(query): report error if certain function query stable has duplicate
timestamps

TD-19892
2022-11-11 15:22:00 +08:00
Ganlin Zhao a0e13e04fe fix(query): add mode with selectivity rows 2022-11-03 18:27:43 +08:00
chenhaoran 5fc0e2341e test:supplement muti-branch for query fuction testcase 2022-10-28 16:35:13 +08:00
Hui Li 4ac9c3694f
Merge pull request #17669 from taosdata/fix/coverage_glzhao
fix(query): comment out unused functions for code coverage
2022-10-26 20:49:21 +08:00
Ganlin Zhao 547e7694a8 fix(query): comment out unsed functions for code coverage 2022-10-26 15:06:56 +08:00
Ganlin Zhao c3362223d9 fix(query): comment out unsed functions for code coverage 2022-10-26 13:39:56 +08:00
dapan1121 3fa1f535f1
Merge pull request #17609 from taosdata/fix/interpStable
fix: remove interp stable limitation and add test case
2022-10-26 09:59:42 +08:00
Ganlin Zhao 3125f3ea26 remove unused code 2022-10-25 16:42:52 +08:00
Ganlin Zhao 7fa93222f4 remove interp unused code 2022-10-25 16:40:56 +08:00
Ganlin Zhao cf60bf98d6 fix: add elapsed partial/merge 2022-10-25 16:37:01 +08:00
Ganlin Zhao 3a4dc8f0da Merge branch '3.0' into fix/coverity_glzhao 2022-10-25 09:47:46 +08:00
Ganlin Zhao 1935d349d4 fix: coverity issues
CID: 399493
2022-10-24 17:47:12 +08:00
dapan1121 af6a9aeb13 fix: remove interp stable limitation and add test case 2022-10-24 17:46:21 +08:00
Haojun Liao f7fdeecc44
Merge branch '3.0' into feature/3_liaohj 2022-10-19 11:23:55 +08:00
Haojun Liao c7c99ae346 fix(query): update last_row callback function. 2022-10-19 09:33:54 +08:00
Haojun Liao e4940e82c5 fix(query): update last_row callback function. 2022-10-18 17:54:29 +08:00
Alex Duan e2637ab6a7 fix(query): improve performance for first function x10 quickly 2022-10-18 17:22:41 +08:00
Xiaoyu Wang 0228caaa0d enh: last cache optimize 2022-10-18 11:38:59 +08:00
Haojun Liao c8b180b129 fix(query): update the cached last query. 2022-10-17 19:14:06 +08:00
Haojun Liao 68842b9c64 refactor: do some internal refactor. 2022-10-10 14:55:58 +08:00
Xiaoyu Wang 73642ac3a2 feat: sql command 'show table tags from table_name' 2022-09-29 16:16:49 +08:00
Xiaoyu Wang 8caa36299c feat: add the _irowts pseudo column to output the timestamp of the interp 2022-09-27 15:23:24 +08:00
Xiaoyu Wang 0321b8c088 fix: avg function data scan optimization 2022-09-22 15:08:07 +08:00
Ganlin Zhao 0a6ec7c0d7 fix(query): restrict max/min function input type not including timestamp
type
2022-09-14 11:17:12 +08:00
Haojun Liao 717e6706bb fix(query): add data required flag for avg against super table. 2022-09-09 14:26:44 +08:00
Ganlin Zhao a34da949bc fix(query): support timestap type in max/min function
TD-16780
2022-08-31 18:57:45 +08:00
Ganlin Zhao c49aa8e1f9 fix(query): support timestap type in max/min function
TD-16780
2022-08-31 18:51:29 +08:00
Ganlin Zhao 0fbb4d7f9a fix(query): restrict using JSON type as input in string functions
TD-18664
2022-08-26 16:18:21 +08:00
Ganlin Zhao ab85eeaaaa fix(query): restrict using JSON type as input in string functions
TD-18664
2022-08-26 16:15:08 +08:00
Ganlin Zhao df2224988f fix(query): restrict interp input type to numerical type only
TD-18487
2022-08-18 17:02:52 +08:00
Ganlin Zhao cb4a40a7ca fix error 2022-08-17 11:43:14 +08:00
Ganlin Zhao b58402ba3b fix(query): set proper ltrim/rtrim result schema bytes
TD-18431
2022-08-17 11:33:37 +08:00
Ganlin Zhao cca1cacd35 fix(query): set proper ltrim/rtrim result schema bytes
TD-18431
2022-08-17 11:29:55 +08:00
Ganlin Zhao 4c654f8a17 fix(query): set proper ltrim/rtrim result schema bytes
TD-18431
2022-08-17 11:25:32 +08:00
slzhou 5b54707501 Merge branch '3.0' of github.com:taosdata/TDengine into szhou/fixbug 2022-08-12 14:19:32 +08:00
slzhou 3c27c8b8fd fix: sample/top/bottom is not allowed with fill clause 2022-08-12 10:20:09 +08:00
Ganlin Zhao c017e4387f fix(query): forbid interp in super table query
TD-18331
2022-08-11 16:17:38 +08:00
Xiaoyu Wang 0829f6611c Merge remote-tracking branch 'origin/feature/3.0_query_optimize' into feature/3.0_query_optimize_wxy 2022-08-09 10:20:32 +08:00
Xiaoyu Wang 7e1917f105 enh: refine the data requirements of the partition physical plan 2022-08-09 09:55:03 +08:00
Haojun Liao 4d8842c21f other:merge 3.0 2022-08-08 16:37:37 +08:00
Xiaoyu Wang 5ae9fdff7a enh: increase explain output, enhance debugging function 2022-08-08 10:00:10 +08:00
Ganlin Zhao e808f67518 Merge branch '3.0' into fix/TD-18115 2022-08-05 10:03:17 +08:00
Haojun Liao 3a5fcefc8d other: merge 3.0 2022-08-04 17:24:51 +08:00
Xiaoyu Wang a068db58a1 fix: the problem that drop stable does not clear the catalog cache 2022-08-04 15:09:56 +08:00
Haojun Liao 3566e8cf46 enh(query): diff support timestamp column type. 2022-08-04 15:02:05 +08:00
Ganlin Zhao 6b6f85e22f fix(query): fix interp function core dump
TD-18115
2022-08-03 21:24:47 +08:00
Ganlin Zhao a646bba398
Merge pull request #15716 from taosdata/fix/TD-18010
fix(query): add interp param check
2022-08-03 20:33:59 +08:00
Haojun Liao 2a8da022de refactor: do some internal refactor. 2022-08-03 19:01:06 +08:00
Ganlin Zhao 479367207b fix(query): add interp param check
TD-18113
2022-08-03 18:58:47 +08:00
Xiaoyu Wang 335c8b4d8a fix: timeline function validity check 2022-08-03 18:17:12 +08:00
Haojun Liao 074500805c refactor: set filter function for last_row function. 2022-08-03 15:34:42 +08:00
Haojun Liao be51233c24 refactor: add dynamic prune for last partial. 2022-08-02 14:26:54 +08:00
Ganlin Zhao 3ebd338097 enh(query): add selectivity support for mavg functions
TD-18050
2022-08-01 15:41:05 +08:00
Ganlin Zhao 62407b9e9f enh(query): add selectivity support for csum functions
TD-18050
2022-08-01 15:33:38 +08:00
Ganlin Zhao b4fa0d61b4 enh(query): add selectivity support for statecount/stateduration
functions

TD-18050
2022-08-01 15:16:29 +08:00
Liu Jicong c67f5f10a2
Merge pull request #15595 from taosdata/feature/stream
enh(wal): add lock to guarantee read behaviour
2022-07-30 17:09:20 +08:00
Liu Jicong 6d67b17113 enh(wal): add lock to guarantee read behaviour 2022-07-30 16:35:49 +08:00
Liu Jicong f24f2eff42 merge 3.0 2022-07-30 13:03:12 +08:00
Ganlin Zhao 54f2e4af05 fix(query): fix hyperloglog return inconsistent values
TD-17574
2022-07-30 10:55:33 +08:00
Haojun Liao d29a60499a
Merge pull request #15554 from taosdata/feature/3_liaohj
refactor: do some internal refactor and opt query performance.
2022-07-29 22:34:37 +08:00
Liu Jicong 6c250e75cd fix(stream): forbid histogram in stream 2022-07-29 21:28:18 +08:00
Xiaoyu Wang 31126d5390 Merge remote-tracking branch 'origin/3.0' into feature/3.0_wxy 2022-07-29 20:41:39 +08:00
Haojun Liao 9ee13ca254 other: merge 3.0 2022-07-29 18:03:42 +08:00
Xiaoyu Wang 41b8ef7d93 enh: cumulative functions support mixed use 2022-07-29 18:00:48 +08:00
Ganlin Zhao f280721687
Merge pull request #15541 from taosdata/fix/TD-17009
fix(query): fix apercentile with interval crash issue
2022-07-29 16:55:17 +08:00
Ganlin Zhao 77c24996ba fix(query): fix histogram function with interval result not match
TD-17740
2022-07-29 15:04:40 +08:00
Ganlin Zhao 4768285ca2 fix(query): fix apercentile with interval crash issue
TD-17009
TD-17783
2022-07-29 14:13:09 +08:00
Haojun Liao 78bbfcb4bc refactor: do some internal refactor and opt query performance. 2022-07-29 09:56:03 +08:00
Ganlin Zhao af305d1f51 Merge branch '3.0' into enh/TD-17659 2022-07-26 14:15:11 +08:00
Ganlin Zhao 7c6fbd1a09 Merge branch '3.0' into fix/TD-17795 2022-07-25 20:44:35 +08:00
Ganlin Zhao 4ad4ffd6f5 fix(query): fix apercentile merge function percent parameter
not reserved cause crash_gen report invalid read

TD-17795
2022-07-25 20:40:58 +08:00
Ganlin Zhao e8da4f429c enh(query): add derivative function selectivity
TD-17659
2022-07-25 16:04:28 +08:00
Haojun Liao 90b3e77698 fix(query):disable merge for project operator in stream processing. 2022-07-25 16:02:49 +08:00
Ganlin Zhao db48f118a1 enh(query): add selectivity for diff function
TD-17659
2022-07-25 15:34:46 +08:00
Xiaoyu Wang ba0518e2b9 fix: plan problem when functions that requires a timeline is used directly in a super table 2022-07-22 15:06:01 +08:00
Xiaoyu Wang dceec792bc
Merge pull request #15237 from taosdata/feature/3.0_debug_wxy
fix: plan problem caused by project eliminate optimize
2022-07-22 13:51:07 +08:00
Xiaoyu Wang 958f3586c0 fix: the problem of data loss when interval is used for outer query 2022-07-22 11:41:18 +08:00
Haojun Liao 7522c2edf3 refactor: do some internal refactor. 2022-07-21 20:44:36 +08:00
Ganlin Zhao a2dd6f0ce5 fix(query): fix memory leak in histogram param validation
TD-17598
2022-07-20 14:42:03 +08:00
Ganlin Zhao 6be20e951d fix(query): forbid use null as input for first/last/last_row 2022-07-19 20:21:43 +08:00
Ganlin Zhao 1f75dbb862 Merge branch '3.0' into fix/TD-17511 2022-07-19 19:50:49 +08:00
Xiaoyu Wang 68dd74bcb3
Merge pull request #15117 from taosdata/feature/3.0_debug_wxy
feat: sql command 'show tags from table_name'
2022-07-19 19:49:59 +08:00
Ganlin Zhao d9d6409249 fix(query): forbid use null as input for first/last/last_row
TD-17526
2022-07-19 19:47:46 +08:00
Xiaoyu Wang 6d5d6746f2 feat: sql command 'show tags from table_name' 2022-07-19 18:51:06 +08:00
Ganlin Zhao 78e8d84855 fix(query): change cast function input limit from 1000->4096 bytes
TD-17497
2022-07-19 16:47:24 +08:00
Xiaoyu Wang 1ad25d9e98 feat: sql command 'show tags from table_name' 2022-07-19 16:25:45 +08:00
Xiaoyu Wang 6ab444016b feat: sql command 'show tags from table_name' 2022-07-19 15:35:51 +08:00
Ganlin Zhao 8eb17a28d2 Merge branch '3.0' into fix/TD-17400 2022-07-18 17:36:56 +08:00
Ganlin Zhao f57ad234dc fix(query): fix function first/last do not support constant input
TD-17400
2022-07-18 16:59:03 +08:00
afwerar fa96e64ee0 os: win format negative timestamp error 2022-07-18 16:53:20 +08:00
Ganlin Zhao 5b12fcb3a1 feat(query): add selective function scalar version 2022-07-18 15:23:28 +08:00
Ganlin Zhao 75371098a6 refactor 2022-07-18 15:14:20 +08:00
Ganlin Zhao a4d86275d2 feat(query): add first/last/last_row scalar version 2022-07-18 15:06:21 +08:00
Ganlin Zhao a80027736d feat(query): add top/bottom function scalar version 2022-07-18 14:47:53 +08:00
Ganlin Zhao e37bdbd5c9 feat(query): add histogram function scalar version
TD-17344
2022-07-18 13:37:18 +08:00
Ganlin Zhao 107cb15dbd feat(query): add stateduration function scalar version 2022-07-17 22:52:40 +08:00
Ganlin Zhao 810be53374 fix(query): fix state operator check 2022-07-16 17:01:44 +08:00
Ganlin Zhao 3bcd669fcd feat(query): add statecount function scalar version 2022-07-16 16:46:03 +08:00
Ganlin Zhao 790cecf123 feat(query): add diff function scalar version 2022-07-16 15:04:45 +08:00
Ganlin Zhao 2d1428eadd feat(query): add csum function scalar version
TD-17344
2022-07-16 14:13:45 +08:00
Ganlin Zhao a57d2417c3 feat(query): add hyperloglog scalar version
TD-17344
2022-07-16 14:01:49 +08:00
Ganlin Zhao 59839f0d0a feat(query): add mavg scalar version 2022-07-16 13:41:42 +08:00
Ganlin Zhao 8caa4bcad0 feat(query): add twa function scalar version 2022-07-16 13:34:24 +08:00
Ganlin Zhao 566ab93f83 add derivative scalar version 2022-07-16 11:37:05 +08:00
Ganlin Zhao 598c57e059 feat(query): add irate function scalar version
TD-17344
2022-07-16 11:26:04 +08:00
Ganlin Zhao 624e7b7183 Merge branch '3.0' into feat/agg_client_api 2022-07-16 10:53:57 +08:00
Xiaoyu Wang 2db0b4075f
Merge pull request #14952 from taosdata/feature/3.0_debug_wxy
fix: multi-rows function validity check
2022-07-16 10:11:30 +08:00
Ganlin Zhao cbd2d8f8a2 fix(query): add spread function scalar version
TD-17344
2022-07-15 17:54:45 +08:00
Haojun Liao 50aa7f9cfa Merge branch '3.0' into feature/3_liaohj 2022-07-15 17:37:47 +08:00