xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Merge branch 'develop' into feature/mergeimport

This commit is contained in:
Hongze Cheng 2019-11-26 10:06:40 +08:00
parent a3464134c8 bb5a0f8241
commit 0ba6a7cdbf
93 changed files with 5058 additions and 3666 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
documentation/tdenginedocs-en/super-table/index.html
View File

@ -6,9 +6,9 @@
<p>Like a table, you can create, show, delete and describe STables. Most query operations on tables can be applied to STable too, including the aggregation and selector functions. For queries on a STable, if no tags filter, the operations are applied to all the tables created via this STable. If there is a tag filter, the operations are applied only to a subset of the tables which satisfy the tag filter conditions. It will be very convenient to use tags to put devices into different groups for aggregation.</p>
<a class='anchor' id='Create-a-STable'></a><h2>Create a STable</h2>
<p>Similiar to creating a standard table, syntax is: </p>
<pre><code class="mysql language-mysql">CREATE TABLE &lt;stable_name&gt; (&lt;field_name&gt; TIMESTAMP, field_name1 field_type,…) TAGS(tag_name tag_type, …)</code></pre>
<pre><code class="mysql language-mysql">CREATE TABLE &lt;stable_name&gt; (&lt;field_name&gt; TIMESTAMP, field_name1 field_type, ...) TAGS(tag_name tag_type, ...)</code></pre>
<p>New keyword "tags" is introduced, where tag_name is the tag name, and tag_type is the associated data type. </p>
<p>Note：</p>
<p>Note: </p>
<ol>
<li>The bytes of all tags together shall be less than 512 </li>
<li>Tag's data type can not be time stamp or nchar</li>
@ -30,12 +30,12 @@ tags (location binary(20), type int)</code></pre>
create table t4 using thermometer tags ('shanghai', 20);
create table t5 using thermometer tags ('new york', 10);</code></pre>
<a class='anchor' id='Aggregate-Tables-via-STable'></a><h2>Aggregate Tables via STable</h2>
<p>You can group a set of tables together by specifying the tags filter condition, then apply the aggregation operations. The result set can be grouped and ordered based on tag value. Syntax is：</p>
<pre><code class="mysql language-mysql">SELECT function&lt;field_name&gt;,…
<p>You can group a set of tables together by specifying the tags filter condition, then apply the aggregation operations. The result set can be grouped and ordered based on tag value. Syntax is:</p>
<pre><code class="mysql language-mysql">SELECT function&lt;field_name&gt;, ...
FROM &lt;stable_name&gt;
WHERE &lt;tag_name&gt; &lt;[=|&lt;=|&gt;=|&lt;&gt;] values..&gt; ([AND|OR] …)
WHERE &lt;tag_name&gt; &lt;[=|&lt;=|&gt;=|&lt;&gt;] values..&gt; ([AND|OR] ...
INTERVAL (&lt;time range&gt;)
GROUP BY &lt;tag_name&gt;, &lt;tag_name&gt;…
GROUP BY &lt;tag_name&gt;, &lt;tag_name&gt; ...
ORDER BY &lt;tag_name&gt; &lt;asc|desc&gt;
SLIMIT &lt;group_limit&gt;
SOFFSET &lt;group_offset&gt;
@ -75,9 +75,9 @@ INTERVAL(10M)</code></pre>
<pre><code class="mysql language-mysql">DROP TABLE &lt;stable_name&gt;</code></pre>
<p>To delete a STable, all the tables created via this STable shall be deleted first, otherwise, it will fail.</p>
<a class='anchor' id='List-the-Associated-Tables-of-a-STable'></a><h3>List the Associated Tables of a STable</h3>
<pre><code class="mysql language-mysql">SELECT TBNAME,[TAG_NAME,…] FROM &lt;stable_name&gt; WHERE &lt;tag_name&gt; &lt;[=|=&lt;|&gt;=|&lt;&gt;] values..&gt; ([AND|OR] …)</code></pre>
<pre><code class="mysql language-mysql">SELECT TBNAME,[TAG_NAME, ...] FROM &lt;stable_name&gt; WHERE &lt;tag_name&gt; &lt;[=|=&lt;|&gt;=|&lt;&gt;] values..&gt; ([AND|OR] ...)</code></pre>
<p>It will list all the tables which satisfy the tag filter conditions. The tables are all created from this specific STable. TBNAME is a new keyword introduced, it is the table name associated with the STable. </p>
<pre><code class="mysql language-mysql">SELECT COUNT(TBNAME) FROM &lt;stable_name&gt; WHERE &lt;tag_name&gt; &lt;[=|=&lt;|&gt;=|&lt;&gt;] values..&gt; ([AND|OR] …)</code></pre>
<pre><code class="mysql language-mysql">SELECT COUNT(TBNAME) FROM &lt;stable_name&gt; WHERE &lt;tag_name&gt; &lt;[=|=&lt;|&gt;=|&lt;&gt;] values..&gt; ([AND|OR] ...)</code></pre>
<p>The above SQL statement will list the number of tables in a STable, which satisfy the filter condition.</p>
<a class='anchor' id='Management-of-Tags'></a><h2>Management of Tags</h2>
<p>You can add, delete and change the tags for a STable, and you can change the tag value of a table. The SQL commands are listed below. </p>

8
packaging/deb/DEBIAN/preinst
View File

@ -9,13 +9,13 @@ fi
if pidof taosd &> /dev/null; then
if pidof systemd &> /dev/null; then
${csudo} systemctl stop taosd || :
elif $(which insserv &> /dev/null); then
${csudo} service taosd stop || :
elif $(which update-rc.d &> /dev/null); then
elif $(which service &> /dev/null); then
${csudo} service taosd stop || :
else
pid=$(ps -ef | grep "taosd" | grep -v "grep" | awk '{print $2}')
${csudo} kill -9 ${pid} || :
if [ -n "$pid" ]; then
${csudo} kill -9 $pid || :
fi
fi
echo "Stop taosd service success!"
sleep 1

4
packaging/deb/DEBIAN/prerm
View File

@ -35,6 +35,8 @@ else
${csudo} rm -f ${data_link_dir} || :
pid=$(ps -ef | grep "taosd" | grep -v "grep" | awk '{print $2}')
${csudo} kill -9 ${pid} || :
if [ -n "$pid" ]; then
${csudo} kill -9 $pid || :
fi
fi

3
packaging/deb/makedeb.sh
View File

@ -24,8 +24,7 @@ fi
mkdir -p ${pkg_dir}
cd ${pkg_dir}
versioninfo=$(${script_dir}/../tools/get_version.sh ${script_dir}/../../src/util/src/version.c)
libfile="libtaos.so.${versioninfo}"
libfile="libtaos.so.${tdengine_ver}"
# create install dir
install_home_path="/usr/local/taos"

32
packaging/release.sh
View File

@ -3,7 +3,9 @@
# Generate the deb package for ubunt, or rpm package for centos, or tar.gz package for other linux os
set -e
#set -x
# set -x
armver=$1
curr_dir=$(pwd)
script_dir="$(dirname $(readlink -f $0))"
@ -110,21 +112,28 @@ echo "char gitinfo[128] = \"$(git rev-parse --verify HEAD)\";" >> ${versioninfo
echo "char buildinfo[512] = \"Built by ${USER} at ${build_time}\";" >> ${versioninfo}
# 2. cmake executable file
#default use debug mode
compile_mode="debug"
if [[ $1 == "Release" ]] || [[ $1 == "release" ]]; then
compile_mode="Release"
fi
compile_dir="${top_dir}/${compile_mode}"
compile_dir="${top_dir}/debug"
if [ -d ${compile_dir} ]; then
${csudo} rm -rf ${compile_dir}
${csudo} rm -rf ${compile_dir}
fi
${csudo} mkdir -p ${compile_dir}
cd ${compile_dir}
${csudo} cmake -DCMAKE_BUILD_TYPE=${compile_mode} ${top_dir}
${csudo} make
# arm only support lite ver
if [ -z "$armver" ]; then
cmake ${top_dir}/../
elif [ "$armver" == "arm64" ]; then
cmake ${top_dir}/../ -DVERSION=lite -DARMVER=arm64
elif [ "$armver" == "arm32" ]; then
cmake ${top_dir}/../ -DVERSION=lite -DARMVER=arm32
else
echo "input parameter error!!!"
return
fi
make
cd ${curr_dir}
@ -153,7 +162,8 @@ ${csudo} ./makerpm.sh ${compile_dir} ${output_dir} ${version}
echo "do tar.gz package for all systems"
cd ${script_dir}/tools
${csudo} ./makepkg.sh ${compile_dir} ${version} "${build_time}"
${csudo} ./makepkg.sh ${compile_dir} ${version} "${build_time}" ${armver}
${csudo} ./makeclient.sh ${compile_dir} ${version} "${build_time}" ${armver}
# 4. Clean up temporary compile directories
#${csudo} rm -rf ${compile_dir}

16
packaging/rpm/tdengine.spec
View File

@ -39,8 +39,7 @@ echo topdir: %{_topdir}
echo version: %{_version}
echo buildroot: %{buildroot}
versioninfo=$(%{_compiledir}/../packaging/tools/get_version.sh ../../src/util/src/version.c)
libfile="libtaos.so.${versioninfo}"
libfile="libtaos.so.%{_version}"
# create install path, and cp file
mkdir -p %{buildroot}%{homepath}/bin
@ -80,18 +79,17 @@ fi
if pidof taosd &> /dev/null; then
if pidof systemd &> /dev/null; then
${csudo} systemctl stop taosd || :
elif $(which insserv &> /dev/null); then
${csudo} service taosd stop || :
elif $(which update-rc.d &> /dev/null); then
elif $(which service &> /dev/null); then
${csudo} service taosd stop || :
else
pid=$(ps -ef | grep "taosd" | grep -v "grep" | awk '{print $2}')
${csudo} kill -9 ${pid} || :
if [ -n "$pid" ]; then
${csudo} kill -9 $pid || :
fi
fi
echo "Stop taosd service success!"
sleep 1
fi
# if taos.cfg already softlink, remove it
if [ -f %{cfg_install_dir}/taos.cfg ]; then
${csudo} rm -f %{homepath}/cfg/taos.cfg || :
@ -146,7 +144,9 @@ if [ $1 -eq 0 ];then
${csudo} rm -f ${data_link_dir} || :
pid=$(ps -ef | grep "taosd" | grep -v "grep" | awk '{print $2}')
${csudo} kill -9 ${pid} || :
if [ -n "$pid" ]; then
${csudo} kill -9 $pid || :
fi
fi
fi

111
packaging/tools/install.sh
View File

@ -1,6 +1,6 @@
#!/bin/bash
#
# This file is used to install TAOS time-series database on linux systems. The operating system
# This file is used to install database on linux systems. The operating system
# is required to use systemd to manage services at boot
set -e
@ -41,19 +41,58 @@ if command -v sudo > /dev/null; then
csudo="sudo"
fi
initd_mod=0
service_mod=2
if pidof systemd &> /dev/null; then
service_mod=0
elif $(which update-rc.d &> /dev/null); then
elif $(which service &> /dev/null); then
service_mod=1
service_config_dir="/etc/init.d"
if $(which chkconfig &> /dev/null); then
initd_mod=1
elif $(which insserv &> /dev/null); then
initd_mod=2
elif $(which update-rc.d &> /dev/null); then
initd_mod=3
else
service_mod=2
fi
else
service_mod=2
fi
# get the operating system type for using the corresponding init file
# ubuntu/debian(deb), centos/fedora(rpm), others: opensuse, redhat, ..., no verification
#osinfo=$(awk -F= '/^NAME/{print $2}' /etc/os-release)
osinfo=$(cat /etc/os-release | grep "NAME" | cut -d '"' -f2)
#echo "osinfo: ${osinfo}"
os_type=0
if echo $osinfo | grep -qwi "ubuntu" ; then
echo "this is ubuntu system"
os_type=1
elif echo $osinfo | grep -qwi "debian" ; then
echo "this is debian system"
os_type=1
elif echo $osinfo | grep -qwi "Kylin" ; then
echo "this is Kylin system"
os_type=1
elif echo $osinfo | grep -qwi "centos" ; then
echo "this is centos system"
os_type=2
elif echo $osinfo | grep -qwi "fedora" ; then
echo "this is fedora system"
os_type=2
else
echo "this is other linux system"
os_type=0
fi
function kill_taosd() {
pid=$(ps -ef | grep "taosd" | grep -v "grep" | awk '{print $2}')
${csudo} kill -9 pid || :
if [ -n "$pid" ]; then
${csudo} kill -9 $pid || :
fi
}
function install_main_path() {
@ -89,7 +128,7 @@ function install_bin() {
function install_lib() {
# Remove links
${csudo} rm -f ${lib_link_dir}/libtaos.* || :
${csudo} rm -f ${lib_link_dir}/libtaos.* || :
${csudo} cp -rf ${script_dir}/driver/* ${install_main_dir}/driver && ${csudo} chmod 777 ${install_main_dir}/driver/*
@ -117,6 +156,7 @@ function install_config() {
${csudo} ln -s ${cfg_install_dir}/taos.cfg ${install_main_dir}/cfg
}
function install_log() {
${csudo} rm -rf ${log_dir} || :
${csudo} mkdir -p ${log_dir} && ${csudo} chmod 777 ${log_dir}
@ -139,14 +179,26 @@ function install_examples() {
}
function clean_service_on_sysvinit() {
restart_config_str="taos:2345:respawn:${service_config_dir}/taosd start"
#restart_config_str="taos:2345:respawn:${service_config_dir}/taosd start"
#${csudo} sed -i "\|${restart_config_str}|d" /etc/inittab || :
if pidof taosd &> /dev/null; then
${csudo} service taosd stop || :
fi
${csudo} sed -i "\|${restart_config_str}|d" /etc/inittab || :
if ((${initd_mod}==1)); then
${csudo} chkconfig --del taosd || :
elif ((${initd_mod}==2)); then
${csudo} insserv -r taosd || :
elif ((${initd_mod}==3)); then
${csudo} update-rc.d -f taosd remove || :
fi
${csudo} rm -f ${service_config_dir}/taosd || :
${csudo} update-rc.d -f taosd remove || :
${csudo} init q || :
if $(which init &> /dev/null); then
${csudo} init q || :
fi
}
function install_service_on_sysvinit() {
@ -155,14 +207,27 @@ function install_service_on_sysvinit() {
sleep 1
# Install taosd service
${csudo} cp -f ${script_dir}/init.d/taosd ${install_main_dir}/init.d
${csudo} cp ${script_dir}/init.d/taosd ${service_config_dir} && ${csudo} chmod a+x ${service_config_dir}/taosd
restart_config_str="taos:2345:respawn:${service_config_dir}/taosd start"
${csudo} grep -q -F "$restart_config_str" /etc/inittab || ${csudo} bash -c "echo '${restart_config_str}' >> /etc/inittab"
# TODO: for centos, change here
${csudo} update-rc.d taosd defaults
# chkconfig mysqld on
if ((${os_type}==1)); then
${csudo} cp -f ${script_dir}/init.d/taosd.deb ${install_main_dir}/init.d/taosd
${csudo} cp ${script_dir}/init.d/taosd.deb ${service_config_dir}/taosd && ${csudo} chmod a+x ${service_config_dir}/taosd
elif ((${os_type}==2)); then
${csudo} cp -f ${script_dir}/init.d/taosd.rpm ${install_main_dir}/init.d/taosd
${csudo} cp ${script_dir}/init.d/taosd.rpm ${service_config_dir}/taosd && ${csudo} chmod a+x ${service_config_dir}/taosd
fi
#restart_config_str="taos:2345:respawn:${service_config_dir}/taosd start"
#${csudo} grep -q -F "$restart_config_str" /etc/inittab || ${csudo} bash -c "echo '${restart_config_str}' >> /etc/inittab"
if ((${initd_mod}==1)); then
${csudo} chkconfig --add taosd || :
${csudo} chkconfig --level 2345 taosd on || :
elif ((${initd_mod}==2)); then
${csudo} insserv taosd || :
${csudo} insserv -d taosd || :
elif ((${initd_mod}==3)); then
${csudo} update-rc.d taosd defaults || :
fi
}
function clean_service_on_systemd() {
@ -212,7 +277,7 @@ function install_service() {
elif ((${service_mod}==1)); then
install_service_on_sysvinit
else
# must manual start taosd
# must manual stop taosd
kill_taosd
fi
}
@ -274,7 +339,7 @@ function update_TDengine() {
return 1
fi
echo -e "${GREEN}Start to update TDEngine...${NC}"
echo -e "${GREEN}Start to update TDengine...${NC}"
# Stop the service if running
if pidof taosd &> /dev/null; then
if ((${service_mod}==0)); then
@ -306,8 +371,7 @@ function update_TDengine() {
if ((${service_mod}==0)); then
echo -e "${GREEN_DARK}To start TDengine ${NC}: ${csudo} systemctl start taosd${NC}"
elif ((${service_mod}==1)); then
echo -e "${GREEN_DARK}To start TDengine ${NC}: ${csudo} update-rc.d taosd default ${RED} for the first time${NC}"
echo -e " : ${csudo} service taosd start ${RED} after${NC}"
echo -e "${GREEN_DARK}To start TDengine ${NC}: ${csudo} service taosd start${NC}"
else
echo -e "${GREEN_DARK}To start TDengine ${NC}: ./taosd${NC}"
fi
@ -316,7 +380,7 @@ function update_TDengine() {
echo
echo -e "\033[44;32;1mTDengine is updated successfully!${NC}"
else
install_bin $1
install_bin
install_config
echo
@ -334,7 +398,7 @@ function install_TDengine() {
fi
tar -zxf taos.tar.gz
echo -e "${GREEN}Start to install TDEngine...${NC}"
echo -e "${GREEN}Start to install TDengine...${NC}"
install_main_path
@ -362,10 +426,9 @@ function install_TDengine() {
if ((${service_mod}==0)); then
echo -e "${GREEN_DARK}To start TDengine ${NC}: ${csudo} systemctl start taosd${NC}"
elif ((${service_mod}==1)); then
echo -e "${GREEN_DARK}To start TDengine ${NC}: ${csudo} update-rc.d taosd default ${RED} for the first time${NC}"
echo -e " : ${csudo} service taosd start ${RED} after${NC}"
echo -e "${GREEN_DARK}To start TDengine ${NC}: ${csudo} service taosd start${NC}"
else
echo -e "${GREEN_DARK}To start TDengine ${NC}: ./taosd${NC}"
echo -e "${GREEN_DARK}To start TDengine ${NC}: taosd${NC}"
fi
echo -e "${GREEN_DARK}To access TDengine ${NC}: use ${GREEN_UNDERLINE}taos${NC} in shell${NC}"

199
packaging/tools/install_client.sh
View File

@ -1,4 +1,201 @@
#!/bin/bash
#
# This file is used to install TDengine client on linux systems. The operating system
# is required to use systemd to manage services at boot
set -e
#set -x
# -----------------------Variables definition---------------------
script_dir=$(dirname $(readlink -m "$0"))
${script_dir}/install.sh client
# Dynamic directory
data_dir="/var/lib/taos"
log_dir="/var/log/taos"
log_link_dir="/usr/local/taos/log"
cfg_install_dir="/etc/taos"
bin_link_dir="/usr/bin"
lib_link_dir="/usr/lib"
inc_link_dir="/usr/include"
#install main path
install_main_dir="/usr/local/taos"
# old bin dir
bin_dir="/usr/local/taos/bin"
# Color setting
RED='\033[0;31m'
GREEN='\033[1;32m'
GREEN_DARK='\033[0;32m'
GREEN_UNDERLINE='\033[4;32m'
NC='\033[0m'
csudo=""
if command -v sudo > /dev/null; then
csudo="sudo"
fi
update_flag=0
function kill_client() {
pid=$(ps -ef | grep "taos" | grep -v "grep" | awk '{print $2}')
if [ -n "$pid" ]; then
${csudo} kill -9 $pid || :
fi
}
function install_main_path() {
#create install main dir and all sub dir
${csudo} rm -rf ${install_main_dir} || :
${csudo} mkdir -p ${install_main_dir}
${csudo} mkdir -p ${install_main_dir}/cfg
${csudo} mkdir -p ${install_main_dir}/bin
${csudo} mkdir -p ${install_main_dir}/connector
${csudo} mkdir -p ${install_main_dir}/driver
${csudo} mkdir -p ${install_main_dir}/examples
${csudo} mkdir -p ${install_main_dir}/include
}
function install_bin() {
# Remove links
${csudo} rm -f ${bin_link_dir}/taos || :
${csudo} rm -f ${bin_link_dir}/taosdump || :
${csudo} rm -f ${bin_link_dir}/rmtaos || :
${csudo} cp -r ${script_dir}/bin/* ${install_main_dir}/bin && ${csudo} chmod 0555 ${install_main_dir}/bin/*
#Make link
[ -x ${install_main_dir}/bin/taos ] && ${csudo} ln -s ${install_main_dir}/bin/taos ${bin_link_dir}/taos || :
[ -x ${install_main_dir}/bin/taosdump ] && ${csudo} ln -s ${install_main_dir}/bin/taosdump ${bin_link_dir}/taosdump || :
[ -x ${install_main_dir}/bin/remove_client.sh ] && ${csudo} ln -s ${install_main_dir}/bin/remove_client.sh ${bin_link_dir}/rmtaos || :
}
function clean_lib() {
sudo rm -f /usr/lib/libtaos.so || :
sudo rm -rf ${lib_dir} || :
}
function install_lib() {
# Remove links
${csudo} rm -f ${lib_link_dir}/libtaos.* || :
${csudo} cp -rf ${script_dir}/driver/* ${install_main_dir}/driver && ${csudo} chmod 777 ${install_main_dir}/driver/*
${csudo} ln -s ${install_main_dir}/driver/libtaos.* ${lib_link_dir}/libtaos.so.1
${csudo} ln -s ${lib_link_dir}/libtaos.so.1 ${lib_link_dir}/libtaos.so
}
function install_header() {
${csudo} rm -f ${inc_link_dir}/taos.h ${inc_link_dir}/taoserror.h || :
${csudo} cp -f ${script_dir}/inc/* ${install_main_dir}/include && ${csudo} chmod 644 ${install_main_dir}/include/*
${csudo} ln -s ${install_main_dir}/include/taos.h ${inc_link_dir}/taos.h
${csudo} ln -s ${install_main_dir}/include/taoserror.h ${inc_link_dir}/taoserror.h
}
function install_config() {
#${csudo} rm -f ${install_main_dir}/cfg/taos.cfg || :
if [ ! -f ${cfg_install_dir}/taos.cfg ]; then
${csudo} mkdir -p ${cfg_install_dir}
[ -f ${script_dir}/cfg/taos.cfg ] && ${csudo} cp ${script_dir}/cfg/taos.cfg ${cfg_install_dir}
${csudo} chmod 644 ${cfg_install_dir}/*
fi
${csudo} cp -f ${script_dir}/cfg/taos.cfg ${install_main_dir}/cfg/taos.cfg.org
${csudo} ln -s ${cfg_install_dir}/taos.cfg ${install_main_dir}/cfg
}
function install_log() {
${csudo} rm -rf ${log_dir} || :
${csudo} mkdir -p ${log_dir} && ${csudo} chmod 777 ${log_dir}
${csudo} ln -s ${log_dir} ${install_main_dir}/log
}
function install_connector() {
${csudo} cp -rf ${script_dir}/connector/* ${install_main_dir}/connector
}
function install_examples() {
if [ -d ${script_dir}/examples ]; then
${csudo} cp -rf ${script_dir}/examples/* ${install_main_dir}/examples
fi
}
function update_TDengine() {
# Start to update
if [ ! -e taos.tar.gz ]; then
echo "File taos.tar.gz does not exist"
exit 1
fi
tar -zxf taos.tar.gz
echo -e "${GREEN}Start to update TDengine client...${NC}"
# Stop the client shell if running
if pidof taos &> /dev/null; then
kill_client
sleep 1
fi
install_main_path
install_log
install_header
install_lib
install_connector
install_examples
install_bin
install_config
echo
echo -e "\033[44;32;1mTDengine client is updated successfully!${NC}"
rm -rf $(tar -tf taos.tar.gz)
}
function install_TDengine() {
# Start to install
if [ ! -e taos.tar.gz ]; then
echo "File taos.tar.gz does not exist"
exit 1
fi
tar -zxf taos.tar.gz
echo -e "${GREEN}Start to install TDengine client...${NC}"
install_main_path
install_log
install_header
install_lib
install_connector
install_examples
install_bin
install_config
echo
echo -e "\033[44;32;1mTDengine client is installed successfully!${NC}"
rm -rf $(tar -tf taos.tar.gz)
}
## ==============================Main program starts from here============================
# Install or updata client and client
# if server is already install, don't install client
if [ -e ${bin_dir}/taosd ]; then
echo -e "\033[44;32;1mThere are already installed TDengine server, so don't need install client!${NC}"
exit 0
fi
if [ -x ${bin_dir}/taos ]; then
update_flag=1
update_TDengine
else
install_TDengine
fi

79
packaging/tools/makeclient.sh Executable file
View File

@ -0,0 +1,79 @@
#!/bin/bash
#
# Generate tar.gz package for linux client
set -e
set -x
curr_dir=$(pwd)
compile_dir=$1
version=$2
build_time=$3
armver=$4
script_dir="$(dirname $(readlink -f $0))"
top_dir="$(readlink -m ${script_dir}/../..)"
# create compressed install file.
build_dir="${compile_dir}/build"
code_dir="${top_dir}/src"
release_dir="${top_dir}/release"
community_dir="${script_dir}/../../../community/src"
package_name='linux'
install_dir="${release_dir}/TDengine-client-enterprise-${version}-${package_name}-$(echo ${build_time}| tr ': ' -)"
# Directories and files.
bin_files="${build_dir}/bin/taos ${build_dir}/bin/taosdump ${script_dir}/remove_client.sh"
lib_files="${build_dir}/lib/libtaos.so.${version}"
header_files="${community_dir}/inc/taos.h ${community_dir}/inc/taoserror.h"
cfg_dir="${top_dir}/packaging/cfg"
install_files="${script_dir}/install_client.sh"
# make directories.
mkdir -p ${install_dir}
mkdir -p ${install_dir}/inc && cp ${header_files} ${install_dir}/inc
mkdir -p ${install_dir}/cfg && cp ${cfg_dir}/taos.cfg ${install_dir}/cfg/taos.cfg
mkdir -p ${install_dir}/bin && cp ${bin_files} ${install_dir}/bin && chmod a+x ${install_dir}/bin/*
cd ${install_dir}
tar -zcv -f taos.tar.gz * --remove-files || :
cd ${curr_dir}
cp ${install_files} ${install_dir} && chmod a+x ${install_dir}/install*
# Copy example code
mkdir -p ${install_dir}/examples
cp -r ${top_dir}/tests/examples/c ${install_dir}/examples
cp -r ${top_dir}/tests/examples/JDBC ${install_dir}/examples
cp -r ${top_dir}/tests/examples/matlab ${install_dir}/examples
cp -r ${top_dir}/tests/examples/python ${install_dir}/examples
cp -r ${top_dir}/tests/examples/R ${install_dir}/examples
cp -r ${top_dir}/tests/examples/go ${install_dir}/examples
# Copy driver
mkdir -p ${install_dir}/driver
cp ${lib_files} ${install_dir}/driver
# Copy connector
connector_dir="${community_dir}/connector"
mkdir -p ${install_dir}/connector
cp ${build_dir}/lib/*.jar ${install_dir}/connector
cp -r ${connector_dir}/grafana ${install_dir}/connector/
cp -r ${connector_dir}/python ${install_dir}/connector/
cp -r ${connector_dir}/go ${install_dir}/connector
# Copy release note
# cp ${script_dir}/release_note ${install_dir}
# exit 1
cd ${release_dir}
if [ -z "$armver" ]; then
tar -zcv -f "$(basename ${install_dir}).tar.gz" $(basename ${install_dir}) --remove-files
elif [ "$armver" == "arm64" ]; then
tar -zcv -f "$(basename ${install_dir})-arm64.tar.gz" $(basename ${install_dir}) --remove-files
elif [ "$armver" == "arm32" ]; then
tar -zcv -f "$(basename ${install_dir})-arm32.tar.gz" $(basename ${install_dir}) --remove-files
fi
cd ${curr_dir}

42
packaging/tools/makepkg.sh
View File

@ -6,6 +6,7 @@ curr_dir=$(pwd)
compile_dir=$1
version=$2
build_time=$3
armver=$4
script_dir="$(dirname $(readlink -f $0))"
top_dir="$(readlink -m ${script_dir}/../..)"
@ -20,11 +21,10 @@ install_dir="${release_dir}/taos-${version}-${package_name}-$(echo ${build_time}
# Directories and files.
bin_files="${build_dir}/bin/taosd ${build_dir}/bin/taos ${build_dir}/bin/taosdemo ${build_dir}/bin/taosdump ${script_dir}/remove.sh"
versioninfo=$(${script_dir}/get_version.sh ${code_dir}/util/src/version.c)
lib_files="${build_dir}/lib/libtaos.so.${versioninfo}"
lib_files="${build_dir}/lib/libtaos.so.${version}"
header_files="${code_dir}/inc/taos.h ${code_dir}/inc/taoserror.h"
cfg_files="${top_dir}/packaging/cfg/*.cfg"
install_files="${script_dir}/install.sh ${script_dir}/install_client.sh"
cfg_dir="${top_dir}/packaging/cfg"
install_files="${script_dir}/install.sh"
# Init file
#init_dir=${script_dir}/deb
@ -33,14 +33,16 @@ install_files="${script_dir}/install.sh ${script_dir}/install_client.sh"
#fi
#init_files=${init_dir}/taosd
# temp use rpm's taosd. TODO: later modify according to os type
init_files=${script_dir}/../rpm/taosd
init_file_deb=${script_dir}/../deb/taosd
init_file_rpm=${script_dir}/../rpm/taosd
# make directories.
mkdir -p ${install_dir}
mkdir -p ${install_dir}/inc && cp ${header_files} ${install_dir}/inc
mkdir -p ${install_dir}/cfg && cp ${cfg_files} ${install_dir}/cfg
mkdir -p ${install_dir}/cfg && cp ${cfg_dir}/taos.cfg ${install_dir}/cfg/taos.cfg
mkdir -p ${install_dir}/bin && cp ${bin_files} ${install_dir}/bin && chmod a+x ${install_dir}/bin/*
mkdir -p ${install_dir}/init.d && cp ${init_files} ${install_dir}/init.d
mkdir -p ${install_dir}/init.d && cp ${init_file_deb} ${install_dir}/init.d/taosd.deb
mkdir -p ${install_dir}/init.d && cp ${init_file_rpm} ${install_dir}/init.d/taosd.rpm
cd ${install_dir}
tar -zcv -f taos.tar.gz * --remove-files || :
@ -50,12 +52,13 @@ cp ${install_files} ${install_dir} && chmod a+x ${install_dir}/install*
# Copy example code
mkdir -p ${install_dir}/examples
cp -r ${top_dir}/tests/examples/c ${install_dir}/examples
cp -r ${top_dir}/tests/examples/JDBC ${install_dir}/examples
cp -r ${top_dir}/tests/examples/matlab ${install_dir}/examples
cp -r ${top_dir}/tests/examples/python ${install_dir}/examples
cp -r ${top_dir}/tests/examples/R ${install_dir}/examples
cp -r ${top_dir}/tests/examples/go ${install_dir}/examples
examples_dir="${top_dir}/tests/examples"
cp -r ${examples_dir}/c ${install_dir}/examples
cp -r ${examples_dir}/JDBC ${install_dir}/examples
cp -r ${examples_dir}/matlab ${install_dir}/examples
cp -r ${examples_dir}/python ${install_dir}/examples
cp -r ${examples_dir}/R ${install_dir}/examples
cp -r ${examples_dir}/go ${install_dir}/examples
# Copy driver
mkdir -p ${install_dir}/driver
@ -64,18 +67,23 @@ cp ${lib_files} ${install_dir}/driver
# Copy connector
connector_dir="${code_dir}/connector"
mkdir -p ${install_dir}/connector
cp ${build_dir}/lib/*.jar ${install_dir}/connector
cp -r ${connector_dir}/grafana ${install_dir}/connector/
cp -r ${connector_dir}/python ${install_dir}/connector/
cp -r ${connector_dir}/go ${install_dir}/connector
cp ${build_dir}/lib/*.jar ${install_dir}/connector
# Copy release note
cp ${script_dir}/release_note ${install_dir}
# cp ${script_dir}/release_note ${install_dir}
# exit 1
cd ${release_dir}
tar -zcv -f "$(basename ${install_dir}).tar.gz" $(basename ${install_dir}) --remove-files
if [ -z "$armver" ]; then
tar -zcv -f "$(basename ${install_dir}).tar.gz" $(basename ${install_dir}) --remove-files
elif [ "$armver" == "arm64" ]; then
tar -zcv -f "$(basename ${install_dir})-arm64.tar.gz" $(basename ${install_dir}) --remove-files
elif [ "$armver" == "arm32" ]; then
tar -zcv -f "$(basename ${install_dir})-arm32.tar.gz" $(basename ${install_dir}) --remove-files
fi
cd ${curr_dir}

58
packaging/tools/post.sh
View File

@ -42,14 +42,18 @@ initd_mod=0
service_mod=2
if pidof systemd &> /dev/null; then
service_mod=0
elif $(which insserv &> /dev/null); then
elif $(which service &> /dev/null); then
service_mod=1
initd_mod=1
service_config_dir="/etc/init.d"
elif $(which update-rc.d &> /dev/null); then
service_mod=1
initd_mod=2
service_config_dir="/etc/init.d"
if $(which chkconfig &> /dev/null); then
initd_mod=1
elif $(which insserv &> /dev/null); then
initd_mod=2
elif $(which update-rc.d &> /dev/null); then
initd_mod=3
else
service_mod=2
fi
else
service_mod=2
fi
@ -57,7 +61,9 @@ fi
function kill_taosd() {
# ${csudo} pkill -f taosd || :
pid=$(ps -ef | grep "taosd" | grep -v "grep" | awk '{print $2}')
${csudo} kill -9 ${pid} || :
if [ -n "$pid" ]; then
${csudo} kill -9 $pid || :
fi
}
function install_include() {
@ -103,20 +109,26 @@ function install_config() {
}
function clean_service_on_sysvinit() {
restart_config_str="taos:2345:respawn:${service_config_dir}/taosd start"
#restart_config_str="taos:2345:respawn:${service_config_dir}/taosd start"
#${csudo} sed -i "\|${restart_config_str}|d" /etc/inittab || :
if pidof taosd &> /dev/null; then
${csudo} service taosd stop || :
fi
${csudo} sed -i "\|${restart_config_str}|d" /etc/inittab || :
${csudo} rm -f ${service_config_dir}/taosd || :
if ((${initd_mod}==1)); then
${csudo} grep -q -F "taos" /etc/inittab && ${csudo} insserv -r taosd || :
${csudo} chkconfig --del taosd || :
elif ((${initd_mod}==2)); then
${csudo} grep -q -F "taos" /etc/inittab && ${csudo} update-rc.d -f taosd remove || :
${csudo} insserv -r taosd || :
elif ((${initd_mod}==3)); then
${csudo} update-rc.d -f taosd remove || :
fi
${csudo} rm -f ${service_config_dir}/taosd || :
if $(which init &> /dev/null); then
${csudo} init q || :
fi
# ${csudo} update-rc.d -f taosd remove || :
${csudo} init q || :
}
function install_service_on_sysvinit() {
@ -127,18 +139,24 @@ function install_service_on_sysvinit() {
# Install taosd service
${csudo} cp %{init_d_dir}/taosd ${service_config_dir} && ${csudo} chmod a+x ${service_config_dir}/taosd
restart_config_str="taos:2345:respawn:${service_config_dir}/taosd start"
#restart_config_str="taos:2345:respawn:${service_config_dir}/taosd start"
#${csudo} grep -q -F "$restart_config_str" /etc/inittab || ${csudo} bash -c "echo '${restart_config_str}' >> /etc/inittab"
${csudo} grep -q -F "$restart_config_str" /etc/inittab || ${csudo} bash -c "echo '${restart_config_str}' >> /etc/inittab"
# TODO: for centos, change here
${csudo} update-rc.d taosd defaults
# chkconfig mysqld on
if ((${initd_mod}==1)); then
${csudo} chkconfig --add taosd || :
${csudo} chkconfig --level 2345 taosd on || :
elif ((${initd_mod}==2)); then
${csudo} insserv taosd || :
${csudo} insserv -d taosd || :
elif ((${initd_mod}==3)); then
${csudo} update-rc.d taosd defaults || :
fi
}
function clean_service_on_systemd() {
taosd_service_config="${service_config_dir}/taosd.service"
# taosd service already is stoped before install
# taosd service already is stoped before install in preinst script
#if systemctl is-active --quiet taosd; then
# echo "TDengine is running, stopping it..."
# ${csudo} systemctl stop taosd &> /dev/null || echo &> /dev/null

44
packaging/tools/preun.sh
View File

@ -26,22 +26,27 @@ initd_mod=0
service_mod=2
if pidof systemd &> /dev/null; then
service_mod=0
elif $(which insserv &> /dev/null); then
elif $(which service &> /dev/null); then
service_mod=1
initd_mod=1
service_config_dir="/etc/init.d"
elif $(which update-rc.d &> /dev/null); then
service_mod=1
initd_mod=2
service_config_dir="/etc/init.d"
if $(which chkconfig &> /dev/null); then
initd_mod=1
elif $(which insserv &> /dev/null); then
initd_mod=2
elif $(which update-rc.d &> /dev/null); then
initd_mod=3
else
service_mod=2
fi
else
service_mod=2
fi
function kill_taosd() {
pid=$(ps -ef | grep "taosd" | grep -v "grep" | awk '{print $2}')
${csudo} kill -9 ${pid} || :
if [ -n "$pid" ]; then
${csudo} kill -9 $pid || :
fi
}
function clean_service_on_systemd() {
@ -57,20 +62,27 @@ function clean_service_on_systemd() {
}
function clean_service_on_sysvinit() {
restart_config_str="taos:2345:respawn:${service_config_dir}/taosd start"
#restart_config_str="taos:2345:respawn:${service_config_dir}/taosd start"
#${csudo} sed -i "\|${restart_config_str}|d" /etc/inittab || :
if pidof taosd &> /dev/null; then
echo "TDengine taosd is running, stopping it..."
${csudo} service taosd stop || :
fi
${csudo} sed -i "\|${restart_config_str}|d" /etc/inittab || :
${csudo} rm -f ${service_config_dir}/taosd || :
if ((${initd_mod}==1)); then
${csudo} grep -q -F "taos" /etc/inittab && ${csudo} insserv -r taosd || :
${csudo} chkconfig --del taosd || :
elif ((${initd_mod}==2)); then
${csudo} grep -q -F "taos" /etc/inittab && ${csudo} update-rc.d -f taosd remove || :
${csudo} insserv -r taosd || :
elif ((${initd_mod}==3)); then
${csudo} update-rc.d -f taosd remove || :
fi
${csudo} rm -f ${service_config_dir}/taosd || :
if $(which init &> /dev/null); then
${csudo} init q || :
fi
# ${csudo} update-rc.d -f taosd remove || :
${csudo} init q || :
}
function clean_service() {
@ -79,7 +91,7 @@ function clean_service() {
elif ((${service_mod}==1)); then
clean_service_on_sysvinit
else
# must manual start taosd
# must manual stop taosd
kill_taosd
fi
}

47
packaging/tools/remove.sh
View File

@ -1,6 +1,6 @@
#!/bin/bash
#
# Script to stop the service and uninstall tdengine, but retain the config, data and log files.
# Script to stop the service and uninstall TDengine, but retain the config, data and log files.
RED='\033[0;31m'
GREEN='\033[1;32m'
@ -27,21 +27,27 @@ initd_mod=0
service_mod=2
if pidof systemd &> /dev/null; then
service_mod=0
elif $(which insserv &> /dev/null); then
elif $(which service &> /dev/null); then
service_mod=1
initd_mod=1
service_config_dir="/etc/init.d"
elif $(which update-rc.d &> /dev/null); then
service_mod=1
initd_mod=2
service_config_dir="/etc/init.d"
if $(which chkconfig &> /dev/null); then
initd_mod=1
elif $(which insserv &> /dev/null); then
initd_mod=2
elif $(which update-rc.d &> /dev/null); then
initd_mod=3
else
service_mod=2
fi
else
service_mod=2
fi
function kill_taosd() {
pid=$(ps -ef | grep "taosd" | grep -v "grep" | awk '{print $2}')
${csudo} kill -9 ${pid} || :
if [ -n "$pid" ]; then
${csudo} kill -9 $pid || :
fi
}
function clean_bin() {
@ -87,20 +93,27 @@ function clean_service_on_systemd() {
}
function clean_service_on_sysvinit() {
restart_config_str="taos:2345:respawn:${service_config_dir}/taosd start"
#restart_config_str="taos:2345:respawn:${service_config_dir}/taosd start"
#${csudo} sed -i "\|${restart_config_str}|d" /etc/inittab || :
if pidof taosd &> /dev/null; then
echo "TDengine taosd is running, stopping it..."
${csudo} service taosd stop || :
fi
${csudo} sed -i "\|${restart_config_str}|d" /etc/inittab || :
${csudo} rm -f ${service_config_dir}/taosd || :
if ((${initd_mod}==1)); then
${csudo} grep -q -F "taos" /etc/inittab && ${csudo} insserv -r taosd || :
${csudo} chkconfig --del taosd || :
elif ((${initd_mod}==2)); then
${csudo} grep -q -F "taos" /etc/inittab && ${csudo} update-rc.d -f taosd remove || :
${csudo} insserv -r taosd || :
elif ((${initd_mod}==3)); then
${csudo} update-rc.d -f taosd remove || :
fi
${csudo} rm -f ${service_config_dir}/taosd || :
if $(which init &> /dev/null); then
${csudo} init q || :
fi
# ${csudo} update-rc.d -f taosd remove || :
${csudo} init q || :
}
function clean_service() {
@ -109,7 +122,7 @@ function clean_service() {
elif ((${service_mod}==1)); then
clean_service_on_sysvinit
else
# must manual start taosd
# must manual stop taosd
kill_taosd
fi
}
@ -140,4 +153,4 @@ elif echo $osinfo | grep -qwi "centos" ; then
${csudo} rpm -e --noscripts tdengine || :
fi
echo -e "${GREEN}TDEngine is removed successfully!${NC}"
echo -e "${GREEN}TDengine is removed successfully!${NC}"

75
packaging/tools/remove_client.sh Executable file
View File

@ -0,0 +1,75 @@
#!/bin/bash
#
# Script to stop the client and uninstall database, but retain the config and log files.
set -e
# set -x
RED='\033[0;31m'
GREEN='\033[1;32m'
NC='\033[0m'
#install main path
install_main_dir="/usr/local/taos"
log_link_dir="/usr/local/taos/log"
cfg_link_dir="/usr/local/taos/cfg"
bin_link_dir="/usr/bin"
lib_link_dir="/usr/lib"
inc_link_dir="/usr/include"
csudo=""
if command -v sudo > /dev/null; then
csudo="sudo"
fi
function kill_client() {
#pid=$(ps -ef | grep "taos" | grep -v "grep" | awk '{print $2}')
if [ -n "$(pidof taos)" ]; then
${csudo} kill -9 $pid || :
fi
}
function clean_bin() {
# Remove link
${csudo} rm -f ${bin_link_dir}/taos || :
${csudo} rm -f ${bin_link_dir}/taosump || :
${csudo} rm -f ${bin_link_dir}/rmtaos || :
}
function clean_lib() {
# Remove link
${csudo} rm -f ${lib_link_dir}/libtaos.* || :
}
function clean_header() {
# Remove link
${csudo} rm -f ${inc_link_dir}/taos.h || :
${csudo} rm -f ${inc_link_dir}/taoserror.h || :
}
function clean_config() {
# Remove link
${csudo} rm -f ${cfg_link_dir}/* || :
}
function clean_log() {
# Remove link
${csudo} rm -rf ${log_link_dir} || :
}
# Stop client.
kill_client
# Remove binary file and links
clean_bin
# Remove header file.
clean_header
# Remove lib file
clean_lib
# Remove link log directory
clean_log
# Remove link configuration file
clean_config
${csudo} rm -rf ${install_main_dir}
echo -e "${GREEN}TDengine client is removed successfully!${NC}"

21
src/inc/tsql.h → src/client/inc/tscSQLParser.h
View File

@ -23,23 +23,10 @@ extern "C" {
#include "taos.h"
#include "tsqldef.h"
#include "ttypes.h"
#define TK_SPACE 200
#define TK_COMMENT 201
#define TK_ILLEGAL 202
#define TK_HEX 203 // hex number 0x123
#define TK_OCT 204 // oct number
#define TK_BIN 205 // bin format data 0b111
#define TK_FILE 206
#define TSQL_SO_ASC 1
#define TSQL_SO_DESC 0
#include "taosmsg.h"
#define MAX_TOKEN_LEN 30
#define TSQL_TBNAME "TBNAME"
#define TSQL_TBNAME_L "tbname"
// token type
enum {
TSQL_NODE_TYPE_EXPR = 0x1,
@ -277,8 +264,7 @@ typedef struct tSQLExpr {
uint32_t nSQLOptr; // TK_FUNCTION: sql function, TK_LE: less than(binary expr)
// the full sql string of function(col, param), which is actually the raw
// field name,
// since the function name is kept in nSQLOptr already
// field name, since the function name is kept in nSQLOptr already
SSQLToken operand;
struct tSQLExprList *pParam; // function parameters
@ -345,8 +331,6 @@ tSQLExprList *tSQLExprListAppend(tSQLExprList *pList, tSQLExpr *pNode, SSQLToken
void tSQLExprListDestroy(tSQLExprList *pList);
int32_t tSQLSyntaxNodeToString(tSQLExpr *pNode, char *dst);
SQuerySQL *tSetQuerySQLElems(SSQLToken *pSelectToken, tSQLExprList *pSelection, tVariantList *pFrom, tSQLExpr *pWhere,
tVariantList *pGroupby, tVariantList *pSortOrder, SSQLToken *pInterval,
SSQLToken *pSliding, tVariantList *pFill, SLimitVal *pLimit, SLimitVal *pGLimit);
@ -378,6 +362,7 @@ tDCLSQL *tTokenListAppend(tDCLSQL *pTokenList, SSQLToken *pToken);
void setCreateDBSQL(SSqlInfo *pInfo, int32_t type, SSQLToken *pToken, SCreateDBInfo *pDB, SSQLToken *pIgExists);
void setCreateAcctSQL(SSqlInfo *pInfo, int32_t type, SSQLToken *pName, SSQLToken *pPwd, SCreateAcctSQL *pAcctInfo);
void setDefaultCreateDbOption(SCreateDBInfo *pDBInfo);
// prefix show db.tables;
void setDBName(SSQLToken *pCpxName, SSQLToken *pDB);

6
src/client/inc/tscUtil.h
View File

@ -127,7 +127,7 @@ int tscAllocPayload(SSqlCmd* pCmd, int size);
void tscFieldInfoSetValFromSchema(SFieldInfo* pFieldInfo, int32_t index, SSchema* pSchema);
void tscFieldInfoSetValFromField(SFieldInfo* pFieldInfo, int32_t index, TAOS_FIELD* pField);
void tscFieldInfoSetValue(SFieldInfo* pFieldInfo, int32_t index, int8_t type, char* name, int16_t bytes);
void tscFieldInfoSetValue(SFieldInfo* pFieldInfo, int32_t index, int8_t type, const char* name, int16_t bytes);
void tscFieldInfoUpdateVisible(SFieldInfo* pFieldInfo, int32_t index, bool visible);
void tscFieldInfoCalOffset(SSqlCmd* pCmd);
@ -143,7 +143,9 @@ void tscClearFieldInfo(SFieldInfo* pFieldInfo);
void addExprParams(SSqlExpr* pExpr, char* argument, int32_t type, int32_t bytes, int16_t tableIndex);
SSqlExpr* tscSqlExprInsert(SSqlCmd* pCmd, int32_t index, int16_t functionId, SColumnIndex* pColIndex, int16_t type,
int16_t size, /*int16_t colId,*/ int16_t interSize);
int16_t size, int16_t interSize);
SSqlExpr* tscSqlExprInsertEmpty(SSqlCmd* pCmd, int32_t index, int16_t functionId);
SSqlExpr* tscSqlExprUpdate(SSqlCmd* pCmd, int32_t index, int16_t functionId, int16_t srcColumnIndex, int16_t type,
int16_t size);

11
src/client/inc/tsclient.h
View File

@ -35,7 +35,7 @@ extern "C" {
#include "tlog.h"
#include "tscCache.h"
#include "tsdb.h"
#include "tsql.h"
#include "tscSQLParser.h"
#include "tsqlfunction.h"
#include "tutil.h"
@ -92,7 +92,12 @@ enum _sql_cmd {
*/
TSDB_SQL_RETRIEVE_EMPTY_RESULT,
TSDB_SQL_RESET_CACHE,
TSDB_SQL_RESET_CACHE, // 40
TSDB_SQL_SERV_STATUS,
TSDB_SQL_CURRENT_DB,
TSDB_SQL_SERV_VERSION,
TSDB_SQL_CLI_VERSION,
TSDB_SQL_CURRENT_USER,
TSDB_SQL_CFG_LOCAL,
TSDB_SQL_MAX
@ -346,7 +351,7 @@ typedef struct _tsc_obj {
char user[TSDB_USER_LEN];
char pass[TSDB_KEY_LEN];
char acctId[TSDB_DB_NAME_LEN];
char db[TSDB_DB_NAME_LEN];
char db[TSDB_METER_ID_LEN];
char sversion[TSDB_VERSION_LEN];
char writeAuth : 1;
char superAuth : 1;

2977
src/client/src/sql.c Normal file
View File

File diff suppressed because it is too large Load Diff

10
src/client/src/tscAst.c
View File

@ -21,8 +21,12 @@
#include "tschemautil.h"
#include "tsdb.h"
#include "tskiplist.h"
#include "tsqldef.h"
#include "tsqlfunction.h"
#include "tstoken.h"
#include "ttypes.h"
#include "tutil.h"
#include "tscSQLParser.h"
/*
*
@ -37,10 +41,10 @@
*/
static tSQLSyntaxNode *tSQLSyntaxNodeCreate(SSchema *pSchema, int32_t numOfCols, SSQLToken *pToken);
static void tSQLSyntaxNodeDestroy(tSQLSyntaxNode *pNode, void (*fp)(void *));
static void tSQLSyntaxNodeDestroy(tSQLSyntaxNode *pNode, void (*fp)(void *));
static tSQLSyntaxNode *createSyntaxTree(SSchema *pSchema, int32_t numOfCols, char *str, int32_t *i);
static void destroySyntaxTree(tSQLSyntaxNode *);
static void destroySyntaxTree(tSQLSyntaxNode *);
static uint8_t isQueryOnPrimaryKey(const char *primaryColumnName, const tSQLSyntaxNode *pLeft,
const tSQLSyntaxNode *pRight);
@ -255,7 +259,7 @@ static tSQLSyntaxNode *createSyntaxTree(SSchema *pSchema, int32_t numOfCols, cha
t0 = tStrGetToken(str, i, false, 0, NULL);
if (t0.n == 0 || t0.type == TK_RP) {
if (pLeft->nodeType != TSQL_NODE_EXPR) { // if left is not the expr, it is not a legal expr
if (pLeft->nodeType != TSQL_NODE_EXPR) { // if left is not the expr, it is not a legal expr
tSQLSyntaxNodeDestroy(pLeft, NULL);
return NULL;
}

5
src/client/src/tscAsync.c
View File

@ -22,8 +22,9 @@
#include "tscUtil.h"
#include "tsclient.h"
#include "tsocket.h"
#include "tsql.h"
#include "tscSQLParser.h"
#include "tutil.h"
#include "tnote.h"
void tscProcessFetchRow(SSchedMsg *pMsg);
void tscProcessAsyncRetrieve(void *param, TAOS_RES *tres, int numOfRows);
@ -55,6 +56,8 @@ void taos_query_a(TAOS *taos, const char *sqlstr, void (*fp)(void *, TAOS_RES *,
return;
}
taosNotePrintTsc(sqlstr);
SSqlObj *pSql = (SSqlObj *)malloc(sizeof(SSqlObj));
if (pSql == NULL) {
tscError("failed to malloc sqlObj");

100
src/client/src/tscLocal.c
View File

@ -26,6 +26,8 @@
#include "tschemautil.h"
#include "tsocket.h"
static void tscSetLocalQueryResult(SSqlObj *pSql, const char *val, const char *columnName, size_t valueLength);
static int32_t getToStringLength(const char *pData, int32_t length, int32_t type) {
char buf[512] = {0};
@ -39,7 +41,7 @@ static int32_t getToStringLength(const char *pData, int32_t length, int32_t type
case TSDB_DATA_TYPE_DOUBLE: {
#ifdef _TD_ARM_32_
double dv = 0;
*(int64_t*)(&dv) = *(int64_t*)pData;
*(int64_t *)(&dv) = *(int64_t *)pData;
len = sprintf(buf, "%f", dv);
#else
len = sprintf(buf, "%lf", *(double *)pData);
@ -47,12 +49,11 @@ static int32_t getToStringLength(const char *pData, int32_t length, int32_t type
if (strncasecmp("nan", buf, 3) == 0) {
len = 4;
}
}
break;
} break;
case TSDB_DATA_TYPE_FLOAT: {
#ifdef _TD_ARM_32_
float fv = 0;
*(int32_t*)(&fv) = *(int32_t*)pData;
*(int32_t *)(&fv) = *(int32_t *)pData;
len = sprintf(buf, "%f", fv);
#else
len = sprintf(buf, "%f", *(float *)pData);
@ -60,8 +61,7 @@ static int32_t getToStringLength(const char *pData, int32_t length, int32_t type
if (strncasecmp("nan", buf, 3) == 0) {
len = 4;
}
}
break;
} break;
case TSDB_DATA_TYPE_TIMESTAMP:
case TSDB_DATA_TYPE_BIGINT:
len = sprintf(buf, "%lld", *(int64_t *)pData);
@ -203,23 +203,21 @@ static int32_t tscSetValueToResObj(SSqlObj *pSql, int32_t rowLen) {
case TSDB_DATA_TYPE_FLOAT: {
#ifdef _TD_ARM_32_
float fv = 0;
*(int32_t*)(&fv) = *(int32_t*)pTagValue;
*(int32_t *)(&fv) = *(int32_t *)pTagValue;
sprintf(target, "%f", fv);
#else
sprintf(target, "%f", *(float *)pTagValue);
#endif
}
break;
} break;
case TSDB_DATA_TYPE_DOUBLE: {
#ifdef _TD_ARM_32_
double dv = 0;
*(int64_t*)(&dv) = *(int64_t*)pTagValue;
*(int64_t *)(&dv) = *(int64_t *)pTagValue;
sprintf(target, "%lf", dv);
#else
sprintf(target, "%lf", *(double *)pTagValue);
#endif
}
break;
} break;
case TSDB_DATA_TYPE_TINYINT:
sprintf(target, "%d", *(int8_t *)pTagValue);
break;
@ -391,6 +389,68 @@ static int tscProcessQueryTags(SSqlObj *pSql) {
}
}
static void tscProcessCurrentUser(SSqlObj *pSql) {
SSqlExpr* pExpr = tscSqlExprGet(&pSql->cmd, 0);
tscSetLocalQueryResult(pSql, pSql->pTscObj->user, pExpr->aliasName, TSDB_USER_LEN);
}
static void tscProcessCurrentDB(SSqlObj *pSql) {
char db[TSDB_DB_NAME_LEN + 1] = {0};
extractDBName(pSql->pTscObj->db, db);
// no use db is invoked before.
if (strlen(db) == 0) {
setNull(db, TSDB_DATA_TYPE_BINARY, TSDB_DB_NAME_LEN);
}
SSqlExpr* pExpr = tscSqlExprGet(&pSql->cmd, 0);
tscSetLocalQueryResult(pSql, db, pExpr->aliasName, TSDB_DB_NAME_LEN);
}
static void tscProcessServerVer(SSqlObj *pSql) {
const char* v = pSql->pTscObj->sversion;
SSqlExpr* pExpr = tscSqlExprGet(&pSql->cmd, 0);
tscSetLocalQueryResult(pSql, v, pExpr->aliasName, tListLen(pSql->pTscObj->sversion));
}
static void tscProcessClientVer(SSqlObj *pSql) {
SSqlExpr* pExpr = tscSqlExprGet(&pSql->cmd, 0);
tscSetLocalQueryResult(pSql, version, pExpr->aliasName, strlen(version));
}
static void tscProcessServStatus(SSqlObj *pSql) {
STscObj* pObj = pSql->pTscObj;
if (pObj->pHb != NULL) {
if (pObj->pHb->res.code == TSDB_CODE_NETWORK_UNAVAIL) {
pSql->res.code = TSDB_CODE_NETWORK_UNAVAIL;
return;
}
} else {
if (pSql->res.code == TSDB_CODE_NETWORK_UNAVAIL) {
return;
}
}
SSqlExpr* pExpr = tscSqlExprGet(&pSql->cmd, 0);
tscSetLocalQueryResult(pSql, "1", pExpr->aliasName, 2);
}
void tscSetLocalQueryResult(SSqlObj *pSql, const char *val, const char *columnName, size_t valueLength) {
SSqlCmd *pCmd = &pSql->cmd;
SSqlRes *pRes = &pSql->res;
pCmd->numOfCols = 1;
pCmd->order.order = TSQL_SO_ASC;
tscFieldInfoSetValue(&pCmd->fieldsInfo, 0, TSDB_DATA_TYPE_BINARY, columnName, valueLength);
tscInitResObjForLocalQuery(pSql, 1, valueLength);
TAOS_FIELD *pField = tscFieldInfoGetField(pCmd, 0);
strncpy(pRes->data, val, pField->bytes);
}
int tscProcessLocalCmd(SSqlObj *pSql) {
SSqlCmd *pCmd = &pSql->cmd;
@ -402,13 +462,23 @@ int tscProcessLocalCmd(SSqlObj *pSql) {
pSql->res.code = (uint8_t)tscProcessQueryTags(pSql);
} else if (pCmd->command == TSDB_SQL_RETRIEVE_EMPTY_RESULT) {
/*
* pass the qhandle check, in order to call partial release function to
* free allocated resources and remove the SqlObj from linked list
* set the qhandle to be 1 in order to pass the qhandle check, and to call partial release function to
* free allocated resources and remove the SqlObj from sql query linked list
*/
pSql->res.qhandle = 0x1; // pass the qhandle check
pSql->res.qhandle = 0x1;
pSql->res.numOfRows = 0;
} else if (pCmd->command == TSDB_SQL_RESET_CACHE) {
taosClearDataCache(tscCacheHandle);
} else if (pCmd->command == TSDB_SQL_SERV_VERSION) {
tscProcessServerVer(pSql);
} else if (pCmd->command == TSDB_SQL_CLI_VERSION) {
tscProcessClientVer(pSql);
} else if (pCmd->command == TSDB_SQL_CURRENT_USER) {
tscProcessCurrentUser(pSql);
} else if (pCmd->command == TSDB_SQL_CURRENT_DB) {
tscProcessCurrentDB(pSql);
} else if (pCmd->command == TSDB_SQL_SERV_STATUS) {
tscProcessServStatus(pSql);
} else {
pSql->res.code = TSDB_CODE_INVALID_SQL;
tscError("%p not support command:%d", pSql, pCmd->command);

2
src/client/src/tscPrepare.c
View File

@ -15,7 +15,7 @@
#include "taos.h"
#include "tsclient.h"
#include "tsql.h"
#include "tscSQLParser.h"
#include "tscUtil.h"
#include "ttimer.h"
#include "taosmsg.h"

310
src/client/src/tscSQLParser.c
View File

@ -25,7 +25,8 @@
#include "tscUtil.h"
#include "tschemautil.h"
#include "tsclient.h"
#include "tsql.h"
#include "tscSQLParser.h"
#pragma GCC diagnostic ignored "-Wunused-variable"
#define DEFAULT_PRIMARY_TIMESTAMP_COL_NAME "_c0"
@ -48,18 +49,15 @@ typedef struct SColumnIdListRes {
SColumnList list;
} SColumnIdListRes;
static SSqlExpr* doAddProjectCol(SSqlCmd* pCmd, int32_t fieldIDInResult, int32_t colIdx, int32_t tableIndex);
static SSqlExpr* doAddProjectCol(SSqlCmd* pCmd, int32_t outputIndex, int32_t colIdx, int32_t tableIndex);
static int32_t setShowInfo(SSqlObj* pSql, SSqlInfo* pInfo);
static bool has(tFieldList* pFieldList, int32_t offset, char* name);
static char* getAccountId(SSqlObj* pSql);
static bool has(tFieldList* pFieldList, int32_t startIdx, const char* name);
static void getCurrentDBName(SSqlObj* pSql, SSQLToken* pDBToken);
static bool hasSpecifyDB(SSQLToken* pTableName);
static bool validateTableColumnInfo(tFieldList* pFieldList, SSqlCmd* pCmd);
static bool validateTagParams(tFieldList* pTagsList, tFieldList* pFieldList, SSqlCmd* pCmd);
static int32_t setObjFullName(char* fullName, char* account, SSQLToken* pDB, SSQLToken* tableName, int32_t* len);
@ -68,20 +66,18 @@ static void getColumnName(tSQLExprItem* pItem, char* resultFieldName, int32_t nL
static void getRevisedName(char* resultFieldName, int32_t functionId, int32_t maxLen, char* columnName);
static int32_t addExprAndResultField(SSqlCmd* pCmd, int32_t colIdx, tSQLExprItem* pItem);
static int32_t insertResultField(SSqlCmd* pCmd, int32_t fieldIDInResult, SColumnList* pIdList, int16_t bytes,
int8_t type, char* fieldName);
static int32_t changeFunctionID(int32_t optr, int16_t* functionId);
static int32_t parseSelectClause(SSqlCmd* pCmd, tSQLExprList* pSelection, bool isMetric);
static void setErrMsg(SSqlCmd* pCmd, const char* pzErrMsg);
static int32_t parseSelectClause(SSqlCmd* pCmd, tSQLExprList* pSelection, bool isMetric);
static bool validateIpAddress(char* ip);
static bool hasUnsupportFunctionsForMetricQuery(SSqlCmd* pCmd);
static bool functionCompatibleCheck(SSqlCmd* pCmd);
static void setColumnOffsetValueInResultset(SSqlCmd* pCmd);
static void setColumnOffsetValueInResultset(SSqlCmd* pCmd);
static int32_t parseGroupbyClause(SSqlCmd* pCmd, tVariantList* pList);
static int32_t parseIntervalClause(SSqlCmd* pCmd, SQuerySQL* pQuerySql);
@ -94,7 +90,6 @@ static int32_t parseFillClause(SSqlCmd* pCmd, SQuerySQL* pQuerySQL);
static int32_t parseOrderbyClause(SSqlCmd* pCmd, SQuerySQL* pQuerySql, SSchema* pSchema, int32_t numOfCols);
static int32_t tsRewriteFieldNameIfNecessary(SSqlCmd* pCmd);
static bool validateOneTags(SSqlCmd* pCmd, TAOS_FIELD* pTagField);
static int32_t setAlterTableInfo(SSqlObj* pSql, struct SSqlInfo* pInfo);
static int32_t validateSqlFunctionInStreamSql(SSqlCmd* pCmd);
static int32_t buildArithmeticExprString(tSQLExpr* pExpr, char** exprString);
@ -105,18 +100,22 @@ static int32_t validateLocalConfig(tDCLSQL* pOptions);
static int32_t validateColumnName(char* name);
static int32_t setKillInfo(SSqlObj* pSql, struct SSqlInfo* pInfo);
static bool hasTimestampForPointInterpQuery(SSqlCmd* pCmd);
static void updateTagColumnIndex(SSqlCmd* pCmd, int32_t tableIndex);
static int32_t parseLimitClause(SSqlObj* pSql, SQuerySQL* pQuerySql);
static bool validateOneTags(SSqlCmd* pCmd, TAOS_FIELD* pTagField);
static bool hasTimestampForPointInterpQuery(SSqlCmd* pCmd);
static void updateTagColumnIndex(SSqlCmd* pCmd, int32_t tableIndex);
static int32_t parseCreateDBOptions(SCreateDBInfo* pCreateDbSql, SSqlCmd* pCmd);
static int32_t parseLimitClause(SSqlObj* pSql, SQuerySQL* pQuerySql);
static int32_t parseCreateDBOptions(SSqlCmd* pCmd, SCreateDBInfo* pCreateDbSql);
static int32_t getColumnIndexByNameEx(SSQLToken* pToken, SSqlCmd* pCmd, SColumnIndex* pIndex);
static int32_t getTableIndexByName(SSQLToken* pToken, SSqlCmd* pCmd, SColumnIndex* pIndex);
static int32_t optrToString(tSQLExpr* pExpr, char** exprString);
static int32_t getMeterIndex(SSQLToken* pTableToken, SSqlCmd* pCmd, SColumnIndex* pIndex);
static int32_t doFunctionsCompatibleCheck(SSqlObj* pSql);
static int32_t doLocalQueryProcess(SQuerySQL* pQuerySql, SSqlCmd* pCmd);
static int32_t tscCheckCreateDbParams(SSqlCmd* pCmd, SCreateDbMsg *pCreate);
static SColumnList getColumnList(int32_t num, int16_t tableIndex, int32_t columnIndex);
static int32_t getMeterIndex(SSQLToken* pTableToken, SSqlCmd* pCmd, SColumnIndex* pIndex);
static int32_t doFunctionsCompatibleCheck(SSqlObj* pSql);
static int32_t tscQueryOnlyMetricTags(SSqlCmd* pCmd, bool* queryOnMetricTags) {
assert(QUERY_IS_STABLE_QUERY(pCmd->type));
@ -320,7 +319,7 @@ int32_t tscToSQLCmd(SSqlObj* pSql, struct SSqlInfo* pInfo) {
return ret;
}
if (parseCreateDBOptions(pCreateDB, pCmd) != TSDB_CODE_SUCCESS) {
if (parseCreateDBOptions(pCmd, pCreateDB) != TSDB_CODE_SUCCESS) {
return TSDB_CODE_INVALID_SQL;
}
@ -877,7 +876,7 @@ int32_t tscToSQLCmd(SSqlObj* pSql, struct SSqlInfo* pInfo) {
case TSQL_QUERY_METER: {
SQuerySQL* pQuerySql = pInfo->pQueryInfo;
assert(pQuerySql != NULL && pQuerySql->from->nExpr > 0);
assert(pQuerySql != NULL && (pQuerySql->from == NULL || pQuerySql->from->nExpr > 0));
const char* msg0 = "invalid table name";
const char* msg1 = "table name too long";
@ -896,6 +895,19 @@ int32_t tscToSQLCmd(SSqlObj* pSql, struct SSqlInfo* pInfo) {
return TSDB_CODE_INVALID_SQL;
}
/*
* handle the sql expression without from subclause
* select current_database();
* select server_version();
* select client_version();
* select server_state();
*/
if (pQuerySql->from == NULL) {
assert(pQuerySql->fillType == NULL && pQuerySql->pGroupby == NULL && pQuerySql->pWhere == NULL &&
pQuerySql->pSortOrder == NULL);
return doLocalQueryProcess(pQuerySql, pCmd);
}
if (pQuerySql->from->nExpr > TSDB_MAX_JOIN_TABLE_NUM) {
setErrMsg(pCmd, msg7);
return TSDB_CODE_INVALID_SQL;
@ -1120,7 +1132,7 @@ int32_t parseIntervalClause(SSqlCmd* pCmd, SQuerySQL* pQuerySql) {
SMeterMetaInfo* pMeterMetaInfo = tscGetMeterMetaInfo(pCmd, 0);
if (pQuerySql->interval.type == 0) {
if (pQuerySql->interval.type == 0 || pQuerySql->interval.n == 0) {
return TSDB_CODE_SUCCESS;
}
@ -1497,7 +1509,7 @@ bool validateOneColumn(SSqlCmd* pCmd, TAOS_FIELD* pColField) {
}
/* is contained in pFieldList or not */
static bool has(tFieldList* pFieldList, int32_t startIdx, char* name) {
static bool has(tFieldList* pFieldList, int32_t startIdx, const char* name) {
for (int32_t j = startIdx; j < pFieldList->nField; ++j) {
if (strncasecmp(name, pFieldList->p[j].name, TSDB_COL_NAME_LEN) == 0) return true;
}
@ -4944,7 +4956,7 @@ int32_t validateFunctionsInIntervalOrGroupbyQuery(SSqlCmd* pCmd) {
// projection query on primary timestamp, the selectivity function needs to be present.
if (pExpr->functionId == TSDB_FUNC_PRJ && pExpr->colInfo.colId == PRIMARYKEY_TIMESTAMP_COL_INDEX) {
bool hasSelectivity = false;
for(int32_t j = 0; j < pCmd->fieldsInfo.numOfOutputCols; ++j) {
for (int32_t j = 0; j < pCmd->fieldsInfo.numOfOutputCols; ++j) {
SSqlExpr* pEx = tscSqlExprGet(pCmd, j);
if ((aAggs[pEx->functionId].nStatus & TSDB_FUNCSTATE_SELECTIVITY) == TSDB_FUNCSTATE_SELECTIVITY) {
hasSelectivity = true;
@ -4958,7 +4970,7 @@ int32_t validateFunctionsInIntervalOrGroupbyQuery(SSqlCmd* pCmd) {
}
if (pExpr->functionId == TSDB_FUNC_PRJ || pExpr->functionId == TSDB_FUNC_DIFF ||
pExpr->functionId == TSDB_FUNC_ARITHM) {
pExpr->functionId == TSDB_FUNC_ARITHM) {
isProjectionFunction = true;
}
}
@ -4971,8 +4983,8 @@ int32_t validateFunctionsInIntervalOrGroupbyQuery(SSqlCmd* pCmd) {
}
typedef struct SDNodeDynConfOption {
char* name;
int32_t len;
char* name; // command name
int32_t len; // name string length
} SDNodeDynConfOption;
int32_t validateDNodeConfig(tDCLSQL* pOptions) {
@ -4980,7 +4992,7 @@ int32_t validateDNodeConfig(tDCLSQL* pOptions) {
return TSDB_CODE_INVALID_SQL;
}
SDNodeDynConfOption DNODE_DYNAMIC_CFG_OPTIONS[14] = {
const SDNodeDynConfOption DNODE_DYNAMIC_CFG_OPTIONS[14] = {
{"resetLog", 8}, {"resetQueryCache", 15}, {"dDebugFlag", 10}, {"rpcDebugFlag", 12},
{"tmrDebugFlag", 12}, {"cDebugFlag", 10}, {"uDebugFlag", 10}, {"mDebugFlag", 10},
{"sdbDebugFlag", 12}, {"httpDebugFlag", 13}, {"monitorDebugFlag", 16}, {"qDebugflag", 10},
@ -4991,7 +5003,7 @@ int32_t validateDNodeConfig(tDCLSQL* pOptions) {
if (pOptions->nTokens == 2) {
// reset log and reset query cache does not need value
for (int32_t i = 0; i < 2; ++i) {
SDNodeDynConfOption* pOption = &DNODE_DYNAMIC_CFG_OPTIONS[i];
const SDNodeDynConfOption* pOption = &DNODE_DYNAMIC_CFG_OPTIONS[i];
if ((strncasecmp(pOption->name, pOptionToken->z, pOptionToken->n) == 0) && (pOption->len == pOptionToken->n)) {
return TSDB_CODE_SUCCESS;
}
@ -5014,7 +5026,7 @@ int32_t validateDNodeConfig(tDCLSQL* pOptions) {
}
for (int32_t i = 2; i < tListLen(DNODE_DYNAMIC_CFG_OPTIONS) - 1; ++i) {
SDNodeDynConfOption* pOption = &DNODE_DYNAMIC_CFG_OPTIONS[i];
const SDNodeDynConfOption* pOption = &DNODE_DYNAMIC_CFG_OPTIONS[i];
if ((strncasecmp(pOption->name, pOptionToken->z, pOptionToken->n) == 0) && (pOption->len == pOptionToken->n)) {
/* options is valid */
@ -5129,7 +5141,7 @@ int32_t parseLimitClause(SSqlObj* pSql, SQuerySQL* pQuerySql) {
}
if (UTIL_METER_IS_METRIC(pMeterMetaInfo)) {
bool queryOnTags = false;
bool queryOnTags = false;
if (tscQueryOnlyMetricTags(pCmd, &queryOnTags) != TSDB_CODE_SUCCESS) {
return TSDB_CODE_INVALID_SQL;
}
@ -5187,57 +5199,46 @@ int32_t parseLimitClause(SSqlObj* pSql, SQuerySQL* pQuerySql) {
return TSDB_CODE_SUCCESS;
}
static void setCreateDBOption(SCreateDbMsg* pMsg, SCreateDBInfo* pCreateDb) {
pMsg->precision = TSDB_TIME_PRECISION_MILLI; // millisecond by default
static int32_t setKeepOption(SSqlCmd* pCmd, SCreateDbMsg* pMsg, SCreateDBInfo* pCreateDb) {
const char* msg = "invalid number of options";
pMsg->daysToKeep = htonl(-1);
pMsg->daysToKeep1 = htonl(-1);
pMsg->daysToKeep2 = htonl(-1);
pMsg->blocksPerMeter = (pCreateDb->numOfBlocksPerTable == 0) ? htons(-1) : htons(pCreateDb->numOfBlocksPerTable);
pMsg->compression = (pCreateDb->compressionLevel == 0) ? -1 : pCreateDb->compressionLevel;
pMsg->commitLog = (pCreateDb->commitLog == 0) ? -1 : pCreateDb->commitLog;
pMsg->commitTime = (pCreateDb->commitTime == 0) ? htonl(-1) : htonl(pCreateDb->commitTime);
pMsg->maxSessions = (pCreateDb->tablesPerVnode == 0) ? htonl(-1) : htonl(pCreateDb->tablesPerVnode);
pMsg->cacheNumOfBlocks.fraction = (pCreateDb->numOfAvgCacheBlocks == 0) ? -1 : pCreateDb->numOfAvgCacheBlocks;
pMsg->cacheBlockSize = (pCreateDb->cacheBlockSize == 0) ? htonl(-1) : htonl(pCreateDb->cacheBlockSize);
pMsg->rowsInFileBlock = (pCreateDb->rowPerFileBlock == 0) ? htonl(-1) : htonl(pCreateDb->rowPerFileBlock);
pMsg->daysPerFile = (pCreateDb->daysPerFile == 0) ? htonl(-1) : htonl(pCreateDb->daysPerFile);
pMsg->replications = (pCreateDb->replica == 0) ? -1 : pCreateDb->replica;
}
int32_t parseCreateDBOptions(SCreateDBInfo* pCreateDbSql, SSqlCmd* pCmd) {
const char* msg0 = "invalid number of options";
const char* msg1 = "invalid time precision";
SCreateDbMsg* pMsg = (SCreateDbMsg*)(pCmd->payload + tsRpcHeadSize + sizeof(SMgmtHead));
setCreateDBOption(pMsg, pCreateDbSql);
if (pCreateDbSql->keep != NULL) {
switch (pCreateDbSql->keep->nExpr) {
tVariantList* pKeep = pCreateDb->keep;
if (pKeep != NULL) {
switch (pKeep->nExpr) {
case 1:
pMsg->daysToKeep = htonl(pCreateDbSql->keep->a[0].pVar.i64Key);
pMsg->daysToKeep = htonl(pKeep->a[0].pVar.i64Key);
break;
case 2: {
pMsg->daysToKeep = htonl(pCreateDbSql->keep->a[0].pVar.i64Key);
pMsg->daysToKeep1 = htonl(pCreateDbSql->keep->a[1].pVar.i64Key);
pMsg->daysToKeep = htonl(pKeep->a[0].pVar.i64Key);
pMsg->daysToKeep1 = htonl(pKeep->a[1].pVar.i64Key);
break;
}
case 3: {
pMsg->daysToKeep = htonl(pCreateDbSql->keep->a[0].pVar.i64Key);
pMsg->daysToKeep1 = htonl(pCreateDbSql->keep->a[1].pVar.i64Key);
pMsg->daysToKeep2 = htonl(pCreateDbSql->keep->a[2].pVar.i64Key);
pMsg->daysToKeep = htonl(pKeep->a[0].pVar.i64Key);
pMsg->daysToKeep1 = htonl(pKeep->a[1].pVar.i64Key);
pMsg->daysToKeep2 = htonl(pKeep->a[2].pVar.i64Key);
break;
}
default: {
setErrMsg(pCmd, msg0);
setErrMsg(pCmd, msg);
return TSDB_CODE_INVALID_SQL;
}
}
}
SSQLToken* pToken = &pCreateDbSql->precision;
return TSDB_CODE_SUCCESS;
}
static int32_t setTimePrecisionOption(SSqlCmd* pCmd, SCreateDbMsg* pMsg, SCreateDBInfo* pCreateDbInfo) {
const char* msg = "invalid time precision";
pMsg->precision = TSDB_TIME_PRECISION_MILLI; // millisecond by default
SSQLToken* pToken = &pCreateDbInfo->precision;
if (pToken->n > 0) {
pToken->n = strdequote(pToken->z);
@ -5246,10 +5247,10 @@ int32_t parseCreateDBOptions(SCreateDBInfo* pCreateDbSql, SSqlCmd* pCmd) {
// time precision for this db: million second
pMsg->precision = TSDB_TIME_PRECISION_MILLI;
} else if (strncmp(pToken->z, TSDB_TIME_PRECISION_MICRO_STR, pToken->n) == 0 &&
strlen(TSDB_TIME_PRECISION_MICRO_STR) == pToken->n) {
strlen(TSDB_TIME_PRECISION_MICRO_STR) == pToken->n) {
pMsg->precision = TSDB_TIME_PRECISION_MICRO;
} else {
setErrMsg(pCmd, msg1);
setErrMsg(pCmd, msg);
return TSDB_CODE_INVALID_SQL;
}
}
@ -5257,6 +5258,39 @@ int32_t parseCreateDBOptions(SCreateDBInfo* pCreateDbSql, SSqlCmd* pCmd) {
return TSDB_CODE_SUCCESS;
}
static void setCreateDBOption(SCreateDbMsg* pMsg, SCreateDBInfo* pCreateDb) {
pMsg->blocksPerMeter = htons(pCreateDb->numOfBlocksPerTable);
pMsg->compression = pCreateDb->compressionLevel;
pMsg->commitLog = (char) pCreateDb->commitLog;
pMsg->commitTime = htonl(pCreateDb->commitTime);
pMsg->maxSessions = htonl(pCreateDb->tablesPerVnode);
pMsg->cacheNumOfBlocks.fraction = pCreateDb->numOfAvgCacheBlocks;
pMsg->cacheBlockSize = htonl(pCreateDb->cacheBlockSize);
pMsg->rowsInFileBlock = htonl(pCreateDb->rowPerFileBlock);
pMsg->daysPerFile = htonl(pCreateDb->daysPerFile);
pMsg->replications = pCreateDb->replica;
}
int32_t parseCreateDBOptions(SSqlCmd* pCmd, SCreateDBInfo* pCreateDbSql) {
SCreateDbMsg* pMsg = (SCreateDbMsg*)(pCmd->payload + tsRpcHeadSize + sizeof(SMgmtHead));
setCreateDBOption(pMsg, pCreateDbSql);
if (setKeepOption(pCmd, pMsg, pCreateDbSql) != TSDB_CODE_SUCCESS) {
return TSDB_CODE_INVALID_SQL;
}
if (setTimePrecisionOption(pCmd, pMsg, pCreateDbSql) != TSDB_CODE_SUCCESS) {
return TSDB_CODE_INVALID_SQL;
}
if (tscCheckCreateDbParams(pCmd, pMsg) != TSDB_CODE_SUCCESS) {
return TSDB_CODE_INVALID_SQL;
}
return TSDB_CODE_SUCCESS;
}
void tscAddTimestampColumn(SSqlCmd* pCmd, int16_t functionId, int16_t tableIndex) {
// the first column not timestamp column, add it
SSqlExpr* pExpr = NULL;
@ -5365,9 +5399,8 @@ static void doUpdateSqlFunctionForTagPrj(SSqlCmd* pCmd) {
static void doUpdateSqlFunctionForColPrj(SSqlCmd* pCmd) {
for (int32_t i = 0; i < pCmd->fieldsInfo.numOfOutputCols; ++i) {
SSqlExpr *pExpr = tscSqlExprGet(pCmd, i);
SSqlExpr* pExpr = tscSqlExprGet(pCmd, i);
if (pExpr->functionId == TSDB_FUNC_PRJ) {
bool qualifiedCol = false;
for (int32_t j = 0; j < pCmd->groupbyExpr.numOfGroupCols; ++j) {
if (pExpr->colInfo.colId == pCmd->groupbyExpr.columnInfo[j].colId) {
@ -5686,3 +5719,150 @@ int32_t doFunctionsCompatibleCheck(SSqlObj* pSql) {
return checkUpdateTagPrjFunctions(pCmd);
}
}
int32_t doLocalQueryProcess(SQuerySQL* pQuerySql, SSqlCmd* pCmd) {
const char* msg1 = "only one expression allowed";
const char* msg2 = "invalid expression in select clause";
const char* msg3 = "invalid function";
tSQLExprList* pExprList = pQuerySql->pSelection;
if (pExprList->nExpr != 1) {
setErrMsg(pCmd, msg1);
return TSDB_CODE_INVALID_SQL;
}
tSQLExpr* pExpr = pExprList->a[0].pNode;
if (pExpr->operand.z == NULL) {
setErrMsg(pCmd, msg2);
return TSDB_CODE_INVALID_SQL;
}
SDNodeDynConfOption functionsInfo[5] = {{"database()", 10},
{"server_version()", 16},
{"server_status()", 15},
{"client_version()", 16},
{"current_user()", 14}};
int32_t index = -1;
for (int32_t i = 0; i < tListLen(functionsInfo); ++i) {
if (strncasecmp(functionsInfo[i].name, pExpr->operand.z, functionsInfo[i].len) == 0 &&
functionsInfo[i].len == pExpr->operand.n) {
index = i;
break;
}
}
SSqlExpr* pExpr1 = tscSqlExprInsertEmpty(pCmd, 0, TSDB_FUNC_TAG_DUMMY);
if (pExprList->a[0].aliasName != NULL) {
strncpy(pExpr1->aliasName, pExprList->a[0].aliasName, tListLen(pExpr1->aliasName));
} else {
strncpy(pExpr1->aliasName, functionsInfo[index].name, tListLen(pExpr1->aliasName));
}
switch (index) {
case 0:
pCmd->command = TSDB_SQL_CURRENT_DB;
return TSDB_CODE_SUCCESS;
case 1:
pCmd->command = TSDB_SQL_SERV_VERSION;
return TSDB_CODE_SUCCESS;
case 2:
pCmd->command = TSDB_SQL_SERV_STATUS;
return TSDB_CODE_SUCCESS;
case 3:
pCmd->command = TSDB_SQL_CLI_VERSION;
return TSDB_CODE_SUCCESS;
case 4:
pCmd->command = TSDB_SQL_CURRENT_USER;
return TSDB_CODE_SUCCESS;
default: {
setErrMsg(pCmd, msg3);
return TSDB_CODE_INVALID_SQL;
}
}
}
// can only perform the parameters based on the macro definitation
int32_t tscCheckCreateDbParams(SSqlCmd* pCmd, SCreateDbMsg *pCreate) {
char msg[512] = {0};
if (pCreate->commitLog != -1 && (pCreate->commitLog < 0 || pCreate->commitLog > 1)) {
snprintf(msg, tListLen(msg), "invalid db option commitLog: %d, only 0 or 1 allowed", pCreate->commitLog);
setErrMsg(pCmd, msg);
return TSDB_CODE_INVALID_SQL;
}
if (pCreate->replications != -1 &&
(pCreate->replications < TSDB_REPLICA_MIN_NUM || pCreate->replications > TSDB_REPLICA_MAX_NUM)) {
snprintf(msg, tListLen(msg), "invalid db option replications: %d valid range: [%d, %d]", pCreate->replications, TSDB_REPLICA_MIN_NUM,
TSDB_REPLICA_MAX_NUM);
setErrMsg(pCmd, msg);
return TSDB_CODE_INVALID_SQL;
}
int32_t val = htonl(pCreate->daysPerFile);
if (val != -1 && (val < TSDB_FILE_MIN_PARTITION_RANGE || val > TSDB_FILE_MAX_PARTITION_RANGE)) {
snprintf(msg, tListLen(msg), "invalid db option daysPerFile: %d valid range: [%d, %d]", val,
TSDB_FILE_MIN_PARTITION_RANGE, TSDB_FILE_MAX_PARTITION_RANGE);
setErrMsg(pCmd, msg);
return TSDB_CODE_INVALID_SQL;
}
val = htonl(pCreate->rowsInFileBlock);
if (val != -1 && (val < TSDB_MIN_ROWS_IN_FILEBLOCK || val > TSDB_MAX_ROWS_IN_FILEBLOCK)) {
snprintf(msg, tListLen(msg), "invalid db option rowsInFileBlock: %d valid range: [%d, %d]", val,
TSDB_MIN_ROWS_IN_FILEBLOCK, TSDB_MAX_ROWS_IN_FILEBLOCK);
setErrMsg(pCmd, msg);
return TSDB_CODE_INVALID_SQL;
}
val = htonl(pCreate->cacheBlockSize);
if (val != -1 && (val < TSDB_MIN_CACHE_BLOCK_SIZE || val > TSDB_MAX_CACHE_BLOCK_SIZE)) {
snprintf(msg, tListLen(msg), "invalid db option cacheBlockSize: %d valid range: [%d, %d]", val,
TSDB_MIN_CACHE_BLOCK_SIZE, TSDB_MAX_CACHE_BLOCK_SIZE);
setErrMsg(pCmd, msg);
return TSDB_CODE_INVALID_SQL;
}
val = htonl(pCreate->maxSessions);
if (val != -1 && (val < TSDB_MIN_TABLES_PER_VNODE || val > TSDB_MAX_TABLES_PER_VNODE)) {
snprintf(msg, tListLen(msg), "invalid db option maxSessions: %d valid range: [%d, %d]", val, TSDB_MIN_TABLES_PER_VNODE,
TSDB_MAX_TABLES_PER_VNODE);
setErrMsg(pCmd, msg);
return TSDB_CODE_INVALID_SQL;
}
if (pCreate->precision != -1 &&
(pCreate->precision != TSDB_TIME_PRECISION_MILLI && pCreate->precision != TSDB_TIME_PRECISION_MICRO)) {
snprintf(msg, tListLen(msg), "invalid db option timePrecision: %d valid value: [%d, %d]", pCreate->precision, TSDB_TIME_PRECISION_MILLI,
TSDB_TIME_PRECISION_MICRO);
setErrMsg(pCmd, msg);
return TSDB_CODE_INVALID_SQL;
}
if (pCreate->cacheNumOfBlocks.fraction != -1 && (pCreate->cacheNumOfBlocks.fraction < TSDB_MIN_AVG_BLOCKS ||
pCreate->cacheNumOfBlocks.fraction > TSDB_MAX_AVG_BLOCKS)) {
snprintf(msg, tListLen(msg), "invalid db option ablocks: %f valid value: [%d, %d]", pCreate->cacheNumOfBlocks.fraction,
TSDB_MIN_AVG_BLOCKS, TSDB_MAX_AVG_BLOCKS);
setErrMsg(pCmd, msg);
return TSDB_CODE_INVALID_SQL;
}
val = htonl(pCreate->commitTime);
if (val != -1 && (val < TSDB_MIN_COMMIT_TIME_INTERVAL || val > TSDB_MAX_COMMIT_TIME_INTERVAL)) {
snprintf(msg, tListLen(msg), "invalid db option commitTime: %d valid range: [%d, %d]", val,
TSDB_MIN_COMMIT_TIME_INTERVAL, TSDB_MAX_COMMIT_TIME_INTERVAL);
setErrMsg(pCmd, msg);
return TSDB_CODE_INVALID_SQL;
}
if (pCreate->compression != -1 &&
(pCreate->compression < TSDB_MIN_COMPRESSION_LEVEL || pCreate->compression > TSDB_MAX_COMPRESSION_LEVEL)) {
snprintf(msg, tListLen(msg), "invalid db option compression: %d valid range: [%d, %d]", pCreate->compression, TSDB_MIN_COMPRESSION_LEVEL,
TSDB_MAX_COMPRESSION_LEVEL);
setErrMsg(pCmd, msg);
return TSDB_CODE_INVALID_SQL;
}
return TSDB_CODE_SUCCESS;
}

45
src/client/src/tscSQLParserImpl.c
View File

@ -14,8 +14,10 @@
*/
#include "os.h"
#include "taosmsg.h"
#include "tglobalcfg.h"
#include "tsql.h"
#include "tlog.h"
#include "tscSQLParser.h"
#include "tstoken.h"
#include "ttime.h"
#include "tutil.h"
@ -500,7 +502,7 @@ void tSQLSetColumnType(TAOS_FIELD *pField, SSQLToken *type) {
SQuerySQL *tSetQuerySQLElems(SSQLToken *pSelectToken, tSQLExprList *pSelection, tVariantList *pFrom, tSQLExpr *pWhere,
tVariantList *pGroupby, tVariantList *pSortOrder, SSQLToken *pInterval,
SSQLToken *pSliding, tVariantList *pFill, SLimitVal *pLimit, SLimitVal *pGLimit) {
assert(pSelection != NULL && pFrom != NULL && pInterval != NULL && pLimit != NULL && pGLimit != NULL);
assert(pSelection != NULL);
SQuerySQL *pQuery = calloc(1, sizeof(SQuerySQL));
pQuery->selectToken = *pSelectToken;
@ -512,13 +514,23 @@ SQuerySQL *tSetQuerySQLElems(SSQLToken *pSelectToken, tSQLExprList *pSelection,
pQuery->pSortOrder = pSortOrder;
pQuery->pWhere = pWhere;
pQuery->limit = *pLimit;
pQuery->slimit = *pGLimit;
if (pLimit != NULL) {
pQuery->limit = *pLimit;
}
if (pGLimit != NULL) {
pQuery->slimit = *pGLimit;
}
if (pInterval != NULL) {
pQuery->interval = *pInterval;
}
if (pSliding != NULL) {
pQuery->sliding = *pSliding;
}
pQuery->interval = *pInterval;
pQuery->sliding = *pSliding;
pQuery->fillType = pFill;
return pQuery;
}
@ -732,3 +744,22 @@ void setCreateAcctSQL(SSqlInfo *pInfo, int32_t type, SSQLToken *pName, SSQLToken
tTokenListAppend(pInfo->pDCLInfo, pPwd);
}
}
void setDefaultCreateDbOption(SCreateDBInfo *pDBInfo) {
pDBInfo->numOfBlocksPerTable = -1;
pDBInfo->compressionLevel = -1;
pDBInfo->commitLog = -1;
pDBInfo->commitTime = -1;
pDBInfo->tablesPerVnode = -1;
pDBInfo->numOfAvgCacheBlocks = -1;
pDBInfo->cacheBlockSize = -1;
pDBInfo->rowPerFileBlock = -1;
pDBInfo->daysPerFile = -1;
pDBInfo->replica = -1;
pDBInfo->keep = NULL;
memset(&pDBInfo->precision, 0, sizeof(SSQLToken));
}

1
src/client/src/tscSecondaryMerge.c
View File

@ -13,7 +13,6 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include "os.h"
#include "tlosertree.h"
#include "tscSecondaryMerge.h"

45
src/client/src/tscServer.c
View File

@ -24,7 +24,7 @@
#include "tsclient.h"
#include "tscompression.h"
#include "tsocket.h"
#include "tsql.h"
#include "tscSQLParser.h"
#include "ttime.h"
#include "ttimer.h"
#include "tutil.h"
@ -59,6 +59,22 @@ void tscPrintMgmtIp() {
}
#endif
/*
* For each management node, try twice at least in case of poor network situation.
* If the client start to connect to a non-management node from the client, and the first retry may fail due to
* the poor network quality. And then, the second retry get the response with redirection command.
* The retry will not be executed since only *two* retry is allowed in case of single management node in the cluster.
* Therefore, we need to multiply the retry times by factor of 2 to fix this problem.
*/
static int32_t tscGetMgmtConnMaxRetryTimes() {
int32_t factor = 2;
#ifdef CLUSTER
return tscMgmtIpList.numOfIps * factor;
#else
return 1*factor;
#endif
}
void tscProcessHeartBeatRsp(void *param, TAOS_RES *tres, int code) {
STscObj *pObj = (STscObj *)param;
if (pObj == NULL) return;
@ -134,18 +150,17 @@ void tscProcessActivityTimer(void *handle, void *tmrId) {
tscProcessSql(pObj->pHb);
}
//TODO HANDLE error from mgmt
void tscGetConnToMgmt(SSqlObj *pSql, uint8_t *pCode) {
STscObj *pTscObj = pSql->pTscObj;
#ifdef CLUSTER
if (pSql->retry < tscMgmtIpList.numOfIps) {
if (pSql->retry < tscGetMgmtConnMaxRetryTimes()) {
*pCode = 0;
pSql->retry++;
pSql->index = pSql->index % tscMgmtIpList.numOfIps;
if (pSql->cmd.command > TSDB_SQL_READ && pSql->index == 0) pSql->index = 1;
void *thandle = taosGetConnFromCache(tscConnCache, tscMgmtIpList.ip[pSql->index], TSC_MGMT_VNODE, pTscObj->user);
#else
if (pSql->retry < 1) {
if (pSql->retry < tscGetMgmtConnMaxRetryTimes()) {
*pCode = 0;
pSql->retry++;
void *thandle = taosGetConnFromCache(tscConnCache, tsServerIp, TSC_MGMT_VNODE, pTscObj->user);
@ -184,6 +199,15 @@ void tscGetConnToMgmt(SSqlObj *pSql, uint8_t *pCode) {
pSql->vnode = TSC_MGMT_VNODE;
#endif
}
// the pSql->res.code is the previous error(status) code.
if (pSql->thandle == NULL && pSql->retry >= pSql->maxRetry) {
if (pSql->res.code != TSDB_CODE_SUCCESS && pSql->res.code != TSDB_CODE_ACTION_IN_PROGRESS) {
*pCode = pSql->res.code;
}
tscError("%p reach the max retry:%d, code:%d", pSql, pSql->retry, *pCode);
}
}
void tscGetConnToVnode(SSqlObj *pSql, uint8_t *pCode) {
@ -435,8 +459,8 @@ void *tscProcessMsgFromServer(char *msg, void *ahandle, void *thandle) {
}
} else {
uint16_t rspCode = pMsg->content[0];
#ifdef CLUSTER
#ifdef CLUSTER
if (rspCode == TSDB_CODE_REDIRECT) {
tscTrace("%p it shall be redirected!", pSql);
@ -1795,7 +1819,7 @@ int tscBuildCreateDbMsg(SSqlObj *pSql) {
pMsg += sizeof(SMgmtHead);
pCreateDbMsg = (SCreateDbMsg *)pMsg;
strcpy(pCreateDbMsg->db, pMeterMetaInfo->name);
strncpy(pCreateDbMsg->db, pMeterMetaInfo->name, tListLen(pCreateDbMsg->db));
pMsg += sizeof(SCreateDbMsg);
msgLen = pMsg - pStart;
@ -2017,7 +2041,7 @@ int tscBuildDropDbMsg(SSqlObj *pSql) {
pMsg += sizeof(SMgmtHead);
pDropDbMsg = (SDropDbMsg *)pMsg;
strcpy(pDropDbMsg->db, pMeterMetaInfo->name);
strncpy(pDropDbMsg->db, pMeterMetaInfo->name, tListLen(pDropDbMsg->db));
pDropDbMsg->ignoreNotExists = htons(pCmd->existsCheck ? 1 : 0);
@ -3795,7 +3819,14 @@ void tscInitMsgs() {
tscProcessMsgRsp[TSDB_SQL_SHOW] = tscProcessShowRsp;
tscProcessMsgRsp[TSDB_SQL_RETRIEVE] = tscProcessRetrieveRspFromVnode; // rsp handled by same function.
tscProcessMsgRsp[TSDB_SQL_DESCRIBE_TABLE] = tscProcessDescribeTableRsp;
tscProcessMsgRsp[TSDB_SQL_RETRIEVE_TAGS] = tscProcessTagRetrieveRsp;
tscProcessMsgRsp[TSDB_SQL_CURRENT_DB] = tscProcessTagRetrieveRsp;
tscProcessMsgRsp[TSDB_SQL_CURRENT_USER] = tscProcessTagRetrieveRsp;
tscProcessMsgRsp[TSDB_SQL_SERV_VERSION] = tscProcessTagRetrieveRsp;
tscProcessMsgRsp[TSDB_SQL_CLI_VERSION] = tscProcessTagRetrieveRsp;
tscProcessMsgRsp[TSDB_SQL_SERV_STATUS] = tscProcessTagRetrieveRsp;
tscProcessMsgRsp[TSDB_SQL_RETRIEVE_EMPTY_RESULT] = tscProcessEmptyResultRsp;
tscProcessMsgRsp[TSDB_SQL_RETRIEVE_METRIC] = tscProcessRetrieveMetricRsp;

17
src/client/src/tscSql.c
View File

@ -24,9 +24,10 @@
#include "tsclient.h"
#include "tscompression.h"
#include "tsocket.h"
#include "tsql.h"
#include "tscSQLParser.h"
#include "ttimer.h"
#include "tutil.h"
#include "tnote.h"
TAOS *taos_connect_imp(const char *ip, const char *user, const char *pass, const char *db, uint16_t port, void (*fp)(void *, TAOS_RES *, int),
void *param, void **taos) {
@ -63,12 +64,8 @@ TAOS *taos_connect_imp(const char *ip, const char *user, const char *pass, const
#ifdef CLUSTER
if (ip && ip[0]) {
tscMgmtIpList.numOfIps = 2;
strcpy(tscMgmtIpList.ipstr[0], ip);
tscMgmtIpList.ip[0] = inet_addr(ip);
strcpy(tscMgmtIpList.ipstr[1], ip);
tscMgmtIpList.ip[1] = inet_addr(ip);
}
#else
if (ip && ip[0]) {
@ -156,7 +153,7 @@ TAOS *taos_connect_imp(const char *ip, const char *user, const char *pass, const
TAOS *taos_connect(const char *ip, const char *user, const char *pass, const char *db, uint16_t port) {
if (ip == NULL || (ip != NULL && (strcmp("127.0.0.1", ip) == 0 || strcasecmp("localhost", ip) == 0))) {
#ifdef CLUSTER
ip = tsPrivateIp;
ip = tsMasterIp;
#else
ip = tsServerIpStr;
#endif
@ -245,9 +242,9 @@ int taos_query_imp(STscObj* pObj, SSqlObj* pSql) {
pRes->qhandle = 0;
pSql->thandle = NULL;
if (pRes->code != TSDB_CODE_SUCCESS) return pRes->code;
tscDoQuery(pSql);
if (pRes->code == TSDB_CODE_SUCCESS) {
tscDoQuery(pSql);
}
tscTrace("%p SQL result:%d, %s pObj:%p", pSql, pRes->code, taos_errstr(pObj), pObj);
if (pRes->code != TSDB_CODE_SUCCESS) {
@ -274,6 +271,8 @@ int taos_query(TAOS *taos, const char *sqlstr) {
return pRes->code;
}
taosNotePrintTsc(sqlstr);
void *sql = realloc(pSql->sqlstr, sqlLen + 1);
if (sql == NULL) {
pRes->code = TSDB_CODE_CLI_OUT_OF_MEMORY;

2
src/client/src/tscStream.c
View File

@ -15,7 +15,7 @@
#include "os.h"
#include "tlog.h"
#include "tsql.h"
#include "tscSQLParser.h"
#include "ttime.h"
#include "ttimer.h"
#include "tutil.h"

2
src/client/src/tscSyntaxtreefunction.c
View File

@ -16,7 +16,7 @@
#include "os.h"
#include "tscSyntaxtreefunction.h"
#include "tsql.h"
#include "tscSQLParser.h"
#include "ttypes.h"
#include "tutil.h"

8
src/client/src/tscSystem.c
View File

@ -45,6 +45,10 @@ extern int tscEmbedded;
int tscNumOfThreads;
static pthread_once_t tscinit = PTHREAD_ONCE_INIT;
extern int tsTscEnableRecordSql;
extern int tsNumOfLogLines;
void taosInitNote(int numOfNoteLines, int maxNotes, char* lable);
void tscCheckDiskUsage(void *para, void *unused) {
taosGetDisk();
taosTmrReset(tscCheckDiskUsage, 1000, NULL, tscTmr, &tscCheckDiskUsageTmr);
@ -85,6 +89,10 @@ void taos_init_imp() {
taosSetCoreDump();
if (tsTscEnableRecordSql != 0) {
taosInitNote(tsNumOfLogLines / 10, 1, (char*)"tsc_note");
}
#ifdef CLUSTER
tscMgmtIpList.numOfIps = 2;
strcpy(tscMgmtIpList.ipstr[0], tsMasterIp);

17
src/client/src/tscUtil.c
View File

@ -720,7 +720,7 @@ static void evic(SFieldInfo* pFieldInfo, int32_t index) {
}
}
static void setValueImpl(TAOS_FIELD* pField, int8_t type, char* name, int16_t bytes) {
static void setValueImpl(TAOS_FIELD* pField, int8_t type, const char* name, int16_t bytes) {
pField->type = type;
strncpy(pField->name, name, TSDB_COL_NAME_LEN);
pField->bytes = bytes;
@ -764,7 +764,7 @@ void tscFieldInfoUpdateVisible(SFieldInfo* pFieldInfo, int32_t index, bool visib
}
}
void tscFieldInfoSetValue(SFieldInfo* pFieldInfo, int32_t index, int8_t type, char* name, int16_t bytes) {
void tscFieldInfoSetValue(SFieldInfo* pFieldInfo, int32_t index, int8_t type, const char* name, int16_t bytes) {
ensureSpace(pFieldInfo, pFieldInfo->numOfOutputCols + 1);
evic(pFieldInfo, index);
@ -896,6 +896,19 @@ static void _exprEvic(SSqlExprInfo* pExprInfo, int32_t index) {
}
}
SSqlExpr* tscSqlExprInsertEmpty(SSqlCmd* pCmd, int32_t index, int16_t functionId) {
SSqlExprInfo* pExprInfo = &pCmd->exprsInfo;
_exprCheckSpace(pExprInfo, pExprInfo->numOfExprs + 1);
_exprEvic(pExprInfo, index);
SSqlExpr* pExpr = &pExprInfo->pExprs[index];
pExpr->functionId = functionId;
pExprInfo->numOfExprs++;
return pExpr;
}
SSqlExpr* tscSqlExprInsert(SSqlCmd* pCmd, int32_t index, int16_t functionId, SColumnIndex* pColIndex, int16_t type,
int16_t size, int16_t interSize) {
SMeterMetaInfo* pMeterMetaInfo = tscGetMeterMetaInfo(pCmd, pColIndex->tableIndex);

15
src/connector/go/src/taosSql/rows.go
View File

@ -118,14 +118,15 @@ func (rows *taosSqlRows) ColumnTypeScanType(i int) reflect.Type {
return rows.rs.columns[i].scanType()
}
func (rows *taosSqlRows) Close() (err error) {
mc := rows.mc
if mc == nil {
return nil
func (rows *taosSqlRows) Close() error {
if rows.mc != nil {
result := C.taos_use_result(rows.mc.taos)
if result != nil {
C.taos_free_result(result)
}
rows.mc = nil
}
rows.mc = nil
return err
return nil
}
func (rows *taosSqlRows) HasNextResultSet() (b bool) {

2
src/connector/go/src/taosSql/taosSqlCgo.go
View File

@ -39,7 +39,7 @@ func (mc *taosConn) taosConnect(ip, user, pass, db string, port int) (taos unsaf
defer C.free(unsafe.Pointer(cpass))
defer C.free(unsafe.Pointer(cdb))
taosObj := C.taos_connect(cip, cuser, cpass, cdb, (C.int)(port))
taosObj := C.taos_connect(cip, cuser, cpass, cdb, (C.ushort)(port))
if taosObj == nil {
return nil, errors.New("taos_connect() fail!")
}

36
src/inc/sql.y
View File

@ -26,8 +26,7 @@
#include <string.h>
#include <assert.h>
#include <stdbool.h>
#include "tsql.h"
#include "tscSQLParser.h"
#include "tutil.h"
}
@ -113,7 +112,7 @@ cmd ::= DROP TABLE ifexists(Y) ids(X) cpxName(Z). {
}
cmd ::= DROP DATABASE ifexists(Y) ids(X). { setDCLSQLElems(pInfo, DROP_DATABASE, 2, &X, &Y); }
cmd ::= DROP DNODE IP(X). { setDCLSQLElems(pInfo, DROP_DNODE, 1, &X); }
cmd ::= DROP DNODE IPTOKEN(X). { setDCLSQLElems(pInfo, DROP_DNODE, 1, &X); }
cmd ::= DROP USER ids(X). { setDCLSQLElems(pInfo, DROP_USER, 1, &X); }
cmd ::= DROP ACCOUNT ids(X). { setDCLSQLElems(pInfo, DROP_ACCOUNT, 1, &X); }
@ -129,8 +128,8 @@ cmd ::= DESCRIBE ids(X) cpxName(Y). {
/////////////////////////////////THE ALTER STATEMENT////////////////////////////////////////
cmd ::= ALTER USER ids(X) PASS ids(Y). { setDCLSQLElems(pInfo, ALTER_USER_PASSWD, 2, &X, &Y); }
cmd ::= ALTER USER ids(X) PRIVILEGE ids(Y). { setDCLSQLElems(pInfo, ALTER_USER_PRIVILEGES, 2, &X, &Y);}
cmd ::= ALTER DNODE IP(X) ids(Y). { setDCLSQLElems(pInfo, ALTER_DNODE, 2, &X, &Y); }
cmd ::= ALTER DNODE IP(X) ids(Y) ids(Z). { setDCLSQLElems(pInfo, ALTER_DNODE, 3, &X, &Y, &Z); }
cmd ::= ALTER DNODE IPTOKEN(X) ids(Y). { setDCLSQLElems(pInfo, ALTER_DNODE, 2, &X, &Y); }
cmd ::= ALTER DNODE IPTOKEN(X) ids(Y) ids(Z). { setDCLSQLElems(pInfo, ALTER_DNODE, 3, &X, &Y, &Z); }
cmd ::= ALTER LOCAL ids(X). { setDCLSQLElems(pInfo, ALTER_LOCAL, 1, &X); }
cmd ::= ALTER LOCAL ids(X) ids(Y). { setDCLSQLElems(pInfo, ALTER_LOCAL, 2, &X, &Y); }
cmd ::= ALTER DATABASE ids(X) alter_db_optr(Y). { SSQLToken t = {0}; setCreateDBSQL(pInfo, ALTER_DATABASE, &X, &Y, &t);}
@ -155,7 +154,7 @@ ifnotexists(X) ::= . {X.n = 0;}
/////////////////////////////////THE CREATE STATEMENT///////////////////////////////////////
//create option for dnode/db/user/account
cmd ::= CREATE DNODE IP(X). { setDCLSQLElems(pInfo, CREATE_DNODE, 1, &X);}
cmd ::= CREATE DNODE IPTOKEN(X). { setDCLSQLElems(pInfo, CREATE_DNODE, 1, &X);}
cmd ::= CREATE ACCOUNT ids(X) PASS ids(Y) acct_optr(Z).
{ setCreateAcctSQL(pInfo, CREATE_ACCOUNT, &X, &Y, &Z);}
cmd ::= CREATE DATABASE ifnotexists(Z) ids(X) db_optr(Y). { setCreateDBSQL(pInfo, CREATE_DATABASE, &X, &Y, &Z);}
@ -219,7 +218,8 @@ comp(Y) ::= COMP INTEGER(X). { Y = X; }
prec(Y) ::= PRECISION STRING(X). { Y = X; }
%type db_optr {SCreateDBInfo}
db_optr ::= . {}
db_optr(Y) ::= . {setDefaultCreateDbOption(&Y);}
db_optr(Y) ::= db_optr(Z) tables(X). { Y = Z; Y.tablesPerVnode = strtol(X.z, NULL, 10); }
db_optr(Y) ::= db_optr(Z) cache(X). { Y = Z; Y.cacheBlockSize = strtol(X.z, NULL, 10); }
db_optr(Y) ::= db_optr(Z) replica(X). { Y = Z; Y.replica = strtol(X.z, NULL, 10); }
@ -234,7 +234,7 @@ db_optr(Y) ::= db_optr(Z) prec(X). { Y = Z; Y.precision = X; }
db_optr(Y) ::= db_optr(Z) keep(X). { Y = Z; Y.keep = X; }
%type alter_db_optr {SCreateDBInfo}
alter_db_optr(Y) ::= . { memset(&Y, 0, sizeof(SCreateDBInfo));}
alter_db_optr(Y) ::= . { setDefaultCreateDbOption(&Y);}
alter_db_optr(Y) ::= alter_db_optr(Z) replica(X). { Y = Z; Y.replica = strtol(X.z, NULL, 10); }
alter_db_optr(Y) ::= alter_db_optr(Z) tables(X). { Y = Z; Y.tablesPerVnode = strtol(X.z, NULL, 10); }
@ -350,6 +350,14 @@ select(A) ::= SELECT(T) selcollist(W) from(X) where_opt(Y) interval_opt(K) fill_
A = tSetQuerySQLElems(&T, W, X, Y, P, Z, &K, &S, F, &L, &G);
}
// Support for the SQL exprssion without from & where subclauses, e.g.,
// select current_database(),
// select server_version(), select client_version(),
// select server_state();
select(A) ::= SELECT(T) selcollist(W). {
A = tSetQuerySQLElems(&T, W, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
}
// selcollist is a list of expressions that are to become the return
// values of the SELECT statement. The "*" in statements like
// "SELECT * FROM ..." is encoded as a special expression with an opcode of TK_ALL.
@ -392,7 +400,7 @@ tmvar(A) ::= VARIABLE(X). {A = X;}
%type interval_opt {SSQLToken}
interval_opt(N) ::= INTERVAL LP tmvar(E) RP. {N = E; }
interval_opt(N) ::= . {N.n = 0; }
interval_opt(N) ::= . {N.n = 0; N.z = NULL; N.type = 0; }
%type fill_opt {tVariantList*}
%destructor fill_opt {tVariantListDestroy($$);}
@ -413,7 +421,7 @@ fill_opt(N) ::= FILL LP ID(Y) RP. {
%type sliding_opt {SSQLToken}
sliding_opt(K) ::= SLIDING LP tmvar(E) RP. {K = E; }
sliding_opt(K) ::= . {K.n = 0; }
sliding_opt(K) ::= . {K.n = 0; K.z = NULL; K.type = 0; }
%type orderby_opt {tVariantList*}
%destructor orderby_opt {tVariantListDestroy($$);}
@ -642,12 +650,12 @@ cmd ::= ALTER TABLE ids(X) cpxName(F) SET TAG ids(Y) EQ tagitem(Z). {
}
////////////////////////////////////////kill statement///////////////////////////////////////
cmd ::= KILL CONNECTION IP(X) COLON(Z) INTEGER(Y). {X.n += (Z.n + Y.n); setDCLSQLElems(pInfo, KILL_CONNECTION, 1, &X);}
cmd ::= KILL STREAM IP(X) COLON(Z) INTEGER(Y) COLON(K) INTEGER(F). {X.n += (Z.n + Y.n + K.n + F.n); setDCLSQLElems(pInfo, KILL_STREAM, 1, &X);}
cmd ::= KILL QUERY IP(X) COLON(Z) INTEGER(Y) COLON(K) INTEGER(F). {X.n += (Z.n + Y.n + K.n + F.n); setDCLSQLElems(pInfo, KILL_QUERY, 1, &X);}
cmd ::= KILL CONNECTION IPTOKEN(X) COLON(Z) INTEGER(Y). {X.n += (Z.n + Y.n); setDCLSQLElems(pInfo, KILL_CONNECTION, 1, &X);}
cmd ::= KILL STREAM IPTOKEN(X) COLON(Z) INTEGER(Y) COLON(K) INTEGER(F). {X.n += (Z.n + Y.n + K.n + F.n); setDCLSQLElems(pInfo, KILL_STREAM, 1, &X);}
cmd ::= KILL QUERY IPTOKEN(X) COLON(Z) INTEGER(Y) COLON(K) INTEGER(F). {X.n += (Z.n + Y.n + K.n + F.n); setDCLSQLElems(pInfo, KILL_QUERY, 1, &X);}
%fallback ID ABORT AFTER ASC ATTACH BEFORE BEGIN CASCADE CLUSTER CONFLICT COPY DATABASE DEFERRED
DELIMITERS DESC DETACH EACH END EXPLAIN FAIL FOR GLOB IGNORE IMMEDIATE INITIALLY INSTEAD
LIKE MATCH KEY OF OFFSET RAISE REPLACE RESTRICT ROW STATEMENT TRIGGER VIEW ALL
COUNT SUM AVG MIN MAX FIRST LAST TOP BOTTOM STDDEV PERCENTILE APERCENTILE LEASTSQUARES HISTOGRAM DIFF
SPREAD TWA INTERP LAST_ROW NOW IP SEMI NONE PREV LINEAR IMPORT METRIC TBNAME JOIN METRICS STABLE.
SPREAD TWA INTERP LAST_ROW NOW IPTOKEN SEMI NONE PREV LINEAR IMPORT METRIC TBNAME JOIN METRICS STABLE NULL.

9
src/inc/taosmsg.h
View File

@ -309,7 +309,7 @@ typedef struct {
} SCreateMsg;
typedef struct {
char db[TSDB_DB_NAME_LEN];
char db[TSDB_METER_ID_LEN];
short ignoreNotExists;
} SDropDbMsg, SUseDbMsg;
@ -583,7 +583,7 @@ typedef struct {
int64_t compStorage;
int64_t pointsWritten;
uint8_t syncStatus;
uint8_t reserved;
uint8_t reserved[15];
} SVnodeLoad;
typedef struct {
@ -594,6 +594,11 @@ typedef struct {
// NOTE: sizeof(SVnodeCfg) < TSDB_FILE_HEADER_LEN/4
typedef struct {
char acct[TSDB_USER_LEN];
/*
* the message is too large, so it may will overwrite the cfg information in meterobj.v*
* recover to origin codes
*/
//char db[TSDB_METER_ID_LEN+2]; // 8bytes align
char db[TSDB_DB_NAME_LEN];
uint32_t vgId;
int32_t maxSessions;

4
src/inc/tast.h
View File

@ -25,7 +25,7 @@ extern "C" {
#include <stdlib.h>
#include "taosmsg.h"
#include "tsql.h"
#include "ttypes.h"
struct tSQLBinaryExpr;
struct SSchema;
@ -49,7 +49,7 @@ typedef struct tQueryInfo {
int32_t offset; // offset value in tags
int32_t colIdx; // index of column in schema
uint8_t optr; // expression operator
SSchema sch; // schema of tags
SSchema sch; // schema of tags
tVariant q; // query condition value on the specific schema, filter expression
__compar_fn_t compare; // filter function
} tQueryInfo;

2
src/inc/tglobalcfg.h
View File

@ -150,6 +150,8 @@ extern int tsHttpEnableRecordSql;
extern int tsTelegrafUseFieldNum;
extern int tsAdminRowLimit;
extern int tsTscEnableRecordSql;
extern char tsMonitorDbName[];
extern char tsInternalPass[];
extern int tsMonitorInterval;

64
src/inc/tnote.h Normal file
View File

@ -0,0 +1,64 @@
/*
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
*
* This program is free software: you can use, redistribute, and/or modify
* it under the terms of the GNU Affero General Public License, version 3
* or later ("AGPL"), as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef TDENGINE_TNOTE_H
#define TDENGINE_TNOTE_H
#ifdef __cplusplus
extern "C" {
#endif
#include "unistd.h"
#include "os.h"
#include "tutil.h"
#include "tglobalcfg.h"
#define MAX_NOTE_LINE_SIZE 66000
#define NOTE_FILE_NAME_LEN 300
typedef struct _taosNoteInfo {
int taosNoteFileNum ;
int taosNoteMaxLines;
int taosNoteLines;
char taosNoteName[NOTE_FILE_NAME_LEN];
int taosNoteFlag;
int taosNoteFd;
int taosNoteOpenInProgress;
pthread_mutex_t taosNoteMutex;
}taosNoteInfo;
void taosNotePrint(taosNoteInfo * pNote, const char * const format, ...);
extern taosNoteInfo m_HttpNote;
extern taosNoteInfo m_TscNote;
extern int tsHttpEnableRecordSql;
extern int tsTscEnableRecordSql;
#define taosNotePrintHttp(...) \
if (tsHttpEnableRecordSql) { \
taosNotePrint(&m_HttpNote, __VA_ARGS__); \
}
#define taosNotePrintTsc(...) \
if (tsTscEnableRecordSql) { \
taosNotePrint(&m_TscNote, __VA_ARGS__); \
}
#ifdef __cplusplus
}
#endif
#endif

46
src/inc/tsdb.h
View File

@ -44,40 +44,6 @@ extern "C" {
#define TSDB_TIME_PRECISION_MILLI_STR "ms"
#define TSDB_TIME_PRECISION_MICRO_STR "us"
enum _vnode_status {
TSDB_VNODE_STATUS_OFFLINE,
TSDB_VNODE_STATUS_CREATING,
TSDB_VNODE_STATUS_UNSYNCED,
TSDB_VNODE_STATUS_SLAVE,
TSDB_VNODE_STATUS_MASTER,
TSDB_VNODE_STATUS_CLOSING,
TSDB_VNODE_STATUS_DELETING,
};
enum _vnode_sync_status {
STDB_SSTATUS_INIT,
TSDB_SSTATUS_SYNCING,
TSDB_SSTATUS_SYNC_CACHE,
TSDB_SSTATUS_SYNC_FILE,
};
enum _dnode_status {
TSDB_DNODE_STATUS_OFFLINE,
TSDB_DNODE_STATUS_READY
};
enum _dnode_balance_status {
LB_DNODE_STATE_BALANCED,
LB_DNODE_STATE_BALANCING,
LB_DNODE_STATE_OFFLINE_REMOVING,
LB_DNODE_STATE_SHELL_REMOVING
};
enum _vgroup_status {
LB_VGROUP_STATE_READY,
LB_VGROUP_STATE_UPDATE
};
#define TSDB_DATA_TYPE_BOOL 1 // 1 bytes
#define TSDB_DATA_TYPE_TINYINT 2 // 1 byte
#define TSDB_DATA_TYPE_SMALLINT 3 // 2 bytes
@ -180,8 +146,8 @@ enum _vgroup_status {
#define TSDB_MAX_MPEERS 5
#define TSDB_MAX_MGMT_IPS (TSDB_MAX_MPEERS+1)
//#define TSDB_REPLICA_MAX_NUM 3
#define TSDB_REPLICA_MIN_NUM 1
#define TSDB_REPLICA_MAX_NUM 3
#define TSDB_TBNAME_COLUMN_INDEX (-1)
#define TSDB_MULTI_METERMETA_MAX_NUM 100000 // maximum batch size allowed to load metermeta
@ -196,9 +162,6 @@ enum _vgroup_status {
#define TSDB_MIN_COMPRESSION_LEVEL 0
#define TSDB_MAX_COMPRESSION_LEVEL 2
#define TSDB_MIN_CACHE_BLOCKS_PER_METER 32
#define TSDB_MAX_CACHE_BLOCKS_PER_METER 40960
#define TSDB_MIN_COMMIT_TIME_INTERVAL 30
#define TSDB_MAX_COMMIT_TIME_INTERVAL 40960
@ -211,7 +174,9 @@ enum _vgroup_status {
#define TSDB_MIN_CACHE_BLOCKS 100
#define TSDB_MAX_CACHE_BLOCKS 409600
#define TSDB_MIN_AVG_BLOCKS 2
#define TSDB_MAX_AVG_BLOCKS 2048
#define TSDB_DEFAULT_AVG_BLOCKS 4
#define TSDB_MIN_TABLES_PER_VNODE 1
#define TSDB_MAX_TABLES_PER_VNODE 220000
@ -238,7 +203,7 @@ enum _vgroup_status {
#define TSDB_MAX_RPC_THREADS 5
#define TSDB_QUERY_TYPE_QUERY 0 // normal query
#define TSDB_QUERY_TYPE_QUERY 0 // normal query
#define TSDB_QUERY_TYPE_FREE_RESOURCE 0x01U // free qhandle at vnode
/*
@ -254,6 +219,9 @@ enum _vgroup_status {
#define TSDB_QUERY_TYPE_PROJECTION_QUERY 0x40U // select *,columns... query
#define TSDB_QUERY_TYPE_JOIN_SEC_STAGE 0x80U // join sub query at the second stage
#define TSQL_SO_ASC 1
#define TSQL_SO_DESC 0
#ifdef __cplusplus
}
#endif

3
src/inc/tsqldef.h
View File

@ -80,7 +80,7 @@
#define TK_TABLE 62
#define TK_DATABASE 63
#define TK_DNODE 64
#define TK_IP 65
#define TK_IPTOKEN 65
#define TK_USER 66
#define TK_ACCOUNT 67
#define TK_USE 68
@ -210,7 +210,6 @@
#define TK_JOIN 192
#define TK_METRICS 193
#define TK_STABLE 194
#define TK_QUESTION 195
#endif

1
src/inc/tsqlfunction.h
View File

@ -24,7 +24,6 @@ extern "C" {
#include <stdint.h>
#include "trpc.h"
#include "tsql.h"
#include "ttypes.h"
#define TSDB_FUNC_INVALID_ID -1

66
src/inc/tstatus.h
View File

@ -20,11 +20,73 @@
extern "C" {
#endif
enum _TSDB_VG_STATUS {
TSDB_VG_STATUS_READY,
TSDB_VG_STATUS_IN_PROGRESS,
TSDB_VG_STATUS_COMMITLOG_INIT_FAILED,
TSDB_VG_STATUS_INIT_FAILED,
TSDB_VG_STATUS_FULL
};
enum _TSDB_DB_STATUS {
TSDB_DB_STATUS_READY,
TSDB_DB_STATUS_DROPPING,
TSDB_DB_STATUS_DROP_FROM_SDB
};
enum _TSDB_VN_STATUS {
TSDB_VN_STATUS_OFFLINE,
TSDB_VN_STATUS_CREATING,
TSDB_VN_STATUS_UNSYNCED,
TSDB_VN_STATUS_SLAVE,
TSDB_VN_STATUS_MASTER,
TSDB_VN_STATUS_CLOSING,
TSDB_VN_STATUS_DELETING,
};
enum _TSDB_VN_SYNC_STATUS {
TSDB_VN_SYNC_STATUS_INIT,
TSDB_VN_SYNC_STATUS_SYNCING,
TSDB_VN_SYNC_STATUS_SYNC_CACHE,
TSDB_VN_SYNC_STATUS_SYNC_FILE
};
enum _TSDB_VN_DROP_STATUS {
TSDB_VN_DROP_STATUS_READY,
TSDB_VN_DROP_STATUS_DROPPING
};
enum _TSDB_DN_STATUS {
TSDB_DN_STATUS_OFFLINE,
TSDB_DN_STATUS_READY
};
enum _TSDB_DN_LB_STATUS {
TSDB_DN_LB_STATUS_BALANCED,
TSDB_DN_LB_STATUS_BALANCING,
TSDB_DN_LB_STATUS_OFFLINE_REMOVING,
TSDB_DN_LB_STATE_SHELL_REMOVING
};
enum _TSDB_VG_LB_STATUS {
TSDB_VG_LB_STATUS_READY,
TSDB_VG_LB_STATUS_UPDATE
};
enum _TSDB_VN_STREAM_STATUS {
TSDB_VN_STREAM_STATUS_STOP,
TSDB_VN_STREAM_STATUS_START
};
const char* taosGetVgroupStatusStr(int vgroupStatus);
const char* taosGetDbStatusStr(int dbStatus);
const char* taosGetVnodeStatusStr(int vnodeStatus);
const char* taosGetDnodeStatusStr(int dnodeStatus);
const char* taosGetDnodeBalanceStateStr(int dnodeBalanceStatus);
const char* taosGetVnodeSyncStatusStr(int vnodeSyncStatus);
const char* taosGetVnodeDropStatusStr(int dropping);
const char* taosGetDnodeStatusStr(int dnodeStatus);
const char* taosGetDnodeLbStatusStr(int dnodeBalanceStatus);
const char* taosGetVgroupLbStatusStr(int vglbStatus);
const char* taosGetVnodeStreamStatusStr(int vnodeStreamStatus);
#ifdef __cplusplus
}

16
src/inc/tstoken.h
View File

@ -22,6 +22,18 @@ extern "C" {
#include <stdbool.h>
#define TK_SPACE 200
#define TK_COMMENT 201
#define TK_ILLEGAL 202
#define TK_HEX 203 // hex number 0x123
#define TK_OCT 204 // oct number
#define TK_BIN 205 // bin format data 0b111
#define TK_FILE 206
#define TK_QUESTION 207 // denoting the placeholder of "?",when invoking statement bind query
#define TSQL_TBNAME "TBNAME"
#define TSQL_TBNAME_L "tbname"
// used to denote the minimum unite in sql parsing
typedef struct SSQLToken {
uint32_t n;
@ -29,10 +41,6 @@ typedef struct SSQLToken {
char * z;
} SSQLToken;
#if 0
char *tscGetToken(char *string, char **token, int *tokenLen);
#endif
/**
* tokenizer for sql string
* @param z

2
src/modules/http/inc/httpHandle.h
View File

@ -63,7 +63,7 @@
#define HTTP_WRITE_RETRY_TIMES 500
#define HTTP_WRITE_WAIT_TIME_MS 5
#define HTTP_EXPIRED_TIME 60000
#define HTTP_DELAY_CLOSE_TIME_MS 1000
#define HTTP_DELAY_CLOSE_TIME_MS 500
#define HTTP_COMPRESS_IDENTITY 0
#define HTTP_COMPRESS_GZIP 2

10
src/modules/http/src/httpServer.c
View File

@ -107,9 +107,9 @@ void httpCleanUpContextTimer(HttpContext *pContext) {
}
void httpCleanUpContext(HttpContext *pContext) {
httpTrace("context:%p, start the clean up operation", pContext);
atomic_val_compare_exchange_ptr(&pContext->signature, pContext, 0);
if (pContext->signature != NULL) {
httpTrace("context:%p, start the clean up operation, sig:%p", pContext, pContext->signature);
void *sig = atomic_val_compare_exchange_ptr(&pContext->signature, pContext, 0);
if (sig == NULL) {
httpTrace("context:%p is freed by another thread.", pContext);
return;
}
@ -527,8 +527,8 @@ void httpAcceptHttpConnection(void *arg) {
totalFds += pServer->pThreads[i].numOfFds;
}
if (totalFds > tsHttpCacheSessions * 20) {
httpError("fd:%d, ip:%s:%u, totalFds:%d larger than httpCacheSessions:%d*20, refuse connection",
if (totalFds > tsHttpCacheSessions * 100) {
httpError("fd:%d, ip:%s:%u, totalFds:%d larger than httpCacheSessions:%d*100, refuse connection",
connFd, inet_ntoa(clientAddr.sin_addr), htons(clientAddr.sin_port), totalFds, tsHttpCacheSessions);
taosCloseSocket(connFd);
continue;

6
src/modules/http/src/httpSql.c
View File

@ -24,11 +24,11 @@
#include "httpResp.h"
#include "taos.h"
#include "tsclient.h"
#include "tnote.h"
void *taos_connect_a(char *ip, char *user, char *pass, char *db, uint16_t port, void (*fp)(void *, TAOS_RES *, int),
void *param, void **taos);
void httpProcessMultiSql(HttpContext *pContext);
void taosNotePrint(const char * const format, ...);
void httpProcessMultiSqlRetrieveCallBack(void *param, TAOS_RES *result, int numOfRows) {
HttpContext *pContext = (HttpContext *)param;
@ -165,7 +165,7 @@ void httpProcessMultiSql(HttpContext *pContext) {
char *sql = httpGetCmdsString(pContext, cmd->sql);
httpDump("context:%p, fd:%d, ip:%s, user:%s, process pos:%d, start query, sql:%s", pContext, pContext->fd,
pContext->ipstr, pContext->user, multiCmds->pos, sql);
taosNotePrint(sql);
taosNotePrintHttp(sql);
taos_query_a(pContext->session->taos, sql, httpProcessMultiSqlCallBack, (void *)pContext);
}
@ -298,7 +298,7 @@ void httpProcessSingleSqlCmd(HttpContext *pContext) {
httpDump("context:%p, fd:%d, ip:%s, user:%s, start query, sql:%s", pContext, pContext->fd, pContext->ipstr,
pContext->user, sql);
taosNotePrint(sql);
taosNotePrintHttp(sql);
taos_query_a(pSession->taos, sql, httpProcessSingleSqlCallBack, (void *)pContext);
}

6
src/modules/http/src/httpSystem.c
View File

@ -42,7 +42,7 @@
#endif
static HttpServer *httpServer = NULL;
void taosInitNote(int numOfNoteLines, int maxNotes);
void taosInitNote(int numOfNoteLines, int maxNotes, char* lable);
int httpInitSystem() {
taos_init();
@ -61,7 +61,7 @@ int httpInitSystem() {
pthread_mutex_init(&httpServer->serverMutex, NULL);
if (tsHttpEnableRecordSql != 0) {
taosInitNote(tsNumOfLogLines / 10, 1);
taosInitNote(tsNumOfLogLines / 10, 1, (char*)"http_note");
}
restInitHandle(httpServer);
adminInitHandle(httpServer);
@ -89,7 +89,7 @@ int httpStartSystem() {
}
if (httpServer->timerHandle == NULL) {
httpServer->timerHandle = taosTmrInit(tsHttpCacheSessions * 20 + 100, 1000, 60000, "http");
httpServer->timerHandle = taosTmrInit(tsHttpCacheSessions * 100 + 100, 200, 60000, "http");
}
if (httpServer->timerHandle == NULL) {
httpError("http init timer failed");

4
src/modules/monitor/src/monitorSystem.c
View File

@ -246,9 +246,9 @@ void monitorStopSystem() {
}
#ifdef CLUSTER
monitorLPrint("dnode:%s is stopped", tsPrivateIp);
monitorLPrint("dnode:%s monitor module is stopped", tsPrivateIp);
#else
monitorLPrint("dnode:%s is stopped", tsInternalIp);
monitorLPrint("dnode:%s monitor module is stopped", tsInternalIp);
#endif
monitor->state = MONITOR_STATE_STOPPED;
taosLogFp = NULL;

1
src/os/linux/inc/os.h
View File

@ -224,6 +224,7 @@ int64_t str2int64(char *str);
void taosSetCoreDump();
void taosBlockSIGPIPE();
#define BUILDIN_CLZL(val) __builtin_clzl(val)
#define BUILDIN_CLZ(val) __builtin_clz(val)

16
src/os/linux/src/tlinux.c
View File

@ -287,8 +287,10 @@ ssize_t tsendfile(int dfd, int sfd, off_t *offset, size_t size) {
ssize_t sentbytes;
while (leftbytes > 0) {
// TODO : Think to check if file is larger than 1GB
if (leftbytes > 1000000000) leftbytes = 1000000000;
/*
* TODO : Think to check if file is larger than 1GB
*/
//if (leftbytes > 1000000000) leftbytes = 1000000000;
sentbytes = sendfile(dfd, sfd, offset, leftbytes);
if (sentbytes == -1) {
if (errno == EINTR) {
@ -339,3 +341,13 @@ bool taosSkipSocketCheck() {
return false;
}
void taosBlockSIGPIPE() {
sigset_t signal_mask;
sigemptyset(&signal_mask);
sigaddset(&signal_mask, SIGPIPE);
int rc = pthread_sigmask(SIG_BLOCK, &signal_mask, NULL);
if (rc != 0) {
pError("failed to block SIGPIPE");
}
}

10
src/os/linux/src/tsystem.c
View File

@ -586,9 +586,13 @@ void taosKillSystem() {
kill(tsProcId, 2);
}
extern int tsEnableCoreFile;
int _sysctl(struct __sysctl_args *args );
void taosSetCoreDump() {
if (0 == tsEnableCoreFile) {
return;
}
// 1. set ulimit -c unlimited
struct rlimit rlim;
struct rlimit rlim_new;
@ -608,6 +612,7 @@ void taosSetCoreDump() {
pPrint("the new unlimited para: rlim_cur=%d, rlim_max=%d", rlim.rlim_cur, rlim.rlim_max);
}
#ifndef _TD_ARM_
// 2. set the path for saving core file
struct __sysctl_args args;
int old_usespid = 0;
@ -649,9 +654,10 @@ void taosSetCoreDump() {
}
pPrint("The new core_uses_pid[%d]: %d", old_len, old_usespid);
#endif
#if 0
// 3. set the path for saving core file
// 3. create the path for saving core file
int status;
char coredump_dir[32] = "/var/log/taosdump";
if (opendir(coredump_dir) == NULL) {

3
src/rpc/src/tudp.c
View File

@ -506,6 +506,8 @@ void *taosInitUdpConnection(char *ip, uint16_t port, char *label, int threads, v
pConn->localPort = (int16_t)ntohs(sin.sin_port);
}
strcpy(pConn->label, label);
if (pthread_create(&pConn->thread, &thAttr, taosRecvUdpData, pConn) != 0) {
tError("%s failed to create thread to process UDP data, reason:%s", label, strerror(errno));
taosCloseSocket(pConn->fd);
@ -513,7 +515,6 @@ void *taosInitUdpConnection(char *ip, uint16_t port, char *label, int threads, v
return NULL;
}
strcpy(pConn->label, label);
pConn->shandle = shandle;
pConn->processData = fp;
pConn->index = i;

65
src/sdb/src/sdbEngine.c
View File

@ -24,7 +24,12 @@
extern char version[];
const int16_t sdbFileVersion = 0;
int sdbExtConns = 0;
#ifdef CLUSTER
int sdbMaster = 0;
#else
int sdbMaster = 1;
#endif
void *(*sdbInitIndexFp[])(int maxRows, int dataSize) = {sdbOpenStrHash, sdbOpenIntHash, sdbOpenIntHash};
@ -98,7 +103,7 @@ int sdbOpenSdbFile(SSdbTable *pTable) {
pTable->header.swVersion = swVersion.iversion;
pTable->header.sdbFileVersion = sdbFileVersion;
if (taosCalcChecksumAppend(0, (uint8_t *)(&pTable->header), size) < 0) {
sdbError("failed to get file header checksum, file: %s", pTable->fn);
sdbError("failed to get file header checksum, file:%s", pTable->fn);
tclose(pTable->fd);
return -1;
}
@ -120,17 +125,17 @@ int sdbOpenSdbFile(SSdbTable *pTable) {
ssize_t tsize = read(pTable->fd, &(pTable->header), size);
if (tsize < size) {
sdbError("failed to read sdb file header, file: %s", pTable->fn);
sdbError("failed to read sdb file header, file:%s", pTable->fn);
tclose(pTable->fd);
return -1;
}
if (pTable->header.swVersion != swVersion.iversion) {
sdbWarn("sdb file %s version not match software version", pTable->fn);
sdbWarn("sdb file:%s version not match software version", pTable->fn);
}
if (!taosCheckChecksumWhole((uint8_t *)(&pTable->header), size)) {
sdbError("sdb file header is broken since checksum mismatch, file: %s", pTable->fn);
sdbError("sdb file header is broken since checksum mismatch, file:%s", pTable->fn);
tclose(pTable->fd);
return -1;
}
@ -173,17 +178,19 @@ int sdbInitTableByFile(SSdbTable *pTable) {
total_size = sizeof(SRowHead) + pTable->maxRowSize + sizeof(TSCKSUM);
SRowHead *rowHead = (SRowHead *)malloc(total_size);
if (rowHead == NULL) {
sdbError("failed to allocate row head memory, sdb: %s", pTable->name);
sdbError("failed to allocate row head memory, sdb:%s", pTable->name);
return -1;
}
sdbTrace("open sdb file:%s for read", pTable->fn);
// Loop to read sdb file row by row
while (1) {
memset(rowHead, 0, total_size);
bytes = read(pTable->fd, rowHead, sizeof(SRowHead));
if (bytes < 0) {
sdbError("failed to read sdb file: %s", pTable->fn);
sdbError("failed to read sdb file:%s", pTable->fn);
goto sdb_exit1;
}
@ -196,8 +203,8 @@ int sdbInitTableByFile(SSdbTable *pTable) {
}
if (rowHead->rowSize < 0 || rowHead->rowSize > pTable->maxRowSize) {
sdbError("error row size in sdb file: %s rowSize: %d maxRowSize: %d", pTable->fn, rowHead->rowSize,
pTable->maxRowSize);
sdbError("error row size in sdb file:%s, id:%d rowSize:%d maxRowSize:%d",
pTable->fn, rowHead->id, rowHead->rowSize, pTable->maxRowSize);
pTable->size += sizeof(SRowHead);
continue;
}
@ -208,13 +215,13 @@ int sdbInitTableByFile(SSdbTable *pTable) {
bytes = read(pTable->fd, rowHead->data, rowHead->rowSize + sizeof(TSCKSUM));
if (bytes < rowHead->rowSize + sizeof(TSCKSUM)) {
// TODO: Here may cause pTable->size not end of the file
sdbError("failed to read sdb file: %s id: %d rowSize: %d", pTable->fn, rowHead->id, rowHead->rowSize);
sdbError("failed to read sdb file:%s id:%d rowSize:%d", pTable->fn, rowHead->id, rowHead->rowSize);
break;
}
real_size = sizeof(SRowHead) + rowHead->rowSize + sizeof(TSCKSUM);
if (!taosCheckChecksumWhole((uint8_t *)rowHead, real_size)) {
sdbError("error sdb checksum, sdb: %s id: %d, skip", pTable->name, rowHead->id);
sdbError("error sdb checksum, sdb:%s id:%d, skip", pTable->name, rowHead->id);
pTable->size += real_size;
continue;
}
@ -225,7 +232,7 @@ int sdbInitTableByFile(SSdbTable *pTable) {
if (pMetaRow == NULL) { // New object
if (rowHead->id < 0) {
/* assert(0); */
sdbError("error sdb negative id: %d, sdb: %s, skip", rowHead->id, pTable->name);
sdbError("error sdb negative id:%d, sdb:%s, skip", rowHead->id, pTable->name);
} else {
rowMeta.id = rowHead->id;
// TODO: Get rid of the rowMeta.offset and rowSize
@ -366,7 +373,7 @@ int64_t sdbInsertRow(void *handle, void *row, int rowSize) {
pTable->id++;
sdbVersion++;
if (pTable->keyType == SDB_KEYTYPE_AUTO) {
// TODO: here need to change
// TODO:here need to change
*((uint32_t *)pObj) = ++pTable->autoIndex;
(*(pTable->appTool))(SDB_TYPE_ENCODE, pObj, rowHead->data, pTable->maxRowSize, &(rowHead->rowSize));
}
@ -376,7 +383,7 @@ int64_t sdbInsertRow(void *handle, void *row, int rowSize) {
rowHead->delimiter = SDB_DELIMITER;
rowHead->id = pTable->id;
if (taosCalcChecksumAppend(0, (uint8_t *)rowHead, real_size) < 0) {
sdbError("failed to get checksum while inserting, sdb: %s", pTable->name);
sdbError("failed to get checksum while inserting, sdb:%s", pTable->name);
pthread_mutex_unlock(&pTable->mutex);
tfree(rowHead);
return -1;
@ -470,7 +477,7 @@ int sdbDeleteRow(void *handle, void *row) {
total_size = sizeof(SRowHead) + rowSize + sizeof(TSCKSUM);
rowHead = (SRowHead *)malloc(total_size);
if (rowHead == NULL) {
sdbError("failed to allocate row head memory, sdb: %s", pTable->name);
sdbError("failed to allocate row head memory, sdb:%s", pTable->name);
return -1;
}
memset(rowHead, 0, total_size);
@ -486,7 +493,7 @@ int sdbDeleteRow(void *handle, void *row) {
rowHead->id = -(pTable->id);
memcpy(rowHead->data, row, rowSize);
if (taosCalcChecksumAppend(0, (uint8_t *)rowHead, total_size) < 0) {
sdbError("failed to get checksum while inserting, sdb: %s", pTable->name);
sdbError("failed to get checksum while inserting, sdb:%s", pTable->name);
pthread_mutex_unlock(&pTable->mutex);
tfree(rowHead);
return -1;
@ -498,7 +505,7 @@ int sdbDeleteRow(void *handle, void *row) {
sdbFinishCommit(pTable);
pTable->numOfRows--;
// TODO: Change the update list here
// TODO:Change the update list here
sdbAddIntoUpdateList(pTable, SDB_TYPE_DELETE, pMetaRow);
switch (pTable->keyType) {
case SDB_KEYTYPE_STRING:
@ -556,7 +563,7 @@ int sdbUpdateRow(void *handle, void *row, int updateSize, char isUpdated) {
total_size = sizeof(SRowHead) + pTable->maxRowSize + sizeof(TSCKSUM);
SRowHead *rowHead = (SRowHead *)malloc(total_size);
if (rowHead == NULL) {
sdbError("failed to allocate row head memory, sdb: %s", pTable->name);
sdbError("failed to allocate row head memory, sdb:%s", pTable->name);
return -1;
}
memset(rowHead, 0, total_size);
@ -585,7 +592,7 @@ int sdbUpdateRow(void *handle, void *row, int updateSize, char isUpdated) {
rowHead->delimiter = SDB_DELIMITER;
rowHead->id = pTable->id;
if (taosCalcChecksumAppend(0, (uint8_t *)rowHead, real_size) < 0) {
sdbError("failed to get checksum, sdb: %s id: %d", pTable->name, rowHead->id);
sdbError("failed to get checksum, sdb:%s id:%d", pTable->name, rowHead->id);
pthread_mutex_unlock(&pTable->mutex);
tfree(rowHead);
return -1;
@ -640,7 +647,7 @@ int sdbBatchUpdateRow(void *handle, void *row, int rowSize) {
if (pTable == NULL || row == NULL || rowSize <= 0) return -1;
pMeta = sdbGetRowMeta(handle, row);
if (pMeta == NULL) {
sdbTrace("table: %s, record is not there, batch update failed", pTable->name);
sdbTrace("table:%s, record is not there, batch update failed", pTable->name);
return -1;
}
@ -650,7 +657,7 @@ int sdbBatchUpdateRow(void *handle, void *row, int rowSize) {
total_size = sizeof(SRowHead) + pTable->maxRowSize + sizeof(TSCKSUM);
SRowHead *rowHead = (SRowHead *)malloc(total_size);
if (rowHead == NULL) {
sdbError("failed to allocate row head memory, sdb: %s", pTable->name);
sdbError("failed to allocate row head memory, sdb:%s", pTable->name);
return -1;
}
@ -748,12 +755,14 @@ void sdbResetTable(SSdbTable *pTable) {
return;
}
sdbTrace("open sdb file:%s for update", pTable->fn);
while (1) {
memset(rowHead, 0, total_size);
bytes = read(pTable->fd, rowHead, sizeof(SRowHead));
if (bytes < 0) {
sdbError("failed to read sdb file: %s", pTable->fn);
sdbError("failed to read sdb file:%s", pTable->fn);
tfree(rowHead);
return;
}
@ -767,7 +776,7 @@ void sdbResetTable(SSdbTable *pTable) {
}
if (rowHead->rowSize < 0 || rowHead->rowSize > pTable->maxRowSize) {
sdbError("error row size in sdb file: %s rowSize: %d maxRowSize: %d", pTable->fn, rowHead->rowSize,
sdbError("error row size in sdb file:%s rowSize:%d maxRowSize:%d", pTable->fn, rowHead->rowSize,
pTable->maxRowSize);
pTable->size += sizeof(SRowHead);
continue;
@ -775,13 +784,13 @@ void sdbResetTable(SSdbTable *pTable) {
bytes = read(pTable->fd, rowHead->data, rowHead->rowSize + sizeof(TSCKSUM));
if (bytes < rowHead->rowSize + sizeof(TSCKSUM)) {
sdbError("failed to read sdb file: %s id: %d rowSize: %d", pTable->fn, rowHead->id, rowHead->rowSize);
sdbError("failed to read sdb file:%s id:%d rowSize:%d", pTable->fn, rowHead->id, rowHead->rowSize);
break;
}
real_size = sizeof(SRowHead) + rowHead->rowSize + sizeof(TSCKSUM);
if (!taosCheckChecksumWhole((uint8_t *)rowHead, real_size)) {
sdbError("error sdb checksum, sdb: %s id: %d, skip", pTable->name, rowHead->id);
sdbError("error sdb checksum, sdb:%s id:%d, skip", pTable->name, rowHead->id);
pTable->size += real_size;
continue;
}
@ -790,10 +799,10 @@ void sdbResetTable(SSdbTable *pTable) {
pMetaRow = sdbGetRow(pTable, rowHead->data);
if (pMetaRow == NULL) { // New object
if (rowHead->id < 0) {
sdbError("error sdb negative id: %d, sdb: %s, skip", rowHead->id, pTable->name);
sdbError("error sdb negative id:%d, sdb:%s, skip", rowHead->id, pTable->name);
} else {
rowMeta.id = rowHead->id;
// TODO: Get rid of the rowMeta.offset and rowSize
// TODO:Get rid of the rowMeta.offset and rowSize
rowMeta.offset = pTable->size;
rowMeta.rowSize = rowHead->rowSize;
rowMeta.row = (*(pTable->appTool))(SDB_TYPE_DECODE, NULL, rowHead->data, rowHead->rowSize, NULL);
@ -826,7 +835,7 @@ void sdbResetTable(SSdbTable *pTable) {
sdbTrace("table:%s is updated, sdbVerion:%ld id:%ld", pTable->name, sdbVersion, pTable->id);
}
// TODO: A problem here : use snapshot file to sync another node will cause
// TODO:A problem here :use snapshot file to sync another node will cause
// problem
void sdbSaveSnapShot(void *handle) {
SSdbTable *pTable = (SSdbTable *)handle;
@ -856,7 +865,7 @@ void sdbSaveSnapShot(void *handle) {
total_size = sizeof(SRowHead) + pTable->maxRowSize + sizeof(TSCKSUM);
SRowHead *rowHead = (SRowHead *)malloc(total_size);
if (rowHead == NULL) {
sdbError("failed to allocate memory while saving SDB snapshot, sdb: %s", pTable->name);
sdbError("failed to allocate memory while saving SDB snapshot, sdb:%s", pTable->name);
return;
}
memset(rowHead, 0, size);

20
src/system/detail/inc/mgmt.h
View File

@ -45,17 +45,6 @@ extern int mgmtShellConns;
extern int mgmtDnodeConns;
extern char mgmtDirectory[];
enum _TSDB_VG_STATUS {
TSDB_VG_STATUS_READY,
TSDB_VG_STATUS_IN_PROGRESS,
TSDB_VG_STATUS_COMMITLOG_INIT_FAILED,
TSDB_VG_STATUS_INIT_FAILED,
TSDB_VG_STATUS_FULL
};
enum _TSDB_DB_STATUS { TSDB_DB_STATUS_READY, TSDB_DB_STATUS_DROPPING, TSDB_DB_STATUS_DROP_FROM_SDB };
enum _TSDB_VN_STATUS { TSDB_VN_STATUS_READY, TSDB_VN_STATUS_DROPPING };
typedef struct {
uint32_t privateIp;
@ -86,7 +75,7 @@ typedef struct {
uint16_t slot;
int32_t customScore; // config by user
float lbScore; // calc in balance function
int16_t lbState; // set in balance function
int16_t lbStatus; // set in balance function
int16_t lastAllocVnode; // increase while create vnode
SVnodeLoad vload[TSDB_MAX_VNODES];
char reserved[16];
@ -148,7 +137,7 @@ typedef struct _vg_obj {
int32_t numOfMeters;
int32_t lbIp;
int32_t lbTime;
int8_t lbState;
int8_t lbStatus;
char reserved[16];
char updateEnd[1];
struct _vg_obj *prev, *next;
@ -157,6 +146,9 @@ typedef struct _vg_obj {
} SVgObj;
typedef struct _db_obj {
/*
* this length will cause the storage structure to change, rollback
*/
char name[TSDB_DB_NAME_LEN + 1];
int64_t createdTime;
SDbCfg cfg;
@ -426,7 +418,7 @@ void mgmtSetDnodeShellRemoving(SDnodeObj *pDnode);
void mgmtSetDnodeUnRemove(SDnodeObj *pDnode);
void mgmtStartBalanceTimer(int mseconds);
void mgmtSetDnodeOfflineOnSdbChanged();
void mgmtUpdateVgroupState(SVgObj *pVgroup, int lbState, int srcIp);
void mgmtUpdateVgroupState(SVgObj *pVgroup, int lbStatus, int srcIp);
bool mgmtAddVnode(SVgObj *pVgroup, SDnodeObj *pSrcDnode, SDnodeObj *pDestDnode);
void mgmtSetModuleInDnode(SDnodeObj *pDnode, int moduleType);

1
src/system/detail/inc/mgmtUtil.h
View File

@ -31,6 +31,7 @@ int32_t mgmtFindTagCol(STabObj * pMetric, const char * tagName);
int32_t mgmtGetTagsLength(STabObj* pMetric, int32_t col);
bool mgmtCheckIsMonitorDB(char *db, char *monitordb);
int32_t mgmtCheckDBParams(SCreateDbMsg *pCreate);
int32_t mgmtRetrieveMetersFromMetric(SMetricMetaMsg* pInfo, int32_t tableIndex, tQueryResultset* pRes);
int32_t mgmtDoJoin(SMetricMetaMsg* pMetricMetaMsg, tQueryResultset* pRes);

3
src/system/detail/inc/vnodeQueryImpl.h
View File

@ -174,7 +174,7 @@ void queryOnBlock(SMeterQuerySupportObj* pSupporter, int64_t* primaryKeys, int32
SBlockInfo* pBlockBasicInfo, SMeterDataInfo* pDataHeadInfoEx, SField* pFields,
__block_search_fn_t searchFn);
SMeterDataInfo** vnodeFilterQualifiedMeters(SQInfo* pQInfo, int32_t vid, SQueryFileInfo* pQueryFileInfo,
SMeterDataInfo** vnodeFilterQualifiedMeters(SQInfo* pQInfo, int32_t vid, int32_t fileIndex,
tSidSet* pSidSet, SMeterDataInfo* pMeterDataInfo, int32_t* numOfMeters);
int32_t vnodeGetVnodeHeaderFileIdx(int32_t* fid, SQueryRuntimeEnv* pRuntimeEnv, int32_t order);
@ -194,6 +194,7 @@ uint32_t getDataBlocksForMeters(SMeterQuerySupportObj* pSupporter, SQuery* pQuer
int32_t numOfMeters, SQueryFileInfo* pQueryFileInfo, SMeterDataInfo** pMeterDataInfo);
int32_t LoadDatablockOnDemand(SCompBlock* pBlock, SField** pFields, int8_t* blkStatus, SQueryRuntimeEnv* pRuntimeEnv,
int32_t fileIdx, int32_t slotIdx, __block_search_fn_t searchFn, bool onDemand);
char *vnodeGetHeaderFileData(SQueryRuntimeEnv *pRuntimeEnv, int32_t fileIndex);
/**
* Create SMeterQueryInfo.

56
src/system/detail/inc/vnodeRead.h
View File

@ -57,9 +57,9 @@ typedef struct SQueryFileInfo {
char lastFilePath[256];
int32_t defaultMappingSize; /* default mapping size */
int32_t headerFd; /* file handler */
char* pHeaderFileData; /* mmap header files */
size_t headFileSize;
int32_t headerFd; /* file handler */
char* pHeaderFileData; /* mmap header files */
size_t headFileSize;
int32_t dataFd;
char* pDataFileData;
size_t dataFileSize;
@ -107,44 +107,40 @@ typedef struct SOutputRes {
} SOutputRes;
typedef struct RuntimeEnvironment {
SPositionInfo startPos; /* the start position, used for secondary/third iteration */
SPositionInfo endPos; /* the last access position in query, served as the start pos of reversed order query */
SPositionInfo nextPos; /* start position of the next scan */
SData* colDataBuffer[TSDB_MAX_COLUMNS];
SResultInfo* resultInfo;
// Indicate if data block is loaded, the block is first/last/internal block
int8_t blockStatus;
int32_t unzipBufSize;
SData* primaryColBuffer;
char* unzipBuffer;
char* secondaryUnzipBuffer;
SQuery* pQuery;
SMeterObj* pMeterObj;
SQLFunctionCtx* pCtx;
SQueryLoadBlockInfo loadBlockInfo; /* record current block load information */
SPositionInfo startPos; /* the start position, used for secondary/third iteration */
SPositionInfo endPos; /* the last access position in query, served as the start pos of reversed order query */
SPositionInfo nextPos; /* start position of the next scan */
SData* colDataBuffer[TSDB_MAX_COLUMNS];
SResultInfo* resultInfo;
uint8_t blockStatus; // Indicate if data block is loaded, the block is first/last/internal block
int32_t unzipBufSize;
SData* primaryColBuffer;
char* unzipBuffer;
char* secondaryUnzipBuffer;
SQuery* pQuery;
SMeterObj* pMeterObj;
SQLFunctionCtx* pCtx;
SQueryLoadBlockInfo loadBlockInfo; /* record current block load information */
SQueryLoadCompBlockInfo loadCompBlockInfo; /* record current compblock information in SQuery */
/*
* header files info, avoid to iterate the directory, the data is acquired
* during in query preparation function
*/
SQueryFileInfo* pHeaderFiles;
uint32_t numOfFiles; /* number of files of one vnode during query execution */
int16_t numOfRowsPerPage;
int16_t offset[TSDB_MAX_COLUMNS];
int16_t scanFlag; /* denotes reversed scan of data or not */
SQueryFileInfo* pVnodeFiles;
uint32_t numOfFiles; // the total available number of files for this virtual node during query execution
int32_t mmapedHFileIndex; // the mmaped header file, NOTE: only one header file can be mmap.
int16_t numOfRowsPerPage;
int16_t offset[TSDB_MAX_COLUMNS];
int16_t scanFlag; // denotes reversed scan of data or not
SInterpolationInfo interpoInfo;
SData** pInterpoBuf;
SOutputRes* pResult; // reference to SQuerySupporter->pResult
void* hashList;
int32_t usedIndex; // assigned SOutputRes in list
STSBuf* pTSBuf;
STSCursor cur;
SQueryCostSummary summary;
STSBuf* pTSBuf;
STSCursor cur;
SQueryCostSummary summary;
} SQueryRuntimeEnv;
/* intermediate result during multimeter query involves interval */

2
src/system/detail/inc/vnodeUtil.h
View File

@ -81,6 +81,8 @@ int32_t vnodeSetMeterState(SMeterObj* pMeterObj, int32_t state);
void vnodeClearMeterState(SMeterObj* pMeterObj, int32_t state);
bool vnodeIsMeterState(SMeterObj* pMeterObj, int32_t state);
void vnodeSetMeterDeleting(SMeterObj* pMeterObj);
int32_t vnodeSetMeterInsertImportStateEx(SMeterObj* pObj, int32_t st);
bool vnodeIsSafeToDeleteMeter(SVnodeObj* pVnode, int32_t sid);
void vnodeFreeColumnInfo(SColumnInfo* pColumnInfo);
bool isGroupbyNormalCol(SSqlGroupbyExpr* pExpr);

54
src/system/detail/src/dnodeMgmt.c
View File

@ -116,7 +116,7 @@ int vnodeProcessCreateMeterRequest(char *pMsg, int msgLen, SMgmtObj *pObj) {
goto _over;
}
if (pVnode->syncStatus == TSDB_SSTATUS_SYNCING) {
if (pVnode->syncStatus == TSDB_VN_SYNC_STATUS_SYNCING) {
code = vnodeSaveCreateMsgIntoQueue(pVnode, pMsg, msgLen);
dTrace("vid:%d, create msg is saved into sync queue", vid);
} else {
@ -331,7 +331,7 @@ int vnodeProcessVPeerCfg(char *msg, int msgLen, SMgmtObj *pMgmtObj) {
return -1;
}
if (vnodeList[vnode].vnodeStatus == TSDB_VNODE_STATUS_CREATING) {
if (vnodeList[vnode].vnodeStatus == TSDB_VN_STATUS_CREATING) {
dTrace("vid:%d, vnode is still under creating", vnode);
return 0;
}
@ -350,51 +350,53 @@ int vnodeProcessVPeerCfg(char *msg, int msgLen, SMgmtObj *pMgmtObj) {
pCfg->rowsInFileBlock = htonl(pCfg->rowsInFileBlock);
if (pCfg->replications > 0) {
dTrace("vid:%d, vpeer cfg received, replica:%d session:%d, vnodeList replica:%d session:%d",
vnode, pCfg->replications, pCfg->maxSessions, vnodeList[vnode].cfg.replications, vnodeList[vnode].cfg.maxSessions);
dPrint("vid:%d, vpeer cfg received, replica:%d session:%d, vnodeList replica:%d session:%d, acct:%s db:%s",
vnode, pCfg->replications, pCfg->maxSessions, vnodeList[vnode].cfg.replications, vnodeList[vnode].cfg.maxSessions,
pCfg->acct, pCfg->db);
for (i = 0; i < pCfg->replications; ++i) {
pMsg->vpeerDesc[i].vnode = htonl(pMsg->vpeerDesc[i].vnode);
pMsg->vpeerDesc[i].ip = htonl(pMsg->vpeerDesc[i].ip);
dTrace("vid:%d, vpeer:%d ip:0x%x vid:%d ", vnode, i, pMsg->vpeerDesc[i].ip, pMsg->vpeerDesc[i].vnode);
dPrint("vid:%d, vpeer:%d ip:0x%x vid:%d ", vnode, i, pMsg->vpeerDesc[i].ip, pMsg->vpeerDesc[i].vnode);
}
}
if (vnodeList[vnode].cfg.maxSessions == 0) {
dTrace("vid:%d, vnode is empty", vnode);
dPrint("vid:%d, vnode is empty", vnode);
if (pCfg->maxSessions > 0) {
if (vnodeList[vnode].vnodeStatus == TSDB_VNODE_STATUS_OFFLINE) {
dTrace("vid:%d, status:%s, start to create vnode", vnode, taosGetVnodeStatusStr(vnodeList[vnode].vnodeStatus));
if (vnodeList[vnode].vnodeStatus == TSDB_VN_STATUS_OFFLINE) {
dPrint("vid:%d, status:%s, start to create vnode", vnode, taosGetVnodeStatusStr(vnodeList[vnode].vnodeStatus));
return vnodeCreateVnode(vnode, pCfg, pMsg->vpeerDesc);
} else {
dTrace("vid:%d, status:%s, cannot preform create vnode operation", vnode, taosGetVnodeStatusStr(vnodeList[vnode].vnodeStatus));
dPrint("vid:%d, status:%s, cannot preform create vnode operation", vnode, taosGetVnodeStatusStr(vnodeList[vnode].vnodeStatus));
return TSDB_CODE_INVALID_VNODE_STATUS;
}
}
} else {
dTrace("vid:%d, vnode is not empty", vnode);
dPrint("vid:%d, vnode is not empty", vnode);
if (pCfg->maxSessions > 0) {
if (vnodeList[vnode].vnodeStatus == TSDB_VNODE_STATUS_DELETING) {
dTrace("vid:%d, status:%s, wait vnode delete finished", vnode, taosGetVnodeStatusStr(vnodeList[vnode].vnodeStatus));
if (vnodeList[vnode].vnodeStatus == TSDB_VN_STATUS_DELETING) {
dPrint("vid:%d, status:%s, wait vnode delete finished", vnode, taosGetVnodeStatusStr(vnodeList[vnode].vnodeStatus));
} else {
dTrace("vid:%d, status:%s, start to update vnode", vnode, taosGetVnodeStatusStr(vnodeList[vnode].vnodeStatus));
}
/*
if (pCfg->maxSessions != vnodeList[vnode].cfg.maxSessions) {
vnodeCleanUpOneVnode(vnode);
}
dPrint("vid:%d, status:%s, start to update vnode", vnode, taosGetVnodeStatusStr(vnodeList[vnode].vnodeStatus));
vnodeConfigVPeers(vnode, pCfg->replications, pMsg->vpeerDesc);
vnodeSaveVnodeCfg(vnode, pCfg, pMsg->vpeerDesc);
if (pCfg->maxSessions != vnodeList[vnode].cfg.maxSessions) {
vnodeCleanUpOneVnode(vnode);
}
if (pCfg->maxSessions != vnodeList[vnode].cfg.maxSessions) {
vnodeUpdateHeadFile(vnode, vnodeList[vnode].cfg.maxSessions, pCfg->maxSessions);
vnodeList[vnode].cfg.maxSessions = pCfg->maxSessions;
vnodeOpenVnode(vnode);
vnodeConfigVPeers(vnode, pCfg->replications, pMsg->vpeerDesc);
vnodeSaveVnodeCfg(vnode, pCfg, pMsg->vpeerDesc);
/*
if (pCfg->maxSessions != vnodeList[vnode].cfg.maxSessions) {
vnodeUpdateHeadFile(vnode, vnodeList[vnode].cfg.maxSessions, pCfg->maxSessions);
vnodeList[vnode].cfg.maxSessions = pCfg->maxSessions;
vnodeOpenVnode(vnode);
}
*/
}
*/
return 0;
} else {
dTrace("vid:%d, status:%s, start to delete vnode", vnode, taosGetVnodeStatusStr(vnodeList[vnode].vnodeStatus));
dPrint("vid:%d, status:%s, start to delete vnode", vnode, taosGetVnodeStatusStr(vnodeList[vnode].vnodeStatus));
vnodeRemoveVnode(vnode);
}
}

2
src/system/detail/src/dnodeSystem.c
View File

@ -136,6 +136,8 @@ int dnodeInitSystem() {
return -1;
}
vnodeInitMgmtIp();
tsPrintGlobalConfig();
dPrint("Server IP address is:%s", tsInternalIp);

92
src/system/detail/src/mgmtDb.c
View File

@ -20,6 +20,7 @@
#include "mgmtBalance.h"
#include "mgmtUtil.h"
#include "tschemautil.h"
#include "tstatus.h"
void *dbSdb = NULL;
int tsDbUpdateSize;
@ -139,90 +140,31 @@ int mgmtCheckDbParams(SCreateDbMsg *pCreate) {
if (pCreate->replications < 0) pCreate->replications = tsReplications; //
if (pCreate->rowsInFileBlock < 0) pCreate->rowsInFileBlock = tsRowsInFileBlock; //
if (pCreate->cacheNumOfBlocks.fraction < 0) pCreate->cacheNumOfBlocks.fraction = tsAverageCacheBlocks; //
//-1 for balance
if (pCreate->replications <= 0 || pCreate->replications > TSDB_REPLICA_MAX_NUM) {
mTrace("invalid db option replications: %d", pCreate->replications);
if (mgmtCheckDBParams(pCreate) != TSDB_CODE_SUCCESS) {
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->commitLog < 0 || pCreate->commitLog > 1) {
mTrace("invalid db option commitLog: %d", pCreate->commitLog);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->replications < TSDB_REPLICA_MIN_NUM || pCreate->replications > TSDB_REPLICA_MAX_NUM) {
mTrace("invalid db option replications: %d", pCreate->replications);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->daysPerFile < TSDB_FILE_MIN_PARTITION_RANGE || pCreate->daysPerFile > TSDB_FILE_MAX_PARTITION_RANGE) {
mTrace("invalid db option daysPerFile: %d valid range: %d--%d", pCreate->daysPerFile, TSDB_FILE_MIN_PARTITION_RANGE,
TSDB_FILE_MAX_PARTITION_RANGE);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->daysToKeep1 > pCreate->daysToKeep2 || pCreate->daysToKeep2 > pCreate->daysToKeep) {
mTrace("invalid db option daystokeep1: %d, daystokeep2: %d, daystokeep: %d", pCreate->daysToKeep1,
pCreate->daysToKeep2, pCreate->daysToKeep);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->daysToKeep1 < TSDB_FILE_MIN_PARTITION_RANGE || pCreate->daysToKeep1 < pCreate->daysPerFile) {
mTrace("invalid db option daystokeep: %d", pCreate->daysToKeep);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->rowsInFileBlock < TSDB_MIN_ROWS_IN_FILEBLOCK || pCreate->rowsInFileBlock > TSDB_MAX_ROWS_IN_FILEBLOCK) {
mTrace("invalid db option rowsInFileBlock: %d valid range: %d--%d", pCreate->rowsInFileBlock,
TSDB_MIN_ROWS_IN_FILEBLOCK, TSDB_MAX_ROWS_IN_FILEBLOCK);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->cacheBlockSize < TSDB_MIN_CACHE_BLOCK_SIZE || pCreate->cacheBlockSize > TSDB_MAX_CACHE_BLOCK_SIZE) {
mTrace("invalid db option cacheBlockSize: %d valid range: %d--%d", pCreate->cacheBlockSize,
TSDB_MIN_CACHE_BLOCK_SIZE, TSDB_MAX_CACHE_BLOCK_SIZE);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->maxSessions < TSDB_MIN_TABLES_PER_VNODE || pCreate->maxSessions > TSDB_MAX_TABLES_PER_VNODE) {
mTrace("invalid db option maxSessions: %d valid range: %d--%d", pCreate->maxSessions, TSDB_MIN_TABLES_PER_VNODE,
TSDB_MAX_TABLES_PER_VNODE);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->precision != TSDB_TIME_PRECISION_MILLI && pCreate->precision != TSDB_TIME_PRECISION_MICRO) {
mTrace("invalid db option timePrecision: %d valid value: %d,%d", pCreate->precision, TSDB_TIME_PRECISION_MILLI,
TSDB_TIME_PRECISION_MICRO);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->cacheNumOfBlocks.fraction < 0 || pCreate->cacheNumOfBlocks.fraction > TSDB_MAX_AVG_BLOCKS) {
mTrace("invalid db option ablocks: %d valid value: %d,%d", pCreate->precision, 0, TSDB_MAX_AVG_BLOCKS);
return TSDB_CODE_INVALID_OPTION;
} else {
pCreate->cacheNumOfBlocks.totalBlocks = (int32_t)(pCreate->cacheNumOfBlocks.fraction * pCreate->maxSessions);
}
pCreate->cacheNumOfBlocks.totalBlocks = (int32_t)(pCreate->cacheNumOfBlocks.fraction * pCreate->maxSessions);
if (pCreate->cacheNumOfBlocks.totalBlocks > TSDB_MAX_CACHE_BLOCKS) {
mTrace("invalid db option cacheNumOfBlocks: %d valid range: %d", pCreate->cacheNumOfBlocks.totalBlocks,
TSDB_MAX_CACHE_BLOCKS);
mTrace("invalid db option cacheNumOfBlocks: %d valid range: [%d, %d]", pCreate->cacheNumOfBlocks.totalBlocks,
TSDB_MIN_CACHE_BLOCKS, TSDB_MAX_CACHE_BLOCKS);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->commitTime < TSDB_MIN_COMMIT_TIME_INTERVAL || pCreate->commitTime > TSDB_MAX_COMMIT_TIME_INTERVAL) {
mTrace("invalid db option commitTime: %d valid range: %d-%d", pCreate->commitTime, TSDB_MIN_COMMIT_TIME_INTERVAL,
TSDB_MAX_COMMIT_TIME_INTERVAL);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->compression > TSDB_MAX_COMPRESSION_LEVEL) {
mTrace("invalid db option compression: %d", pCreate->compression, TSDB_MIN_COMMIT_TIME_INTERVAL,
TSDB_MAX_COMMIT_TIME_INTERVAL);
return TSDB_CODE_INVALID_OPTION;
// calculate the blocks per table
if (pCreate->blocksPerMeter < 0) {
pCreate->blocksPerMeter = pCreate->cacheNumOfBlocks.totalBlocks / 4;
}
if (pCreate->blocksPerMeter < 0) pCreate->blocksPerMeter = pCreate->cacheNumOfBlocks.totalBlocks / 4;
if (pCreate->blocksPerMeter > pCreate->cacheNumOfBlocks.totalBlocks * 3 / 4) {
pCreate->blocksPerMeter = pCreate->cacheNumOfBlocks.totalBlocks * 3 / 4;
}
if (pCreate->blocksPerMeter < 4) pCreate->blocksPerMeter = 4;
if (pCreate->blocksPerMeter < TSDB_MIN_AVG_BLOCKS) {
pCreate->blocksPerMeter = TSDB_MIN_AVG_BLOCKS;
}
pCreate->maxSessions++;
@ -281,8 +223,8 @@ int mgmtSetDbDropping(SDbObj *pDb) {
if (pDnode == NULL) continue;
SVnodeLoad *pVload = &pDnode->vload[pVnodeGid->vnode];
if (pVload->dropStatus != TSDB_VN_STATUS_DROPPING) {
pVload->dropStatus = TSDB_VN_STATUS_DROPPING;
if (pVload->dropStatus != TSDB_VN_DROP_STATUS_DROPPING) {
pVload->dropStatus = TSDB_VN_DROP_STATUS_DROPPING;
mPrint("dnode:%s vnode:%d db:%s set to dropping status", taosIpStr(pDnode->privateIp), pVnodeGid->vnode, pDb->name);
if (mgmtUpdateDnode(pDnode) < 0) {
@ -315,10 +257,10 @@ bool mgmtCheckDropDbFinished(SDbObj *pDb) {
SDnodeObj *pDnode = mgmtGetDnode(pVnodeGid->ip);
if (pDnode == NULL) continue;
if (pDnode->status == TSDB_DNODE_STATUS_OFFLINE) continue;
if (pDnode->status == TSDB_DN_STATUS_OFFLINE) continue;
SVnodeLoad *pVload = &pDnode->vload[pVnodeGid->vnode];
if (pVload->dropStatus == TSDB_VN_STATUS_DROPPING) {
if (pVload->dropStatus == TSDB_VN_DROP_STATUS_DROPPING) {
mTrace("dnode:%s, vnode:%d db:%s wait dropping", taosIpStr(pDnode->privateIp), pVnodeGid->vnode, pDb->name);
return false;
}
@ -444,7 +386,7 @@ int mgmtAlterDb(SAcctObj *pAcct, SAlterDbMsg *pAlter) {
SVgObj *pVgroup = pDb->pHead;
while (pVgroup != NULL) {
mgmtUpdateVgroupState(pVgroup, LB_VGROUP_STATE_UPDATE, 0);
mgmtUpdateVgroupState(pVgroup, TSDB_VG_LB_STATUS_UPDATE, 0);
if (oldReplicaNum < pDb->cfg.replications) {
if (!mgmtAddVnode(pVgroup, NULL, NULL)) {
mWarn("db:%s vgroup:%d not enough dnode to add vnode", pAlter->db, pVgroup->vgId);

13
src/system/detail/src/mgmtDnode.c
View File

@ -44,20 +44,21 @@ void mgmtSetDnodeMaxVnodes(SDnodeObj *pDnode) {
pDnode->openVnodes = 0;
#ifdef CLUSTER
pDnode->status = TSDB_DNODE_STATUS_OFFLINE;
pDnode->status = TSDB_DN_STATUS_OFFLINE;
#else
pDnode->status = TSDB_DNODE_STATUS_READY;
pDnode->status = TSDB_DN_STATUS_READY;
#endif
}
void mgmtCalcNumOfFreeVnodes(SDnodeObj *pDnode) {
int totalVnodes = 0;
mTrace("dnode:%s, begin calc free vnodes", taosIpStr(pDnode->privateIp));
for (int i = 0; i < pDnode->numOfVnodes; ++i) {
SVnodeLoad *pVload = pDnode->vload + i;
if (pVload->vgId != 0) {
mTrace("dnode:%s, calc free vnodes, exist vnode:%d, vgroup:%d, state:%d %s, dropstate:%d %s, syncstatus:%d %s",
taosIpStr(pDnode->privateIp), i, pVload->vgId,
mTrace("%d-dnode:%s, calc free vnodes, exist vnode:%d, vgroup:%d, state:%d %s, dropstate:%d %s, syncstatus:%d %s",
totalVnodes, taosIpStr(pDnode->privateIp), i, pVload->vgId,
pVload->status, taosGetDnodeStatusStr(pVload->status),
pVload->dropStatus, taosGetVnodeDropStatusStr(pVload->dropStatus),
pVload->syncStatus, taosGetVnodeSyncStatusStr(pVload->syncStatus));
@ -66,7 +67,7 @@ void mgmtCalcNumOfFreeVnodes(SDnodeObj *pDnode) {
}
pDnode->numOfFreeVnodes = pDnode->numOfVnodes - totalVnodes;
mTrace("dnode:%s, calc free vnodes, numOfVnodes:%d, numOfFreeVnodes:%d, totalVnodes:%d",
mTrace("dnode:%s, numOfVnodes:%d, numOfFreeVnodes:%d, totalVnodes:%d",
taosIpStr(pDnode->privateIp), pDnode->numOfVnodes, pDnode->numOfFreeVnodes, totalVnodes);
}
@ -201,7 +202,7 @@ int mgmtRetrieveDnodes(SShowObj *pShow, char *data, int rows, SConnObj *pConn) {
cols++;
pWrite = data + pShow->offset[cols] * rows + pShow->bytes[cols] * numOfRows;
strcpy(pWrite, taosGetDnodeBalanceStateStr(pDnode->lbState));
strcpy(pWrite, taosGetDnodeLbStatusStr(pDnode->lbStatus));
cols++;
tinet_ntoa(ipstr, pDnode->publicIp);

28
src/system/detail/src/mgmtDnodeInt.c
View File

@ -43,6 +43,11 @@ int mgmtProcessMeterCfgMsg(char *cont, int contLen, SDnodeObj *pObj) {
SMeterCfgMsg *pCfg = (SMeterCfgMsg *)cont;
SVgObj * pVgroup;
if (!sdbMaster) {
taosSendSimpleRspToDnode(pObj, TSDB_MSG_TYPE_METER_CFG_RSP, TSDB_CODE_REDIRECT);
return 0;
}
int vnode = htonl(pCfg->vnode);
int sid = htonl(pCfg->sid);
@ -51,6 +56,7 @@ int mgmtProcessMeterCfgMsg(char *cont, int contLen, SDnodeObj *pObj) {
taosSendSimpleRspToDnode(pObj, TSDB_MSG_TYPE_METER_CFG_RSP, TSDB_CODE_SERV_OUT_OF_MEMORY);
return 0;
}
pMsg = pStart;
if (vnode < pObj->numOfVnodes) {
@ -87,10 +93,18 @@ int mgmtProcessVpeerCfgMsg(char *cont, int contLen, SDnodeObj *pObj) {
SVpeerCfgMsg *pCfg = (SVpeerCfgMsg *)cont;
SVgObj * pVgroup = NULL;
if (!sdbMaster) {
taosSendSimpleRspToDnode(pObj, TSDB_MSG_TYPE_VPEER_CFG_RSP, TSDB_CODE_REDIRECT);
return 0;
}
int vnode = htonl(pCfg->vnode);
pStart = taosBuildRspMsgToDnode(pObj, TSDB_MSG_TYPE_VPEER_CFG_RSP);
if (pStart == NULL) return 0;
if (pStart == NULL) {
taosSendSimpleRspToDnode(pObj, TSDB_MSG_TYPE_VPEER_CFG_RSP, TSDB_CODE_SERV_OUT_OF_MEMORY);
return 0;
}
pMsg = pStart;
if (vnode < pObj->numOfVnodes) pVgroup = mgmtGetVgroup(pObj->vload[vnode].vgId);
@ -99,7 +113,7 @@ int mgmtProcessVpeerCfgMsg(char *cont, int contLen, SDnodeObj *pObj) {
*pMsg = 0;
pMsg++;
pMsg = mgmtBuildVpeersIe(pMsg, pVgroup, vnode);
mTrace("dnode:%s, vnode:%d, vgroup:%d, send create meter msg, code:%d", taosIpStr(pObj->privateIp), vnode, pVgroup->vgId, *pMsg);
mTrace("dnode:%s, vnode:%d, vgroup:%d, send create vnode msg, code:%d", taosIpStr(pObj->privateIp), vnode, pVgroup->vgId, *pMsg);
} else {
mTrace("dnode:%s, vnode:%d, no vgroup info, vgroup:%d", taosIpStr(pObj->privateIp), vnode, pObj->vload[vnode].vgId);
*pMsg = TSDB_CODE_NOT_ACTIVE_VNODE;
@ -121,14 +135,20 @@ int mgmtProcessFreeVnodeRsp(char *msg, int msgLen, SDnodeObj *pObj) { return 0;
int mgmtProcessVPeersRsp(char *msg, int msgLen, SDnodeObj *pObj) {
STaosRsp *pRsp = (STaosRsp *)msg;
if (!sdbMaster) {
taosSendSimpleRspToDnode(pObj, TSDB_MSG_TYPE_VPEERS_RSP, TSDB_CODE_REDIRECT);
return 0;
}
SDbObj *pDb = mgmtGetDb(pRsp->more);
if (!pDb) {
mError("dnode:%s, db not find, code:%d", taosIpStr(pObj->privateIp), pRsp->code);
mError("dnode:%s, db:%s not find, code:%d", taosIpStr(pObj->privateIp), pRsp->more, pRsp->code);
return 0;
}
if (pDb->vgStatus != TSDB_VG_STATUS_IN_PROGRESS) {
mTrace("dnode:%s, db:%s vpeer rsp already disposed, vgroup status:%d code:%d", taosIpStr(pObj->privateIp), pRsp->more, pDb->vgStatus, pRsp->code);
mTrace("dnode:%s, db:%s vpeer rsp already disposed, vgroup status:%s code:%d",
taosIpStr(pObj->privateIp), pRsp->more, taosGetVgroupStatusStr(pDb->vgStatus), pRsp->code);
return 0;
}

5
src/system/detail/src/mgmtMeter.c
View File

@ -27,6 +27,7 @@
#include "tsqlfunction.h"
#include "ttime.h"
#include "vnodeTagMgmt.h"
#include "tstatus.h"
extern int64_t sdbVersion;
@ -617,7 +618,7 @@ int mgmtCreateMeter(SDbObj *pDb, SCreateTableMsg *pCreate) {
if (pDb->vgStatus == TSDB_VG_STATUS_IN_PROGRESS) {
mgmtDestroyMeter(pMeter);
mTrace("table:%s, vgroup in creating progress", pCreate->meterId);
//mTrace("table:%s, vgroup in creating progress", pCreate->meterId);
return TSDB_CODE_ACTION_IN_PROGRESS;
}
@ -661,7 +662,7 @@ int mgmtCreateMeter(SDbObj *pDb, SCreateTableMsg *pCreate) {
pMeter->uid = (((uint64_t)pMeter->gid.vgId) << 40) + ((((uint64_t)pMeter->gid.sid) & ((1ul << 24) - 1ul)) << 16) +
((uint64_t)sdbVersion & ((1ul << 16) - 1ul));
mTrace("table:%s, create table in vgroup, vgId:%d sid:%d vnode:%d uid:%d db:%s",
mTrace("table:%s, create table in vgroup, vgId:%d sid:%d vnode:%d uid:%ld db:%s",
pMeter->meterId, pVgroup->vgId, sid, pVgroup->vnodeGid[0].vnode, pMeter->uid, pDb->name);
} else {
pMeter->uid = (((uint64_t)pMeter->createdTime) << 16) + ((uint64_t)sdbVersion & ((1ul << 16) - 1ul));

6
src/system/detail/src/mgmtShell.c
View File

@ -21,6 +21,7 @@
#include "mgmtProfile.h"
#include "taosmsg.h"
#include "tlog.h"
#include "tstatus.h"
#pragma GCC diagnostic push
@ -973,7 +974,10 @@ int mgmtProcessCreateTableMsg(char *pMsg, int msgLen, SConnObj *pConn) {
}
}
if (code != 0) {
if (code == 1) {
//mTrace("table:%s, wait vgroup create finish", pCreate->meterId, code);
}
else if (code != 0) {
mError("table:%s, failed to create table, code:%d", pCreate->meterId, code);
} else {
mTrace("table:%s, table is created by %s", pCreate->meterId, pConn->pUser->user);

73
src/system/detail/src/mgmtUtil.c
View File

@ -92,3 +92,76 @@ bool mgmtCheckIsMonitorDB(char *db, char *monitordb) {
size_t len = strlen(dbName);
return (strncasecmp(dbName, monitordb, len) == 0 && len == strlen(monitordb));
}
int32_t mgmtCheckDBParams(SCreateDbMsg *pCreate) {
if (pCreate->commitLog < 0 || pCreate->commitLog > 1) {
mError("invalid db option commitLog: %d, only 0 or 1 allowed", pCreate->commitLog);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->replications < TSDB_REPLICA_MIN_NUM || pCreate->replications > TSDB_REPLICA_MAX_NUM) {
mError("invalid db option replications: %d valid range: [%d, %d]", pCreate->replications, TSDB_REPLICA_MIN_NUM,
TSDB_REPLICA_MAX_NUM);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->daysPerFile < TSDB_FILE_MIN_PARTITION_RANGE || pCreate->daysPerFile > TSDB_FILE_MAX_PARTITION_RANGE) {
mError("invalid db option daysPerFile: %d valid range: [%d, %d]", pCreate->daysPerFile, TSDB_FILE_MIN_PARTITION_RANGE,
TSDB_FILE_MAX_PARTITION_RANGE);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->daysToKeep1 > pCreate->daysToKeep2 || pCreate->daysToKeep2 > pCreate->daysToKeep) {
mError("invalid db option daystokeep1: %d, daystokeep2: %d, daystokeep: %d", pCreate->daysToKeep1,
pCreate->daysToKeep2, pCreate->daysToKeep);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->daysToKeep1 < TSDB_FILE_MIN_PARTITION_RANGE || pCreate->daysToKeep1 < pCreate->daysPerFile) {
mError("invalid db option daystokeep: %d", pCreate->daysToKeep);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->rowsInFileBlock < TSDB_MIN_ROWS_IN_FILEBLOCK || pCreate->rowsInFileBlock > TSDB_MAX_ROWS_IN_FILEBLOCK) {
mError("invalid db option rowsInFileBlock: %d valid range: [%d, %d]", pCreate->rowsInFileBlock,
TSDB_MIN_ROWS_IN_FILEBLOCK, TSDB_MAX_ROWS_IN_FILEBLOCK);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->cacheBlockSize < TSDB_MIN_CACHE_BLOCK_SIZE || pCreate->cacheBlockSize > TSDB_MAX_CACHE_BLOCK_SIZE) {
mError("invalid db option cacheBlockSize: %d valid range: [%d, %d]", pCreate->cacheBlockSize,
TSDB_MIN_CACHE_BLOCK_SIZE, TSDB_MAX_CACHE_BLOCK_SIZE);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->maxSessions < TSDB_MIN_TABLES_PER_VNODE || pCreate->maxSessions > TSDB_MAX_TABLES_PER_VNODE) {
mError("invalid db option maxSessions: %d valid range: [%d, %d]", pCreate->maxSessions, TSDB_MIN_TABLES_PER_VNODE,
TSDB_MAX_TABLES_PER_VNODE);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->precision != TSDB_TIME_PRECISION_MILLI && pCreate->precision != TSDB_TIME_PRECISION_MICRO) {
mError("invalid db option timePrecision: %d valid value: [%d, %d]", pCreate->precision, TSDB_TIME_PRECISION_MILLI,
TSDB_TIME_PRECISION_MICRO);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->cacheNumOfBlocks.fraction < TSDB_MIN_AVG_BLOCKS || pCreate->cacheNumOfBlocks.fraction > TSDB_MAX_AVG_BLOCKS) {
mError("invalid db option ablocks: %f valid value: [%d, %d]", pCreate->cacheNumOfBlocks.fraction, 0, TSDB_MAX_AVG_BLOCKS);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->commitTime < TSDB_MIN_COMMIT_TIME_INTERVAL || pCreate->commitTime > TSDB_MAX_COMMIT_TIME_INTERVAL) {
mError("invalid db option commitTime: %d valid range: [%d, %d]", pCreate->commitTime, TSDB_MIN_COMMIT_TIME_INTERVAL,
TSDB_MAX_COMMIT_TIME_INTERVAL);
return TSDB_CODE_INVALID_OPTION;
}
if (pCreate->compression < TSDB_MIN_COMPRESSION_LEVEL || pCreate->compression > TSDB_MAX_COMPRESSION_LEVEL) {
mError("invalid db option compression: %d valid range: [%d, %d]", pCreate->compression, TSDB_MIN_COMPRESSION_LEVEL,
TSDB_MAX_COMPRESSION_LEVEL);
return TSDB_CODE_INVALID_OPTION;
}
return TSDB_CODE_SUCCESS;
}

3
src/system/detail/src/mgmtVgroup.c
View File

@ -19,6 +19,7 @@
#include "mgmt.h"
#include "tschemautil.h"
#include "tlog.h"
#include "tstatus.h"
void * vgSdb = NULL;
int tsVgUpdateSize;
@ -312,7 +313,7 @@ int mgmtRetrieveVgroups(SShowObj *pShow, char *data, int rows, SConnObj *pConn)
cols++;
pWrite = data + pShow->offset[cols] * rows + pShow->bytes[cols] * numOfRows;
strcpy(pWrite, pVgroup->lbState ? "updating" : "ready");
strcpy(pWrite, taosGetVgroupLbStatusStr(pVgroup->lbStatus));
cols++;
for (int i = 0; i < maxReplica; ++i) {

7
src/system/detail/src/vnodeCache.c
View File

@ -20,6 +20,7 @@
#include "vnode.h"
#include "vnodeCache.h"
#include "vnodeUtil.h"
#include "tstatus.h"
void vnodeSearchPointInCache(SMeterObj *pObj, SQuery *pQuery);
void vnodeProcessCommitTimer(void *param, void *tmrId);
@ -77,7 +78,7 @@ void *vnodeOpenCachePool(int vnode) {
}
}
dTrace("vid:%d, cache pool is allocated:0x%x", vnode, pCachePool);
dPrint("vid:%d, cache pool is allocated:0x%x", vnode, pCachePool);
return pCachePool;
@ -102,7 +103,7 @@ void vnodeCloseCachePool(int vnode) {
taosTmrStopA(&pVnode->commitTimer);
if (pVnode->commitInProcess) pthread_cancel(pVnode->commitThread);
dTrace("vid:%d, cache pool closed, count:%d", vnode, pCachePool->count);
dPrint("vid:%d, cache pool closed, count:%d", vnode, pCachePool->count);
int maxAllocBlock = (1024 * 1024 * 1024) / pVnode->cfg.cacheBlockSize;
while (blockId < pVnode->cfg.cacheNumOfBlocks.totalBlocks) {
@ -297,7 +298,7 @@ pthread_t vnodeCreateCommitThread(SVnodeObj *pVnode) {
taosTmrStopA(&pVnode->commitTimer);
if (pVnode->vnodeStatus == TSDB_VNODE_STATUS_UNSYNCED) {
if (pVnode->vnodeStatus == TSDB_VN_STATUS_UNSYNCED) {
taosTmrReset(vnodeProcessCommitTimer, pVnode->cfg.commitTime * 1000, pVnode, vnodeTmrCtrl, &pVnode->commitTimer);
dTrace("vid:%d, it is in unsyc state, commit later", pVnode->vnode);
return pVnode->commitThread;

2
src/system/detail/src/vnodeCommit.c
View File

@ -235,7 +235,7 @@ int vnodeInitCommit(int vnode) {
}
pVnode->pWrite += size;
dTrace("vid:%d, commit log is initialized", vnode);
dPrint("vid:%d, commit log is initialized", vnode);
return 0;
}

10
src/system/detail/src/vnodeFile.c
View File

@ -125,10 +125,8 @@ int vnodeCreateHeadDataFile(int vnode, int fileId, char *headName, char *dataNam
if (symlink(dDataName, dataName) != 0) return -1;
if (symlink(dLastName, lastName) != 0) return -1;
dTrace(
"vid:%d, fileId:%d, empty header file:%s dataFile:%s lastFile:%s on "
"disk:%s is created ",
vnode, fileId, headName, dataName, lastName, path);
dPrint("vid:%d, fileId:%d, empty header file:%s dataFile:%s lastFile:%s on disk:%s is created ",
vnode, fileId, headName, dataName, lastName, path);
return 0;
}
@ -322,7 +320,7 @@ int vnodeOpenCommitFiles(SVnodeObj *pVnode, int noTempLast) {
vnodeRecoverFromPeer(pVnode, fileId);
goto _error;
} else {
dTrace("vid:%d, data file:%s is opened to write", vnode, name);
dPrint("vid:%d, data file:%s is opened to write", vnode, name);
}
// open last file
@ -427,7 +425,7 @@ void vnodeRemoveFile(int vnode, int fileId) {
remove(dDataName);
remove(dLastName);
dTrace("vid:%d fileId:%d on disk: %s is removed, numOfFiles:%d maxFiles:%d", vnode, fileId, path,
dPrint("vid:%d fileId:%d on disk: %s is removed, numOfFiles:%d maxFiles:%d", vnode, fileId, path,
pVnode->numOfFiles, pVnode->maxFiles);
}

24
src/system/detail/src/vnodeMeter.c
View File

@ -24,6 +24,7 @@
#include "vnodeMgmt.h"
#include "vnodeShell.h"
#include "vnodeUtil.h"
#include "tstatus.h"
#pragma GCC diagnostic ignored "-Wpointer-sign"
@ -112,7 +113,7 @@ FILE *vnodeOpenMeterObjFile(int vnode) {
fp = fopen(fileName, "r+");
if (fp != NULL) {
if (vnodeCheckFileIntegrity(fp) < 0) {
dError("file:%s is corrupted, need to restore it first", fileName);
dError("file:%s is corrupted, need to restore it first, exit program", fileName);
fclose(fp);
// todo: how to recover
@ -376,7 +377,11 @@ int vnodeOpenMetersVnode(int vnode) {
fseek(fp, TSDB_FILE_HEADER_LEN * 2 / 4, SEEK_SET);
fread(&pVnode->cfg, sizeof(SVnodeCfg), 1, fp);
if (vnodeIsValidVnodeCfg(&pVnode->cfg) == false) {
dError("vid:%d, maxSessions:%d cacheBlockSize:%d replications:%d daysPerFile:%d daysToKeep:%d invalid, clear it",
vnode, pVnode->cfg.maxSessions, pVnode->cfg.cacheBlockSize, pVnode->cfg.replications,
pVnode->cfg.daysPerFile, pVnode->cfg.daysToKeep);
pVnode->cfg.maxSessions = 0; // error in vnode file
return 0;
}
@ -484,7 +489,7 @@ int vnodeCreateMeterObj(SMeterObj *pNew, SConnSec *pSec) {
vnodeSaveMeterObjToFile(pNew);
// vnodeCreateMeterMgmt(pNew, pSec);
vnodeCreateStream(pNew);
dTrace("vid:%d sid:%d id:%s, meterObj is created, uid:%ld", pNew->vnode, pNew->sid, pNew->meterId, pNew->uid);
dTrace("vid:%d, sid:%d id:%s, meterObj is created, uid:%ld", pNew->vnode, pNew->sid, pNew->meterId, pNew->uid);
}
return code;
@ -572,7 +577,7 @@ int vnodeInsertPoints(SMeterObj *pObj, char *cont, int contLen, char source, voi
dTrace("vid:%d sid:%d id:%s, cache is full, freePoints:%d, notFreeSlots:%d", pObj->vnode, pObj->sid, pObj->meterId,
pObj->freePoints, pPool->notFreeSlots);
vnodeProcessCommitTimer(pVnode, NULL);
return TSDB_CODE_ACTION_IN_PROGRESS;
return code;
}
// FIXME: Here should be after the comparison of sversions.
@ -596,7 +601,7 @@ int vnodeInsertPoints(SMeterObj *pObj, char *cont, int contLen, char source, voi
}
pData = pSubmit->payLoad;
code = 0;
code = TSDB_CODE_SUCCESS;
TSKEY firstKey = *((TSKEY *)pData);
TSKEY lastKey = *((TSKEY *)(pData + pObj->bytesPerPoint * (numOfPoints - 1)));
@ -609,8 +614,11 @@ int vnodeInsertPoints(SMeterObj *pObj, char *cont, int contLen, char source, voi
return TSDB_CODE_TIMESTAMP_OUT_OF_RANGE;
}
for (i = 0; i < numOfPoints; ++i) {
// meter will be dropped, abort current insertion
if ((code = vnodeSetMeterInsertImportStateEx(pObj, TSDB_METER_STATE_INSERT)) != TSDB_CODE_SUCCESS) {
goto _over;
}
for (i = 0; i < numOfPoints; ++i) { // meter will be dropped, abort current insertion
if (pObj->state >= TSDB_METER_STATE_DELETING) {
dWarn("vid:%d sid:%d id:%s, meter is dropped, abort insert, state:%d", pObj->vnode, pObj->sid, pObj->meterId,
pObj->state);
@ -652,6 +660,7 @@ int vnodeInsertPoints(SMeterObj *pObj, char *cont, int contLen, char source, voi
pVnode->version++;
pthread_mutex_unlock(&(pVnode->vmutex));
vnodeClearMeterState(pObj, TSDB_METER_STATE_INSERT);
_over:
dTrace("vid:%d sid:%d id:%s, %d out of %d points are inserted, lastKey:%ld source:%d, vnode total storage: %ld",
@ -709,7 +718,8 @@ void vnodeUpdateMeter(void *param, void *tmrId) {
SVnodeObj* pVnode = &vnodeList[pNew->vnode];
if (pVnode->meterList == NULL) {
dTrace("vid:%d sid:%d id:%s, vnode is deleted, abort update schema", pNew->vnode, pNew->sid, pNew->meterId);
dTrace("vid:%d sid:%d id:%s, vnode is deleted, status:%s, abort update schema",
pNew->vnode, pNew->sid, pNew->meterId, taosGetVnodeStatusStr(vnodeList[pNew->vnode].vnodeStatus));
free(pNew->schema);
free(pNew);
return;

307
src/system/detail/src/vnodeQueryImpl.c
View File

@ -47,7 +47,7 @@ static int32_t readDataFromDiskFile(int fd, SQInfo *pQInfo, SQueryFileInfo *pQue
__read_data_fn_t readDataFunctor[2] = {copyDataFromMMapBuffer, readDataFromDiskFile};
static void vnodeInitLoadCompBlockInfo(SQueryLoadCompBlockInfo *pCompBlockLoadInfo);
static void vnodeInitLoadCompBlockInfo(SQueryLoadCompBlockInfo *pCompBlockLoadInfo);
static int32_t moveToNextBlock(SQueryRuntimeEnv *pRuntimeEnv, int32_t step, __block_search_fn_t searchFn,
bool loadData);
static int32_t doMergeMetersResultsToGroupRes(SMeterQuerySupportObj *pSupporter, SQuery *pQuery,
@ -57,7 +57,7 @@ static int32_t doMergeMetersResultsToGroupRes(SMeterQuerySupportObj *pSupporter,
static TSKEY getTimestampInCacheBlock(SCacheBlock *pBlock, int32_t index);
static TSKEY getTimestampInDiskBlock(SQueryRuntimeEnv *pRuntimeEnv, int32_t index);
static void savePointPosition(SPositionInfo *position, int32_t fileId, int32_t slot, int32_t pos);
static void savePointPosition(SPositionInfo *position, int32_t fileId, int32_t slot, int32_t pos);
static int32_t getNextDataFileCompInfo(SQueryRuntimeEnv *pRuntimeEnv, SMeterObj *pMeterObj, int32_t step);
static void setGroupOutputBuffer(SQueryRuntimeEnv *pRuntimeEnv, SOutputRes *pResult);
@ -195,7 +195,7 @@ static bool vnodeIsCompBlockInfoLoaded(SQueryRuntimeEnv *pRuntimeEnv, SMeterObj
// if vnodeFreeFields is called, the pQuery->pFields is NULL
if (pLoadCompBlockInfo->fileListIndex == fileIndex && pLoadCompBlockInfo->sid == pMeterObj->sid &&
pQuery->pFields != NULL && pQuery->fileId > 0) {
assert(pRuntimeEnv->pHeaderFiles[fileIndex].fileID == pLoadCompBlockInfo->fileId && pQuery->numOfBlocks > 0);
assert(pRuntimeEnv->pVnodeFiles[fileIndex].fileID == pLoadCompBlockInfo->fileId && pQuery->numOfBlocks > 0);
return true;
}
@ -207,7 +207,7 @@ static void vnodeSetCompBlockInfoLoaded(SQueryRuntimeEnv *pRuntimeEnv, int32_t f
pLoadCompBlockInfo->sid = sid;
pLoadCompBlockInfo->fileListIndex = fileIndex;
pLoadCompBlockInfo->fileId = pRuntimeEnv->pHeaderFiles[fileIndex].fileID;
pLoadCompBlockInfo->fileId = pRuntimeEnv->pVnodeFiles[fileIndex].fileID;
}
static void vnodeInitLoadCompBlockInfo(SQueryLoadCompBlockInfo *pCompBlockLoadInfo) {
@ -247,6 +247,60 @@ static void vnodeInitDataBlockInfo(SQueryLoadBlockInfo *pBlockLoadInfo) {
pBlockLoadInfo->fileListIndex = -1;
}
static void doUnmapHeaderFileData(SQueryRuntimeEnv* pRuntimeEnv) {
if (pRuntimeEnv->mmapedHFileIndex >= 0) {
assert(pRuntimeEnv->mmapedHFileIndex < pRuntimeEnv->numOfFiles && pRuntimeEnv->mmapedHFileIndex >= 0);
SQueryFileInfo *otherVnodeFiles = &pRuntimeEnv->pVnodeFiles[pRuntimeEnv->mmapedHFileIndex];
munmap(otherVnodeFiles->pHeaderFileData, otherVnodeFiles->headFileSize);
otherVnodeFiles->pHeaderFileData = NULL;
pRuntimeEnv->mmapedHFileIndex = -1;
}
assert(pRuntimeEnv->mmapedHFileIndex == -1);
}
/**
* mmap the data file into memory. For each query, only one header file is allowed to mmap into memory, in order to
* avoid too many mmapped files at the save time to cause OS return the message of "Cannot allocate memory",
* during query processing.
*
* @param pRuntimeEnv
* @param fileIndex
* @return the return value may be null, so any invoker needs to check the returned value
*/
char *vnodeGetHeaderFileData(SQueryRuntimeEnv *pRuntimeEnv, int32_t fileIndex) {
assert(fileIndex >= 0 && fileIndex < pRuntimeEnv->numOfFiles);
SQuery *pQuery = pRuntimeEnv->pQuery;
SQInfo *pQInfo = (SQInfo *)GET_QINFO_ADDR(pQuery); // only for log output
SQueryFileInfo *pVnodeFiles = &pRuntimeEnv->pVnodeFiles[fileIndex];
size_t size = pVnodeFiles->headFileSize;
if (pVnodeFiles->pHeaderFileData == NULL) {
assert(pRuntimeEnv->mmapedHFileIndex != fileIndex);
doUnmapHeaderFileData(pRuntimeEnv); // do close the other mmaped header file
pVnodeFiles->pHeaderFileData = mmap(NULL, size, PROT_READ, MAP_SHARED, pVnodeFiles->headerFd, 0);
if (pVnodeFiles->pHeaderFileData == MAP_FAILED) {
pVnodeFiles->pHeaderFileData = NULL;
dError("QInfo:%p failed to mmap header file:%s, size:%lld, %s", pQInfo, pVnodeFiles->headerFilePath, size,
strerror(errno));
} else {
pRuntimeEnv->mmapedHFileIndex = fileIndex; // set the value in case of success mmap file
if (madvise(pVnodeFiles->pHeaderFileData, size, MADV_SEQUENTIAL) == -1) {
dError("QInfo:%p failed to advise kernel the usage of header file, reason:%s", pQInfo, strerror(errno));
}
}
} else {
assert(pRuntimeEnv->mmapedHFileIndex == fileIndex);
}
return pVnodeFiles->pHeaderFileData;
}
/*
* read comp block info from header file
*
@ -256,8 +310,7 @@ static int vnodeGetCompBlockInfo(SMeterObj *pMeterObj, SQueryRuntimeEnv *pRuntim
SQInfo *pQInfo = (SQInfo *)GET_QINFO_ADDR(pQuery);
SVnodeCfg * pCfg = &vnodeList[pMeterObj->vnode].cfg;
SQueryFileInfo *pQueryFileInfo = &pRuntimeEnv->pHeaderFiles[fileIndex];
int32_t fd = pQueryFileInfo->headerFd;
SQueryFileInfo *pQueryFileInfo = &pRuntimeEnv->pVnodeFiles[fileIndex];
int64_t st = taosGetTimestampUs();
@ -273,8 +326,11 @@ static int vnodeGetCompBlockInfo(SMeterObj *pMeterObj, SQueryRuntimeEnv *pRuntim
pSummary->numOfSeek++;
#if 1
char *data = pRuntimeEnv->pHeaderFiles[fileIndex].pHeaderFileData;
UNUSED(fd);
char *data = vnodeGetHeaderFileData(pRuntimeEnv, fileIndex);
if (data == NULL) {
return -1; // failed to load the header file data into memory
}
#else
char *data = calloc(1, tmsize + TSDB_FILE_HEADER_LEN);
read(fd, data, tmsize + TSDB_FILE_HEADER_LEN);
@ -353,7 +409,7 @@ static int vnodeGetCompBlockInfo(SMeterObj *pMeterObj, SQueryRuntimeEnv *pRuntim
int64_t et = taosGetTimestampUs();
qTrace("QInfo:%p vid:%d sid:%d id:%s, fileId:%d, load compblock info, size:%d, elapsed:%f ms", pQInfo,
pMeterObj->vnode, pMeterObj->sid, pMeterObj->meterId, pRuntimeEnv->pHeaderFiles[fileIndex].fileID,
pMeterObj->vnode, pMeterObj->sid, pMeterObj->meterId, pRuntimeEnv->pVnodeFiles[fileIndex].fileID,
compBlockSize, (et - st) / 1000.0);
pSummary->totalCompInfoSize += compBlockSize;
@ -379,7 +435,7 @@ static void setExecParams(SQuery *pQuery, SQLFunctionCtx *pCtx, int64_t StartQue
char *primaryColumnData, int32_t size, int32_t functionId, SField *pField, bool hasNull,
int32_t blockStatus, void *param, int32_t scanFlag);
void createGroupResultBuf(SQuery *pQuery, SOutputRes *pOneResult, bool isMetricQuery);
void createGroupResultBuf(SQuery *pQuery, SOutputRes *pOneResult, bool isMetricQuery);
static void destroyGroupResultBuf(SOutputRes *pOneOutputRes, int32_t nOutputCols);
static int32_t binarySearchForBlockImpl(SCompBlock *pBlock, int32_t numOfBlocks, TSKEY skey, int32_t order) {
@ -439,10 +495,10 @@ static int32_t moveMMapWindow(SQueryFileInfo *pQueryFileInfo, uint64_t offset) {
}
/*
* 1. there is import data that locate farther from the beginning, but with less timestamp, so we need to move the
* window backwards
* 2. otherwise, move the mmaping window forward
*/
* 1. there is import data that locate farther from the beginning, but with less timestamp, so we need to move the
* window backwards
* 2. otherwise, move the mmaping window forward
*/
upperBnd = (offset / pQueryFileInfo->defaultMappingSize + 1) * pQueryFileInfo->defaultMappingSize - 1;
/* unmap previous buffer */
@ -636,7 +692,7 @@ static int32_t loadDataBlockIntoMem(SCompBlock *pBlock, SField **pField, SQueryR
SMeterObj *pMeterObj = pRuntimeEnv->pMeterObj;
SData ** sdata = pRuntimeEnv->colDataBuffer;
SQueryFileInfo *pQueryFileInfo = &pRuntimeEnv->pHeaderFiles[fileIdx];
SQueryFileInfo *pQueryFileInfo = &pRuntimeEnv->pVnodeFiles[fileIdx];
SData ** primaryTSBuf = &pRuntimeEnv->primaryColBuffer;
void * tmpBuf = pRuntimeEnv->unzipBuffer;
@ -850,9 +906,11 @@ SCacheBlock *getCacheDataBlock(SMeterObj *pMeterObj, SQuery *pQuery, int32_t slo
}
if (pMeterObj != pBlock->pMeterObj || pBlock->blockId > pQuery->blockId) {
dWarn("QInfo:%p vid:%d sid:%d id:%s, cache block is overwritten, slot:%d blockId:%d qBlockId:%d, meterObj:%p, blockMeterObj:%p",
GET_QINFO_ADDR(pQuery), pMeterObj->vnode, pMeterObj->sid, pMeterObj->meterId, pQuery->slot, pBlock->blockId,
pQuery->blockId, pMeterObj, pBlock->pMeterObj);
dWarn(
"QInfo:%p vid:%d sid:%d id:%s, cache block is overwritten, slot:%d blockId:%d qBlockId:%d, meterObj:%p, "
"blockMeterObj:%p",
GET_QINFO_ADDR(pQuery), pMeterObj->vnode, pMeterObj->sid, pMeterObj->meterId, pQuery->slot, pBlock->blockId,
pQuery->blockId, pMeterObj, pBlock->pMeterObj);
return NULL;
}
@ -938,7 +996,7 @@ static bool getQualifiedDataBlock(SMeterObj *pMeterObj, SQueryRuntimeEnv *pRunti
break;
}
dError("QInfo:%p fileId:%d total numOfBlks:%d blockId:%d into memory failed due to error in disk files",
dError("QInfo:%p fileId:%d total numOfBlks:%d blockId:%d load into memory failed due to error in disk files",
GET_QINFO_ADDR(pQuery), pQuery->fileId, pQuery->numOfBlocks, blkIdx);
blkIdx += step;
}
@ -1110,9 +1168,9 @@ static int32_t blockwiseApplyAllFunctions(SQueryRuntimeEnv *pRuntimeEnv, int32_t
for (int32_t k = 0; k < pQuery->numOfOutputCols; ++k) {
int32_t functionId = pQuery->pSelectExpr[k].pBase.functionId;
// if (!functionNeedToExecute(pRuntimeEnv, &pCtx[k], functionId)) {
// continue;
// }
// if (!functionNeedToExecute(pRuntimeEnv, &pCtx[k], functionId)) {
// continue;
// }
SField dummyField = {0};
@ -1135,8 +1193,8 @@ static int32_t blockwiseApplyAllFunctions(SQueryRuntimeEnv *pRuntimeEnv, int32_t
TSKEY ts = QUERY_IS_ASC_QUERY(pQuery) ? pQuery->skey : pQuery->ekey;
int64_t alignedTimestamp = taosGetIntervalStartTimestamp(ts, pQuery->nAggTimeInterval, pQuery->intervalTimeUnit,
pQuery->precision);
int64_t alignedTimestamp =
taosGetIntervalStartTimestamp(ts, pQuery->nAggTimeInterval, pQuery->intervalTimeUnit, pQuery->precision);
setExecParams(pQuery, &pCtx[k], alignedTimestamp, dataBlock, (char *)primaryKeyCol, forwardStep, functionId,
tpField, hasNull, pRuntimeEnv->blockStatus, &sasArray[k], pRuntimeEnv->scanFlag);
}
@ -1405,8 +1463,8 @@ static int32_t rowwiseApplyAllFunctions(SQueryRuntimeEnv *pRuntimeEnv, int32_t *
char *dataBlock = getDataBlocks(pRuntimeEnv, data, &sasArray[k], k, *forwardStep, isDiskFileBlock);
TSKEY ts = QUERY_IS_ASC_QUERY(pQuery) ? pQuery->skey : pQuery->ekey;
int64_t alignedTimestamp = taosGetIntervalStartTimestamp(ts, pQuery->nAggTimeInterval, pQuery->intervalTimeUnit,
pQuery->precision);
int64_t alignedTimestamp =
taosGetIntervalStartTimestamp(ts, pQuery->nAggTimeInterval, pQuery->intervalTimeUnit, pQuery->precision);
setExecParams(pQuery, &pCtx[k], alignedTimestamp, dataBlock, (char *)primaryKeyCol, (*forwardStep), functionId,
pFields, hasNull, pRuntimeEnv->blockStatus, &sasArray[k], pRuntimeEnv->scanFlag);
@ -1655,18 +1713,18 @@ int32_t vnodeGetVnodeHeaderFileIdx(int32_t *fid, SQueryRuntimeEnv *pRuntimeEnv,
}
/* set the initial file for current query */
if (order == TSQL_SO_ASC && *fid < pRuntimeEnv->pHeaderFiles[0].fileID) {
*fid = pRuntimeEnv->pHeaderFiles[0].fileID;
if (order == TSQL_SO_ASC && *fid < pRuntimeEnv->pVnodeFiles[0].fileID) {
*fid = pRuntimeEnv->pVnodeFiles[0].fileID;
return 0;
} else if (order == TSQL_SO_DESC && *fid > pRuntimeEnv->pHeaderFiles[pRuntimeEnv->numOfFiles - 1].fileID) {
*fid = pRuntimeEnv->pHeaderFiles[pRuntimeEnv->numOfFiles - 1].fileID;
} else if (order == TSQL_SO_DESC && *fid > pRuntimeEnv->pVnodeFiles[pRuntimeEnv->numOfFiles - 1].fileID) {
*fid = pRuntimeEnv->pVnodeFiles[pRuntimeEnv->numOfFiles - 1].fileID;
return pRuntimeEnv->numOfFiles - 1;
}
int32_t numOfFiles = pRuntimeEnv->numOfFiles;
if (order == TSQL_SO_DESC && *fid > pRuntimeEnv->pHeaderFiles[numOfFiles - 1].fileID) {
*fid = pRuntimeEnv->pHeaderFiles[numOfFiles - 1].fileID;
if (order == TSQL_SO_DESC && *fid > pRuntimeEnv->pVnodeFiles[numOfFiles - 1].fileID) {
*fid = pRuntimeEnv->pVnodeFiles[numOfFiles - 1].fileID;
return numOfFiles - 1;
}
@ -1674,12 +1732,12 @@ int32_t vnodeGetVnodeHeaderFileIdx(int32_t *fid, SQueryRuntimeEnv *pRuntimeEnv,
int32_t i = 0;
int32_t step = 1;
while (i<numOfFiles && * fid> pRuntimeEnv->pHeaderFiles[i].fileID) {
while (i<numOfFiles && * fid> pRuntimeEnv->pVnodeFiles[i].fileID) {
i += step;
}
if (i < numOfFiles && *fid <= pRuntimeEnv->pHeaderFiles[i].fileID) {
*fid = pRuntimeEnv->pHeaderFiles[i].fileID;
if (i < numOfFiles && *fid <= pRuntimeEnv->pVnodeFiles[i].fileID) {
*fid = pRuntimeEnv->pVnodeFiles[i].fileID;
return i;
} else {
return -1;
@ -1688,12 +1746,12 @@ int32_t vnodeGetVnodeHeaderFileIdx(int32_t *fid, SQueryRuntimeEnv *pRuntimeEnv,
int32_t i = numOfFiles - 1;
int32_t step = -1;
while (i >= 0 && *fid < pRuntimeEnv->pHeaderFiles[i].fileID) {
while (i >= 0 && *fid < pRuntimeEnv->pVnodeFiles[i].fileID) {
i += step;
}
if (i >= 0 && *fid >= pRuntimeEnv->pHeaderFiles[i].fileID) {
*fid = pRuntimeEnv->pHeaderFiles[i].fileID;
if (i >= 0 && *fid >= pRuntimeEnv->pVnodeFiles[i].fileID) {
*fid = pRuntimeEnv->pVnodeFiles[i].fileID;
return i;
} else {
return -1;
@ -1723,6 +1781,7 @@ int32_t getNextDataFileCompInfo(SQueryRuntimeEnv *pRuntimeEnv, SMeterObj *pMeter
break;
}
// failed to mmap header file into memory will cause the retrieval of compblock info failed
if (vnodeGetCompBlockInfo(pMeterObj, pRuntimeEnv, fid) > 0) {
break;
}
@ -1821,13 +1880,13 @@ void setExecParams(SQuery *pQuery, SQLFunctionCtx *pCtx, int64_t startQueryTimes
}
// set the output buffer for the selectivity + tag query
static void setCtxTagColumnInfo(SQuery* pQuery, SQueryRuntimeEnv* pRuntimeEnv) {
static void setCtxTagColumnInfo(SQuery *pQuery, SQueryRuntimeEnv *pRuntimeEnv) {
if (isSelectivityWithTagsQuery(pQuery)) {
int32_t num = 0;
SQLFunctionCtx *pCtx = NULL;
int16_t tagLen = 0;
SQLFunctionCtx ** pTagCtx = calloc(pQuery->numOfOutputCols, POINTER_BYTES);
SQLFunctionCtx **pTagCtx = calloc(pQuery->numOfOutputCols, POINTER_BYTES);
for (int32_t i = 0; i < pQuery->numOfOutputCols; ++i) {
SSqlFuncExprMsg *pSqlFuncMsg = &pQuery->pSelectExpr[i].pBase;
if (pSqlFuncMsg->functionId == TSDB_FUNC_TAG_DUMMY || pSqlFuncMsg->functionId == TSDB_FUNC_TS_DUMMY) {
@ -1973,7 +2032,7 @@ _error_clean:
tfree(pRuntimeEnv->resultInfo);
tfree(pRuntimeEnv->pCtx);
for(int32_t i = 0; i < pRuntimeEnv->pQuery->numOfCols; ++i) {
for (int32_t i = 0; i < pRuntimeEnv->pQuery->numOfCols; ++i) {
tfree(pRuntimeEnv->colDataBuffer[i]);
}
@ -1993,7 +2052,7 @@ static void teardownQueryRuntimeEnv(SQueryRuntimeEnv *pRuntimeEnv) {
}
dTrace("QInfo:%p teardown runtime env", GET_QINFO_ADDR(pRuntimeEnv->pQuery));
for(int32_t i = 0; i < pRuntimeEnv->pQuery->numOfCols; ++i) {
for (int32_t i = 0; i < pRuntimeEnv->pQuery->numOfCols; ++i) {
tfree(pRuntimeEnv->colDataBuffer[i]);
}
@ -2024,7 +2083,7 @@ static void teardownQueryRuntimeEnv(SQueryRuntimeEnv *pRuntimeEnv) {
}
for (int32_t i = 0; i < pRuntimeEnv->numOfFiles; ++i) {
SQueryFileInfo *pQFileInfo = &(pRuntimeEnv->pHeaderFiles[i]);
SQueryFileInfo *pQFileInfo = &(pRuntimeEnv->pVnodeFiles[i]);
if (pQFileInfo->pHeaderFileData != NULL && pQFileInfo->pHeaderFileData != MAP_FAILED) {
munmap(pQFileInfo->pHeaderFileData, pQFileInfo->headFileSize);
}
@ -2038,9 +2097,9 @@ static void teardownQueryRuntimeEnv(SQueryRuntimeEnv *pRuntimeEnv) {
tclose(pQFileInfo->lastFd);
}
if (pRuntimeEnv->pHeaderFiles != NULL) {
if (pRuntimeEnv->pVnodeFiles != NULL) {
pRuntimeEnv->numOfFiles = 0;
free(pRuntimeEnv->pHeaderFiles);
free(pRuntimeEnv->pVnodeFiles);
}
if (pRuntimeEnv->pInterpoBuf != NULL) {
@ -2783,19 +2842,24 @@ int64_t loadRequiredBlockIntoMem(SQueryRuntimeEnv *pRuntimeEnv, SPositionInfo *p
* currently opened file is not the start file, reset to the start file
*/
int32_t fileIdx = vnodeGetVnodeHeaderFileIdx(&pQuery->fileId, pRuntimeEnv, pQuery->order.order);
if (fileIdx < 0) {
if (fileIdx < 0) { // ignore the files on disk
dError("QInfo:%p failed to get data file:%d", GET_QINFO_ADDR(pQuery), pQuery->fileId);
// ignore the files on disk
position->fileId = -1;
return -1;
}
/*
* NOTE: the compblock information may not be loaded yet, here loaded it firstly
* NOTE:
* The compblock information may not be loaded yet, here loaded it firstly.
* If the compBlock info is loaded, it wont be loaded again.
*
* If failed to load comp block into memory due some how reasons, e.g., empty header file/not enough memory
*/
int32_t numOfBlocks = vnodeGetCompBlockInfo(pMeterObj, pRuntimeEnv, fileIdx);
assert(numOfBlocks > 0);
if (numOfBlocks <= 0) {
position->fileId = -1;
return -1;
}
nextTimestamp = getTimestampInDiskBlock(pRuntimeEnv, pQuery->pos);
}
@ -2865,71 +2929,28 @@ static int file_order_comparator(const void *p1, const void *p2) {
*/
static int32_t vnodeOpenVnodeDBFiles(SQInfo *pQInfo, SQueryFileInfo *pVnodeFiles, int32_t fid, int32_t vnodeId,
char *fileName, char *prefix) {
__off_t size = 0;
// __off_t size = 0;
pVnodeFiles->fileID = fid;
pVnodeFiles->defaultMappingSize = DEFAULT_DATA_FILE_MMAP_WINDOW_SIZE;
snprintf(pVnodeFiles->headerFilePath, 256, "%s%s", prefix, fileName);
#if 1
pVnodeFiles->headerFd = open(pVnodeFiles->headerFilePath, O_RDONLY);
#else
int32_t *val = (int32_t *)taosGetStrHashData(fileHandleHashList, pVnodeFiles->headerFilePath);
if (val == NULL) {
pVnodeFiles->headerFd = open(pVnodeFiles->headerFilePath, O_RDONLY);
taosAddStrHash(fileHandleHashList, pVnodeFiles->headerFilePath, (char *)&pVnodeFiles->headerFd);
} else {
pVnodeFiles->headerFd = *val;
}
#endif
if (!VALIDFD(pVnodeFiles->headerFd)) {
dError("QInfo:%p failed open header file:%s reason:%s", pQInfo, pVnodeFiles->headerFilePath, strerror(errno));
goto _clean;
}
struct stat fstat;
struct stat fstat = {0};
if (stat(pVnodeFiles->headerFilePath, &fstat) < 0) return -1;
pVnodeFiles->headFileSize = fstat.st_size;
size = fstat.st_size;
pVnodeFiles->pHeaderFileData = mmap(NULL, size, PROT_READ, MAP_SHARED, pVnodeFiles->headerFd, 0);
if (pVnodeFiles->pHeaderFileData == MAP_FAILED) {
dError("QInfo:%p failed to map header file:%s, %s", pQInfo, pVnodeFiles->headerFilePath, strerror(errno));
goto _clean;
}
/* even the advise failed, continue.. */
if (madvise(pVnodeFiles->pHeaderFileData, size, MADV_SEQUENTIAL) == -1) {
dError("QInfo:%p failed to advise kernel the usage of header files, reason:%s", pQInfo, strerror(errno));
}
snprintf(pVnodeFiles->dataFilePath, 256, "%sv%df%d.data", prefix, vnodeId, fid);
snprintf(pVnodeFiles->lastFilePath, 256, "%sv%df%d.last", prefix, vnodeId, fid);
#if 1
pVnodeFiles->dataFd = open(pVnodeFiles->dataFilePath, O_RDONLY);
pVnodeFiles->lastFd = open(pVnodeFiles->lastFilePath, O_RDONLY);
#else
val = (int32_t *)taosGetStrHashData(fileHandleHashList, pVnodeFiles->dataFilePath);
if (val == NULL) {
pVnodeFiles->dataFd = open(pVnodeFiles->dataFilePath, O_RDONLY);
taosAddStrHash(fileHandleHashList, pVnodeFiles->dataFilePath, (char *)&pVnodeFiles->dataFd);
} else {
pVnodeFiles->dataFd = *val;
}
#endif
if (!VALIDFD(pVnodeFiles->dataFd)) {
dError("QInfo:%p failed to open data file:%s, reason:%s", pQInfo, pVnodeFiles->dataFilePath, strerror(errno));
goto _clean;
}
if (!VALIDFD(pVnodeFiles->lastFd)) {
dError("QInfo:%p failed to open last file:%s, reason:%s", pQInfo, pVnodeFiles->lastFilePath, strerror(errno));
goto _clean;
}
if (stat(pVnodeFiles->dataFilePath, &fstat) < 0) return -1;
pVnodeFiles->dataFileSize = fstat.st_size;
@ -2953,13 +2974,9 @@ static int32_t vnodeOpenVnodeDBFiles(SQInfo *pQInfo, SQueryFileInfo *pVnodeFiles
}
#endif
return 0;
return TSDB_CODE_SUCCESS;
_clean:
if (pVnodeFiles->pHeaderFileData != MAP_FAILED && pVnodeFiles->pDataFileData != NULL) {
munmap(pVnodeFiles->pHeaderFileData, pVnodeFiles->headFileSize);
pVnodeFiles->pHeaderFileData = NULL;
}
#if DEFAULT_IO_ENGINE == IO_ENGINE_MMAP
if (pVnodeFiles->pDataFileData != MAP_FAILED && pVnodeFiles->pDataFileData != NULL) {
@ -2987,10 +3004,10 @@ static void vnodeOpenAllFiles(SQInfo *pQInfo, int32_t vnodeId) {
char suffix[] = ".head";
struct dirent *pEntry = NULL;
int32_t alloc = 4; // default allocated size
size_t alloc = 4; // default allocated size
SQueryRuntimeEnv *pRuntimeEnv = &(pQInfo->pMeterQuerySupporter->runtimeEnv);
pRuntimeEnv->pHeaderFiles = calloc(1, sizeof(SQueryFileInfo) * alloc);
pRuntimeEnv->pVnodeFiles = calloc(1, sizeof(SQueryFileInfo) * alloc);
SVnodeObj *pVnode = &vnodeList[vnodeId];
while ((pEntry = readdir(pDir)) != NULL) {
@ -3026,11 +3043,11 @@ static void vnodeOpenAllFiles(SQInfo *pQInfo, int32_t vnodeId) {
if (++pRuntimeEnv->numOfFiles > alloc) {
alloc = alloc << 1;
pRuntimeEnv->pHeaderFiles = realloc(pRuntimeEnv->pHeaderFiles, alloc * sizeof(SQueryFileInfo));
memset(&pRuntimeEnv->pHeaderFiles[alloc >> 1], 0, (alloc >> 1) * sizeof(SQueryFileInfo));
pRuntimeEnv->pVnodeFiles = realloc(pRuntimeEnv->pVnodeFiles, alloc * sizeof(SQueryFileInfo));
memset(&pRuntimeEnv->pVnodeFiles[alloc >> 1], 0, (alloc >> 1) * sizeof(SQueryFileInfo));
}
SQueryFileInfo *pVnodeFiles = &pRuntimeEnv->pHeaderFiles[pRuntimeEnv->numOfFiles - 1];
SQueryFileInfo *pVnodeFiles = &pRuntimeEnv->pVnodeFiles[pRuntimeEnv->numOfFiles - 1];
int32_t ret = vnodeOpenVnodeDBFiles(pQInfo, pVnodeFiles, fid, vnodeId, pEntry->d_name, dbFilePathPrefix);
if (ret < 0) {
memset(pVnodeFiles, 0, sizeof(SQueryFileInfo)); // reset information
@ -3043,7 +3060,7 @@ static void vnodeOpenAllFiles(SQInfo *pQInfo, int32_t vnodeId) {
dTrace("QInfo:%p find %d data files in %s to be checked", pQInfo, pRuntimeEnv->numOfFiles, dbFilePathPrefix);
/* order the files information according their names */
qsort(pRuntimeEnv->pHeaderFiles, (size_t)pRuntimeEnv->numOfFiles, sizeof(SQueryFileInfo), file_order_comparator);
qsort(pRuntimeEnv->pVnodeFiles, (size_t)pRuntimeEnv->numOfFiles, sizeof(SQueryFileInfo), file_order_comparator);
}
static void updateOffsetVal(SQueryRuntimeEnv *pRuntimeEnv, SBlockInfo *pBlockInfo, void *pBlock) {
@ -3122,8 +3139,8 @@ static bool onlyOneQueryType(SQuery *pQuery, int32_t functId, int32_t functIdDst
for (int32_t i = 0; i < pQuery->numOfOutputCols; ++i) {
int32_t functionId = pQuery->pSelectExpr[i].pBase.functionId;
if (functionId == TSDB_FUNC_TS || functionId == TSDB_FUNC_TS_DUMMY ||
functionId == TSDB_FUNC_TAG || functionId == TSDB_FUNC_TAG_DUMMY) {
if (functionId == TSDB_FUNC_TS || functionId == TSDB_FUNC_TS_DUMMY || functionId == TSDB_FUNC_TAG ||
functionId == TSDB_FUNC_TAG_DUMMY) {
continue;
}
@ -3141,7 +3158,8 @@ static bool onlyLastQuery(SQuery *pQuery) { return onlyOneQueryType(pQuery, TSDB
static void changeExecuteScanOrder(SQuery *pQuery, bool metricQuery) {
// in case of point-interpolation query, use asc order scan
char msg[] = "QInfo:%p scan order changed for %s query, old:%d, new:%d, qrange exchanged, old qrange:%lld-%lld, "
char msg[] =
"QInfo:%p scan order changed for %s query, old:%d, new:%d, qrange exchanged, old qrange:%lld-%lld, "
"new qrange:%lld-%lld";
// descending order query
@ -3440,12 +3458,12 @@ void pointInterpSupporterSetData(SQInfo *pQInfo, SPointInterpoSupporter *pPointI
// only the function of interp needs the corresponding information
if (pCtx->functionId != TSDB_FUNC_INTERP) {
continue;
continue;
}
pCtx->numOfParams = 4;
SInterpInfo * pInterpInfo = (SInterpInfo *)pRuntimeEnv->pCtx[i].aOutputBuf;
SInterpInfo *pInterpInfo = (SInterpInfo *)pRuntimeEnv->pCtx[i].aOutputBuf;
pInterpInfo->pInterpDetail = calloc(1, sizeof(SInterpInfoDetail));
@ -3624,6 +3642,7 @@ int32_t vnodeQuerySingleMeterPrepare(SQInfo *pQInfo, SMeterObj *pMeterObj, SMete
// dataInCache requires lastKey value
pQuery->lastKey = pQuery->skey;
pSupporter->runtimeEnv.mmapedHFileIndex = -1; // set the initial value
vnodeInitDataBlockInfo(&pSupporter->runtimeEnv.loadBlockInfo);
vnodeInitLoadCompBlockInfo(&pSupporter->runtimeEnv.loadCompBlockInfo);
@ -3793,6 +3812,7 @@ int32_t vnodeMultiMeterQueryPrepare(SQInfo *pQInfo, SQuery *pQuery, void *param)
changeExecuteScanOrder(pQuery, true);
pSupporter->runtimeEnv.mmapedHFileIndex = -1; // set the initial value
vnodeInitDataBlockInfo(&pSupporter->runtimeEnv.loadBlockInfo);
vnodeInitLoadCompBlockInfo(&pSupporter->runtimeEnv.loadCompBlockInfo);
@ -3917,13 +3937,13 @@ void vnodeDecMeterRefcnt(SQInfo *pQInfo) {
}
// todo merge with doRevisedResultsByLimit
void UNUSED_FUNC truncateResultByLimit(SQInfo *pQInfo, int64_t * final, int32_t *interpo) {
void UNUSED_FUNC truncateResultByLimit(SQInfo *pQInfo, int64_t *final, int32_t *interpo) {
SQuery *pQuery = &(pQInfo->query);
if (pQuery->limit.limit > 0 && ((* final) + pQInfo->pointsRead > pQuery->limit.limit)) {
int64_t num = (* final) + pQInfo->pointsRead - pQuery->limit.limit;
if (pQuery->limit.limit > 0 && ((*final) + pQInfo->pointsRead > pQuery->limit.limit)) {
int64_t num = (*final) + pQInfo->pointsRead - pQuery->limit.limit;
(*interpo) -= num;
(* final) -= num;
(*final) -= num;
setQueryStatus(pQuery, QUERY_COMPLETED); // query completed
}
@ -3974,7 +3994,7 @@ TSKEY getTimestampInDiskBlock(SQueryRuntimeEnv *pRuntimeEnv, int32_t index) {
// the fields info is not loaded, load it into memory
if (pQuery->pFields == NULL || pQuery->pFields[pQuery->slot] == NULL) {
loadDataBlockFieldsInfo(pRuntimeEnv, &pRuntimeEnv->pHeaderFiles[fileIndex], pBlock, &pQuery->pFields[pQuery->slot]);
loadDataBlockFieldsInfo(pRuntimeEnv, &pRuntimeEnv->pVnodeFiles[fileIndex], pBlock, &pQuery->pFields[pQuery->slot]);
}
SET_DATA_BLOCK_LOADED(pRuntimeEnv->blockStatus);
@ -4006,7 +4026,7 @@ static void getFirstDataBlockInCache(SQueryRuntimeEnv *pRuntimeEnv) {
}
}
//TODO handle case that the cache is allocated but not assign to SMeterObj
// TODO handle case that the cache is allocated but not assign to SMeterObj
void getQueryPositionForCacheInvalid(SQueryRuntimeEnv *pRuntimeEnv, __block_search_fn_t searchFn) {
SQuery * pQuery = pRuntimeEnv->pQuery;
SQInfo * pQInfo = (SQInfo *)GET_QINFO_ADDR(pQuery);
@ -4464,17 +4484,18 @@ static void doMerge(SQueryRuntimeEnv *pRuntimeEnv, int64_t timestamp, tFilePage
pCtx[i].hasNull = true;
pCtx[i].nStartQueryTimestamp = timestamp;
pCtx[i].aInputElemBuf = ((char *) inputSrc->data) +
((int32_t) pRuntimeEnv->offset[i] * pRuntimeEnv->numOfRowsPerPage) + pCtx[i].outputBytes * inputIdx;
pCtx[i].aInputElemBuf = ((char *)inputSrc->data) +
((int32_t)pRuntimeEnv->offset[i] * pRuntimeEnv->numOfRowsPerPage) +
pCtx[i].outputBytes * inputIdx;
//in case of tag column, the tag information should be extracted from input buffer
// in case of tag column, the tag information should be extracted from input buffer
if (functionId == TSDB_FUNC_TAG_DUMMY || functionId == TSDB_FUNC_TAG) {
tVariantDestroy(&pCtx[i].tag);
tVariantCreateFromBinary(&pCtx[i].tag, pCtx[i].aInputElemBuf, pCtx[i].inputBytes, pCtx[i].inputType);
}
}
for(int32_t i = 0; i < pQuery->numOfOutputCols; ++i) {
for (int32_t i = 0; i < pQuery->numOfOutputCols; ++i) {
int32_t functionId = pQuery->pSelectExpr[i].pBase.functionId;
if (functionId == TSDB_FUNC_TAG_DUMMY) {
continue;
@ -4942,7 +4963,8 @@ void disableFunctForSuppleScan(SQueryRuntimeEnv *pRuntimeEnv, int32_t order) {
}
}
}
} else {
} else { // TODO ERROR!!
// need to handle for each query result, not just the single runtime ctx.
for (int32_t i = 0; i < pQuery->numOfOutputCols; ++i) {
pRuntimeEnv->pCtx[i].order = (pRuntimeEnv->pCtx[i].order) ^ 1;
int32_t functId = pQuery->pSelectExpr[i].pBase.functionId;
@ -5460,19 +5482,26 @@ static int32_t offsetComparator(const void *pLeft, const void *pRight) {
* @param pMeterHeadDataInfo
* @return
*/
SMeterDataInfo **vnodeFilterQualifiedMeters(SQInfo *pQInfo, int32_t vid, SQueryFileInfo *pQueryFileInfo,
tSidSet *pSidSet, SMeterDataInfo *pMeterDataInfo, int32_t *numOfMeters) {
SQuery * pQuery = &pQInfo->query;
SMeterDataInfo **vnodeFilterQualifiedMeters(SQInfo *pQInfo, int32_t vid, int32_t fileIndex, tSidSet *pSidSet,
SMeterDataInfo *pMeterDataInfo, int32_t *numOfMeters) {
SQuery *pQuery = &pQInfo->query;
SMeterQuerySupportObj *pSupporter = pQInfo->pMeterQuerySupporter;
SMeterSidExtInfo ** pMeterSidExtInfo = pSupporter->pMeterSidExtInfo;
SQueryRuntimeEnv * pRuntimeEnv = &pSupporter->runtimeEnv;
SQueryFileInfo * pQueryFileInfo = &pRuntimeEnv->pVnodeFiles[fileIndex];
SVnodeObj *pVnode = &vnodeList[vid];
char * pHeaderData = pQueryFileInfo->pHeaderFileData;
char * pHeaderFileData = vnodeGetHeaderFileData(pRuntimeEnv, fileIndex);
if (pHeaderFileData == NULL) { // failed to load header file into buffer
return 0;
}
int32_t tmsize = sizeof(SCompHeader) * (pVnode->cfg.maxSessions) + sizeof(TSCKSUM);
// file is corrupted, abort query in current file
if (validateHeaderOffsetSegment(pQInfo, pQueryFileInfo->headerFilePath, vid, pHeaderData, tmsize) < 0) {
if (validateHeaderOffsetSegment(pQInfo, pQueryFileInfo->headerFilePath, vid, pHeaderFileData, tmsize) < 0) {
*numOfMeters = 0;
return 0;
}
@ -5523,7 +5552,7 @@ SMeterDataInfo **vnodeFilterQualifiedMeters(SQInfo *pQInfo, int32_t vid, SQueryF
int64_t headerOffset = TSDB_FILE_HEADER_LEN + sizeof(SCompHeader) * pMeterObj->sid;
SCompHeader *compHeader = (SCompHeader *)(pHeaderData + headerOffset);
SCompHeader *compHeader = (SCompHeader *)(pHeaderFileData + headerOffset);
if (compHeader->compInfoOffset == 0) {
continue;
@ -6467,7 +6496,7 @@ int32_t LoadDatablockOnDemand(SCompBlock *pBlock, SField **pFields, int8_t *blkS
int32_t fileIdx, int32_t slotIdx, __block_search_fn_t searchFn, bool onDemand) {
SQuery * pQuery = pRuntimeEnv->pQuery;
SMeterObj * pMeterObj = pRuntimeEnv->pMeterObj;
SQueryFileInfo *pQueryFileInfo = &pRuntimeEnv->pHeaderFiles[fileIdx];
SQueryFileInfo *pQueryFileInfo = &pRuntimeEnv->pVnodeFiles[fileIdx];
TSKEY *primaryKeys = (TSKEY *)pRuntimeEnv->primaryColBuffer->data;
@ -6869,7 +6898,7 @@ bool vnodeHasRemainResults(void *handle) {
// query has completed
if (Q_STATUS_EQUAL(pQuery->over, QUERY_COMPLETED | QUERY_NO_DATA_TO_CHECK)) {
TSKEY ekey = taosGetRevisedEndKey(pSupporter->rawEKey, pQuery->order.order, pQuery->nAggTimeInterval,
TSKEY ekey = taosGetRevisedEndKey(pSupporter->rawEKey, pQuery->order.order, pQuery->nAggTimeInterval,
pQuery->intervalTimeUnit, pQuery->precision);
int32_t numOfTotal = taosGetNumOfResultWithInterpo(pInterpoInfo, (TSKEY *)pRuntimeEnv->pInterpoBuf[0]->data,
remain, pQuery->nAggTimeInterval, ekey, pQuery->pointsToRead);
@ -6914,18 +6943,18 @@ static int32_t resultInterpolate(SQInfo *pQInfo, tFilePage **data, tFilePage **p
return numOfRes;
}
static void doCopyQueryResultToMsg(SQInfo *pQInfo, int32_t numOfRows, char* data, int32_t* size) {
SMeterObj* pObj = pQInfo->pObj;
SQuery* pQuery = &pQInfo->query;
static void doCopyQueryResultToMsg(SQInfo *pQInfo, int32_t numOfRows, char *data, int32_t *size) {
SMeterObj *pObj = pQInfo->pObj;
SQuery * pQuery = &pQInfo->query;
int tnumOfRows = vnodeList[pObj->vnode].cfg.rowsInFileBlock;
int tnumOfRows = vnodeList[pObj->vnode].cfg.rowsInFileBlock;
int32_t dataSize = pQInfo->query.rowSize * numOfRows;
if (dataSize >= tsCompressMsgSize && tsCompressMsgSize > 0) {
char* compBuf = malloc((size_t) dataSize);
char *compBuf = malloc((size_t)dataSize);
// for metric query, bufIndex always be 0.
char* d = compBuf;
char *d = compBuf;
for (int32_t col = 0; col < pQuery->numOfOutputCols; ++col) { // pQInfo->bufIndex == 0
int32_t bytes = pQuery->pSelectExpr[col].resBytes;
@ -6937,7 +6966,7 @@ static void doCopyQueryResultToMsg(SQInfo *pQInfo, int32_t numOfRows, char* data
dTrace("QInfo:%p compress rsp msg, before:%d, after:%d", pQInfo, dataSize, *size);
free(compBuf);
} else { // for metric query, bufIndex always be 0.
} else { // for metric query, bufIndex always be 0.
for (int32_t col = 0; col < pQuery->numOfOutputCols; ++col) { // pQInfo->bufIndex == 0
int32_t bytes = pQuery->pSelectExpr[col].resBytes;
@ -6958,9 +6987,9 @@ static void doCopyQueryResultToMsg(SQInfo *pQInfo, int32_t numOfRows, char* data
* @param numOfRows the number of rows that are not returned in current retrieve
* @return
*/
int32_t vnodeCopyQueryResultToMsg(void *handle, char *data, int32_t numOfRows, int32_t* size) {
int32_t vnodeCopyQueryResultToMsg(void *handle, char *data, int32_t numOfRows, int32_t *size) {
SQInfo *pQInfo = (SQInfo *)handle;
SQuery * pQuery = &pQInfo->query;
SQuery *pQuery = &pQInfo->query;
assert(pQuery->pSelectExpr != NULL && pQuery->numOfOutputCols > 0);
@ -6998,7 +7027,7 @@ int32_t vnodeQueryResultInterpolate(SQInfo *pQInfo, tFilePage **pDst, tFilePage
while (1) {
numOfRows = taosNumOfRemainPoints(&pRuntimeEnv->interpoInfo);
TSKEY ekey = taosGetRevisedEndKey(pSupporter->rawEKey, pQuery->order.order, pQuery->nAggTimeInterval,
TSKEY ekey = taosGetRevisedEndKey(pSupporter->rawEKey, pQuery->order.order, pQuery->nAggTimeInterval,
pQuery->intervalTimeUnit, pQuery->precision);
int32_t numOfFinalRows =
taosGetNumOfResultWithInterpo(&pRuntimeEnv->interpoInfo, (TSKEY *)pDataSrc[0]->data, numOfRows,

20
src/system/detail/src/vnodeQueryProcess.c
View File

@ -27,7 +27,7 @@
#include "vnodeQueryImpl.h"
#define ALL_CACHE_BLOCKS_CHECKED(q) \
((q)->slot == (q)->currentSlot && QUERY_IS_ASC_QUERY(q) || (q)->slot == (q)->firstSlot && (!QUERY_IS_ASC_QUERY(q)))
(((q)->slot == (q)->currentSlot && QUERY_IS_ASC_QUERY(q)) || ((q)->slot == (q)->firstSlot && (!QUERY_IS_ASC_QUERY(q))))
#define FORWARD_CACHE_BLOCK_CHECK_SLOT(slot, step, maxblocks) (slot) = ((slot) + (step) + (maxblocks)) % (maxblocks);
@ -289,13 +289,15 @@ static SMeterDataInfo *queryOnMultiDataFiles(SQInfo *pQInfo, SMeterDataInfo *pMe
pQuery->fileId = fid;
pSummary->numOfFiles++;
SQueryFileInfo *pQueryFileInfo = &pRuntimeEnv->pHeaderFiles[fileIdx];
char * pHeaderData = pQueryFileInfo->pHeaderFileData;
SQueryFileInfo *pQueryFileInfo = &pRuntimeEnv->pVnodeFiles[fileIdx];
char *pHeaderData = vnodeGetHeaderFileData(pRuntimeEnv, fileIdx);
if (pHeaderData == NULL) { // failed to mmap header file into buffer, ignore current file, try next
continue;
}
int32_t numOfQualifiedMeters = 0;
SMeterDataInfo **pReqMeterDataInfo = vnodeFilterQualifiedMeters(
pQInfo, vnodeId, pQueryFileInfo, pSupporter->pSidSet, pMeterDataInfo, &numOfQualifiedMeters);
dTrace("QInfo:%p file:%s, %d meters qualified", pQInfo, pQueryFileInfo->dataFilePath, numOfQualifiedMeters);
SMeterDataInfo **pReqMeterDataInfo = vnodeFilterQualifiedMeters(pQInfo, vnodeId, fileIdx, pSupporter->pSidSet,
pMeterDataInfo, &numOfQualifiedMeters);
if (pReqMeterDataInfo == NULL) {
dError("QInfo:%p failed to allocate memory to perform query processing, abort", pQInfo);
@ -305,6 +307,8 @@ static SMeterDataInfo *queryOnMultiDataFiles(SQInfo *pQInfo, SMeterDataInfo *pMe
return NULL;
}
dTrace("QInfo:%p file:%s, %d meters qualified", pQInfo, pQueryFileInfo->dataFilePath, numOfQualifiedMeters);
// none of meters in query set have pHeaderData in this file, try next file
if (numOfQualifiedMeters == 0) {
fid += step;
@ -500,7 +504,7 @@ static int64_t doCheckMetersInGroup(SQInfo *pQInfo, int32_t index, int32_t start
#if DEFAULT_IO_ENGINE == IO_ENGINE_MMAP
for (int32_t i = 0; i < pRuntimeEnv->numOfFiles; ++i) {
resetMMapWindow(&pRuntimeEnv->pHeaderFiles[i]);
resetMMapWindow(&pRuntimeEnv->pVnodeFiles[i]);
}
#endif
@ -670,7 +674,7 @@ static void vnodeMultiMeterMultiOutputProcessor(SQInfo *pQInfo) {
#if DEFAULT_IO_ENGINE == IO_ENGINE_MMAP
for (int32_t i = 0; i < pRuntimeEnv->numOfFiles; ++i) {
resetMMapWindow(&pRuntimeEnv->pHeaderFiles[i]);
resetMMapWindow(&pRuntimeEnv->pVnodeFiles[i]);
}
#endif

9
src/system/detail/src/vnodeShell.c
View File

@ -28,6 +28,7 @@
#include "vnodeRead.h"
#include "vnodeUtil.h"
#include "vnodeStore.h"
#include "tstatus.h"
#pragma GCC diagnostic ignored "-Wint-conversion"
extern int tsMaxQueues;
@ -101,7 +102,7 @@ void *vnodeProcessMsgFromShell(char *msg, void *ahandle, void *thandle) {
// if ( vnodeList[vnode].status != TSDB_STATUS_MASTER && pMsg->msgType != TSDB_MSG_TYPE_RETRIEVE ) {
#ifdef CLUSTER
if (vnodeList[vnode].vnodeStatus != TSDB_VNODE_STATUS_MASTER) {
if (vnodeList[vnode].vnodeStatus != TSDB_VN_STATUS_MASTER) {
taosSendSimpleRsp(thandle, pMsg->msgType + 1, TSDB_CODE_NOT_READY);
dTrace("vid:%d sid:%d, shell msg is ignored since in state:%d", vnode, sid, vnodeList[vnode].vnodeStatus);
} else {
@ -188,19 +189,21 @@ int vnodeOpenShellVnode(int vnode) {
return -1;
}
dTrace("vid:%d, sessions:%d, shell is opened", vnode, pCfg->maxSessions);
dPrint("vid:%d, sessions:%d, shell is opened", vnode, pCfg->maxSessions);
return TSDB_CODE_SUCCESS;
}
static void vnodeDelayedFreeResource(void *param, void *tmrId) {
int32_t vnode = *(int32_t*) param;
dTrace("vid:%d, start to free resources", vnode);
dTrace("vid:%d, start to free resources for 500ms arrived", vnode);
taosCloseRpcChann(pShellServer, vnode); // close connection
tfree(shellList[vnode]); //free SShellObj
tfree(param);
memset(vnodeList + vnode, 0, sizeof(SVnodeObj));
dTrace("vid:%d, status set to %s", vnode, taosGetVnodeStatusStr(vnodeList[vnode].vnodeStatus));
vnodeCalcOpenVnodes();
}

43
src/system/detail/src/vnodeStore.c
View File

@ -59,7 +59,7 @@ static int vnodeInitStoreVnode(int vnode) {
}
pthread_mutex_init(&(pVnode->vmutex), NULL);
dTrace("vid:%d, storage initialized, version:%ld fileId:%d numOfFiles:%d", vnode, pVnode->version, pVnode->fileId,
dPrint("vid:%d, storage initialized, version:%ld fileId:%d numOfFiles:%d", vnode, pVnode->version, pVnode->fileId,
pVnode->numOfFiles);
return 0;
@ -78,12 +78,12 @@ int vnodeOpenVnode(int vnode) {
return TSDB_CODE_SUCCESS;
}
if (!(pVnode->vnodeStatus == TSDB_VNODE_STATUS_OFFLINE || pVnode->vnodeStatus == TSDB_VNODE_STATUS_CREATING)) {
if (!(pVnode->vnodeStatus == TSDB_VN_STATUS_OFFLINE || pVnode->vnodeStatus == TSDB_VN_STATUS_CREATING)) {
dError("vid:%d, status:%s, cannot enter open operation", vnode, taosGetVnodeStatusStr(pVnode->vnodeStatus));
return TSDB_CODE_INVALID_VNODE_STATUS;
}
dTrace("vid:%d, status:%s, start to open", vnode, taosGetVnodeStatusStr(pVnode->vnodeStatus));
dPrint("vid:%d, status:%s, start to open", vnode, taosGetVnodeStatusStr(pVnode->vnodeStatus));
pthread_mutex_lock(&dmutex);
// not enough memory, abort
@ -104,7 +104,7 @@ int vnodeOpenVnode(int vnode) {
vnodeOpenStreams(pVnode, NULL);
#endif
dTrace("vid:%d, vnode is opened, openVnodes:%d, status:%s", vnode, tsOpenVnodes, taosGetVnodeStatusStr(pVnode->vnodeStatus));
dPrint("vid:%d, vnode is opened, openVnodes:%d, status:%s", vnode, tsOpenVnodes, taosGetVnodeStatusStr(pVnode->vnodeStatus));
return TSDB_CODE_SUCCESS;
}
@ -140,12 +140,12 @@ static int vnodeCloseVnode(int vnode) {
return TSDB_CODE_SUCCESS;
}
if (pVnode->vnodeStatus == TSDB_VNODE_STATUS_DELETING) {
dTrace("vid:%d, status:%s, another thread performed delete operation", vnode, taosGetVnodeStatusStr(pVnode->vnodeStatus));
if (pVnode->vnodeStatus == TSDB_VN_STATUS_DELETING) {
dPrint("vid:%d, status:%s, another thread performed delete operation", vnode, taosGetVnodeStatusStr(pVnode->vnodeStatus));
return TSDB_CODE_SUCCESS;
} else {
dTrace("vid:%d, status:%s, enter close operation", vnode, taosGetVnodeStatusStr(pVnode->vnodeStatus));
pVnode->vnodeStatus = TSDB_VNODE_STATUS_CLOSING;
dPrint("vid:%d, status:%s, enter close operation", vnode, taosGetVnodeStatusStr(pVnode->vnodeStatus));
pVnode->vnodeStatus = TSDB_VN_STATUS_CLOSING;
}
// set the meter is dropped flag
@ -154,8 +154,8 @@ static int vnodeCloseVnode(int vnode) {
return TSDB_CODE_ACTION_IN_PROGRESS;
}
dTrace("vid:%d, status:%s, enter delete operation", vnode, taosGetVnodeStatusStr(pVnode->vnodeStatus));
pVnode->vnodeStatus = TSDB_VNODE_STATUS_DELETING;
dPrint("vid:%d, status:%s, enter delete operation", vnode, taosGetVnodeStatusStr(pVnode->vnodeStatus));
pVnode->vnodeStatus = TSDB_VN_STATUS_DELETING;
vnodeCloseStream(vnodeList + vnode);
vnodeCancelCommit(vnodeList + vnode);
@ -178,12 +178,12 @@ static int vnodeCloseVnode(int vnode) {
int vnodeCreateVnode(int vnode, SVnodeCfg *pCfg, SVPeerDesc *pDesc) {
char fileName[128];
if (vnodeList[vnode].vnodeStatus != TSDB_VNODE_STATUS_OFFLINE) {
if (vnodeList[vnode].vnodeStatus != TSDB_VN_STATUS_OFFLINE) {
dError("vid:%d, status:%s, cannot enter create operation", vnode, taosGetVnodeStatusStr(vnodeList[vnode].vnodeStatus));
return TSDB_CODE_INVALID_VNODE_STATUS;
}
vnodeList[vnode].vnodeStatus = TSDB_VNODE_STATUS_CREATING;
vnodeList[vnode].vnodeStatus = TSDB_VN_STATUS_CREATING;
sprintf(fileName, "%s/vnode%d", tsDirectory, vnode);
mkdir(fileName, 0755);
@ -235,7 +235,7 @@ static void vnodeRemoveDataFiles(int vnode) {
if (tcode >= 0) {
remove(dfilePath);
dTrace("Data file %s is removed, link file %s", dfilePath, linkFile);
dPrint("Data file %s is removed, link file %s", dfilePath, linkFile);
}
} else {
remove(de->d_name);
@ -250,7 +250,7 @@ static void vnodeRemoveDataFiles(int vnode) {
sprintf(vnodeDir, "%s/vnode%d", tsDirectory, vnode);
rmdir(vnodeDir);
dTrace("vid:%d, vnode is removed!", vnode);
dPrint("vid:%d, vnode is removed, status:%s", vnode, taosGetVnodeStatusStr(vnodeList[vnode].vnodeStatus));
}
int vnodeRemoveVnode(int vnode) {
@ -258,10 +258,10 @@ int vnodeRemoveVnode(int vnode) {
if (vnodeList[vnode].cfg.maxSessions > 0) {
SVnodeObj* pVnode = &vnodeList[vnode];
if (pVnode->vnodeStatus == TSDB_VNODE_STATUS_CREATING
|| pVnode->vnodeStatus == TSDB_VNODE_STATUS_OFFLINE
|| pVnode->vnodeStatus == TSDB_VNODE_STATUS_DELETING) {
dError("vid:%d, status:%s, cannot enter close/delete operation", vnode, taosGetVnodeStatusStr(pVnode->vnodeStatus));
if (pVnode->vnodeStatus == TSDB_VN_STATUS_CREATING
|| pVnode->vnodeStatus == TSDB_VN_STATUS_OFFLINE
|| pVnode->vnodeStatus == TSDB_VN_STATUS_DELETING) {
dTrace("vid:%d, status:%s, cannot enter close/delete operation", vnode, taosGetVnodeStatusStr(pVnode->vnodeStatus));
return TSDB_CODE_ACTION_IN_PROGRESS;
} else {
int32_t ret = vnodeCloseVnode(vnode);
@ -269,11 +269,12 @@ int vnodeRemoveVnode(int vnode) {
return ret;
}
dTrace("vid:%d, status:%s, do delete operation", vnode, taosGetVnodeStatusStr(pVnode->vnodeStatus));
vnodeRemoveDataFiles(vnode);
}
} else {
dTrace("vid:%d, max sessions:%d, this vnode already dropped!!!", vnode, vnodeList[vnode].cfg.maxSessions);
dPrint("vid:%d, max sessions:%d, this vnode already dropped!!!", vnode, vnodeList[vnode].cfg.maxSessions);
vnodeList[vnode].cfg.maxSessions = 0; //reset value
vnodeCalcOpenVnodes();
}
@ -325,7 +326,7 @@ void vnodeCleanUpOneVnode(int vnode) {
again = 1;
if (vnodeList[vnode].pCachePool) {
vnodeList[vnode].vnodeStatus = TSDB_VNODE_STATUS_OFFLINE;
vnodeList[vnode].vnodeStatus = TSDB_VN_STATUS_OFFLINE;
vnodeClosePeerVnode(vnode);
}
@ -354,7 +355,7 @@ void vnodeCleanUpVnodes() {
for (int vnode = 0; vnode < TSDB_MAX_VNODES; ++vnode) {
if (vnodeList[vnode].pCachePool) {
vnodeList[vnode].vnodeStatus = TSDB_VNODE_STATUS_OFFLINE;
vnodeList[vnode].vnodeStatus = TSDB_VN_STATUS_OFFLINE;
vnodeClosePeerVnode(vnode);
}
}

27
src/system/detail/src/vnodeStream.c
View File

@ -17,6 +17,7 @@
#include "taosmsg.h"
#include "vnode.h"
#include "vnodeUtil.h"
#include "tstatus.h"
/* static TAOS *dbConn = NULL; */
void vnodeCloseStreamCallback(void *param);
@ -55,14 +56,11 @@ void vnodeProcessStreamRes(void *param, TAOS_RES *tres, TAOS_ROW row) {
contLen += sizeof(SSubmitMsg);
int32_t numOfPoints = 0;
int32_t code = vnodeInsertPoints(pObj, (char *)pMsg, contLen, TSDB_DATA_SOURCE_SHELL, NULL, pObj->sversion,
&numOfPoints, taosGetTimestamp(vnodeList[pObj->vnode].cfg.precision));
int32_t state = vnodeSetMeterState(pObj, TSDB_METER_STATE_INSERT);
if (state == TSDB_METER_STATE_READY) {
vnodeInsertPoints(pObj, (char *)pMsg, contLen, TSDB_DATA_SOURCE_SHELL, NULL, pObj->sversion, &numOfPoints, taosGetTimestamp(vnodeList[pObj->vnode].cfg.precision));
vnodeClearMeterState(pObj, TSDB_METER_STATE_INSERT);
} else {
dError("vid:%d sid:%d id:%s, failed to insert continuous query results, state:%d", pObj->vnode, pObj->sid,
pObj->meterId, state);
if (code != TSDB_CODE_SUCCESS) {
dError("vid:%d sid:%d id:%s, failed to insert continuous query results", pObj->vnode, pObj->sid, pObj->meterId);
}
assert(numOfPoints >= 0 && numOfPoints <= 1);
@ -80,7 +78,7 @@ void vnodeOpenStreams(void *param, void *tmrId) {
SVnodeObj *pVnode = (SVnodeObj *)param;
SMeterObj *pObj;
if (pVnode->streamRole == 0) return;
if (pVnode->streamRole == TSDB_VN_STREAM_STATUS_STOP) return;
if (pVnode->meterList == NULL) return;
taosTmrStopA(&pVnode->streamTimer);
@ -119,7 +117,7 @@ void vnodeCreateStream(SMeterObj *pObj) {
SVnodeObj *pVnode = vnodeList + pObj->vnode;
if (pVnode->streamRole == 0) return;
if (pVnode->streamRole == TSDB_VN_STREAM_STATUS_STOP) return;
if (pObj->pStream) return;
dTrace("vid:%d sid:%d id:%s stream:%s is created", pObj->vnode, pObj->sid, pObj->meterId, pObj->pSql);
@ -154,7 +152,7 @@ void vnodeRemoveStream(SMeterObj *pObj) {
// Close all streams in a vnode
void vnodeCloseStream(SVnodeObj *pVnode) {
SMeterObj *pObj;
dTrace("vid:%d, stream is closed, old role:%d", pVnode->vnode, pVnode->streamRole);
dPrint("vid:%d, stream is closed, old role %s", pVnode->vnode, taosGetVnodeStreamStatusStr(pVnode->streamRole));
// stop stream computing
for (int sid = 0; sid < pVnode->cfg.maxSessions; ++sid) {
@ -171,17 +169,18 @@ void vnodeCloseStream(SVnodeObj *pVnode) {
void vnodeUpdateStreamRole(SVnodeObj *pVnode) {
/* SMeterObj *pObj; */
int newRole = (pVnode->vnodeStatus == TSDB_VNODE_STATUS_MASTER) ? 1 : 0;
int newRole = (pVnode->vnodeStatus == TSDB_VN_STATUS_MASTER) ? TSDB_VN_STREAM_STATUS_START : TSDB_VN_STREAM_STATUS_STOP;
if (newRole != pVnode->streamRole) {
dTrace("vid:%d, stream role is changed to:%d", pVnode->vnode, newRole);
dPrint("vid:%d, stream role is changed from %s to %s",
pVnode->vnode, taosGetVnodeStreamStatusStr(pVnode->streamRole), taosGetVnodeStreamStatusStr(newRole));
pVnode->streamRole = newRole;
if (newRole) {
if (newRole == TSDB_VN_STREAM_STATUS_START) {
vnodeOpenStreams(pVnode, NULL);
} else {
vnodeCloseStream(pVnode);
}
} else {
dTrace("vid:%d, stream role is keep to:%d", pVnode->vnode, newRole);
dPrint("vid:%d, stream role is keep to %s", pVnode->vnode, taosGetVnodeStreamStatusStr(pVnode->streamRole));
}
}

31
src/system/detail/src/vnodeUtil.c
View File

@ -283,6 +283,7 @@ SSqlFunctionExpr* vnodeCreateSqlFunctionExpr(SQueryMeterMsg* pQueryMsg, int32_t*
int32_t param = pExprs[i].pBase.arg[0].argValue.i64;
if (getResultDataInfo(type, bytes, pExprs[i].pBase.functionId, param, &pExprs[i].resType, &pExprs[i].resBytes,
&pExprs[i].interResBytes, 0, isSuperTable) != TSDB_CODE_SUCCESS) {
*code = TSDB_CODE_INVALID_QUERY_MSG;
return NULL;
}
@ -626,16 +627,16 @@ void vnodeUpdateQueryColumnIndex(SQuery* pQuery, SMeterObj* pMeterObj) {
return;
}
for(int32_t i = 0; i < pQuery->numOfOutputCols; ++i) {
SSqlFuncExprMsg* pSqlExprMsg = &pQuery->pSelectExpr[i].pBase;
for(int32_t k = 0; k < pQuery->numOfOutputCols; ++k) {
SSqlFuncExprMsg* pSqlExprMsg = &pQuery->pSelectExpr[k].pBase;
if (pSqlExprMsg->functionId == TSDB_FUNC_ARITHM || pSqlExprMsg->colInfo.flag == TSDB_COL_TAG) {
continue;
}
SColIndexEx* pColIndexEx = &pSqlExprMsg->colInfo;
for(int32_t j = 0; j < pQuery->numOfCols; ++j) {
if (pColIndexEx->colId == pQuery->colList[j].data.colId) {
pColIndexEx->colIdx = pQuery->colList[j].colIdx;
for(int32_t f = 0; f < pQuery->numOfCols; ++f) {
if (pColIndexEx->colId == pQuery->colList[f].data.colId) {
pColIndexEx->colIdx = pQuery->colList[f].colIdx;
break;
}
}
@ -668,6 +669,26 @@ void vnodeSetMeterDeleting(SMeterObj* pMeterObj) {
pMeterObj->state |= TSDB_METER_STATE_DELETING;
}
int32_t vnodeSetMeterInsertImportStateEx(SMeterObj* pObj, int32_t st) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t state = vnodeSetMeterState(pObj, st);
if (state != TSDB_METER_STATE_READY) {//return to denote import is not performed
if (vnodeIsMeterState(pObj, TSDB_METER_STATE_DELETING)) {
dTrace("vid:%d sid:%d id:%s, meter is deleted, state:%d", pObj->vnode, pObj->sid, pObj->meterId,
pObj->state);
code = TSDB_CODE_NOT_ACTIVE_TABLE;
} else {// waiting for 300ms by default and try again
dTrace("vid:%d sid:%d id:%s, try submit again since in state:%d", pObj->vnode, pObj->sid,
pObj->meterId, pObj->state);
code = TSDB_CODE_ACTION_IN_PROGRESS;
}
}
return code;
}
bool vnodeIsSafeToDeleteMeter(SVnodeObj* pVnode, int32_t sid) {
SMeterObj* pObj = pVnode->meterList[sid];

2
src/system/lite/src/dnodeMgmt.spec.c
View File

@ -90,4 +90,6 @@ void vnodeProcessMsgFromMgmtSpec(SSchedMsg *sched) {
int vnodeInitMgmt() { return 0; }
void vnodeInitMgmtIp() {}
int vnodeSaveCreateMsgIntoQueue(SVnodeObj *pVnode, char *pMsg, int msgLen) { return 0; }

13
src/system/lite/src/mgmtBalance.spec.c
View File

@ -15,6 +15,7 @@
#define _DEFAULT_SOURCE
#include "mgmtBalance.h"
#include "tstatus.h"
void mgmtStartBalanceTimer(int mseconds) {}
@ -25,10 +26,11 @@ void mgmtCleanupBalance() {}
int mgmtAllocVnodes(SVgObj *pVgroup) {
int selectedVnode = -1;
SDnodeObj *pDnode = &dnodeObj;
int lastAllocVode = pDnode->lastAllocVnode;
for (int i = 0; i < pDnode->numOfVnodes; i++) {
int vnode = (i + pDnode->lastAllocVnode) % pDnode->numOfVnodes;
if (pDnode->vload[vnode].vgId == 0 && pDnode->vload[vnode].status == TSDB_VN_STATUS_READY) {
int vnode = (i + lastAllocVode) % pDnode->numOfVnodes;
if (pDnode->vload[vnode].vgId == 0 && pDnode->vload[vnode].status == TSDB_VN_STATUS_OFFLINE) {
selectedVnode = vnode;
break;
}
@ -38,8 +40,7 @@ int mgmtAllocVnodes(SVgObj *pVgroup) {
mError("vgroup:%d alloc vnode failed, free vnodes:%d", pVgroup->vgId, pDnode->numOfFreeVnodes);
return -1;
} else {
mTrace("vgroup:%d allocate vnode:%d, last allocated vnode:%d", pVgroup->vgId, selectedVnode,
pDnode->lastAllocVnode);
mTrace("vgroup:%d allocate vnode:%d, last allocated vnode:%d", pVgroup->vgId, selectedVnode, lastAllocVode);
pVgroup->vnodeGid[0].vnode = selectedVnode;
pDnode->lastAllocVnode = selectedVnode + 1;
if (pDnode->lastAllocVnode >= pDnode->numOfVnodes) pDnode->lastAllocVnode = 0;
@ -53,8 +54,8 @@ bool mgmtCheckModuleInDnode(SDnodeObj *pDnode, int moduleType) {
bool mgmtCheckVnodeReady(SDnodeObj *pDnode, SVgObj *pVgroup, SVnodeGid *pVnode) { return true; }
void mgmtUpdateDnodeState(SDnodeObj *pDnode, int lbState) {}
void mgmtUpdateDnodeState(SDnodeObj *pDnode, int lbStatus) {}
void mgmtUpdateVgroupState(SVgObj *pVgroup, int lbState, int srcIp) {}
void mgmtUpdateVgroupState(SVgObj *pVgroup, int lbStatus, int srcIp) {}
bool mgmtAddVnode(SVgObj *pVgroup, SDnodeObj *pSrcDnode, SDnodeObj *pDestDnode) { return false; }

5
src/system/lite/src/mgmtDnode.spec.c
View File

@ -15,6 +15,7 @@
#define _DEFAULT_SOURCE
#include "mgmt.h"
#include "tstatus.h"
SDnodeObj dnodeObj;
extern uint32_t tsRebootTime;
@ -30,13 +31,13 @@ int mgmtInitDnodes() {
dnodeObj.createdTime = (int64_t)tsRebootTime * 1000;
dnodeObj.lastReboot = tsRebootTime;
dnodeObj.numOfCores = (uint16_t)tsNumOfCores;
dnodeObj.status = TSDB_DNODE_STATUS_READY;
dnodeObj.status = TSDB_DN_STATUS_READY;
dnodeObj.alternativeRole = TSDB_DNODE_ROLE_ANY;
dnodeObj.numOfTotalVnodes = tsNumOfTotalVnodes;
dnodeObj.thandle = (void*)(1); //hack way
if (dnodeObj.numOfVnodes == TSDB_INVALID_VNODE_NUM) {
mgmtSetDnodeMaxVnodes(&dnodeObj);
mPrint("first access, set total vnodes:%d", dnodeObj.numOfVnodes);
mPrint("dnode first access, set total vnodes:%d", dnodeObj.numOfVnodes);
}
return 0;
}

17
src/system/lite/src/mgmtDnodeInt.spec.c
View File

@ -23,6 +23,7 @@
#include "tutil.h"
#include "vnode.h"
#include "tsystem.h"
#include "tstatus.h"
extern void *dmQhandle;
void * mgmtStatusTimer = NULL;
@ -82,7 +83,7 @@ void mgmtCleanUpDnodeInt() {}
void mgmtProcessDnodeStatus(void *handle, void *tmrId) {
SDnodeObj *pObj = &dnodeObj;
pObj->openVnodes = tsOpenVnodes;
pObj->status = TSDB_DNODE_STATUS_READY;
pObj->status = TSDB_DN_STATUS_READY;
float memoryUsedMB = 0;
taosGetSysMemory(&memoryUsedMB);
@ -93,10 +94,10 @@ void mgmtProcessDnodeStatus(void *handle, void *tmrId) {
SVnodeObj * pVnode = vnodeList + vnode;
// wait vnode dropped
if (pVload->dropStatus == TSDB_VN_STATUS_DROPPING) {
if (pVload->dropStatus == TSDB_VN_DROP_STATUS_DROPPING) {
if (vnodeList[vnode].cfg.maxSessions <= 0) {
pVload->dropStatus = TSDB_VN_STATUS_READY;
pVload->status = TSDB_VN_STATUS_READY;
pVload->dropStatus = TSDB_VN_DROP_STATUS_READY;
pVload->status = TSDB_VN_STATUS_OFFLINE;
mPrint("dnode:%s, vid:%d, drop finished", taosIpStr(pObj->privateIp), vnode);
taosTmrStart(mgmtMonitorDbDrop, 10000, NULL, mgmtTmr);
}
@ -107,7 +108,7 @@ void mgmtProcessDnodeStatus(void *handle, void *tmrId) {
}
pVload->vnode = vnode;
pVload->status = TSDB_VN_STATUS_READY;
pVload->status = TSDB_VN_STATUS_MASTER;
pVload->totalStorage = pVnode->vnodeStatistic.totalStorage;
pVload->compStorage = pVnode->vnodeStatistic.compStorage;
pVload->pointsWritten = pVnode->vnodeStatistic.pointsWritten;
@ -116,7 +117,7 @@ void mgmtProcessDnodeStatus(void *handle, void *tmrId) {
SVgObj *pVgroup = mgmtGetVgroup(vgId);
if (pVgroup == NULL) {
mError("vgroup:%d is not there, but associated with vnode %d", vgId, vnode);
pVload->dropStatus = TSDB_VN_STATUS_DROPPING;
pVload->dropStatus = TSDB_VN_DROP_STATUS_DROPPING;
continue;
}
@ -126,9 +127,9 @@ void mgmtProcessDnodeStatus(void *handle, void *tmrId) {
continue;
}
if (pVload->vgId == 0 || pVload->dropStatus == TSDB_VN_STATUS_DROPPING) {
if (pVload->vgId == 0 || pVload->dropStatus == TSDB_VN_DROP_STATUS_DROPPING) {
mError("vid:%d, mgmt not exist, drop it", vnode);
pVload->dropStatus = TSDB_VN_STATUS_DROPPING;
pVload->dropStatus = TSDB_VN_DROP_STATUS_DROPPING;
}
}

2
src/system/lite/src/vnodeFile.spec.c
View File

@ -97,7 +97,7 @@ int vnodeCheckNewHeaderFile(int fd, SVnodeObj *pVnode) {
}
_correct_exit:
dTrace("vid: %d new header file %s is correct", pVnode->vnode, pVnode->nfn);
dPrint("vid: %d new header file %s is correct", pVnode->vnode, pVnode->nfn);
tfree(pBlocks);
tfree(pHeader);
return 0;

4
src/system/lite/src/vnodePeer.spec.c
View File

@ -31,8 +31,8 @@ void vnodeBroadcastStatusToUnsyncedPeer(SVnodeObj *pVnode) {}
int vnodeOpenPeerVnode(int vnode) {
SVnodeObj *pVnode = vnodeList + vnode;
pVnode->vnodeStatus = (pVnode->cfg.replications > 1) ? TSDB_VNODE_STATUS_UNSYNCED : TSDB_VNODE_STATUS_MASTER;
dTrace("vid:%d, status:%s numOfPeers:%d", vnode, taosGetVnodeStatusStr(pVnode->vnodeStatus), pVnode->cfg.replications - 1);
pVnode->vnodeStatus = (pVnode->cfg.replications > 1) ? TSDB_VN_STATUS_UNSYNCED : TSDB_VN_STATUS_MASTER;
dPrint("vid:%d, status:%s numOfPeers:%d", vnode, taosGetVnodeStatusStr(pVnode->vnodeStatus), pVnode->cfg.replications - 1);
vnodeUpdateStreamRole(pVnode);
return 0;
}

2556
src/util/src/sql.c
View File

File diff suppressed because it is too large Load Diff

1
src/util/src/textbuffer.c
View File

@ -18,7 +18,6 @@
#include "taosmsg.h"
#include "textbuffer.h"
#include "tlog.h"
#include "tsql.h"
#include "tsqlfunction.h"
#include "ttime.h"
#include "ttypes.h"

22
src/util/src/tglobalcfg.c
View File

@ -84,7 +84,7 @@ short tsCheckHeaderFile = 0;
int tsSessionsPerVnode = 1000;
int tsCacheBlockSize = 16384; // 256 columns
int tsAverageCacheBlocks = 4;
int tsAverageCacheBlocks = TSDB_DEFAULT_AVG_BLOCKS;
int tsRowsInFileBlock = 4096;
float tsFileBlockMinPercent = 0.05;
@ -92,10 +92,10 @@ float tsFileBlockMinPercent = 0.05;
short tsNumOfBlocksPerMeter = 100;
short tsCommitTime = 3600; // seconds
short tsCommitLog = 1;
short tsCompression = 2;
short tsCompression = TSDB_MAX_COMPRESSION_LEVEL;
short tsDaysPerFile = 10;
int tsDaysToKeep = 3650;
int tsReplications = 1;
int tsReplications = TSDB_REPLICA_MIN_NUM;
int tsNumOfMPeers = 3;
int tsMaxShellConns = 2000;
@ -162,6 +162,9 @@ int tsHttpEnableRecordSql = 0;
int tsTelegrafUseFieldNum = 0;
int tsAdminRowLimit = 10240;
int tsTscEnableRecordSql = 0;
int tsEnableCoreFile = 0;
int tsRpcTimer = 300;
int tsRpcMaxTime = 600; // seconds;
@ -507,7 +510,7 @@ static void doInitGlobalConfig() {
0, TSDB_MAX_VNODES, 0, TSDB_CFG_UTYPE_NONE);
tsInitConfigOption(cfg++, "tables", &tsSessionsPerVnode, TSDB_CFG_VTYPE_INT,
TSDB_CFG_CTYPE_B_CONFIG | TSDB_CFG_CTYPE_B_SHOW,
4, 220000, 0, TSDB_CFG_UTYPE_NONE);
TSDB_MIN_TABLES_PER_VNODE, TSDB_MAX_TABLES_PER_VNODE, 0, TSDB_CFG_UTYPE_NONE);
tsInitConfigOption(cfg++, "cache", &tsCacheBlockSize, TSDB_CFG_VTYPE_INT,
TSDB_CFG_CTYPE_B_CONFIG | TSDB_CFG_CTYPE_B_SHOW,
100, 1048576, 0, TSDB_CFG_UTYPE_BYTE);
@ -519,7 +522,7 @@ static void doInitGlobalConfig() {
0, 1.0, 0, TSDB_CFG_UTYPE_NONE);
tsInitConfigOption(cfg++, "ablocks", &tsAverageCacheBlocks, TSDB_CFG_VTYPE_INT,
TSDB_CFG_CTYPE_B_CONFIG | TSDB_CFG_CTYPE_B_SHOW,
2, 128, 0, TSDB_CFG_UTYPE_NONE);
TSDB_MIN_AVG_BLOCKS, TSDB_MAX_AVG_BLOCKS, 0, TSDB_CFG_UTYPE_NONE);
tsInitConfigOption(cfg++, "tblocks", &tsNumOfBlocksPerMeter, TSDB_CFG_VTYPE_SHORT,
TSDB_CFG_CTYPE_B_CONFIG | TSDB_CFG_CTYPE_B_SHOW,
32, 4096, 0, TSDB_CFG_UTYPE_NONE);
@ -767,6 +770,14 @@ static void doInitGlobalConfig() {
0, 255, 0,
TSDB_CFG_UTYPE_NONE);
tsInitConfigOption(cfg++, "tscEnableRecordSql", &tsTscEnableRecordSql, TSDB_CFG_VTYPE_INT,
TSDB_CFG_CTYPE_B_CONFIG,
1, 100000, 0, TSDB_CFG_UTYPE_NONE);
tsInitConfigOption(cfg++, "enableCoreFile", &tsEnableCoreFile, TSDB_CFG_VTYPE_INT,
TSDB_CFG_CTYPE_B_CONFIG,
1, 100000, 0, TSDB_CFG_UTYPE_NONE);
// version info
tsInitConfigOption(cfg++, "gitinfo", gitinfo, TSDB_CFG_VTYPE_STRING,
TSDB_CFG_CTYPE_B_SHOW | TSDB_CFG_CTYPE_B_CLIENT,
@ -779,6 +790,7 @@ static void doInitGlobalConfig() {
0, 0, 0, TSDB_CFG_UTYPE_NONE);
tsGlobalConfigNum = (int)(cfg - tsGlobalConfig);
assert(tsGlobalConfigNum <= TSDB_CFG_MAX_NUM);
}
static pthread_once_t initGlobalConfig = PTHREAD_ONCE_INIT;

171
src/util/src/tnote.c
View File

@ -13,42 +13,50 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include "os.h"
#include "tutil.h"
#include "tglobalcfg.h"
#include "tnote.h"
#define MAX_NOTE_LINE_SIZE 66000
#define NOTE_FILE_NAME_LEN 300
taosNoteInfo m_HttpNote;
taosNoteInfo m_TscNote;
static int taosNoteFileNum = 1;
static int taosNoteMaxLines = 0;
static int taosNoteLines = 0;
static char taosNoteName[NOTE_FILE_NAME_LEN];
static int taosNoteFlag = 0;
static int taosNoteFd = -1;
static int taosNoteOpenInProgress = 0;
static pthread_mutex_t taosNoteMutex;
void taosNotePrint(const char * const format, ...);
int taosOpenNoteWithMaxLines(char *fn, int maxLines, int maxNoteNum);
int taosOpenNoteWithMaxLines(char *fn, int maxLines, int maxNoteNum, taosNoteInfo * pNote);
void taosInitNote(int numOfNoteLines, int maxNotes)
void taosInitNote(int numOfNoteLines, int maxNotes, char* lable)
{
taosNoteInfo * pNote = NULL;
char temp[128] = { 0 };
sprintf(temp, "%s/taosnote", logDir);
if (taosOpenNoteWithMaxLines(temp, numOfNoteLines, maxNotes) < 0)
if (strcasecmp(lable, "http_note") == 0) {
pNote = &m_HttpNote;
sprintf(temp, "%s/httpnote", logDir);
} else if (strcasecmp(lable, "tsc_note") == 0) {
pNote = &m_TscNote;
sprintf(temp, "%s/tscnote-%d", logDir, getpid());
} else {
return;
}
memset(pNote, 0, sizeof(taosNoteInfo));
pNote->taosNoteFileNum = 1;
//pNote->taosNoteMaxLines = 0;
//pNote->taosNoteLines = 0;
//pNote->taosNoteFlag = 0;
pNote->taosNoteFd = -1;
//pNote->taosNoteOpenInProgress = 0;
if (taosOpenNoteWithMaxLines(temp, numOfNoteLines, maxNotes, pNote) < 0)
fprintf(stderr, "failed to init note file\n");
taosNotePrint("==================================================");
taosNotePrint("=================== new note ===================");
taosNotePrint("==================================================");
taosNotePrint(pNote, "==================================================");
taosNotePrint(pNote, "=================== new note ===================");
taosNotePrint(pNote, "==================================================");
}
void taosCloseNoteByFd(int oldFd);
bool taosLockNote(int fd)
void taosCloseNoteByFd(int oldFd, taosNoteInfo * pNote);
bool taosLockNote(int fd, taosNoteInfo * pNote)
{
if (fd < 0) return false;
if (taosNoteFileNum > 1) {
if (pNote->taosNoteFileNum > 1) {
int ret = (int)(flock(fd, LOCK_EX | LOCK_NB));
if (ret == 0) {
return true;
@ -58,11 +66,11 @@ bool taosLockNote(int fd)
return false;
}
void taosUnLockNote(int fd)
void taosUnLockNote(int fd, taosNoteInfo * pNote)
{
if (fd < 0) return;
if (taosNoteFileNum > 1) {
if (pNote->taosNoteFileNum > 1) {
flock(fd, LOCK_UN | LOCK_NB);
}
}
@ -70,50 +78,51 @@ void taosUnLockNote(int fd)
void *taosThreadToOpenNewNote(void *param)
{
char name[NOTE_FILE_NAME_LEN];
taosNoteInfo * pNote = (taosNoteInfo *)param;
taosNoteFlag ^= 1;
taosNoteLines = 0;
sprintf(name, "%s.%d", taosNoteName, taosNoteFlag);
pNote->taosNoteFlag ^= 1;
pNote->taosNoteLines = 0;
sprintf(name, "%s.%d", pNote->taosNoteName, pNote->taosNoteFlag);
umask(0);
int fd = open(name, O_WRONLY | O_CREAT | O_TRUNC, S_IRWXU | S_IRWXG | S_IRWXO);
taosLockNote(fd);
taosLockNote(fd, pNote);
lseek(fd, 0, SEEK_SET);
int oldFd = taosNoteFd;
taosNoteFd = fd;
taosNoteLines = 0;
taosNoteOpenInProgress = 0;
taosNotePrint("=============== new note is opened =============");
int oldFd = pNote->taosNoteFd;
pNote->taosNoteFd = fd;
pNote->taosNoteLines = 0;
pNote->taosNoteOpenInProgress = 0;
taosNotePrint(pNote, "=============== new note is opened =============");
taosCloseNoteByFd(oldFd);
taosCloseNoteByFd(oldFd, pNote);
return NULL;
}
int taosOpenNewNote()
int taosOpenNewNote(taosNoteInfo * pNote)
{
pthread_mutex_lock(&taosNoteMutex);
pthread_mutex_lock(&pNote->taosNoteMutex);
if (taosNoteLines > taosNoteMaxLines && taosNoteOpenInProgress == 0) {
taosNoteOpenInProgress = 1;
if (pNote->taosNoteLines > pNote->taosNoteMaxLines && pNote->taosNoteOpenInProgress == 0) {
pNote->taosNoteOpenInProgress = 1;
taosNotePrint("=============== open new note ==================");
taosNotePrint(pNote, "=============== open new note ==================");
pthread_t pattern;
pthread_attr_t attr;
pthread_attr_init(&attr);
pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
pthread_create(&pattern, &attr, taosThreadToOpenNewNote, NULL);
pthread_create(&pattern, &attr, taosThreadToOpenNewNote, (void*)pNote);
pthread_attr_destroy(&attr);
}
pthread_mutex_unlock(&taosNoteMutex);
pthread_mutex_unlock(&pNote->taosNoteMutex);
return taosNoteFd;
return pNote->taosNoteFd;
}
bool taosCheckNoteIsOpen(char *noteName)
bool taosCheckNoteIsOpen(char *noteName, taosNoteInfo * pNote)
{
int exist = access(noteName, F_OK);
if (exist != 0) {
@ -126,8 +135,8 @@ bool taosCheckNoteIsOpen(char *noteName)
return true;
}
if (taosLockNote(fd)) {
taosUnLockNote(fd);
if (taosLockNote(fd, pNote)) {
taosUnLockNote(fd, pNote);
close(fd);
return false;
}
@ -137,80 +146,80 @@ bool taosCheckNoteIsOpen(char *noteName)
}
}
void taosGetNoteName(char *fn)
void taosGetNoteName(char *fn, taosNoteInfo * pNote)
{
if (taosNoteFileNum > 1) {
for (int i = 0; i < taosNoteFileNum; i++) {
if (pNote->taosNoteFileNum > 1) {
for (int i = 0; i < pNote->taosNoteFileNum; i++) {
char fileName[NOTE_FILE_NAME_LEN];
sprintf(fileName, "%s%d.0", fn, i);
bool file1open = taosCheckNoteIsOpen(fileName);
bool file1open = taosCheckNoteIsOpen(fileName, pNote);
sprintf(fileName, "%s%d.1", fn, i);
bool file2open = taosCheckNoteIsOpen(fileName);
bool file2open = taosCheckNoteIsOpen(fileName, pNote);
if (!file1open && !file2open) {
sprintf(taosNoteName, "%s%d", fn, i);
sprintf(pNote->taosNoteName, "%s%d", fn, i);
return;
}
}
}
strcpy(taosNoteName, fn);
strcpy(pNote->taosNoteName, fn);
}
int taosOpenNoteWithMaxLines(char *fn, int maxLines, int maxNoteNum)
int taosOpenNoteWithMaxLines(char *fn, int maxLines, int maxNoteNum, taosNoteInfo * pNote)
{
char name[NOTE_FILE_NAME_LEN] = "\0";
struct stat notestat0, notestat1;
int size;
taosNoteMaxLines = maxLines;
taosNoteFileNum = maxNoteNum;
taosGetNoteName(fn);
pNote->taosNoteMaxLines = maxLines;
pNote->taosNoteFileNum = maxNoteNum;
taosGetNoteName(fn, pNote);
strcpy(name, fn);
strcat(name, ".0");
// if none of the note files exist, open 0, if both exists, open the old one
if (stat(name, &notestat0) < 0) {
taosNoteFlag = 0;
pNote->taosNoteFlag = 0;
}
else {
strcpy(name, fn);
strcat(name, ".1");
if (stat(name, &notestat1) < 0) {
taosNoteFlag = 1;
pNote->taosNoteFlag = 1;
}
else {
taosNoteFlag = (notestat0.st_mtime > notestat1.st_mtime) ? 0 : 1;
pNote->taosNoteFlag = (notestat0.st_mtime > notestat1.st_mtime) ? 0 : 1;
}
}
sprintf(name, "%s.%d", taosNoteName, taosNoteFlag);
pthread_mutex_init(&taosNoteMutex, NULL);
sprintf(name, "%s.%d", pNote->taosNoteName, pNote->taosNoteFlag);
pthread_mutex_init(&pNote->taosNoteMutex, NULL);
umask(0);
taosNoteFd = open(name, O_WRONLY | O_CREAT, S_IRWXU | S_IRWXG | S_IRWXO);
pNote->taosNoteFd = open(name, O_WRONLY | O_CREAT, S_IRWXU | S_IRWXG | S_IRWXO);
if (taosNoteFd < 0) {
if (pNote->taosNoteFd < 0) {
fprintf(stderr, "failed to open note file:%s reason:%s\n", name, strerror(errno));
return -1;
}
taosLockNote(taosNoteFd);
taosLockNote(pNote->taosNoteFd, pNote);
// only an estimate for number of lines
struct stat filestat;
fstat(taosNoteFd, &filestat);
fstat(pNote->taosNoteFd, &filestat);
size = (int)filestat.st_size;
taosNoteLines = size / 60;
pNote->taosNoteLines = size / 60;
lseek(taosNoteFd, 0, SEEK_END);
lseek(pNote->taosNoteFd, 0, SEEK_END);
return 0;
}
void taosNotePrint(const char * const format, ...)
void taosNotePrint(taosNoteInfo * pNote, const char * const format, ...)
{
va_list argpointer;
char buffer[MAX_NOTE_LINE_SIZE];
@ -222,7 +231,7 @@ void taosNotePrint(const char * const format, ...)
gettimeofday(&timeSecs, NULL);
curTime = timeSecs.tv_sec;
ptm = localtime_r(&curTime, &Tm);
len = sprintf(buffer, "%02d/%02d %02d:%02d:%02d.%06d ", ptm->tm_mon + 1, ptm->tm_mday, ptm->tm_hour, ptm->tm_min, ptm->tm_sec, (int)timeSecs.tv_usec);
len = sprintf(buffer, "%02d/%02d %02d:%02d:%02d.%06d %lx ", ptm->tm_mon + 1, ptm->tm_mday, ptm->tm_hour, ptm->tm_min, ptm->tm_sec, (int)timeSecs.tv_usec, pthread_self());
va_start(argpointer, format);
len += vsnprintf(buffer + len, MAX_NOTE_LINE_SIZE - len, format, argpointer);
@ -233,26 +242,26 @@ void taosNotePrint(const char * const format, ...)
buffer[len++] = '\n';
buffer[len] = 0;
if (taosNoteFd >= 0) {
twrite(taosNoteFd, buffer, (unsigned int)len);
if (pNote->taosNoteFd >= 0) {
twrite(pNote->taosNoteFd, buffer, (unsigned int)len);
if (taosNoteMaxLines > 0) {
taosNoteLines++;
if ((taosNoteLines > taosNoteMaxLines) && (taosNoteOpenInProgress == 0))
taosOpenNewNote();
if (pNote->taosNoteMaxLines > 0) {
pNote->taosNoteLines++;
if ((pNote->taosNoteLines > pNote->taosNoteMaxLines) && (pNote->taosNoteOpenInProgress == 0))
taosOpenNewNote(pNote);
}
}
}
void taosCloseNote()
void taosCloseNote(taosNoteInfo * pNote)
{
taosCloseNoteByFd(taosNoteFd);
taosCloseNoteByFd(pNote->taosNoteFd, pNote);
}
void taosCloseNoteByFd(int fd)
void taosCloseNoteByFd(int fd, taosNoteInfo * pNote)
{
if (fd >= 0) {
taosUnLockNote(fd);
taosUnLockNote(fd, pNote);
close(fd);
}
}

105
src/util/src/tstatus.c
View File

@ -15,52 +15,89 @@
#include "taosmsg.h"
#include "tsdb.h"
#include "tstatus.h"
const char* taosGetVgroupStatusStr(int vgroupStatus) {
switch (vgroupStatus) {
case TSDB_VG_STATUS_READY: return "ready";
case TSDB_VG_STATUS_IN_PROGRESS: return "inprogress";
case TSDB_VG_STATUS_COMMITLOG_INIT_FAILED: return "commitlog_init_failed";
case TSDB_VG_STATUS_INIT_FAILED: return "init_failed";
case TSDB_VG_STATUS_FULL: return "full";
default: return "undefined";
}
}
const char* taosGetDbStatusStr(int dbStatus) {
switch (dbStatus) {
case TSDB_DB_STATUS_READY: return "ready";
case TSDB_DB_STATUS_DROPPING: return "dropping";
case TSDB_DB_STATUS_DROP_FROM_SDB: return "drop_from_sdb";
default: return "undefined";
}
}
const char* taosGetVnodeStatusStr(int vnodeStatus) {
switch (vnodeStatus) {
case TSDB_VNODE_STATUS_OFFLINE:return "offline";
case TSDB_VNODE_STATUS_CREATING: return "creating";
case TSDB_VNODE_STATUS_UNSYNCED: return "unsynced";
case TSDB_VNODE_STATUS_SLAVE: return "slave";
case TSDB_VNODE_STATUS_MASTER: return "master";
case TSDB_VNODE_STATUS_CLOSING: return "closing";
case TSDB_VNODE_STATUS_DELETING: return "deleting";
default: return "undefined";
}
}
const char* taosGetDnodeStatusStr(int dnodeStatus) {
switch (dnodeStatus) {
case TSDB_DNODE_STATUS_OFFLINE: return "offline";
case TSDB_DNODE_STATUS_READY: return "ready";
default: return "undefined";
}
}
const char* taosGetDnodeBalanceStateStr(int dnodeBalanceStatus) {
switch (dnodeBalanceStatus) {
case LB_DNODE_STATE_BALANCED: return "balanced";
case LB_DNODE_STATE_BALANCING: return "balancing";
case LB_DNODE_STATE_OFFLINE_REMOVING: return "offline removing";
case LB_DNODE_STATE_SHELL_REMOVING: return "removing";
default: return "undefined";
case TSDB_VN_STATUS_OFFLINE: return "offline";
case TSDB_VN_STATUS_CREATING: return "creating";
case TSDB_VN_STATUS_UNSYNCED: return "unsynced";
case TSDB_VN_STATUS_SLAVE: return "slave";
case TSDB_VN_STATUS_MASTER: return "master";
case TSDB_VN_STATUS_CLOSING: return "closing";
case TSDB_VN_STATUS_DELETING: return "deleting";
default: return "undefined";
}
}
const char* taosGetVnodeSyncStatusStr(int vnodeSyncStatus) {
switch (vnodeSyncStatus) {
case STDB_SSTATUS_INIT: return "init";
case TSDB_SSTATUS_SYNCING: return "syncing";
case TSDB_SSTATUS_SYNC_CACHE: return "sync_cache";
case TSDB_SSTATUS_SYNC_FILE: return "sync_file";
default: return "undefined";
case TSDB_VN_SYNC_STATUS_INIT: return "init";
case TSDB_VN_SYNC_STATUS_SYNCING: return "syncing";
case TSDB_VN_SYNC_STATUS_SYNC_CACHE: return "sync_cache";
case TSDB_VN_SYNC_STATUS_SYNC_FILE: return "sync_file";
default: return "undefined";
}
}
const char* taosGetVnodeDropStatusStr(int dropping) {
switch (dropping) {
case 0: return "ready";
case 1: return "dropping";
default: return "undefined";
case TSDB_VN_DROP_STATUS_READY: return "ready";
case TSDB_VN_DROP_STATUS_DROPPING: return "dropping";
default: return "undefined";
}
}
const char* taosGetDnodeStatusStr(int dnodeStatus) {
switch (dnodeStatus) {
case TSDB_DN_STATUS_OFFLINE: return "offline";
case TSDB_DN_STATUS_READY: return "ready";
default: return "undefined";
}
}
const char* taosGetDnodeLbStatusStr(int dnodeBalanceStatus) {
switch (dnodeBalanceStatus) {
case TSDB_DN_LB_STATUS_BALANCED: return "balanced";
case TSDB_DN_LB_STATUS_BALANCING: return "balancing";
case TSDB_DN_LB_STATUS_OFFLINE_REMOVING: return "offline removing";
case TSDB_DN_LB_STATE_SHELL_REMOVING: return "removing";
default: return "undefined";
}
}
const char* taosGetVgroupLbStatusStr(int vglbStatus) {
switch (vglbStatus) {
case TSDB_VG_LB_STATUS_READY: return "ready";
case TSDB_VG_LB_STATUS_UPDATE: return "updating";
default: return "undefined";
}
}
const char* taosGetVnodeStreamStatusStr(int vnodeStreamStatus) {
switch (vnodeStreamStatus) {
case TSDB_VN_STREAM_STATUS_START: return "start";
case TSDB_VN_STREAM_STATUS_STOP: return "stop";
default: return "undefined";
}
}

142
src/util/src/tstoken.c
View File

@ -1,142 +0,0 @@
/*
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
*
* This program is free software: you can use, redistribute, and/or modify
* it under the terms of the GNU Affero General Public License, version 3
* or later ("AGPL"), as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <ctype.h>
#include <pthread.h>
#include <stdbool.h>
#include <stdint.h>
#include <string.h>
#include "os.h"
#include "shash.h"
#include "tstoken.h"
void shiftStr(char *dst, char *src);
static char operator[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, '$', '%', '&', 0, '(', ')', '*', '+',
0, '-', 0, '/', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, '<', '=', '>', 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, '[', 0, ']', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, '|', 0, 0, 0};
static char delimiter[] = {
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ',', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ';', 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
};
bool isCharInDelimiter(char c, char *delimiter) {
for (int i = 0; i < strlen(delimiter); i++) {
if (delimiter[i] == c) return true;
}
return false;
}
char *tscGetTokenDelimiter(char *string, char **token, int *tokenLen, char *delimiters) {
while (*string != 0) {
if (isCharInDelimiter(*string, delimiters)) {
++string;
} else {
break;
}
}
*token = string;
char *str = string;
*tokenLen = 0;
while (*str != 0) {
if (!isCharInDelimiter(*str, delimiters)) {
*tokenLen = *tokenLen + 1;
str++;
} else {
break;
}
}
return string;
}
char *tscGetToken(char *string, char **token, int *tokenLen) {
char quote = 0;
while (*string != 0) {
if (delimiter[*string]) {
++string;
} else {
break;
}
}
char quotaChar = 0;
if (*string == '\'' || *string == '\"') {
quote = 1;
quotaChar = *string;
string++;
}
*token = string;
/* not in string, return token */
if (*string > 0 && operator[*string] && quote == 0) {
string++;
/* handle the case: insert into tabx using stable1 tags(-1)/tags(+1)
* values(....) */
if (operator[*string] &&(*string != '(' && *string != ')' && *string != '-' && *string != '+'))
*tokenLen = 2;
else
*tokenLen = 1;
return *token + *tokenLen;
}
while (*string != 0) {
if (quote) {
// handle escape situation: '\"', the " should not be eliminated
if (*string == quotaChar) {
if (*(string - 1) != '\\') {
break;
} else {
shiftStr(string - 1, string);
}
} else {
++string;
}
} else {
if (delimiter[*string]) break;
if (operator[*string]) break;
++string;
}
}
*tokenLen = (int)(string - *token);
if (quotaChar != 0 && *string != 0 && *(string + 1) != 0) {
return string + 1;
} else {
return string;
}
}
void shiftStr(char *dst, char *src) {
int32_t i = 0;
do {
dst[i] = src[i];
i++;
} while (delimiter[src[i]] == 0);
src[i - 1] = ' ';
}

54
src/util/src/ttimer.c
View File

@ -14,6 +14,7 @@
*/
#include "os.h"
#include <inttypes.h>
#include "tlog.h"
#include "tsched.h"
#include "ttime.h"
@ -253,13 +254,13 @@ static void processExpiredTimer(void* handle, void* arg) {
timer->executedBy = taosGetPthreadId();
uint8_t state = atomic_val_compare_exchange_8(&timer->state, TIMER_STATE_WAITING, TIMER_STATE_EXPIRED);
if (state == TIMER_STATE_WAITING) {
const char* fmt = "%s timer[id=%lld, fp=%p, param=%p] execution start.";
const char* fmt = "%s timer[id=" PRIuPTR ", fp=%p, param=%p] execution start.";
tmrTrace(fmt, timer->ctrl->label, timer->id, timer->fp, timer->param);
(*timer->fp)(timer->param, (tmr_h)timer->id);
atomic_store_8(&timer->state, TIMER_STATE_STOPPED);
fmt = "%s timer[id=%lld, fp=%p, param=%p] execution end.";
fmt = "%s timer[id=" PRIuPTR ", fp=%p, param=%p] execution end.";
tmrTrace(fmt, timer->ctrl->label, timer->id, timer->fp, timer->param);
}
removeTimer(timer->id);
@ -267,7 +268,7 @@ static void processExpiredTimer(void* handle, void* arg) {
}
static void addToExpired(tmr_obj_t* head) {
const char* fmt = "%s adding expired timer[id=%lld, fp=%p, param=%p] to queue.";
const char* fmt = "%s adding expired timer[id=" PRIuPTR ", fp=%p, param=%p] to queue.";
while (head != NULL) {
uintptr_t id = head->id;
@ -281,7 +282,7 @@ static void addToExpired(tmr_obj_t* head) {
schedMsg.thandle = NULL;
taosScheduleTask(tmrQhandle, &schedMsg);
tmrTrace("timer[id=%lld] has been added to queue.", id);
tmrTrace("timer[id=" PRIuPTR "] has been added to queue.", id);
head = next;
}
}
@ -295,7 +296,7 @@ static uintptr_t doStartTimer(tmr_obj_t* timer, TAOS_TMR_CALLBACK fp, int msecon
timer->ctrl = ctrl;
addTimer(timer);
const char* fmt = "%s timer[id=%lld, fp=%p, param=%p] started";
const char* fmt = "%s timer[id=" PRIuPTR ", fp=%p, param=%p] started";
tmrTrace(fmt, ctrl->label, timer->id, timer->fp, timer->param);
if (mseconds == 0) {
@ -380,40 +381,37 @@ static void taosTimerLoopFunc(int signo) {
}
static bool doStopTimer(tmr_obj_t* timer, uint8_t state) {
bool reusable = false;
if (state == TIMER_STATE_WAITING) {
bool reusable = false;
if (removeFromWheel(timer)) {
removeTimer(timer->id);
// only safe to reuse the timer when timer is removed from the wheel.
// we cannot guarantee the thread safety of the timr in all other cases.
reusable = true;
}
const char* fmt = "%s timer[id=%lld, fp=%p, param=%p] is cancelled.";
const char* fmt = "%s timer[id=" PRIuPTR ", fp=%p, param=%p] is cancelled.";
tmrTrace(fmt, timer->ctrl->label, timer->id, timer->fp, timer->param);
} else if (state != TIMER_STATE_EXPIRED) {
return reusable;
}
if (state != TIMER_STATE_EXPIRED) {
// timer already stopped or cancelled, has nothing to do in this case
} else if (timer->executedBy == taosGetPthreadId()) {
return false;
}
if (timer->executedBy == taosGetPthreadId()) {
// taosTmrReset is called in the timer callback, should do nothing in this
// case to avoid dead lock. note taosTmrReset must be the last statement
// of the callback funtion, will be a bug otherwise.
} else {
assert(timer->executedBy != taosGetPthreadId());
const char* fmt = "%s timer[id=%lld, fp=%p, param=%p] fired, waiting...";
tmrTrace(fmt, timer->ctrl->label, timer->id, timer->fp, timer->param);
for (int i = 1; atomic_load_8(&timer->state) != TIMER_STATE_STOPPED; i++) {
if (i % 1000 == 0) {
sched_yield();
}
}
fmt = "%s timer[id=%lld, fp=%p, param=%p] stopped.";
tmrTrace(fmt, timer->ctrl->label, timer->id, timer->fp, timer->param);
return false;
}
return reusable;
// timer callback is executing in another thread, we SHOULD wait it to stop,
// BUT this may result in dead lock if current thread are holding a lock which
// the timer callback need to acquire. so, we HAVE TO return directly.
const char* fmt = "%s timer[id=" PRIuPTR ", fp=%p, param=%p] is executing and cannot be stopped.";
tmrTrace(fmt, timer->ctrl->label, timer->id, timer->fp, timer->param);
return false;
}
bool taosTmrStop(tmr_h timerId) {
@ -421,7 +419,7 @@ bool taosTmrStop(tmr_h timerId) {
tmr_obj_t* timer = findTimer(id);
if (timer == NULL) {
tmrTrace("timer[id=%lld] does not exist", id);
tmrTrace("timer[id=" PRIuPTR "] does not exist", id);
return false;
}
@ -448,7 +446,7 @@ bool taosTmrReset(TAOS_TMR_CALLBACK fp, int mseconds, void* param, void* handle,
bool stopped = false;
tmr_obj_t* timer = findTimer(id);
if (timer == NULL) {
tmrTrace("%s timer[id=%lld] does not exist", ctrl->label, id);
tmrTrace("%s timer[id=" PRIuPTR "] does not exist", ctrl->label, id);
} else {
uint8_t state = atomic_val_compare_exchange_8(&timer->state, TIMER_STATE_WAITING, TIMER_STATE_CANCELED);
if (!doStopTimer(timer, state)) {
@ -463,7 +461,7 @@ bool taosTmrReset(TAOS_TMR_CALLBACK fp, int mseconds, void* param, void* handle,
return stopped;
}
tmrTrace("%s timer[id=%lld] is reused", ctrl->label, timer->id);
tmrTrace("%s timer[id=" PRIuPTR "] is reused", ctrl->label, timer->id);
// wait until there's no other reference to this timer,
// so that we can reuse this timer safely.

11
src/util/src/ttokenizer.c
View File

@ -13,14 +13,12 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <ctype.h>
#include <pthread.h>
#include <string.h>
#include "os.h"
#include "shash.h"
#include "tsql.h"
#include "tutil.h"
#include "tsqldef.h"
#include "tstoken.h"
#include "ttypes.h"
// All the keywords of the SQL language are stored in a hash table
typedef struct SKeyword {
@ -96,7 +94,6 @@ static SKeyword keywordTable[] = {
{"TABLE", TK_TABLE},
{"DATABASE", TK_DATABASE},
{"DNODE", TK_DNODE},
{"IP", TK_IP},
{"USER", TK_USER},
{"ACCOUNT", TK_ACCOUNT},
{"USE", TK_USE},
@ -523,7 +520,7 @@ uint32_t tSQLGetToken(char* z, uint32_t* tokenType) {
}
if (seg == 4) { // ip address
*tokenType = TK_IP;
*tokenType = TK_IPTOKEN;
return i;
}

3
src/util/src/ttypes.c
View File

@ -16,7 +16,6 @@
#include "os.h"
#include "taos.h"
#include "tsdb.h"
#include "tsql.h"
#include "tsqldef.h"
#include "ttypes.h"
#include "tutil.h"
@ -67,6 +66,8 @@ bool isValidDataType(int32_t type, int32_t length) {
void tVariantCreate(tVariant *pVar, SSQLToken *token) { tVariantCreateFromString(pVar, token->z, token->n, token->type); }
void tVariantCreateFromString(tVariant *pVar, char *pz, uint32_t len, uint32_t type) {
memset(pVar, 0, sizeof(tVariant));
switch (type) {
case TSDB_DATA_TYPE_BOOL: {
int32_t k = strncasecmp(pz, "true", 4);

2
src/util/src/tutil.c
View File

@ -490,7 +490,7 @@ char *taosIpStr(uint32_t ipInt) {
static int ipStrIndex = 0;
char *ipStr = ipStrArray[(ipStrIndex++) % 3];
sprintf(ipStr, "0x%x:%d.%d.%d.%d", ipInt, ipInt & 0xFF, (ipInt >> 8) & 0xFF, (ipInt >> 16) & 0xFF, ipInt >> 24);
sprintf(ipStr, "0x%x:%u.%u.%u.%u", ipInt, ipInt & 0xFF, (ipInt >> 8) & 0xFF, (ipInt >> 16) & 0xFF, (uint8_t)(ipInt >> 24));
return ipStr;
}