本公司涉及到一部分数据推送的业务，然后因为是配置数据传输任务的方式做的，但是有时候数据任务会出现问题，可能不推送等等情况，所以需要有个程序监控是否数据推送完成，简单就是检测规则就是定时跑源数据表和推送目标表的最大ID，然后进行比较。我用Java开发，成本有点大，就直接用Shell脚本简单写一下。
#! /bin/bash
# 将定时跑批任务的结果数据到一张表里
MANAGE_USERNAME=root
MANAGE_PASSWORD=root
MANAGE_PORT=3306
MANAGE_IPADDRESS=192.168.21.103# 目标库
PRE_USERNAME=root
PRE_PASSWORD=root
PRE_PORT=3306
PRE_IPADDRESS=192.168.21.104# 查询需要跑批的数据表
tables_names=`mysql -u${MANAGE_USERNAME} -p${MANAGE_PASSWORD} -h ${MANAGE_IPADDRESS} -P${MANAGE_PORT} -e "select table_name from test.table_push_info where table_status=0;" | awk 'NR>1{print $1}'`
for name in $tables_names;# 打印看一下效果echo $name
拿到表之后就可以用来跑批
max_id=`mysql -u${PRE_USERNAME} -p${PRE_PASSWORD} -h ${PRE_IPADDRESS} -P${PRE_PORT} -e "select COALESCE(id,0) from test.$name order by id desc limit 1;" | tail -1`
# 因为存在如果是没有数据，返回空，这边就判断为空的时候，插入0if [ ! $max_id ];thenmysql -u${MANAGE_USERNAME} -p${MANAGE_PASSWORD} -h ${MANAGE_IPADDRESS} -P${MANAGE_PORT} test -e "update test.table_push_info set max_id=0,update_time=now() where table_name='$name';" else mysql -u${MANAGE_USERNAME} -p${MANAGE_PASSWORD} -h ${MANAGE_IPADDRESS} -P${MANAGE_PORT} test -e "update test.table_push_info set max_id=$max_id,update_time=now() where table_name='$name';" fi
上述就是简单完成了
#! /bin/bash
# 将定时跑批任务的结果数据到一张表里
MANAGE_USERNAME=root
MANAGE_PASSWORD=root
MANAGE_PORT=3306
MANAGE_IPADDRESS=192.168.21.103# 目标库
PRE_USERNAME=root
PRE_PASSWORD=root
PRE_PORT=3306
PRE_IPADDRESS=192.168.21.104# 查询需要跑批的数据表
tables_names=`mysql -u${MANAGE_USERNAME} -p${MANAGE_PASSWORD} -h ${MANAGE_IPADDRESS} -P${MANAGE_PORT} -e "select table_name from test.table_push_info where table_status=0;" | awk 'NR>1{print $1}'`
for name in $tables_names;
do max_id=`mysql -u${PRE_USERNAME} -p${PRE_PASSWORD} -h ${PRE_IPADDRESS} -P${PRE_PORT} -e "select COALESCE(id,0) from test.$name order by id desc limit 1;" | tail -1`# 因为存在如果是没有数据，返回空，这边就判断为空的时候，插入0if [ ! $max_id ];thenmysql -u${MANAGE_USERNAME} -p${MANAGE_PASSWORD} -h ${MANAGE_IPADDRESS} -P${MANAGE_PORT} test -e "update test.table_push_info set max_id=0,update_time=now() where table_name='$name';" else mysql -u${MANAGE_USERNAME} -p${MANAGE_PASSWORD} -h ${MANAGE_IPADDRESS} -P${MANAGE_PORT} test -e "update test.table_push_info set max_id=$max_id,update_time=now() where table_name='$name';" fi
done
之后将将该程序部署到crontab就好
> chmod +x update_id.sh
> crontab -e
0 * * * * /data/update_id.sh
这样就可以每小时执行一次脚本
)