Sqoop安装与配置-shell脚本一键安装配置

文章目录

前言
一、使用shell脚本一键安装
- 1. 复制脚本
- 2. 增加执行权限
- 3. 执行脚本
- 4. 加载用户环境变量
- 5. 查看是否安装成功
总结

前言

本文介绍了如何使用Shell脚本一键安装Sqoop。Sqoop是一个用于在Apache Hadoop和结构化数据存储（如关系数据库）之间传输数据的工具。通过使用Sqoop，您可以方便地将数据从关系数据库导入到Hadoop集群中进行分析，并将结果数据导出回关系数据库。

在本文中，我们将提供一个Shell脚本，该脚本将自动下载、安装和配置Sqoop。您只需按照步骤执行脚本，即可完成Sqoop的安装和配置过程。

一、使用shell脚本一键安装

1. 复制脚本

首先，将以下脚本内容复制并保存为/tmp/install_sqoop.sh文件。

#!/bin/bashinstallDir="/opt/module/sqoop"
sqoop_version="1.4.6"
mysql_connector_version="5.1.37"if [ -z "$(command -v wget)" ]; thensudo yum install -y wgetecho "wget安装完成"
fi
if [ -z "$JAVA_HOME" ]; thenecho "JAVA_HOME未设置，请安装jdk1.8，设置Java环境变量再来执行此脚本"exit 1
fi
if [ ! -d "${installDir}" ]; thensudo mkdir -p "${installDir}"if [ $? -eq 0 ]; thenecho "安装目录${installDir}已创建"elseecho "请确保您有足够的权限来创建目录，请增加权限后再次执行"exit 1fi
fi
if [ ! -f /tmp/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz ]; thenwget https://archive.apache.org/dist/sqoop/$sqoop_version/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz -P /tmp/if [ $? -eq 0 ]; thenecho "sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz下载成功"elseecho "sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz下载失败，请重试或手动下载到/tmp目录下再次执行"echo "下载地址：https://archive.apache.org/dist/sqoop/$sqoop_version/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz"exit 1fi
fiif [ -d $installDir/sqoop-$sqoop_version ]; thenecho "$installDir/sqoop-$sqoop_version 已存在，正在删除..."sudo rm -rf $installDir/sqoop-$sqoop_version
fi
tar -zxvf /tmp/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz -C $installDir
if [ $? -eq 0 ]; thenecho "/tmp/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz解压成功"
elseecho "/tmp/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha.tar.gz解压失败，请查看异常信息后重试"exit 1
fi#设置sqoop环境变量
if [ -z "$SQOOP_HOME" ]; thenecho >> ~/.bashrcecho '#SQOOP_HOME' >> ~/.bashrcecho "export SQOOP_HOME=$installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha" >> ~/.bashrcecho 'export PATH=$PATH:$SQOOP_HOME/bin' >> ~/.bashrc
elseecho "SQOOP_HOME已有设置：$SQOOP_HOME"
ficp $installDir/sqoop-"$sqoop_version".bin__hadoop-2.0.4-alpha/conf/sqoop-env-template.sh $installDir/sqoop-"$sqoop_version".bin__hadoop-2.0.4-alpha/conf/sqoop-env.sh
if [ $? -ne 0 ]; thenecho "cp $installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/conf/sqoop-env-template.sh $installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/conf/sqoop-env.sh 失败"exit 1
fi
#设置bin/hadoop所在的路径
if [ -n "$HADOOP_HOME" ]; thensed -i "s|^#export HADOOP_COMMON_HOME=.*|export HADOOP_COMMON_HOME=$HADOOP_HOME|" "$installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/conf/sqoop-env.sh"if [ $? -ne 0 ]; thenecho "修改export HADOOP_COMMON_HOME失败"exit 1fi
fi
#设置hadoop-*-core.jar可用的路径
if [ -n "$HADOOP_HOME" ]; thensed -i "s|^#export HADOOP_MAPRED_HOME=.*|export HADOOP_MAPRED_HOME=$HADOOP_HOME|" "$installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/conf/sqoop-env.sh"if [ $? -ne 0 ]; thenecho "修改export HADOOP_MAPRED_HOME失败"exit 1fi
fi
#设置bin/hbase所在路径
if [ -n "$HBASE_HOME" ]; thensed -i "s|^#export HBASE_HOME=.*|export HBASE_HOME=$HBASE_HOME|" "$installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/conf/sqoop-env.sh"if [ $? -ne 0 ]; thenecho "修改export HBASE_HOME失败"exit 1fi
fi
#设置bin/hive可用的路径
if [ -n "$HIVE_HOME" ]; thensed -i "s|^#export HIVE_HOME=.*|export HIVE_HOME=$HIVE_HOME|" "$installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/conf/sqoop-env.sh"if [ $? -ne 0 ]; thenecho "修改export HIVE_HOME失败"exit 1fi
fi
#设置zookeeper config dir所在的路径
if [ -n "$ZOOKEEPER_HOME" ]; thensed -i "s|^#export ZOOCFGDIR=.*|export ZOOCFGDIR=$ZOOKEEPER_HOME/conf|" "$installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/conf/sqoop-env.sh"if [ $? -ne 0 ]; thenecho "修改export ZOOCFGDIR失败"exit 1fi
fiif [ ! -f /tmp/mysql-connector-java-"$mysql_connector_version".jar ]; thenwget https://repo1.maven.org/maven2/mysql/mysql-connector-java/"$mysql_connector_version"/mysql-connector-java-"$mysql_connector_version".jar -P /tmpif [ $? -eq 0 ]; thenecho "mysql驱动下载成功"elseecho "mysql驱动下载失败，请手动下载到/tmp目录下再次执行次脚本"echo "下载地址：https://repo1.maven.org/maven2/mysql/mysql-connector-java/$mysql_connector_version/mysql-connector-java-$mysql_connector_version.jar"exit 1fi
ficp /tmp/mysql-connector-java-"$mysql_connector_version".jar $installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/lib
if [ $? -ne 0 ]; thenecho "cp /tmp/mysql-connector-java-"$mysql_connector_version".jar $installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/lib 失败"exit 1
ficp $HIVE_HOME/lib/hive-common-*.jar $installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/lib
if [ $? -ne 0 ]; thenecho "cp $HIVE_HOME/lib/hive-common-*.jar $installDir/sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha/lib 失败"exit 1
fiecho "sqoop-$sqoop_version.bin__hadoop-2.0.4-alpha 下载、安装、配置成功"exit 0