flume获取mysql日志到hdfs_Hadoop实战：Flume输入日志到HDFS报错解决

使用Flume把日志存储到HDFS，在启动时报错如下：

2017-06-16 08:58:32,634 (conf-file-poller-0) [ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:146)] Failed to start agent because dependencies were not found in classpath. Error follows.

java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType

at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:235)

at org.apache.flume.conf.Configurables.configure(Configurables.java:41)

at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:411)

at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)

at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:141)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 12 more

2017-06-1608:58:32,634(conf-file-poller-0)[ERROR-org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:146)]Failedtostartagentbecausedependencieswerenotfoundinclasspath.Errorfollows.

java.lang.NoClassDefFoundError:org/apache/hadoop/io/SequenceFile$CompressionType

atorg.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:235)

atorg.apache.flume.conf.Configurables.configure(Configurables.java:41)

atorg.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:411)

atorg.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)

atorg.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:141)

atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

atjava.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

atjava.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

atjava.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

atjava.lang.Thread.run(Thread.java:748)

Causedby:java.lang.ClassNotFoundException:org.apache.hadoop.io.SequenceFile$CompressionType

atjava.net.URLClassLoader.findClass(URLClassLoader.java:381)

atjava.lang.ClassLoader.loadClass(ClassLoader.java:424)

atsun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)

atjava.lang.ClassLoader.loadClass(ClassLoader.java:357)

...12more

其实这个报错信息真的不太友好，看不错哪里有问题。第一眼看上去一直以为是JAVA的classpath有问题，后来在网上看到一篇文章：安装Flume遇到的问题及解决，说是缺少Hadoop相关的jar包导致，想想也对。解决这个问题也很简单，就是在Flume主机上解压好Hadoop的二进制安装包，然后输出Hadoop环境变量即可，Flume会根据环境变量自动找到相关的依赖jar包。

由于我这里的Flume是独立安装，没有部署在Hadoop机器上，所以需要在Flume主机上安装一个Hadoop。

Hadoop二进制包下载自行去国内源或者官网搞定。

$ tar xvf hadoop-2.8.0.tar.gz -C /usr/local/

$ ln -sv /usr/local/hadoop-2.8.0/ /usr/local/hadoop

$ useradd hadoop

$ passwd hadoop

$tarxvfhadoop-2.8.0.tar.gz-C/usr/local/

$ln-sv/usr/local/hadoop-2.8.0//usr/local/hadoop

$useraddhadoop

$passwdhadoop

然后最主要的就是输出Hadoop环境变量，编辑环境配置文件/etc/profile.d/hadoop.sh，定义类似如下环境变量，设定Hadoop的运行环境。

#!/bin/bash

export HADOOP_PREFIX="/usr/local/hadoop"

export PATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin

export HADOOP_COMMON_HOME=${HADOOP_PREFIX}

export HADOOP_HDFS_HOME=${HADOOP_PREFIX}

export HADOOP_MAPRED_HOME=${HADOOP_PREFIX}

export HADOOP_YARN_HOME=${HADOOP_PREFIX}

#!/bin/bash

exportHADOOP_PREFIX="/usr/local/hadoop"

exportPATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin

exportHADOOP_COMMON_HOME=${HADOOP_PREFIX}

exportHADOOP_HDFS_HOME=${HADOOP_PREFIX}

exportHADOOP_MAPRED_HOME=${HADOOP_PREFIX}

exportHADOOP_YARN_HOME=${HADOOP_PREFIX}

$ source /etc/profile

$source/etc/profile

再次运行Flume-ng就应该可以了。

另外，当Flume-ng正常运行后，写入HDFS时报错：java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=root, access=WRITE, inode=”/”:hadoop:supergroup:drwxr-xr-x.

这个提示很明显，就是没有写入权限(因为你当前运行flume-ng的用户不是Hadoop用户)，解决方案也很简单，就是切换到Hadoop用户执行flume-ng命令即可。

$ chown hadoop.hadoop -R /usr/local/flume/

$ su - hadoop

$ flume-ng .....

$chownhadoop.hadoop-R/usr/local/flume/

$su-hadoop

$flume-ng.....

或者开启HDFS允许所有用户进行文件写入，默认可能你没有开启。需要在hdfs-site.xml配置文件中添加一项属性定义：

dfs.permissions

false

dfs.permissions

false