Hadoop64位版本安装后遇到的警告处理

在使用hadoop的过程中，会遇到一个警告，内容如下：

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

对于这个问题网上很多说法是由于系统位数和所下载的hadoop的位数不同造成的，说到这里就需要看一下自己的hadoop的位数了，查看方法如下：

1.进入到hadoop的安装文件夹下；

2.进入如下目录：

3.看到上图中的libhadoop.so后，使用file命令：

到这里，就可以完全知道了自己的hadoop的版本是32位的还是64位的了；

如果确实是因为位数不一样，ok那么只能选择下载源代码然后自己编译了；但是我这里遇到的不是这个问题，因为我的是64位的操作系统，并且我的hadoop也是64位的；那么问题出在哪里呢？

编辑一下/etc/profile，让hadoop打印日志到console中，来看一下；

1.给/etc/profile 中加入如下内容：

export HADOOP_ROOT_LOGGER=DEBUG,console

截图如下：

然后source一下/etc/profile让它生效：

source /etc/profile

2.执行任意的hadoop命令，来看一下弹出的警告信息，主要内容如下

[hadoop@hadoop2 ~]$ hdfs dfs -ls /
17/01/13 14:04:39 DEBUG util.Shell: setsid exited with exit code 0
17/01/13 14:04:39 DEBUG conf.Configuration: parsing URL jar:file:/home/hadoop/app/hadoop-2.5.2/share/hadoop/common/hadoop-common-2.5.2.jar!/core-default.xml
17/01/13 14:04:39 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@6e90891
17/01/13 14:04:39 DEBUG conf.Configuration: parsing URL file:/home/hadoop/app/hadoop-2.5.2/etc/hadoop/core-site.xml
17/01/13 14:04:39 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@3021eb3f
17/01/13 14:04:40 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[Rate of successful kerberos logins and latency (milliseconds)], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops)
17/01/13 14:04:40 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[Rate of failed kerberos logins and latency (milliseconds)], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops)
17/01/13 14:04:40 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[GetGroups], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops)
17/01/13 14:04:40 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
17/01/13 14:04:41 DEBUG security.Groups:  Creating new Groups object
17/01/13 14:04:41 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
17/01/13 14:04:41 DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: /home/hadoop/app/hadoop-2.5.2/lib/native/libhadoop.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/hadoop/app/hadoop-2.5.2/lib/native/libhadoop.so)
17/01/13 14:04:41 DEBUG util.NativeCodeLoader: java.library.path=/home/hadoop/app/hadoop-2.5.2/lib/native
17/01/13 14:04:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/01/13 14:04:41 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Falling back to shell based
17/01/13 14:04:41 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
17/01/13 14:04:41 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
17/01/13 14:04:41 DEBUG security.UserGroupInformation: hadoop login
17/01/13 14:04:41 DEBUG security.UserGroupInformation: hadoop login commit
17/01/13 14:04:41 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: hadoop
17/01/13 14:04:41 DEBUG security.UserGroupInformation: UGI loginUser:hadoop (auth:SIMPLE)
17/01/13 14:04:42 DEBUG hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
17/01/13 14:04:42 DEBUG hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = false
17/01/13 14:04:42 DEBUG hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
17/01/13 14:04:42 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = 
17/01/13 14:04:42 DEBUG hdfs.HAUtil: No HA service delegation token found for logical URI hdfs://ns1/
17/01/13 14:04:42 DEBUG hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
17/01/13 14:04:42 DEBUG hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = false
17/01/13 14:04:42 DEBUG hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
17/01/13 14:04:42 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = 
17/01/13 14:04:42 DEBUG retry.RetryUtils: multipleLinearRandomRetry = null
17/01/13 14:04:42 DEBUG ipc.Server: rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWrapper, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@440b2a8c
17/01/13 14:04:42 DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@2301799f
17/01/13 14:04:43 DEBUG shortcircuit.DomainSocketFactory: Both short-circuit local reads and UNIX domain socket are disabled.
17/01/13 14:04:43 DEBUG ipc.Client: The ping interval is 60000 ms.
17/01/13 14:04:43 DEBUG ipc.Client: Connecting to hadoop1/192.168.1.232:9000
17/01/13 14:04:43 DEBUG ipc.Client: IPC Client (382257985) connection to hadoop1/192.168.1.232:9000 from hadoop: starting, having connections 1
17/01/13 14:04:43 DEBUG ipc.Client: IPC Client (382257985) connection to hadoop1/192.168.1.232:9000 from hadoop sending #0
17/01/13 14:04:43 DEBUG ipc.Client: IPC Client (382257985) connection to hadoop1/192.168.1.232:9000 from hadoop got value #0
17/01/13 14:04:43 INFO retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over hadoop1/192.168.1.232:9000. Trying to fail over immediately.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standbyat org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1688)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1258)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3684)at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:803)at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:779)at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:415)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)at org.apache.hadoop.ipc.Client.call(Client.java:1411)at org.apache.hadoop.ipc.Client.call(Client.java:1364)at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:707)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1785)at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1068)at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)at org.apache.hadoop.fs.Globber.glob(Globber.java:265)at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:224)at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:207)at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)at org.apache.hadoop.fs.shell.Command.run(Command.java:154)at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
17/01/13 14:04:43 DEBUG retry.RetryUtils: multipleLinearRandomRetry = null
17/01/13 14:04:43 DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@2301799f
17/01/13 14:04:43 DEBUG ipc.Client: The ping interval is 60000 ms.
17/01/13 14:04:43 DEBUG ipc.Client: Connecting to hadoop2/192.168.1.233:9000
17/01/13 14:04:43 DEBUG ipc.Client: IPC Client (382257985) connection to hadoop2/192.168.1.233:9000 from hadoop sending #0
17/01/13 14:04:43 DEBUG ipc.Client: IPC Client (382257985) connection to hadoop2/192.168.1.233:9000 from hadoop: starting, having connections 2
17/01/13 14:04:43 DEBUG ipc.Client: IPC Client (382257985) connection to hadoop2/192.168.1.233:9000 from hadoop got value #0
17/01/13 14:04:43 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 8ms
17/01/13 14:04:43 DEBUG ipc.Client: IPC Client (382257985) connection to hadoop2/192.168.1.233:9000 from hadoop sending #1
17/01/13 14:04:43 DEBUG ipc.Client: IPC Client (382257985) connection to hadoop2/192.168.1.233:9000 from hadoop got value #1
17/01/13 14:04:43 DEBUG ipc.ProtobufRpcEngine: Call: getListing took 6ms
Found 3 items
-rw-r--r--   3 hadoop supergroup  179161400 2017-01-09 13:35 /apache-storm-1.0.2.tar.gz
-rw-r--r--   3 hadoop supergroup  147197492 2017-01-07 12:28 /hadoop-2.5.2.tar.gz
drwxr-xr-x   - hadoop supergroup          0 2017-01-12 08:54 /hbase
17/01/13 14:04:43 DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@2301799f
17/01/13 14:04:43 DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@2301799f
17/01/13 14:04:43 DEBUG ipc.Client: removing client from cache: org.apache.hadoop.ipc.Client@2301799f
17/01/13 14:04:43 DEBUG ipc.Client: stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@2301799f
17/01/13 14:04:43 DEBUG ipc.Client: Stopping client
17/01/13 14:04:43 DEBUG ipc.Client: IPC Client (382257985) connection to hadoop2/192.168.1.233:9000 from hadoop: closed
17/01/13 14:04:43 DEBUG ipc.Client: IPC Client (382257985) connection to hadoop2/192.168.1.233:9000 from hadoop: stopped, remaining connections 1
17/01/13 14:04:43 DEBUG ipc.Client: IPC Client (382257985) connection to hadoop1/192.168.1.232:9000 from hadoop: closed
17/01/13 14:04:43 DEBUG ipc.Client: IPC Client (382257985) connection to hadoop1/192.168.1.232:9000 from hadoop: stopped, remaining connections 0

绿色的部分不用管，主要是因为我这个是集群中的standby的节点造成的重试现象；主要看红色部分，说来说去就是一个问题