-
问题描述
(1) windows电脑向aliyun进行hdfs api文件追加操作时出现下方错误:DEBUG [Thread-3] (DFSOutputStream.java:1503) - Connecting to datanode *.*.*.*:9866 DEBUG [IPC Client (927327686) connection to centos8/*.*.*.*:9820 from root] (Client.java:1192) - IPC Client (927327686) connection to centos8/*.*.*.*:9820 from root: closed DEBUG [IPC Client (927327686) connection to centos8/*.*.*.*:9820 from root] (Client.java:987) - IPC Client (927327686) connection to centos8/*.*.*.*:9820 from root: stopped, remaining connections 0INFO [Thread-3] (DFSOutputStream.java:1343) - Exception in createBlockOutputStream java.net.ConnectException: Connection timed out: no further informationat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1508)at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1284)at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237)at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
(2) windows电脑向aliyun进行mapreduce api操作时出现下方错误:
WARN [LocalJobRunner Map Task Executor #0] (BlockReaderFactory.java:716) - I/O error constructing remote block reader.
-
问题原因
客户端创建文件先访问NameNode服务器进行创建文件Meta信息,以及文件树,此时访问NameNode的IP,NameNode服务器创建文件成功后,会返回对应dataNode的服务器节点,但此时的节点信息中的ip是与NameNode在同一网络下的内网IP,客户端是无法访问,也就无法写入。 -
问题解决
(1) 在hdfs-site.xml中添加配置<property><name>dfs.client.use.datanode.hostname</name><value>true</value></property>
(2) 或者在idea代码中添加
configuration.set("dfs.client.use.datanode.hostname", "true");