实验三： HDFS Java API编程实践

实验题目

HDFS Java API编程实践

实验目的

熟悉HDFS操作常用的Java API。

实验平台

操作系统：Linux

Hadoop版本：2.6.0或以上版本

JDK版本：1.6或以上版本

Java IDE：Eclipse

实验内容和要求

编写一个Java程序，新建一个HDFS文件，并向其中写入你的名字；

        import org.apache.hadoop.conf.Configuration;  

        import org.apache.hadoop.fs.FileSystem;

        import org.apache.hadoop.fs.FSDataOutputStream;

        import org.apache.hadoop.fs.Path;

        public class Chapter3 {    

                public static void main(String[] args) {

                        try {

                                Configuration conf = new Configuration();  

                                conf.set("fs.defaultFS","hdfs://localhost:9000");

                                conf.set("fs.hdfs.impl","org.apache.hadoop.hdfs.DistributedFileSystem");

                                FileSystem fs = FileSystem.get(conf);

                                byte[] buff = "Hello world".getBytes(); // 要写入的内容

                                String filename = "LQL"; //要写入的文件名

                                FSDataOutputStream os = fs.create(new Path(filename));

                                os.write(buff,0,buff.length);

                                System.out.println("Create:"+ filename);

                                os.close();

                                fs.close();

                        } catch (Exception e) {  

                                e.printStackTrace();  

                        }  

                }  

        }

编写一个Java程序，判断HDFS上是否存在某个文件（上一步建立的文件是否存在）？

        import org.apache.hadoop.conf.Configuration;

        import org.apache.hadoop.fs.FileSystem;

        import org.apache.hadoop.fs.Path;

        public class Chapter3 {

                public static void main(String[] args) {

                            try {

                                    String filename = "test";

                                    Configuration conf = new Configuration();

                                    conf.set("fs.defaultFS","hdfs://localhost:9000");

                                    conf.set("fs.hdfs.impl","org.apache.hadoop.hdfs.DistributedFileSystem");

                                    FileSystem fs = FileSystem.get(conf);

                                    if(fs.exists(new Path(filename))){

                                            System.out.println("文件存在");

                                    }else{

                                            System.out.println("文件不存在");

                                    }

                                    fs.close();

                        } catch (Exception e) {

                                e.printStackTrace();

                        }

                }

        }

编写一个Java程序，打开一个HDFS中的文件（上一步建立的文件），并读取其中的数据，输出到标准输出；

        import java.io.BufferedReader;

        import java.io.InputStreamReader;

        import org.apache.hadoop.conf.Configuration;

        import org.apache.hadoop.fs.FileSystem;

        import org.apache.hadoop.fs.Path;

        import org.apache.hadoop.fs.FSDataInputStream;

        public class Chapter3 {

                public static void main(String[] args) {

                        try {

                                Configuration conf = new Configuration();

                                conf.set("fs.defaultFS","hdfs://localhost:9000");

                                conf.set("fs.hdfs.impl","org.apache.hadoop.hdfs.DistributedFileSystem");

                                FileSystem fs = FileSystem.get(conf);

                                Path file = new Path("test");

                                FSDataInputStream getIt = fs.open(file);

                                BufferedReader d = new BufferedReader(new InputStreamReader(getIt));

                                String content = d.readLine(); //读取文件一行

                                System.out.println(content);

                                d.close(); //关闭文件

                                fs.close(); //关闭hdfs

                        } catch (Exception e) {

                                e.printStackTrace();

                        }

                }

        }

遇到的问题和解决方法

【遇到的问题】：

Hadoop安全模式问题:

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.n0amenode.SafeModeException):Cannot create file /user/hadoop/LQL. Namenode is in safe mode.

尝试在Hadoop分布式文件系统(HDFS)上创建文件时遇到了问题，因为NameNode处于安全模式(Safe Mode)。

【解决方法】：

Hadoop的NameNode在启动时会进入安全模式，这是为了保护数据的完整性，防止在系统启动时数据不一致时进行写操作。在安全模式中，大多数写操作（如创建、删除文件或目录）将被阻止。根据日志，NameNode将在22秒后自动退出安全模式。

使用HDFS的命令行工具，通过执行hdfs dfsadmin -safemode leave命令来退出安全模式。