hadoop使用内置包进行性能测试
hadoop使用内置包进行性能测试
- hadoop使用内置包进行性能测试
- TestDFSIO read & write
- NNBench
- MRBench
- SliveTest
TestDFSIO read & write
TestDFSIO 是一个 Hadoop 自带的基准测试工具，用于测试 HDFS 的读写性能。它会模拟大量的文件读写操作，并输出相应的性能指标。
测试内容：
- TestDFSIO 使用了两个阶段进行测试 — 写入（Write）和读取（Read）。在写入阶段，它将生成指定数量和大小的文件，并将其写入到 HDFS 中。在读取阶段，它将从 HDFS 读取这些文件并计算读取速度。
测试参数：
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
private static final Logger LOG = LoggerFactory.getLogger(TestDFSIO.class);private static final int DEFAULT_BUFFER_SIZE = 1000000;private static final String BASE_FILE_NAME = "test_io_";private static final String DEFAULT_RES_FILE_NAME = "TestDFSIO_results.log";private static final long MEGA = ByteMultiple.MB.value();private static final int DEFAULT_NR_BYTES = 128;private static final int DEFAULT_NR_FILES = 4;private static final String USAGE ="Usage: " + TestDFSIO.class.getSimpleName() +" [genericOptions]" +" -read [-random | -backward | -skip [-skipSize Size]] |" +" -write | -append | -truncate | -clean" +" [-compression codecClassName]" +" [-nrFiles N]" +" [-size Size[B|KB|MB|GB|TB]]" +" [-resFile resultFileName] [-bufferSize Bytes]" +" [-storagePolicy storagePolicyName]" +" [-erasureCodePolicy erasureCodePolicyName]";
-write：指定写入阶段的测试。-read：指定读取阶段的测试。-nrFiles <number>：要创建的文件数量。-size <size>：每个文件的大。-resFile <filename>：指定结果输出文件的路径。-bufferSize <bufferSize>：此参数用于指定读写文件时使用的缓冲区大小。compression <codecClassName>：指定在写入文件时使用的压缩编解码器。storagePolicy <storagePolicyName>：用于在写入文件时指定存储策略。存储策略决定了数据在 HDFS 中的复制和分布方式。erasureCodePolicy <erasureCodePolicyName>：用于在写入文件时指定纠删码（Erasure Code）策略。
执行语句示例：
创建 10 个大小为 100MB 的文件，并将结果输出到当前路径下的TestDFSIO_results.log文件中。
hadoop jar hadoop-mapreduce-client-jobclient-3.3.6-tests.jar TestDFSIO -write -nrFiles 10 -size 100MB
读取10 个大小为 100MB 的文件，并将结果输出到当前路径下的TestDFSIO_results.log文件中。
hadoop jar hadoop-mapreduce-client-test.jar TestDFSIO -read -nrFiles 10 -fileSize 100
TestDFSIO_results.log保存内容：
----- TestDFSIO ----- : writeDate & time: Thu Dec 14 13:21:57 CST 2023Number of files: 10Total MBytes processed: 1000Throughput mb/sec: 49.72Average IO rate mb/sec: 190.11IO rate std deviation: 75.6Test exec time sec: 27.01----- TestDFSIO ----- : readDate & time: Thu Dec 14 13:36:56 CST 2023Number of files: 10Total MBytes processed: 1000Throughput mb/sec: 631.31Average IO rate mb/sec: 655.3IO rate std deviation: 127.5Test exec time sec: 8.03
NNBench
Hadoop NNbench 是一个用于测试 HDFS （Hadoop 分布式文件系统）性能的工具。它可以模拟多个客户端同时进行读写操作，以评估 HDFS 的并发访问能力和吞吐量。
测试内容：
- 数据生成：NNbench 会生成一系列指定大小的数据块（Blocks），这些数据块将被写入到 HDFS 中。数据块的大小由
-blockSize参数指定。 - 写入测试：多个客户端同时向 HDFS 写入生成的数据块，可以设置写入的线程数和每个线程的写入次数。写入操作的参数由
-operation create_write指定。 - 读取测试：多个客户端同时从 HDFS 读取之前写入的数据块。读取操作的参数由
-operation open_read指定。
测试参数：
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
Options:-operation <Available operations are create_write open_read rename delete. This option is mandatory>* NOTE: The open_read, rename and delete operations assume that the files they operate on, are already available. The create_write operation must be run before running the other operations.-maps <number of maps. default is 1. This is not mandatory>-reduces <number of reduces. default is 1. This is not mandatory>-startTime <time to start, given in seconds from the epoch. Make sure this is far enough into the future, so all maps (operations) will start at the same time. default is launch time + 2 mins. This is not mandatory>-blockSize <Block size in bytes. default is 1. This is not mandatory>-bytesToWrite <Bytes to write. default is 0. This is not mandatory>-bytesPerChecksum <Bytes per checksum for the files. default is 1. This is not mandatory>-numberOfFiles <number of files to create. default is 1. This is not mandatory>-replicationFactorPerFile <Replication factor for the files. default is 1. This is not mandatory>-baseDir <base DFS path. default is /benchmarks/NNBench. Supports cross-cluster access by using full path with schema and cluster. This is not mandatory>-readFileAfterOpen <true or false. if true, it reads the file and reports the average time to read. This is valid with the open_read operation. default is false. This is not mandatory>-help: Display the help statement
-operation <write|read>：指定进行写入或读取操作。-maps <numMaps>：指定 MapReduce 作业中使用的 Mapper 数量。-reduces <numReduces>：指定 MapReduce 作业中使用的 Reducer 数量。-blockSize <blockSize>：指定生成的数据块的大小。-bytesToWrite <bytesToWrite>：指定总共写入的数据大小。-bytesPerChecksum <bytesPerChecksum>：指定每个校验和的大小。-numberOfFiles <numberOfFiles>：指定创建/读取的文件数。-replicationFactorPerFile <replicationFactor>：指定写入数据块的副本数量。baseDir <baseDir>：基础DFS路径，默认值为/backages/NNBench。readFileAfterOpen <true|false>：如果为true，则读取文件并报告平均读取时间。只对open_read操作有效，默认为false。...
执行语句示例：
写入操作（-opertation create_write），采用 12 个 Mapper 和 6 个 Reducer 来处理数据，每个数据块大小为 64MB（-blockSize 67108864），总共写入 128MB 的数据（-bytesToWrite 134217728），写入文件数为1（-numberOfFiles 1），并且将数据块副本数设置为 2（-replicationFactor 2），基础DFS路径为(/benchmarks/NNBench-’hostname -s’)，测试结果输出到当前路径下的NNBench_results.log文件中。
hadoop jar hadoop-mapreduce-client-jobclient-3.3.6-tests.jar nnbench -operation create_write -maps 12 -reduces 6 -blockSize 67108864 -bytesToWrite 134217728 -numberOfFiles 1 -replicationFactorPerFile 2 -basedir /benchmarks/NNBench-’hostname -s’
读取操作（-opertation open_read），采用 12 个 Mapper 和 6 个 Reducer 来处理数据，每个数据块大小为 64MB（-blockSize 67108864），总共写入 128MB 的数据（-bytesToWrite 134217728），写入文件数为1（-numberOfFiles 1），并且将数据块副本数设置为 2（-replicationFactor 2），报告平均读取时间（-readFileAfterOpen true）基础DFS路径为(/benchmarks/NNBench-’hostname -s’)，测试结果输出到当前路径下的NNBench_results.log文件中。
hadoop jar hadoop-mapreduce-client-jobclient-3.3.6-tests.jar nnbench -operation open_read -maps 12 -reduces 6 -blockSize 67108864 -bytesToWrite 134217728 -numberOfFiles 1 -replicationFactorPerFile 2 -readFileAfterOpen true -basedir /benchmarks/NNBench-’hostname -s’
NNBench_results.log
-------------- NNBench -------------- : Version: NameNode Benchmark 0.4Date & time: 2023-12-14 14:11:35,409Test Operation: create_writeStart time: 2023-12-14 14:11:20,732Maps to run: 12Reduces to run: 6Block Size (bytes): 67108864Bytes to write: 134217728Bytes per checksum: 1Number of files: 1Replication factor: 2Successful file operations: 12# maps that missed the barrier: 0# exceptions: 0TPS: Create/Write/Close: 2
Avg exec time (ms): Create/Write/Close: 882.25Avg Lat (ms): Create/Write: 670.9166666666666Avg Lat (ms): Close: 109.58333333333333RAW DATA: AL Total #1: 8051RAW DATA: AL Total #2: 1315RAW DATA: TPS Total (ms): 10587RAW DATA: Longest Map Time (ms): 11954.0RAW DATA: Late maps: 0RAW DATA: # of exceptions: 0-------------- NNBench -------------- : Version: NameNode Benchmark 0.4Date & time: 2023-12-14 14:17:34,927Test Operation: open_readStart time: 2023-12-14 14:17:26,840Maps to run: 12Reduces to run: 6Block Size (bytes): 67108864Bytes to write: 134217728Bytes per checksum: 1Number of files: 1Replication factor: 2Successful file operations: 12# maps that missed the barrier: 0# exceptions: 0TPS: Open/Read: 2Avg Exec time (ms): Open/Read: 320.75Avg Lat (ms): Open: 3.0Avg Lat (ms): Read: 175.75RAW DATA: AL Total #1: 36RAW DATA: AL Total #2: 2109RAW DATA: TPS Total (ms): 3849RAW DATA: Longest Map Time (ms): 4837.0RAW DATA: Late maps: 0RAW DATA: # of exceptions: 0
MRBench
Hadoop Mrbench 是一个用于测试 Hadoop MapReduce 的性能的工具。它可以模拟 MapReduce 作业的执行过程，以评估 Hadoop 集群的并发处理能力和吞吐量。
测试内容：
- 数据生成：Mrbench 会生成指定数量的输入数据块（blocks），每个数据块的大小由
-blockSize参数指定。 - MapReduce 作业执行：通过调度 MapReduce 作业来模拟实际的计算任务。作业的类型、数量和规模由
-numRuns、-maps和-reduces参数指定。 - 数据读取：测试过程中，将对生成的数据块进行读取操作，以验证数据的正确性。
测试参数：
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MRBench.java
Usage: mrbench [-baseDir <base DFS path for output/input, default is /benchmarks/MRBench>] [-jar <local path to job jar file containing Mapper and Reducer implementations, default is current jar file>] [-numRuns <number of times to run the job, default is 1>] [-maps <number of maps for each run, default is 2>] [-reduces <number of reduces for each run, default is 1>] [-inputLines <number of input lines to generate, default is 1>] [-inputType <type of input to generate, one of ascending (default), descending, random>] [-verbose]
-maps <numMaps>：指定 MapReduce 作业中使用的 Mapper 数量。-reduces <numReduces>：指定 MapReduce 作业中使用的 Reducer 数量。-numRuns <numRuns>：指定要执行的 MapReduce 作业的次数。-inputLines：指定要输入的数据行数...
模拟运行10次 MapReduce 作业（-numRuns 10），使用 12 个 Mapper 和 6 个 Reducer 来处理数据（-maps 12 -reduces 6），每个文件输入数据行数为50。
执行语句示例：
hadoop jar hadoop-mapreduce-client-jobclient-3.3.6-tests.jar mrbench -maps 12 -reduces 6 -numRuns 10 -inputLines 50
DataLines Maps Reduces AvgTime (milliseconds)
50 12 6 2683
SliveTest
SliveTest 测试的主要目的是评估 Hadoop 文件系统（HDFS）和 MapReduce 框架在高负载、并发访问以及数据完整性方面的表现。
由于 SliveTest 类是作为 Hadoop MapReduce 项目中的一个测试类存在的，它通常不作为独立工具来使用，而是通过运行整个测试套件进行测试。
测试参数：
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/SliveTest.java
usage: SliveTest 0.1.0-append <arg> pct,distribution where distribution is one ofbeg,end,uniform,mid-appendSize <arg> Min,max for size to append(min=max=MAX_LONG=blocksize)-baseDir <arg> Base directory path-blockSize <arg> Min,max for dfs file block size-cleanup <arg> Cleanup & remove directory after reporting-create <arg> pct,distribution where distribution is one ofbeg,end,uniform,mid-delete <arg> pct,distribution where distribution is one ofbeg,end,uniform,mid-dirSize <arg> Max files per directory-duration <arg> Duration of a map task in seconds (MAX_INT for nolimit)-exitOnError Exit on first error-files <arg> Max total number of files-help Usage information-ls <arg> pct,distribution where distribution is one ofbeg,end,uniform,mid-maps <arg> Number of maps-mkdir <arg> pct,distribution where distribution is one ofbeg,end,uniform,mid-ops <arg> Max number of operations per map-packetSize <arg> Dfs write packet size-queue <arg> Queue name-read <arg> pct,distribution where distribution is one ofbeg,end,uniform,mid-readSize <arg> Min,max for size to read (min=max=MAX_LONG=readentire file)-reduces <arg> Number of reduces-rename <arg> pct,distribution where distribution is one ofbeg,end,uniform,mid-replication <arg> Min,max value for replication amount-resFile <arg> Result file name-seed <arg> Random number seed-sleep <arg> Min,max for millisecond of random sleep to perform(between operations)-truncate <arg> pct,distribution where distribution is one ofbeg,end,uniform,mid-truncateSize <arg> Min,max for size to truncate(min=max=MAX_LONG=blocksize)-truncateWait <arg> Should wait for truncate recovery-writeSize <arg> Min,max for size to write(min=max=MAX_LONG=blocksize)
-append <arg>：追加操作的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。-appendSize <arg>：追加操作的最小和最大文件大小。如果未指定，则默认为MAX_LONG=blocksize。-baseDir <arg>：基础目录路径，即执行操作的根目录。-blockSize <arg>：HDFS 文件块大小的最小和最大值。-cleanup <arg>：在报告后进行清理并删除目录。<arg>可以是true或false。-create <arg>：创建文件的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。-delete <arg>：删除操作的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。-dirSize <arg>：每个目录的最大文件数。-duration <arg>：每个 Map 任务的持续时间（秒）。设置为MAX_INT表示没有时间限制。-exitOnError：在出现错误时立即退出。-files <arg>：总文件数的最大值。-help：显示使用信息。-ls <arg>：列出操作的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。-maps <arg>：Map 任务的数量。-mkdir <arg>：创建目录的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。-ops <arg>：每个 Map 任务的最大操作数。-packetSize <arg>：DFS 写入数据包的大小。-queue <arg>：指定作业提交到的队列名称。-read <arg>：读取操作的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。-readSize <arg>：读取操作的最小和最大文件大小。如果未指定，则默认为MAX_LONG=read entire file。-reduces <arg>：Reduce 任务的数量。-rename <arg>：重命名操作的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。-replication <arg>：文件的最小和最大副本数量。-resFile <arg>：结果文件名。-seed <arg>：随机数种子。-sleep <arg>：执行操作之间的随机休眠时间，单位为毫秒。指定一个最小值和最大值。-truncate <arg>：截断操作的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。-truncateSize <arg>：截断操作的最小和最大文件大小。如果未指定，则默认为MIN_LONG=blocksize。-truncateWait <arg>：是否等待截断恢复完成。<arg>可以是true或false。-writeSize <arg>: …
hadoop org.apache.hadoop.fs.slive.SliveTest -maps 12 -reduces 6 -files 5 -ops 5 -truncateWait true -cleanup true
执行完毕后如果没有设置 -cleanup true 则需要手动清理测试痕迹：
hdfs dfs -rm -r -f hdfs://hadoop-system-namenode:9000/test/slive/slive/output
2023-12-14 16:51:05,863 INFO slive.SliveTest: Reporting on job:
2023-12-14 16:51:05,863 INFO slive.SliveTest: Writing report using contents of /test/slive/slive/output
2023-12-14 16:51:05,925 INFO slive.SliveTest: Report results being placed to logging output and to file /opt/hadoop/share/hadoop/mapreduce/part-0000
2023-12-14 16:51:05,926 INFO slive.ReportWriter: Basic report for operation type AppendOp
2023-12-14 16:51:05,926 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,926 INFO slive.ReportWriter: Measurement "files_not_found" = 1
2023-12-14 16:51:05,926 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Basic report for operation type CreateOp
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "bytes_written" = 67108864
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "milliseconds_taken" = 3018
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "successes" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Rate for measurement "bytes_written" = 21.206 MB/sec
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Rate for measurement "op_count" = 0.331 operations/sec
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Rate for measurement "successes" = 0.331 successes/sec
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Basic report for operation type DeleteOp
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "failures" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Basic report for operation type ListOp
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "files_not_found" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Basic report for operation type MkdirOp
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "milliseconds_taken" = 35
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "successes" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Rate for measurement "op_count" = 28.571 operations/sec
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Rate for measurement "successes" = 28.571 successes/sec
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Basic report for operation type ReadOp
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "files_not_found" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Basic report for operation type RenameOp
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "failures" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Basic report for operation type SliveMapper
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "milliseconds_taken" = 4486
2023-12-14 16:51:05,928 INFO slive.ReportWriter: Measurement "op_count" = 8
2023-12-14 16:51:05,928 INFO slive.ReportWriter: Rate for measurement "op_count" = 1.783 operations/sec
2023-12-14 16:51:05,928 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,928 INFO slive.ReportWriter: Basic report for operation type TruncateOp
2023-12-14 16:51:05,928 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,928 INFO slive.ReportWriter: Measurement "files_not_found" = 1
2023-12-14 16:51:05,928 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,928 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,928 INFO slive.SliveTest: Cleaning up job:
2023-12-14 16:51:05,928 INFO slive.SliveTest: Attempting to recursively delete /test/slive/slive
)
覆盖优化 - 附代码)
