hadoop2.7 伪分布

hadoop 2.7.3伪分布式环境运行官方wordcount

hadoop 2.7.3伪分布式模式运行wordcount

基本环境：
系统：win7
虚机环境：virtualBox
虚机：centos 7
hadoop版本：2.7.3

本次以伪分布式模式来运行wordcount。

参考：

  • hadoop docs

​

1 hadoop环境

伪分布式就是将多个hadoop组件部署在一台机器上。因此涉及到各组件的配置，以及机器信任关系。

### 准备一个全新的环境
# cd /home/jungle/hadoop
# tar -zxvf hadoop-2.7.3.tar.gz
# mv hadoop-2.7.3 hadoop-daemon
# cd /home/jungle/hadoop/hadoop-daemon/

1.1 修改hadoop配置

  • core-site.xml
# vi etc/hadoop/core-site.xml
<configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value></property>
</configuration>
  • hdfs-site.xml
# vi etc/hadoop/hdfs-site.xml
<configuration><property><name>dfs.replication</name><value>1</value></property>
</configuration>

1.2 信任关系

# ssh-keygen -t rsa
# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys# ps
# ssh localhost
### 登陆本机# ps
### 确认两次ps显示的终端是不同的tty，则成功了

1.3 格式化hdfs

# hadoop fs -ls /
Found 20 items
-rw-r--r--   1 root root          0 2016-12-30 12:26 /1
dr-xr-xr-x   - root root      45056 2016-12-30 13:06 /bin
dr-xr-xr-x   - root root       4096 2016-12-29 20:09 /boot
drwxr-xr-x   - root root       3120 2017-01-06 18:31 /dev
drwxr-xr-x   - root root       8192 2017-01-06 18:32 /etc
# ... 是linux文件系统 # hdfs namenode -format
17/01/06 19:29:51 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = localhost/127.0.0.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.7.3
#...STARTUP_MSG:   java = 1.8.0_111
************************************************************/
17/01/06 19:29:51 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
17/01/06 19:29:51 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-ee109ab5-d5f1-4919-a1c6-5ff4de21a03f
17/01/06 19:29:52 INFO namenode.FSNamesystem: No KeyProvider found.
17/01/06 19:29:52 INFO namenode.FSNamesystem: fsLock is fair:true
17/01/06 19:29:52 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
17/01/06 19:29:52 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
17/01/06 19:29:52 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
17/01/06 19:29:52 INFO blockmanagement.BlockManager: The block deletion will start around 2017 Jan 06 19:29:52
17/01/06 19:29:52 INFO util.GSet: Computing capacity for map BlocksMap
17/01/06 19:29:52 INFO util.GSet: VM type       = 64-bit
17/01/06 19:29:52 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
17/01/06 19:29:52 INFO util.GSet: capacity      = 2^21 = 2097152 entries
17/01/06 19:29:52 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
17/01/06 19:29:52 INFO blockmanagement.BlockManager: defaultReplication         = 3
17/01/06 19:29:52 INFO blockmanagement.BlockManager: maxReplication             = 512
17/01/06 19:29:52 INFO blockmanagement.BlockManager: minReplication             = 1
17/01/06 19:29:52 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
17/01/06 19:29:52 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
17/01/06 19:29:52 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
17/01/06 19:29:52 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
17/01/06 19:29:52 INFO namenode.FSNamesystem: fsOwner             = jungle (auth:SIMPLE)
17/01/06 19:29:52 INFO namenode.FSNamesystem: supergroup          = supergroup
17/01/06 19:29:52 INFO namenode.FSNamesystem: isPermissionEnabled = true
17/01/06 19:29:52 INFO namenode.FSNamesystem: HA Enabled: false
17/01/06 19:29:52 INFO namenode.FSNamesystem: Append Enabled: true
17/01/06 19:29:52 INFO util.GSet: Computing capacity for map INodeMap
17/01/06 19:29:52 INFO util.GSet: VM type       = 64-bit
17/01/06 19:29:52 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
17/01/06 19:29:52 INFO util.GSet: capacity      = 2^20 = 1048576 entries
17/01/06 19:29:52 INFO namenode.FSDirectory: ACLs enabled? false
17/01/06 19:29:52 INFO namenode.FSDirectory: XAttrs enabled? true
17/01/06 19:29:52 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
17/01/06 19:29:52 INFO namenode.NameNode: Caching file names occuring more than 10 times
17/01/06 19:29:52 INFO util.GSet: Computing capacity for map cachedBlocks
17/01/06 19:29:52 INFO util.GSet: VM type       = 64-bit
17/01/06 19:29:52 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
17/01/06 19:29:52 INFO util.GSet: capacity      = 2^18 = 262144 entries
17/01/06 19:29:52 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
17/01/06 19:29:52 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
17/01/06 19:29:52 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
17/01/06 19:29:52 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
17/01/06 19:29:52 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
17/01/06 19:29:52 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
17/01/06 19:29:52 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
17/01/06 19:29:53 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
17/01/06 19:29:53 INFO util.GSet: Computing capacity for map NameNodeRetryCache
17/01/06 19:29:53 INFO util.GSet: VM type       = 64-bit
17/01/06 19:29:53 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
17/01/06 19:29:53 INFO util.GSet: capacity      = 2^15 = 32768 entries
17/01/06 19:29:53 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1788036100-127.0.0.1-1483702193052
17/01/06 19:29:53 INFO common.Storage: Storage directory /tmp/hadoop-jungle/dfs/name has been successfully formatted.
17/01/06 19:29:53 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-jungle/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
17/01/06 19:29:53 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-jungle/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 353 bytes saved in 0 seconds.
17/01/06 19:29:53 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/01/06 19:29:53 INFO util.ExitUtil: Exiting with status 0
17/01/06 19:29:53 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
************************************************************/

列出上面的日志，可以看到操作结果。其中最重要的应该就是基于linux文件系统存储的hdfs：

# ls -l /tmp/hadoop-jungle/dfs/name/current/
total 16
-rw-rw-r--. 1 jungle jungle 353 Jan  6 19:29 fsimage_0000000000000000000
-rw-rw-r--. 1 jungle jungle  62 Jan  6 19:29 fsimage_0000000000000000000.md5
-rw-rw-r--. 1 jungle jungle   2 Jan  6 19:29 seen_txid
-rw-rw-r--. 1 jungle jungle 201 Jan  6 19:29 VERSION

1.4 安装jps

如上篇中只安装了java。还需要安装jps等工具

# yum install java-1.8.0-openjdk-devel#jps
4497 Jps

2 启动hadoop

2.1 启动hdfs

# sbin/start-dfs.sh 
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/jungle/hadoop/hadoop-daemon/logs/hadoop-jungle-namenode-localhost.out
localhost: starting datanode, logging to /home/jungle/hadoop/hadoop-daemon/logs/hadoop-jungle-datanode-localhost.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is 6a:67:9f:8b:84:64:db:19:1a:ba:86:4f:f1:9a:1c:82.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /home/jungle/hadoop/hadoop-daemon/logs/hadoop-jungle-secondarynamenode-localhost.out# echo $?
0# ls -ltr logs/ 
total 96
-rw-rw-r--. 1 jungle jungle     0 Jan  6 20:17 SecurityAuth-jungle.audit
-rw-rw-r--. 1 jungle jungle   716 Jan  6 20:17 hadoop-jungle-namenode-localhost.out
-rw-rw-r--. 1 jungle jungle   716 Jan  6 20:17 hadoop-jungle-datanode-localhost.out
-rw-rw-r--. 1 jungle jungle 29280 Jan  6 20:17 hadoop-jungle-namenode-localhost.log
-rw-rw-r--. 1 jungle jungle 25370 Jan  6 20:17 hadoop-jungle-datanode-localhost.log
-rw-rw-r--. 1 jungle jungle   716 Jan  6 20:17 hadoop-jungle-secondarynamenode-localhost.out
-rw-rw-r--. 1 jungle jungle 22386 Jan  6 20:17 hadoop-jungle-secondarynamenode-localhost.log# jps
4977 SecondaryNameNode
4802 DataNode
4660 NameNode
5095 Jps

如上可以看到，已经启动了NameNode及SecondaryNameNode。以及DataNode。相应的，日志文件下也有对应的out和log文件。


# ls -l /tmp/hadoop-jungle/dfs/name/current/
total 3036
-rw-rw-r--. 1 jungle jungle      42 Jan  6 20:18 edits_0000000000000000001-0000000000000000002
-rw-rw-r--. 1 jungle jungle 1048576 Jan  6 20:18 edits_0000000000000000003-0000000000000000003
-rw-rw-r--. 1 jungle jungle 1048576 Jan  8 14:56 edits_inprogress_0000000000000000004
-rw-rw-r--. 1 jungle jungle     353 Jan  6 20:18 fsimage_0000000000000000002
-rw-rw-r--. 1 jungle jungle      62 Jan  6 20:18 fsimage_0000000000000000002.md5
-rw-rw-r--. 1 jungle jungle     353 Jan  8 14:56 fsimage_0000000000000000003
-rw-rw-r--. 1 jungle jungle      62 Jan  8 14:56 fsimage_0000000000000000003.md5
-rw-rw-r--. 1 jungle jungle       2 Jan  8 14:56 seen_txid
-rw-rw-r--. 1 jungle jungle     201 Jan  8 14:56 VERSION### pid 
# ls -l /tmp/hadoop-jungle-*
-rw-rw-r--. 1 jungle jungle 5 Jan  8 14:56 /tmp/hadoop-jungle-datanode.pid
-rw-rw-r--. 1 jungle jungle 5 Jan  8 14:56 /tmp/hadoop-jungle-namenode.pid
-rw-rw-r--. 1 jungle jungle 5 Jan  8 14:56 /tmp/hadoop-jungle-secondarynamenode.pid

2.2 检查页面

先关闭防火墙。

 
# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor prest: enabled)
   Active: inactive (dead) since Sun 2017-01-08 15:12:58 CST; 8s ago
     Docs: man:firewalld(1)
  Process: 681 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (cod=exited, status=0/SUCCESS)Main PID: 681 (code=exited, status=0/SUCCESS)# systemctl disable firewalld.service
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

2.3 访问NameNode

链接：

http://192.168.1.111:50070/

修改环境变量。因为之前在单机模式下有另一个测试目录。将其从hadoop-local改回hadoop-daemon

# vi ~/.bashrc
### export HADOOP_INSTALL=/home/jungle/hadoop/hadoop-local
export HADOOP_INSTALL=/home/jungle/hadoop/hadoop-daemon# source ~/.bashrc

操作hdfs

# hadoop fs -ls /
### 输出为空，根目录下没有任何内容# hdfs dfs -mkdir /user
# hadoop fs -ls /
Found 1 items
drwxr-xr-x   - jungle supergroup          0 2017-01-08 15:57 /user# hdfs dfs -mkdir /user/test
# hadoop fs -ls /user/
Found 1 items
drwxr-xr-x   - jungle supergroup          0 2017-01-08 15:57 /user/test# hadoop fs -put ../hadoop-local/dataLocal/input/ /user/test# hadoop fs -ls /user/test
Found 1 items
drwxr-xr-x   - jungle supergroup          0 2017-01-08 16:02 /user/test/input# hadoop fs -ls /user/test/input
Found 2 items
-rw-r--r--   1 jungle supergroup         37 2017-01-08 16:02 /user/test/input/file1.txt
-rw-r--r--   1 jungle supergroup         70 2017-01-08 16:02 /user/test/input/file2.txt

访问：

  • http://192.168.1.111:50070/explorer.html#
  • http://192.168.1.111:50070/explorer.html#/user/test/input

3 wordcount

# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /user/test/input/ /user/test/output# bin/hadoop fs -ls /user/test/output
Found 2 items
-rw-r--r--   1 jungle supergroup          0 2017-01-08 16:11 /user/test/output/_SUCCESS
-rw-r--r--   1 jungle supergroup         82 2017-01-08 16:11 /user/test/output/part-r-00000bin/hadoop fs -cat /user/test/output/part-r-00000
I   1
am  1
bye 2
great   1
hadoop. 3
hello   3
is  1
jungle. 2
software    1
the 1
world.  2

4 使用yarn

启动yarn

# jps
4803 DataNode
4979 SecondaryNameNode
4661 NameNode
6309 Jps# sbin/start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /home/jungle/hadoop/hadoop-daemon/logs/yarn-jungle-resourcemanager-localhost.localdomain.out
localhost: starting nodemanager, logging to /home/jungle/hadoop/hadoop-daemon/logs/yarn-jungle-nodemanager-localhost.localdomain.out# jps
4803 DataNode
4979 SecondaryNameNode
6355 ResourceManager
4661 NameNode
6477 NodeManager
6750 Jps# hadoop fs -ls /user/test/
Found 2 items
drwxr-xr-x   - jungle supergroup          0 2017-01-08 16:02 /user/test/input
drwxr-xr-x   - jungle supergroup          0 2017-01-08 16:11 /user/test/output# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /user/test/input/ /user/test/output2# hadoop fs -ls /user/test/
Found 3 items
drwxr-xr-x   - jungle supergroup          0 2017-01-08 16:02 /user/test/input
drwxr-xr-x   - jungle supergroup          0 2017-01-08 16:11 /user/test/output
drwxr-xr-x   - jungle supergroup          0 2017-01-08 16:25 /user/test/output2# hadoop fs -ls /user/test/output2
Found 2 items
-rw-r--r--   1 jungle supergroup          0 2017-01-08 16:25 /user/test/output2/_SUCCESS
-rw-r--r--   1 jungle supergroup         82 2017-01-08 16:25 /user/test/output2/part-r-00000# hadoop fs -cat /user/test/output2/part-r-00000
I   1
am  1
bye 2
great   1
hadoop. 3
hello   3
is  1
jungle. 2
software    1
the 1
world.  2

执行日志：

17/01/08 16:25:32 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
17/01/08 16:25:32 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
17/01/08 16:25:32 INFO input.FileInputFormat: Total input paths to process : 2
17/01/08 16:25:33 INFO mapreduce.JobSubmitter: number of splits:2
17/01/08 16:25:33 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local247232145_0001
17/01/08 16:25:33 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
17/01/08 16:25:33 INFO mapreduce.Job: Running job: job_local247232145_0001
17/01/08 16:25:33 INFO mapred.LocalJobRunner: OutputCommitter set in config null
17/01/08 16:25:33 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/01/08 16:25:33 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
17/01/08 16:25:33 INFO mapred.LocalJobRunner: Waiting for map tasks
17/01/08 16:25:33 INFO mapred.LocalJobRunner: Starting task: attempt_local247232145_0001_m_000000_0
17/01/08 16:25:33 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/01/08 16:25:33 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
17/01/08 16:25:33 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/test/input/file2.txt:0+70
17/01/08 16:25:35 INFO mapreduce.Job: Job job_local247232145_0001 running in uber mode : false
17/01/08 16:25:35 INFO mapreduce.Job:  map 0% reduce 0%
17/01/08 16:25:35 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/01/08 16:25:35 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/01/08 16:25:35 INFO mapred.MapTask: soft limit at 83886080
17/01/08 16:25:35 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/01/08 16:25:35 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/01/08 16:25:37 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/01/08 16:25:37 INFO mapred.LocalJobRunner: 
17/01/08 16:25:37 INFO mapred.MapTask: Starting flush of map output
17/01/08 16:25:37 INFO mapred.MapTask: Spilling map output
17/01/08 16:25:37 INFO mapred.MapTask: bufstart = 0; bufend = 114; bufvoid = 104857600
17/01/08 16:25:37 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
17/01/08 16:25:38 INFO mapred.MapTask: Finished spill 0
17/01/08 16:25:38 INFO mapred.Task: Task:attempt_local247232145_0001_m_000000_0 is done. And is in the process of committing
17/01/08 16:25:38 INFO mapred.LocalJobRunner: map
17/01/08 16:25:38 INFO mapred.Task: Task 'attempt_local247232145_0001_m_000000_0' done.
17/01/08 16:25:38 INFO mapred.LocalJobRunner: Finishing task: attempt_local247232145_0001_m_000000_0
17/01/08 16:25:38 INFO mapred.LocalJobRunner: Starting task: attempt_local247232145_0001_m_000001_0
17/01/08 16:25:38 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/01/08 16:25:38 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
17/01/08 16:25:38 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/test/input/file1.txt:0+37
17/01/08 16:25:38 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/01/08 16:25:38 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/01/08 16:25:38 INFO mapred.MapTask: soft limit at 83886080
17/01/08 16:25:38 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/01/08 16:25:38 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/01/08 16:25:38 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/01/08 16:25:38 INFO mapred.LocalJobRunner: 
17/01/08 16:25:38 INFO mapred.MapTask: Starting flush of map output
17/01/08 16:25:38 INFO mapred.MapTask: Spilling map output
17/01/08 16:25:38 INFO mapred.MapTask: bufstart = 0; bufend = 65; bufvoid = 104857600
17/01/08 16:25:38 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214372(104857488); length = 25/6553600
17/01/08 16:25:38 INFO mapred.MapTask: Finished spill 0
17/01/08 16:25:38 INFO mapred.Task: Task:attempt_local247232145_0001_m_000001_0 is done. And is in the process of committing
17/01/08 16:25:38 INFO mapred.LocalJobRunner: map
17/01/08 16:25:38 INFO mapred.Task: Task 'attempt_local247232145_0001_m_000001_0' done.
17/01/08 16:25:38 INFO mapred.LocalJobRunner: Finishing task: attempt_local247232145_0001_m_000001_0
17/01/08 16:25:38 INFO mapred.LocalJobRunner: map task executor complete.
17/01/08 16:25:38 INFO mapreduce.Job:  map 100% reduce 0%
17/01/08 16:25:39 INFO mapred.LocalJobRunner: Waiting for reduce tasks
17/01/08 16:25:39 INFO mapred.LocalJobRunner: Starting task: attempt_local247232145_0001_r_000000_0
17/01/08 16:25:39 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/01/08 16:25:39 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
17/01/08 16:25:39 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6bef8a0a
17/01/08 16:25:39 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10
17/01/08 16:25:39 INFO reduce.EventFetcher: attempt_local247232145_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
17/01/08 16:25:39 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local247232145_0001_m_000000_0 decomp: 98 len: 102 to MEMORY
17/01/08 16:25:39 INFO reduce.InMemoryMapOutput: Read 98 bytes from map-output for attempt_local247232145_0001_m_000000_0
17/01/08 16:25:39 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 98, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->98
17/01/08 16:25:39 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local247232145_0001_m_000001_0 decomp: 68 len: 72 to MEMORY
17/01/08 16:25:39 INFO reduce.InMemoryMapOutput: Read 68 bytes from map-output for attempt_local247232145_0001_m_000001_0
17/01/08 16:25:39 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 68, inMemoryMapOutputs.size() -> 2, commitMemory -> 98, usedMemory ->166
17/01/08 16:25:39 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
17/01/08 16:25:39 INFO mapred.LocalJobRunner: 2 / 2 copied.
17/01/08 16:25:39 INFO reduce.MergeManagerImpl: finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
17/01/08 16:25:40 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptorat org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
17/01/08 16:25:40 INFO mapred.Merger: Merging 2 sorted segments
17/01/08 16:25:40 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 156 bytes
17/01/08 16:25:40 INFO reduce.MergeManagerImpl: Merged 2 segments, 166 bytes to disk to satisfy reduce memory limit
17/01/08 16:25:40 INFO reduce.MergeManagerImpl: Merging 1 files, 168 bytes from disk
17/01/08 16:25:40 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
17/01/08 16:25:40 INFO mapred.Merger: Merging 1 sorted segments
17/01/08 16:25:40 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 160 bytes
17/01/08 16:25:40 INFO mapred.LocalJobRunner: 2 / 2 copied.
17/01/08 16:25:40 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
17/01/08 16:25:40 INFO mapred.Task: Task:attempt_local247232145_0001_r_000000_0 is done. And is in the process of committing
17/01/08 16:25:40 INFO mapred.LocalJobRunner: 2 / 2 copied.
17/01/08 16:25:40 INFO mapred.Task: Task attempt_local247232145_0001_r_000000_0 is allowed to commit now
17/01/08 16:25:40 INFO output.FileOutputCommitter: Saved output of task 'attempt_local247232145_0001_r_000000_0' to hdfs://localhost:9000/user/test/output2/_temporary/0/task_local247232145_0001_r_000000
17/01/08 16:25:40 INFO mapred.LocalJobRunner: reduce > reduce
17/01/08 16:25:40 INFO mapred.Task: Task 'attempt_local247232145_0001_r_000000_0' done.
17/01/08 16:25:40 INFO mapred.LocalJobRunner: Finishing task: attempt_local247232145_0001_r_000000_0
17/01/08 16:25:40 INFO mapred.LocalJobRunner: reduce task executor complete.
17/01/08 16:25:40 INFO mapreduce.Job:  map 100% reduce 100%
17/01/08 16:25:40 INFO mapreduce.Job: Job job_local247232145_0001 completed successfully
17/01/08 16:25:41 INFO mapreduce.Job: Counters: 35File System CountersFILE: Number of bytes read=889201FILE: Number of bytes written=1745401FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=284HDFS: Number of bytes written=82HDFS: Number of read operations=22HDFS: Number of large read operations=0HDFS: Number of write operations=5Map-Reduce FrameworkMap input records=3Map output records=18Map output bytes=179Map output materialized bytes=174Input split bytes=224Combine input records=18Combine output records=14Reduce input groups=11Reduce shuffle bytes=174Reduce input records=14Reduce output records=11Spilled Records=28Shuffled Maps =2Failed Shuffles=0Merged Map outputs=2GC time elapsed (ms)=117Total committed heap usage (bytes)=457912320Shuffle ErrorsBAD_ID=0CONNECTION=0IO_ERROR=0WRONG_LENGTH=0WRONG_MAP=0WRONG_REDUCE=0File Input Format Counters Bytes Read=107File Output Format Counters Bytes Written=82

 

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.mzph.cn/news/387768.shtml

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈email:809451989@qq.com，一经查实，立即删除！

相关文章

iPhone手机屏幕尺寸（分辨率）

iPhone手机屏幕尺寸（分辨率）

第一代iPhone2G屏幕为3.5英吋&#xff0c;分辨率为320*480像素&#xff0c;比例为3:2。 第二代iPhone3G屏幕为3.5英吋&#xff0c;分辨率为320*480像素&#xff0c;比例为3:2。 第三代iPhone3GS屏幕为3.5英吋&#xff0c;分辨率为320*480像素&#xff0c;比例为3:2。 第四代iPh…
阅读更多...
[Java in NetBeans] Lesson 06. Custom classes

[Java in NetBeans] Lesson 06. Custom classes

这个课程的参考视频和图片来自youtube。 主要学到的知识点有&#xff1a; Constructors: A special method called when an object of the class is createdproperty pattern and encapsulation(封装): hide the implementation details from the user, so when the class is b…
阅读更多...
UDP打洞NAT大致分为下面四类 P2P

UDP打洞NAT大致分为下面四类 P2P

NAT大致分为下面四类 1) Full Cone 这种NAT内部的机器A连接过外网机器C后,NAT会打开一个端口.然后外网的任何发到这个打开的端口的UDP数据报都可以到达A.不管是不是C发过来的. 例如 A:192.168.8.100 NAT:202.100.100.100 C:292.88.88.88 A(192.168.8.100:5000) -> NAT(202.1…
阅读更多...
让内核突破512字节的限制

让内核突破512字节的限制

转载于:https://www.cnblogs.com/ZHONGZHENHUA/p/10124237.html
阅读更多...
高频算法面试题（字符串） 242. 有效的字母异位词

高频算法面试题（字符串） 242. 有效的字母异位词

leetcode 242. 有效的字母异位词 给定两个字符串 s 和 t &#xff0c;编写一个函数来判断 t 是否是 s 的一个字母异位词。示例 1: 输入: s "anagram", t "nagaram" 输出: true 复制代码示例 2: 输入: s "rat", t "car" 输出: fals…
阅读更多...
struts2的漏洞

struts2的漏洞

文章前半部分来自团队小伙伴阿德马的总结&#xff0c;后半部分的Poc和Exp是小编匆忙之际借鉴而来&#xff0c;感谢写Poc和Exp的伙伴~ 安恒给官方上报的&#xff0c;然后官方选择了1个对国内来说比较敏感的时期发了公告出来&#xff0c;好蛋疼。 该漏洞的CVE编号是CVE-2017-56…
阅读更多...
Java Statement PK PrepareStatement

Java Statement PK PrepareStatement

PreparedStatement是用来执行SQL查询语句的API之一&#xff0c;Java提供了 Statement、PreparedStatement 和 CallableStatement三种方式来执行查询语句&#xff0c;其中 Statement 用于通用查询&#xff0c; PreparedStatement 用于执行参数化查询&#xff0c;而 CallableStat…
阅读更多...
mysql在linux 下安装

mysql在linux 下安装

安装环境&#xff1a;系统是 centos6.5 1、下载 下载地址&#xff1a;http://dev.mysql.com/downloads/mysql/5.6.html#downloads 下载版本&#xff1a;我这里选择的5.6.33&#xff0c;通用版&#xff0c;linux下64位 也可以直接复制64位的下载地址&#xff0c;通过命令下载&a…
阅读更多...
Leetcode PHP题解--D47 868. Binary Gap

Leetcode PHP题解--D47 868. Binary Gap

2019独角兽企业重金招聘Python工程师标准>>> D47 868. Binary Gap 题目链接 868. Binary Gap 题目分析 给定一个数字&#xff0c;计算其二进制表示中&#xff0c;出现的两个1最大距离。 思路 当然是先转换成二进制了。再进行遍历。 当只有一个1时&#xff0c;返回0。…
阅读更多...
[洛谷P5048][Ynoi2019模拟赛]Yuno loves sqrt technology III

[洛谷P5048][Ynoi2019模拟赛]Yuno loves sqrt technology III

题目大意&#xff1a;有$n(n\leqslant5\times10^5)$个数&#xff0c;$m(m\leqslant5\times10^5)$个询问&#xff0c;每个询问问区间$[l,r]$中众数的出现次数 题解&#xff1a;分块&#xff0c;设块大小为$S$&#xff0c;先可以预处理出两两块之间的众数出现次数&#xff0c;复杂…
阅读更多...
C#接口实现多态

C#接口实现多态

我比较喜欢对感兴趣的理论进行反复的理解甚至理解背诵下来&#xff0c;接下来再复习一下什么叫多态&#xff08;哈哈哈&#xff09; 多态&#xff1a;在同一粒度视图下对相同类型的事物不做区别的统一处理 接下来看一下接口和引擎类是如何实现多态的&#xff1a; 一、 1、创建了…
阅读更多...
docker 网络 不好用 docker: Error response from daemon: failed to create endpoint jovial_wing on network b

docker 网络 不好用 docker: Error response from daemon: failed to create endpoint jovial_wing on network b

启动容器时&#xff0c;有可能会遇到如下问题&#xff0c;比如启动redis容器&#xff1a; sudo docker run -d -p 6379:6379 --name redis redis:latest Linux代码docker: Error response from daemon: failed to create endpoint redis on network bridge: iptables failed: …
阅读更多...
hadoop-hdfs-存储模型-架构模型-角色介绍

hadoop-hdfs-存储模型-架构模型-角色介绍

转载于:https://www.cnblogs.com/LXL616/p/10803978.html
阅读更多...
docker 镜像 导入导出

docker 镜像 导入导出

很喜欢玩docker&#xff0c;但最新遇到一个问题&#xff0c;公司给的新机器的dns有问题&#xff0c;导致pull不下来镜像。 没办法了&#xff0c;没有镜像什么神马都干不了&#xff0c;又不能花很多时间去搭建私有的镜像库&#xff0c;只有另寻办法了。 废话少说&#xff0c;经…
阅读更多...
使用Nginx+uWSGI部署Django项目

使用Nginx+uWSGI部署Django项目

1.linux安装python3环境 参考链接&#xff1a;https://www.cnblogs.com/zzqit/p/10087680.html 2.安装uwsgi pip3 install uwsgiln -s /usr/local/python3/bin/uwsgi /usr/local/bin/uwsgi #建立软链接uwsgi --version #检查安装成功 3.基于uwsgidjango项目部署 django项目目…
阅读更多...
Nagios使用check_mysql_health插件监控Mysql主机

Nagios使用check_mysql_health插件监控Mysql主机

基本信息 Nagios&#xff1a;Nagios core 4.4.3Nagios Plugins&#xff1a;check_mysql_health 2.2.2Mysql-server: 192.168.0.91db user&#xff1a;db操作流程&#xff1a;下载插件->安装插件->配置command->添加主机->添加服务安装插件 下载 wget https://labs.…
阅读更多...
lsof使用

lsof使用

简介 lsof(list open files)是一个列出当前系统打开文件的工具。在linux环境下&#xff0c;任何事物都以文件的形式存在&#xff0c;通过文件不仅仅可以访问常规数据&#xff0c;还可以访问网络连接和硬件。所以如传输控制协议 (TCP) 和用户数据报协议 (UDP) 套接字等&#xf…
阅读更多...
解题：2017清华集训 无限之环

解题：2017清华集训 无限之环

题面 费用流 把每种水管再拆出来四个方向的接头&#xff0c;然后根据水管的形状连出旋转时的代价。最后黑白染色成二分图&#xff0c;然后白点对应的接头向黑点对应的接头连边&#xff0c;源点向白点自己连边&#xff0c;黑点自己向汇点连边。 怎么连边&#xff1f;我是大力讨论…
阅读更多...
Node.js学习之（第二章：exports和module.exports）

Node.js学习之（第二章：exports和module.exports）

前言 Node中&#xff0c;每个模块都有一个exports接口对象&#xff0c;我们需要把公共的方法或者字符串挂载在这个接口对象中&#xff0c;其他的模块才可以使用。 Node.js中只有模块作用域&#xff0c;默认两个模块之间的变量&#xff0c;方法互不冲突&#xff0c;互不影响&…
阅读更多...
docker命令及挂载

docker命令及挂载

常用命令所有镜像:docker images当前执行:docker ps提交保存docker容器: docker commit进入到对应服务:docker attach <container id>已经执行带容器:docker ps -l根据名称启动通过8081端口察看docker容器里的8080:docker run -i -t -d -p 8081:8080 -p23:22 ubuntu:ubun…
阅读更多...
最新文章

Copyright @ 2022~2023