Hadoop3：HDFS的Fsimage和Edits文件介绍

一、概念

在这里插入图片描述
Fsimage文件：HDFS文件系统元数据的一个永久性的检查点，其中包含HDFS文件系统的所有目
录和文件inode的序列化信息。

Edits文件：存放HDFS文件系统的所有更新操作的路径，文件系统客户端执行的所有写操作首先
会被记录到Edits文件中。

seen_txid文件：保存的是一个数字，就是最后一个edits_的数字
在这里插入图片描述

每次NameNode启动的时候都会将Fsimage文件读入内存，加载Edits里面的更新操作，保证内存
中的元数据信息是最新的、同步的，可以看成NameNode启动的时候就将Fsimage和Edits文件进行了合并。

二、查看Fsimage和Edits文件

1、查看Fsimage文件

hdfs oiv -p XML -i fsimage_0000000000000000349 -o /opt/tmp/fsimage_0000000000000000349.xml

补充：修改目录所属用户及所属组

sudo chown atguigu:atguigu /opt/tmp/

xml文件内容
它保存的就是web页面的目录信息，但是，它没有记录文件块对应的DataNode节点信息，这是因为，在集群启动后，要求DataNode上报数据块信息，并间隔性的上报最新块信息。
在这里插入图片描述

<?xml version="1.0"?>
<fsimage><version><layoutVersion>-64</layoutVersion><onDiskVersion>1</onDiskVersion><oivRevision>ba631c436b806728f8ec2f54ab1e289526c90579</oivRevision></version>
<NameSection><namespaceId>1559270131</namespaceId><genstampV1>1000</genstampV1><genstampV2>1035</genstampV2><genstampV1Limit>0</genstampV1Limit><lastAllocatedBlockId>1073741857</lastAllocatedBlockId><txid>349</txid></NameSection>
<ErasureCodingSection>
<erasureCodingPolicy>
<policyId>1</policyId><policyName>RS-6-3-1024k</policyName><cellSize>1048576</cellSize><policyState>DISABLED</policyState><ecSchema>
<codecName>rs</codecName><dataUnits>6</dataUnits><parityUnits>3</parityUnits></ecSchema>
</erasureCodingPolicy><erasureCodingPolicy>
<policyId>2</policyId><policyName>RS-3-2-1024k</policyName><cellSize>1048576</cellSize><policyState>DISABLED</policyState><ecSchema>
<codecName>rs</codecName><dataUnits>3</dataUnits><parityUnits>2</parityUnits></ecSchema>
</erasureCodingPolicy><erasureCodingPolicy>
<policyId>3</policyId><policyName>RS-LEGACY-6-3-1024k</policyName><cellSize>1048576</cellSize><policyState>DISABLED</policyState><ecSchema>
<codecName>rs-legacy</codecName><dataUnits>6</dataUnits><parityUnits>3</parityUnits></ecSchema>
</erasureCodingPolicy><erasureCodingPolicy>
<policyId>4</policyId><policyName>XOR-2-1-1024k</policyName><cellSize>1048576</cellSize><policyState>DISABLED</policyState><ecSchema>
<codecName>xor</codecName><dataUnits>2</dataUnits><parityUnits>1</parityUnits></ecSchema>
</erasureCodingPolicy><erasureCodingPolicy>
<policyId>5</policyId><policyName>RS-10-4-1024k</policyName><cellSize>1048576</cellSize><policyState>DISABLED</policyState><ecSchema>
<codecName>rs</codecName><dataUnits>10</dataUnits><parityUnits>4</parityUnits></ecSchema>
</erasureCodingPolicy></ErasureCodingSection><INodeSection><lastInodeId>16452</lastInodeId><numInodes>30</numInodes><inode><id>16385</id><type>DIRECTORY</type><name></name><mtime>1715590504629</mtime><permission>atguigu:supergroup:0755</permission><nsquota>9223372036854775807</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16386</id><type>DIRECTORY</type><name>tmp</name><mtime>1714552374783</mtime><permission>atguigu:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16387</id><type>DIRECTORY</type><name>hadoop-yarn</name><mtime>1714396386802</mtime><permission>atguigu:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16388</id><type>DIRECTORY</type><name>staging</name><mtime>1714552295219</mtime><permission>atguigu:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16389</id><type>DIRECTORY</type><name>history</name><mtime>1714396386869</mtime><permission>atguigu:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16390</id><type>DIRECTORY</type><name>done</name><mtime>1714552573060</mtime><permission>atguigu:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16391</id><type>DIRECTORY</type><name>done_intermediate</name><mtime>1714552378760</mtime><permission>atguigu:supergroup:1777</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16392</id><type>DIRECTORY</type><name>input1</name><mtime>1715589612250</mtime><permission>atguigu:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16393</id><type>FILE</type><name>cls.txt</name><replication>3</replication><mtime>1714550964453</mtime><atime>1715589568850</atime><preferredBlockSize>134217728</preferredBlockSize><permission>atguigu:supergroup:0644</permission><blocks><block><id>1073741825</id><genstamp>1001</genstamp><numBytes>35</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16395</id><type>DIRECTORY</type><name>atguigu</name><mtime>1714552295219</mtime><permission>atguigu:supergroup:0700</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16396</id><type>DIRECTORY</type><name>.staging</name><mtime>1714552411438</mtime><permission>atguigu:supergroup:0700</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16404</id><type>DIRECTORY</type><name>logs</name><mtime>1714552374827</mtime><permission>atguigu:atguigu:1777</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16405</id><type>DIRECTORY</type><name>atguigu</name><mtime>1714552374831</mtime><permission>atguigu:atguigu:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16406</id><type>DIRECTORY</type><name>logs-tfile</name><mtime>1716191443316</mtime><permission>atguigu:atguigu:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16408</id><type>DIRECTORY</type><name>atguigu</name><mtime>1714552573116</mtime><permission>atguigu:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16410</id><type>DIRECTORY</type><name>output</name><mtime>1714552409823</mtime><permission>atguigu:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16416</id><type>FILE</type><name>part-r-00000</name><replication>3</replication><mtime>1714552409568</mtime><atime>1715240513984</atime><preferredBlockSize>134217728</preferredBlockSize><permission>atguigu:supergroup:0644</permission><blocks><block><id>1073741836</id><genstamp>1012</genstamp><numBytes>38</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16418</id><type>FILE</type><name>_SUCCESS</name><replication>3</replication><mtime>1714552409826</mtime><atime>1714552409823</atime><preferredBlockSize>134217728</preferredBlockSize><permission>atguigu:supergroup:0644</permission><storagePolicyId>0</storagePolicyId></inode>
<inode><id>16426</id><type>DIRECTORY</type><name>2024</name><mtime>1714552573060</mtime><permission>atguigu:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16427</id><type>DIRECTORY</type><name>05</name><mtime>1714552573060</mtime><permission>atguigu:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16428</id><type>DIRECTORY</type><name>01</name><mtime>1715236457191</mtime><permission>atguigu:supergroup:0770</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16433</id><type>FILE</type><name>weiguo.txt</name><replication>3</replication><mtime>1715240756249</mtime><atime>1715256474920</atime><preferredBlockSize>134217728</preferredBlockSize><permission>atguigu:supergroup:0644</permission><blocks><block><id>1073741845</id><genstamp>1021</genstamp><numBytes>7</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16434</id><type>FILE</type><name>wuguo.txt</name><replication>3</replication><mtime>1715240867021</mtime><atime>1715240866821</atime><preferredBlockSize>134217728</preferredBlockSize><permission>atguigu:supergroup:0644</permission><blocks><block><id>1073741846</id><genstamp>1022</genstamp><numBytes>6</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16436</id><type>DIRECTORY</type><name>jinguo</name><mtime>1715256921684</mtime><permission>atguigu:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16437</id><type>FILE</type><name>shuguo.txt</name><replication>10</replication><mtime>1715256804728</mtime><atime>1715256804492</atime><preferredBlockSize>134217728</preferredBlockSize><permission>atguigu:supergroup:0644</permission><blocks><block><id>1073741848</id><genstamp>1026</genstamp><numBytes>14</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16439</id><type>DIRECTORY</type><name>xiyou</name><mtime>1715589490184</mtime><permission>atguigu:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16448</id><type>DIRECTORY</type><name>huaguoshan</name><mtime>1715589000149</mtime><permission>atguigu:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16449</id><type>FILE</type><name>sunwukong.txt</name><replication>1</replication><mtime>1715589000729</mtime><atime>1715589000149</atime><preferredBlockSize>134217728</preferredBlockSize><permission>atguigu:supergroup:0644</permission><blocks><block><id>1073741855</id><genstamp>1033</genstamp><numBytes>9</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
<inode><id>16451</id><type>DIRECTORY</type><name>input</name><mtime>1715590504629</mtime><permission>atguigu:supergroup:0755</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16452</id><type>FILE</type><name>test.txt</name><replication>1</replication><mtime>1715590504933</mtime><atime>1715590504629</atime><preferredBlockSize>134217728</preferredBlockSize><permission>atguigu:supergroup:0644</permission><blocks><block><id>1073741857</id><genstamp>1035</genstamp><numBytes>11</numBytes></block>
</blocks>
<storagePolicyId>0</storagePolicyId></inode>
</INodeSection>
<INodeReferenceSection></INodeReferenceSection><SnapshotSection><snapshotCounter>0</snapshotCounter><numSnapshots>0</numSnapshots></SnapshotSection>
<INodeDirectorySection><directory><parent>16385</parent><child>16393</child><child>16451</child><child>16392</child><child>16436</child><child>16410</child><child>16386</child><child>16439</child></directory>
<directory><parent>16386</parent><child>16387</child><child>16404</child></directory>
<directory><parent>16387</parent><child>16388</child></directory>
<directory><parent>16388</parent><child>16395</child><child>16389</child></directory>
<directory><parent>16389</parent><child>16390</child><child>16391</child></directory>
<directory><parent>16390</parent><child>16426</child></directory>
<directory><parent>16391</parent><child>16408</child></directory>
<directory><parent>16395</parent><child>16396</child></directory>
<directory><parent>16404</parent><child>16405</child></directory>
<directory><parent>16405</parent><child>16406</child></directory>
<directory><parent>16410</parent><child>16418</child><child>16416</child></directory>
<directory><parent>16426</parent><child>16427</child></directory>
<directory><parent>16427</parent><child>16428</child></directory>
<directory><parent>16436</parent><child>16437</child><child>16433</child><child>16434</child></directory>
<directory><parent>16439</parent><child>16448</child></directory>
<directory><parent>16448</parent><child>16449</child></directory>
<directory><parent>16451</parent><child>16452</child></directory>
</INodeDirectorySection>
<FileUnderConstructionSection></FileUnderConstructionSection>
<SecretManagerSection><currentId>0</currentId><tokenSequenceNumber>0</tokenSequenceNumber><numDelegationKeys>0</numDelegationKeys><numTokens>0</numTokens></SecretManagerSection><CacheManagerSection><nextDirectiveId>1</nextDirectiveId><numDirectives>0</numDirectives><numPools>0</numPools></CacheManagerSection>
</fsimage>

2、查看Edits文件

hdfs oev -p XML -i edits_inprogress_0000000000000000351 -o /opt/tmp/edits_inprogress_0000000000000000351.xml

在这里插入图片描述
思考：NameNode如何确定下次开机启动的时候合并哪些Edits？
合并那些ID大于fsimage尾号的edits文件，加载到内存中，这时，内存里面就有最新最全的元数据信息。