Hadoop集群(二) HDFS搭建-白红宇

Hadoop集群(二) HDFS搭建

阅读量：6205 次

发布时间：2019-06-21

本文共 19680 字，大约阅读时间需要 65 分钟。

HDFS只是Hadoop最基本的一个服务，很多其他服务，都是基于HDFS展开的。所以部署一个HDFS集群，是很核心的一个动作，也是大数据平台的开始。

安装Hadoop集群，首先需要有Zookeeper才可以完成安装。如果没有Zookeeper，请先部署一套Zookeeper。另外，JDK以及物理主机的一些设置等。请参考：

下面开始HDFS的安装

HDFS主机分配

192.168.67.101 c6701

--Namenode+datanode

192.168.67.102 c6702

--datanode

192.168.67.103 c6703

--datanode

1. 安装HDFS，解压hadoop-2.6.0-EDH-0u2.tar.gz

我同时下载2.6和2.7版本的软件，先安装2.6，然后在执行2.6到2.7的升级步骤

                         
useradd hdfs
echo 
"hdfs:hdfs"
| chpasswd
su - hdfs
cd /tmp/software
tar -zxvf hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs/
mkdir -p /data/hadoop/
temp
mkdir -p /data/hadoop/journal 
mkdir -p /data/hadoop/hdfs/
name
mkdir -p /data/hadoop/hdfs/data
chown -R hdfs:hdfs /data/hadoop
chown -R hdfs:hdfs /data/hadoop/
temp
chown -R hdfs:hdfs /data/hadoop/journal 
chown -R hdfs:hdfs /data/hadoop/hdfs/
name
chown -R hdfs:hdfs /data/hadoop/hdfs/data 
$ pwd
/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop
        
      

2. 修改core-site.xml对应的参数

                         
$ cat core-site.xml
<configuration>
 
<!
-- 指定hdfs的nameservice为ns -->
 
<property>   
      
<
name
>fs.defaultFS</
name
>   
      
<value>hdfs://ns</value>   
 
</property>
 
<!
--指定hadoop数据临时存放目录-->
 
<property>
      
<
name
>hadoop.tmp.dir</
name
>
      
<value>/data/hadoop/
temp
</value>
 
</property> 
                         
 
<property>   
      
<
name
>io.file.buffer.
size
</
name
>   
      
<value>4096</value>   
 
</property>
 
<!
--指定zookeeper地址-->
 
<property>
      
<
name
>ha.zookeeper.quorum</
name
>
      
<value>c6701:2181,c6702:2181,c6703:2181</value>
 
</property>
 
</configuration>
        
      

3. 修改hdfs-site.xml对应的参数

                         
cat hdfs-site.xml
<configuration>
    
<!
--指定hdfs的nameservice为ns，需要和core-site.xml中的保持一致,并且ns如果改，整个文件中，全部的ns要都修改，保持统一 -->   
    
<property>   
        
<
name
>dfs.nameservices</
name
>   
        
<value>ns</value>   
    
</property> 
    
<!
-- ns下面有两个NameNode，分别是nn1，nn2 -->
    
<property>
      
<
name
>dfs.ha.namenodes.ns</
name
>
      
<value>nn1,nn2</value>
    
</property>
    
<!
-- nn1的RPC通信地址 -->
    
<property>
      
<
name
>dfs.namenode.rpc-address.ns.nn1</
name
>
      
<value>c6701:9000</value>
    
</property>
    
<!
-- nn1的http通信地址 -->
    
<property>
        
<
name
>dfs.namenode.http-address.ns.nn1</
name
>
        
<value>c6701:50070</value>
    
</property>
    
<!
-- nn2的RPC通信地址 -->
    
<property>
        
<
name
>dfs.namenode.rpc-address.ns.nn2</
name
>
        
<value>c6702:9000</value>
    
</property>
    
<!
-- nn2的http通信地址 -->
    
<property>
        
<
name
>dfs.namenode.http-address.ns.nn2</
name
>
        
<value>c6702:50070</value>
    
</property>
    
<!
-- 指定NameNode的元数据在JournalNode上的存放位置 -->
    
<property>
        
<
name
>dfs.namenode.shared.edits.dir</
name
>
        
<value>qjournal://c6701:8485;c6702:8485;c6703:8485/ns</value>
    
</property>
    
<!
-- 指定JournalNode在本地磁盘存放数据的位置 -->
    
<property>
          
<
name
>dfs.journalnode.edits.dir</
name
>
          
<value>/data/hadoop/journal</value>
    
</property>
    
<!
-- 开启NameNode故障时自动切换 -->
    
<property>
          
<
name
>dfs.ha.automatic-failover.enabled</
name
>
          
<value>
true
</value>
    
</property>
    
<!
-- 配置失败自动切换实现方式 -->
    
<property>
            
<
name
>dfs.client.failover.proxy.provider.ns</
name
>
            
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    
</property>
    
<!
-- 配置隔离机制 -->
    
<property>
            
<
name
>dfs.ha.fencing.methods</
name
>
            
<value>sshfence</value>
    
</property>
    
<!
-- 使用隔离机制时需要ssh免登陆 -->
    
<property>
            
<
name
>dfs.ha.fencing.ssh.private-
key
-files</
name
>
            
<value>/home/hdfs/.ssh/id_rsa</value>
    
</property>
                             
    
<property>   
        
<
name
>dfs.namenode.
name
.dir</
name
>   
        
<value>/data/hadoop/hdfs/
name
</value>   
    
</property>   
   
    
<property>   
        
<
name
>dfs.datanode.data.dir</
name
>   
        
<value>/data/hadoop/hdfs/data</value>   
    
</property>   
   
    
<property>   
      
<
name
>dfs.replication</
name
>   
      
<value>2</value>   
    
</property> 
    
<!
-- 在NN和DN上开启WebHDFS (REST API)功能,不是必须 -->                                                                   
    
<property>   
      
<
name
>dfs.webhdfs.enabled</
name
>   
      
<value>
true
</value>   
    
</property>   
</configuration>
        
      

4. 添加slaves文件

                         
$ more slaves
c6701
c6702
c6703

--- 安装C6702的hdfs---

5. 创建c6702的用户，并为hdfs用户ssh免密

                         
ssh c6702 
"useradd hdfs"
ssh c6702 
"echo "
hdfs:hdfs
" | chpasswd"
ssh-copy-id  hdfs@c6702
        
      

6. 拷贝软件

                         
scp -r /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz root@c6702:/tmp/software/.
ssh c6702 
"chmod 777 /tmp/software/*"

7. 创建目录，解压软件

                         
ssh hdfs@c6702 
"mkdir hdfs"
ssh hdfs@c6702 
"tar -zxvf /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs"
ssh hdfs@c6702 
"ls -al hdfs"
ssh hdfs@c6702 
"ls -al hdfs/hadoop*"
        
      

复制配置文件

                         
ssh hdfs@c6702 
"rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml"
ssh hdfs@c6702 
"rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml"
scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml
scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml
scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves
        
      

创建hdfs需要的目录

                         
ssh root@c6702 
"mkdir -p /data/hadoop"
ssh root@c6702 
" chown -R hdfs:hdfs  /data/hadoop"
ssh hdfs@c6702 
"mkdir -p /data/hadoop/temp"
ssh hdfs@c6702 
"mkdir -p /data/hadoop/journal"
ssh hdfs@c6702 
"mkdir -p /data/hadoop/hdfs/name"
ssh hdfs@c6702 
"mkdir -p /data/hadoop/hdfs/data"
        
      

--- 安装C6703的hdfs---

8. 创建c6703的用户，并为hdfs用户ssh免密

                         
ssh c6703 
"useradd hdfs"
ssh c6703 
"echo "
hdfs:hdfs
" | chpasswd"
ssh-copy-id  hdfs@c6703
        
      

9. 拷贝软件

                         
scp -r /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz root@c6703:/tmp/software/.
ssh c6703 
"chmod 777 /tmp/software/*"
10. 创建目录，解压软件
ssh hdfs@c6703 
"mkdir hdfs"
ssh hdfs@c6703 
"tar -zxvf /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs"
ssh hdfs@c6703 
"ls -al hdfs"
ssh hdfs@c6703 
"ls -al hdfs/hadoop*"
        
      

复制配置文件

                         
ssh hdfs@c6703 
"rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml"
ssh hdfs@c6703 
"rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml"
scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml
scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml
scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves
        
      

创建hdfs需要的目录

                         
ssh root@c6703 
"mkdir -p /data/hadoop"
ssh root@c6703 
" chown -R hdfs:hdfs  /data/hadoop"
ssh hdfs@c6703 
"mkdir -p /data/hadoop/temp"
ssh hdfs@c6703 
"mkdir -p /data/hadoop/journal"
ssh hdfs@c6703 
"mkdir -p /data/hadoop/hdfs/name"
ssh hdfs@c6703 
"mkdir -p /data/hadoop/hdfs/data"
        
      

11. 启动HDFS，先启动三个节点的journalnode

/home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start journalnode

检查状态

                         
$ jps
3958 Jps
3868 JournalNode

12. 然后启动namenode,首次启动namenode之前，先在其中一个节点(主节点)format namenode信息，信息会存在于dfs.namenode.name.dir指定的路径中

                         
 
<
name
>dfs.namenode.
name
.dir</
name
>   
 
<value>/data/hadoop/hdfs/
name
</value>
        
      

                         
$ ./hdfs namenode -format
17/09/26 07:52:17 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = c6701.python279.org/192.168.67.101
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.6.0-EDH-0u2
STARTUP_MSG:   classpath = /home/hdfs/hadoop-2.6.0-EDHxxxxxxxxxx
STARTUP_MSG:   build = http://gitlab-xxxxx
STARTUP_MSG:   java = 1.8.0_144
************************************************************/
17/09/26 07:52:17 INFO namenode.NameNode: registered UNIX signal handlers 
for
[TERM, HUP, 
INT
]
17/09/26 07:52:17 INFO namenode.NameNode: createNameNode [-format]
17/09/26 07:52:18 WARN common.Util: Path /data/hadoop/hdfs/
name
should be specified 
as
a URI 
in
configuration files. Please 
update
hdfs configuration.
17/09/26 07:52:18 WARN common.Util: Path /data/hadoop/hdfs/
name
should be specified 
as
a URI 
in
configuration files. Please 
update
hdfs configuration.
Formatting using clusterid: CID-b2f01411-862f-44b2-a6dc-7d17bd48d522
17/09/26 07:52:18 INFO namenode.FSNamesystem: 
No
KeyProvider found.
17/09/26 07:52:18 INFO namenode.FSNamesystem: fsLock 
is
fair:
true
17/09/26 07:52:18 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
17/09/26 07:52:18 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-
check
=
true
17/09/26 07:52:18 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec 
is
set
to
000:00:00:00.000
17/09/26 07:52:18 INFO blockmanagement.BlockManager: The block deletion will start around 2017 Sep 26 07:52:18
17/09/26 07:52:18 INFO util.GSet: Computing capacity 
for
map BlocksMap
17/09/26 07:52:18 INFO util.GSet: VM type       = 64-
bit
17/09/26 07:52:18 INFO util.GSet: 2.0% 
max
memory 966.7 MB = 19.3 MB
17/09/26 07:52:18 INFO util.GSet: capacity      = 2^21 = 2097152 entries
17/09/26 07:52:18 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=
false
17/09/26 07:52:18 INFO blockmanagement.BlockManager: defaultReplication         = 2
17/09/26 07:52:18 INFO blockmanagement.BlockManager: maxReplication             = 512
17/09/26 07:52:18 INFO blockmanagement.BlockManager: minReplication             = 1
17/09/26 07:52:18 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
17/09/26 07:52:18 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = 
false
17/09/26 07:52:18 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
17/09/26 07:52:18 INFO blockmanagement.BlockManager: encryptDataTransfer        = 
false
17/09/26 07:52:18 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
17/09/26 07:52:18 INFO namenode.FSNamesystem: fsOwner             = hdfs (auth:SIMPLE)
17/09/26 07:52:18 INFO namenode.FSNamesystem: supergroup          = supergroup
17/09/26 07:52:18 INFO namenode.FSNamesystem: isPermissionEnabled = 
true
17/09/26 07:52:18 INFO namenode.FSNamesystem: Determined nameservice ID: ns
17/09/26 07:52:18 INFO namenode.FSNamesystem: HA Enabled: 
true
17/09/26 07:52:18 INFO namenode.FSNamesystem: Append Enabled: 
true
17/09/26 07:52:18 INFO util.GSet: Computing capacity 
for
map INodeMap
17/09/26 07:52:18 INFO util.GSet: VM type       = 64-
bit
17/09/26 07:52:18 INFO util.GSet: 1.0% 
max
memory 966.7 MB = 9.7 MB
17/09/26 07:52:18 INFO util.GSet: capacity      = 2^20 = 1048576 entries
17/09/26 07:52:18 INFO namenode.NameNode: Caching file names occuring more than 10 times
17/09/26 07:52:18 INFO util.GSet: Computing capacity 
for
map cachedBlocks
17/09/26 07:52:18 INFO util.GSet: VM type       = 64-
bit
17/09/26 07:52:18 INFO util.GSet: 0.25% 
max
memory 966.7 MB = 2.4 MB
17/09/26 07:52:18 INFO util.GSet: capacity      = 2^18 = 262144 entries
17/09/26 07:52:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
17/09/26 07:52:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.
min
.datanodes = 0
17/09/26 07:52:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
17/09/26 07:52:18 INFO namenode.FSNamesystem: Retry cache 
on
namenode 
is
enabled
17/09/26 07:52:18 INFO namenode.FSNamesystem: Retry cache will use 0.03 
of
total heap 
and
retry cache entry expiry 
time
is
600000 millis
17/09/26 07:52:18 INFO util.GSet: Computing capacity 
for
map NameNodeRetryCache
17/09/26 07:52:18 INFO util.GSet: VM type       = 64-
bit
17/09/26 07:52:18 INFO util.GSet: 0.029999999329447746% 
max
memory 966.7 MB = 297.0 KB
17/09/26 07:52:18 INFO util.GSet: capacity      = 2^15 = 32768 entries
17/09/26 07:52:18 INFO namenode.NNConf: ACLs enabled? 
false
17/09/26 07:52:18 INFO namenode.NNConf: XAttrs enabled? 
true
17/09/26 07:52:18 INFO namenode.NNConf: Maximum 
size
of
an xattr: 16384
17/09/26 07:52:19 INFO namenode.FSImage: Allocated new BlockPoolId: BP-144216011-192.168.67.101-1506412339757
17/09/26 07:52:19 INFO common.Storage: Storage directory /data/hadoop/hdfs/
name
has been successfully formatted.
17/09/26 07:52:20 INFO namenode.NNStorageRetentionManager: Going 
to
retain 1 images 
with
txid >= 0
17/09/26 07:52:20 INFO util.ExitUtil: Exiting 
with
status 0
17/09/26 07:52:20 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode 
at
c6701.python279.org/192.168.67.101
************************************************************/
        
      

13. standby namenode需要先执行bootstrapstandby，输出如下

                         
[hdfs@c6702 sbin]$ ../bin/hdfs namenode -bootstrapstandby
17/09/26 09:44:58 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = c6702.python279.org/192.168.67.102
STARTUP_MSG:   args = [-bootstrapstandby]
STARTUP_MSG:   version = 2.6.0-EDH-0u2
STARTUP_MSG:   classpath = /home/hdfs/haxxx
STARTUP_MSG:   build = http://gitlab-xxxx
STARTUP_MSG:   java = 1.8.0_144
************************************************************/
17/09/26 09:44:58 INFO namenode.NameNode: registered UNIX signal handlers 
for
[TERM, HUP, 
INT
]
17/09/26 09:44:58 INFO namenode.NameNode: createNameNode [-bootstrapstandby]
17/09/26 09:44:59 WARN common.Util: Path /data/hadoop/hdfs/
name
should be specified 
as
a URI 
in
configuration files. Please 
update
hdfs configuration.
17/09/26 09:44:59 WARN common.Util: Path /data/hadoop/hdfs/
name
should be specified 
as
a URI 
in
configuration files. Please 
update
hdfs configuration.
=====================================================
About 
to
bootstrap Standby ID nn2 
from
:
           
Nameservice ID: ns
        
Other Namenode ID: nn1
  
Other NN
's HTTP address: http://c6701:50070
  
Other NN'
s IPC  address: c6701/192.168.67.101:9000
             
Namespace ID: 793662207
            
Block pool ID: BP-144216011-192.168.67.101-1506412339757
               
Cluster ID: CID-b2f01411-862f-44b2-a6dc-7d17bd48d522
           
Layout version: -60
=====================================================
Re-format filesystem 
in
Storage Directory /data/hadoop/hdfs/
name
? (Y 
or
N) y
17/09/26 09:45:16 INFO common.Storage: Storage directory /data/hadoop/hdfs/
name
has been successfully formatted.
17/09/26 09:45:16 WARN common.Util: Path /data/hadoop/hdfs/
name
should be specified 
as
a URI 
in
configuration files. Please 
update
hdfs configuration.
17/09/26 09:45:16 WARN common.Util: Path /data/hadoop/hdfs/
name
should be specified 
as
a URI 
in
configuration files. Please 
update
hdfs configuration.
17/09/26 09:45:17 INFO namenode.TransferFsImage: Opening 
connection
to
http://c6701:50070/imagetransfer?getimage=1&txid=0&storageInfo=-60:793662207:0:CID-b2f01411-862f-44b2-a6dc-7d17bd48d522
17/09/26 09:45:17 INFO namenode.TransferFsImage: Image Transfer timeout configured 
to
60000 milliseconds
17/09/26 09:45:17 INFO namenode.TransferFsImage: Transfer took 0.01s 
at
0.00 KB/s
17/09/26 09:45:17 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 
size
351 bytes.
17/09/26 09:45:17 INFO util.ExitUtil: Exiting 
with
status 0
17/09/26 09:45:17 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode 
at
c6702.python279.org/192.168.67.102
************************************************************/
        
      

14. 检查状态，namenode还没有启动

                         
[hdfs@c6702 sbin]$ jps
4539 Jps
3868 JournalNode

15. 启动standby namenode，命令和master启动的方式相同

                         
[hdfs@c6702 sbin]$ ./hadoop-daemon.sh start namenode
starting namenode, logging 
to
/home/hdfs/hadoop-2.6.0-EDH-0u2/logs/hadoop-hdfs-namenode-c6702.python279.org.
out
        
      

16. 再次检查，namenode已经启动

                         
[hdfs@c6702 sbin]$ jps
4640 Jps
4570 NameNode
3868 JournalNode

17. 格式化zkfc,让在zookeeper中生成ha节点，在master上执行如下命令，完成格式化

                         
[hdfs@c6701 bin]$ ./hdfs zkfc -formatZK
17/09/26 09:59:20 INFO tools.DFSZKFailoverController: Failover controller configured 
for
NameNode NameNode 
at
c6701/192.168.67.101:9000
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built 
on
02/20/2014 09:09 GMT
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:host.
name
=c6701.python279.org
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_144
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/
local
/jdk1.8.0_144/jre
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/hdfs/hadoop-2.6.0-EDH-0u2/exxxx
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/hdfs/hadoop-2.6.0-EDH-0u2/lib/native
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:os.
name
=Linux
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.el6.x86_64
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:
user
.
name
=hdfs
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:
user
.home=/home/hdfs
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:
user
.dir=/home/hdfs/hadoop-2.6.0-EDH-0u2/bin
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Initiating client 
connection
, connectString=c6701:2181,c6702:2181,c6703:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@20deea7f
17/09/26 09:59:20 INFO zookeeper.ClientCnxn: Opening socket 
connection
to
server c6703.python279.org/192.168.67.103:2181. Will 
not
attempt 
to
authenticate using SASL (unknown error)
17/09/26 09:59:20 INFO zookeeper.ClientCnxn: Socket 
connection
established 
to
c6703.python279.org/192.168.67.103:2181, initiating session
17/09/26 09:59:20 INFO zookeeper.ClientCnxn: Session establishment complete 
on
server c6703.python279.org/192.168.67.103:2181, sessionid = 0x35ebc5163710000, negotiated timeout = 5000
17/09/26 09:59:20 INFO ha.ActiveStandbyElector: Session connected.
17/09/26 09:59:20 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/ns 
in
ZK.
17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Session: 0x35ebc5163710000 closed
17/09/26 09:59:20 INFO zookeeper.ClientCnxn: EventThread shut down
        
      

18. 格式化完成的检查

格式成功后，查看zookeeper中可以看到 <<<<<<<<<<<命令没确认

                         
[zk: localhost:2181(CONNECTED) 1] ls /hadoop-ha

19. 启动zkfc，这个就是为namenode使用的

                         
./hadoop-daemon.sh start zkfc
starting zkfc, logging 
to
/home/hdfs/hadoop-2.6.0-EDH-0u2/logs/hadoop-hdfs-zkfc-c6701.python279.org.
out
$ jps
4272 DataNode
4402 JournalNode
6339 Jps
6277 DFSZKFailoverController
4952 NameNode
        
      

20. 另一个节点启动zkfc，

                         
ssh  hdfs@c6702 
/home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start zkfc
$ jps
4981 Jps
4935 DFSZKFailoverController
4570 NameNode
3868 JournalNode
        
      

21. 注意：进行初始化的时候，必须保证zk集群已经启动了。

1、在ZK中创建znode来存储automatic Failover的数据，任选一个NN执行完成即可：

sh bin/hdfs zkfc -formatZK

2、启动zkfs，在所有的NN节点中执行以下命令：

sh sbin/hadoop-daemon.sh start zkfc

22. 启动datanode

最后启动集群

                         
/home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start zkfc
    
sh sbin/start-dfs.sh

HDFS安装过程中的重点，最后在软件启动过程中，一些初始化操作，很重要。

1. 启动全部的journalnode

2. 在namenode1上执行， hdfs namenode -format

3. 在namenode1上执行，启动namenode1，命令hadoop-daemon.sh start namenode

4. 在namenode2上执行, hdfs namenode -bootstrapstandby

5. 在namenode1上执行，格式化zkfc,在zookeeper中生成HA节点, hdfs zkfc -formatZK

6. 启动zkfc，hadoop-daemon.sh start zkfc。有namenode运行的节点，都要启动ZKFC

7. 启动 datanode

HDFS只是Hadoop最基本的一个模块，这里已经安装完成，可以为后面的Hbase提供服务了。

http://blog.51cto.com/hsbxxl/1971475

转载于:https://www.cnblogs.com/chuancheng/p/8973301.html

你可能感兴趣的文章

从Mysql slave system lock延迟说开去

查看>>

归并排序

查看>>

RecyclerView的下拉刷新和加载更多动画

查看>>

ABAP常见面试问题

查看>>

程序猿是如何解决SQLServer占CPU100%的

查看>>

web.xml

查看>>

HBase-1.2.4LruBlockCache实现分析（一）

物联网商机迸发 LPWAN芯片现身本文转自d1net（转载）

查看>>

【eclipse转idea的第一天】配置idea

查看>>

error: Refusing toundefine while domain managed save image exists