国产99久久精品_欧美日本韩国一区二区_激情小说综合网_欧美一级二级视频_午夜av电影_日本久久精品视频

最新文章專題視頻專題問答1問答10問答100問答1000問答2000關鍵字專題1關鍵字專題50關鍵字專題500關鍵字專題1500TAG最新視頻文章推薦1 推薦3 推薦5 推薦7 推薦9 推薦11 推薦13 推薦15 推薦17 推薦19 推薦21 推薦23 推薦25 推薦27 推薦29 推薦31 推薦33 推薦35 推薦37視頻文章20視頻文章30視頻文章40視頻文章50視頻文章60 視頻文章70視頻文章80視頻文章90視頻文章100視頻文章120視頻文章140 視頻2關鍵字專題關鍵字專題tag2tag3文章專題文章專題2文章索引1文章索引2文章索引3文章索引4文章索引5123456789101112131415文章專題3
問答文章1 問答文章501 問答文章1001 問答文章1501 問答文章2001 問答文章2501 問答文章3001 問答文章3501 問答文章4001 問答文章4501 問答文章5001 問答文章5501 問答文章6001 問答文章6501 問答文章7001 問答文章7501 問答文章8001 問答文章8501 問答文章9001 問答文章9501
當前位置: 首頁 - 科技 - 知識百科 - 正文

hadoop增加新節點實踐

來源:懂視網 責編:小采 時間:2020-11-09 14:46:54
文檔

hadoop增加新節點實踐

hadoop增加新節點實踐:之前已經有了namenode和datanode1,現在要新增節點datanode2 第一步:修改將要增加節點的主機名 hadoop@datanode1:~$ vim /etc/hostname datanode2 第二步:修改host文件 hadoop@datanode1:~$ vim /etc/hosts 192.16
推薦度:
導讀hadoop增加新節點實踐:之前已經有了namenode和datanode1,現在要新增節點datanode2 第一步:修改將要增加節點的主機名 hadoop@datanode1:~$ vim /etc/hostname datanode2 第二步:修改host文件 hadoop@datanode1:~$ vim /etc/hosts 192.16

之前已經有了namenode和datanode1,現在要新增節點datanode2 第一步:修改將要增加節點的主機名 hadoop@datanode1:~$ vim /etc/hostname datanode2 第二步:修改host文件 hadoop@datanode1:~$ vim /etc/hosts 192.168.8.4 datanode2 127.0.0.1 localhost 127.0

之前已經有了namenode和datanode1,現在要新增節點datanode2
第一步:修改將要增加節點的主機名
hadoop@datanode1:~$ vim /etc/hostname
datanode2
第二步:修改host文件
hadoop@datanode1:~$ vim /etc/hosts
192.168.8.4 datanode2
127.0.0.1 localhost
127.0.1.1 ubuntu
192.168.8.2 namenode
192.168.8.3 datanode1
192.168.8.4 datanode2(增加了這個)

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
第三步:修改ip
\
第四步:重啟
第五步:ssh免密碼配置
1.生成密鑰
hadoop@datanode2:~$ ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
/home/hadoop/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
34:45:84:85:6e:f3:9e:7a:c0:f1:a4:ef:bf:30:a6:74 hadoop@datanode2
The key's randomart image is:
+--[ RSA 2048]----+
| *= |
| o. |
| .o |
| .=.. |
| oSB |
| + o |
| .+E. |
| . +=o |
| o+..o. |
+-----------------+
2.把公鑰傳給namenode
hadoop@datanode2:~$ cd ~/.ssh
hadoop@datanode2:~/.ssh$ ls
authorized_keys id_rsa id_rsa.pub known_hosts
hadoop@datanode2:~/.ssh$ scp ./id_rsa.pub hadoop@namenode:/home/hadoop
hadoop@namenode's password:
id_rsa.pub 100% 398 0.4KB/s 00:00
3.把公鑰追加到authorized_keys
hadoop@namenode:~/.ssh$ cat ../id_rsa.pub >> authorized_keys
hadoop@namenode:~/.ssh$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDuOOD8R7OfNSUhGPZhQWCfC0yTeM6+txWSo3LiJjEWZbH512ymKIEiNRjCzTiRjLEqWGadAPVbip3jLuOHFpk89v7D6q8QH4ilBjLtsaVxmhb77w3yGrXlHJ8+g3QtS8VmjGEyZ86oeM5F9UM8F8QmK9mxXOWhqt3xvufetr7o7acV3APEHH1hvvkFImim2sT/iNi/Nxsch176byUS6y86gOTgznVH8OIx8MDmdKSLjqWPSCTrpvXPESlZvpLm4YSN2cYoKaxcedaynzOhXgAC0GLdq1k07eFmerUwpBT+xTzTRJPquYawK+MPf6+lnLm89u+bewdBZLdunCKhbCK3 hadoop@ubuntu3
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCssQnDzo5uhPn93bVqj+nEpzgQBipc1WgasOeFQV7ljyNlFHhOPVS6G3oHpvSrbjg3aK1MqxmCw0VokuuO5eoHwqh0alQw46eEmunzrnwuhhFpAU9V4t7LJ5pYuxZOioXbsJKxCetOY6G2lKRmyk2Z/MIMpPW+UFebt150+oYXcKKYSBBJoLmThH3bWW2CesAokIe8gCQ3rIYsHfA8rNuwxEnrL8fC2XlWODTahjHD5bymBO4rd3uiJxuTv7/r243t0hrimjhJ7uUIyPcIRYDchPmmO9DFVEBtYloLmqQQs/ZOxDiX7GF+YK7KC7Ayo1kL8VuwP90dqIhpaJmP96zV hadoop@ubuntu2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDbeTMrOtMZ8gurJyzoSVFpJbtXzUYDElXJcfm0O+FRpigxoIePPHiQc5vi7kabnLSiEv+94YDMclxZpXFjR0TXz6IJOVdPxFPqovY+GzrYVXEXj3HhbBWKC4sFUvGFGSZr8rM3R5OE2wYIZzOKdX9c6Ak5uIE7BUSuXzaiFctYXIvu37TObYZ44vDQGv9/mPsqP4Qnyx4czTLD1VmOeUHA5iQTKLt4K0HNE3i+a3mEEBMxBwETUI/6dcmvTxjEe7cy48YPadr5UT0/xgTub/OdmkBfvfT6fPDVlHtRP5jQiFapFyzL/BXiObqkSlrJbLKWTczS8J6SfsKWsSZfOPzL hadoop@datanode2
4.把公鑰傳給其節點
hadoop@namenode:~$ scp ./.ssh/authorized_keys hadoop@datanode1:/home/hadoop/.ssh/authorized_keys
authorized_keys 100% 1190 1.2KB/s 00:00
hadoop@namenode:~$ scp ./.ssh/authorized_keys hadoop@datanode2:/home/hadoop/.ssh/authorized_keys
authorized_keys 100% 1190 1.2KB/s 00:00
5.一個錯誤

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@ WARNING: UNPROTECTED PRIVATE KEY FILE! @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Permissions 0644 for '/home/jiangqixiang/.ssh/id_dsa' are too open.

It is recommended that your private key files are NOT accessible by others.

This private key will be ignored.

bad permissions: ignore key: /home/youraccount/.ssh/id_dsa 解決方法:

chmod 700 id_rsa

第六步:修改namenode的配置文件

hadoop@namenode:~$ cd hadoop-1.2.1/conf

hadoop@namenode:~/hadoop-1.2.1/conf$ vim slaves

datanode1

datanode2

第七步:負載均衡

hadoop@namenode:~/hadoop-1.2.1/conf$ start-balancer.sh

Warning: $HADOOP_HOME is deprecated.

starting balancer, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-balancer-namenode.out

以下摘自其他博客

1)如果不balance,那么cluster會把新的數據都存放在新的node上,這樣會降低Map Reduce的工作效率

2)threshold是平衡閾值,默認是10%,值越低各節點越平衡,但消耗時間也更長

/app/hadoop/bin/start-balancer.sh -threshold 0.1

3)在namenode的配置文件 hdfs-site.xml 可以加上balance的帶寬(默認值就是1M):

  dfs.balance.bandwidthPerSec

  1048576

  

    Specifies the maximum amount of bandwidth that each datanode

    can utilize for the balancing purpose in term of

    the number of bytes per second.

  

第八步:測試是否有效

1.啟動hadoop

hadoop@namenode:~/hadoop-1.2.1$ start-all.sh

Warning: $HADOOP_HOME is deprecated.

starting namenode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-namenode-namenode.out

datanode2: starting datanode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-datanode2.out

datanode1: starting datanode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-datanode1.out

namenode: starting secondarynamenode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-secondarynamenode-namenode.out

starting jobtracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-jobtracker-namenode.out

datanode2: starting tasktracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-datanode2.out

datanode1: starting tasktracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-datanode1.out

hadoop@namenode:~/hadoop-1.2.1$

2.錯誤

運行wordcount程序時出現錯誤

hadoop@namenode:~/hadoop-1.2.1$ hadoop jar hadoop-examples-1.2.1.jar wordcount in out

Warning: $HADOOP_HOME is deprecated.

14/09/12 08:40:39 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop cause:org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.mapred.SafeModeException: JobTracker is in safe mode

at org.apache.hadoop.mapred.JobTracker.checkSafeMode(JobTracker.java:5188)

at org.apache.hadoop.mapred.JobTracker.getStagingAreaDir(JobTracker.java:3677)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)

org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.mapred.SafeModeException: JobTracker is in safe mode

at org.apache.hadoop.mapred.JobTracker.checkSafeMode(JobTracker.java:5188)

at org.apache.hadoop.mapred.JobTracker.getStagingAreaDir(JobTracker.java:3677)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)

at org.apache.hadoop.ipc.Client.call(Client.java:1113)

at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)

at org.apache.hadoop.mapred.$Proxy2.getStagingAreaDir(Unknown Source)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)

at org.apache.hadoop.mapred.$Proxy2.getStagingAreaDir(Unknown Source)

at org.apache.hadoop.mapred.JobClient.getStagingAreaDir(JobClient.java:1309)

at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:102)

at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)

at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)

at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)

at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)

at org.apache.hadoop.examples.WordCount.main(WordCount.java:82)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)

at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)

at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

解決方法:

hadoop@namenode:~/hadoop-1.2.1$ hadoop dfsadmin -safemode leave

Warning: $HADOOP_HOME is deprecated.

Safe mode is OFF

3.再次測試

hadoop@namenode:~/hadoop-1.2.1$ hadoop jar hadoop-examples-1.2.1.jar wordcount in out

Warning: $HADOOP_HOME is deprecated.

14/09/12 08:48:26 INFO input.FileInputFormat: Total input paths to process : 2

14/09/12 08:48:26 INFO util.NativeCodeLoader: Loaded the native-hadoop library

14/09/12 08:48:26 WARN snappy.LoadSnappy: Snappy native library not loaded

14/09/12 08:48:28 INFO mapred.JobClient: Running job: job_201409120827_0003

14/09/12 08:48:29 INFO mapred.JobClient: map 0% reduce 0%

14/09/12 08:48:47 INFO mapred.JobClient: map 50% reduce 0%

14/09/12 08:48:48 INFO mapred.JobClient: map 100% reduce 0%

14/09/12 08:48:57 INFO mapred.JobClient: map 100% reduce 33%

14/09/12 08:48:59 INFO mapred.JobClient: map 100% reduce 100%

14/09/12 08:49:02 INFO mapred.JobClient: Job complete: job_201409120827_0003

14/09/12 08:49:02 INFO mapred.JobClient: Counters: 30

14/09/12 08:49:02 INFO mapred.JobClient: Job Counters

14/09/12 08:49:02 INFO mapred.JobClient: Launched reduce tasks=1

14/09/12 08:49:02 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=27285

14/09/12 08:49:02 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0

14/09/12 08:49:02 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0

14/09/12 08:49:02 INFO mapred.JobClient: Rack-local map tasks=1

14/09/12 08:49:02 INFO mapred.JobClient: Launched map tasks=2

14/09/12 08:49:02 INFO mapred.JobClient: Data-local map tasks=1

14/09/12 08:49:02 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=12080

14/09/12 08:49:02 INFO mapred.JobClient: File Output Format Counters

14/09/12 08:49:02 INFO mapred.JobClient: Bytes Written=48

14/09/12 08:49:02 INFO mapred.JobClient: FileSystemCounters

14/09/12 08:49:02 INFO mapred.JobClient: FILE_BYTES_READ=104

14/09/12 08:49:02 INFO mapred.JobClient: HDFS_BYTES_READ=265

14/09/12 08:49:02 INFO mapred.JobClient: FILE_BYTES_WRITTEN=177680

14/09/12 08:49:02 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=48

14/09/12 08:49:02 INFO mapred.JobClient: File Input Format Counters

14/09/12 08:49:02 INFO mapred.JobClient: Bytes Read=45

14/09/12 08:49:02 INFO mapred.JobClient: Map-Reduce Framework

14/09/12 08:49:02 INFO mapred.JobClient: Map output materialized bytes=110

14/09/12 08:49:02 INFO mapred.JobClient: Map input records=2

14/09/12 08:49:02 INFO mapred.JobClient: Reduce shuffle bytes=110

14/09/12 08:49:02 INFO mapred.JobClient: Spilled Records=18

14/09/12 08:49:02 INFO mapred.JobClient: Map output bytes=80

14/09/12 08:49:02 INFO mapred.JobClient: Total committed heap usage (bytes)=248127488

14/09/12 08:49:02 INFO mapred.JobClient: CPU time spent (ms)=8560

14/09/12 08:49:02 INFO mapred.JobClient: Combine input records=9

14/09/12 08:49:02 INFO mapred.JobClient: SPLIT_RAW_BYTES=220

14/09/12 08:49:02 INFO mapred.JobClient: Reduce input records=9

14/09/12 08:49:02 INFO mapred.JobClient: Reduce input groups=7

14/09/12 08:49:02 INFO mapred.JobClient: Combine output records=9

14/09/12 08:49:02 INFO mapred.JobClient: Physical memory (bytes) snapshot=322252800

14/09/12 08:49:02 INFO mapred.JobClient: Reduce output records=7

14/09/12 08:49:02 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1042149376

14/09/12 08:49:02 INFO mapred.JobClient: Map output records=9

hadoop@namenode:~/hadoop-1.2.1$ hadoop fs -cat out/*

Warning: $HADOOP_HOME is deprecated.

heheh 1

hello 2

it's 1

ll 1

the 2

think 1

why 1

cat: File does not exist: /user/hadoop/out/_logs

聲明:本網頁內容旨在傳播知識,若有侵權等問題請及時與本網聯系,我們將在第一時間刪除處理。TEL:177 7030 7066 E-MAIL:11247931@qq.com

文檔

hadoop增加新節點實踐

hadoop增加新節點實踐:之前已經有了namenode和datanode1,現在要新增節點datanode2 第一步:修改將要增加節點的主機名 hadoop@datanode1:~$ vim /etc/hostname datanode2 第二步:修改host文件 hadoop@datanode1:~$ vim /etc/hosts 192.16
推薦度:
標簽: 添加 增加 有了
  • 熱門焦點

最新推薦

猜你喜歡

熱門推薦

專題
Top
主站蜘蛛池模板: 国产在视频 | 国产小视频在线免费观看 | 夜夜骑首页 | 成人免费一级毛片在线播放视频 | 日韩欧美精品在线观看 | 欧美日韩激情 | 在线观看欧美日韩 | 亚洲欧美另类日本 | 欧美成人一级视频 | 日韩精品免费视频 | 国产精品免费观看 | 成人精品视频一区二区三区 | 亚洲欧洲日产国码一级毛片 | 91麻精品国产91久久久久 | 国产欧美第一页 | 欧美一区二区三区视频 | 亚洲视频在线免费观看 | 国产成人精品日本亚洲语音2 | 成人国产一区二区三区精品 | 91亚洲欧美综合高清在线 | 欧美精品久久久久久久久大尺度 | 99热成人精品国产免国语的 | 中文字幕免费在线播放 | 一道本一区二区三区 | 国产淫语对白在线 | 久久精品人 | 欧美在线视频观看 | 亚欧在线观看 | 中文字幕第一页亚洲 | 亚洲一区中文 | 国产ssss在线观看极品 | 国产在线91区精品 | 欧美日本一区 | 国产一区二区在线播放 | 精品国产一二三区在线影院 | 久久久久久国产精品免费 | 久久这里只有精品国产 | 国产精品资源在线播放 | 伊人精品成人久久综合欧美 | 国产精品久久久久久久久鸭 | 黑人群性xxx |