hadoopHA配置文件-创新互联

fs.defaultFS hdfs://rongxinhadoop

创新互联公司坚持“要么做到，要么别承诺”的工作理念，服务领域包括：成都做网站、网站建设、企业官网、英文网站、手机端网站、网站推广等服务，满足客户于互联网时代的永胜网站设计、移动媒体设计的需求，帮助企业找到有效的互联网解决方案。努力成为您成熟可靠的网络建设合作伙伴！

这里的 mycluster为HA集群的逻辑名，与hdfs-site.xml中的dfs.nameservices配置一致 hadoop.tmp.dir /data/hadoop1/HAtmp3

这里的路径默认是NameNode、DataNode、JournalNode等存放数据的公共目录。用户也可单独指定每类数据的存储目录。这里目录结构需要自己先创建好

ha.zookeeper.quorum master:2181,slave1:2181,slave2:2181

这里是zk集群配置中各节点的地址和端口。

注意：数量一定是奇数而且和zoo.cfg中配置的一致

--------------------------------------------------------------------------------------------------

dfs.replication 2 配置副本数量

dfs.namenode.name.dir file:/data/hadoop1/HAname3 namenode元数据存储目录 dfs.datanode.data.dir file:/data/hadoop1/HAdata3 datanode数据存储目录 dfs.nameservices rongxinhadoop 指定HA命名服务，可随意起名， core-site.xml中fs.defaultFS配置需要引用它

dfs.ha.namenodes.rongxinhadoop nn1,nn2 指定集群下NameNode逻辑名

dfs.namenode.rpc-address.rongxinhadoop.nn1 master:9000

dfs.namenode.rpc-address.rongxinhadoop.nn2 slave1:9000

dfs.namenode.http-address.rongxinhadoop.nn1 master:50070

dfs.namenode.http-address.rongxinhadoop.nn2 slave1:50070

dfs.namenode.servicerpc-address.rongxinhadoop.nn1 master:53310 dfs.namenode.servicerpc-address.rongxinhadoop.nn2 slave1:53310

dfs.ha.automatic-failover.enabled.rongxinhadoop true 故障失败是否自动切换 dfs.namenode.shared.edits.dir qjournal://master:8485;slave1:8485;slave2:8485/rongxinhadoop 配置JournalNode，包含三部分：

1.qjournal 前缀表名协议；

2.然后就是三台部署JournalNode的主机host/ip:端口，三台机器之间用分号分隔；

3.最后的hadoop-journal是journalnode的命名空间，可以随意取名。 dfs.journalnode.edits.dir /data/hadoop1/HAjournal3/ journalnode的本地数据存放目录 dfs.client.failover.proxy.provider.rongxinhadoop

org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider 指定mycluster出故障时执行故障切换的类

dfs.ha.fencing.methods sshfence ssh的操作方式执行故障切换

dfs.ha.fencing.ssh.private-key-files /home/hadoop1/.ssh/id_rsa 如果使用ssh进行故障切换，使用ssh通信时用的密钥存储的位置

dfs.ha.fencing.ssh.connect-timeout 1000 dfs.namenode.handler.count 10

--------------------------------------------------------------------------------------------------

mapreduce.framework.name yarn mapreduce.jobhistory.address master:10020 mapreduce.jobhistory.webapp.address master:19888 mapreduce.jobhistory.intermediate-done-dir /data/hadoop1/mr_history/HAtmp3 Directory where history files are written by MapReduce jobs. mapreduce.jobhistory.done-dir /data/hadoop1/mr_history/HAdone3 Directory where history files are managed by the MR JobHistory Server.

--------------------------------------------------------------------------------------------------

yarn.resourcemanager.ha.enabled

true

yarn.resourcemanager.cluster-id

clusterrm

yarn.resourcemanager.ha.rm-ids

rm1,rm2

yarn.resourcemanager.hostname.rm1

master

yarn.resourcemanager.hostname.rm2

slave1

yarn.resourcemanager.recovery.enabled

true

yarn.resourcemanager.store.class

org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore

yarn.resourcemanager.zk-address

master:2181,slave1:2181,slave2:2181

yarn.nodemanager.aux-services

mapreduce_shuffle

yarn.nodemanager.aux-services.mapreduce.shuffle.class

org.apache.hadoop.mapred.ShuffleHandler

yarn.log-aggregation-enable

true

The hostname of the Timeline service web application.

yarn.timeline-service.hostname

master

Address for the Timeline server to start the RPC server.

yarn.timeline-service.address

master:10200

The http address of the Timeline service web application.

yarn.timeline-service.webapp.address

master:8188

The https address of the Timeline service web application.

yarn.timeline-service.webapp.https.address

master:8190

Handler thread count to serve the client RPC requests.

yarn.timeline-service.handler-thread-count

Enables cross-origin support (CORS) for web services where cross-origin web response headers are needed. For example, javascript making a web services request to the timeline server.

yarn.timeline-service.http-cross-origin.enabled

false

Comma separated list of origins that are allowed for web services needing cross-origin (CORS) support. Wildcards (*) and patterns allowed

yarn.timeline-service.http-cross-origin.allowed-origins

Comma separated list of methods that are allowed for web services needing cross-origin (CORS) support.

yarn.timeline-service.http-cross-origin.allowed-methods

GET,POST,HEAD

Comma separated list of headers that are allowed for web services needing cross-origin (CORS) support.

yarn.timeline-service.http-cross-origin.allowed-headers

X-Requested-With,Content-Type,Accept,Origin

The number of seconds a pre-flighted request can be cached for web services needing cross-origin (CORS) support.

yarn.timeline-service.http-cross-origin.max-age

1800

Indicate to clients whether Timeline service is enabled or not. If enabled, the TimelineClient library used by end-users will post entities and events to the Timeline server.

yarn.timeline-service.enabled

true

Store class name for timeline store.

yarn.timeline-service.store-class

org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore

Enable age off of timeline store data.

yarn.timeline-service.ttl-enable

true

Time to live for timeline store data in milliseconds.

yarn.timeline-service.ttl-ms

604800000

另外有需要云服务器可以了解下创新互联scvps.cn，海内外云服务器15元起步，三天无理由+7*72小时售后在线，公司持有idc许可证，提供“云服务器、裸金属服务器、高防服务器、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案，具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势，专为企业上云打造定制，能够满足用户丰富、多元化的应用场景需求。

网页名称：hadoopHA配置文件-创新互联
文章转载：http://azwzsj.com/article/csjsss.html

hadoopHA配置文件-创新互联

其他资讯