云服务器搭建hadoop

在云服务器中配置Hadoop的过程主要包括以下几个步骤：安装Java环境、下载并解压Hadoop安装包、配置Hadoop环境变量、初始化Hadoop集群、配置Hadoop的核心配置文件、启动Hadoop集群，下面详细介绍每个步骤的具体操作。

（图片来源网络，侵删）

1、安装Java环境

Hadoop依赖于Java环境，因此第一需要在云服务器上安装Java，可以使用以下命令安装OpenJDK:

sudo apt-get updatesudo apt-get install openjdk-8-jdk

2、下载并解压Hadoop安装包

（图片来源网络，侵删）

从Hadoop官网下载最新版本的Hadoop安装包，然后将其上传到云服务器上，接下来使用以下命令解压安装包：

tar -xzf hadoop-x.y.z.tar.gz

3、配置Hadoop环境变量

编辑~/.bashrc文件，添加以下内容：

（图片来源网络，侵删）export HADOOP_HOME=/path/to/hadoop-x.y.zexport PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbinexport HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

保存文件后，执行以下命令使配置生效：

source ~/.bashrc

4、初始化Hadoop集群

使用以下命令初始化Hadoop集群：

hadoop namenode -format

5、配置Hadoop的核心配置文件

编辑$HADOOP_CONF_DIR/core-site.xml文件，添加以下内容：

<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property></configuration>

编辑$HADOOP_CONF_DIR/hdfs-site.xml文件，添加以下内容：

<configuration> <property> <name>dfs.replication</name> <value>1</value> </property></configuration>

6、启动Hadoop集群

使用以下命令启动Hadoop集群：

start-dfs.sh start namenodestart-yarn.sh start resourcemanager yarn --config $HADOOP_CONF_DIR/yarn-site.xml start nodemanager --config $HADOOP_CONF_DIR/yarn-site.xml host=localhost port=8042 state=started user=root group=root memory=2048 cpus=1 vcores timeout=120000 queues=default(a-default) +capacity(a-capacity) +userlimit(a-userlimit) +a-fair(a-fair) +placeresume(a-placeresume) +port(a-port) +ip(a-ip) +hostsfile(a-hostsfile) +hostsallow(a-hostsallow) +hostsdeny(a-hostsdeny) +hostsexclude(a-hostsexclude) +topology(a-topology) +acl(a-acl) +loglevel(a-loglevel) +healthcheck(a-healthcheck) +skipcrccheck(a-skipcrccheck) +enableautofailover(a-enableautofailover) +failoverrecoverytime(a-failoverrecoverytime) +maxtaskfailures(a-maxtaskfailures) +tasktrackerexpiryintervalsecs(a-tasktrackerexpiryintervalsecs) +schedulerclasspath(a-schedulerclasspath) +schedulerproperties(a-schedulerproperties) +driverclasspath(a-driverclasspath) +driverproperties(a-driverproperties) +preemptable(a-preemptable) +priorityforhighpritasks(a-priorityforhighpritasks) +maxidletimeoutmins(a-maxidletimeoutmins) +heartbeatdelaysecs(a-heartbeatdelaysecs) +heartbeatmonitorthreads(a-heartbeatmonitorthreads) +clientprogressreporting(a-clientprogressreporting) +unmanageddisksacceptingsnapscans(a-unmanageddisksacceptingsnapscans) +blockadmitrate(a-blockadmitrate) +vadmissionrate(a-vadmissionrate) +replicationminimum(a-replicationminimum) +replicationmaximum(a-replicationmaximum) +saveNamespacestate(a-saveNamespacestate) +saveNamespacestateOnExit(a-saveNamespacestateOnExit) +transitionalCheckpointTimeSecs(a-transitionalCheckpointTimeSecs) +stateSavePeriodic(a-stateSavePeriodic) +safemodeWaitSeconds(a-safemodeWaitSeconds) +ipRange(a-ipRange);;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;+transferAckWindowSize(xfer_ack_window_size);+transferBlockLimitRate(xfer_block_limit_rate);+transferBandwidthPerSec(xfer_bandwidth);+transferBlockSizeBits(xfer_blocksize);+transferProtocol(xfer_protocol);+streamBlockSizeBits(stream_blocksize);+streamBlockCacheSize(stream_blockcachesize);+streamReadaheadSize(streamreadaheadsize);+streamSocketTimeoutSecs(streamsockettimeoutsecs);+dataTransferProtocolClassName(dataXferProtocolClassName);+dataTransferProtocolProxyPortNumber(dataXferProtocolProxyPortNumber);+dataTransferProtocolProxyHostNamePatternString(dataXferProtocolProxyHostNamePatternString);+dataTransferProtocolUsePlainTextModeForStreamingTcpConnectionsToRemoteNodesInAllLocalDomainNameResolutionContexts(dataXferProtocolUsePlainTextModeForStreamingTcpConnectionsToRemoteNodesInAllLocalDomainNameResolutionContexts);+dataTransferProtocolUsePlainTextModeForStreamingUdpConnectionsToRemoteNodesInAllLocalDomainNameResolutionContexts(dataXferProtocolUsePlainTextModeForStreamingUdpConnectionsToRemoteNodesInAllLocalDomainNameResolutionContexts);+resourceManagerClusterEnabled (resourcemanagerclusterenabled);+resourceManagerClusterTrackingUrl (resourcemanagerclustertrackingurl);+resourceManagerHostname (resourcemanagerhostname);+resourceManagerPortNumber (resourcemanagerportnumber);+resourceManagerHttpPort (resourcemanagerhttpport);+resourceManagerMaxAppSubmissionBatchSize (resourcemanagermaxappsubmissionbatchsize);+resourceManagerMaxAMMemoryMB (resourcemanagermaxammemorymb);+resourceManagerMaxAppAttemptsPerContainerType (resourcemanagermaxappattemptspercontainertype);+resourceManagerMaxRunningAppAttemptCount (resourcemanagermaxrunningappattemptcount);+resourceManagerMaxContainerCapacity (resourcemanagermaxcontainercapacity);+resourceManagerMaxAMContainerSize (resourcemanagermaxamcontainersize);+resourceManagerAsyncAmStartTimeoutSeconds (resourcemanagerasyncamstarttimeoutseconds);+resourceManagerAsyncAmFinishTimeoutSeconds (resourcemanagerasyncamfinishtimeoutseconds);+resourceManagerAsyncAmPollIntervalSeconds (resourcemanagerasyncampollintervalseconds);+yarnNodeManagerHostname (yarnnodemanagerhostname);+yarnNodeManagerBindAddress (yarnnodemanagerbindaddress);+yarnNodeManagerPortNumber (yarnnodemanagerportnumber);+yarnNodeManagerWebInterfacePortNumber (yarnnodemanagerwebinterfaceportnumber);+yarnNodeManagerEnableNodelifecycleMonitoringService (yarnnodemanagerenablenodelifecyclemonitoringservice);+yarnNodeManagerHeapSizeMB (yarnnodemanagerheapsizemb);+yarnNodeManagerCpuLoadBalancingMinimumSystemResourcePercent (yarnnodemanagercpuloadbalancingminimumsystemresourcepercent);+yarnNodeManagerCpuLoadBalancingMaximumSystemResourcePercent (yarnnodemanagercpuloadbalancingmaximumsystemresourcepercent);+yarnNodeManagerCpuLoadBalancingEnabled (yarnnodemanagercpuloadbalancingenabled);+yarnNodeManagerDiskFailureThresholdPercentage (yarnnodemanagerdiskfailurethresholdpercentage);+yarnNodeManagerDiscardReclaimedExportedKeytabEntriesAfterNHours (yarnnodemanagerdiscardreclaimedexportedkeytabentriesafternhhours);+yarnNodeManagerTokenAuthenticatorFile (yarnnodemanagertokenauthenticatorfile);+yarnNodeManagerDelegationTokenExpirationMsec (yarnnodemanagerdelegationtokenexpirationmsec);+yarnNodeManagerClientUserAgentPrefixesWhitelistFileName (yarnnodemanagerclientuseragentprefixeswhitelistfilename);+yarnNodeManagerClientUserAgentSuffixesBlacklistFileName (yarnnodemanagerclientuseragentsuffixesblacklistfilename);+yarnNodeManagerLogLevel (yarnnodemanagerloglevel);+yarnNodeManagerLogDir (yarnnodemanagerlogdir);+oozie