【运维与安装】Hadoop 3.3.1 ：Setting up a Single Node Cluster.

最近公司搞国产化项目里用到的hadoop 2.10.1版本 不支持国产化hadoop 3.1.1 版本支持ARM架构的国产CPU。hadoop的版本修改势必会影响项目里对API、restfulAPI、flink with hadoop相关使用所以本地先搭建一个hadoop 3.1.1 测试一下版本升级对项目的影响。参考官网搭建一个本地的hadoop官网Hadoop: Setting up a Single Node Cluster.文章目录1. Required Software2. Download3. Prepare to Start the Hadoop Cluster4. HDFS on a Single Node4.1 configurationa. etc/hadoop/core-site.xml:b. etc/hadoop/hdfs-site.xml:Setup passphraseless ssh (省略)4.2. Executiona. Format the filesystem:b. Start NameNode daemon and DataNode daemon:c. Browse the web interface for the NameNode; by default it is available at:d. When you’re done, stop the daemons with:5. YARN on a Single Node5.1. Configure parameters as follows:5.2. Start ResourceManager daemon and NodeManager daemon:5.3. Browse the web interface for the ResourceManager; by default it is available at:5.4. When you’re done, stop the daemons with:1. Required Softwarejava 和 sshJava™ must be installed. Recommended Java versions are described at HadoopJavaVersionsssh must be installed and sshd must be running to use the Hadoop scripts thatmanage remote Hadoop daemonsif the optional start and stop scripts are to be used.Additionally, it is recommmended that pdsh also be installed for better ssh resource management.2. DownloadTo get a Hadoop distribution, download a recent stable release from one of the Apache Download Mirrors.3. Prepare to Start the Hadoop Cluster解压并在 hadoop-env.sh 设置java home。Unpack the downloaded Hadoop distribution. In the distribution, edit the file etc/hadoop/hadoop-env.sh to define some parameters as follows:#setto the root of your Java installationexportJAVA_HOME/Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home尝试命令$ bin/hadoopThis will display the usage documentation for the hadoop script.Now you are ready to start your Hadoop cluster in one of the three supported modes:我们这次关注单节点的安装Pseudo-Distributed Operation (伪集群模式)Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process.4. HDFS on a Single Node4.1 configurationa. etc/hadoop/core-site.xml:要配置一个数据目录用于存储数据否则可能hadoop默认选择的目录没有权限使用。org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:Directory/private/tmp/hadoop-lianggao/dfs/name is in an inconsistent state:storage directory doesnotexistorisnotaccessible.at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:392)at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:243)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1197)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:779)at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:677)at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:764)at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:1018)at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:991)at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1767)at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1832)configurationpropertynamefs.defaultFS/namevaluehdfs://localhost:9000/value/propertypropertynamehadoop.tmp.dir/namevalue/Users/lianggao/MyWorkSpace/002install/hadoop-3.3.1/hadoop_repo/value/property/configurationb. etc/hadoop/hdfs-site.xml:configurationpropertynamedfs.replication/namevalue1/value/property/configurationSetup passphraseless ssh (省略)如需要参考官网官网4.2. Executiona. Format the filesystem:$ bin/hdfs namenode-format看到这些信息格式化成功2022-08-1217:43:11,039INFO common.Storage:Storage directory/Users/lianggao/MyWorkSpace/002install/hadoop-3.3.1/hadoop_repo/dfs/name has been successfully formatted.2022-08-1217:43:11,069INFO namenode.FSImageFormatProtobuf:Saving image file/Users/lianggao/MyWorkSpace/002install/hadoop-3.3.1/hadoop_repo/dfs/name/current/fsimage.ckpt_0000000000000000000usingno compression2022-08-1217:43:11,200INFO namenode.FSImageFormatProtobuf:Image file/Users/lianggao/MyWorkSpace/002install/hadoop-3.3.1/hadoop_repo/dfs/name/current/fsimage.ckpt_0000000000000000000 of size403bytes saved in0seconds.如果格式化失败需要先删除nn的管理目录因为格式化的时候是创建了nn文件的管理目录 common.Storage: Storage directory /data/hadoopdata/name has been successfully formatted.b. Start NameNode daemon and DataNode daemon:$ sbin/start-dfs.shc. Browse the web interface for the NameNode; by default it is available at:NameNode - http://localhost:9870/d. When you’re done, stop the daemons with:$ sbin/stop-dfs.sh5. YARN on a Single Node5.1. Configure parameters as follows:etc/hadoop/mapred-site.xml:configurationpropertynamemapreduce.framework.name/namevalueyarn/value/propertypropertynamemapreduce.application.classpath/namevalue$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*/value/property/configurationetc/hadoop/yarn-site.xml:configurationpropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/propertypropertynameyarn.nodemanager.env-whitelist/namevalueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME/value/property/configuration5.2. Start ResourceManager daemon and NodeManager daemon:$ sbin/start-yarn.sh5.3. Browse the web interface for the ResourceManager; by default it is available at:ResourceManager - http://localhost:8088/5.4. When you’re done, stop the daemons with:$ sbin/stop-yarn.sh