容器化搭建可以看前面的搭建hadoop,我的3个节点是在3个容器内搭建的分别为master,slave1,slave2此后我直接说节点一.spark的搭建1.将需要的包导入master节点并且解压[rootlocalhost ~]# docker cp spark-3.1.1-bin-hadoop3.2.tgz master:/root[rootmaster ~]# tar -zxvf spark-3.1.1-bin-hadoop3.2.tgz -C /opt/module/2.配置环境变量(1)添加如下几行[rootmaster module]# vim /etc/profileexportSPARK_HOME/opt/module/spark-3.1.1-bin-hadoop3.2exportPATH$PATH:$SPARK_HOME/bin(2)使环境变量生效[rootmaster module]# source /etc/profile(3)验证是否成功[rootmaster module]# spark-submit --versionWelcome to ____ __ / __/__ ___ _____/ /__ _\\/ _\/ _ / __/ _/ /___/ .__/\_,_/_/ /_/\_\version3.1.1 /_/ Using Scala version2.12.10, Java HotSpot(TM)64-Bit Server VM,1.8.0_371 Branch HEAD Compiled by user ubuntu on2021-02-22T01:33:19Z Revision 1d550c4e90275ab418b9161925049239227f3dc9 Url https://github.com/apache/spark Type--helpformoreinformation.3.配置相关环境[rootmaster hadoop]# cd /opt/module/spark-3.1.1-bin-hadoop3.2/conf/[rootmaster conf]# cp spark-env.sh.template spark-env.sh[rootmaster conf]# cp workers.template workers[rootmaster conf]# vim spark-env.sh #添加如下几行exportJAVA_HOME/opt/module/jdk1.8.0_371exportHADOOP_HOME/opt/module/hadoop-3.2.1exportYARN_CONF_DIR/opt/module/hadoop-3.2.1/etc/hadoopexportHADOOP_CONF_DIR/opt/module/hadoop-3.2.1/etc/hadoop[rootmaster conf]# vim workers #添加3行,删除原来的localhostmaster slave1 slave24.把master节点的传给其他两个节点[rootmaster conf]# scp -r /etc/profile slave1:/etc/profile[rootmaster conf]# scp -r /etc/profile slave2:/etc/profile[rootmaster conf]# scp -r /opt/module/spark-3.1.1-bin-hadoop3.2/ slave1:/opt/module/[rootmaster conf]# scp -r /opt/module/spark-3.1.1-bin-hadoop3.2/ slave2:/opt/module/5.测试[rootmaster spark-3.1.1-bin-hadoop3.2]# ./bin/run-example SparkPi 10Assignments Map()2025-10-2017:58:46,196 INFO scheduler.TaskSetManager: Finished task5.0instage0.0(TID5)in47ms on master(executor driver)(6/10)2025-10-2017:58:46,198 INFO executor.Executor: Running task7.0instage0.0(TID7)2025-10-2017:58:46,211 INFO executor.Executor: Finished task6.0instage0.0(TID6).957bytes result sent to driver2025-10-2017:58:46,213 INFO scheduler.TaskSetManager: Starting task8.0instage0.0(TID8)(master, executor driver, partition8, PROCESS_LOCAL,4578bytes)taskResourceAssignments Map()2025-10-2017:58:46,214 INFO scheduler.TaskSetManager: Finished task6.0instage0.0(TID6)in36ms on master(executor driver)(7/10)2025-10-2017:58:46,217 INFO executor.Executor: Running task8.0instage0.0(TID8)2025-10-2017:58:46,233 INFO executor.Executor: Finished task7.0instage0.0(TID7).957bytes result sent to driver2025-10-2017:58:46,238 INFO scheduler.TaskSetManager: Starting task9.0instage0.0(TID9)(master, executor driver, partition9, PROCESS_LOCAL,4578bytes)taskResourceAssignments Map()2025-10-2017:58:46,239 INFO scheduler.TaskSetManager: Finished task7.0instage0.0(TID7)in44ms on master(executor driver)(8/10)2025-10-2017:58:46,239 INFO executor.Executor: Running task9.0instage0.0(TID9)2025-10-2017:58:46,255 INFO executor.Executor: Finished task8.0instage0.0(TID8).957bytes result sent to driver2025-10-2017:58:46,263 INFO scheduler.TaskSetManager: Finished task8.0instage0.0(TID8)in51ms on master(executor driver)(9/10)2025-10-2017:58:46,273 INFO executor.Executor: Finished task9.0instage0.0(TID9).957bytes result sent to driver2025-10-2017:58:46,274 INFO scheduler.TaskSetManager: Finished task9.0instage0.0(TID9)in37ms on master(executor driver)(10/10)2025-10-2017:58:46,276 INFO scheduler.TaskSchedulerImpl: Removed TaskSet0.0, whose tasks have all completed, from pool2025-10-2017:58:46,285 INFO scheduler.DAGScheduler: ResultStage0(reduce at SparkPi.scala:38)finishedin1.230s2025-10-2017:58:46,292 INFO scheduler.DAGScheduler: Job0is finished. Cancelling potential speculative or zombie tasksforthis job2025-10-2017:58:46,292 INFO scheduler.TaskSchedulerImpl: Killing all running tasksinstage0: Stage finished2025-10-2017:58:46,296 INFO scheduler.DAGScheduler: Job0finished: reduce at SparkPi.scala:38, took1.313123s Pi is roughly3.1379711379711382025-10-2017:58:46,315 INFO server.AbstractConnector: Stopped Spark53cdecf6{HTTP/1.1,(http/1.1)}{0.0.0.0:4040}2025-10-2017:58:46,321 INFO ui.SparkUI: Stopped Spark web UI at http://master:40402025-10-2017:58:46,345 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!2025-10-2017:58:46,390 INFO memory.MemoryStore: MemoryStore cleared2025-10-2017:58:46,391 INFO storage.BlockManager: BlockManager stopped2025-10-2017:58:46,407 INFO storage.BlockManagerMaster: BlockManagerMaster stopped2025-10-2017:58:46,411 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:OutputCommitCoordinator stopped!2025-10-2017:58:46,444 INFO spark.SparkContext: Successfully stopped SparkContext2025-10-2017:58:46,449 INFO util.ShutdownHookManager: Shutdown hook called2025-10-2017:58:46,449 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-46db54ca-ca2c-4114-b9fb-58f0fc3a62b42025-10-2017:58:46,452 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-97095aab-a35e-4bb4-9edb-9177e4b23cb7输出如下说明没问题二.Hbase的搭建1.将需要的包导入master节点并且解压[rootlocalhost ~]# docker cp hbase-2.2.3-bin.tar.gz master:/root[rootmaster ~]# tar -zxvf hbase-2.2.3-bin.tar.gz -C /opt/module/[rootmaster ~]# cd /opt/module/[rootmaster module]# mv hbase-2.2.3 hbase2.配置环境变量[rootmaster ~]# vim /etc/profileexportHBASE_HOME/opt/module/hbaseexportPATH$PATH:$HBASE_HOME/bin[rootmaster module]# source /etc/profile[rootmaster module]# hbase versionSLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.slf4j.impl.Log4jLoggerFactory]HBase2.2.3 Source code repository git://hao-OptiPlex-7050/home/hao/open_source/hbaserevision6a830d87542b766bd3dc4cfdee28655f62de3974 Compiled by hao on2020? 01?10? ???18:27:51 CST Fromsourcewith checksum 097925184b85f6995e20da5462b10f3f3.配置相关环境[rootmaster conf]# vim hbase-env.shexportJAVA_HOME/opt/module/jdk1.8.0_371exportHBASE_MANAGES_ZKfalse[rootmaster conf]# vim hbase-site.xmlconfigurationpropertynamehbase.rootdir/namevaluehdfs://master:9000/hbase/value/propertypropertynamehbase.cluster.distributed/namevaluetrue/value/propertypropertynamehbase.zookeeper.quorum/namevaluemaster,slave1,slave2/value/propertypropertynamehbase.unsafe.stream.capability.enforce/namevaluefalse/value/property/configuration[rootmaster conf]# vim regionserversmaster slave1 slave2[rootmaster conf]# cp /opt/module/hadoop-3.2.1/etc/hadoop/core-site.xml /opt/module/hbase/conf/[rootmaster conf]# cp /opt/module/hadoop-3.2.1/etc/hadoop/hdfs-site.xml /opt/module/hbase/conf/4把master节点的传给其他两个节点跟前面一样这里就不写了之后都是5.启动[rootmaster conf]# start-hbase.shSLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.slf4j.impl.Log4jLoggerFactory]running master, logging to /opt/module/hbase/logs/hbase-root-master-master.out slave2: running regionserver, logging to /opt/module/hbase/bin/../logs/hbase-root-regionserver-slave2.out slave1: running regionserver, logging to /opt/module/hbase/bin/../logs/hbase-root-regionserver-slave1.out master: running regionserver, logging to /opt/module/hbase/bin/../logs/hbase-root-regionserver-master.out[rootmaster conf]# hbase shell # 启动 HBase 交互式命令行工具SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.slf4j.impl.Log4jLoggerFactory]HBase Shell Usehelpto get list of supported commands. Useexitto quit this interactive shell. For Reference, please visit: http://hbase.apache.org/2.0/book.html#shellVersion2.2.3, r6a830d87542b766bd3dc4cfdee28655f62de3974,2020年 01月10日 星期五18:27:51 CST Took0.0034seconds hbase(main):001:0三.hudi,maven的搭建1.将需要的包导入master节点并且解压[rootlocalhost ~]# docker cp apache-maven-3.6.3-bin.tar.gz master:/root[rootlocalhost ~]# docker cp hudi-0.11.0.src.tgz master:/root[rootmaster ~]# tar -zxvf apache-maven-3.6.3-bin.tar.gz -C /opt/module/[rootmaster ~]# tar -zxvf hudi-0.11.0.src.tgz -C /opt/module/[rootmaster modulel# mv apache-maven-3.6.3/ maven2.配置环境变量[rootmaster ~]# vim /etc/profileexportMAVEN_HOME/opt/module/mavenexportPATH$PATH:$MAVEN_HOME/bin[rootmaster module]# source /etc/profile[rootmaster ~]# mvn -vApache Maven3.6.3(cecedd343002696d0abb50b32b541b8a6ba2883f)Maven home: /opt/module/maven Java version:1.8.0_371, vendor: Oracle Corporation, runtime: /opt/module/jdk1.8.0_371/jre Default locale: en_US, platform encoding: ANSI_X3.4-1968 OS name:linux, version:3.10.0-1160.el7.x86_64, arch:amd64, family:unix3.配置相关环境[rootmaster ~]# cd /opt/module/maven/conf/[rootmaster conf]# vim settings.xml/locallocalRepository/path/to/local/repo/localRepository有需要在这里改仓库位置 /mirrorsmirroridnexus-aliyun/idmirrorOfcentral/mirrorOfnameNexus aliyun/nameurlhttp://maven.aliyun.com/nexus/content/groups/public/url/mirror[rootmaster conf]# cd /opt/module/hudi-0.11.0/[rootmaster hudi-0.11.0]# vim pom.xml/repoidnexus-aliyun/idnamenexus-aliyun/nameurlhttp://maven.aliyun.com/nexus/content/groups/public/url/hadoophadoop.version3.2.1/hadoop.versionhive.version3.1.2/hive.version[rootmaster hudi-0.11.0]# cd hudi-common/src/main/java/org/apache/hudi/common/table/log/block/[rootmaster block]# vim HoodieParquetDataBlock.java/try try(FSDataOutputStream outputStreamnew FSDataOutputStream(baos,satas:null)){4.启动[rootmaster block]# spark-shell \--jars/opt/software/hudi-spark3.1-bundle_2.12-0.12.0.jar\--confspark.serializerorg.apache.spark.serializer.KryoSerializer\--confspark.sql.extensionsorg.apache.spark.sql.hudi.HoodieSparkSessionExtension2025-10-2020:14:29,222 INFO repl.Main: Created Spark session with Hive support Spark context Web UI available at http://master:4040 Spark context available assc(masterlocal[*], appidlocal-1760991268524). Spark session available asspark.Welcome to ____ __ / __/__ ___ _____/ /__ _\\/ _\/ _ / __/ _/ /___/ .__/\_,_/_/ /_/\_\version3.1.1 /_/ Using Scala version2.12.10(Java HotSpot(TM)64-Bit Server VM, Java1.8.0_371)Typeinexpressions to have them evaluated. Type :helpformoreinformation. scala四.flink的搭建1.将需要的包导入master节点并且解压[rootlocalhost ~]# docker cp flink-1.14.0-bin-scala_2.12.tgz master:/root[rootmaster ~]# tar -zxvf flink-1.14.0-bin-scala_2.12.tgz -C /opt/module/[rootmaster ~]# cd /opt/modulerootmaster module]#:mv flink-1.14.0 flink2.配置环境变量[rootmaster ~]# vim /etc/profileexportFLINK_HOME/opt/module/flinkexportPATH$PATH:$FLINK_HOME/binexportHADOOP_CLASSPATH$(hadoop classpath)[rootmaster module]# source /etc/profile[rootmaster ~]# flink --versionSLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found bindingin[jar:file:/opt/module/flink/lib/log4j-slf4j-impl-2.14.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found bindingin[jar:file:/opt/module/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is oftype[org.apache.logging.slf4j.Log4jLoggerFactory]Version:1.14.0, Commit ID: 460b3863.配置相关环境[rootmaster ~]# cd /opt/module/flink/conf/[rootmaster conf]# vim flink-conf.yamlclassloader.check-leaked-classloader:falsejobmanager.rpc.address: master[rootmaster conf]# vim mastersmaster:8081[rootmaster conf]# vim workersmaster[rootmaster conf]# vim zoo.cfgserver.1master:2888:38884把master节点的传给其他两个节点5.测试[rootmaster conf]# flink run -m yarn-cluster -p 2 -yjm 2G -ytm 2G $FLINK_HOME/examples/batch/WordCount.jar(rather,1)(regard,1)(remember,1)(resolution,1)(respect,1)(rub,1)(sins,1)(sleep,5)(slings,1)(so,1)(soft,1)(spurns,1)(suffer,1)(sweat,1)(take,1)(that,7)(their,1)(there,2)(thought,1)(time,1)(to,15)(under,1)(undiscover,1)(we,4)(weary,1)(when,2)(whips,1)(who,2)(whose,1)(will,1)(wish,1)(would,2)(wrong,1)(you,1)
企业数字化 ERP 产品动态
相关推荐
PSP与Switch游戏上安卓:Tyranor-Next外置模拟器跳转(PPSSPP/Eden)使用指南 【免费下载链接】Tyranor-Next 多引擎视觉小说(Galgame)聚合启动器,面向 Android 平台。内置 Kirikiri / ONScripter / Tyrano / Artemis / Siglus / FVP 六套引擎运行环境,并支持 RenPy、RPG Maker RGSS 外置 APK 引擎模块与 PSP… · 2026/11/7 15:38:10
(最新)基于Springboot高校实验室预约系统【附源码+文档】 /* MD / 富文本中的 .toc(含博客园搬家等嵌套结构);.toc-box 在侧栏,不受影响 */#content_views .toc,/* 编辑器常在目录前后插入空 p(:empty 仍占 20px),一并去掉避免顶空隙 */#content_views.markdown_views > p:empty:has(+ .toc),#content_views.markdown_views … · 2026/11/19 9:03:02
怎么证明策略能赚钱?Zipline 量化交易框架从零上手完整教程 怎么证明策略能赚钱?Zipline 量化交易框架从零上手完整教程 【免费下载链接】zipline Zipline, a Pythonic Algorithmic Trading Library 项目地址: https://gitcode.com/gh_mirrors/zi/zipline
策略想法听起来不错,可怎么证明它真能赚钱… · 2026/11/12 17:29:16
黄石低压电工证办理:工地太忙根本没时间复习考试?官方报名入口了解一下 黄石低压电工证办理:工地太忙根本没时间复习考试?官方报名入口了解一下 工地太忙根本没时间复习考试?你不是一个人在战斗。黄石低压电工证办理这件事,很多人都是被催着去办,结果一拖就拖了半年,甚至几年。说实话,这事儿真不是小事,但也不用太紧张,关… · 2026/11/19 9:02:58
苏州哪里考电工证快与2026河南最新政策的关联 苏州哪里考电工证快与2026河南最新政策的关联 你是不是正为了找工作发愁?手里没个电工证,连面试资格都没?2026年河南最新政策已经明确,特种作业操作证将成为建筑、制造等行业的准入门槛,没有证,连工地都进不去。而苏州作为长三角重要城市,电工… · 2026/11/19 9:02:58
厦门电工证复审体检表怎么查?电子证书能异地复审吗 厦门电工证复审体检表怎么查?电子证书能异地复审吗 之前考的证在外省能不能转过来?这个问题,我见过太多人问。其实,厦门电工证复审体检表并不难搞,关键是你有没有搞懂电子证书的查询与使用。今天就从实际案例出发,帮你理清楚流程和注意事项。… · 2026/11/19 9:02:58
进网作业电工证挂靠官方回应有效期多久?刚上岗电工必看! 进网作业电工证挂靠官方回应有效期多久?刚上岗电工必看! 你还在为进网作业电工证挂靠问题头疼?找工作被卡在证书上,眼看着同行都上手干活,你却连面试机会都没?官方回应来了!河南省应急管理厅明确规定,进网作业电工证挂靠不仅关系到你能不能上岗,还直… · 2026/11/19 9:02:58
金华特种工考证地点怎么找?证书过期了不知道怎么复审延期 金华特种工考证地点怎么找?证书过期了不知道怎么复审延期 证书过期了不知道怎么复审延期?你不是一个人,很多人在特种作业岗位上打拼多年,却对证书的含金量和管理流程一知半解。特别是像金华这样的地区,特种工考证地点分散,复审流程不透明,很多人一不小… · 2026/11/19 9:02:58
低压电工作业禁止标志怎么报名?别让培训费打水漂 低压电工作业禁止标志怎么报名?别让培训费打水漂 怕考不过白交培训费?低压电工作业禁止标志是很多人报考低压电工作业证前必须搞清楚的关键点,直接影响你能不能顺利上手干活,也决定你能不能拿到证书。别急,这篇文章手把手教你从怎么报名到考试全流程,避… · 2026/11/19 9:01:58
长春初级电工证怎么考?通过率揭秘全在这里 长春初级电工证怎么考?通过率揭秘全在这里 之前考的证在外省能不能转过来?这几乎是每个想从事电工工作的人都会问的问题。特别是那些在外地打拼多年、手里握着一张“外地电工证”的人,到了长春想继续干这一行,第一反应就是“我的证能用吗?”。今天我们就… · 2026/11/18 8:19:55
国内电工证报考条件与考试时间 低压电工证怎么考?学历不高也能报名,官方报名入口全在这 你是不是经常听说“电工证是入行门槛”,但又担心自己学历不够、经验不足,根本报不了名?别急,今天就带你从零了解【国内电工证】的报考流程、考试内容,还有怎么在 官方报名入口… · 2026/11/18 8:19:53
电工作业时如何断电?电子证书到底能不能全国通用? 电工作业时如何断电?电子证书到底能不能全国通用? 实操考试心里没底怕挂科,特别是电工作业时如何断电这一操作,不仅关系到你能不能拿到证书,更关系到以后能不能顺利上岗。很多人觉得这不过就是“断个电”,但其实这里面的门道可不少,稍有不慎就可能出错… · 2026/11/18 14:38:14
姑苏区哪里考电工证好考前押题怎么找 姑苏区哪里考电工证好考前押题怎么找 工地太忙根本没时间复习考试,电工证考前还总找不到靠谱的 考前押题 资源,真让人头疼。如果你也是在苏州姑苏区从事建筑或电力行业的,那这些建议你一定要看完。 为什么姑苏区考电工证的人这么多?… · 2026/11/19 0:00:12
罗定能考电工证吗?外省证转过来流程全解析+郑州报考避坑指南 罗定能考电工证吗?外省证转过来流程全解析+郑州报考避坑指南 之前考的证在外省能不能转过来?这几乎是每个特种作业证书持有者都会遇到的问题,特别是像你这样已经在外地工作多年,想回老家罗定发展或者重新规划职业路径的人。别急,今天就给你一条清晰的路… · 2026/11/19 0:00:12
罗定能考电工证吗 电工证含金量到底有多少? 罗定能考电工证吗 电工证含金量到底有多少? 之前考的证在外省能不能转过来,这几乎是每个准备考电工证的朋友最先想到的问题。特别是在罗定这样的地方,很多人的电工证都是在外地考的,现在想在当地继续使用,但不知道怎么办。实际上,电工证的 含金量… · 2026/11/19 0:00:12
双极步进电机控制实战:DRV8818PWPR与PIC18F86K22驱动调试 /* MD / 富文本中的 .toc(含博客园搬家等嵌套结构);.toc-box 在侧栏,不受影响 */#content_views .toc,/* 编辑器常在目录前后插入空 p(:empty 仍占 20px),一并去掉避免顶空隙 */#content_views.markdown_views > p:empty:has(+ .toc),#content_views.markdown_views … · 2026/11/19 6:33:25
STM32F207ZG + DRV8818PWPR 步进电机驱动实战解析 /* MD / 富文本中的 .toc(含博客园搬家等嵌套结构);.toc-box 在侧栏,不受影响 */#content_views .toc,/* 编辑器常在目录前后插入空 p(:empty 仍占 20px),一并去掉避免顶空隙 */#content_views.markdown_views > p:empty:has(+ .toc),#content_views.markdown_views … · 2026/11/19 6:33:32
DRV8818PWPR+dsPIC30F4013工业级双极步进电机控制方案 /* MD / 富文本中的 .toc(含博客园搬家等嵌套结构);.toc-box 在侧栏,不受影响 */#content_views .toc,/* 编辑器常在目录前后插入空 p(:empty 仍占 20px),一并去掉避免顶空隙 */#content_views.markdown_views > p:empty:has(+ .toc),#content_views.markdown_views … · 2026/11/19 0:22:38