【运维】linkis安装dss保姆级教程与踩坑实践

文章目录一. 安装准备二. 创建用户三. 准备安装包四. 修改配置1. 修改config.sh2. 修改db.sh五、安装和使用1. 执行安装脚本2. 启动服务3. 查看验证是否成功六. 报错处理报错一The user is not logged in报错二dss接口报错报错三执行没有输出结果dss官方安装文档一. 安装准备需要的命令工具(安装时会自动检查并安装如果安装失败则需手动安装)telnet; tar; sed; dos2unix; mysql; yum; java; unzip; zip; expect需要安装的软件MySQL (5.5); JDK (1.8.0_141以上); NginxTips: 请确保已安装Links二. 创建用户假设部署用户是taiyi账号在所有需要部署的机器上创建部署用户用于安装sudo useradd taiyi改部署用户权限vi /etc/sudoerstaiyi ALL(ALL) NOPASSWD: NOPASSWD: ALL三. 准备安装包下载安装包DSS Release-1.1.1或者自己编译这里选择了一键安装的dss安装包安装起来比较方便。DSS 一键安装部署包的层级目录结构如下├── dss_install# 一键部署主目录├── bin# 用于一键安装以及一键启动 DSS├── conf# 一键部署的参数配置目录├── wedatasphere-dss-x.x.x-dist.tar.gz# DSS后端安装包├── wedatasphere-dss-web-x.x.x-dist.zip# DSS前端安装包四. 修改配置1. 修改config.sh### deploy user deployUsertaiyi ### DSS Web DSS_NGINX_IPdss安装所在ip或hostname DSS_WEB_PORT8085### DSS VERSION DSS_VERSION1.1.1LINKIS_HOME### EUREKA install information ########## 这些配置会产生dss的nginx配置注意别写默认的127.0.0.1### You can access it in your browser at the address below:http://${EUREKA_INSTALL_IP}:${EUREKA_PORT}### Microservices Service Registration Discovery Center LINKIS_EUREKA_INSTALL_IPlinkis EUREKA所在节点ip LINKIS_EUREKA_PORT20303#LINKIS_EUREKA_PREFER_IPtrue### Gateway install information LINKIS_GATEWAY_INSTALL_IPlinkis GATEWAY所在节点ip LINKIS_GATEWAY_PORT9001########## ## java applicationdefaultjvm memory export SERVER_HEAP_SIZE512M##sendemail配置只影响DSS工作流中发邮件功能 EMAIL_HOSTsmtp.163.com EMAIL_PORT25EMAIL_USERNAMExxx163.com EMAIL_PASSWORDxxxxx EMAIL_PROTOCOLsmtp ### Save the file path exported by the orchestrator service ORCHESTRATOR_FILE_PATH/home/taiyi/dss1.1.1/orchestrator-service ### Save DSS flow execution service log path EXECUTION_LOG_PATH/home/taiyi/dss1.1.1/logs2. 修改db.sh###forDSS-Server and Eventchecker APPCONN MYSQL_HOSTip MYSQL_PORT3306MYSQL_DBlinkis_dss MYSQL_USERuser MYSQL_PASSWORDpd #主要是配合scriptis一起使用如果不配置会默认尝试通过$HIVE_CONF_DIR 中的配置文件获取 # 或者不配置、或找不到配置则在scriptis使用不了hive HIVE_META_URLjdbc:mysql://hostname:3306/hivedb?createDatabaseIfNotExisttrueamp;useUnicodetrueamp;characterEncodingUTF-8amp;useSSLfalseHIVE_META_USERuserHIVE_META_PASSWORDpd五、安装和使用1. 执行安装脚本cd xx/dss_install/bin sh install.sh2. 启动服务在xx/dss_install/bin目录下执行启动服务脚本sh start-all.sh如果启动产生了错误信息可以查看具体报错原因。启动后各项微服务都会进行通信检测如果有异常则可以帮助用户定位异常日志和原因。3. 查看验证是否成功用户可以在Linkis的Eureka界面查看 DSS 后台各微服务的启动情况默认情况下DSS有7个微服务用户可以使用谷歌浏览器访问以下前端地址http://DSS_NGINX_IP:DSS_WEB_PORT 启动日志会打印此访问地址在xx/dss_install/conf/config.sh中也配置了此地址。六. 报错处理报错一The user is not logged in解决方式手动添加令牌日志分析2023-07-2620:52:04.149[INFO][main]o.a.l.h.d.DWSHttpClient(110)[org$apache$linkis$httpclient$AbstractHttpClient$$addAttempt$1]-The user is not logged in,please log in first,you can set a retry2023-07-2620:52:04.153[ERROR][main]c.w.w.d.c.u.DSSExceptionUtils(42)[lambda$handling$0]-execute failed,reason:org.apache.linkis.httpclient.exception.HttpClientRetryException:errCode:10900,desc:The user is not logged in,please log in first,you can set a retry,message:这里是dds向links的gateway接口请求令牌{method:/api/rest_j/v1/bml/upload,status:-1,message:Token Authentication Failed, token: BML-AUTHtokenUser: taiyi, reason: errCode: 15201 ,desc: Token is not valid or stale(token 令牌无效或已过期)! ,ip: bi50v.poc.bjmd.qihoo.net ,port: 9001 ,serviceKind: linkis-mg-gateway,data:{}},ip:bi50v.poc.bjmd.qihoo.net,port:9002,serviceKind:dss-framework-project-server at org.apache.linkis.httpclient.AbstractHttpClient.org$apache$linkis$httpclient$AbstractHttpClient$$addAttempt$1(AbstractHttpClient.scala:115)~[linkis-httpclient-1.1.1.jar:1.1.1]at org.apache.linkis.httpclient.AbstractHttpClient.execute(AbstractHttpClient.scala:123)~[linkis-httpclient-1.1.1.jar:1.1.1]at org.apache.linkis.httpclient.AbstractHttpClient.execute(AbstractHttpClient.scala:92)~[linkis-httpclient-1.1.1.jar:1.1.1]at org.apache.linkis.bml.client.impl.HttpBmlClient.uploadResource(HttpBmlClient.scala:313)~[linkis-bml-client-1.1.1.jar:1.1.1]at com.webank.wedatasphere.dss.framework.appconn.service.impl.AppConnResourceServiceImpl.upload(AppConnResourceServiceImpl.java:121)~[dss-appconn-framework-1.1.1.jar:?]at通过查看日志堆栈基本可以定位到dss调用links的gateway接口发送令牌以便能成功与linkis连通但因为令牌的原因令牌无效或已过期导致dss无法通过linkis gateway的校验。所以我们需要找到符合逻辑的令牌。通过阅读相关linkis gateway代码知道令牌存在linkis_mg_gateway_auth_token表中而这些令牌是linkis安装时进行初始化的接着看dss提供的令牌是BML-AUTH这里显然没有那既然这样手动添加一个BML-AUTH令牌。重启linkis服务访问前端输入用户名密码进来了。报错二dss接口报错rg.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\n\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadEx ecutor.java:409)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\n\ tat java.lang.Thread.run(Thread.java:748)\nCaused by:org.apache.ibatis.exceptions.PersistenceException:\n### Error querying database.Cause:org.springframework.jdbc.CannotGetJdbcConnectionException:Failed to obtain JDBC Connection;nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException:wait millis6000,active0,maxActive20,creating0,createErrorCount98652\n### The error may exist in URL[jar:file:/home/taiyi/linkis1.3.2/install/lib/linkis-public-enhancements/linkis-ps-publicservice/linkis-metadata-1.3.2.jar!/mapper/common/HiveMetaDao.xml]\n### The error may involve org.apache.linkis.metadata.hive.dao.HiveMetaDao.getAllDbs\n### The error occurredwhileexecuting a query\n### Cause:org.springframework.jdbc.CannotGetJdbcConnectionException:Failed to obtain JDBC Connection;nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException:wait millis6000,active0,maxActive20,creating0,createErrorCount98652\n\tat org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)\n\tat通过观察报错信息getAllDbs所用的连接配置定位到linkis-ps-publicservice模块下的配置出现了问题如下修改配置vim conf/db.shvim conf/linkis-ps-publicservice.properties### Provide the DB information of Hive metadata database.### Attention!If there are special characters like,they need to be enclosed in quotation marks.HIVE_META_URLjdbc:mysql://hive_meta_hive_db:3306/hivedb?createDatabaseIfNotExisttrueamp;useUnicodetrueamp;characterEncodingUTF-8amp;useSSLfalse而错误的原因就是Attention! If there are special characters like “”, they need to be enclosed in quotation marks.报错三执行没有输出结果执行没有输出结果报错如下具体是输出结果的路径配置的的不对而输出结果路径是在安装时配置的如下在install脚本中安装后存在两个位置修改vim install/conf/linkis-env.shvim linkis-cg-entrance.properties重启linkis再执行Script