【hive】Install hive using mysql as hive metadata service

文章目录一. Requirements二. Installing Hive from a Stable Release三. Running Hive四. Running Hive CLI五.Running HiveServer2 and Beeline1. 下载安装mysql2. 下载mysql驱动3. 配置hive-site.xml4. 初始化元数据库5. 通过beeline进行连接一. RequirementsUsers are strongly advised to start moving to Java 1.8 (see HIVE-8607).Hadoop 2.x (preferred), 本文实际上使用的是hadoop3.0.3The instructions in this document are applicable to Linux and Mac.二. Installing Hive from a Stable Release下载页面Hive ReleasesIndex of /hive解压tar-xzvf hive-x.y.z.tar.gz设置HIVE_HOME 环境变量/etc/profile$ cd hive-x.y.z $ export HIVE_HOME{{pwd}}$ export PATH$HIVE_HOME/bin:$PATH三. Running HiveHive uses Hadoop, so:设置hadoop环境变量创建hdfs目录In addition, you must use below HDFS commands to create /tmp and /user/hive/warehouse (aka hive.metastore.warehouse.dir) and set them chmod gw before you can create a table in Hive.$ $HADOOP_HOME/bin/hadoop fs-mkdir/tmp $ $HADOOP_HOME/bin/hadoop fs-mkdir-p/user/hive/warehouse $ $HADOOP_HOME/bin/hadoop fs-chmod gw/tmp $ $HADOOP_HOME/bin/hadoop fs-chmod gw/user/hive/warehouse四. Running Hive CLIhive Logging initialized using configuration in jar:file:/home/taiyi/apache-hive-3.1.3-bin/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async:true Hive-on-MR is deprecated in Hive2and may not be available in the future versions.Consider using a different executionengine(i.e.spark,tez)or using Hive1.X releases.hive因为没有初始化元数据实际上还不能使用五.Running HiveServer2 and BeelineStarting from Hive 2.1, we need to run the schematool command below as an initialization step. For example, we can use “derby” as db type.注意实际作者操作的过程中没有使用derby初始化的元数据并不能使用接下来使用mysql作为hive的元数据库。1. 下载安装mysql安装、设置新用户、对新用户赋权等这里略2. 下载mysql驱动下载mysql驱动mysql-connector-java-8.0.11.jar上传到$HIVE_HOME/lib 文件夹下3. 配置hive-site.xmlhive-site.xml位于$HIVE_HOME/conf文件夹下。默认情况下并没有这个文件所有的参数配置在hive-default.xml.template中。因此第一次使用时需要手动创建这个文件。接下来配置元数据库configurationpropertynamehive.server2.enable.doAs/namevaluefalse/valuedescriptionSetting this property to true will have HiveServer2 execute Hive operations as the user making the calls to it.当设置为true时会话将以连接用户的权限来执行操作。会出现如下报错 当设置为false时HiveServer2会话将使用HiveServer2服务的运行用户的身份来执行查询。/description/propertypropertynamejavax.jdo.option.ConnectionURL/namevaluejdbc:mysql://hostname:3306/hivedb?allowPublicKeyRetrievaltrueamp;createDatabaseIfNotExisttrueamp;useUnicodetrueamp;characterEncodingUTF-8amp;useSSLfalse/valuedescriptionJDBC connect stringfora JDBC metastore allowPublicKeyRetrievaltrue用于允许在连接MySQL数据库时检索公钥。它的作用是允许连接到MySQL服务器时自动检索服务器的SSL公钥。/description/propertypropertynamejavax.jdo.option.ConnectionDriverName/namevaluecom.mysql.cj.jdbc.Driver/valuedescriptionDriver class namefora JDBC metastore 注意mysql的驱动类要和放入的驱动包一致/description/propertypropertynamejavax.jdo.option.ConnectionUserName/namevalueusername/valuedescriptionusername to use against metastore database/description/propertypropertynamejavax.jdo.option.ConnectionPassword/namevaluepassword/valuedescriptionpassword to use against metastore database/description/property/configurationhive.server2.enable.doAs相关报错Connecting to jdbc:hive2://localhost:1000023/07/3015:08:04[main]:WARN jdbc.HiveConnection:Failed to connect to localhost:10000Error:Could not open client transport with JDBC Uri:jdbc:hive2://localhost:10000:Failed to open new session:java.lang.RuntimeException:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):User:taiyi is not allowed to impersonateanonymous(state08S01,code0)allowPublicKeyRetrievaltrue可预防的报错Public Key Retrieval is not allowed但要注意AllowPublicKeyRetrievalTrue 可能会导致恶意的代理通过中间人攻击(MITM)获取到明文密码所以默认是关闭的必须显式开启4. 初始化元数据库schematool-dbType mysql-initSchema5. 通过beeline进行连接HiveServer2 (introduced in Hive 0.11) has its own CLI called Beeline. HiveCLI is now deprecated in favor of Beeline, as it lacks the multi-user, security, and other capabilities of HiveServer2.To run HiveServer2 and Beeline from shell:启动server nohup hive--service hiveserver2/tmp/hiveserver2.log21启动metastore nohup hive--service metastore/tmp/hivemeta.log21beeline-u jdbc:hive2://localhost:10000如果localhost连接不了则改为具体机器的hostname