kylin问题总结

一、点击加载hive表出现如下异常：

java.lang.NoClassDefFoundError: org/apache/hadoop/hive/cli/CliSessionState

java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/session/SessionState

解决：将hive lib文件夹下的lib拷贝到kylin lib文件夹下。

查看了下bin里的find-hive-dependency.sh，有设置hive_dependency 包含了所有hive lib下的jar包，但不知道为啥没起作用。所以只好拷贝操作了

在网上看到一个人说，修改bin目录下的kylin.sh

在HBASE_CLASSPATH_PREFIX把hive_dependency 加上。经测可用。

二、解决jar问题，hive表依然加载不出来（点击加载树，一直在转圈）

查看log，如：tail -f -n200 /opt/app/apache-kylin-2.0.0-bin/logs/kylin.log

发现没有异常，弄得我直挠头。尝试点了下reload table按钮，终于报出了异常，但是后台log依然没打出来，报出异常提示连接不上元数据。

hive不知道什么时候挂了，输入： hive --service metastore & 启动一下元数据。终于看到加载的hive表了

为了配置kylin，/etc/profile文件加入如下：

export JAVA_HOME=/opt/app/jdk1.8.0_131
PATH=$PATH:/$JAVA_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
JRE_HOME=$JAVA_HOME/jre
export HADOOP_HOME=/opt/app/hadoop-2.8.0
PATH=$PATH:$HADOOP_HOME/bin:$PATH
export HIVE_HOME=/opt/app/apache-hive-2.1.1-bin
export HCAT_HOME=$HIVE_HOME/hcatalog
export HIVE_CONF=$HIVE_HOME/conf
PATH=$PATH:$HIVE_HOME/bin:$PATH
export HBASE_HOME=/opt/app/hbase-1.3.1
PATH=$PATH:$HBASE_HOME/bin:$PATH
#export HIVE_CONF=/opt/app/apache-hive-2.1.1-bin/conf
#PATH=$PATH:$HIVE_HOME/bin:$PATH
export KYLIN_HOME=/opt/app/apache-kylin-2.0.0-bin
PATH=$PATH:$KYLIN_HOME/bin:$PATH
#export KYLIN_HOME=/opt/app/kylin/
# zookeeper env 本人zookeeper用了hbase自带的，此没有用到
export ZOOKEEPER_HOME=/opt/app/zookeeper-3.4.6
export PATH=$ZOOKEEPER_HOME/bin:$PATH

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

问题：将hadoop format，重新搭建环境，报异常： org.apache.hadoop.hbase.TableExistsException: kylin_metadata_user

解决：

1.如果是独立的zookeeper

在zookeeper的bin目录下有一个zkCli.sh ,运行进入；
进入后运行 rmr  /hbase , 然后quit退出；

2.如果是用hbase自带的zookeeper

用 hbase zkcli进入

进入后运行 rmr  /hbase , 然后quit退出；

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

问题：File does not exist: hdfs://localhost:9000/home/hadoop/data/mapred/staging/root1629090427/.staging/job_local1629090427_0006/libjars/hive-metastore-1.2.1.jar

是由于mapred-site.xml、yarn没有配置正确造成的。

mapred-site.xml

<configuration><property>    <name>mapreduce.reduce.java.opts</name>    <value>-Xms2000m -Xmx4600m</value></property><property>    <name>mapreduce.map.memory.mb</name>    <value>5120</value></property><property>    <name>mapreduce.reduce.input.buffer.percent</name>    <value>0.5</value></property> <property>   <name>mapreduce.reduce.memory.mb</name>   <value>2048</value> </property><property>    <name>mapred.tasktracker.reduce.tasks.maximum</name>    <value>2</value></property><property>    <name>mapreduce.framework.name</name>    <value>yarn</value></property><property>     <name>mapreduce.jobhistory.address</name>     <value>localhost:10020</value>  </property>  <property>     <name>yarn.app.mapreduce.am.staging-dir</name>     <value>/home/hadoop/data/mapred/staging</value>  </property>  <property>     <name>mapreduce.jobhistory.intermediate-done-dir</name>     <value>${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate</value>  </property>  <property>     <name>mapreduce.jobhistory.done-dir</name>     <value>${yarn.app.mapreduce.am.staging-dir}/history/done</value>  </property></configuration>

yarn-site.xml

<configuration><!-- Site specific YARN configuration properties -->    <property>    <name>yarn.nodemanager.aux-services</name>    <value>mapreduce_shuffle</value>    </property><property>    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>    <value>org.apache.hadoop.mapred.ShuffleHandler</value></property> <property>        <name>mapreduce.jobtracker.staging.root.dir</name>        <value>/home/hadoop/data/mapred/staging</value></property>  <property>   <name>yarn.app.mapreduce.am.staging-dir</name>    <value>/home/hadoop/data/mapred/staging</value>  </property>  <property>    <name>yarn.resourcemanager.hostname</name>    <value>localhost</value>  </property></configuration>

记得启动 /hadoop-2.8.0/sbin/mr-jobhistory-daemon.sh start historyserver

---------------------------------------------------------------------------

问题：build cube没有成功，得到如下log

Counters: 41File System CountersFILE: Number of bytes read=0FILE: Number of bytes written=310413FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=1700HDFS: Number of bytes written=0HDFS: Number of read operations=4HDFS: Number of large read operations=0HDFS: Number of write operations=0Job Counters Failed map tasks=1Failed reduce tasks=6Killed reduce tasks=3Launched map tasks=2Launched reduce tasks=9Other local map tasks=1Data-local map tasks=1Total time spent by all maps in occupied slots (ms)=10602Total time spent by all reduces in occupied slots (ms)=76818Total time spent by all map tasks (ms)=10602Total time spent by all reduce tasks (ms)=76818Total vcore-seconds taken by all map tasks=10602Total vcore-seconds taken by all reduce tasks=76818Total megabyte-seconds taken by all map tasks=10856448Total megabyte-seconds taken by all reduce tasks=78661632Map-Reduce FrameworkMap input records=0Map output records=3Map output bytes=42Map output materialized bytes=104Input split bytes=1570Combine input records=3Combine output records=3Spilled Records=3

Failed Shuffles=0Merged Map outputs=0GC time elapsed (ms)=168CPU time spent (ms)=3790Physical memory (bytes) snapshot=356413440Virtual memory (bytes) snapshot=2139308032Total committed heap usage (bytes)=195035136File Input Format Counters Bytes Read=0

在mapred-site.xml中加上如下配置：

1. <property>    <name>mapreduce.reduce.java.opts</name>    <value>-Xms2000m -Xmx4600m</value></property><property>    <name>mapreduce.map.memory.mb</name>    <value>5120</value></property><property>    <name>mapreduce.reduce.input.buffer.percent</name>    <value>0.5</value></property> <property>   <name>mapreduce.reduce.memory.mb</name>   <value>2048</value> </property>