hadoop要点（二）

1、用./bin/spark-shell启动spark时遇到异常：java.net.BindException: Can't assign requested address: Service 'sparkDriver' failed after 16 retries!

解决方法：add export SPARK_LOCAL_IP="127.0.0.1" to spark-env.sh

2、java Kafka producer error:ERROR kafka.utils.Utils$ - fetching topic metadata for topics [Set(words_topic)] from broker [ArrayBuffer(id:0,host: xxxxxx,port:9092)] failed

解决方法：Set 'advertised.host.name' on server.properties of Kafka broker to server's realIP(same to producer's 'metadata.broker.list' property)

3、java.net.NoRouteToHostException: No route to host

解决方法：zookeeper的IP要配对

4、Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)   java.net.UnknownHostException: linux-pic4.site:

解决方法：add your hostname to /etc/hosts: 127.0.0.1 localhost linux-pic4.site

5、org.apache.spark.SparkException: A master URL must be set in your configuration

解决方法：SparkConf sparkConf = new SparkConf().setAppName("JavaDirectKafkaWordCount").setMaster("local");

6、Failed to locate the winutils binary in the hadoop binary path

解决方法:先安装好hadoop

7、启动spark时： Failed to get database default, returning NoSuchObjectException

解决方法：1)Copy winutils.exe from here(https://github.com/steveloughran/winutils/tree/master/hadoop-2.6.0/bin) to some folder say, C:\Hadoop\bin. Set HADOOP_HOME to C:\Hadoop.2）Open admin command prompt. Run C:\Hadoop\bin\winutils.exe chmod 777 /tmp/hive

8、org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true.

解决方法：Use this constructor JavaStreamingContext(sparkContext: JavaSparkContext, batchDuration: Duration)  替代  new JavaStreamingContext(sparkConf, Durations.seconds(5));

9、Reconnect due to socket error: java.nio.channels.ClosedChannelException

解决方法：kafka服务器broker ip写对

10、java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute

解决方法：tranformation最后一步产生的那个RDD必须有相应Action操作，例如massages.print()等

11、经验：spark中数据写入ElasticSearch的操作必须在action中以RDD为单位执行

12、 Problem binding to [0.0.0.0:50010] java.net.BindException: Address already in use;

解决方法：master和slave配置成同一个IP导致的，要配成不同IP

13、CALL TO LOCALHOST/127.0.0.1:9000

解决方法：host配置正确，/etc/sysconfig/network    /etc/hosts    /etc/sysconfig/network-scripts/ifcfg-eth0

13、打开namenode:50070页面，Datanode Infomation只显示一个节点

解决方法：SSH配置错误导致，主机名一定要严格匹配，重新配置ssh免密码登录

14、经验：搭建集群时要首先配置好主机名，并重启机器让配置的主机名生效

15、INFO hdfs.DFSClient: Exception in createBlockOutputStream  java.net.NoRouteToHostException: No route to host

解决方法：如果主从节点能相互ping通，那就关掉防火墙 service iptables stop

16、经验：不要随意格式化HDFS，这会带来数据版本不一致等诸多问题，格式化前要清空数据文件夹

17、namenode1: ssh: connect to host namenode1 port 22: Connection refused

解决方法：sshd被关闭或没安装导致，which sshd检查是否安装，若已经安装，则sshd restart，并ssh 本机hostname，检查是否连接成功

18、Log aggregation has not completed or is not enabled.

解决方法：在yarn-site.xml中增加相应配置，以支持日志聚合

19、failed to launch org.apache.spark.deploy.history.History Server full log in

解决方法：正确配置spark-defaults.xml,spark-en.sh中SPARK_HISTORY_OPTS属性

20、Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

解决方法：yarn-lient模式出现的异常，暂时无解

21、hadoop的文件不能下载以及YARN中Tracking UI不能访问历史日志

解决方法：windows系统不能解析域名所致，把hosts文件hostname复制到windows的hosts中

22、经验：HDFS文件路径写法为：hdfs://master:9000/文件路径，这里的master是namenode的hostname,9000是hdfs端口号。

23、Yarn JobHistory Error: Failed redirect for container

解决方法：将 <value>http://<LOG_SERVER_HOSTNAME>:19888/jobhistory/logs</value>  配置到yarn-site.xml中，重启yarn和JobHistoryServer

24、通过hadoop UI访问hdfs文件夹时，出现提示 Permission denied: user=dr.who

解决方法：namonode节点终端执行：hdfs dfs -chmod -R 755 /

25、经验：Spark的Driver只有在Action时才会收到结果
26、经验：Spark需要全局聚合变量时应当使用累加器（Accumulator）
27、经验：Kafka以topic与consumer group划分关系，一个topic的消息会被订阅它的消费者组全部消费，如果希望某个consumer使用topic的全部消息，可将该组只设一个消费者，每个组的消费者数目不能大于topic的partition总数，否则多出的consumer将无消可费

28、java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;

解决方法：统一ES版本，尽量避免直接在spark中创建ES client

29、eturned Bad Request(400) - failed to parse;Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes; Bailing out..

解决方法：写入ES的数据格式纠正

30、java.util.concurrent.TimeoutException: Cannot receive any reply in 120 seconds

解决方法：确保所有节点之间能够免密码登录

31、集群模式下，spark无法向elasticsearch写入数据

解决方法：采用这种写入方式（带上es配置的Map参数）results.foreachRDD(javaRDD -> {JavaEsSpark.saveToEs(javaRDD, esSchema, cfg);return null;});

32、经验：所有自定义类要实现serializable接口，否则在集群中无法生效
33、经验：resources资源文件读取要在Spark Driver端进行，以局部变量方式传给闭包函数

34、通过nio读取资源文件时，java.nio.file.FileSystemNotFoundException  at com.sun.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:171)

解决方法：打成jar包后URI发生变化所致，形如jar:file:/C:/path/to/my/project.jar!/my-folder，要采用以下解析方式，

final Map<String, String> env = new HashMap<>();
final String[] array = uri.toString().split("!");
final FileSystem fs = FileSystems.newFileSystem(URI.create(array[0]), env);
final Path path = fs.getPath(array[1]);
35、经验：DStream流转化只产生临时流对象，如果要继续使用，需要一个引用指向该临时流对象
36、经验：提交到yarn cluster的作业不能直接print到控制台，要用log4j输出到日志文件中

37、java.io.NotSerializableException: org.apache.log4j.Logger

解决方法：序列化类中不能包含不可序列化对象，you have to prevent logger instance from default serializabtion process, either make it transient or static. Making it static final is preferred option due to many reason because if you make it transient than after deserialization logger instance will be null and any logger.debug() call will result in NullPointerException in Java because neither constructor not instance initializer block is called during deserialization. By making it static and final you ensure that its thread-safe and all instance of Customer class can share same logger instance, By the way this error is also one of the reason Why Logger should be declared static and final in Java program. 

38、log4j:WARN Unsupported encoding

解决方法：1.把UTF改成小写utf-8    2.设置编码那行有空格

39、MapperParsingException[Malformed content, must start with an object

解决方法：采用接口JavaEsSpark.saveJsonToEs，因为saveToEs只能处理对象不能处理字符串

40、 ERROR ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application

解决方法：资源不能分配过大,或者没有把.setMaster("local[*]")去掉

41、WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)

解决方法：配置文件broker编号要写对，命令中的IP写真实IP

42、 User class threw exception: org.apache.spark.SparkException: org.apache.spark.SparkException: Couldn't find leaders for Set([mywaf,7], [mywaf,1])

解决方法：正确配置kafka，并重新创建topic

43、在ES界面发现有节点shard分片不显示

解决方法：该节点磁盘容量不足，清理磁盘增加容量

44、The method updateStateByKey(Function2<List<String>,Optional<S>,Optional<S>>, int) in the type JavaPairDStream<String,String> is not applicable for the arguments (Function2<List<String>,Optional<String>,Optional<String>>, int)

解决方法：Spark use com.google.common.base.Optional not jdk default package java.util.Optional

45、NativeCrc32.nativeComputeChunkedSumsByteArray

解决方法：配置eclipse的hadoop-home，bin和system32文件夹中加入64位的2.6版本的hadoop.dll

46、经验：Spark Streaming包含三种计算模式：nonstate 、stateful 、window

47、Yarn的RM单点故障

解决方法：通过三节点zookeeper集群和yarn-site.xml配置文件完成Yarn HA

48、经验：kafka可通过配置文件使用自带的zookeeper集群
49、经验：Spark一切操作归根结底是对RDD的操作

50、如何保证kafka消息队列的强有序

解决方法：把需要强有序的topic只设置一个partition