Spark连接需Kerberos认证的HBase

来源：互联网发布：压缩空气管道计算软件编辑：程序博客网时间：2024/06/01 13:30

Prerequisite

krb5.conf 或 krb5.ini
xx.keytab
core-site.xml
hbase-core.xml

Codes

hBaseConfig.addResource("hbase-site.xml")hBaseConfig.addResource("core-site.xml")UserGroupInformation.setConfiguration(hBaseConfig);UserGroupInformation ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI("name@xyz.com", keyTab);UserGroupInformation.setLoginUser(ugi);ugi.doAs(new PrivilegedAction<Void>() {  public Void run() {    connection = ConnectionFactory.createConnection(conf);    //hbase operation    return null;  }});

Submit cmd

spark-submit --class com.myclass \--master yarn \--deploy-mode cluster \--num-executors 2 \--driver-memory 1G \--executor-memory 1G \--executor-cores 2 \--queue myqueue \--conf spark.app.name=MyAPP \--conf spark.streaming.kafka.maxRatePerPartition=100 \--conf spark.streaming.stopGracefullyOnShutDown=true \--conf spark.yarn.security.credentials.hbase.enable=true \--conf spark.driver.extraJavaOptions='-Dlog4j.configuration=log4j -Djava.security.krb5.conf=krb5.conf -Dkeytab.file=mykeytab.keytab' \--conf spark.executor.extraJavaOptions='-Dlog4j.configuration=log4j -Djava.security.krb5.conf=krb5.conf -Dkeytab.file=mykeytab.keytab' \--conf spark.yarn.dist.archives=/path/core-site.xml,/path/hbase-site.xml,/path/krb5.conf,/path/mykeytab.keytab,/path/log4j.properties / \/path/myproject-1.0-SNAPSHOT.jar

TIPS

1. spark yarn 模式下提交时可用--files, --conf spark.yarn.dist.archives, --conf spark.yarn.dist.files 后接逗号分隔符的文件全路径将文件提交到每个运行节点.2. spark yarn 模式下测试情况来看只能用ugi.doAs形式访问HBase,否则会报错认证不通过.3. 如果使用submit命令中使用了--keytab和--principal,--conf spark.yarn.dist.archives提交的keytab文件会找不到. 详情参照spark官方文档security和yarn

References

Spark can’t connect to HBase using Kerberos in Cluster mode
HBase中的TTL应用
Spark读写Hbase的二种方式对比
Spark Streaming接收kafka数据，输出到HBase
Spark Streaming with HBase
SparkStreamingHBaseExample
Spark读写Hbase的二种方式对比
kafka-spark-streaming-to-hbase
New in Cloudera Labs: SparkOnHBase
Spark官档Security
Spark官档Yarn
Spark官档Configuration

阅读全文

0 0