基因数据处理68之avocado的配置文件默认无法从hdfs读取
来源:互联网 发布:js正则匹配网址 编辑:程序博客网 时间:2024/06/05 00:31
(1)配置文件设置为hdfs路劲有问题
val configFile = "hdfs://219.219.220.149:9000/xubo/avocado/avocado-sample-configs/basic.properties"
报错:
hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem/BWAMEMSparkAll/test$ ./GcdssCallVariant2.sh start:fqFile:hdfs://219.219.220.149:9000/xubo/avocado/artificial/artificial.realigned.samfaFile:hdfs://219.219.220.149:9000/xubo/avocado/artificial/artificial.faconfigFile:hdfs://219.219.220.149:9000/xubo/avocado/avocado-sample-configs/basic.propertiesoutput:hdfs://219.219.220.149:9000/xubo/avocado/test201605281620AvocadoZidai060505fqLoad.count:10faLoad.count:1faLoad:hdfs://219.219.220.149:9000/xubo/avocado/test201605281620AvocadoZidai060505tempPath:/tmp/config8081833845300900496tempFilePath:/tmp/config8081833845300900496/temp.propertiesException in thread "main" org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source hdfs://219.219.220.149:9000/xubo/avocado/avocado-sample-configs/basic.propertiesat org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:217)at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:197)at org.apache.commons.configuration.AbstractHierarchicalFileConfiguration.load(AbstractHierarchicalFileConfiguration.java:164)at org.apache.commons.configuration.AbstractHierarchicalFileConfiguration.<init>(AbstractHierarchicalFileConfiguration.java:91)at org.apache.commons.configuration.plist.PropertyListConfiguration.<init>(PropertyListConfiguration.java:159)at org.gcdss.cli.Gcdss.<init>(Gcdss.scala:84)at org.gcdss.cli.Gcdss$.apply(Gcdss.scala:46)at org.gcdss.cli.GcdssCallVariant$.main(GcdssCallVariant.scala:64)at org.gcdss.cli.GcdssCallVariant.main(GcdssCallVariant.scala)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
(2)将配置文件设置为本地路劲可以通过:
val configFile = "/home/hadoop/xubo/data/testTools/basic.properties"
记录:
hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem/BWAMEMSparkAll/test$ ./GcdssCallVariant2.sh start:fqFile:hdfs://219.219.220.149:9000/xubo/avocado/artificial/artificial.realigned.samfaFile:hdfs://219.219.220.149:9000/xubo/avocado/artificial/artificial.faconfigFile:/home/hadoop/xubo/data/testTools/basic.propertiesoutput:hdfs://219.219.220.149:9000/xubo/avocado/test201605281620AvocadoZidai060505fqLoad.count:10faLoad.count:1faLoad:hdfs://219.219.220.149:9000/xubo/avocado/test201605281620AvocadoZidai060505tempPath:/tmp/config5809810828492689651tempFilePath:/tmp/config5809810828492689651/temp.propertiesLoading reads in from hdfs://219.219.220.149:9000/xubo/avocado/artificial/artificial.realigned.samSLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementationSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.read parquet: *************end*************
参考
【1】https://github.com/xubo245/AdamLearning【2】https://github.com/bigdatagenomics/adam/ 【3】https://github.com/xubo245/SparkLearning【4】http://spark.apache.org【5】http://stackoverflow.com/questions/28166667/how-to-pass-d-parameter-or-environment-variable-to-spark-job 【6】http://stackoverflow.com/questions/28840438/how-to-override-sparks-log4j-properties-per-driver
研究成果:
【1】 [BIBM] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Chao Wang, and Xuehai Zhou, "Distributed Gene Clinical Decision Support System Based on Cloud Computing", in IEEE International Conference on Bioinformatics and Biomedicine. (BIBM 2017, CCF B)【2】 [IEEE CLOUD] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Xuehai Zhou. Efficient Distributed Smith-Waterman Algorithm Based on Apache Spark (CLOUD 2017, CCF-C).【3】 [CCGrid] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Jinhong Zhou, Xuehai Zhou. DSA: Scalable Distributed Sequence Alignment System Using SIMD Instructions. (CCGrid 2017, CCF-C).【4】more: https://github.com/xubo245/Publications
Help
If you have any questions or suggestions, please write it in the issue of this project or send an e-mail to me: xubo245@mail.ustc.edu.cnWechat: xu601450868QQ: 601450868
阅读全文
0 0
- 基因数据处理68之avocado的配置文件默认无法从hdfs读取
- 基因数据处理73之从HDFS读取fasta文件存为Adam的parquet文件
- 基因数据处理25之avocado安装
- 基因数据处理28之avocado运行
- 基因数据处理66之avocado集群运行
- 基因数据处理74之从HDFS读取vcf文件存为Adam的parquet文件(有问题)
- 基因数据处理75之从HDFS读取vcf文件存为Adam的parquet文件(成功)
- 基因数据处理30之avocado运行avocado-cli中的avocado问题1和2
- 基因数据处理76之从HDFS读取fasta并统计条数
- 基因数据处理29之avocado运行snap-basic有问题
- 基因数据处理32之Avocado运行记录(人造数据集)
- 基因数据处理33之Avocado运行记录(参考基因组)
- 基因数据处理31之avocado运行avocado-cli中的avocado问题3-变异识别找不到RecordGroupSample(null)
- 基因数据处理62之snap默认无法处理大于400bp的reads
- 基因数据处理85之adam-0.18.2无法读取0.14.0使用adamSave存储的数据
- 基因数据处理78之从vcf使用不同的方法读取结果不一样
- 基因数据处理80之disease的DataProcessing
- 基因数据处理77之从vcf文件中提取某条染色体的数据
- LeetCode27. Remove Element
- 基因数据处理64之bwamem处理500bp和1000bp的记录
- 基因数据处理65之bwa处理500bp和1000bp的记录
- 基因数据处理66之avocado集群运行
- 基因数据处理67之bwa建立索引时间
- 基因数据处理68之avocado的配置文件默认无法从hdfs读取
- Scrapy_part1_初识SCRAPY和安装
- 基因数据处理69之bowtie安装与使用
- ll -bash: ls: command not found
- 02爬虫---浏览器的模拟Headers属性
- 基因数据处理70之Picard安装没成功
- 基因数据处理71之GRCH38 的chr14提取
- LeetCode28. Implement strStr()
- 关于支持向量机(SVM)