Hadoop 程式編譯

Hadoop 程式編譯

 

1. 練習 0 : hello : 印出我們的 key value 為何
2. 練習 1 : Word Count 初級版
3. 練習 2 : Word Count 進階版

 

練習 0 : hello : 印出我們的 key value 為何¶

• 下載 nchc-example.jar

  $ wget http://trac.nchc.org.tw/cloud/raw-attachment/wiki/Hadoop_Lab4/nchc-example.jar

• 執行客製化的hadoop 程式

$ bin/hadoop jar nchc-example.jar

• 輸出結果

  ******************************************歡迎使用 NCHC.Hadoop 的運算功能指令：  Hadoop jar nchc-example-*.jar <功能>功能：  wordcount: 計算輸入資料夾內分別在每個檔案的字數統計  mwc:       統合計算所有輸入檔的字數統計  grep:      算出包指定字串的個數  nchcgrep:  整合來源檔內的每個字與其所有出現的所在列  hello:     印出內容並統計累進字數******************************************

• 使用hello

  $ bin/hadoop jar nchc-example.jar hello

• 使用提示：

  hello <inDir> <outDir> <m> <r>

• 範例：

$ bin/hadoop jar nchc-example.jar hello lab3_input lab4_out6 1 1

練習 1 : Word Count 初級版¶

• 上傳內容到hdfs內

$ cd /opt/hadoop$ mkdir lab4_input$ echo "I like NCHC Cloud Course." > lab4_input/input1$ echo "I like nchc Cloud Course, and we enjoy this course." > lab4_input/input2$ bin/hadoop fs -put lab4_input lab4_input$ bin/hadoop fs -ls lab4_input

• 下載 WordCount.java 並存到/opt/hadoop;

  $ wget http://secuse.nchc.org.tw/class/WordCount.java

• 運作程式

$ mkdir MyJava$ javac -classpath hadoop-*-core.jar -d MyJava WordCount.java$ jar -cvf wordcount.jar -C MyJava .$ bin/hadoop jar wordcount.jar WordCount lab4_input/ lab4_out1/$ bin/hadoop fs -cat lab4_out1/part-00000

• lab4_out1 執行結果

  Cloud 2Course, 1Course. 1I 2NCHC  1and 1course. 1enjoy 1like  2nchc  1this  1we  1

---

練習 2 : Word Count 進階版¶

$ echo "/." >pattern.txt && echo "/," >>pattern.txt$ bin/hadoop fs -put pattern.txt ./$ mkdir MyJava2

• 下載 WordCount2.java 並存到/opt/hadoop;

  $ wget http://trac.nchc.org.tw/cloud/raw-attachment/wiki/Hadoop_Lab4/WordCount2.java

$ javac -classpath hadoop-*-core.jar -d MyJava2 WordCount2.java$ jar -cvf wordcount2.jar -C MyJava2 .$ bin/hadoop jar wordcount2.jar WordCount2 lab4_input lab4_out2 -skip pattern.txt$ bin/hadoop fs -cat lab4_out2/part-00000

• lab4_out2 執行結果

  Cloud 2Course  2I 2NCHC  1and 1course  1enjoy 1like  2nchc  1this  1we  1

$ bin/hadoop jar wordcount2.jar WordCount2 -Dwordcount.case.sensitive=false lab4_input lab4_out3 -skip pattern.txt$ bin/hadoop fs -cat lab4_out3/part-00000

• lab4_out3 執行結果

  and 1cloud 2course  3enjoy 1i 2like  2nchc  2this  1we  1

Attachments

• WordCount2.java (4.2 kB) - added by waue 4 days ago.
• nchc-example.jar (23.2 kB) - added by waue 4 days ago.
• WordCount.java (2.6 kB) - added by waue 2 days ago.