StreamTokenizer

所谓token就是一个一个的单词，数字，符号，可以作为整体一次读入。而StreamTokennizer能够识别这些token.

import java.io.IOException;import java.io.StreamTokenizer;import java.io.StringReader;public class tokenNizer {/** * @param args * @throws IOException */public static void main(String[] args) throws IOException {// TODO Auto-generated method stubStreamTokenizer tokenizer = new StreamTokenizer(new StringReader("Mary had 1 little lamb..."));while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {if (tokenizer.ttype == StreamTokenizer.TT_WORD) {System.out.println(tokenizer.sval);} else if (tokenizer.ttype == StreamTokenizer.TT_NUMBER) {System.out.println(tokenizer.nval);} else if (tokenizer.ttype == StreamTokenizer.TT_EOL) {System.out.println();}}}}

输出：

Mary
had
1.0
little
lamb...

原文：http://tutorials.jenkov.com/java-io/streamtokenizer.html