c4 Internet Address - Some useful Program

SpamCheck

许多服务器要监视“垃圾邮件”，然后通知客户端它要访问的host是否是垃圾邮件。这种实时的黑洞列表要求尽可能快，并且负载很大，可能是百万级的。

解决这个问题要反应尽可能快，最好是有缓存，负载问题可以通过分布式服务器完成.可以用web server 来完成，SOAP,UDP,自定义协议等。实际上这种服务器可以DNS实现。

public static final String BLACKHOLE = "sbl.spamhaus.org";public static void main(String[] args) throws SocketException, UnknownHostException {for (String arg : args) {if (isSpammer(arg)) {System.out.println(arg + " is a known spammer.");} else {System.out.println(arg + " appears legitimate.");}}}private static boolean isSpammer(String arg) {try {InetAddress address = InetAddress.getByName(arg);byte[] quad = address.getAddress();String query = BLACKHOLE;for (byte octet : quad) {int unsignedByte = octet < 0 ? octet + 256 : octet;query = unsignedByte + "." + query;}System.out.println(InetAddress.getByName(query).getHostName());System.out.println(InetAddress.getByName(query).getHostAddress());return true;} catch (UnknownHostException e) {return false;}}

使用这种技术要注意对blackhole list和地址的维护。服务器被攻击，对所有请求都拒绝回应等问题要考虑到。

Processing Web Server Logfiles

205.160.186.76 unknown - [17/Jun/2013:22:53:58 -0500]                               "GET /bgs/greenbg.gif HTTP 1.0" 200 50

上面那条记录表示来着 205.160.186.76 的浏览器请求资源 /bgs/greenbg.gif ，并且成功请求到，资源大小是 50 bytes

public class Weblog {  public static void main(String[] args) {    try (FileInputStream fin =  new FileInputStream(args[0]);      Reader in = new InputStreamReader(fin);      BufferedReader bin = new BufferedReader(in);) {      for (String entry = bin.readLine();        entry != null;        entry = bin.readLine()) {        // separate out the IP address        int index = entry.indexOf(' ');        String ip = entry.substring(0, index);        String theRest = entry.substring(index);        // Ask DNS for the hostname and print it out        try {          InetAddress address = InetAddress.getByName(ip);          System.out.println(address.getHostName() + theRest);        } catch (UnknownHostException ex) {          System.err.println(entry);        }      }    } catch (IOException ex) {      System.out.println("Exception: " + ex);    }  }}

InetAddress会缓存结果，所以同样的ip地址，不会再次访问DNS。

但上面的程序可以改造一下，变得更快！因为上面的程序花了非常多的时间在“等待”DNS的反应结果。这个时候，用多线程正好解决该问题。一个线程读取log entry，读到的entry交给其他线程去执行。但要注意到，可能log entry有很多很多，那如果每条log entry都启动一个线程的话，那VM几下就会被干趴下，所以这里要用线程池。

public class LookupTask implements Callable<String> {  private String line;  public LookupTask(String line) {    this.line = line;  }  @Override  public String call() {    try {      // separate out the IP address      int index = line.indexOf(' ');      String address = line.substring(0, index);      String theRest = line.substring(index);      String hostname = InetAddress.getByName(address).getHostName();      return hostname + " " + theRest;    } catch (Exception ex) {      return line;    }  }}

// Requires Java 7 for try-with-resources and multi-catchpublic class PooledWeblog {  private final static int NUM_THREADS = 4;  public static void main(String[] args) throws IOException {    ExecutorService executor = Executors.newFixedThreadPool(NUM_THREADS);    Queue<LogEntry> results = new LinkedList<LogEntry>();    try (BufferedReader in = new BufferedReader(      new InputStreamReader(new FileInputStream(args[0]), "UTF-8"));) {      for (String entry = in.readLine(); entry != null; entry = in.readLine()) {        LookupTask task = new LookupTask(entry);        Future<String> future = executor.submit(task);        LogEntry result = new LogEntry(entry, future);        results.add(result);      }    }    // Start printing the results. This blocks each time a result isn't ready.    for (LogEntry result : results) {      try {        System.out.println(result.future.get());      } catch (InterruptedException | ExecutionException ex) {        System.out.println(result.original);      }    }    executor.shutdown();  }  private static class LogEntry {    String original;    Future<String> future;    LogEntry(String original, Future<String> future) {     this.original = original;     this.future = future;    }  }}

不完全科学的统计，上述方法比第一种方法要快10-50倍！

但上面的程序还有一个设计上的downside！logfile可能会是很大很大，那queue就会很大，程序就会消耗很多很多内存！避免这个问题方法可以是，将output工作放在一个单独的线程中，和input共享一个queue，早先处理的entry可以先打印出来，不必等所有entry都放到queue后再去output。但这个会引起另一个问题，你需要一个单独的signal来告知output已经完成了，因为queue为空并不能保证output已经完成，最简单的办法是count input 的条数和output的条数一致！