使用网页抓取技术获取自己World Community Grid任务信息的经历

在获取World Community Grid任务页面的时候经历了很多的周折。开始的时候，认为这个页面和其他的页面相似，直接Post登陆信息即可获取到登陆Cookie，但是此方法失败，远端会返回登陆错误的页面。然后就搁置了好长时间，直到发现IE9开发者工具提供了比较友好的网页抓包数据，不过和WireShark获得的数据差不多，只是过滤了和HTTP协议无关的数据，更容易查看。

在捕获登陆操作以后，可以发现浏览器向服务器发送了怎样的请求，同时也可以知道服务器给予了怎样的响应。

上图为跟踪到的登录时的POST请求，可以看到请求的真实地址为/j_security_check，从页面的js代码中也可以看到这一点。然后根据请求的内容将对应的键值对写入request.Header中即可。让我走最多弯路的就是Cookie的处理了，开始我将Cookie的内容赋值给request.Header["Cookie"]，结果始终无法登陆成功，经过一系列的尝试，最后想起来request是使用CookieContainer来存储发送的Cookie的，直接将捕获到的“__utma”等键值对写到CookieContainer对象中，再将此对象赋值给request.CookieContainer，就可以成功登陆并且获取JSESSIONID和访问认证信息等Cookie了，然后就可以访问对应的页面获取想要的内容了。

基本代码如下：

string url = "https://secure.worldcommunitygrid.org/j_security_check";string resulturl = "https://secure.worldcommunitygrid.org/ms/viewBoincResults.do?filterDevice=0&filterStatus=4&projectId=-1&sortBy=returnedTime&pageNum=1";HttpWebRequest request = null;HttpWebResponse response = null;Stream requestStream = null;CookieContainer cc = new CookieContainer();try{string formData = "j_username=zl860628&j_password=" + Constant.PassWord1;ASCIIEncoding encoding = new ASCIIEncoding();byte[] data = encoding.GetBytes(formData);//将请求时的Cookie加入cc.Add(new Cookie("__utma", "2464182.1748491084.1301203514.1301203514.1301234717.2", "/", "secure.worldcommunitygrid.org"));cc.Add(new Cookie("__utmz", "2464182.1301203514.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none)", "/", "secure.worldcommunitygrid.org"));cc.Add(new Cookie("__utmc", "2464182", "/", "secure.worldcommunitygrid.org"));cc.Add(new Cookie("__utmb", "2464182", "/", "secure.worldcommunitygrid.org"));request = (HttpWebRequest)WebRequest.Create(url);request.Method = "POST";request.ContentType = "application/x-www-form-urlencoded";request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13";request.Accept = "text/html,application/xhtml+xml,*/*";request.ContentLength = data.Length;request.CookieContainer = cc;request.KeepAlive = true;request.Timeout = 100 * 1000;request.Headers["Accept-Encoding"] = "gzip, deflate";request.Headers["Accept-Language"] = "en";request.Headers["Accept-Charset"] = "GB2312,utf-8";request.Headers["Cookie"] = "__utma=2464182.1748491084.1301203514.1301203514.1301234717.2; __utmz=2464182.1301203514.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); __utmc=2464182; __utmb=2464182";//request.Headers["Cookie"] = "__utmb=2464182";requestStream = request.GetRequestStream();requestStream.Write(data, 0, data.Length);response = (HttpWebResponse)request.GetResponse();response.Cookies = cc.GetCookies(request.RequestUri);//获取登陆后的Cookiecc.Add(response.Cookies);//使用得到的Cookie，获取任务结果页面string html = transportTool.GetAndGetHTML(resulturl, cc, Encoding.Default);}catch (WebException ex){}finally{if (response != null) response.Close();if (request != null) request.Abort();}