蘑菇街商品数据抓取

抓取蘑菇街商品数据主要用于研究，不用于商业用途。
so
1.获取分类列表页面，如 洁面
2.查看这个页面的源代码会发现主要的数据请求都是异步Ajax实现的，上面页面就是一个空的框架壳子，所以我们需要找到Ajax请求地址
3.
4.去页面代码中找到实现这个新的请求地址的代码

<!--    ajax 参数 param-name里填参数名 value填参数值-->    <input type="hidden" class="ajax_url" value="http://list.mogujie.com/search" />    <input type="hidden" class="ajax_param" param-name="action" value="magic" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="currentId" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="preId" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="cKey" value="pc-wall-v1" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="fcid" value="51899" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="page" value="1" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="sort" value="pop" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="fid" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="minPrice" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="maxPrice" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="itemMarks" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="userMarks" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="title" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="f" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="ad" value="2" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="stitle" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="tag" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="book" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="section" value="0" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="cpc_offset" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="showW" value="220" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="showH" value="330" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="width" value="220" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="height" value="330" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="ptpPartC" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="_mgjuuid" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="is_admin" value="" autocomplete="off"/>    <input type="hidden" class="ajax_param" param-name="userId" value="" autocomplete="off"/>

5.很好，我们只需要利用这些hidden元素生成url就行了

self.re_ajax_params = r"<input type=\"hidden\" class=\"ajax_param\" param-name=\"(.*?)\" value=\"(.*?)\""def get_ajax_url(self, content, item):        params = get_re_findall(content, self.re_ajax_params)        url = "http://list.mogujie.com/search?"        url += "&".join([p[0] + "=" + p[1] for p in params])

6.于是剩下的事情就很简单了，直接利用生成的url抓取json就行了。
代码片段

@gen.coroutine    def get_links_from_url(self, item, proxy_ip=None):        try:            request = httpclient.HTTPRequest(url=item[0], method="GET", request_timeout=300)            if proxy_ip is not None:                request.proxy_host, request.proxy_port = proxy_ip.split(":")                if not isinstance(request.proxy_port, int): request.proxy_port = int(request.proxy_port)            curl_httpclient.AsyncHTTPClient.configure(curl_httpclient.CurlAsyncHTTPClient)            response = yield curl_httpclient.AsyncHTTPClient().fetch(request)            html = response.body if isinstance(response.body, unicode) else response.body.decode("utf-8")            urls = []            #html            if item[1] == 0:                urls = self.get_ajax_url(html, item)            elif item[1] == 1:   # json                if item[2] == 1:  # first page                    urls = self.parse_next_page(html)                    for url in urls:                        self.db.insertItemWithKey(self.list_fetching_table,{"url":url}, {"p1":url[1],"p2":url[2]})                    self.parse_json_obj(html)                else:   # page 1..N                    self.parse_json_obj(html)            print urls            print('fetched %s' % (item[0]))            self.db.insertItemWithKey(self.list_fetched_table, {"url": item[0]},                                      {"url": item[0], "update_time": get_current_timestamp()})            self.db.remove(self.list_fetching_table, {"url": item[0]})        except Exception as e:            print('Exception: %s %s' % (e, item[0]))            raise gen.Return((False,[]))        raise gen.Return((True,urls))