scrapy splash 实现下滑加载

来源：互联网发布：小说网站源码带手机版编辑：程序博客网时间：2024/04/30 07:18

实现滚轴下拉加载页面的splash script(Lua 脚本)

方法1

function main(splash, args)  splash:set_viewport_size(1028, 10000)  splash:go(args.url)  local scroll_to = splash:jsfunc("window.scrollTo")  scroll_to(0, 2000)  splash:wait(5)  return {png=splash:png()}end

方法2

function main(splash, args)  splash:set_viewport_size(1028, 10000)  splash:go(args.url)  splash.scroll_position={0,2000}  splash:wait(5)  return {png=splash:png()}end

爬虫实现下滑加载

    def start_requests(self):        script = """                function main(splash)                    splash:set_viewport_size(1028, 10000)                    splash:go(splash.args.url)                    local scroll_to = splash:jsfunc("window.scrollTo")                    scroll_to(0, 2000)                    splash:wait(15)                    return {                        html = splash:html()                    }                end                """        for url in self.start_urls:            yield Request(url,callback=self.parse_info_index,meta = {                'dont_redirect': True,                'splash':{                    'args':{'lua_source':script,'images':0},                    'endpoint':'execute',                }            })

阅读全文

0 0