Python安装Scrapy爬虫工具
来源:互联网 发布:如何解析json数据 编辑:程序博客网 时间:2024/05/17 23:23
原文出自:http://www.cnblogs.com/txw1958/archive/2012/07/12/scrapy_installation_introduce.html感谢作者。
一、 Scrapy简介
Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
官方主页: http://www.scrapy.org/
二、 安装Python2.7
官方主页:http://www.python.org/
下载地址:http://www.python.org/ftp/python/2.7.3/python-2.7.3.msi
1) 安装python
安装目录:D:\Python27
2) 添加环境变量
略System Properties -> Advanced -> Environment Variables - >System Variables -> Path -> Edit
3) 验证环境变量
T:\>set PathPath=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;D:\Rational\common;D:\Rational\ClearCase\bin;D:\Python27;D:\Python27\ScriptsPATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH
4) 验证Python
T:\>pythonPython 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32Type "help", "copyright", "credits" or "license" for more information.>>> exit()T:\>
三、 安装Twisted
Twisted is an event-driven networking engine written in Python and licensed under the open source
1) 安装setuptools
Download, build, install, upgrade, and uninstall Python packages -- easily!
官方主页:http://pypi.python.org/pypi/setuptools
下载地址:http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11.win32-py2.7.exe
安装过程:略
2) 安装Zope.Interface
官方主页:http://pypi.python.org/pypi/zope.interface/
下载地址:http://pypi.python.org/packages/2.7/z/zope.interface/zope.interface-4.0.1-py2.7-win32.egg
安装过程:
T:\>d:D:\>cd D:\Python27\ScriptsD:\Python27\Scripts>easy_install.exe zope.interface-4.0.1-py2.7-win32.eggProcessing zope.interface-4.0.1-py2.7-win32.eggcreating d:\python27\lib\site-packages\zope.interface-4.0.1-py2.7-win32.eggExtracting zope.interface-4.0.1-py2.7-win32.egg to d:\python27\lib\site-packagesAdding zope.interface 4.0.1 to easy-install.pth fileInstalled d:\python27\lib\site-packages\zope.interface-4.0.1-py2.7-win32.eggProcessing dependencies for zope.interface==4.0.1Finished processing dependencies for zope.interface==4.0.1D:\Python27\Scripts>
验证安装:
D:\Python27\Scripts>pythonPython 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32Type "help", "copyright", "credits" or "license" for more information.>>> import zope.interface>>>
3) 安装Twisted
官方主页:http://twistedmatrix.com/trac/wiki/TwistedProject
下载地址:http://pypi.python.org/packages/2.7/T/Twisted/Twisted-12.1.0.win32-py2.7.msi
安装过程:略
四、 安装w3lib
官方主页:http://pypi.python.org/pypi/w3lib
下载地址: http://pypi.python.org/packages/source/w/w3lib/w3lib-1.2.tar.gz
解压过程:略
安装过程:
T:\w3lib-1.2>python setup.py installrunning installrunning buildrunning build_pycreating buildcreating build\libcreating build\lib\w3libcopying w3lib\encoding.py -> build\lib\w3libcopying w3lib\form.py -> build\lib\w3libcopying w3lib\html.py -> build\lib\w3libcopying w3lib\http.py -> build\lib\w3libcopying w3lib\url.py -> build\lib\w3libcopying w3lib\util.py -> build\lib\w3libcopying w3lib\__init__.py -> build\lib\w3librunning install_libcreating D:\Python27\Lib\site-packages\w3libcopying build\lib\w3lib\encoding.py -> D:\Python27\Lib\site-packages\w3libcopying build\lib\w3lib\form.py -> D:\Python27\Lib\site-packages\w3libcopying build\lib\w3lib\html.py -> D:\Python27\Lib\site-packages\w3libcopying build\lib\w3lib\http.py -> D:\Python27\Lib\site-packages\w3libcopying build\lib\w3lib\url.py -> D:\Python27\Lib\site-packages\w3libcopying build\lib\w3lib\util.py -> D:\Python27\Lib\site-packages\w3libcopying build\lib\w3lib\__init__.py -> D:\Python27\Lib\site-packages\w3libbyte-compiling D:\Python27\Lib\site-packages\w3lib\encoding.py to encoding.pycbyte-compiling D:\Python27\Lib\site-packages\w3lib\form.py to form.pycbyte-compiling D:\Python27\Lib\site-packages\w3lib\html.py to html.pycbyte-compiling D:\Python27\Lib\site-packages\w3lib\http.py to http.pycbyte-compiling D:\Python27\Lib\site-packages\w3lib\url.py to url.pycbyte-compiling D:\Python27\Lib\site-packages\w3lib\util.py to util.pycbyte-compiling D:\Python27\Lib\site-packages\w3lib\__init__.py to __init__.pycrunning install_egg_infoWriting D:\Python27\Lib\site-packages\w3lib-1.2-py2.7.egg-infoT:\w3lib-1.2>
验证安装:
T:\>pythonPython 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32Type "help", "copyright", "credits" or "license" for more information.>>> import w3lib>>>
五、 安装libxml2
官方主页:http://users.skynet.be/sbi/libxml-python/http://pypi.python.org/pypi/pyOpenSSL
下载地址:http://users.skynet.be/sbi/libxml-python/binaries/libxml2-python-2.7.7.win32-py2.7.exe
安装过程:略
验证安装:
T:\>pythonPython 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32Type "help", "copyright", "credits" or "license" for more information.>>> import libxml2>>>
六、 安装pyOpenSSL
官方主页:http://pypi.python.org/pypi/pyOpenSSL
下载地址:http://pypi.python.org/packages/2.7/p/pyOpenSSL/pyOpenSSL-0.13.winxp32-py2.7.msi
安装过程:略
验证安装:
T:\>pythonPython 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32Type "help", "copyright", "credits" or "license" for more information.>>> import OpenSSL>>>
七、 安装Scrapy
官方主页:http://scrapy.org/
下载地址:http://pypi.python.org/packages/source/S/Scrapy/Scrapy-0.14.4.tar.gz
解压过程:略
安装过程:
T:\Scrapy-0.14.4>python setup.py install……Installing easy_install-2.7-script.py script to D:\Python27\ScriptsInstalling easy_install-2.7.exe script to D:\Python27\ScriptsInstalling easy_install-2.7.exe.manifest script to D:\Python27\ScriptsUsing d:\python27\lib\site-packagesFinished processing dependencies for Scrapy==0.14.4T:\Scrapy-0.14.4>
Scrapy 0.14.4 - no active project
Usage:
scrapy <command> [options] [args]
Available commands:
fetch Fetch a URL using the Scrapy downloader
runspider Run a self-contained spider (without creating a project)
settings Get settings values
shell Interactive scraping console
startproject Create new project
version Print Scrapy version
view Open URL in browser, as seen by Scrapy
Use "scrapy <command> -h" to see more info about a command
T:\>
解决方案是下载Python的另一个安装工具PIP,安装PIP之后把C:\Python27\Scripts路径加入环境变量path,在控制台输入pip显示如下表示安装成功:
我们找到下载的pyOpenSSL文件位置,在那里打开cmd直接输入pip pyOpenSSL-0.14.tar.gz回车,接着就安装OK了。
然后用pip安装下Scrapy包,OK,搞定。
- Python安装Scrapy爬虫工具
- python爬虫+scrapy安装
- python爬虫scrapy安装
- Python爬虫:scrapy安装
- win7(win32)安装python 爬虫工具scrapy记录
- win7(win32)安装python 爬虫工具scrapy记录
- mac 下 python 爬虫工具 scrapy 的安装
- <scrapy>python 爬虫框架scrapy安装
- Python爬虫框架Scrapy安装
- Windows安装python爬虫Scrapy
- python爬虫框架scrapy安装
- MAC OS X 安装Scrapy爬虫工具
- 在安装Scrapy爬虫工具时sudo aptitude install python-dev无法安装
- Python爬虫框架Scrapy实战之安装
- 网络爬虫(蜘蛛)Scrapy,Python安装!
- ubuntu14.04安装python爬虫框架Scrapy
- windows-python爬虫之scrapy快速安装
- python爬虫框架Scrapy入门:安装
- Hadoop到底能做什么?怎么用hadoop?
- java 枚举类型enum 的使用
- 屏幕适配
- Thinkcpp 学习笔记 运算符重载
- Oracle数据库,实例,表空间,用户,表之间的关系简析
- Python安装Scrapy爬虫工具
- Cocos2d-X 3.4版-地图无限滚动与边缘检测《赵云要格斗》
- java 中的 '\0'
- block
- HDU 4281 Judges' response 状态压缩 01背包 MTSP
- 笛卡儿积的Java算法实现
- Codeforces Round #290 (Div. 1)B. Fox And Jumping
- linux 进程调度策略
- javascript对url进行encode的两种方式