win7下 scrapy的安装
来源:互联网 发布:试验次数优化方法 编辑:程序博客网 时间:2024/04/30 21:51
学习自: http://scrapy-chs.readthedocs.io/zh_CN/0.24/intro/install.html
http://www.th7.cn/Program/Python/201505/447703.shtml
http://blog.csdn.net/mfcing/article/details/43449441
我的python已经是预先安装好的2.7版本的
原文中 pywin32的网址是:从 http://sourceforge.net/projects/pywin32/ 安装 pywin32
我的2.7 32位是这个网址
https://sourceforge.net/projects/pywin32/files/pywin32/Build%20220/
建议按我的这个链接来 我找过两次没找到 今天这次找了30分钟终于找到了下载链接。。。。。之前下载的都是Readme.txt
首先第一个给的链接是用pip来安装的,我们安装库可以有几种方式
1:python XXX.py install
2:pip
3:easy_install
目前我接触的是这几种,
上面链接中第一个链接是安装pywin32之后安装pip通过pip进行安装的。
后面的是自己下载文件,通过exe文件以及easy_install和 python自己安装的。
下面我贴出来通过自己下载文件安装的帖子之后在我出错的地方标注我的错误之处,这样看帖子的人可以少犯错误进行安装scrapy了
一、 Scrapy简介
Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
官方主页: http://www.scrapy.org/
二、 安装Python2.7
官方主页:http://www.python.org/
下载地址:http://www.python.org/ftp/python/2.7.3/python-2.7.3.msi
1) 安装Python
安装目录:D:\Python27
2) 添加环境变量
略System Properties -> Advanced -> Environment Variables - >System Variables -> Path -> Edit
3) 验证环境变量
T:\>set PathPath=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;D:\Rational\common;D:\Rational\ClearCase\bin;D:\Python27;D:\Python27\ScriptsPATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH
4) 验证Python
T:\>pythonPython 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32Type "help", "copyright", "credits" or "license" for more information.>>> exit()T:\>
三、 安装Twisted
Twisted is an event-driven networking engine written in Python and licensed under the open source
1) 安装setuptools
Download, build, install, upgrade, and uninstall Python packages -- easily!
官方主页:http://pypi.python.org/pypi/setuptools
下载地址:http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11.win32-py2.7.exe
安装过程:(这里是exe文件)
2) 安装Zope.Interface
官方主页:http://pypi.python.org/pypi/zope.interface/
下载地址:http://pypi.python.org/packages/2.7/z/zope.interface/zope.interface-4.0.1-py2.7-win32.egg
安装过程:
T:\>d:D:\>cd D:\Python27\ScriptsD:\Python27\Scripts>easy_install.exe zope.interface-4.0.1-py2.7-win32.eggProcessing zope.interface-4.0.1-py2.7-win32.eggcreating d:\python27\lib\site-packages\zope.interface-4.0.1-py2.7-win32.eggExtracting zope.interface-4.0.1-py2.7-win32.egg to d:\python27\lib\site-packagesAdding zope.interface 4.0.1 to easy-install.pth fileInstalled d:\python27\lib\site-packages\zope.interface-4.0.1-py2.7-win32.eggProcessing dependencies for zope.interface==4.0.1Finished processing dependencies for zope.interface==4.0.1D:\Python27\Scripts>
验证安装:
D:\Python27\Scripts>pythonPython 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32Type "help", "copyright", "credits" or "license" for more information.>>> import zope.interface>>>
(下面贴一段我的测试,正好上面链接的帖子里面也提到了,会有一个类型错误,不过可以使用库,那个错误貌似是easy_install里面的)
D:\Program Files\python27\Scripts>easy_install.exe zope.interface-4.0.1-py2.7-win32.egg
Processing zope.interface-4.0.1-py2.7-win32.egg
creating d:\program files\python27\lib\site-packages\zope.interface-4.0.1-py2.7-win32.egg
Extracting zope.interface-4.0.1-py2.7-win32.egg to d:\program files\python27\lib\site-packages
Adding zope.interface 4.0.1 to easy-install.pth file
Installed d:\program files\python27\lib\site-packages\zope.interface-4.0.1-py2.7-win32.egg
Traceback (most recent call last):
File "D:\Program Files\python27\Scripts\easy_install-script.py", line 8, in <module>
load_entry_point('setuptools==0.6c11', 'console_scripts', 'easy_install')()
File "d:\program files\python27\lib\site-packages\setuptools\command\easy_install.py", line 1712, in main with_ei_usage(lambda:
File "d:\program files\python27\lib\site-packages\setuptools\command\easy_inst
all.py", line 1700, in with_ei_usage
return f()
File "d:\program files\python27\lib\site-packages\setuptools\command\easy_inst
all.py", line 1716, in <lambda>
distclass=DistributionWithoutHelpCommands, **kw
File "D:\Program Files\python27\lib\distutils\core.py", line 151, in setup
dist.run_commands()
File "D:\Program Files\python27\lib\distutils\dist.py", line 953, in run_comma
nds
self.run_command(cmd)
File "D:\Program Files\python27\lib\distutils\dist.py", line 972, in run_comma
nd
cmd_obj.run()
File "d:\program files\python27\lib\site-packages\setuptools\command\easy_inst
all.py", line 211, in run
self.easy_install(spec, not self.no_deps)
File "d:\program files\python27\lib\site-packages\setuptools\command\easy_inst
all.py", line 427, in easy_install
return self.install_item(None, spec, tmpdir, deps, True)
File "d:\program files\python27\lib\site-packages\setuptools\command\easy_inst
all.py", line 478, in install_item
self.process_distribution(spec, dist, deps)
File "d:\program files\python27\lib\site-packages\setuptools\command\easy_inst
all.py", line 514, in process_distribution
distreq.project_name, distreq.specs, requirement.extras
TypeError: __init__() takes exactly 2 arguments (4 given)
D:\Program Files\python27\Scripts>python
Python 2.7.13 on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import zope.interface(这里没有错误就是可以使用的)
>>>
3) 安装Twisted
官方主页:http://twistedmatrix.com/trac/wiki/TwistedProject
下载地址:http://pypi.python.org/packages/2.7/T/Twisted/Twisted-12.1.0.win32-py2.7.msi
安装过程:(这里是exe文件)
四、 安装w3lib
官方主页:http://pypi.python.org/pypi/w3lib
下载地址: http://pypi.python.org/packages/source/w/w3lib/w3lib-1.2.tar.gz
解压过程:略
安装过程:
T:\w3lib-1.2>python setup.py installrunning installrunning buildrunning build_pycreating buildcreating build\libcreating build\lib\w3libcopying w3lib\encoding.py -> build\lib\w3libcopying w3lib\form.py -> build\lib\w3libcopying w3lib\html.py -> build\lib\w3libcopying w3lib\http.py -> build\lib\w3libcopying w3lib\url.py -> build\lib\w3libcopying w3lib\util.py -> build\lib\w3libcopying w3lib\__init__.py -> build\lib\w3librunning install_libcreating D:\Python27\Lib\site-packages\w3libcopying build\lib\w3lib\encoding.py -> D:\Python27\Lib\site-packages\w3libcopying build\lib\w3lib\form.py -> D:\Python27\Lib\site-packages\w3libcopying build\lib\w3lib\html.py -> D:\Python27\Lib\site-packages\w3libcopying build\lib\w3lib\http.py -> D:\Python27\Lib\site-packages\w3libcopying build\lib\w3lib\url.py -> D:\Python27\Lib\site-packages\w3libcopying build\lib\w3lib\util.py -> D:\Python27\Lib\site-packages\w3libcopying build\lib\w3lib\__init__.py -> D:\Python27\Lib\site-packages\w3libbyte-compiling D:\Python27\Lib\site-packages\w3lib\encoding.py to encoding.pycbyte-compiling D:\Python27\Lib\site-packages\w3lib\form.py to form.pycbyte-compiling D:\Python27\Lib\site-packages\w3lib\html.py to html.pycbyte-compiling D:\Python27\Lib\site-packages\w3lib\http.py to http.pycbyte-compiling D:\Python27\Lib\site-packages\w3lib\url.py to url.pycbyte-compiling D:\Python27\Lib\site-packages\w3lib\util.py to util.pycbyte-compiling D:\Python27\Lib\site-packages\w3lib\__init__.py to __init__.pycrunning install_egg_infoWriting D:\Python27\Lib\site-packages\w3lib-1.2-py2.7.egg-infoT:\w3lib-1.2>
验证安装:
T:\>pythonPython 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32Type "help", "copyright", "credits" or "license" for more information.>>> import w3lib>>>
五、 安装libxml2
官方主页:http://users.skynet.be/sbi/libxml-python/http://pypi.python.org/pypi/pyOpenSSL
下载地址:http://users.skynet.be/sbi/libxml-python/binaries/libxml2-python-2.7.7.win32-py2.7.exe
安装过程:略(这里我开了翻墙打开的)
验证安装:
T:\>pythonPython 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32Type "help", "copyright", "credits" or "license" for more information.>>> import libxml2>>>
六、 安装pyOpenSSL
官方主页:http://pypi.python.org/pypi/pyOpenSSL
下载地址:http://pypi.python.org/packages/2.7/p/pyOpenSSL/pyOpenSSL-0.13.winxp32-py2.7.msi
安装过程:(这里是exe文件)
验证安装:(下面是我的错误)
>>> import OpenSSL
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build\bdist.win32\egg\OpenSSL\__init__.py", line 8, in <module>
File "build\bdist.win32\egg\OpenSSL\rand.py", line 10, in <module>
ImportError: No module named six
之后去找six包 提供了python2与3的差异性。
下载地址:
https://pypi.python.org/pypi/six/1.6.1
安装过程,进入文件夹下 python setup.py install
验证安装: import six
仍然缺: No module named cryptography.hazmat.bindings.openssl.binding
https://github.com/MattTW/HoneyAlarmServer/issues/1
https://bugs.archlinux.org/task/39388#comment120549
两篇文章我看到的是后面的版本可以解决这个问题,之后我就直接安装scrapy了 import scrapy 是可以的。之后我用下面那个方法
pip下载地址。不过一般自带的:
https://pypi.python.org/pypi/pip
将D:\Program Files\python27\Scripts 加入环境变量为了 让你在命令行里面随意使用pip而不用必须在他的文件夹下。
这个步骤成功的话(也就是在命令行下无论在哪个文件夹下 都可以使用pip了 之后进入到下载安装包的位置 pip install pyOpenSSL-16.2.0.tar.gz)
我下载的是这个包,之后pip帮你把所有的支持包也下载了。完成时候是这样的
Successfully installed asn1crypto-0.21.1 cffi-1.9.1 cryptography-1.8.1 enum34-1.
1.6 idna-2.5 ipaddress-1.0.18 packaging-16.8 pycparser-2.17 pyparsing-2.2.0
可以测试下, import OpenSSL 没问题
pip install Scrapy-0.14.4.tar.gz
没问题和上一步骤一样的
之后就可以用了 scrapy
(不过我在深深的怀疑直接用pip scrapy的支持会不会都下载下来~~而且 还是linux安装简单。。。)
T:\>python
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32Type "help", "copyright", "credits" or "license" for more information.>>> import OpenSSL>>>
七、 安装Scrapy
官方主页:http://scrapy.org/
下载地址:http://pypi.python.org/packages/source/S/Scrapy/Scrapy-0.14.4.tar.gz
解压过程:略
安装过程:
T:\Scrapy-0.14.4>python setup.py install……Installing easy_install-2.7-script.py script to D:\Python27\ScriptsInstalling easy_install-2.7.exe script to D:\Python27\ScriptsInstalling easy_install-2.7.exe.manifest script to D:\Python27\ScriptsUsing d:\python27\lib\site-packagesFinished processing dependencies for Scrapy==0.14.4T:\Scrapy-0.14.4>
Scrapy 0.14.4 - no active project
Usage:
scrapy <command> [options] [args]
Available commands:
fetch Fetch a URL using the Scrapy downloader
runspider Run a self-contained spider (without creating a project)
settings Get settings values
shell Interactive scraping console
startproject Create new project
version Print Scrapy version
view Open URL in browser, as seen by Scrapy
Use "scrapy <command> -h" to see more info about a command
T:\>
解决方案是下载Python的另一个安装工具PIP,安装PIP之后把C:\Python27\Scripts路径加入环境变量path,在控制台输入pip显示如下表示安装成功:
我们找到下载的pyOpenSSL文件位置,在那里打开cmd直接输入pip pyOpenSSL-0.14.tar.gz回车,接着就安装OK了。
然后用pip安装下Scrapy包,OK,搞定。
- win7下 scrapy的安装
- win7下安装Scrapy
- Win7下 python安装Scrapy
- [python3]win7/win10下安装scrapy
- win7环境下安装爬虫框架 scrapy
- 【scrapy】debian下scrapy的安装
- 【scrapy】windows下scrapy的安装
- 无语的scrapy安装过程 win7
- Win7 python2.7 Scrapy的安装
- Linux 下Scrapy的安装
- Ubuntu下Scrapy的安装
- mac下scrapy的安装
- windows下scrapy 的安装
- win10下scrapy的安装
- win7下安装scrapy(开源python爬虫)
- 64位win7环境下进行Scrapy安装
- win7 32bit python3.6下安装scrapy
- win7 64位 scrapy安装
- war包和jar包的区别
- shell 中 grep、sed、awk 命令
- 《JAVA与模式》之访问者模式
- eclipse启动报错:
- HDU 1181(记忆化搜索)
- win7下 scrapy的安装
- 制作U盘启动盘装fedoras的几种方法
- 编程世界里面的特有名词
- Spring AOP
- UIViewController属性presentedViewController,presentingViewController和parentViewController区别
- angularjs+chosen的使用备忘
- 操作系统之处理机管理
- hask
- Java NIO入门与详解