Kali 安装scrapy爬虫框架
来源:互联网 发布:matlab遗传算法实例 编辑:程序博客网 时间:2024/05/18 12:40
参考http://www.linuxidc.com/Linux/2012-07/66236.htm
准备工作
Requirements
Python 2.5, 2.6, 2.7 (3.x is not yet supported)
Twisted 2.5.0, 8.0 or above (Windows users: you’ll need to install Zope.Interface and maybe pywin32 because of this Twisted bug)
w3lib
lxml or libxml2 (if using libxml2, version 2.6.28 or above is highly recommended)
simplejson (not required if using Python 2.6 or above)
pyopenssl (for HTTPS support. Optional, but highly recommended)
---------------------------------------------
Twisted安装过程
sudo apt-get install python-twisted python-libxml2 python-simplejson
安装完成后进入python,测试Twisted是否安装成功
pyOpenSSL
wget http://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.13.tar.gz#md5=767bca18a71178ca353dff9e10941929
tar -zxvf pyOpenSSL-0.13.tar.gz
cd pyOpenSSL-0.13
sudo python setup.py install
pycrypto
wget http://pypi.python.org/packages/source/p/pycrypto/pycrypto-2.5.tar.gz#md5=783e45d4a1a309e03ab378b00f97b291
tar -zxvf pycrypto-2.5.tar.gz
cd pycrypto-2.5
sudo python setup.py install
测试是否安装成功
$python
>>> import Crypto
>>> import twisted.conch.ssh.transport
>>> print Crypto.PublicKey.RSA
<module 'Crypto.PublicKey.RSA' from '/usr/python/lib/python2.5/site-packages/Crypto/PublicKey/RSA.pyc'>
>>> import OpenSSL
>>> import twisted.internet.ssl
>>> twisted.internet.ssl
<module 'twisted.internet.ssl' from '/usr/python/lib/python2.5/site-packages/Twisted-10.1.0-py2.5-linux-i686.egg/twisted/internet/ssl.pyc'>
如果出现类似提示,说明pyOpenSSL模块已经安装成功了,否则,请检查上面的安装过程(OpenSSL需要pycrypto)。
w3lib
sudo easy_install -U w3lib
Scrapy
wget http://pypi.python.org/packages/source/S/Scrapy/Scrapy-0.14.3.tar.gz#md5=59f1225f7692f28fa0f78db3d34b3850
tar -zxvf Scrapy-0.14.3.tar.gz
cd Scrapy-0.14.3
sudo python setup.py install
Scrapy安装验证
经过上面的安装和配置过程,已经完成了Scrapy的安装,我们可以通过如下命令行来验证一下:
$ scrapy
Scrapy 0.14.3 - no active project
Usage:
scrapy <command> [options] [args]
Available commands:
fetch Fetch a URL using the Scrapy downloader
runspider Run a self-contained spider (without creating a project)
settings Get settings values
shell Interactive scraping console
startproject Create new project
version Print Scrapy version
view Open URL in browser, as seen by Scrapy
- Kali 安装Scrapy爬虫框架
- Kali 安装scrapy爬虫框架
- 爬虫框架scrapy安装
- <scrapy>python 爬虫框架scrapy安装
- 安装Twisted、Scrapy爬虫框架
- Python爬虫框架Scrapy安装
- python爬虫框架scrapy安装
- 爬虫实践之爬虫框架Scrapy安装
- Python_Ubuntu 12.04 安装Twisted、Scrapy爬虫框架
- Python爬虫框架Scrapy实战之安装
- ubuntu14.04安装python爬虫框架Scrapy
- Lubuntu14.04(Ubuntu)安装爬虫框架Scrapy
- python爬虫框架Scrapy入门:安装
- RedHat下完美安装scrapy爬虫框架
- scrapy windows 安装教程 python 爬虫框架
- ubuntu下安装scrapy爬虫框架
- Mac下安装爬虫框架Scrapy
- 爬虫框架Scrapy的安装与简介
- Java Web高性能开发--前端高性能
- jvm_内存溢出_运行时常量池溢出
- DDPush开源推送框架源码分析之Client到DDPush(UDP模式)
- STL面试题
- Struts2.1.6+Spring2.5.6+Hibernate3.3.1全注解实例详解(四)
- Kali 安装scrapy爬虫框架
- 解决perl编译问题
- Trie树算法
- 对近期数个项目的整理和展望
- 例题1.7 偶数矩阵 UVa11464
- Leetcode Reverse Integer
- STM32定时器输出比较模式中的疑惑【转】
- Struts2.1.6+Spring2.5.6+Hibernate3.3.1全注解实例详解(五)
- uva439