MAC系统下基于Python3版本安装Scrapy

输入 pip install scrapy 执行 scrapy 安装

?

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

➜ ~ pip install Scrapy

Collecting Scrapy

 Using cached Scrapy-1.4.0-py2.py3-none-any.whl

Collecting lxml (from Scrapy)

 Using cached lxml-4.1.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl

Collecting PyDispatcher>=2.0.5 (from Scrapy)

 Using cached PyDispatcher-2.0.5.tar.gz

Collecting Twisted>=13.1.0 (from Scrapy)

 Using cached Twisted-17.9.0.tar.bz2

Requirement already satisfied: pyOpenSSL in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Scrapy)

Collecting queuelib (from Scrapy)

 Using cached queuelib-1.4.2-py2.py3-none-any.whl

Collecting cssselect>=0.9 (from Scrapy)

 Using cached cssselect-1.0.1-py2.py3-none-any.whl

Collecting parsel>=1.1 (from Scrapy)

 Using cached parsel-1.2.0-py2.py3-none-any.whl

Collecting service-identity (from Scrapy)

 Using cached service_identity-17.0.0-py2.py3-none-any.whl

Collecting six>=1.5.2 (from Scrapy)

 Using cached six-1.11.0-py2.py3-none-any.whl

Collecting w3lib>=1.17.0 (from Scrapy)

 Using cached w3lib-1.18.0-py2.py3-none-any.whl

Requirement already satisfied: zope.interface>=3.6.0 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Twisted>=13.1.0->Scrapy)

Collecting constantly>=15.1 (from Twisted>=13.1.0->Scrapy)

 Using cached constantly-15.1.0-py2.py3-none-any.whl

Collecting incremental>=16.10.1 (from Twisted>=13.1.0->Scrapy)

 Using cached incremental-17.5.0-py2.py3-none-any.whl

Collecting Automat>=0.3.0 (from Twisted>=13.1.0->Scrapy)

 Using cached Automat-0.6.0-py2.py3-none-any.whl

Collecting hyperlink>=17.1.1 (from Twisted>=13.1.0->Scrapy)

 Using cached hyperlink-17.3.1-py2.py3-none-any.whl

Collecting pyasn1 (from service-identity->Scrapy)

 Using cached pyasn1-0.3.7-py2.py3-none-any.whl

Collecting pyasn1-modules (from service-identity->Scrapy)

 Using cached pyasn1_modules-0.1.5-py2.py3-none-any.whl

Collecting attrs (from service-identity->Scrapy)

 Using cached attrs-17.2.0-py2.py3-none-any.whl

Requirement already satisfied: setuptools in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from zope.interface>=3.6.0->Twisted>=13.1.0->Scrapy)

Installing collected packages: lxml, PyDispatcher, constantly, incremental, six, attrs, Automat, hyperlink, Twisted, queuelib, cssselect, w3lib, parsel, pyasn1, pyasn1-modules, service-identity, Scrapy

Exception:

Traceback (most recent call last):

 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main

 status = self.run(options, args)

 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 342, in run

 prefix=options.prefix_path,

 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 784, in install

 **kwargs

 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 851, in install

 self.move_wheel_files(self.source_dir, root=root, prefix=prefix)

 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 1064, in move_wheel_files

 isolated=self.isolated,

 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 345, in move_wheel_files

 clobber(source, lib_dir, True)

 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 316, in clobber

 ensure_dir(destdir)

 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 83, in ensure_dir

 os.makedirs(path)

 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 157, in makedirs

 mkdir(name, mode)

OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/lxml'

出现 OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/lxml' 错误

4. 尝试重新安装lxml,执行 sudo pip install lxml

?

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

➜ ~ sudo pip install lxml

The directory '/Users/wangruofeng/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

The directory '/Users/wangruofeng/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

Collecting lxml

 Downloading lxml-4.1.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (8.7MB)

 100% |████████████████████████████████| 8.7MB 97kB/s

Installing collected packages: lxml

Successfully installed lxml-4.1.0

➜ ~ sudo pip install scrapy

The directory '/Users/wangruofeng/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

The directory '/Users/wangruofeng/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

Collecting scrapy

 Downloading Scrapy-1.4.0-py2.py3-none-any.whl (248kB)

 100% |████████████████████████████████| 256kB 1.5MB/s

Requirement already satisfied: lxml in /Library/Python/2.7/site-packages (from scrapy)

Collecting PyDispatcher>=2.0.5 (from scrapy)

 Downloading PyDispatcher-2.0.5.tar.gz

Collecting Twisted>=13.1.0 (from scrapy)

 Downloading Twisted-17.9.0.tar.bz2 (3.0MB)

 100% |████████████████████████████████| 3.0MB 371kB/s

Requirement already satisfied: pyOpenSSL in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from scrapy)

Collecting queuelib (from scrapy)

 Downloading queuelib-1.4.2-py2.py3-none-any.whl

Collecting cssselect>=0.9 (from scrapy)

 Downloading cssselect-1.0.1-py2.py3-none-any.whl

Collecting parsel>=1.1 (from scrapy)

 Downloading parsel-1.2.0-py2.py3-none-any.whl

Collecting service-identity (from scrapy)

 Downloading service_identity-17.0.0-py2.py3-none-any.whl

Collecting six>=1.5.2 (from scrapy)

 Downloading six-1.11.0-py2.py3-none-any.whl

Collecting w3lib>=1.17.0 (from scrapy)

 Downloading w3lib-1.18.0-py2.py3-none-any.whl

Requirement already satisfied: zope.interface>=3.6.0 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Twisted>=13.1.0->scrapy)

Collecting constantly>=15.1 (from Twisted>=13.1.0->scrapy)

 Downloading constantly-15.1.0-py2.py3-none-any.whl

Collecting incremental>=16.10.1 (from Twisted>=13.1.0->scrapy)

 Downloading incremental-17.5.0-py2.py3-none-any.whl

Collecting Automat>=0.3.0 (from Twisted>=13.1.0->scrapy)

 Downloading Automat-0.6.0-py2.py3-none-any.whl

Collecting hyperlink>=17.1.1 (from Twisted>=13.1.0->scrapy)

 Downloading hyperlink-17.3.1-py2.py3-none-any.whl (73kB)

 100% |████████████████████████████████| 81kB 1.4MB/s

Collecting pyasn1 (from service-identity->scrapy)

 Downloading pyasn1-0.3.7-py2.py3-none-any.whl (63kB)

 100% |████████████████████████████████| 71kB 2.8MB/s

Collecting pyasn1-modules (from service-identity->scrapy)

 Downloading pyasn1_modules-0.1.5-py2.py3-none-any.whl (60kB)

 100% |████████████████████████████████| 61kB 2.5MB/s

Collecting attrs (from service-identity->scrapy)

 Downloading attrs-17.2.0-py2.py3-none-any.whl

Requirement already satisfied: setuptools in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from zope.interface>=3.6.0->Twisted>=13.1.0->scrapy)

Installing collected packages: PyDispatcher, constantly, incremental, six, attrs, Automat, hyperlink, Twisted, queuelib, cssselect, w3lib, parsel, pyasn1, pyasn1-modules, service-identity, scrapy

 Running setup.py install for PyDispatcher ... done

 Found existing installation: six 1.4.1

 DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.

 Uninstalling six-1.4.1:

 Successfully uninstalled six-1.4.1

 Running setup.py install for Twisted ... done

Successfully installed Automat-0.6.0 PyDispatcher-2.0.5 Twisted-17.9.0 attrs-17.2.0 constantly-15.1.0 cssselect-1.0.1 hyperlink-17.3.1 incremental-17.5.0 parsel-1.2.0 pyasn1-0.3.7 pyasn1-modules-0.1.5 queuelib-1.4.2 scrapy-1.4.0 service-identity-17.0.0 six-1.11.0 w3lib-1.18.0

成功安装lxml-4.1.0

5. 再次尝试安装scrapy，执行 sudo pip install scrapy

?

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

➜ ~ sudo pip install scrapy

The directory '/Users/wangruofeng/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

The directory '/Users/wangruofeng/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

Collecting scrapy

 Downloading Scrapy-1.4.0-py2.py3-none-any.whl (248kB)

 100% |████████████████████████████████| 256kB 1.5MB/s

Requirement already satisfied: lxml in /Library/Python/2.7/site-packages (from scrapy)

Collecting PyDispatcher>=2.0.5 (from scrapy)

 Downloading PyDispatcher-2.0.5.tar.gz

Collecting Twisted>=13.1.0 (from scrapy)

 Downloading Twisted-17.9.0.tar.bz2 (3.0MB)

 100% |████████████████████████████████| 3.0MB 371kB/s

Requirement already satisfied: pyOpenSSL in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from scrapy)

Collecting queuelib (from scrapy)

 Downloading queuelib-1.4.2-py2.py3-none-any.whl

Collecting cssselect>=0.9 (from scrapy)

 Downloading cssselect-1.0.1-py2.py3-none-any.whl

Collecting parsel>=1.1 (from scrapy)

 Downloading parsel-1.2.0-py2.py3-none-any.whl

Collecting service-identity (from scrapy)

 Downloading service_identity-17.0.0-py2.py3-none-any.whl

Collecting six>=1.5.2 (from scrapy)

 Downloading six-1.11.0-py2.py3-none-any.whl

Collecting w3lib>=1.17.0 (from scrapy)

 Downloading w3lib-1.18.0-py2.py3-none-any.whl

Requirement already satisfied: zope.interface>=3.6.0 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Twisted>=13.1.0->scrapy)

Collecting constantly>=15.1 (from Twisted>=13.1.0->scrapy)

 Downloading constantly-15.1.0-py2.py3-none-any.whl

Collecting incremental>=16.10.1 (from Twisted>=13.1.0->scrapy)

 Downloading incremental-17.5.0-py2.py3-none-any.whl

Collecting Automat>=0.3.0 (from Twisted>=13.1.0->scrapy)

 Downloading Automat-0.6.0-py2.py3-none-any.whl

Collecting hyperlink>=17.1.1 (from Twisted>=13.1.0->scrapy)

 Downloading hyperlink-17.3.1-py2.py3-none-any.whl (73kB)

 100% |████████████████████████████████| 81kB 1.4MB/s

Collecting pyasn1 (from service-identity->scrapy)

 Downloading pyasn1-0.3.7-py2.py3-none-any.whl (63kB)

 100% |████████████████████████████████| 71kB 2.8MB/s

Collecting pyasn1-modules (from service-identity->scrapy)

 Downloading pyasn1_modules-0.1.5-py2.py3-none-any.whl (60kB)

 100% |████████████████████████████████| 61kB 2.5MB/s

Collecting attrs (from service-identity->scrapy)

 Downloading attrs-17.2.0-py2.py3-none-any.whl

Requirement already satisfied: setuptools in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from zope.interface>=3.6.0->Twisted>=13.1.0->scrapy)

Installing collected packages: PyDispatcher, constantly, incremental, six, attrs, Automat, hyperlink, Twisted, queuelib, cssselect, w3lib, parsel, pyasn1, pyasn1-modules, service-identity, scrapy

 Running setup.py install for PyDispatcher ... done

 Found existing installation: six 1.4.1

 DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.

 Uninstalling six-1.4.1:

 Successfully uninstalled six-1.4.1

 Running setup.py install for Twisted ... done

Successfully installed Automat-0.6.0 PyDispatcher-2.0.5 Twisted-17.9.0 attrs-17.2.0 constantly-15.1.0 cssselect-1.0.1 hyperlink-17.3.1 incremental-17.5.0 parsel-1.2.0 pyasn1-0.3.7 pyasn1-modules-0.1.5 queuelib-1.4.2 scrapy-1.4.0 service-identity-17.0.0 six-1.11.0 w3lib-1.18.0

6. 执行 scrapy 出现下面错误

?

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

➜ ~ scrapy

Traceback (most recent call last):

 File"/usr/local/bin/scrapy", line 7,in<module>

 fromscrapy.cmdline importexecute

 File"/Library/Python/2.7/site-packages/scrapy/cmdline.py", line 9,in<module>

 fromscrapy.crawler importCrawlerProcess

 File"/Library/Python/2.7/site-packages/scrapy/crawler.py", line 7,in<module>

 fromtwisted.internet importreactor, defer

 File"/Library/Python/2.7/site-packages/twisted/internet/reactor.py", line 38,in<module>

 fromtwisted.internet importdefault

 File"/Library/Python/2.7/site-packages/twisted/internet/default.py", line 56,in<module>

 install=_getInstallFunction(platform)

 File"/Library/Python/2.7/site-packages/twisted/internet/default.py", line 50,in_getInstallFunction

 fromtwisted.internet.selectreactor importinstall

 File"/Library/Python/2.7/site-packages/twisted/internet/selectreactor.py", line 18,in<module>

 fromtwisted.internet importposixbase

 File"/Library/Python/2.7/site-packages/twisted/internet/posixbase.py", line 18,in<module>

 fromtwisted.internet importerror, udp, tcp

 File"/Library/Python/2.7/site-packages/twisted/internet/tcp.py", line 28,in<module>

 fromtwisted.internet._newtls import(

 File"/Library/Python/2.7/site-packages/twisted/internet/_newtls.py", line 21,in<module>

 fromtwisted.protocols.tls importTLSMemoryBIOFactory, TLSMemoryBIOProtocol

 File"/Library/Python/2.7/site-packages/twisted/protocols/tls.py", line 63,in<module>

 fromtwisted.internet._sslverify import_setAcceptableProtocols

 File"/Library/Python/2.7/site-packages/twisted/internet/_sslverify.py", line 38,in<module>

 TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1,

AttributeError:'module'object has no attribute 'OP_NO_TLSv1_1'

需要更新 OpenSSL 库，执行 sudo pip install --upgrade pyopenssl

?

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

➜ ~ sudo pip install --upgrade pyopenssl

Password:

The directory '/Users/wangruofeng/Library/Caches/pip/http'or its parent directory isnot owned by the current user andthe cache has been disabled. Please check the permissions andowner of that directory. If executing pip with sudo, you may want sudo's -H flag.

The directory '/Users/wangruofeng/Library/Caches/pip'or its parent directory isnot owned by the current user andcaching wheels has been disabled. check the permissions andowner of that directory. If executing pip with sudo, you may want sudo's -H flag.

Collecting pyopenssl

 Downloading pyOpenSSL-17.3.0-py2.py3-none-any.whl (51kB)

 100%|████████████████████████████████| 51kB132kB/s

Requirement already up-to-date: six>=1.5.2in /Library/Python/2.7/site-packages (frompyopenssl)

Collecting cryptography>=1.9(frompyopenssl)

 Downloading cryptography-2.1.1-cp27-cp27m-macosx_10_6_intel.whl (1.5MB)

 100%|████████████████████████████████| 1.5MB938kB/s

Collecting cffi>=1.7; platform_python_implementation !="PyPy" (fromcryptography>=1.9->pyopenssl)

 Downloading cffi-1.11.2-cp27-cp27m-macosx_10_6_intel.whl (238kB)

 100%|████████████████████████████████| 245kB2.2MB/s

Collecting enum34; python_version < "3"(fromcryptography>=1.9->pyopenssl)

 Downloading enum34-1.1.6-py2-none-any.whl

Collecting idna>=2.1(fromcryptography>=1.9->pyopenssl)

 Downloading idna-2.6-py2.py3-none-any.whl (56kB)

 100%|████████████████████████████████| 61kB3.1MB/s

Collecting asn1crypto>=0.21.0(fromcryptography>=1.9->pyopenssl)

 Downloading asn1crypto-0.23.0-py2.py3-none-any.whl (99kB)

 100%|████████████████████████████████| 102kB2.7MB/s

Collecting ipaddress; python_version < "3"(fromcryptography>=1.9->pyopenssl)

 Downloading ipaddress-1.0.18-py2-none-any.whl

Collecting pycparser (fromcffi>=1.7; platform_python_implementation !="PyPy"->cryptography>=1.9->pyopenssl)

 Downloading pycparser-2.18.tar.gz (245kB)

 100%|████████████████████████████████| 256kB3.6MB/s

Installing collected packages: pycparser, cffi, enum34, idna, asn1crypto, ipaddress, cryptography, pyopenssl

 Running setup.py install forpycparser ... done

 Found existing installation: pyOpenSSL 0.13.1

 DEPRECATION: Uninstalling a distutils installed project (pyopenssl) has been deprecated andwill be removed ina future version. This isdue to the fact that uninstalling a distutils project will only partially uninstall the project.

 Uninstalling pyOpenSSL-0.13.1:

 Successfully uninstalled pyOpenSSL-0.13.1

Successfully installed asn1crypto-0.23.0cffi-1.11.2cryptography-2.1.1enum34-1.1.6idna-2.6ipaddress-1.0.18pycparser-2.18pyopenssl-17.3.0

更新 OpenSSL 成功，再次尝试执行 scrapy

?

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

➜ ~ scrapy 

Scrapy1.4.0- no active project

  

Usage:

 scrapy <command> [options] [args]

  

Available commands:

 bench Run quick benchmark test

 fetch Fetch a URL using the Scrapy downloader

 genspider Generate new spider using pre-defined templates

 runspider Run a self-contained spider (without creating a project)

 settings Get settings values

 shell Interactive scraping console

 startproject Create new project

 versionPrintScrapy version

 viewOpenURL inbrowser, as seen by Scrapy

  

 [ more ] More commands available when run fromproject directory

  

Use"scrapy <command> -h" to see more info about a command

出现上面内容，表明安装成功。现在可以通过 scrapy 创建一个爬虫项目了

7. 进入到你项目的目录,执行 scrapy startproject firstscrapy创建 firstscrapy 爬虫项目

?

1

2

3

4

5

6

7

8

➜ PycharmProjects scrapy startproject firstscrapy

New Scrapy project 'firstscrapy', using template directory '/Library/Python/2.7/site-packages/scrapy/templates/project', created in:

 /Users/wangruofeng/PycharmProjects/firstscrapy

  

You can start your first spider with:

 cd firstscrapy

 scrapy genspider example example.com

➜ PycharmProjects

出现上面内容表明项目创建成功，但是使用的是2.7版本的Python怎么切换到3.6版本呢？

8. 使用 PyCharm IDE 打开刚才的项目，执行 command + , 打开偏好设置菜单，在Project里面选择 Projiect interpreter 来切换你需要依赖的Python库的版本，配置结束。