【Python笔记】源码编译安装Python时，如何支持自定义安装的高版本openssl库

最近有个小需求想使用Scrapy库做抓取，但公司开发机操作系统版本老旧，导致系统默认的openssl库版本也很低（OpenSSL 0.9.7a Feb 19 2003），最终导致安装Scrapy非常麻烦。趁着元旦假期，决定用自己安装好的高版本openssl库作为依赖，重新编译安装开发机的Python环境。

安装及其后使用中遇到一些关于openssl的问题，或许对自己后续工作或别人有参考价值，作为笔记，记录于此。

源码编译安装流程如下：

1. Python官网下载源码包（version 2.7.8）并解压

2. 通过configure配置Makefile参数
假定先前已在/home/slvher/tools/openssl-1.0.1j路径下成功安装好高版本openssl，那么，可以通过下面的环境变量来设置gcc的编译/链接参数：

export LDFLAGS="-lssl -lcrypto -Wl,-rpath=/home/slvher/tools/openssl-1.0.1j/lib/"export CPPFLAGS="-I/home/slvher/tools/openssl-1.0.1j/include"

然后运行configure工具：

./configure --prefix=/home/slvher/tools/Python-2.7.8 

configure运行完后，会生成Makefile，check无误后，执行make && make install就能完成Python安装。

3. 验证Python支持的ssl版本
Python安装成功且其bin路径加入PATH环境变量后，可通过下面的命令来验证是否正确支持高版本的openssl库：

python -c "import ssl; print ssl.OPENSSL_VERSION"

例如我机器上的输出结果如下：

OpenSSL 1.0.1j 15 Oct 2014

4. 后续使用依赖openssl的包时，可能遇到的问题及解决思路
在依赖了高版本openssl库（普通用户自定义安装在非系统标准路径下）的Python环境下，通过easy_install或pip安装Python第三方包时，可能会报类似于下面的错误（以pip安装Scrapy为例）：

此处省略上文...[Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!此处省略下文...

受StackOverflow这篇帖子python easy_install fails with SSL certificate error for all packages的启发，结合pip的安装日志（本文最后会贴出报错日志），最终确定报错的原因是由于安装过程中setuptools会通过https访问PyPI来拉取某个库依赖的其它包，而通过https访问PyPI源时，它使用的是高版本的openssl，但其加载的certificate证书却是与系统默认的旧版openssl库配套的证书，所以会报出"certificate verify failed"的错误。
关于这个错误，有人在github给pip提了issue (pip --cert not working as expected)，但实际上这应该是setuptools的问题。

解决思路1：
为setuptools指定新的证书地址，具体方法可参考这篇资料OpenSSL Errors and Rails–Certificate Verify Failed，但我按帖中提到的export SSL_CERT_FILE的方法试验过，报错依旧。估计这个方法对Ruby on Rails环境奏效，Python环境未必有疗效。

解决思路2：
setuptools是由于ssl证书认证失败而报错，且setuptools加载ca证书的地址是硬编码的，并未没有开放参数设置给使用者。因此，思路1的方法暂时无解。作为替代方案，根据github/pip关于这个问题的讨论帖，我们可以绕过setuptools，直接用pip去拉取依赖：

pip install --cert /home/slvher/tools/https-ca/https/cacert.pem -U -i https://pypi.python.org/simple six

不出意外的话，依赖包（上面的pip install命令中以Scrapy依赖的six包为例）会拉取并编译成功，这表明依赖了自定义安装的高版本openssl库的Python环境其实是可以正常工作的，这个报错可能是setuptools的小bug而已。

下面是上文提到的运行"pip install Scrapy"会由于ssl certificate证书太旧导致报错的日志片段。
可以看到，出问题的地方从"Running setup.py install for cryptography"那行（第33行）开始，而setup.py install调用了setuptools，所以可以推断，认证失败的确是由setuptools引起。而用pip直接拉取依赖包不会报错也证实了这个推断。

$ pip install ScrapyRequirement already satisfied (use --upgrade to upgrade): Scrapy in ./lib/python2.7/site-packages/Scrapy-0.24.4-py2.7.eggCollecting Twisted>=10.0.0 (from Scrapy)  Using cached Twisted-14.0.2.tar.bz2Collecting w3lib>=1.8.0 (from Scrapy)  Using cached w3lib-1.10.0-py2.py3-none-any.whlCollecting queuelib (from Scrapy)  Using cached queuelib-1.2.2-py2.py3-none-any.whlCollecting lxml (from Scrapy)  Using cached lxml-3.4.1.tar.gz    /home/slvher/tools/Python-2.7.8/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'      warnings.warn(msg)    Building lxml version 3.4.1.    Building without Cython.    Using build configuration of libxslt 1.1.26    Building against libxml2/libxslt in the following directory: /home/slvher/.jumbo/libCollecting pyOpenSSL (from Scrapy)  Using cached pyOpenSSL-0.14.tar.gzCollecting cssselect>=0.9 (from Scrapy)  Using cached cssselect-0.9.1.tar.gzCollecting six>=1.5.2 (from Scrapy)  Using cached six-1.8.0-py2.py3-none-any.whlCollecting zope.interface>=3.6.0 (from Twisted>=10.0.0->Scrapy)  Using cached zope.interface-4.1.2.tar.gzCollecting cryptography>=0.2.1 (from pyOpenSSL->Scrapy)  Using cached cryptography-0.7.1.tar.gzRequirement already satisfied (use --upgrade to upgrade): setuptools in ./lib/python2.7/site-packages/setuptools-10.1-py2.7.egg (from zope.interface>=3.6.0->Twisted>=10.0.0->Scrapy)Requirement already satisfied (use --upgrade to upgrade): cffi>=0.8 in ./lib/python2.7/site-packages (from cryptography>=0.2.1->pyOpenSSL->Scrapy)Requirement already satisfied (use --upgrade to upgrade): enum34 in ./lib/python2.7/site-packages (from cryptography>=0.2.1->pyOpenSSL->Scrapy)Requirement already satisfied (use --upgrade to upgrade): pyasn1 in ./lib/python2.7/site-packages (from cryptography>=0.2.1->pyOpenSSL->Scrapy)Requirement already satisfied (use --upgrade to upgrade): pycparser in ./lib/python2.7/site-packages (from cffi>=0.8->cryptography>=0.2.1->pyOpenSSL->Scrapy)Installing collected packages: cryptography, zope.interface, six, cssselect, pyOpenSSL, lxml, queuelib, w3lib, Twisted    Running setup.py install for cryptography    Download error on https://pypi.python.org/simple/six/: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!    Couldn't find index page for 'six' (maybe misspelled?)    Download error on https://pypi.python.org/simple/: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!    No local packages or download links found for six>=1.4.1    Traceback (most recent call last):      File "<string>", line 1, in <module>      File "/tmp/pip-build-G9LiJL/cryptography/setup.py", line 337, in <module>        **keywords_with_side_effects(sys.argv)      File "/home/slvher/tools/Python-2.7.8/lib/python2.7/distutils/core.py", line 111, in setup        _setup_distribution = dist = klass(attrs)      File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 266, in __init__      File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 312, in fetch_build_eggs      File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 753, in resolve      File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1005, in best_match      File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1017, in obtain      File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 379, in fetch_build_egg      File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 613, in easy_install    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('six>=1.4.1')    Complete output from command /home/slvher/tools/Python-2.7.8/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-G9LiJL/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-eZ3kZg-record/install-record.txt --single-version-externally-managed --compile:    Download error on https://pypi.python.org/simple/six/: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!        Couldn't find index page for 'six' (maybe misspelled?)        Download error on https://pypi.python.org/simple/: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!        No local packages or download links found for six>=1.4.1        Traceback (most recent call last):          File "<string>", line 1, in <module>          File "/tmp/pip-build-G9LiJL/cryptography/setup.py", line 337, in <module>            **keywords_with_side_effects(sys.argv)          File "/home/slvher/tools/Python-2.7.8/lib/python2.7/distutils/core.py", line 111, in setup            _setup_distribution = dist = klass(attrs)          File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 266, in __init__          File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 312, in fetch_build_eggs          File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 753, in resolve          File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1005, in best_match          File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1017, in obtain          File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 379, in fetch_build_egg          File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 613, in easy_install        distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('six>=1.4.1')        ----------------------------------------    Command "/home/slvher/tools/Python-2.7.8/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-G9LiJL/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-eZ3kZg-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-G9LiJL/cryptography

【参考资料】
1. python easy_install fails with SSL certificate error for all packages 
2. pip --cert not working as expected 

======================== EOF =======================