ubuntu14.04下利用pypi-server搭建本地的pip库

应实验室需要，自己动手实现了一个pip本地仓库。其实搭建起来很简单，重点在于对本地不存在包的处理。这里我设置的机制是当本地不存在所请求的包时继续维持连接，同时从阿里的pip镜像源中将该包下载到本地以供用户下载。于是，问题主要在于获取所缺失包的信息。

我的解决方法是：当pypi-server在本地找不到所请求包时跳转到主机的另一端口，在此端口上运行一个监听程序，从HTTP请求中获取所要下载的包，然后从阿里的pip镜像库中将该包下载到本地。

1.下载pypi-server

跳转下载

2.pypi-server的使用

pypi-server [option] packages_path

中间可以跟相关的参数，最后跟上本地存储路径

这里只说明一些我所用到的参数：

-p 指明pypi-server所在的端口

--disable-fallback 禁止在找不到包的时候跳转到pypi官网链接

--fallback-url [url]指明找不到包需要跳转的链接

如：pypi-server -p 8080 --fallback-url http://localhost:9000 ~/pypi-packages

pypi-server运行在主机的8080端口，若找不到所请求的包，则跳转到主机的9000端口，本地存储路径为~/pypi-packages

3.监听程序

前一个版本的在使用过程中遇到问题，即pip用户在找不到包时会不断的发送请求，于是本地仓库中便会有大量重复的包。

解决方法：下载包之前先判断当前目录下是否已经存在，存在则直接跳过.....

<span style="font-size:18px;"></span>

<span style="font-size:18px;">#coding:utf-8import threadingimport socketimport reimport osimport urllibimport urllib.request as urllib2encoding = 'utf-8'BUFSIZE = 1024# a read thread, read data from remoteclass Reader(threading.Thread):<span style="white-space:pre"></span>def __init__(self, client):<span style="white-space:pre"></span>threading.Thread.__init__(self)<span style="white-space:pre"></span>self.client = client<span style="white-space:pre"></span>self.index_url="http://mirrors.aliyun.com/pypi/simple/"<span style="white-space:pre"></span>self.dir_url="http://mirrors.aliyun.com/pypi/packages/source/"<span style="white-space:pre"></span>self.pkgname=""<span style="white-space:pre"></span>self.pkgdir=[]<span style="white-space:pre"></span>self.pkglist=[]<span style="white-space:pre"></span>self.storepath="~/pypi/"        <span style="white-space:pre"></span>def run(self):<span style="white-space:pre"></span>while True:<span style="white-space:pre"></span>f=open("listen.log","a+")<span style="white-space:pre"></span>data = self.client.recv(BUFSIZE)<span style="white-space:pre"></span>if(data):<span style="white-space:pre"></span>string = bytes.decode(data, encoding)<span style="white-space:pre"></span>f.writelines(string)<span style="white-space:pre"></span>self.pkgname=re.findall("GET /(.*?)/",string)<span style="white-space:pre"></span>self.pkgname="".join(self.pkgname)<span style="white-space:pre"></span>f.writelines("*******the pkgname: "+self.pkgname+"*******")<span style="white-space:pre"></span>self.getpkgdir()<span style="white-space:pre"></span>pkgdir=self.pkgdir<span style="white-space:pre"></span>for pkg in pkgdir:<span style="white-space:pre"></span>if os.path.exists(self.storepath+pkg[1]):<span style="white-space:pre"></span>print("pkg exists!")<span style="white-space:pre"></span>break<span style="white-space:pre"></span>else:<span style="white-space:pre"></span>self.download(self.dir_url+pkg[0])<span style="white-space:pre"></span>else:<span style="white-space:pre"></span>break<span style="white-space:pre"></span>peername=self.client.getpeername()<span style="white-space:pre"></span>f.writelines("close:"+str(peername))<span style="white-space:pre"></span>f.writelines("\n=============================================\n")<span style="white-space:pre"></span>f.close()<span style="white-space:pre"></span>def getpkgdir(self):<span style="white-space:pre"></span>pkg_url=self.index_url+self.pkgname<span style="white-space:pre"></span>request=urllib2.Request(pkg_url)<span style="white-space:pre"></span>response=urllib2.urlopen(request)<span style="white-space:pre"></span>data=response.read().decode("utf-8")<span style="white-space:pre"></span>self.pkgdir=re.findall('<a href=".*?source/(.*?)" rel="internal">(.*?)</a>',data)<span style="white-space:pre"></span>def download(self,url):<span style="white-space:pre"></span>os.system("wget -P %s %s"%(self.storepath,url))    <span style="white-space:pre"></span>def readline(self):<span style="white-space:pre"></span>rec = self.inputs.readline()<span style="white-space:pre"></span>if rec:<span style="white-space:pre"></span>string = bytes.decode(rec, encoding)<span style="white-space:pre"></span>if len(string)>2:<span style="white-space:pre"></span>string = string[0:-2]<span style="white-space:pre"></span>else:<span style="white-space:pre"></span>string = ' '<span style="white-space:pre"></span>else:<span style="white-space:pre"></span>string = False<span style="white-space:pre"></span>return string# a listen thread, listen remote connect# when a remote machine request to connect, it will create a read thread to handleclass Listener(threading.Thread):<span style="white-space:pre"></span>def __init__(self, port):<span style="white-space:pre"></span>threading.Thread.__init__(self)<span style="white-space:pre"></span>self.port = port<span style="white-space:pre"></span>self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)<span style="white-space:pre"></span>self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)<span style="white-space:pre"></span>self.sock.bind(("0.0.0.0", port))<span style="white-space:pre"></span>self.sock.listen(0)<span style="white-space:pre"></span>def run(self):<span style="white-space:pre"></span>print("listener started")<span style="white-space:pre"></span>while True:<span style="white-space:pre"></span>client, cltadd = self.sock.accept()<span style="white-space:pre"></span>Reader(client).start()<span style="white-space:pre"></span>cltadd = cltadd<span style="white-space:pre"></span>print("accept a connect")lst  = Listener(8080)   # create a listen threadlst.start() # then start</span>

<span style="font-size:18px;"></span>

4.pip下载

pip install [packagename] -i http://localhost:8080/simple

利用上述命令则可以从本地pip库中下载所需的包

或者利用下述办法，可直接利用pip install [packagename]命令从本地源下载：

<span style="font-size:18px;"></span><pre name="code" class="plain"><span style="font-size:18px;">//在主文件夹下新建.pip文件夹</span>

<span style="font-size:18px;">mkdir ~/.pip</span>

<span style="font-size:18px;"></span>

<span style="font-size:18px;">cd ~/.pip</span>

<span style="font-size:18px;"></span><pre name="code" class="plain"><span style="font-size:18px;">//在文件夹.pip中新建pip.conf文件</span>

vim pip.conf

<span style="font-size:18px;"></span>

<span style="font-size:18px;"></span>

<span style="font-size:18px;">//在pip.conf文件中添加以下内容[global]index-url = http://localhost:8080/simple</span>

<span style="font-size:18px;"></span>