Mitmproxy

(PS:/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mitmproxy)

Pre——konwledge

SNI(Server Name Indication)

Server Name Indication是用来改善SSL(Secure Socket Layer)和TLS(Transport Layer Security)的一项特性。它允许客户端在服务器端向其发送证书之前请求服务器的域名。这对于在虚拟主机模式使用TLS是必要的。

TLS背景

加密一个面向流的通讯会话最常用的方法之一就是使用TLS协议。比如，当用户在浏览器的地址栏里面输入https时就是在使用这个协议。

为了确认用户想要连接的站点就是浏览器实际连接到的站点，TLS使用包含站点域名的数字签名证书。客户端软件(比如浏览器)通常信任这个证书，如果这个证书是由其内置信任的认证机构签发的。
在TLS启动阶段，客户端软件比较用户输入的URI的域名部分与在服务器证书里面找到的域名部分，如果比较失败，浏览器会提示用户，这个站点的证书存在问题。

缺点

SSL v2的设计顺应经典的公钥基础设施PKI(public key infrastructure)设计，后者认为一个服务器只提供一个服务从而也就只使用一个证书。这意味着服务器可以在TLS启动的早期阶段发送或提交证书，因为它知道它在为哪个域服务。

HTTP服务器开启虚拟主机支持后，每个服务器通过相同的地址可以为很多域提供服务。服务器检查每一个请求来决定它在为哪个域服务。这个信息通常从HTTP请求头获得。不幸的是，当设置了TLS加密，服务器在读取HTTP请求里面的域名之前已经向客户端提交了证书，也就是已经为默认域提供了服务。

因此，这种为虚拟主机提供安全的简单途径经常导致使用了错误的数字证书，从而导致浏览器对用户发出警告。

钓鱼连接

实际上，这意味着每一个HTTP服务器只能为一个域提供安全浏览。而事实上每一个web服务器都为很多域提供服务，结果就是其他的域无法使用安全通信，从而处于危险境地。此外，安全浏览的缺乏使浏览器无法认证服务器，亦即它无法校验站点是否真的是属于宣称它的那个人或实体。钓鱼的一个重要因素是他们企图通过虚假站点来获取用户的信息。使用SSL或者TLS安全连接，浏览器可以基于它的证书来认证站点。钓鱼站点不会作为一个欺骗性的站点得到认证，浏览器会警告这个安全风险。然而，没有安全HTTP就没有标准的方法去认证服务器，使这种钓鱼的企图很容易就能实现。

修正

一个叫做SNI的TLS扩展通过发送虚拟域的名字做为TSL协商的一部分修正了这个问题。这会使服务器更早的切换到正确的虚拟域，并且发送给浏览器包含正确名字的数字证书。

行动

在2005年，人们意识到从SSL v2到TLS没有很容易的升级路径，并且站点不得不升级他们的软件来。为了尽快的推进，Mozilla宣告完全抛弃对SSL v2的支持。Firefox社区确信其余的站点会升级他们的服务器到SSL v3或TLS v1。

从2005年开始，CAcert在虚拟服务器上用不同的方法使用TLS来进行试验，大部分试验是不满意并且不实际的。比如，可以使用subjectAltName在一个数字证书中包含多个域，但是这是一个证书，意味着所有的域必须被一个人拥有并控制，并且每次域列表发生变化，证书必须重新发放。

2004年，EdelKey project为OpenSSL里面的TLS/SNI开发了一个补丁。2006年这个补丁进入OpenSSL的开发分支，2007年，它被向后移植到了OpenSSL 0.9.8，也就是当前的发行版本。

支持状况

支持SNI的浏览器

• Mozilla Firefox 2.0 or later
• Opera 8.0 or later (the TLS 1.1 protocol must be enabled)
• Internet Explorer 7 or later on Windows Vista or higher
• Google Chrome (Vista, not XP)
• Safari 3.2.1 Mac OS X 10.5.6

支持SNI的服务器

• Apache 2.2.12 or later using mod_ssl (or alternatively with experimental mod_gnutls)
• Cherokee if compiled with TLS support
• Versions of lighttpd 1.4.x and 1.5.x with patch
• Nginx with an accompanying OpenSSL built with SNI support
• acWEB with OpenSSL 0.9.8j and later (on Windows)

支持SNI的库

• Mozilla NSS 3.11.1 client side only
• OpenSSL
  0.9.8f (released 11 Oct 2007) – not compiled in by default, can be compiled in with config option ‘–enable-tlsext’.
  0.9.8j (released 07 Jan 2009) – now compiled in by default
  Unreleased 0.9.9 is likely to include SNI compiled in by default.
• GNU TLS

不支持SNI的操作系统,浏览器和库

客户端

• Internet Explorer 6 or earlier and any IE version on Windows XP , windows 2003 or earlier
• Konqueror/KDE in any version

服务器端

• Microsoft Internet Information Server IIS (As of 2009).
• Apache Tomcat 8 or earlier

库

• Qt
• Mozilla NSS server side
• Python

windows和IE

可以看得出，windows XP和windows 2003 server系统上的任何IE版本浏览器都是不支持SNI的, vista及以后系统上的IE 7及更高版本的IE浏览器支持SNI。IE6及更早版本的IE浏览器在任何系统上都是不支持SNI的。

tomcat

tomcat当前的稳定版8尚不支持SNI,tomcat 9才会支持,以后可能会backport到tomcat 8和7。可以使用nginx反向https代理后端的tomcat,参见[2]

SNI测试

用浏览器或其他https客户端比如wget等访问SNI测试站点https://sni.velox.ch/即可以知道浏览器或客户端是否支持SNI。

References:

[1]Server Name Indication
[2]Setting up NGINX SSL reverse proxy for Tomcat
[3]HowTo setup Tomcat serving two SSL Certificates using SNI?

Installation

Installation On Ubuntu

Ubuntu comes with Python but we need to install pip, python-dev and several libraries. This was tested on a fully patched installation of Ubuntu 14.04.

>>> sudo apt-get install python-pip python-dev libffi-dev libssl-dev libxml2-dev libxslt1-dev libjpeg8-dev zlib1g-dev>>> sudo pip install mitmproxy

Once installation is complete you can run mitmproxy or mitmdump from a terminal.

On Ubuntu 12.04 (and other systems with an outdated version of pip), you may need to update pip using

pip install -U pip 

before installing mitmproxy.

Installation From Source (Ubuntu)

If you would like to install mitmproxy directly from the master branch on GitHub or would like to get set up to contribute to the project, install the dependencies as you would for a regular mitmproxy installation (see Installation On Ubuntu). Then see the Hacking section of the README on GitHub.

Installation On Mac OS X

The easiest way to get up and running on OSX is to download the pre-built binary packages from mitmproxy.org.

There are a few bits of customization you might want to do to make mitmproxy comfortable to use on OSX. The default color scheme is optimized for a dark background terminal, but you can select a palette for a light terminal background with the

--palette

option. You can use the OSX open program to create a simple and effective

 ~/.mailcap 

file to view request and response bodies:

application/*; /usr/bin/open -Wn %saudio/*; /usr/bin/open -Wn %simage/*; /usr/bin/open -Wn %svideo/*; /usr/bin/open -Wn %s

Once installation is complete you can run mitmproxy or mitmdump from a terminal.

Installation From Source (Mac OS X)

If you would like to install mitmproxy directly from the master branch on GitHub or would like to get set up to contribute to the project, there are a few OS X specific things to keep in mind.

• Make sure that XCode is installed from the App Store, and that the command-line tools have been downloaded (XCode/Preferences/Downloads).

• If you’re running a Python interpreter installed with homebrew (or similar), you may have to install some dependencies by hand.
  Then see the Hacking section of the README on GitHub.

Installation On Windows

PSPlease note that mitmdump is the only component of mitmproxy that is supported on Windows at the moment.There is no interactive user interface on Windows.

First, install the latest version of Python 2.7 from the Python website. If you already have an older version of Python 2.7 installed, make sure to install pip (pip is included in Python 2.7.9+ by default).

Next, add Python and the Python Scripts directory to your PATH variable. You can do this easily by running the following in powershell:

>>> [Environment]::SetEnvironmentVariable("Path", "$env:Path;C:\Python27;C:\Python27\Scripts", "User")

Now, you can install mitmproxy by running

>>> pip install mitmproxy

Once the installation is complete, you can run mitmdump from a command prompt.

Installation From Source (Windows)

If you would like to install mitmproxy directly from the master branch on GitHub or would like to get set up to contribute to the project, install Python as outlined above, then see the Hacking section of the README on GitHub.

Certificates

Introduction

Mitmproxy can decrypt encrypted traffic on the fly, as long as the client trusts its built-in certificate authority. Usually this means that the mitmproxy CA certificates have to be installed on the client device.

Quick Setup

By far the easiest way to install the mitmproxy certificates is to use the built-in certificate installation app. To do this, just start mitmproxy and configure your target device with the correct proxy settings. Now start a browser on the device, and visit the magic domain mitm.it. You should see something like this:

Installing the mitmproxy CA certificate manually

Sometimes using the quick install app is not an option - Java or the iOS Simulator spring to mind - or you just need to do it manually for some other reason. Below is a list of pointers to manual certificate installation documentation for some common platforms.

The mitmproxy CA cert is located in ~/.mitmproxy after it has been generated at the first start of mitmproxy.

iOShttp://kb.mit.edu/confluence/pages/viewpage.action?pageId=152600377iOS SimulatorSee https://github.com/ADVTOOLS/ADVTrustStore#how-to-use-advtruststoreJavaSee http://docs.oracle.com/cd/E19906-01/820-4916/geygn/index.htmlAndroid/Android SimulatorSee http://wiki.cacert.org/FAQ/ImportRootCert#Android_Phones_.26_TabletsWindowsSee http://windows.microsoft.com/en-ca/windows/import-export-certificates-private-keys#1TC=windows-7Windows (automated)>>> certutil.exe -importpfx mitmproxy-ca-cert.p12See also: https://technet.microsoft.com/en-us/library/cc732443.aspxMac OS XSee https://support.apple.com/kb/PH7297?locale=en_USUbuntu/DebianSee http://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate/94861#94861Mozilla FirefoxSee https://wiki.mozilla.org/MozillaRootCertificate#Mozilla_FirefoxChrome on LinuxSee https://code.google.com/p/chromium/wiki/LinuxCertManagement

The mitmproxy certificate authority

The first time mitmproxy or mitmdump is run, the mitmproxy Certificate Authority (CA) is created in the config directory (~/.mitmproxy by default). This CA is used for on-the-fly generation of dummy certificates for each of the SSL sites that your client visits. Since your browser won’t trust the mitmproxy CA out of the box, you will see an SSL certificate warning every time you visit a new SSL domain through mitmproxy. When you are testing a single site through a browser, just accepting the bogus SSL cert manually is not too much trouble, but there are a many circumstances where you will want to configure your testing system or browser to trust the mitmproxy CA as a signing root authority. For security reasons, the mitmproxy CA is generated uniquely on the first start and is not shared between mitmproxy installations on different devices.

Certificate Pinning

Some applications employ Certificate Pinning to prevent man-in-the-middle attacks. This means that mitmproxy and mitmdump’s certificates will not be accepted by these applications without modifying them. It is recommended to use the Ignore Domains feature in order to prevent mitmproxy and mitmdump from intercepting traffic to these specific domains. If you want to intercept the pinned connections, you need to patch the application manually. For Android and (jailbroken) iOS devices, various tools exist to accomplish this.

CA and cert files

The files created by mitmproxy in the .mitmproxy directory are as follows:

mitmproxy-ca.pemThe certificate and the private key in PEM format.mitmproxy-ca-cert.pemThe certificate in PEM format.Use this to distribute on most non-Windows platforms.mitmproxy-ca-cert.p12The certificate in PKCS12 format. For use on Windows.mitmproxy-ca-cert.cerSame file as .pem, but with an extension expected by some Android devices.

Using a custom certificate

You can use your own certificate by passing the –cert option to mitmproxy. Mitmproxy then uses the provided certificate for interception of the specified domains instead of generating a certificate signed by its own CA.

The certificate file is expected to be in the PEM format. You can include intermediary certificates right below your leaf certificate, so that you PEM file roughly looks like this:

-----BEGIN PRIVATE KEY-----<private key>-----END PRIVATE KEY----------BEGIN CERTIFICATE-----<cert>-----END CERTIFICATE----------BEGIN CERTIFICATE-----<intermediary cert (optional)>-----END CERTIFICATE-----

For example, you can generate a certificate in this format using these instructions:

>>> openssl genrsa -out cert.key 2048>>> openssl req -new -x509 -key cert.key -out cert.crt    (Specify the mitm domain as Common Name, e.g. *.google.com)>>> cat cert.key cert.crt > cert.pem>>> mitmproxy --cert=cert.pem

Using a custom certificate authority

By default, mitmproxy will use

~/.mitmproxy/mitmproxy-ca.pem 

as the certificate authority to generate certificates for all domains for which no custom certificate is provided (see above). You can use your own certificate authority by passing the –cadir DIRECTORY option to mitmproxy. Mitmproxy will then look for mitmproxy-ca.pem in the specified directory. If no such file exists, it will be generated automatically.

Using a client side certificate
You can use a client certificate by passing the

--client-certs DIRECTORY|FILE 

option to mitmproxy. Using a directory allows certs to be selected based on hostname, while using a filename allows a single specific certificate to be used for all SSL connections. Certificate files must be in the PEM format and should contain both the unencrypted private key and the certificate.

Multiple certs by Hostname

If you’ve specified a directory to

--client-certs, 

then the following behavior will be taken:

If you visit example.org, mitmproxy looks for a file named

example.org.pem

in the specified directory and uses this as the client cert.

How mitmproxy works

Mitmproxy is an enormously flexible tool. Knowing exactly how the proxying process works will help you deploy it creatively, and take into account its fundamental assumptions and how to work around them. This document explains mitmproxy’s proxy mechanism in detail, starting with the simplest unencrypted explicit proxying, and working up to the most complicated interaction - transparent proxying of SSL-protected traffic [1] in the presence of Server Name Indication.

Explicit HTTP

Configuring the client to use mitmproxy as an explicit proxy is the simplest and most reliable way to intercept traffic. The proxy protocol is codified in the HTTP RFC, so the behaviour of both the client and the server is well defined, and usually reliable. In the simplest possible interaction with mitmproxy, a client connects directly to the proxy, and makes a request that looks like this:

GET http://example.com/index.html HTTP/1.1

This is a proxy GET request - an extended form of the vanilla HTTP GET request that includes a schema and host specification, and it includes all the information mitmproxy needs to proceed.

The client connects to the proxy and makes a request.
Mitmproxy connects to the upstream server and simply forwards the request on.

Explicit HTTPS

The process for an explicitly proxied HTTPS connection is quite different. The client connects to the proxy and makes a request that looks like this:

CONNECT example.com:443 HTTP/1.1

A conventional proxy can neither view nor manipulate an SSL-encrypted data stream, so a CONNECT request simply asks the proxy to open a pipe between the client and server. The proxy here is just a facilitator - it blindly forwards data in both directions without knowing anything about the contents. The negotiation of the SSL connection happens over this pipe, and the subsequent flow of requests and responses are completely opaque to the proxy.

The MITM in mitmproxy

This is where mitmproxy’s fundamental trick comes into play. The MITM in its name stands for Man-In-The-Middle - a reference to the process we use to intercept and interfere with these theoretically opaque data streams. The basic idea is to pretend to be the server to the client, and pretend to be the client to the server, while we sit in the middle decoding traffic from both sides. The tricky part is that the Certificate Authority system is designed to prevent exactly this attack, by allowing a trusted third-party to cryptographically sign a server’s SSL certificates to verify that they are legit. If this signature doesn’t match or is from a non-trusted party, a secure client will simply drop the connection and refuse to proceed. Despite the many shortcomings of the CA system as it exists today, this is usually fatal to attempts to MITM an SSL connection for analysis. Our answer to this conundrum is to become a trusted Certificate Authority ourselves. Mitmproxy includes a full CA implementation that generates interception certificates on the fly. To get the client to trust these certificates, we register mitmproxy as a trusted CA with the device manually.

Complication 1: What’s the remote hostname?

To proceed with this plan, we need to know the domain name to use in the interception certificate - the client will verify that the certificate is for the domain it’s connecting to, and abort if this is not the case. At first blush, it seems that the CONNECT request above gives us all we need - in this example, both of these values are “example.com”. But what if the client had initiated the connection as follows:

CONNECT 10.1.1.1:443 HTTP/1.1

Using the IP address is perfectly legitimate because it gives us enough information to initiate the pipe, even though it doesn’t reveal the remote hostname.

Mitmproxy has a cunning mechanism that smooths this over - upstream certificate sniffing. As soon as we see the CONNECT request, we pause the client part of the conversation, and initiate a simultaneous connection to the server. We complete the SSL handshake with the server, and inspect the certificates it used. Now, we use the Common Name in the upstream SSL certificates to generate the dummy certificate for the client. Voila, we have the correct hostname to present to the client, even if it was never specified.

Complication 2: Subject Alternative Name

Enter the next complication. Sometimes, the certificate Common Name is not, in fact, the hostname that the client is connecting to. This is because of the optional Subject Alternative Name field in the SSL certificate that allows an arbitrary number of alternative domains to be specified. If the expected domain matches any of these, the client will proceed, even though the domain doesn’t match the certificate Common Name. The answer here is simple: when we extract the CN from the upstream cert, we also extract the SANs, and add them to the generated dummy certificate.

Complication 3: Server Name Indication

One of the big limitations of vanilla SSL is that each certificate requires its own IP address. This means that you couldn’t do virtual hosting where multiple domains with independent certificates share the same IP address. In a world with a rapidly shrinking IPv4 address pool this is a problem, and we have a solution in the form of the Server Name Indication extension to the SSL and TLS protocols. This lets the client specify the remote server name at the start of the SSL handshake, which then lets the server select the right certificate to complete the process.

SNI breaks our upstream certificate sniffing process, because when we connect without using SNI, we get served a default certificate that may have nothing to do with the certificate expected by the client. The solution is another tricky complication to the client connection process. After the client connects, we allow the SSL handshake to continue until just after the SNI value has been passed to us. Now we can pause the conversation, and initiate an upstream connection using the correct SNI value, which then serves us the correct upstream certificate, from which we can extract the expected CN and SANs.

Putting it all together

Lets put all of this together into the complete explicitly proxied HTTPS flow.

The client makes a connection to mitmproxy, and issues an HTTP CONNECT request.Mitmproxy responds with a 200 Connection Established, as if it has set up the CONNECT pipe.The client believes it’s talking to the remote server, and initiates the SSL connection. It uses SNI to indicate the hostname it is connecting to.Mitmproxy connects to the server, and establishes an SSL connection using the SNI hostname indicated by the client.The server responds with the matching SSL certificate, which contains the CN and SAN values needed to generate the interception certificate.Mitmproxy generates the interception cert, and continues the client SSL handshake paused in step 3.The client sends the request over the established SSL connection.wMitmproxy passes the request on to the server over the SSL connection initiated in step 4.

Transparent HTTP

When a transparent proxy is used, the HTTP/S connection is redirected into a proxy at the network layer, without any client configuration being required. This makes transparent proxying ideal for those situations where you can’t change client behaviour - proxy-oblivious Android applications being a common example.

To achieve this, we need to introduce two extra components. The first is a redirection mechanism that transparently reroutes a TCP connection destined for a server on the Internet to a listening proxy server. This usually takes the form of a firewall on the same host as the proxy server - iptables on Linux or pf on OSX. Once the client has initiated the connection, it makes a vanilla HTTP request, which might look something like this:

GET /index.html HTTP/1.1

Note that this request differs from the explicit proxy variation, in that it omits the scheme and hostname. How, then, do we know which upstream host to forward the request to? The routing mechanism that has performed the redirection keeps track of the original destination for us. Each routing mechanism has a different way of exposing this data, so this introduces the second component required for working transparent proxying: a host module that knows how to retrieve the original destination address from the router. In mitmproxy, this takes the form of a built-in set of modules that know how to talk to each platform’s redirection mechanism. Once we have this information, the process is fairly straight-forward.

The client makes a connection to the server.The router redirects the connection to mitmproxy, which is typically listening on a local port of the same host. Mitmproxy then consults the routing mechanism to establish what the original destination was.Now, we simply read the client’s request...... and forward it upstream.    

Transparent HTTPS

The first step is to determine whether we should treat an incoming connection as HTTPS. The mechanism for doing this is simple - we use the routing mechanism to find out what the original destination port is. By default, we treat all traffic destined for ports 443 and 8443 as SSL.

From here, the process is a merger of the methods we’ve described for transparently proxying HTTP, and explicitly proxying HTTPS. We use the routing mechanism to establish the upstream server address, and then proceed as for explicit HTTPS connections to establish the CN and SANs, and cope with SNI.

The client makes a connection to the server.The router redirects the connection to mitmproxy, which is typically listening on a local port of the same host. Mitmproxy then consults the routing mechanism to establish what the original destination was.The client believes it’s talking to the remote server, and initiates the SSL connection. It uses SNI to indicate the hostname it is connecting to.Mitmproxy connects to the server, and establishes an SSL connection using the SNI hostname indicated by the client.The server responds with the matching SSL certificate, which contains the CN and SAN values needed to generate the interception certificate.Mitmproxy generates the interception cert, and continues the client SSL handshake paused in step 3.The client sends the request over the established SSL connection.Mitmproxy passes the request on to the server over the SSL connection initiated in step 4.

Modes of Operation

Mitmproxy has four modes of operation that allow you to use mitmproxy in a variety of scenarios:

Regular (the default)TransparentReverse ProxyUpstream Proxy

Now, which one should you pick? Use this flow chart: