Microsoft Challenge Authentication Handshake Protocol (MS-CHAP) in Virtual Private Network (VPN)

ChungShun Wei

002923077

Mar. 24, 2005 

1. Introduction

Afterthe telephone has been invented, one can just dial numbers on the telephone andtalk to another who may be living on a different continent from the caller. Thismay seem to be an evolutionary invention 60 years ago. However, today, the Internetcan perform an even powerful communication channel between two end pointsworldwide. Not only the Internet can deliver voice, but it can also transmitall formats of data, such as image, text, and video. The Internet providestremendous convenience for people to communicate with each other around theworld. The Internet also allows businessman to join a conference meeting or toaccess company’s private networks without physically being in the companyoffice. The technology behind this private networking access is called VirtualPrivate Network (VPN) which can authenticate remote users and encrypt the datatraveling through the channel. VPN technology utilizes the Internet to carrydata packets from the source point to the destination host. This seems to behighly insecure when company employees may send confidential informationthrough the public internet. This paper will discuss and analyze MicrosoftChallenge Handshake Authentication Protocol (MS-CHAP) which is theauthentication protocol that Microsoft Windows uses to protect the datatraveling between remote users and VPN servers.

2. VirtualPrivate Network (VPN)

VirtualPrivate Network (VPN) can be seen as a private network that is restricted to outsideaccess for security and confidentiality reasons. VPN is often being deployed bycorporations to establish company private networks and allow remote employeeswho are not physically sitting in local company offices to access the company’sprivate network and logically equal to be connecting to the company’s privatenetwork locally. VPN is also used when two branches of the same company wishesto share information. They can use VPN tunneling through Wide Area Network(WAN) and form a logically equivalent private Local Area Network (LAN) connectionbetween these two branches shown in Figure 2.1.

Ifa remote employee/client wants to connect to a VPN. There are two methods forthe client to initiate the connection. The first method is through a modem onthe client machine. The client can dial a number to a Remote Access Server (VPNserver) that will generate a dedicated phone line connection from the clientmachine to the company’s private network. Then, the data will be sent throughthis dedicated phone line. This is usually called Remote Access Service. Thedisadvantage of this method is that the client sometimes will require dialing along-distance phone call to get to the VPN server. Also, each client that isconnected to the VPN server requires a direct phone line connection. Thismethod is too costly and resource-consuming. The second method is to use a currentexisting internet service from Internet Service Provider (ISP) and connect theclient’s machine to the company’s VPN shown in Figure 2.2. The data are no longtraversing through a phone line; instead, they are routed through the Internetto the VPN server. The advantage of this method is that clients can avoidcostly long-distance phone calls. Also, there is no dedicated phone linerequired for each client who connects to VPN [H02].

 

 

 

 

 

Figure 2.1 Virtual Private Network (VPN) Connecting Two Company Branches

Figure 2.2 A VPN Connection Connecting a Remote Client to a Private Intranet

VPNbasically provides four properties:

1. Encapsulation
2. Authentication
3. Data encryption
4. Address and name server allocation

VPNauthentication is the main focus in the article. Next section, we are going todiscuss in detail on two VPN authentication protocols—Password AuthenticationProtocol (PAP) and Challenge Handshake Authentication Protocol (CHAP). They areboth using symmetric key system to authenticate remote clients. The userinformation including passwords is usually stored on a database server—RemoteAuthentication Dial-in User Service (RADIUS)—sitting locally with the VPNserver [H02].

3.Authentication Protocol

3.1 PasswordAuthentication Protocol (PAP)

Aftera connection is established between the client and the VPN server, the VPNserver can request the connection to use Password Authentication Protocol (PAP)for authentication. Next, the client has to send his/her username and passwordin clear text to the VPN server. This protocol is highly insecure to allowpassword traveling through public internet without encryption. The reason thatany client should ever use PAP is when the client dials into a VPN server thatonly understand clear-text password or don’t support any encrypted passwordprotocol [HR]. Because of this drawback, we are going to move our attention toa more secure authentication protocol—Challenge Handshake AuthenticationProtocol (CHAP).

3.2 ChallengeHandshake Authentication Protocol (CHAP)

ChallengeHandshake Authentication protocol (CHAP) is considered more secure thanPassword Authentication Protocol (PAP) because it encrypts the password as wellas the data. One of the requirements of CHAP is that both end points must havea shared secret key before they can start transmitting any passwords or data.Actually, there are many versions of CHAP existed today. Their fundamental methodologyis similar to one another, but some improvements on security can be discussedamong them. CHAP is the most basic Challenge Handshake Authentication Protocolwhich is not as popular as Microsoft Challenge Handshake AuthenticationProtocol (MS-CHAP). MS-CHAP is basically used in all Windows operating system.MS-CHAP currently has two versions. MS-CHAP version 1 (MS-CHAPv1) has beenfound some serious security vulnerabilities. Therefore, MS-CHAP version 2(MS-CHAPv2) is the most popular authentication protocol used in Windowsoperating system today [HR]. Since MS-CHAPv2 is the most up-to-date CHAPprotocol, we are going to analyze it and will introduce the improvements onMS-CHAPv2 over MS-CHAPv1.

3.3 MicrosoftChallenge Authentication Handshake Protocol, Version 2 (MS-CHAPv2)

Sinceall Challenge Authentication Handshake Protocols are similar with someimprovements on security issues, we will focus our attention to MS-CHAPv2 anddiscuss some differences and improvements between MS-CHAPv1 and MS-CHAPv2. Let’sfirst look at the authentication procedure of MS-CHAPv2 [SB99].

1. Client requests a login challenge from the server.
2. The server generates a 16-byte random challenge and sends to the client. (In MS-CHAPv1, an 8-byte challenge is generated instead of 16-byte challenge).
3. On receipt of the 16-byte random challenge from the server, the client generates its own 16-byte random challenge called Peer Authenticator Challenge.
4. With the 16-byte random challenge from server, 16-byte Peer Authenticator Challenge, and the client’s username, the client now concatenates these three values and feed to a SHA-1 hash function. The first 8 byte of the output value from the SHA-1 hash function will be picked out as the 8-byte challenge. A graphic illustration is shown in Figure 3.1.
5. With the 8-byte challenge, the client continues using this challenge and Windows NT hash function to create a 24-byte reply. This computation will also be described later. (This part is identical to MS-CHAPv1 except that MS-CHAPv1 uses both LAN Manager and Windows NT hash functions to hash the client’s password twice and concatenate two 24-byte hash values into a 48-byte reply. Since LAN Manager hash function is considered much weaker than Windows NT hash function. MS-CHAPv2 eliminates the step of hashing client’s password with LAN Manager hash function).
6. The client then sends the 24-byte reply together with the Peer Authenticator Challenge from Step 3.
7. By retrieving the client’s password stored in database based on the client’s username, the server can decrypt the 24-byte reply and compare with the 8-byte challenge. If they are matched, the client is authenticated.
8. The server then uses the 16-byte Peer Authenticator Challenge and the client’s hashed password to create a 20-byte Authenticator Response.
9. After receiving the Authenticator Response, the client computes its own Authenticator Response and compares with the Authenticator Response from the server. If they are matched, the server is authenticated.

 

Figure 3.1 Generating 8-byte Challenge

 

Althoughwe have seen improvements that MS-CHAPv2 has done over MS-CHAPv1, there arestill security and design loopholes that allow attackers to obtain client’spassword. Now, we will look closely at some of these potential security vulnerabilitiesin MS-CHAPv2.

3.3.1 Generating8-byte Challenge

First,let’s explore how the 8-byte challenge in step 4 was actually generated in bothMS-CHAPv1 and MS-CHAPv2. In MS-CHAPv1, this 8-byte challenge is randomlygenerated by the server and sent to the client. Clearly, this is a security problemwhich allows eavesdroppers to acquire this 8-byte challenge by simply listeningto the channel between the server and the client. In contrast, MS-CHAPv2 doesnot require the 8-challenge to be transmitted from the server to the client. Itutilized a server’s 16-byte random challenge and a client’s 16-byte PeerAuthenticator Challenge, and the client’s username to generate the 8-bytechallenge at the client site. This can prevent eavesdroppers to obtain this8-byte challenge. Although this may seem to be a complicated computation andhighly secure design, this step is just a waste in security perspective. Instep 6, the client is actually sending the 24-byte reply together with the PeerAuthenticator Challenge to the server. If the eavesdropper is listening on thechannel from step 2 to step 6, he/she can still find out the 8-byte challenge bycomputing it himself/herself with all the information obtained from the channel.As a result, this improvement of MS-CHAPv2 does not seem to be any securer thanMS-CHAPv1 [SB99].

3.3.2 Generating24-byte MS-CHAP reply

Fromthe step 5 above, the client uses his/her password and the 8-byte challengegenerated from step 4 to create a 24-byte MS-CHAP reply. In this section, thedetail procedure of generating this 24-byte MS-CHAP reply will be examined. Figure3.2 also illustrates this process. First, the client feeds his/her password to WindowsNT hash function and will obtain a 16-byte output value. This 16-byte output valuewill then be padded with five zero bytes to 21 bytes long. Next, the 21-bytestring will be cut into three 7-byte DES keys. In Figure 3.2, I called thesethree keys as DES key 1, DES key 2, DES key 3. Each of these keys will be usedto encrypt the 8-byte challenge. Finally, these three encrypted blocks will beconcatenated into a 24-byte MS-CHAP reply.

 

Figure 3.2 Generating 24-byte MS-CHAPreply

This24-byte MS-CHAP reply is generated through a complicated algorithm. It is alsolong enough to be secure. However, 192 bit long does not mean that the attackermust try all 2¹⁹² possible DES key values to decrypt this MS-CHAPreply. Remember that the last 5 bytes of the 24-byte MS-CHAP reply are allzeros. That is, DES key 3 only has 2 bytes for the attacker to guess. This onlytakes 2¹⁶ tries to find out the real key value. Although there arestill 14 bytes left, it does not take 2¹¹² tries to search the DESkey. Because DES key1 and DES key 2 are used separately to encrypt thechallenge. Only 2 * 2⁵⁶ tries is needed to obtain both DES key 1 andDES key 2. As a result, the total efforts that a attacker requires to find outall three DES keys are (2¹⁶ + 2⁵⁷). Besides using bruteforce method, the attacker could easily speed up the search by dictionary keysearch. Since human passwords are often related or derived from a word orphrase in dictionary. By trying dictionary words as passwords may helpdecreasing the time required to break the password. As a matter of fact, anexperiment was conducted to break the password using a single EFF DES Crackermachine in 9 days. In the article,Cryptanalysis of Microsoft's PPTPAuthentication Extensions (MS-CHAPv2), Schneier and Mudge suggested asimple method to avoid this serious weakness. That is, to generate the MS-CHAPreply by using SHA-1 hash function to hash both 16-byte output from Windows NThash and the 8-byte challenge [SB99]. This technique would be enough to preventthis kind of attack.

4. Conclusion

VPNis the most popular methodology that corporations prefer to allow employees orclients accessing company’s private network. It is extremely important thatdata traveling between the employees and the VPN server is secure becauseconfidential information may be sent through this VPN channel. Because of thesignificance of transmitting confidential data through VPN, Microsoft hasdeveloped its proprietary version of Challenge Handshake AuthenticationProtocol (CHAP) called Microsoft Challenge Handshake Authentication Protocol(MS-CHAP) which allows a VPN server to authenticate its clients. MS-CHAP hastwo versions. Because version 1 was found to have many securityvulnerabilities, Microsoft released MS-CHAP version 2 with securityimprovements over MS-CHAPv1. Although today most of the Windows operatingsystems are using MS-CHAPv2 as the authentication protocol to talk to the WindowsVPN server, researchers still find vulnerabilities on MS-CHAPv2 and actuallyran an experiment to break it. Nevertheless, authentication phase is only apart of security protection in VPN. With the encryption and other securitytechnology that VPN provides, VPN is still the most favorable method that companiesused for their remote access.

5. References

[AZ97] Alexander,Zubair. Is RAS Safe. December 1997.<http://www.windowsitpro.com/Article/ArticleID/135/135.html?Ad=1>.

[FM97] Ford,Merilee; Lew, H. Kim; Spanier, Steve; Stevenson, Tim; Inc Cisco Systems.InternetworkingTechnologies Handbook. Cisco Press. March 1, 1997.

[HR] Haden,Rhys. Data Network Resource: Point to Point Protocol (PPP).<http://www.rhyshaden.com/ppp.htm>. 

[H02] Homebase.com.Virtual Private Networking. Technical References. Nov. 8, 2002 <http://endeavorquest.net:8880/knowbase/vpnsetup.htm>.

[SB98] Schneier,Bruce; Mudge, P. Cryptanalysis of Microsoft's Point-to-Point tunnelingprotocol (PPTP). Technical report, Counterpane Systems, June 1998.<http://citeseer.ist.psu.edu/article/schneier98cryptanalysis.html>.

[SB99] Schneier,Bruce; Mudge, P. Cryptanalysis of Microsoft's PPTP Authentication Extensions(MS-CHAPv2). Technical report, Counterpane Systems, October 19, 1999.<http://www.schneier.com/paper-pptpv2.html>.

[SW96] Simpson W. RFC 1994: PPP Challenge HandshakeAuthentication Protocol (CHAP). August 1996<http://www.cse.ohio-state.edu/cgi-bin/rfc/rfc1994.html>.