HttpWebRequest对象池 HTTP协议 HttpWebRequest和 Socket的一点总结
来源：互联网 发布：win7软件字体乱码 编辑：程序博客网 时间：2024/05/17 04:13
相信接触过网络开发的人对HTTP、HttpWebRequest、Socket这些东西都不陌生吧。它们之间的一些介绍和关系我这里都忽略了。开我们平时开发过程中也是很少有机会接触大什么大并发这个东东，一般大并发我们都需要考虑异步和多线程以及对象池，这个我们以一个简单demo来讲解吧。
主要的调用关系图如下：
类的结构图如下：
一：这里我们依次对这些类做一个简单的说明
HttpRequestInfo：
public string Url：http请求的url字符串，如http://www.baidu.com/
public byte[] PostData：Post请求中的数据
public WebHeaderCollection Headers：请求的头部数据
public bool AllowAutoRedirect ：是否允许301、302自动跳转，如果你想获取请求返回的头部信息，建议一般设置为false
public Dictionary<string, string> ExternalData ：请求过程中附加的数据（如数据记录的ID），便于在成功或失败回调函数中调用
public Action<HttpContextInfo> ActionCompleted ：请求成功后所调用的函数
public Action<HttpRequestException> ActionException：请求失败所调用函数
public HttpRequestInfo Clone()：返回当前对象的一个副本。
HttpResponseInfo：
public Stream ResponseContent ：Http请求返回内容（除头部信息）的对象流
public HttpStatusCode StatusCode：Http返回的状态
public string StatusDescription ：Http状态描述
public WebHeaderCollection Headers：Http返回的头部信息
public string GetString(Encoding coding)：把http返回体中数据流转换为字符串，转换编码就是我们所传参数。
public interface IHttpRequest
{
void GetResponseAsync(HttpRequestInfo request);
bool IsBusy { set; get; }
}
在IHttpRequest接口中，IsBusy属性主要是判断当前对象是否正在使用中，GetResponseAsync方法是真正完成Http请求的方法。
这里我们主要看看HttpRequestFactory的封装吧，管理对象实例的个数，相当于一个对象池，这里的代码主要是基于。net framework2.0的，
首先我们需要2个集合分别管理HttpRequestInfo请求实例和IHttpRequest处理请求实例，
static Queue<HttpRequestInfo> requestTask = new Queue<HttpRequestInfo>();
static List<IHttpRequest> Handlers = new List<IHttpRequest>();
而我们暴露给外部的一个主要方法就是AddRequestTask，
public static void AddRequestTask(HttpRequestInfo info)
{
if (!string.IsNullOrEmpty(info.Url))
{
lock (Lockobj)
{
Interlocked.Increment(ref requestCount);
requestTask.Enqueue(info);
}
}
}
那么这些请求在什么时候被处理了，在一个叫Process方法中处理，
private static void Process(object obj) { while (true) { IHttpRequest handler = GetAvailableHttpRequest(); while (handler == null) { Thread.Sleep(100); handler = GetAvailableHttpRequest(); } HttpRequestInfo task = GetTask(); while (task == null) { Thread.Sleep(100); task = GetTask(); } if (task != null && handler != null) { Interlocked.Decrement(ref requestCount); handler.GetResponseAsync(task); } // Thread.Sleep(10); } }
在这个方法中我们需要调用GetAvailableHttpRequest来获取IHttpRequest处理对象实例，调用GetTask来获取HttpRequestInfo请求实例。如果这2个实例都存在我们调用 IHttpRequest.GetResponseAsync(HttpRequestInfo);方法开始处理http请求。
GetAvailableHttpRequest如下：
private static IHttpRequest GetAvailableHttpRequest() { lock (Lockobj) { for (int i = 0; i < Handlers.Count; i++) { if (!Handlers[i].IsBusy) { return Handlers[i]; } } if (Handlers.Count <= MaxRequestCount) { IHttpRequest handler = (IHttpRequest)Activator.CreateInstance(_httpRequestType); Handlers.Add(handler); return handler; } } return null; //return GetAvailableHttpRequest(); }
在GetAvailableHttpRequest方法中，我们首先在处理对象集合中查找是否有空闲对象，如果有就返回，否则检查当前对象实例个数个数是否达到最大个数，如果没有我们则创建新实例，且加入到集合中，再返回，否者返回null。所以在Process方法中有一个检查，看啊看你返回的IHttpRequest是否为null，请注意这里一定不要用递归来返回有效的IHttpRequest，建议用一个死循环来处理，如果用递归一般不会出现什么问题，但是递归层次嵌套太深就会出现栈溢出错误，我在测试的时候曾经出现过这个问题。GetTask和GetAvailableHttpRequest处理一样。
那么这里的Process方法有在什么地方调用了，在HttpRequestFactory的静态构造函数中调用。
static HttpRequestFactory()
{
MaxRequestCount = 10;
ThreadPool.QueueUserWorkItem(new WaitCallback(Process));
}
到这里我们的一个对象池就构造玩了。
二 现在我们来看看RequestHttpWebRequest是如何处理HTTP请求的。它主要使用HttpWebRequest来处理请求。
这里我们主要使用HttpWebRequest的异步方法，因此我们需要构造一个状态对象StateObjectInfo
class StateObjectInfo : HttpContextInfo
{
internal byte[] Buffer { set; get; } //把返回的流写到HttpResponseInfo.ResponseContent 时用到的暂存数组
internal Stream ReadStream { set; get; }//把返回的流写到HttpResponseInfo.ResponseContent
internal HttpWebRequest HttpWebRequest { set; get; }
internal RequestHttpWebRequest RequestHandler { set; get; }//主要便于后面改IsBusy属性。
}
其GetResponseAsync实现如下：
public void GetResponseAsync(HttpRequestInfo info) { HttpWebRequest webRequest; StateObjectInfo state; InitWebRequest(info, out webRequest, out state); try { if (IsHttpPost) { webRequest.Method = "POST"; webRequest.ContentType = "application/x-www-form-urlencoded"; webRequest.BeginGetRequestStream(EndRequest, state); } else { webRequest.BeginGetResponse(EndResponse, state); } } catch (Exception ex) { HandException(ex, state); } }
其中InitWebRequest的实现如下：
private void InitWebRequest(HttpRequestInfo info, out HttpWebRequest webRequest, out StateObjectInfo state) { IsBusy = true; if (info.PostData != null && info.PostData.Length > 0) { IsHttpPost = true; } else { IsHttpPost = false; } if (info.Url.ToLower().Trim().StartsWith("https")) { IsHttps = true; ServicePointManager.ServerCertificateValidationCallback = new RemoteCertificateValidationCallback(CheckValidationResult); ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3; } else { IsHttps = false; } webRequest = HttpWebRequest.CreateDefault(new Uri(info.Url)) as HttpWebRequest; if (IsHttps) { /*基础连接已经关闭: 发送时发生错误 */ /*无法从传输连接中读取数据: 远程主机强迫关闭了一个现有的连接*/ webRequest.KeepAlive = false; webRequest.ProtocolVersion = HttpVersion.Version10; webRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)"; } webRequest.AllowAutoRedirect = info.AllowAutoRedirect; if (info.Headers != null && info.Headers.Count > 0) { foreach (string key in info.Headers.Keys) { webRequest.Headers.Add(key, info.Headers[key]); } } //webRequest.Proxy = WebProxy.GetDefaultProxy(); //webRequest.Proxy.Credentials = CredentialCache.DefaultCredentials; //webResponse.Headers.Get("Set-Cookie"); state = new StateObjectInfo { Buffer = new byte[1024 * 1024], HttpWebRequest = webRequest, RequestHandler = this, RequestInfo = info, ResponseInfo = new HttpResponseInfo() }; }
关于该类的EndRequest、EndResponse我想就没什么说的了，其中ReadCallBack的实现如下：
void ReadCallBack(IAsyncResult ar) { StateObjectInfo state = ar.AsyncState as StateObjectInfo; try { int read = state.ReadStream.EndRead(ar); if (read > 0) { state.ResponseInfo.ResponseContent.Write(state.Buffer, 0, read); state.ReadStream.BeginRead(state.Buffer, 0, state.Buffer.Length, ReadCallBack, state); } else { state.ReadStream.Close(); state.HttpWebRequest.Abort(); if (state.RequestInfo.ActionCompleted != null) { state.ResponseInfo.ResponseContent.Seek(0, SeekOrigin.Begin); state.RequestInfo.ActionCompleted(state); } state.Buffer = null; state.RequestHandler.IsBusy = false; } } catch (Exception ex) { HandException(ex, state); } }
这里还有一个HandException方法需要我们注意：
private void HandException(Exception ex, StateObjectInfo state) { if (state.ReadStream != null) state.ReadStream.Close(); if (state.HttpWebRequest != null) state.HttpWebRequest.Abort(); state.Buffer = null; if (state.RequestInfo.ActionException != null) { state.RequestInfo.ActionException(new HttpRequestException(state, ex)); } state.RequestHandler.IsBusy = false; }
这里我们在使用HttpWebRequest的时候，在完成使用后一定要关闭请求流。
在我们来看看一个简单的调用把：
public static void DownLoadFile(string remoteurl, string destinationFilePath, string id) { try { if (HasIllegalCharacters(destinationFilePath, false)) { SetFileCopyed(id, "400", "HasIllegalCharacters"); return; } DirectoryInfo dir = new DirectoryInfo(destinationFilePath); FileInfo destinationFile = new FileInfo(destinationFilePath); if (!destinationFile.Directory.Exists) { destinationFile.Directory.Create(); } HttpRequestInfo request = new HttpRequestInfo(remoteurl); request.ActionCompleted = new Action<HttpContextInfo>(x => { if (x.ResponseInfo.StatusCode == HttpStatusCode.OK) { using (Stream wr = File.Open(destinationFilePath, FileMode.OpenOrCreate, FileAccess.Write), sr = x.ResponseInfo.ResponseContent) { byte[] data = new byte[1024 * 1024]; int readcount = sr.Read(data, 0, data.Length); while (readcount > 0) { wr.Write(data, 0, readcount); readcount = sr.Read(data, 0, data.Length); } } SetFileCopyed(id, "200", string.Empty); } else { SetFileCopyed(id, ((int)x.ResponseInfo.StatusCode).ToString(), x.ResponseInfo.StatusDescription); string message = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + " : " + remoteurl + " : " + x.ResponseInfo.StatusDescription; LogManager.LogException(message); } }); request.ActionException = new Action<HttpRequestException>(ex => { Regex reg = new Regex(@"\d{3}",RegexOptions.Compiled); string message = ex.Message; Match m = reg.Match(message); if (m.Success) { SetFileCopyed(id, m.Value, message); } else { SetFileCopyed(id, "503", message); HttpRequestInfo newRequest = ex.HttpContextInfo.RequestInfo.Clone(); request.ActionCompleted = null; request.ActionException = null; HttpRequestFactory.AddRequestTask(newRequest); } message = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + " : " + ex.HttpContextInfo.RequestInfo.Url + " : " + message; LogManager.LogException(message); }); HttpRequestFactory.AddRequestTask(request); } catch (Exception ex) { SetFileCopyed(id, "-1", ex.Message); string message = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + " : " + remoteurl + " : " + ex.Message; LogManager.LogException(message); } } internal static bool HasIllegalCharacters(string path, bool checkAdditional) { for (int i = 0; i < path.Length; i++) { int num2 = path[i]; if (((num2 == 0x22) || (num2 == 60)) || (((num2 == 0x3e) || (num2 == 0x7c)) || (num2 < 0x20))) { return true; } if (checkAdditional && ((num2 == 0x3f) || (num2 == 0x2a))) { return true; } } return false; }
对于这个调用的demo我这里就不多说，不过在调用的时候偶尔会出现：
/*基础连接已经关闭: 发送时发生错误 */
/*无法从传输连接中读取数据: 远程主机强迫关闭了一个现有的连接*/
这样的错误，网上有一些什么改良方法，我测试后都不管用，个人猜测是与网络有关的，即使我用socket来做偶尔也会有一些问题。所以当我们遇到这些网络问题的时候，我们把我们的请求再次加入请求队列中 HttpRequestFactory.AddRequestTask(newRequest);。这一很重要的哦。
HttpWebRequest类对我们做http请求做了很多封装，我们使用也很方便。但是它的性能比我们自己用socket要低很多，同时在一些处理上违背了我们的操作习惯。如我们上面的调用代码：
如果我们http返回状态是403、404...（除200以外）程序没有进入我的else，而是进入我的ActionException方法里面了，这点让我很是不爽。于是萌生了用socket来做http请求的念头。
三 现在我们来看看SocketHttpRequest是如何处理HTTP请求的。它主要使用Socket来处理请求。
SocketHttpRequest和RequestHttpWebRequest一样都是采用对象的异步模式，那么也需要一个状态对象：
class RequestSockeStateObject : HttpContextInfo
{
internal SocketHttpRequest RequestHandler { set; get; }
internal Socket _socket { set; get; } //普通http请求采用socket
internal List<byte> HeaderBuffer { set; get; }
internal byte[] Buffer { set; get; }
internal int ContentLength { set; get; }//http需要接收的数据长度
internal int ReceiveLength { set; get; }//http已经接收的数据长度
internal SslStream SslStream { set; get; }//https请求采用TcpClient，这里需要用到SslStream
}
public void GetResponseAsync(HttpRequestInfo info)
{
RequestSockeStateObject _state;
InitRequestSockeStateObject(info, out _state);
SocketConnection(_state);
}
这里的InitRequestSockeStateObject和RequestHttpWebRequest的InitWebRequest方法差不多，就不在累赘了。
主要看看SocketConnection方法：
void SocketConnection(RequestSockeStateObject _state) { try { Uri uri = new Uri(_state.RequestInfo.Url); IPHostEntry hostEntry = Dns.GetHostEntry(uri.Host); if (IsHttps) { TcpClient tcpclient = new TcpClient(); tcpclient.Connect(hostEntry.AddressList, uri.Port); _state._socket = tcpclient.Client; SslStream sslStream = new SslStream(tcpclient.GetStream(), false, new RemoteCertificateValidationCallback(ValidateServerCertificate), null); sslStream.AuthenticateAsClient(hostEntry.HostName, new X509CertificateCollection(), SslProtocols.Ssl3 | SslProtocols.Tls, false); _state.SslStream = sslStream; Begin_Write(_state); } else { Socket client = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp); client.Connect(hostEntry.AddressList, uri.Port); _state._socket = client; BeginSend(_state); } } catch (Exception ex) { HandException(ex, _state); } }
socket连接是需要IP和端口的，这里我们借用 Uri来获取所需端口，但是一台计算机的ip可能有很多个，实际只有一两个可以连接，所以我们这里需要调用 client.Connect(hostEntry.AddressList, uri.Port)方法，传递一个ip集合。如果是https的话，直接用socket我没有搞定，最后用SslStream搞定，不知道大家有没有其他方法。
其中Begin_Write、End_Write、Complete_Read方法是sslStream异步中所必需的方法，BeginSend、Send_Completed、Receive_Completed、RepeatReceive是socket异步中所需方法。其中Complete_Read和Receive_Completed方法相似。
protected virtual void Complete_Read(IAsyncResult ar) { RequestSockeStateObject state = ar.AsyncState as RequestSockeStateObject; try { int byteCount = state.SslStream.EndRead(ar); if (state.ResponseInfo.Headers.Count < 1) { SetResponseHeaders(state, byteCount); if ((state.ReceiveLength == state.ContentLength && state.ContentLength > 0)) { EndReceive(state); } else { state.SslStream.BeginRead(state.Buffer, 0, state.Buffer.Length, Complete_Read, state); } } else { if (byteCount > 0 && byteCount==state.Buffer.Length) { state.ResponseInfo.ResponseContent.Write(state.Buffer, 0, byteCount); state.SslStream.BeginRead(state.Buffer, 0, state.Buffer.Length, Complete_Read, state); } else { state.ResponseInfo.ResponseContent.Write(state.Buffer, 0, byteCount); EndReceive(state); } } } catch (Exception ex) { HandException(ex, state); } }
如果是第一次接收数据流，我们必须把数据流中http头部信息取出来，再把头部信息以外的数据流写到HttpResponseInfo.ResponseContent中，如果我们已经接收的数据等于我们需要接收的数据，就表示我们已经接收完毕了。如果没有就继续接收数据。在第二次及以后所接收数据过程中，我们需要判断接收数据长度是否小于接收数组的长度，如果小于就表示接收完毕了，否则继续接收。这里的EndReceive方法如下：
void EndReceive(RequestSockeStateObject state) { /* * if (state.RequestInfo.AllowAutoRedirect && (state.ResponseInfo.StatusCode == HttpStatusCode.Found || state.ResponseInfo.StatusCode == HttpStatusCode.MovedPermanently)) { string location = state.ResponseInfo.Headers["Location"]; state.RequestInfo.Url = location; state.RequestInfo.Headers = state.ResponseInfo.Headers; state.RequestInfo.Headers.Remove("Location"); state.RequestInfo.Headers.Add("Referer", location); Begin_Write(state); } */ if (IsHttps) { state.SslStream.Close(); state.SslStream = null; } else { state._socket.Shutdown(SocketShutdown.Both); state._socket.Close(); state._socket = null; } if (state.RequestInfo.ActionCompleted != null) { state.ResponseInfo.ResponseContent.Seek(0, SeekOrigin.Begin); state.RequestInfo.ActionCompleted(state); } state.RequestHandler.IsBusy = false; }
EndReceive方法主要是关闭socket或则SslStream数据流，然后调用ActionCompleted方法。在这里 state.ResponseInfo.ResponseContent.Seek(0, SeekOrigin.Begin);这个方法非常重要，不然在外面的调用方法就必须调用Stream.Seek(0, SeekOrigin.Begin)来吧数据流定位开始位置。
在SocketHttpRequest这个类中，我们是如何来获取发送的http请求信息以及如何解析http返回的header信息了？
首先来看一个GetRequestData方法，它主要是通过RequestInfo实例来获取请求信息：
byte[] GetRequestData(RequestSockeStateObject _state) { StringBuilder bufRequest = new StringBuilder(); Uri uri = new Uri(_state.RequestInfo.Url); if (!IsHttpPost) { bufRequest.Append("GET ").Append(uri.OriginalString).AppendLine(" HTTP/1.1"); } else { bufRequest.Append("POST ").Append(uri.OriginalString).AppendLine(" HTTP/1.1"); string contentLengthkey = "Content-Length"; string contentTypekey = "Content-Type"; List<string> headerKeys = new List<string>(_state.RequestInfo.Headers.AllKeys); if (headerKeys.Contains(contentLengthkey)) { _state.RequestInfo.Headers.Remove(contentLengthkey); } if (headerKeys.Contains(contentTypekey)) { _state.RequestInfo.Headers.Remove(contentTypekey); } _state.RequestInfo.Headers.Add(contentTypekey, "application/x-www-form-urlencoded"); _state.RequestInfo.Headers.Add(contentLengthkey, _state.RequestInfo.PostData.Length.ToString()); } _state.RequestInfo.Headers.Add("Host", uri.Host); _state.RequestInfo.Headers.Add("Connection", "keep-alive"); if (_state.RequestInfo.Headers.Count > 0) { bufRequest.Append(_state.RequestInfo.Headers.ToString()); } byte[] byteData = Encoding.ASCII.GetBytes(bufRequest.ToString()); if (!IsHttpPost) { return byteData; } else { byte[] sendData = new byte[byteData.Length + _state.RequestInfo.PostData.Length]; Array.Copy(byteData, 0, sendData, 0, byteData.Length); Array.Copy(_state.RequestInfo.PostData, 0, sendData, byteData.Length, _state.RequestInfo.PostData.Length); return sendData; } }
有关请求和header信息的字符串建议不要自己拼接，用WebHeaderCollection实例的ToString方法来生成，并且它把最后那个回车换行也生成。
那么如何更具换回的流中的数据来设置它的返回头信息了，这里我们有一个方法SetResponseHeaders：
void SetResponseHeaders(RequestSockeStateObject state, int bytesRead) { try { byte[] tempArray = new byte[bytesRead]; Array.Copy(state.Buffer, 0, tempArray, 0, bytesRead); state.HeaderBuffer.AddRange(tempArray); tempArray = state.HeaderBuffer.ToArray(); string headerSpilt = "\r\n\r\n"; byte[] headerbyte = Encoding.ASCII.GetBytes(headerSpilt); int contentindex = DestTag(tempArray, headerbyte, 0, tempArray.Length); if (contentindex > 0) { string headerStr = Encoding.ASCII.GetString(tempArray, 0, contentindex); int startIndex = contentindex + headerbyte.Length; SetResponseHeaders(headerStr, state); state.ReceiveLength = tempArray.Length - startIndex; state.ResponseInfo.ResponseContent.Write(tempArray, startIndex, tempArray.Length - startIndex); state.HeaderBuffer.Clear(); } } catch (Exception ex) { HandException(ex, state); } }
这里的bytesRead时我第一次接收的数据流长度，首先我们需要在返回流中找到连续的\r\n\r\n 信息，它前面是返回头信息，后面的时返回体信息。这里我们用自定义的DestTag方法来查找。SetResponseHeaders方法如下：
void SetResponseHeaders(string headerStr, RequestSockeStateObject state) { try { string[] headers = headerStr.Split(new string[] { "\r\n" }, StringSplitOptions.RemoveEmptyEntries); string statline = headers[0]; state.ResponseInfo.StatusCode = GetStatusCode(statline); for (int i = 1; i < headers.Length; i++) { int index = headers[i].IndexOf(":"); if (index > 1) { string key = headers[i].Substring(0, index); string value = headers[i].Substring(index + 1); state.ResponseInfo.Headers.Add(key.Trim(), value.Trim()); } } string contentLength = state.ResponseInfo.Headers["Content-Length"]; state.ContentLength = int.Parse(contentLength); state.ReceiveLength = 0; } catch (Exception ex) { HandException(ex, state); } }
以上就是这个类的主要方法,目前SocketHttpRequest对301、302暂不支持。
完整的代码如下：
namespace HttpRequest{ using System; using System.Collections.Generic; using System.Text; using System.Net; [Serializable] public class HttpRequestInfo { public HttpRequestInfo(string url) { Url = url; Headers = new WebHeaderCollection(); ExternalData = new Dictionary<string, string>(); } public string Url { set; get; } public byte[] PostData { set; get; } public WebHeaderCollection Headers { set; get; } public bool AllowAutoRedirect { set; get; } public Dictionary<string, string> ExternalData { set; get; } public Action<HttpContextInfo> ActionCompleted { set; get; } public Action<HttpRequestException> ActionException { set; get; } public HttpRequestInfo Clone() { HttpRequestInfo newobj = new HttpRequestInfo(this.Url) { AllowAutoRedirect = this.AllowAutoRedirect, ActionCompleted = this.ActionCompleted, ActionException = this.ActionException }; if (PostData != null && PostData.Length > 0) { newobj.PostData = new byte[this.PostData.Length]; Array.Copy(this.PostData, 0, newobj.PostData, 0, this.PostData.Length); } if (Headers.Count > 0) { foreach (string key in Headers.Keys) { newobj.Headers.Add(key, Headers[key]); } } if (ExternalData.Count > 0) { foreach (string key in ExternalData.Keys) { newobj.ExternalData.Add(key, ExternalData[key]); } } return newobj; } } public class HttpRequestException : Exception { public HttpRequestException(HttpContextInfo context, Exception ex) : base(ex.Message, ex) { HttpContextInfo = context; } public HttpContextInfo HttpContextInfo { set; get; } }}namespace HttpRequest{ using System; using System.Collections.Generic; using System.Text; using System.IO; using System.Net; [Serializable] public class HttpResponseInfo { public HttpResponseInfo() { ResponseContent = new MemoryStream(); Headers = new WebHeaderCollection(); } public Stream ResponseContent { set; get; } HttpStatusCode _statusCode; public HttpStatusCode StatusCode { set { _statusCode = value; StatusDescription = System.Web.HttpWorkerRequest.GetStatusDescription((int)_statusCode); } get { return _statusCode; } } public string StatusDescription { set; get; } public WebHeaderCollection Headers { set; get; } public string GetString(Encoding coding) { StringBuilder str = new StringBuilder(); Stream sr = ResponseContent; byte[] data = new byte[1024 * 100]; int readcount = sr.Read(data, 0, data.Length); while (readcount > 0) { str.Append(coding.GetString(data, 0, readcount)); readcount = sr.Read(data, 0, data.Length); } ResponseContent.Seek(0, SeekOrigin.Begin); return str.ToString(); } } public class HttpContextInfo { public HttpResponseInfo ResponseInfo { set; get; } public HttpRequestInfo RequestInfo { set; get; } } public interface IHttpRequest { void GetResponseAsync(HttpRequestInfo request); bool IsBusy { set; get; } }}namespace HttpRequest{ using System; using System.Collections.Generic; using System.Text; using System.IO; using System.Data; using System.Data.OleDb; using System.Net; using System.Xml; using System.Net.Security; using System.Security.Cryptography.X509Certificates; public class RequestHttpWebRequest : IHttpRequest { class StateObjectInfo : HttpContextInfo { internal byte[] Buffer { set; get; } internal Stream ReadStream { set; get; } internal HttpWebRequest HttpWebRequest { set; get; } internal RequestHttpWebRequest RequestHandler { set; get; } } private bool CheckValidationResult(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors errors) { //直接确认，否则打不开 return true; } public void GetResponseAsync(HttpRequestInfo info) { HttpWebRequest webRequest; StateObjectInfo state; InitWebRequest(info, out webRequest, out state); try { if (IsHttpPost) { webRequest.Method = "POST"; webRequest.ContentType = "application/x-www-form-urlencoded"; webRequest.BeginGetRequestStream(EndRequest, state); } else { webRequest.BeginGetResponse(EndResponse, state); } } catch (Exception ex) { HandException(ex, state); } } void EndRequest(IAsyncResult ar) { StateObjectInfo state = ar.AsyncState as StateObjectInfo; try { HttpWebRequest webRequest = state.HttpWebRequest as HttpWebRequest; using (Stream stream = webRequest.EndGetRequestStream(ar)) { byte[] data = state.RequestInfo.PostData; stream.Write(data, 0, data.Length); } webRequest.BeginGetResponse(EndResponse, state); } catch (Exception ex) { HandException(ex, state); } } void EndResponse(IAsyncResult ar) { StateObjectInfo state = ar.AsyncState as StateObjectInfo; try { HttpWebResponse webResponse = state.HttpWebRequest.EndGetResponse(ar) as HttpWebResponse; state.ResponseInfo.StatusCode = webResponse.StatusCode; state.ResponseInfo.StatusDescription = webResponse.StatusDescription; foreach (string key in webResponse.Headers.AllKeys) { state.ResponseInfo.Headers.Add(key, webResponse.Headers[key]); } state.ReadStream = webResponse.GetResponseStream(); state.ReadStream.BeginRead(state.Buffer, 0, state.Buffer.Length, ReadCallBack, state); } catch (Exception ex) { HandException(ex, state); } } void ReadCallBack(IAsyncResult ar) { StateObjectInfo state = ar.AsyncState as StateObjectInfo; try { int read = state.ReadStream.EndRead(ar); if (read > 0) { state.ResponseInfo.ResponseContent.Write(state.Buffer, 0, read); state.ReadStream.BeginRead(state.Buffer, 0, state.Buffer.Length, ReadCallBack, state); } else { state.ReadStream.Close(); state.HttpWebRequest.Abort(); if (state.RequestInfo.ActionCompleted != null) { state.ResponseInfo.ResponseContent.Seek(0, SeekOrigin.Begin); state.RequestInfo.ActionCompleted(state); } state.Buffer = null; state.RequestHandler.IsBusy = false; } } catch (Exception ex) { HandException(ex, state); } } private void InitWebRequest(HttpRequestInfo info, out HttpWebRequest webRequest, out StateObjectInfo state) { IsBusy = true; if (info.PostData != null && info.PostData.Length > 0) { IsHttpPost = true; } else { IsHttpPost = false; } if (info.Url.ToLower().Trim().StartsWith("https")) { IsHttps = true; ServicePointManager.ServerCertificateValidationCallback = new RemoteCertificateValidationCallback(CheckValidationResult); ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3; } else { IsHttps = false; } webRequest = HttpWebRequest.CreateDefault(new Uri(info.Url)) as HttpWebRequest; if (IsHttps) { /*基础连接已经关闭: 发送时发生错误 */ /*无法从传输连接中读取数据: 远程主机强迫关闭了一个现有的连接*/ webRequest.KeepAlive = false; webRequest.ProtocolVersion = HttpVersion.Version10; webRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)"; } webRequest.AllowAutoRedirect = info.AllowAutoRedirect; if (info.Headers != null && info.Headers.Count > 0) { foreach (string key in info.Headers.Keys) { webRequest.Headers.Add(key, info.Headers[key]); } } //webRequest.Proxy = WebProxy.GetDefaultProxy(); //webRequest.Proxy.Credentials = CredentialCache.DefaultCredentials; //webResponse.Headers.Get("Set-Cookie"); state = new StateObjectInfo { Buffer = new byte[1024 * 100], HttpWebRequest = webRequest, RequestHandler = this, RequestInfo = info, ResponseInfo = new HttpResponseInfo() }; } private bool IsHttpPost { set; get; } private bool IsHttps { set; get; } public bool IsBusy { set; get; } private void HandException(Exception ex, StateObjectInfo state) { if (state.ReadStream != null) state.ReadStream.Close(); if (state.HttpWebRequest != null) state.HttpWebRequest.Abort(); state.Buffer = null; if (state.RequestInfo.ActionException != null) { state.RequestInfo.ActionException(new HttpRequestException(state, ex)); } state.RequestHandler.IsBusy = false; } }}namespace HttpRequest{ using System; using System.Collections.Generic; using System.Text; using System.Threading; using System.Net; public class HttpRequestFactory { static HttpRequestFactory() { MaxRequestCount = 10; ThreadPool.QueueUserWorkItem(new WaitCallback(Process)); } static readonly object Lockobj = new object(); static long requestCount = 0; static Queue<HttpRequestInfo> requestTask = new Queue<HttpRequestInfo>(); static List<IHttpRequest> Handlers = new List<IHttpRequest>(); public static void AddRequestTask(HttpRequestInfo info) { if (!string.IsNullOrEmpty(info.Url)) { lock (Lockobj) { requestTask.Enqueue(info); Interlocked.Increment(ref requestCount); } } } private static IHttpRequest GetAvailableHttpRequest() { lock (Lockobj) { for (int i = 0; i < Handlers.Count; i++) { if (!Handlers[i].IsBusy) { return Handlers[i]; } } if (Handlers.Count <= MaxRequestCount) { IHttpRequest handler = (IHttpRequest)Activator.CreateInstance(_httpRequestType); Handlers.Add(handler); return handler; } } return null; //return GetAvailableHttpRequest(); } private static HttpRequestInfo GetTask() { HttpRequestInfo task = null; lock (Lockobj) { if (requestTask.Count > 0) { task = requestTask.Dequeue(); return task; } } return task; } private static void Process(object obj) { while (true) { IHttpRequest handler = GetAvailableHttpRequest(); while (handler == null) { Thread.Sleep(500); handler = GetAvailableHttpRequest(); } HttpRequestInfo task = GetTask(); while (task == null) { Thread.Sleep(500); task = GetTask(); } if (task != null && handler != null) { handler.GetResponseAsync(task); Interlocked.Decrement(ref requestCount); Thread.Sleep(50); } } } public static long TaskCount { get { return Interlocked.Read(ref requestCount); } } static int _maxRequestCount = 2; public static int MaxRequestCount { set { _maxRequestCount = value; ServicePointManager.DefaultConnectionLimit = _maxRequestCount * 2; } get { return _maxRequestCount; } } static Type _httpRequestType = typeof(RequestHttpWebRequest); public static void SetHttpRequestType(Type type) { if (type.IsClass && typeof(IHttpRequest).IsAssignableFrom(type)) { _httpRequestType = type; } } }}namespace HttpRequest{ using System; using System.Collections.Generic; using System.Text; using System.Threading; using System.Reflection; using System.IO; public class LogManager { static object lockobj = new object(); static Queue<string> Messages = new Queue<string>(); static long messageCount = 0; static LogManager() { ThreadPool.QueueUserWorkItem(new WaitCallback(Work)); } public static void LogException(Exception ex) { Type type = ex.GetType(); StringBuilder sb = new StringBuilder(); sb.AppendLine(type.ToString() + "--------" + DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss")); PropertyInfo[] properties = type.GetProperties(); foreach (PropertyInfo p in properties) { if (p.PropertyType == typeof(string)) { object msg = p.GetValue(ex, null); if (msg != null) sb.AppendLine(p.Name + ":" + msg.ToString()); } } lock (lockobj) { LogException(sb.ToString()); } } public static void LogException(string msg) { lock (lockobj) { Messages.Enqueue(msg); Interlocked.Increment(ref messageCount); } } static void Work(object obj) { if (!File.Exists(ExceptionLogFilePath)) { FileStream fs = File.Create(ExceptionLogFilePath); fs.Close(); } while (true) { if (MessageCount > 0) { string msg = string.Empty; lock (lockobj) { msg = Messages.Dequeue(); Interlocked.Decrement(ref messageCount); if (!string.IsNullOrEmpty(msg)) { using (StreamWriter sw = new StreamWriter(ExceptionLogFilePath, true, Encoding.UTF8)) { sw.Write(msg); sw.WriteLine(); sw.Flush(); } } } }//end if Thread.Sleep(500); } }//end static long MessageCount { get { return Interlocked.Read(ref messageCount); } } public static string ExceptionLogFilePath { set; get; } }}调用代码 ：
using System;using System.Collections.Generic;using System.Configuration;using System.Data;using System.Data.OleDb;using System.Data.SqlClient;using System.Diagnostics;using System.IO;using System.Net;using System.Reflection;using System.Text;using System.Text.RegularExpressions;using System.Threading;namespace HttpRequest{ class Program { static void Main(string[] args) { LogManager.ExceptionLogFilePath = ConfigurationManager.AppSettings["ExceptionLogFilePath"].Trim(); ConnectionString = ConfigurationManager.AppSettings["ConnectionString"].Trim(); string remoteHostUrl = ConfigurationManager.AppSettings["remoteHostUrl"].Trim(); string destinationBasePath = ConfigurationManager.AppSettings["destinationBasePath"].Trim(); HttpRequestFactory.MaxRequestCount = Convert.ToInt32(ConfigurationManager.AppSettings["MaxRequestCount"]); try { DateTime startTime = DateTime.Now; Console.WriteLine("Start Time:" + startTime.ToLongTimeString()); AsyncDownLoadFiles(remoteHostUrl, destinationBasePath, null, true); DateTime endtime = DateTime.Now; Console.WriteLine("End Time:" + endtime.ToLongTimeString()); TimeSpan tm = endtime - startTime; Console.WriteLine(tm.Hours.ToString() + " Hours " + tm.Minutes.ToString() + " Minutes " + tm.Seconds.ToString() + " Seconds"); } catch (Exception ex) { Console.WriteLine(ex.Message); } Console.ReadLine(); } public static void AsyncDownLoadFiles(string remoteHostUrl, string destinationBasePath, List<string> tables, bool download) { try { List<string> archiveTables = tables; if (archiveTables == null || archiveTables.Count < 1) { archiveTables = GetArchiveTables(); } foreach (string tablename in archiveTables) { string sql = "SELECT ID,file_name_path FROM dbo.Archive_Files WHERE TableName ='" + tablename + "' AND Copyed=0 ORDER BY ID ASC"; DataTable dt = GetData(sql); foreach (DataRow row in dt.Rows) { string id = row["ID"].ToString(); string file_name_path = row["file_name_path"].ToString(); file_name_path = file_name_path.Substring(2); if (download) { AsyncDownLoadFile(remoteHostUrl + file_name_path, destinationBasePath + file_name_path, id); } else { CheckFileExists(destinationBasePath + file_name_path, id); } } while (download) { if (HttpRequestFactory.TaskCount < HttpRequestFactory.MaxRequestCount) { break; } Thread.Sleep(10000); } }//end foreach while (download) { if (HttpRequestFactory.TaskCount < 1) { break; } Thread.Sleep(10000); } bool finishedDownLoad = HasFinishedDownLoad(); int times = 0; while (!finishedDownLoad && times < HttpRequestFactory.TaskCount) { Thread.Sleep(10000); finishedDownLoad = HasFinishedDownLoad(); times++; } } catch (Exception ex) { string message = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + " : " + ex.Message; LogManager.LogException(message); } } public static void AsyncDownLoadFile(string remoteurl, string destinationFilePath, string id) { try { if (HasIllegalCharacters(destinationFilePath, true)) { SetFileCopyed(id, "400", "HasIllegalCharacters"); return; } DirectoryInfo dir = new DirectoryInfo(destinationFilePath); FileInfo destinationFile = new FileInfo(destinationFilePath); if (!destinationFile.Directory.Exists) { destinationFile.Directory.Create(); } HttpRequestInfo request = new HttpRequestInfo(remoteurl); request.ActionCompleted = new Action<HttpContextInfo>(x => { try { if (x.ResponseInfo.StatusCode == HttpStatusCode.OK) { using (Stream wr = File.Open(destinationFilePath, FileMode.OpenOrCreate, FileAccess.Write), sr = x.ResponseInfo.ResponseContent) { byte[] data = new byte[1024 * 100]; int readcount = sr.Read(data, 0, data.Length); while (readcount > 0) { wr.Write(data, 0, readcount); readcount = sr.Read(data, 0, data.Length); } } SetFileCopyed(id, "200", string.Empty); } else { SetFileCopyed(id, ((int)x.ResponseInfo.StatusCode).ToString(), x.ResponseInfo.StatusDescription); } } catch (Exception ea) { SetFileCopyed(id, "-1", ea.Message); } }); request.ActionException = new Action<HttpRequestException>(ex => { try { Regex reg = new Regex(@"\d{3}", RegexOptions.Compiled); string message = ex.Message; Match m = reg.Match(message); if (m.Success) { SetFileCopyed(id, m.Value, message); } else { SetFileCopyed(id, "503", message); HttpRequestInfo newRequest = ex.HttpContextInfo.RequestInfo.Clone(); request.ActionCompleted = null; request.ActionException = null; HttpRequestFactory.AddRequestTask(newRequest); } } catch (Exception ea) { SetFileCopyed(id, "-1", ea.Message); } }); HttpRequestFactory.AddRequestTask(request); } catch (Exception ex) { SetFileCopyed(id, "-1", ex.Message); } } private static void CheckFileExists(string destinationFilePath, string id) { try { if (HasIllegalCharacters(destinationFilePath, true)) { SetFileCopyed(id, "400", "HasIllegalCharacters"); return; } FileInfo destinationFile = new FileInfo(destinationFilePath); if (destinationFile.Exists) { SetFileCopyed(id, "200", string.Empty); } else { SetFileCopyed(id, "404", "Not Found"); } } catch (Exception ex) { string message = ex.Message; SetFileCopyed(id, "-1", message); //message = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + " : " + destinationFilePath + " : " + message; //LogManager.LogException(message); } } internal static bool HasIllegalCharacters(string path, bool checkAdditional) { for (int i = 0; i < path.Length; i++) { int num2 = path[i]; if (((num2 == 0x22) || (num2 == 60)) || (((num2 == 0x3e) || (num2 == 0x7c)) || (num2 < 0x20)))//" ' > | space { return true; } if (checkAdditional && ((num2 == 0x3f) || (num2 == 0x2a)))//? * { return true; } } return false; } private static List<string> GetArchiveTables() { string sql = "SELECT DISTINCT TableName FROM dbo.Archive_Files ORDER BY TableName"; DataTable dt = GetData(sql); List<string> archiveTables = new List<string>(); foreach (DataRow row in dt.Rows) { archiveTables.Add(row["TableName"].ToString()); } return archiveTables; } static bool HasFinishedDownLoad() { string sql = "SELECT COUNT(*) FROM dbo.Archive_Files WITH(NOLOCK) WHERE Copyed=0"; return ExecuteScalar(sql) == 0; } private static bool SetFileCopyed(string id, string statusCode, string error) { string sql = string.Format("UPDATE dbo.Archive_Files SET Copyed={0}, CopyDate=GETDATE() ", statusCode); if (!string.IsNullOrEmpty(error)) { sql += string.Format(" ,CopyError='{0}' ", error); } sql += string.Format(" WHERE ID={0}", id); return ExecuteCmd(sql); } private static DataTable GetData(string sql) { DataTable dt = new DataTable(); try { using (SqlConnection con = new SqlConnection(ConnectionString)) { SqlDataAdapter adapter = new SqlDataAdapter(sql, con); adapter.Fill(dt); } } catch (Exception ex) { string message = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + " : " + ex.Message; LogManager.LogException(message); } return dt; } private static bool ExecuteCmd(string sql) { int resultCount = 0; try { using (SqlConnection connection = new SqlConnection(ConnectionString)) { SqlCommand cmd = new SqlCommand(sql, connection); connection.Open(); cmd.CommandTimeout = 2 * 60; resultCount = cmd.ExecuteNonQuery(); connection.Close(); } } catch (Exception ex) { string message = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + " : " + ex.Message; LogManager.LogException(message); } return resultCount > 0; } static int ExecuteScalar(string sql) { int result = 0; using (SqlConnection con = new SqlConnection(ConnectionString)) { SqlCommand cmd = new SqlCommand(sql, con); cmd.CommandTimeout = 2 * 60; con.Open(); result = Convert.ToInt32(cmd.ExecuteScalar()); con.Close(); } return result; } public static string ConnectionString { set; get; } }}- HttpWebRequest对象池 HTTP协议 HttpWebRequest和 Socket的一点总结
- HttpWebRequest
- HttpWebRequest
- HttpWebRequest
- HttpWebRequest
- HttpWebRequest
- HttpWebRequest
- HttpWebRequest
- httpwebrequest
- HttpWebRequest
- httpwebrequest的referer属性的一点认识
- WebClient 和 HttpWebRequest的区别
- HttpWebRequest 和 HttpWebResponse 的应用
- HttpRequest 和HttpWebRequest的区别
- 用socket来代替HttpWebRequest和HttpWebResponse
- c# HttpWebRequest post和get请求http
- 用socket代替httpwebrequest/httpwebresponse的研究
- HttpWebRequest 模拟Http请求
- hdu1014
- 终于可以在centos下使用QQ啦！
- 面向程序员的数据库访问性能优化法则
- java获取系统属性
- checkbox全选与反选
- HttpWebRequest对象池 HTTP协议 HttpWebRequest和 Socket的一点总结
- mac下直接运行python脚本
- ALSA 接口编程实例——语音聊天
- 在socket创建的时候使用SOCK_CLOEXEC设置close-on-exec
- 服务器开发必须要有个舒服的日志系统： erlang 日志管理系统： lager
- C#学习之结构
- HTML快速入门+常用标记
- 怎样学好hibernate！（http://edu.csdn.net/news/2012822/240ca26f0d7e.shtml）没事下载下来看哦！
- MFC Edit control用法
