[问题记录] curl: (18) transfer closed with outstanding read data remaining 原因分析

问题描述

首先使用get方法请求apache的一个CGI，返回预期结果，然后换成post方法，结果返回如下错误：

curl: (18) transfer closed with outstanding read data remaining

错误的大致意思是：需要读取的数据还没有完成，但是传输数据的连接被关闭了。通过man curl也可以查到返回码18的错误描述，Partial file. Only a part of the file was transferred.

---

服务端CGI的代码很简单，只是构造了一个应答：

#!/bin/bashecho "Content-type: text/html"echo ""echo $REQUEST_METHOD >> /tmp/somelog.gerry# ok, we've sent the header, now send some contentecho  "{\"ret\":0,\"msg\":\"ok\"}"

• 客户端使用get方法调用（可以正常返回）：

$curl -v "http://10.137.142.144/cgi-bin/test.sh?a=b&c=d"* About to connect() to 10.137.142.144 port 80 (#0)*   Trying 10.137.142.144... connected* Connected to 10.137.142.144 (10.137.142.144) port 80 (#0)> GET /cgi-bin/test.sh?a=b&c=d HTTP/1.1> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2> Host: 10.137.142.144> Accept: */*> < HTTP/1.1 200 OK< Date: Fri, 08 Apr 2016 05:49:48 GMT< Server: Apache/2.4.2 (Unix)< Transfer-Encoding: chunked< Content-Type: text/html< {"ret":0,"msg":"ok"}* Connection #0 to host 10.137.142.144 left intact* Closing connection #0

• 客户端使用post方法调用（出现curl: (18)错误）：

$curl -v -d"a=b&c=d" http://10.137.142.144/cgi-bin/test.sh            * About to connect() to 10.137.142.144 port 80 (#0)*   Trying 10.137.142.144... connected* Connected to 10.137.142.144 (10.137.142.144) port 80 (#0)> POST /cgi-bin/test.sh HTTP/1.1> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2> Host: 10.137.142.144> Accept: */*> Content-Length: 7> Content-Type: application/x-www-form-urlencoded> < HTTP/1.1 200 OK< Date: Fri, 08 Apr 2016 05:31:50 GMT< Server: Apache/2.4.2 (Unix)< Transfer-Encoding: chunked< Content-Type: text/html< {"ret":0,"msg":"ok"}* transfer closed with outstanding read data remaining* Closing connection #0curl: (18) transfer closed with outstanding read data remaining

原因分析

错误原因应该是和get和post方法有关。对比两种方法curl调用的输出信息，可以发现应答的HTTP头部内容是一样的，但是在头部多了一个Transfer-Encoding: chunked选项，并不是CGI代码里指定的（问题1）。所以会不会和chunked（分块传输方式）有关？通过查看WIKI Chunked transfer encoding，以及RFC Transfer-Encoding：

Chunked transfer encoding is a data transfer mechanism in version 1.1 of the Hypertext Transfer Protocol (HTTP) in which data is sent in a series of “chunks”. It uses the Transfer-Encoding HTTP header in place of the Content-Length header, which the earlier version of the protocol would otherwise require. Because the Content-Length header is not used, the sender does not need to know the length of the content before it starts transmitting a response to the receiver. Senders can begin transmitting dynamically-generated content before knowing the total size of that content.

The size of each chunk is sent right before the chunk itself so that the receiver can tell when it has finished receiving data for that chunk. The data transfer is terminated by a final chunk of length zero.

An early form of the chunked encoding was proposed in 1994.[2] Later it was standardized in HTTP 1.1.

可得知，Transfer-Encoding: chunked是HTTP/1.1协议默认的编码传输方式。HTTP/1.1之前的协议，比如HTTP/1.0使用的是Content-Length: xxx的传输方式。在使用curl发送请求时，默认使用HTTP/1.1协议，如果要使用HTTP/1.0协议必须显示指定选项-0/--http1.0。因此，这里解释了问题1，为什么应答HTTP头部中默认多了chunked选项。

关于使用chunked方式有什么好处：

• Chunked transfer encoding allows a server to maintain an HTTP persistent connection/HTTP keep-alive/HTTP connection reuse for dynamically generated content. （multiplexed 多路复用，HTTP/1.1默认使用长连接，除非特殊指定不用，而最新的HTTP/2也采用了此思想）
• Chunked encoding allows the sender to send additional header fields after the message body.
• HTTP servers often use compression (gzip or deflate methods) to optimize transmission.

关于使用chunked传输数据的格式说明：

If a Transfer-Encoding field with a value of “chunked” is specified in an HTTP message (either a request sent by a client or the response from the server), the body of the message consists of an unspecified number of chunks, a terminating chunk, trailer, and a final CRLF sequence (i.e. carriage return followed by line feed).

Each chunk starts with the number of octets of the data it embeds expressed as a hexadecimal number in ASCII followed by optional parameters (chunk extension) and a terminating CRLF sequence, followed by the chunk data. The chunk is terminated by CRLF.

If chunk extensions are provided, the chunk size is terminated by a semicolon and followed by the parameters, each also delimited by semicolons. Each parameter is encoded as an extension name followed by an optional equal sign and value. These parameters could be used for a running message digest or digital signature, or to indicate an estimated transfer progress, for instance.

The terminating chunk is a regular chunk, with the exception that its length is zero. It is followed by the trailer, which consists of a (possibly empty) sequence of entity header fields. Normally, such header fields would be sent in the message’s header; however, it may be more efficient to determine them after processing the entire message entity. In that case, it is useful to send those headers in the trailer.

Header fields that regulate the use of trailers are TE (used in requests), and Trailers (used in responses).

客户端使用post方法为什么会出现超时错误，超时时间是多少（问题2）？通过strace分别对get和post方法进行跟踪，可以看到服务端的应答格式：
get方法：

strace -s1024 -tt curl -v http://10.137.142.144/cgi-bin/test.sh?a=b&c=d

13:11:07.350549 recvfrom(3, "HTTP/1.1 200 OK\r\nDate: Mon, 11 Apr 2016 05:11:07 GMT\r\nServer: Apache/2.4.2 (Unix)\r\nTransfer-Encoding: chunked\r\nContent-Type: text/html\r\n\r\n15\r\n{\"ret\":0,\"msg\":\"ok\"}\n\r\n0\r\n\r\n", 16384, 0, NULL, NULL) = 170

post方法：

strace -s1024 -tt curl -v -d”a=b&c=d” http://10.137.142.144/cgi-bin/test.sh

15:41:23.943587 recvfrom(3, "HTTP/1.1 200 OK\r\nDate: Sat, 09 Apr 2016 07:41:23 GMT\r\nServer: Apache/2.4.2 (Unix)\r\nTransfer-Encoding: chunked\r\nContent-Type: text/html\r\n\r\n15\r\n{\"ret\":0,\"msg\":\"ok\"}\n\r\n", 16384, 0, NULL, NULL) = 165

通过上述chunk格式的描述，可以看到get方法HTTP应答的body中：
15\r\n{\”ret\”:0,\”msg\”:\”ok\”}\n\r\n0\r\n\r\n

15（十六进制）表示21个字节，即chunk的数据长度，并以\r\n（CRLF）结束。注意，此长度不包括trailing CRLF (“\r\n”)结尾字符。然后紧接的是，{“ret”:0,”msg”:”ok”}\n，即chunk的数据，并以\r\n结束。最后是，0\r\n，即terminating chunk（一个普通的chunk，只是它的长度为0，客户端可据此判断整条消息都已安全地传输完毕），并以\r\n结束。格式与标准的描述一致。但是，在post方法中，并没有看到terminating chunk，因此客户端不会主动关闭连接，并继续等待服务端的数据，这里可以解释问题2为什么出现超时。

PS：注意，echo会默认在最后添加一个\n，若不想返回此字符，可以使用echo -n。

17:31:37.203111 write(1, "{\"ret\":0,\"msg\":\"ok\"}\n", 21{"ret":0,"msg":"ok"}) = 2117:31:37.203157 poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 1000) = 0 (Timeout)17:31:38.204241 poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)17:31:38.204293 poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 1000) = 0 (Timeout)17:31:39.204741 poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)17:31:39.204796 poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 1000) = 0 (Timeout)17:31:40.205802 poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)17:31:40.205856 poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 1000) = 0 (Timeout)17:31:41.206946 poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)17:31:41.207013 poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 1000) = 1 ([{fd=3, revents=POLLIN|POLLRDNORM}])17:31:42.196723 poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 1 ([{fd=3, revents=POLLIN|POLLRDNORM}])17:31:42.196776 recvfrom(3, "", 16384, 0, NULL, NULL) = 017:31:42.196844 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=405, ...}) = 017:31:42.196915 write(2, "*", 1*)        = 117:31:42.196965 write(2, " ", 1 )        = 117:31:42.197009 write(2, "transfer closed with outstanding read data remaining\n", 53transfer closed with outstanding read data remaining) = 53

关于客户端超时的解释。Apache httpd 2.2及以后版本，HTTP/1.1长连接的超时时间为5秒。超时时间设置比较短的好处是，减少阻塞提高并发能力，避免服务器长时间运行更多的进程或线程以浪费过多的资源。通过上面strace可以验证这个结论，从而进一步解释了问题2超时时间的问题。

解决方法

HTTP的客户端（比如，curl）在读取服务端（Apache2）的应答时，客户端需要知道什么时候数据收完了。特别在persistent connections的场景下，只有当客户端明确判断自己已经收到了完整的应答后，占用的网络连接才能被其他HTTP请求re-used。Content Length & Transfer Encoding介绍了四种客户端判断应答请求是否完整的方法，并可以通过HttpWatch插件查看请求的交互过程。

针对本文提出的post问题，一种解决方法是，CGI在返回HTTP的头部中添加Content-Length选项，明确指定content的长度，这样Apache2就不会再指定Transfer-Encoding: chunked。客户端（curl）根据Content-Length判断数据接收完毕后主动关闭连接。

修改后的CGI代码：

#!/bin/bashecho "Content-type: text/html"echo "Content-Length: 21"echo ""echo $REQUEST_METHOD >> /tmp/somelog.gerry# ok, we've sent the header, now send some contentecho  "{\"ret\":0,\"msg\":\"ok\"}"

使用post方法可以正常返回：

$curl -v -d"a=b&c=d" http://10.137.142.144/cgi-bin/test.sh* About to connect() to 10.137.142.144 port 80 (#0)*   Trying 10.137.142.144... connected* Connected to 10.137.142.144 (10.137.142.144) port 80 (#0)> POST /cgi-bin/test.sh HTTP/1.1> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2> Host: 10.137.142.144> Accept: */*> Content-Length: 7> Content-Type: application/x-www-form-urlencoded> < HTTP/1.1 200 OK< Date: Mon, 11 Apr 2016 11:32:56 GMT< Server: Apache/2.4.2 (Unix)< Content-Length: 21< Content-Type: text/html< {"ret":0,"msg":"ok"}* Connection #0 to host 10.137.142.144 left intact* Closing connection #0

参考
[1] https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol
[2] http://www.jmarshall.com/easy/http/
[3] https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.10