PHP网络数据抓取方法

来源：互联网发布：mac苹果应用商店下载编辑：程序博客网时间：2024/05/17 22:09

常见的抓取网络数据的一些方法。

1. 用 file_get_contents() 以 get 方式获取内容：

$url  = 'http://www.nowamagic.net/php/sock.php';$html = file_get_contents($url);echo $html;

2. 用fopen()打开url，以get方式获取内容：

$url = 'http://www.nowamagic.net/php/sock.php';$fp  = fopen($url, 'r');stream_get_meta_data($fp);$result = '';while(!feof($fp)){    $result .= fgets($fp, 1024);}echo "url body: $result";fclose($fp);

3. 用file_get_contents函数,以post方式获取url

$data = array(    'foo'  => 'bar',    'baz'  => 'boom',    'site' => 'www.nowamagic.net',    'name' => 'nowa magic');$data = http_build_query($data);$options = array(    'http' => array(        'method'  => 'POST',        'header'  => 'Content-type:application/x-www-form-urlencoded',        'content' => $data        'timeout' => 60 *60 // 超时时间（单位:s）    ));$url     = "http://www.nowamagic.net/test2.php";$context = stream_context_create($options);$result  = file_get_contents($url, false, $context);echo $result;

4. 用 fsockopen 函数打开url，以get方式获取完整的数据，包括header和body

$url = 'http://www.nowamagic.net:80/php/sock.php?site=nowamagic.net';function get_url($url,$cookie=false){    $url   = parse_url($url);    $query = $url['path']."?".$url['query'];    echo"Query:".$query;    $fp = fsockopen($url['host'], $url['port']?$url['port']:80 , $errno, $errstr, 30);    if(!$fp)    {        return false;    }    else {        $request = "GET$query HTTP/1.1\r\n";        $request .= "Host:$url[host]\r\n";        $request .="Connection: Close\r\n";        if($cookie)$request.="Cookie:  $cookie\n";        $request.="\r\n";        fwrite($fp,$request);        $result = '';        while(!feof($fp))        {            $result .=@fgets($fp, 1024);        }        fclose($fp);        return $result;    }}//获取url的html部分，去掉headerfunction GetUrlHTML($url,$cookie=false){    $rowdata = get_url($url,$cookie);    if($rowdata)    {        $body=stristr($rowdata,"\r\n\r\n");        $body=substr($body,4,strlen($body));        return $body;    }    return false;}echo get_url($url);echo GetUrlHTML($url);<span style="font-family: Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255);"> </span>

5. 用fsockopen函数打开url，以POST方式获取完整的数据，包括header和body

$url = 'http://www.nowamagic.net:80/php/sock.php?site=nowamagic.net';function HTTP_Post($URL,$data,$cookie, $referer=""){    $URL_Info=parse_url($URL);    if($referer==""){//if not given use this script as referrer       $referer="www.nowamagic.net";    }    foreach($data as $key=>$value){        $values[] = "$key=".urlencode($value);    }    $data_string=implode("&",$values);    if(!isset($URL_Info["port"])){        $URL_Info["port"]=80;    }    $request = '';    $request.="POST".$URL_Info["path"]." HTTP/1.1\n";    $request.="Host:".$URL_Info["host"]."\n";    $request.="Referer:$referer\n";    $request.="Content-type:application/x-www-form-urlencoded\n";    $request.="Content-length:".strlen($data_string)."\n";    $request.="Connection:close\n";    $request.="Cookie:   $cookie\n";    $request.="\n";    $request.=$data_string."\n";    $fp = fsockopen($URL_Info["host"],$URL_Info["port"]);    fputs($fp, $request);    $result = '';    while(!feof($fp))    {        $result .= fgets($fp,1024);    }    fclose($fp);    return $result;}$data = array(    'foo'  => 'bar',    'baz'  => 'boom',    'site' => 'www.nowamagic.net',    'name' => 'nowa magic');$cookie  = '';$referer = 'http://www.nowamagic.net/';echo HTTP_Post($url, $data, $cookie, $referer);

6. 使用curl库，使用curl库之前，可能需要查看一下php.ini是否已经打开了curl扩展。

$url     = 'http://www.nowamagic.net:80/php/sock.php?site=nowamagic.net';$ch      = curl_init();$timeout = 5;curl_setopt ($ch, CURLOPT_URL, $url);curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, $timeout);$file_contents = curl_exec($ch);curl_close($ch);echo $file_contents;

1 0