Larbin Source Code Analysis 3——Class global(updating……)

#include

Collaboration diagram for global:

Class global is the head, the heart, the core of the Larbin, so its complexity is unavoidable and releatively acceptable(Don’t tell me you have been imaging that a webcrawler’s main class should be VERY little and small).

YES, you must be compaining in front  of your lovely computer and saying “Damn! can’t you make it clearer?"

But the fact is You DON’T need to see it clearly. All you have to do is know how complexity the relationsip, because we will dig into it later, don’t worry!

_________________________________________________________________________________________________

0. in main.cc

• /*
• * author: Ailleret
• * comments by: Harold Wang
• * http://blog.csdn.net/hero7935
• */

///////////////////////////////////////////////////////////
// If this thread terminates, the whole program exits
int main (int argc, char *argv[]) {
  // create all the structures
  global glob(argc, argv);               /*initialize all we need*/
#ifdef PROF
  signal (2, handler);                    /*setup signal handler for SIGINT*/
#endif // PROF

#ifndef NOWEBSERVER
  // launch the webserver if needeed
  if (global::httpPort != 0)
    startThread(startWebserver, NULL);
/*if we need a webserver to see the state and condition of the larbin*/
#endif // NOWEBSERVER

  // Start the search
  printf("%s is starting its search/n", global::userAgent);
  time_t old = global::now;

  for (;;)
{                  /*Ok, Never end!*/
    // update time
    global::now = time(NULL);
    if (old != global::now) {
      // this block is called every second
      old = global::now;
      cron();
/*a lot of stats and profiling things. like shall we stop? check if timeouts? etc.*/
    }
    stateMain(-count); /*where is the main loop(just for debugging)*/
    waitBandwidth(&old); /*wait to limit bandwidth usage*/
    stateMain(1);
    for (int i=0; i      global::ansPoll[i] = 0; /*initialize I/O multiplexing*/
/*register the I/O multiplexing fds set which we have intrest in*/
    for (uint i=0; i      global::ansPoll[global::pollfds[i].fd] = global::pollfds[i].revents; 
    global::posPoll = 0; /*initial the pos of the max used field in pollfds to 0*/
    stateMain(2);
    input(); /*if we have inputfiles to handle*/
    stateMain(3);
/*put urls from the queue(including global::URLsPriority || global::URLsPriorityWait || *global::URLsDisk || global::URLsDiskWait) to the queue(global::namedSiteList[u->hostHashCode()]) */ 
    sequencer(); 
    stateMain(4); 
/*get website from dnsSites to start dns request and receive the website’s robot.txt*/
    fetchDns();
    stateMain(5);
/*get a namesite from OKSite and go on crawling its urls*/
    fetchOpen();
    stateMain(6); 
/*check all the connections’s state, I do not finish this item’s comment yet.*/
    checkAll();
    // select
    stateMain(count++);
/*put pollfds to the device waiting queue, waiting for any events in the next 10 millisecond*/
    poll(global::pollfds, global::posPoll, 10); /*posPoll know how many fds we have used*/
    stateMain(7);
  }
}

1. in global.h

 

Note: Destructor ~global() never used. Because the program should never end!

Constructor : initialize almost everything.

Everything is read from the config file (larbin.conf by default)

Here is the call graph for this function: