Studying note of GCC-3.4.6 source (77)

5.6. Prepare the parser

Now it’s going to parse the source file, and the input of the parser must be identifiers of C++. The tool fetching identifiers is called Lexer. It is worth noting that, GCC hasn’t so-called preprocessing pass, because function like cpp_get_token can fetch preprocessed token directly. Obvious, this function is also an important part of the Lexer. Being the first step of parsing, it prepares the parser and the associating Lexer.

 

15112 void

15113 c_parse_file (void)                                                                                  in parser.c

15114 {

15115   bool error_occurred;

15116

15117   the_parser = cp_parser_new ();

15118   push_deferring_access_checks (flag_access_control

15119                             ? dk_no_deferred : dk_no_check);

15120   error_occurred = cp_parser_translation_unit (the_parser);

15121   the_parser = NULL;

15122 }

 

As the data structure for C++ parser, cp_parser is defined as below. Notice that data of this type is managed by GCC garbage collector, for every translation unit, a new parser will be created.

 

1170 typedef struct cp_parser GTY(())                                                                 in parser.c

1171 {

1172   /* The lexer from which we are obtaining tokens.  */

1173   cp_lexer *lexer;

1174

1175   /* The scope in which names should be looked up. If NULL_TREE, then

1176     we look up names in the scope that is currently open in the

1177     source program. If non-NULL, this is either a TYPE or

1178     NAMESPACE_DECL for the scope in which we should look. 

1179

1180     This value is not cleared automatically after a name is looked

1181     up, so we must be careful to clear it before starting a new look

1182     up sequence. (If it is not cleared, then `X::Y' followed by `Z'

1183     will look up `Z' in the scope of `X', rather than the current

1184     scope.) Unfortunately, it is difficult to tell when name lookup

1185     is complete, because we sometimes peek at a token, look it up,

1186     and then decide not to consume it.  */

1187   tree scope;

1188

1189   /* OBJECT_SCOPE and QUALIFYING_SCOPE give the scopes in which the

1190     last lookup took place. OBJECT_SCOPE is used if an expression

1191     like "x->y" or "x.y" was used; it gives the type of "*x" or "x",

1192     respectively. QUALIFYING_SCOPE is used for an expression of the

1193     form "X::Y"; it refers to X.  */

1194   tree object_scope;

1195   tree qualifying_scope;

1196

1197   /* A stack of parsing contexts. All but the bottom entry on the

1198     stack will be tentative contexts.

1199

1200     We parse tentatively in order to determine which construct is in

1201     use in some situations. For example, in order to determine

1202     whether a statement is an expression-statement or a

1203     declaration-statement we parse it tentatively as a

1204     declaration-statement. If that fails, we then reparse the same

1205     token stream as an expression-statement.  */

1206   cp_parser_context *context;

1207

1208   /* True if we are parsing GNU C++. If this flag is not set, then

1209     GNU extensions are not recognized.  */

1210   bool allow_gnu_extensions_p;

1211

1212   /* TRUE if the `>' token should be interpreted as the greater-than

1213     operator. FALSE if it is the end of a template-id or

1214     template-parameter-list.  */

1215   bool greater_than_is_operator_p;

1216

1217   /* TRUE if default arguments are allowed within a parameter list

1218     that starts at this point. FALSE if only a gnu extension makes

1219     them permissible.  */

1220   bool default_arg_ok_p;

1221  

1222   /* TRUE if we are parsing an integral constant-expression. See

1223     [expr.const] for a precise definition.  */

1224   bool integral_constant_expression_p;

1225

1226   /* TRUE if we are parsing an integral constant-expression -- but a

1227     non-constant expression should be permitted as well. This flag

1228     is used when parsing an array bound so that GNU variable-length

1229     arrays are tolerated.  */

1230   bool allow_non_integral_constant_expression_p;

1231

1232   /* TRUE if ALLOW_NON_CONSTANT_EXPRESSION_P is TRUE and something has

1233     been seen that makes the expression non-constant.  */

1234   bool non_integral_constant_expression_p;

1235

1236   /* TRUE if we are parsing the argument to "__offsetof__".  */

1237   bool in_offsetof_p;

1238

1239  /* TRUE if local variable names and `this' are forbidden in the

1240     current context.  */

1241   bool local_variables_forbidden_p;

1242

1243   /* TRUE if the declaration we are parsing is part of a

1244     linkage-specification of the form `extern string-literal

1245     declaration'.  */

1246   bool in_unbraced_linkage_specification_p;

1247

1248   /* TRUE if we are presently parsing a declarator, after the

1249     direct-declarator.  */

1250   bool in_declarator_p;

1251

1252   /* TRUE if we are presently parsing a template-argument-list.  */

1253   bool in_template_argument_list_p;

1254

1255  /* TRUE if we are presently parsing the body of an

1256    iteration-statement.  */

1257   bool in_iteration_statement_p;

1258

1259   /* TRUE if we are presently parsing the body of a switch

1260     statement.  */

1261   bool in_switch_statement_p;

1262

1263   /* TRUE if we are parsing a type-id in an expression context. In

1264     such a situation, both "type (expr)" and "type (type)" are valid

1265     alternatives.  */

1266   bool in_type_id_in_expr_p;

1267

1268   /* If non-NULL, then we are parsing a construct where new type

1269     definitions are not permitted. The string stored here will be

1270     issued as an error message if a type is defined.  */

1271   const char *type_definition_forbidden_message;

1272

1273   /* A list of lists. The outer list is a stack, used for member

1274     functions of local classes. At each level there are two sub-list,

1275     one on TREE_VALUE and one on TREE_PURPOSE. Each of those

1276     sub-lists has a FUNCTION_DECL or TEMPLATE_DECL on their

1277     TREE_VALUE's. The functions are chained in reverse declaration

1278     order.

1279

1280     The TREE_PURPOSE sublist contains those functions with default

1281     arguments that need post processing, and the TREE_VALUE sublist

1282     contains those functions with definitions that need post

1283     processing.

1284

1285     These lists can only be processed once the outermost class being

1286     defined is complete.  */

1287   tree unparsed_functions_queues;

1288

1289   /* The number of classes whose definitions are currently in

1290     progress.  */

1291   unsigned num_classes_being_defined;

1292

1293   /* The number of template parameter lists that apply directly to the

1294     current declaration.  */

1295   unsigned num_template_parameter_lists;

1296 } cp_parser;

 

Function cp_parser_new used to create instances of cp_parser has below definition:

 

2230 static cp_parser *

2231 cp_parser_new (void)                                                                                 in parser.c

2232 {

2233   cp_parser *parser;

2234   cp_lexer *lexer;

2235

2236   /* cp_lexer_new_main is called before calling ggc_alloc because  

2237     cp_lexer_new_main might load a PCH file.  */

2238   lexer = cp_lexer_new_main ();

 

The lexer created by cp_lexer_new_main has following definition, it is also a GC controlled type. Note that in its definition, all pointer members are managed by GC too except next slot at line 212. This next slot reveals that unlike parser that is unique for one translation unit, more lexer will be created temperarily.

 

166  typedef struct cp_lexer GTY (())                                                                  in parser.c

167  {

168    /* The memory allocated for the buffer.  Never NULL.  */

169    cp_token * GTY ((length ("(%h.buffer_end - %h.buffer)"))) buffer;

170    /* A pointer just past the end of the memory allocated for the buffer.  */

171    cp_token * GTY ((skip (""))) buffer_end;

172    /* The first valid token in the buffer, or NULL if none.  */

173    cp_token * GTY ((skip (""))) first_token;

174    /* The next available token. If NEXT_TOKEN is NULL, then there are

175      no more available tokens.  */

176    cp_token * GTY ((skip (""))) next_token;

177    /* A pointer just past the last available token. If FIRST_TOKEN is

178      NULL, however, there are no available tokens, and then this

179      location is simply the place in which the next token read will be

180      placed. If LAST_TOKEN == FIRST_TOKEN, then the buffer is full.

181      When the LAST_TOKEN == BUFFER, then the last token is at the

182      highest memory address in the BUFFER.  */

183    cp_token * GTY ((skip (""))) last_token;

184 

185    /* A stack indicating positions at which cp_lexer_save_tokens was

186      called. The top entry is the most recent position at which we

187      began saving tokens. The entries are differences in token

188      position between FIRST_TOKEN and the first saved token.

189 

190      If the stack is non-empty, we are saving tokens. When a token is

191      consumed, the NEXT_TOKEN pointer will move, but the FIRST_TOKEN

192      pointer will not. The token stream will be preserved so that it

193      can be reexamined later.

194 

195      If the stack is empty, then we are not saving tokens. Whenever a

196      token is consumed, the FIRST_TOKEN pointer will be moved, and the

197      consumed token will be gone forever.  */

198    varray_type saved_tokens;

199 

200    /* The STRING_CST tokens encountered while processing the current

201      string literal.  */

202    varray_type string_tokens;

203 

204    /* True if we should obtain more tokens from the preprocessor; false

205      if we are processing a saved token cache.  */

206    bool main_lexer_p;

207 

208    /* True if we should output debugging information.  */

209    bool debugging_p;

210 

211     /* The next lexer in a linked list of lexers.  */

212    struct cp_lexer *next;

213  } cp_lexer;

 

In previous sections, we have seen that token is represented by type cpp_token, however this type is designed for preprocessor. After preprocessing, preprocessing elements like: maro, assertion, #include directive, etc. are longer exist, and cpp_token is not a fit candidate any more. Replacing it is cp_token for the preprocessed tokens.

 

69    typedef struct cp_token GTY (())                                                                 in parser.c

70    {

71      /* The kind of token.  */

72      ENUM_BITFIELD (cpp_ttype) type : 8;

73      /* If this token is a keyword, this value indicates which keyword.

74        Otherwise, this value is RID_MAX.  */

75      ENUM_BITFIELD (rid) keyword : 8;

76     /* Token flags.  */

77      unsigned char flags;

78      /* The value associated with this token, if any.  */

79      tree value;

80      /* The location at which this token was found.  */

81      location_t location;

82    } cp_token;

 

By comparison, these 2 definitions are quite similar.

5.6.1. Create main Lexer

Every translation-unit should have a main lexer to go with the parser. This main lexer is created by below funciton.

 

301  static cp_lexer *

302  cp_lexer_new_main (void)                                                                          in parser.c

303  {

304    cp_lexer *lexer;

305    cp_token first_token;

306 

307    /* It's possible that lexing the first token will load a PCH file,

308      which is a GC collection point. So we have to grab the first

309      token before allocating any memory.  */

310    cp_lexer_get_preprocessor_token (NULL, &first_token);

311     c_common_no_more_pch ();

312 

313   /* Allocate the memory.  */

314    lexer = ggc_alloc_cleared (sizeof (cp_lexer));

315 

316    /* Create the circular buffer.  */

317    lexer->buffer = ggc_calloc (CP_TOKEN_BUFFER_SIZE, sizeof (cp_token));

318    lexer->buffer_end = lexer->buffer + CP_TOKEN_BUFFER_SIZE;

319 

320    /* There is one token in the buffer.  */

321    lexer->last_token = lexer->buffer + 1;

322    lexer->first_token = lexer->buffer;

323    lexer->next_token = lexer->buffer;

324    memcpy (lexer->buffer, &first_token, sizeof (cp_token));

325 

326    /* This lexer obtains more tokens by calling c_lex.  */

327    lexer->main_lexer_p = true;

328 

329    /* Create the SAVED_TOKENS stack.  */

330    VARRAY_INT_INIT(lexer->saved_tokens, CP_SAVED_TOKENS_SIZE, "saved_tokens");

331   

332    /* Create the STRINGS array.  */

333    VARRAY_TREE_INIT (lexer->string_tokens, 32, "strings");

334 

335    /* Assume we are not debugging.  */

336    lexer->debugging_p = false;

337 

338    return lexer;

339  }

 

Till now, we have read in main input file, header files included by –include options (if there is any), but don’t start parsing token in file. So cp_lexer_get_preprocessor_token below at line 310, will lex the first token. By GCC current implementation and requirement, every source file should include only one precompiled header file, and the precompiled header file should be the first included file, so if current source file uses precompiled header file, the function will cause the header file be read in (remember, first see #include directive, then run_directive calls handle do_include which calls _cpp_stack_include which next calls c_common_read_pch to read in the PCH file). And in ggc_pch_read invoked by c_common_read_pch, if the host system uses paging for memory management, GC garbage collection is triggered.

 

580  static void

581  cp_lexer_get_preprocessor_token (cp_lexer *lexer ATTRIBUTE_UNUSED ,   in parser.c

582                              cp_token *token)

583  {

584    bool done;

585 

586    /* If this not the main lexer, return a terminating CPP_EOF token.  */

587    if (lexer != NULL && !lexer->main_lexer_p)

588    {

589      token->type = CPP_EOF;

590      token->location.line = 0;

591      token->location.file = NULL;

592      token->value = NULL_TREE;

593      token->keyword = RID_MAX;

594 

595      return;

596    }

597 

598    done = false;

599    /* Keep going until we get a token we like.  */

600    while (!done)

601    {

602      /* Get a new token from the preprocessor.  */

603      token->type = c_lex_with_flags (&token->value, &token->flags);

604      /* Issue messages about tokens we cannot process.  */

605      switch (token->type)

606      {

607       case CPP_ATSIGN:

608        case CPP_HASH:

609        case CPP_PASTE:

610          error ("invalid token");

611           break;

612 

613        default:

614          /* This is a good token, so we exit the loop.  */

615          done = true;

616          break;

617      }

618    }

619    /* Now we've got our token.  */

620    token->location = input_location;

621 

622    /* Check to see if this token is a keyword.  */

623    if (token->type == CPP_NAME

624        && C_IS_RESERVED_WORD (token->value))

625    {

626      /* Mark this token as a keyword.  */

627      token->type = CPP_KEYWORD;

628      /* Record which keyword.  */

629      token->keyword = C_RID_CODE (token->value);

630      /* Update the value. Some keywords are mapped to particular

631        entities, rather than simply having the value of the

632        corresponding IDENTIFIER_NODE. For example, `__const' is

633        mapped to `const'.  */

634      token->value = ridpointers[token->keyword];

635    }

636    else

637      token->keyword = RID_MAX;

638  }

 

cp_lexer_get_preprocessor_token is low level routine for lexer, which feeds lexer with preprocessed token. No doubt, “#”, “##’, ‘@’ (line 607, used in Obj-C) are invalidate tokens. Further, preprocessed token sould be identifier or constant, C++ reserves some identifiers as key words, which should be recoginzed here (refer to section Initialize reserved words for C++).

5.6.1.1.    Get preprocessed token

5.6.1.1.1.            Case of identifier

Preprocessed token is represented by cp_token, in which field flags can take following value.

 

619  #define CPP_N_CATEGORY  0x000F                                                        in cpplib.h

620  #define CPP_N_INVALID    0x0000

621  #define CPP_N_INTEGER   0x0001

622  #define CPP_N_FLOATING 0x0002

623 

624  #define CPP_N_WIDTH      0x00F0

625  #define CPP_N_SMALL      0x0010    /* int, float.  */

626  #define CPP_N_MEDIUM   0x0020    /* long, double.  */

627  #define CPP_N_LARGE      0x0040    /* long long, long double.  */

628 

629  #define CPP_N_RADIX       0x0F00

630  #define CPP_N_DECIMAL  0x0100

631  #define CPP_N_HEX    0x0200

632  #define CPP_N_OCTAL       0x0400

633 

634  #define CPP_N_UNSIGNED       0x1000    /* Properties.  */

635  #define CPP_N_IMAGINARY     0x2000

 

There are five groups can be set for flags. For example, for token 0x50, flags will be set with CPP_N_INTEGER, CPP_N_SMALL, CPP_N_HEX, and CPP_N_UNSIGNED. Type, value and flags of preprocessed token is taken by c_lex_with_flags.

 

315  int

316  c_lex_with_flags (tree *value, unsigned char *cpp_flags)                               in c-lex.c

317  {

318    const cpp_token *tok;

319    location_t atloc;

320    static bool no_more_pch;

321 

322  retry:

323    tok = get_nonpadding_token ();

 

The core of get_nonpadding_token is cpp_get_token. As we have seen in previous section, this function is where preprocess takes place. At there, macro definitions are digested into instances of cpp_macro, macro invocation is expanded directly via argument replacement (if it is expected), other directives are covered by varies handles, and kinds preprocessor operator is executed.

 

302  static inline const cpp_token *

303  get_nonpadding_token (void)                                                                      in c-lex.c

304  {

305    const cpp_token *tok;

306    timevar_push (TV_CPP);

307    do

308      tok = cpp_get_token (parse_in);

309    while (tok->type == CPP_PADDING);

310    timevar_pop (TV_CPP);

311  

312    return tok;

313  } 

 

See that get_nonpadding_token still returns cpp_token instead of cp_token.

 

c_lex_with_flags (continue)

 

325  retry_after_at:

326    switch (tok->type)

327    {

328      case CPP_NAME:

329        *value = HT_IDENT_TO_GCC_IDENT (HT_NODE (tok->val.node));

330        break;

331 

332      case CPP_NUMBER:

333      {

334        unsigned int flags = cpp_classify_number (parse_in, tok);

335 

336        switch (flags & CPP_N_CATEGORY)

337        {

338          case CPP_N_INVALID:

339            /* cpplib has issued an error.  */

340            *value = error_mark_node;

341            break;

342 

343          case CPP_N_INTEGER:

344            *value = interpret_integer (tok, flags);

345            break;

346 

347          case CPP_N_FLOATING:

348            *value = interpret_float (tok, flags);

349            break;

350 

351          default:

352            abort ();

353        }

354      }

355      break;

 

At line 328, token of CPP_NAME is an identifier, and HT_IDENT_TO_GCC_IDENT converts the corresponding hashnode into tree node.