Wizard
Software Engineering Project - Wizard
Loading...
Searching...
No Matches
cpp.tokenize Namespace Reference

Classes

class  Token
 

Functions

 _GetString (source, start, i)
 
 _GetChar (source, start, i)
 
 GetTokens (source)
 
 main (argv)
 

Variables

str _letters = 'abcdefghijklmnopqrstuvwxyz'
 
 VALID_IDENTIFIER_CHARS = set(_letters + _letters.upper() + '_0123456789$')
 
 HEX_DIGITS = set('0123456789abcdefABCDEF')
 
 INT_OR_FLOAT_DIGITS = set('01234567890eE-+')
 
 _STR_PREFIXES = set(('R', 'u8', 'u8R', 'u', 'uR', 'U', 'UR', 'L', 'LR'))
 
str UNKNOWN = 'UNKNOWN'
 
str SYNTAX = 'SYNTAX'
 
str CONSTANT = 'CONSTANT'
 
str NAME = 'NAME'
 
str PREPROCESSOR = 'PREPROCESSOR'
 
 WHENCE_STREAM
 
 WHENCE_QUEUE
 

Detailed Description

Tokenize C++ source code.

Function Documentation

◆ GetTokens()

cpp.tokenize.GetTokens ( source)
Returns a sequence of Tokens.

Args:
  source: string of C++ source code.

Yields:
  Token that represents the next token in the source.

◆ main()

cpp.tokenize.main ( argv)
Driver mostly for testing purposes.