Lexical Analyzer

Complete token handling system for source code analysis. More...

Topics
	File I/O
	File chunk handling utilities for lexical analysis.
	Tokenization
	Source code to token conversion.
	Token Validation
	Token validation functions for lexical analysis.

Files
file	lexer_tokenize.h
	Tokenization core implementation.

Detailed Description

Complete token handling system for source code analysis.

This header provides the complete infrastructure for:

• Token type classification (#TokenClassification)
• Individual token representation (#TokenStructure)
• Token collection management (#TokenSet)

Note

The groups form a processing pipeline:

1. Classification categorizes lexemes
2. Structures store the results
3. Collections manage token sequences