LexiC/lexer__tokenize_8h_source.html

#ifndef LEXER_TOKENIZE_H

#define LEXER_TOKENIZE_H


#include "lexer.h"


size_t tokcnt(const char *const line);


void toknz_segtoset(tokset_t *const set,

                    const size_t token_index,

                    const char *const line,

                    const size_t start,

                    const size_t end,

                    const size_t line_no,

                    const tokcat_e category,

                    const size_t column);


tokset_t *toknz(const char *const line);


 // End of Tokenization group


#endif

tokcat_e
tokcat_e
Token category enumeration for categorizing different token types in the pre-processing phase.
Definition lexer.h:48

toknz
tokset_t * toknz(const char *const line)
Tokenizes a line (or multiple lines of code) into a set of tokens.

toknz_segtoset
void toknz_segtoset(tokset_t *const set, const size_t token_index, const char *const line, const size_t start, const size_t end, const size_t line_no, const tokcat_e category, const size_t column)
Tokenizes a segment of a line and stores the resulting token in the token set.

tokcnt
size_t tokcnt(const char *const line)
Counts the number of tokens in a given string (or file content).

lexer.h
Lexical analyzer components for token processing.

tokset_t
Container for multiple tokens.
Definition lexer.h:325