Strategy for grouping specific tokens together #10

New issue

Open

opened 2024-08-08 16:12:03 +02:00 by xqtc · 0 comments

xqtc commented

2024-08-08 16:12:03 +02:00

Owner

Example:

Input (glsl): vec4 uwu = vec4(1., 1., 1., 1.) + vec4(1., 1., 1., 1.)

Output (tokens): Token::VEC4, Token::Identifier("uwu"), Token::EQ_OP, Token::Term(Vec<>)

It could make parsing way more efficient, if we iterate over the tokens and group them together before handing them to the (yet to come) parser. Something like terms and functions should be grouped together. Example: ``` Input (glsl): vec4 uwu = vec4(1., 1., 1., 1.) + vec4(1., 1., 1., 1.) Output (tokens): Token::VEC4, Token::Identifier("uwu"), Token::EQ_OP, Token::Term(Vec<>) ```