With -f --storable-state option re2c generates a lexer that can store its current state, return to the caller, and later resume operations exactly where it left off. The default mode of operation in re2c is a “pull” model, where the lexer “pulls” more input whenever it needs it. However, this mode of operation assumes that the lexer is the owner of the parsing loop, and that may not always be convenient.

Storable state is useful exactly for situations like that: it allows to construct lexers that work in a “push” model, where data is fed to the lexer chunk by chunk. When the lexer needs more input, it stores its state and returns to the caller. Later, when more input becomes available, it resumes operations exactly where it stopped.

Changes needed compared to the “pull” model:

  • Define YYSETSTATE () and YYGETSTATE (state).
  • Define yych, yyaccept and state variables as a part of persistent lexer state. state should be initialized to -1.
  • YYFILL should return to the outer program instead of trying to supply more input. Return code should indicate that lexer needs more input.
  • The outer program should recognize situations when lexer needs more input and respond appropriately.
  • Use /*!getstate:re2c*/ directive if it is necessary to execute any code before entering the lexer.
  • Use configurations state:abort and state:nextlabel to tweak the generated code.