Multiple Input Buffers

Some scanners (such as those which support "include" files) require reading from several input streams. As flex scanners do a large amount of buffering, one cannot control where the next input will be read from by simply writing a YY_INPUT which is sensitive to the scanning context. YY_INPUT is only called when the scanner reaches the end of its buffer, which may be a long time after scanning a statement such as an "include" which requires switching the input source.

To negotiate these sorts of problems, flex provides a mechanism for creating and switching between multiple input buffers. An input buffer is created by using:

YY_BUFFER_STATE yy_create_buffer( FILE *file, int size )

which takes a FILE pointer and a size and creates a buffer associated with the given file and large enough to hold size characters (when in doubt, use YY_BUF_SIZE for the size). It returns a YY_BUFFER_STATE handle, which may then be passed to other routines:

void yy_switch_to_buffer( YY_BUFFER_STATE new_buffer )

switches the scanner's input buffer so subsequent tokens will come from new_buffer. Note that yy_switch_to_buffer may be used by yywrap to sets things up for continued scanning, instead of opening a new file and pointing `yyin' at it.

void yy_delete_buffer( YY_BUFFER_STATE buffer )

is used to reclaim the storage associated with a buffer.

yy_new_buffer is an alias for yy_create_buffer, provided for compatibility with the C++ use of new and delete for creating and destroying dynamic objects.

Finally, the YY_CURRENT_BUFFER macro returns a YY_BUFFER_STATE handle to the current buffer.

Here is an example of using these features for writing a scanner which expands include files (the `<<EOF>>' feature is discussed below):

/* the "incl" state is used for picking up the name
 * of an include file
 */
%x incl

%{
#define MAX_INCLUDE_DEPTH 10
YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH];
int include_stack_ptr = 0;
%}

%%
include             BEGIN(incl);

[a-z]+              ECHO;
[^a-z\n]*\n?        ECHO;

<incl>[ \t]*      /* eat the whitespace */
<incl>[^ \t\n]+   { /* got the include file name */
        if ( include_stack_ptr >= MAX_INCLUDE_DEPTH )
            {
            fprintf( stderr, "Includes nested too deeply" );
            exit( 1 );
            }

        include_stack[include_stack_ptr++] =
            YY_CURRENT_BUFFER;

        yyin = fopen( yytext, "r" );

        if ( ! yyin )
            error( ... );

        yy_switch_to_buffer(
            yy_create_buffer( yyin, YY_BUF_SIZE ) );

        BEGIN(INITIAL);
        }

<<EOF>> {
        if ( --include_stack_ptr < 0 )
            {
            yyterminate();
            }

        else
            yy_switch_to_buffer(
                 include_stack[include_stack_ptr] );
        }