<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE>2.2 Reference</TITLE><LINK href="ozdoc.css" rel="stylesheet" type="text/css"></HEAD><BODY><TABLE align="center" border="0" cellpadding="6" cellspacing="6" class="nav"><TR bgcolor="#DDDDDD"><TD><A href="node3.html#section.scanner.example"><< Prev</A></TD><TD><A href="node2.html">- Up -</A></TD></TR></TABLE><DIV id="section.scanner.reference"><H2><A name="section.scanner.reference">2.2 Reference</A></H2><P> This section is intended to serve as a reference for the user of the Gump Scanner Generator. It details the syntax of the embedded scanner specification language in <A href="node4.html#section.scanner.syntax">Section 2.2.1</A>, which options are supported and how they are specified in <A href="node4.html#section.scanner.params">Section 2.2.2</A> and finally the runtime part of the Scanner Generator, the mixin class <CODE>GumpScanner<SPAN class="keyword">.</SPAN><SPAN class="string">'class'</SPAN></CODE>, in <A href="node4.html#section.scanner.class">Section 2.2.3</A>. </P><DIV id="section.scanner.syntax"><H3><A name="section.scanner.syntax">2.2.1 Syntax of the Scanner Specification Language</A></H3><P> The notation used here for specifying the syntax of the specification language is a variant of <SPAN class="formalism">BNF</SPAN> and is defined in <A href="node8.html#appendix.notation">Appendix A</A>. </P><P> A scanner specification is allowed anywhere as an Oz statement: </P><BLOCKQUOTE><TABLE border="0" cellpadding="0" cellspacing="0"><TR valign="top"><TD><<I>statement</I>></TD><TD align="center"> += </TD><TD><<I>scanner specification</I>></TD></TR></TABLE></BLOCKQUOTE><P> </P><P> It is similar to a class definition, except that it is introduced by the keyword <CODE><SPAN class="keyword">scanner</SPAN></CODE>, must be named by a variable (and not an arbitrary term), since this is used for assigning file names, and allows for additional descriptors after the usual class descriptors. <A name="label82"></A> </P><BLOCKQUOTE><TABLE border="0" cellpadding="0" cellspacing="0"><TR valign="top"><TD><<I>scanner specification</I>></TD><TD align="center"> ::= </TD><TD><CODE><SPAN class="keyword">scanner</SPAN></CODE> <<I>variable</I>></TD></TR><TR valign="top"><TD></TD><TD align="center"></TD><TD>{ <<I>class descriptor</I>> }</TD></TR><TR valign="top"><TD></TD><TD align="center"></TD><TD>{ <<I>method</I>> }</TD></TR><TR valign="top"><TD></TD><TD align="center"></TD><TD>{ <<I>scanner descriptor</I>> }+</TD></TR><TR valign="top"><TD></TD><TD align="center"></TD><TD><CODE><SPAN class="keyword">end</SPAN></CODE></TD></TR></TABLE></BLOCKQUOTE><P> </P><P> A <A name="label83"></A><EM>lexical abbreviation</EM> associates an identifier with a regular expression, which can then be referenced in subsequent lexical abbreviations or any lexical rules by enclosing the identifier in curly brackets. The regular expression is additionally parenthesized when it is expanded. </P><BLOCKQUOTE><TABLE border="0" cellpadding="0" cellspacing="0"><TR valign="top"><TD><<I>lexical abbreviation</I>></TD><TD align="center"> ::= </TD><TD><CODE><SPAN class="keyword">lex</SPAN></CODE> <<I>atom</I>> <SPAN class="terminal">"<CODE>=</CODE>"</SPAN> <<I>regex</I>> <CODE><SPAN class="keyword">end</SPAN></CODE></TD></TR><TR valign="top"><TD></TD><TD align="center"> | </TD><TD><CODE><SPAN class="keyword">lex</SPAN></CODE> <<I>variable</I>> <SPAN class="terminal">"<CODE>=</CODE>"</SPAN> <<I>regex</I>> <CODE><SPAN class="keyword">end</SPAN></CODE></TD></TR></TABLE></BLOCKQUOTE><P> </P><P> The definition of a <A name="label84"></A><EM>lexical rule</EM> is similar to the definition of a method. However, its head consists of a regular expression; when this is matched, the body of the lexical rule is executed (as a method). </P><BLOCKQUOTE><TABLE border="0" cellpadding="0" cellspacing="0"><TR valign="top"><TD><<I>lexical rule</I>></TD><TD align="center"> ::= </TD><TD><CODE><SPAN class="keyword">lex</SPAN></CODE> <<I>regex</I>></TD></TR><TR valign="top"><TD></TD><TD align="center"></TD><TD><<I>in statement</I>></TD></TR><TR valign="top"><TD></TD><TD align="center"></TD><TD><CODE><SPAN class="keyword">end</SPAN></CODE></TD></TR></TABLE></BLOCKQUOTE><P> </P><P> <A name="label85"></A> Regular expressions may be annotated with <A name="label86"></A><EM>lexical modes</EM>. Each lexical mode constitutes an independent sub-scanner: At any time a certain mode is active; in this mode only the regular expressions annotated with it will be matched. All lexical rules defined within the scope of a lexical mode are annotated with this lexical mode. A lexical mode may <EM class="noindex">inherit</EM><A name="label87"></A> from other lexical modes; all regular expressions in these modes are then annotated with the inheriting lexical mode as well. Lexical modes implicitly inherit from all lexical modes they are nested in. Lexical rules written at top-level are annotated with the implicitly declared mode <A name="label88"></A><SPAN class="index"><CODE>INITIAL</CODE></SPAN>. </P><BLOCKQUOTE><TABLE border="0" cellpadding="0" cellspacing="0"><TR valign="top"><TD><<I>lexical mode</I>></TD><TD align="center"> ::= </TD><TD><CODE><SPAN class="keyword">mode</SPAN></CODE> <<I>variable</I>> [ <CODE><SPAN class="keyword">from</SPAN></CODE> { <<I>variable</I>> }+ ]</TD></TR><TR valign="top"><TD></TD><TD align="center"></TD><TD>{ <<I>mode descriptor</I>> }</TD></TR><TR valign="top"><TD></TD><TD align="center"></TD><TD><CODE><SPAN class="keyword">end</SPAN></CODE></TD></TR></TABLE></BLOCKQUOTE><P> </P><BLOCKQUOTE><TABLE border="0" cellpadding="0" cellspacing="0"><TR valign="top"><TD><<I>mode descriptor</I>></TD><TD align="center"> ::= </TD><TD><<I>lexical rule</I>></TD></TR><TR valign="top"><TD></TD><TD align="center"> | </TD><TD><<I>lexical mode</I>></TD></TR></TABLE></BLOCKQUOTE><P> </P><H4><A name="label89">Syntax of Regular Expressions</A></H4><P> <A name="label90"></A> Regular expressions <<I>regex</I>> correspond to the regular expressions used in <A name="label91"></A><SPAN class="index"><SPAN class="tool">flex</SPAN></SPAN> <A href="bib.html#paxson95">[Pax95]</A> with a few exceptions: </P><UL><LI><P>Gump regular expressions are either enclosed in angle brackets or given as Oz strings. </P></LI><LI><P><A name="label92"></A> The angle-bracket annotation with <A name="label93"></A><SPAN class="index">lexical modes</SPAN> is not supported by Gump; use scopes of lexical modes instead. Note that several distinct lexical mode definitions may occur for the same lexical mode name as long as no inheritance cycles are created. </P></LI></UL><P> </P><P> Due to the underlying use of <SPAN class="tool">flex</SPAN>, the names of lexical abbreviations are restricted to the syntax allowed in <SPAN class="tool">flex</SPAN> name definitions. </P><H4><A name="label94">Ambiguities and Errors in the Rule Set</A></H4><P> <A name="label95"></A> Tokenization is performed by a left-to-right scan of the input character stream. If several rules match a prefix of the input, then the rule matching the longest prefix is preferred. If several rules match the same (longest) prefix of the input, then two rules may be applied to disambiguate the match (see <A href="node4.html#section.scanner.params">Section 2.2.2</A> on how to select the rule): </P><DL><DT>First-fit.<A name="label96"></A> </DT><DD><P>The rule notated first in the scanner specification is preferred. In this case, every conflict can be uniquely resolved. Two errors in the rule set are possible: holes and completely covered rules (see below). </P></DD><DT>Best-fit.<A name="label97"></A> </DT><DD><P>Suppose two conflicting rules are rule <I>r</I><SUB>1</SUB> and rule <I>r</I><SUB>2</SUB>, which are annotated by sets of lexical modes <I>S</I><SUB>1</SUB> and <I>S</I><SUB>2</SUB> respectively. Then <I>r</I><SUB>1</SUB> is preferred over <I>r</I><SUB>2</SUB> if and only if the following condition holds: </P><BLOCKQUOTE><P><I>S</I><SUB>1</SUB> is a subset of
<I>S</I><SUB>2</SUB> and L(<I>r</I><SUB>1</SUB>)
is a subset of L(<I>r</I><SUB>2</SUB>)</P></BLOCKQUOTE><P> where L(<I>r</I>) is the language generated by a regular expression <I>r</I>, that is, the set of strings that match <I>r</I>. Intuitively, this rule means that <I>r</I><SUB>1</SUB> is ``more specialized than'' <I>r</I><SUB>2</SUB>. Additionally to the errors possible in the rule set in the first-fit case, here the situation may arise that the rule set is not well-ordered w. r. t. the ``more specialized than'' relation. </P></DD></DL><P> </P><P> <A name="label98"></A> The following errors in the rule set may occur: </P><DL><DT>Holes in the rule set. </DT><DD><P>For some input (in some mode), no true prefix is matched by any rule. Due to the underlying implementation using <SPAN class="tool">flex</SPAN>, this will result in the warning message </P><BLOCKQUOTE class="code"><CODE><SPAN class="string">"-s option given but default rule can be matched"</SPAN></CODE></BLOCKQUOTE><P> If at run-time some such input is encountered, this will result in an error exception </P><BLOCKQUOTE class="code"><CODE><SPAN class="string">"flex scanner jammed"</SPAN></CODE></BLOCKQUOTE><P> </P></DD><DT>Completely covered rules. </DT><DD><P>A rule <I>r</I> is never matched because for every prefix in L(<I>r</I>) exists another rule <I>r</I> which is preferred over <I>r</I>. </P></DD><DT>Non well-orderedness. </DT><DD><P>Two rules <I>r</I><SUB>1</SUB> and <I>r</I><SUB>2</SUB> are in conflict in the best-fit case, but neither is <I>r</I><SUB>1</SUB> more specialized than <I>r</I><SUB>2</SUB> nor the other way round, and no rule or set of rules exists that covers L(<I>r</I><SUB>1</SUB>) intersected
with L(<I>r</I><SUB>2</SUB>). </P></DD></DL><P> </P></DIV><DIV id="section.scanner.params"><H3><A name="section.scanner.params">2.2.2 Parameters to Scanner Generation</A></H3><P> The Gump Scanner Generator supports several configuration parameters, which may be set on a per-scanner basis via the use of macro directives. </P><DIV class="apropos"><P class="margin">Macro Directives</P><P> <A name="label99"></A> Due to the implementation of scanners in <A name="label100"></A><SPAN class="index">C++</SPAN>, a unique prefix is required for each scanner to avoid symbol conflicts when several scanners reside at the same time in the Mozart system. The following macro directive allows this prefix to be changed (the default <CODE>zy</CODE> is all right if only a single scanner is used at any time): </P><BLOCKQUOTE class="code"><CODE><SPAN class="reference">\gumpscannerprefix </SPAN></CODE><<I>atom</I>></BLOCKQUOTE><P> <A name="label101"></A> </P></DIV><DIV class="apropos"><P class="margin">Switches</P><P> <A href="node4.html#table.scanner.switches">Table 2.1</A> summarizes some compiler switches that control the Gump Scanner Generator. </P></DIV></DIV><DIV id="section.scanner.class"><DIV class="table" id="table.scanner.switches"><HR><P><A name="table.scanner.switches"></A></P><P> </P><TABLE align="center" bgcolor="#f0f0e0"><TR valign="top"><TH><P>Switch </P></TH><TH><P>Effect </P></TH></TR><TR valign="top"><TD><P><SPAN class="ignore"><CODE><SPAN class="reference">\switch +</SPAN></CODE></SPAN><CODE><SPAN class="reference">gumpscannerbestfit</SPAN></CODE> <A name="label102"></A> <A name="label103"></A> </P></TD><TD><P>Use best-fit instead of first-fit disambiguating </P></TD></TR><TR valign="top"><TD><P><SPAN class="ignore"><CODE><SPAN class="reference">\switch +</SPAN></CODE></SPAN><CODE><SPAN class="reference">gumpscannercaseless</SPAN></CODE> <A name="label104"></A> </P></TD><TD><P>Generate a case-insensitive scanner </P></TD></TR><TR valign="top"><TD><P><SPAN class="ignore"><CODE><SPAN class="reference">\switch +</SPAN></CODE></SPAN><CODE><SPAN class="reference">gumpscannernowarn</SPAN></CODE> <A name="label105"></A> <A name="label106"></A> <A name="label107"></A> </P></TD><TD><P>Suppress warnings from <A name="label108"></A><SPAN class="index"><SPAN class="tool">flex</SPAN></SPAN> </P></TD></TR></TABLE><P> </P><P class="caption"><STRONG>Table 2.1:</STRONG> Compiler switches for the Gump Scanner Generator.</P><HR></DIV><H3><A name="section.scanner.class">2.2.3 The Mixin Class <CODE>GumpScanner<SPAN class="keyword">.</SPAN><SPAN class="string">'class'</SPAN></CODE></A></H3><P> The module <CODE>GumpScanner</CODE> defines the runtime support needed by Gump-generated scanners. All operations and data are encapsulated in the mixin class <A name="label109"></A><SPAN class="index"><CODE>GumpScanner<SPAN class="keyword">.</SPAN><SPAN class="string">'class'</SPAN></CODE></SPAN> that scanners have to inherit from in order to be executable. </P><DIV class="apropos"><P class="margin">Abstract Members</P><P> The mixin class expects the following features and methods to be defined by derivate classes. (It is a good idea not to define any class members whose name begins with <CODE><SPAN class="keyword">lex</SPAN></CODE>... since these may be used for internals of the Scanner Generator.) </P><DL><DT><CODE><SPAN class="keyword">feat</SPAN> lexer</CODE><A name="label111"></A> </DT><DD><P>This feature must contain the scanner-specific loaded foreign functions, which includes the generated scanner tables. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">lexExecuteAction</SPAN>(</CODE><CODE>+<I>I</I></CODE><CODE>)</CODE><A name="label113"></A> </DT><DD><P>This method is called each time a regular expression is matched. Regular expressions are assigned unique integers; <CODE><I>I</I></CODE> indicates which rule's associated action is to be run. </P></DD></DL><P> </P></DIV><DIV class="apropos"><P class="margin">Provided Members</P><P> The <CODE>GumpScanner<SPAN class="keyword">.</SPAN><SPAN class="string">'class'</SPAN></CODE> class defines some user functionality that is to be used either by users of the generated scanner or by the semantic actions in the scanner itself. </P><DL><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">init</SPAN>()</CODE><A name="label115"></A> </DT><DD><P>This initializes the internal structures of the <CODE>GumpScanner<SPAN class="keyword">.</SPAN><SPAN class="string">'class'</SPAN></CODE>. This must be called before any other method of this class. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">setMode</SPAN>(</CODE><CODE>+<I>I</I></CODE><CODE>)</CODE><A name="label117"></A> </DT><DD><P><A name="label118"></A> <A name="label119"></A> The operation mode of the scanner is set to the lexical mode <CODE><I>I</I></CODE>. Lexical modes are represented internally as integers. Since modes are identified by variables, the class generation phase wraps a <CODE><SPAN class="keyword">local</SPAN> </CODE>...<CODE> <SPAN class="keyword">end</SPAN></CODE> around the class equating the mode variables to the assigned unique integers. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">currentMode</SPAN>(</CODE><CODE>?<I>I</I></CODE><CODE>)</CODE><A name="label121"></A> </DT><DD><P>This returns the integer <CODE><I>I</I></CODE> identifying the lexical mode the scanner currently operates in. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">getAtom</SPAN>(</CODE><CODE>?<I>A</I></CODE><CODE>)</CODE><A name="label123"></A> </DT><DD><P>This method is used to access the lexeme last matched. It is returned as an atom in the variable <CODE><I>A</I></CODE>. Note that if the lexeme contains a <A name="label124"></A><SPAN class="index"><SPAN class="char">NUL</SPAN></SPAN> character (ISO 0) then only the text up to the first <SPAN class="char">NUL</SPAN> but excluding it is returned. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">getString</SPAN>(</CODE><CODE>?<I>S</I></CODE><CODE>)</CODE><A name="label126"></A> </DT><DD><P>This method returns the lexeme as a string in the variable <CODE><I>S</I></CODE>. The restrictions concerning <CODE>getAtom</CODE> do not apply for <CODE>getString</CODE>. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">getLength</SPAN>(</CODE><CODE>?<I>I</I></CODE><CODE>)</CODE><A name="label128"></A> </DT><DD><P>This method returns the length of the lexeme (number of characters matched). </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">putToken</SPAN>(</CODE><CODE>+<I>X</I></CODE><CODE> </CODE><CODE><I>Y</I></CODE><CODE>)</CODE><A name="label130"></A> </DT><DD><P>This method may be used to append a token with token class <CODE><I>X</I></CODE> and value <CODE><I>Y</I></CODE> to the token stream. (Actually, the token class may be an arbitrary Oz value, but atoms and the integers between 0 and 255 are the only representations understood by Gump-generated parsers.) </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">putToken1</SPAN>(</CODE><CODE>+<I>X</I></CODE><CODE>)</CODE><A name="label132"></A> </DT><DD><P>This method may be used to append a token with token class <CODE><I>X</I></CODE> and value <CODE><SPAN class="keyword">unit</SPAN></CODE> to the token stream. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">getToken</SPAN>(</CODE><CODE>?<I>X</I></CODE><CODE> </CODE><CODE><I>Y</I></CODE><CODE>)</CODE><A name="label134"></A> </DT><DD><P>The next token is removed from the token stream and returned. The token class is returned in <CODE><I>X</I></CODE> and its value in <CODE><I>Y</I></CODE>. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">input</SPAN>(</CODE><CODE>?<I>C</I></CODE><CODE>)</CODE><A name="label136"></A> </DT><DD><P>The next (unmatched) character is removed from the character stream and returned in <CODE><I>C</I></CODE>. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">scanFile</SPAN>(</CODE><CODE>+<I>V</I></CODE><CODE>)</CODE><A name="label138"></A> </DT><DD><P><A name="label139"></A> This method causes the currently scanned buffer (if any) to be pushed on a stack of active buffers. A new buffer is created from the file with name <CODE><I>V</I></CODE> and scanned. If the file does not exist, the error exception <CODE>gump(fileNotFound </CODE><CODE><I>V</I></CODE><CODE>)</CODE><A name="label140"></A> with the filename in <CODE><I>V</I></CODE> is raised; the default treatment is the invocation of a custom error printer. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">scanVirtualString</SPAN>(</CODE><CODE>+<I>V</I></CODE><CODE>)</CODE><A name="label142"></A> </DT><DD><P><A name="label143"></A> Like <CODE>scanFile</CODE>, but scans a virtual string <CODE><I>V</I></CODE>. If <CODE><I>V</I></CODE> contains <A name="label144"></A><SPAN class="index"><SPAN class="char">NUL</SPAN></SPAN> characters (ISO 0) then the virtual string is only scanned up to and excluding the first <SPAN class="char">NUL</SPAN> character. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">setInteractive</SPAN>(</CODE><CODE>+<I>B</I></CODE><CODE>)</CODE><A name="label146"></A> </DT><DD><P><A name="label147"></A> Each buffer may be either interactive or non-interactive. An interactive buffer only reads as many characters as are needed to be considered to decide about a match; a non-interactive buffer may read ahead. This method allows the topmost buffer on the stack to be set to interactive (if <CODE><I>B</I></CODE> is <CODE><SPAN class="keyword">true</SPAN></CODE>) or non-interactive (if <CODE><I>B</I></CODE> is <CODE><SPAN class="keyword">false</SPAN></CODE>). New buffers are always created as non-interactive buffers. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">getInteractive</SPAN>(</CODE><CODE>?<I>B</I></CODE><CODE>)</CODE><A name="label149"></A> </DT><DD><P>Whether the topmost buffer on the buffer stack is interactive is returned. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">setBOL</SPAN>(</CODE><CODE>+<I>B</I></CODE><CODE>)</CODE><A name="label151"></A> </DT><DD><P>The <A name="label152"></A><SPAN class="index">beginning-of-line</SPAN> (BOL) flag indicates whether the beginning-of-line regular expression <SPAN class="ignore"><CODE><SPAN class="keyword">lex</SPAN> <SPAN class="string"><</SPAN></CODE></SPAN><CODE><SPAN class="string">^</SPAN></CODE><SPAN class="ignore"><CODE><SPAN class="string">></SPAN></CODE></SPAN> will currently match the input. This flag is true at the beginning of a buffer or after a <A name="label153"></A><SPAN class="index">newline</SPAN> has been scanned. The flag's value may be set at will with this method. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">getBOL</SPAN>(</CODE><CODE>?<I>B</I></CODE><CODE>)</CODE><A name="label155"></A> </DT><DD><P>Returns the current state of the beginning-of-line flag. See the <CODE>setBOL</CODE> method. </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">closeBuffer</SPAN>()</CODE><A name="label157"></A> </DT><DD><P><A name="label158"></A> Closes the topmost buffer on the buffer stack and resumes scanning from the buffer on the new stack top (if any). If the buffer stack is or becomes empty through this operation, only tokens with class <A name="label159"></A><SPAN class="index"><CODE><SPAN class="string">'EOF'</SPAN></CODE></SPAN> and value <CODE><SPAN class="keyword">unit</SPAN></CODE> are returned subsequently (until a new buffer is created). </P></DD><DT><CODE><SPAN class="keyword">meth</SPAN> <SPAN class="functionname">close</SPAN>()</CODE> </DT><DD><P>Closes all buffers on the buffer stack. Before calling any other methods, you should call <CODE>init()</CODE> again. </P></DD></DL><P></P></DIV></DIV></DIV><TABLE align="center" border="0" cellpadding="6" cellspacing="6" class="nav"><TR bgcolor="#DDDDDD"><TD><A href="node3.html#section.scanner.example"><< Prev</A></TD><TD><A href="node2.html">- Up -</A></TD></TR></TABLE><HR><ADDRESS><A href="http://www.ps.uni-sb.de/~kornstae/">Leif Kornstaedt</A><BR><SPAN class="version">Version 1.4.0 (20090610)</SPAN></ADDRESS></BODY></HTML>
distrib
>
Mandriva
>
2010.0
>
i586
>
media
>
contrib-release
>
by-pkgid
>
bd5c3d824c3db63ffd9226c15941e6ad
>
files
>
1353
