## org.apache.lucene.analysis.standard
Interface StandardTokenizerInterface

All Known Implementing Classes:
  StandardTokenizerImpl, StandardTokenizerImpl31, StandardTokenizerImpl34, UAX29URLEmailTokenizerImpl, UAX29URLEmailTokenizerImpl31, UAX29URLEmailTokenizerImpl34, UAX29URLEmailTokenizerImpl36

    public interface StandardTokenizerInterface

Internal interface for supporting versioned grammars.

NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.

Field Summary
`static int` `YYEOF` This character denotes the end of file

Method Summary
`int` `getNextToken()` Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
`void` `getText(CharTermAttribute t)` Copies the matched text into the CharTermAttribute
`int` `yychar()` Returns the current position.
`int` `yylength()` Returns the length of the matched text region.
`void` `yyreset(Reader reader)` Resets the scanner to read from a new input stream.

Field Detail

### YYEOF

    static final int YYEOF

  This character denotes the end of file

  See Also:
    Constant Field Values

Method Detail

### getText

    void getText(CharTermAttribute t)

  Copies the matched text into the CharTermAttribute

### yychar

    int yychar()

  Returns the current position.

### yyreset

    void yyreset(Reader reader)

  Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to `ZZ_INITIAL`.

  Parameters:
    `reader` - the new input stream

### yylength

    int yylength()

  Returns the length of the matched text region.

### getNextToken

    int getNextToken()
                     throws IOException

  Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.

  Returns:
    the next token, `YYEOF` on end of stream
  Throws:
    `IOException` - if any I/O-Error occurs

Copyright © 2000-2013 Apache Software Foundation. All Rights Reserved.