Package org.carrot2.filter.trc.carrot.tokenizer

Interface Summary

HTMLEntityResolver
ITokenizer
TokenizerImplConstants

Class Summary

CommonEntityResolver	Resolve HTML Entity to its text form ( -> " ", & -> "&", etc)
DefaultTokenizer
HTMLAwareTokenizer	Tokenizer that translate common HTML entities to its text form
SimpleCharStream	An implementation of interface CharStream, where the stream is assumed to contain only ASCII characters (without unicode processing).
Token	Describes the input token stream.
Tokenizer	Tokenizer class splits a string into tokens like word, e-mail address, web page address and such.
TokenizerFactory
TokenizerImpl	Implementation of abstract Tokenizer class generated by JavaCC parser generator.
TokenizerImplTokenManager

Exception Summary

ParseException

This exception is thrown when parse errors are encountered.

Error Summary

TokenMgrError