Uses of Class org.apache.lucene.analysis.TokenStream

Uses in package org.apache.lucene.analysis

Classes derived from org.apache.lucene.analysis.TokenStream
class	CharTokenizer An abstract base class for simple, character-oriented tokenizers.
class	ISOLatin1AccentFilter A filter that replaces accented characters in the ISO Latin 1 character set (ISO-8859-1) by their unaccented equivalent.
class	KeywordTokenizer Emits the entire input as a single token.
class	LengthFilter Removes words that are too long and too short from the stream.
class	LetterTokenizer A LetterTokenizer is a tokenizer that divides text at non-letters.
class	LowerCaseFilter Normalizes token text to lower case.
class	LowerCaseTokenizer LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together.
class	PorterStemFilter Transforms the token stream as per the Porter stemming algorithm.
class	StopFilter Removes stop words from a token stream.
class	TokenFilter A TokenFilter is a TokenStream whose input is another token stream.
class	Tokenizer A Tokenizer is a TokenStream whose input is a Reader.
class	WhitespaceTokenizer A WhitespaceTokenizer is a tokenizer that divides text at whitespace.

Constructors with parameter type org.apache.lucene.analysis.TokenStream
ISOLatin1AccentFilter.ISOLatin1AccentFilter(TokenStream input)
LengthFilter.LengthFilter(TokenStream in, int min, int max) Build a filter that removes words that are too long or too short from the text.
LowerCaseFilter.LowerCaseFilter(TokenStream in)
PorterStemFilter.PorterStemFilter(TokenStream in)
StopFilter.StopFilter(TokenStream in, Hashtable stopTable) Constructs a filter which removes words from the input TokenStream that are named in the Hashtable.
StopFilter.StopFilter(TokenStream in, Hashtable stopTable, boolean ignoreCase) Constructs a filter which removes words from the input TokenStream that are named in the Hashtable.
StopFilter.StopFilter(TokenStream in, Set stopWords) Constructs a filter which removes words from the input TokenStream that are named in the Set.
StopFilter.StopFilter(TokenStream input, Set stopWords, boolean ignoreCase) Construct a token stream filtering the given input.
StopFilter.StopFilter(TokenStream input, String[] stopWords) Construct a token stream filtering the given input.
StopFilter.StopFilter(TokenStream in, String[] stopWords, boolean ignoreCase) Constructs a filter which removes words from the input TokenStream that are named in the array of words.
TokenFilter.TokenFilter(TokenStream input) Construct a token stream filtering the given input.

Fields of type org.apache.lucene.analysis.TokenStream
TokenStream	TokenFilter.input The source of tokens for this filter.

Methods with return type org.apache.lucene.analysis.TokenStream
TokenStream	Analyzer.tokenStream(Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.
TokenStream	Analyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.
TokenStream	KeywordAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.
TokenStream	PerFieldAnalyzerWrapper.tokenStream(String fieldName, Reader reader)
TokenStream	SimpleAnalyzer.tokenStream(String fieldName, Reader reader)
TokenStream	StopAnalyzer.tokenStream(String fieldName, Reader reader) Filters LowerCaseTokenizer with StopFilter.
TokenStream	WhitespaceAnalyzer.tokenStream(String fieldName, Reader reader)

Uses in package org.apache.lucene.analysis.br

Classes derived from org.apache.lucene.analysis.TokenStream
class	BrazilianStemFilter Based on GermanStemFilter

Constructors with parameter type org.apache.lucene.analysis.TokenStream
BrazilianStemFilter.BrazilianStemFilter(TokenStream in) Construct a token stream filtering the given input.
BrazilianStemFilter.BrazilianStemFilter(TokenStream in, Hashtable exclusiontable) Builds a BrazilianStemFilter that uses an exclusiontable.
BrazilianStemFilter.BrazilianStemFilter(TokenStream in, Set exclusiontable)

Methods with return type org.apache.lucene.analysis.TokenStream
TokenStream	BrazilianAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Uses in package org.apache.lucene.analysis.cjk

Classes derived from org.apache.lucene.analysis.TokenStream
class	CJKTokenizer CJKTokenizer was modified from StopTokenizer which does a decent job for most European languages.

Methods with return type org.apache.lucene.analysis.TokenStream
TokenStream	CJKAnalyzer.tokenStream(String fieldName, Reader reader) get token stream from input

Uses in package org.apache.lucene.analysis.cn

Classes derived from org.apache.lucene.analysis.TokenStream
class	ChineseFilter Title: ChineseFilter Description: Filter with a stop word table Rule: No digital is allowed.
class	ChineseTokenizer Title: ChineseTokenizer Description: Extract tokens from the Stream using Character.getType() Rule: A Chinese character as a single token Copyright: Copyright (c) 2001 Company: The difference between thr ChineseTokenizer and the CJKTokenizer (id=23545) is that they have different token parsing logic.

Constructors with parameter type org.apache.lucene.analysis.TokenStream
ChineseFilter.ChineseFilter(TokenStream in)

Methods with return type org.apache.lucene.analysis.TokenStream
TokenStream	ChineseAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Uses in package org.apache.lucene.analysis.cz

Methods with return type org.apache.lucene.analysis.TokenStream
TokenStream	CzechAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Uses in package org.apache.lucene.analysis.de

Classes derived from org.apache.lucene.analysis.TokenStream
class	GermanStemFilter A filter that stems German words.

Constructors with parameter type org.apache.lucene.analysis.TokenStream
GermanStemFilter.GermanStemFilter(TokenStream in)
GermanStemFilter.GermanStemFilter(TokenStream in, Hashtable exclusiontable) Builds a GermanStemFilter that uses an exclusiontable.
GermanStemFilter.GermanStemFilter(TokenStream in, Set exclusionSet) Builds a GermanStemFilter that uses an exclusiontable.

Methods with return type org.apache.lucene.analysis.TokenStream
TokenStream	GermanAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Uses in package org.apache.lucene.analysis.el

Classes derived from org.apache.lucene.analysis.TokenStream
class	GreekLowerCaseFilter Normalizes token text to lower case, analyzing given ("greek") charset.

Constructors with parameter type org.apache.lucene.analysis.TokenStream
GreekLowerCaseFilter.GreekLowerCaseFilter(TokenStream in, char[] charset)

Methods with return type org.apache.lucene.analysis.TokenStream
TokenStream	GreekAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Uses in package org.apache.lucene.analysis.fr

Classes derived from org.apache.lucene.analysis.TokenStream
class	FrenchStemFilter A filter that stemms french words.

Constructors with parameter type org.apache.lucene.analysis.TokenStream
FrenchStemFilter.FrenchStemFilter(TokenStream in)
FrenchStemFilter.FrenchStemFilter(TokenStream in, Hashtable exclusiontable) Builds a FrenchStemFilter that uses an exclusiontable.
FrenchStemFilter.FrenchStemFilter(TokenStream in, Set exclusiontable)

Methods with return type org.apache.lucene.analysis.TokenStream
TokenStream	FrenchAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Uses in package org.apache.lucene.analysis.nl

Classes derived from org.apache.lucene.analysis.TokenStream
class	DutchStemFilter A filter that stems Dutch words.

Constructors with parameter type org.apache.lucene.analysis.TokenStream
DutchStemFilter.DutchStemFilter(TokenStream _in)
DutchStemFilter.DutchStemFilter(TokenStream _in, Set exclusiontable) Builds a DutchStemFilter that uses an exclusiontable.
DutchStemFilter.DutchStemFilter(TokenStream _in, Set exclusiontable, Map stemdictionary)

Methods with return type org.apache.lucene.analysis.TokenStream
TokenStream	DutchAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided TextReader.

Uses in package org.apache.lucene.analysis.ru

Classes derived from org.apache.lucene.analysis.TokenStream
class	RussianLetterTokenizer A RussianLetterTokenizer is a tokenizer that extends LetterTokenizer by additionally looking up letters in a given "russian charset".
class	RussianLowerCaseFilter Normalizes token text to lower case, analyzing given ("russian") charset.
class	RussianStemFilter A filter that stems Russian words.

Constructors with parameter type org.apache.lucene.analysis.TokenStream
RussianLowerCaseFilter.RussianLowerCaseFilter(TokenStream in, char[] charset)
RussianStemFilter.RussianStemFilter(TokenStream in, char[] charset)

Methods with return type org.apache.lucene.analysis.TokenStream
TokenStream	RussianAnalyzer.tokenStream(String fieldName, Reader reader) Creates a TokenStream which tokenizes all the text in the provided Reader.

Uses in package org.apache.lucene.analysis.snowball

Classes derived from org.apache.lucene.analysis.TokenStream
class	SnowballFilter A filter that stems words using a Snowball-generated stemmer.

Constructors with parameter type org.apache.lucene.analysis.TokenStream
SnowballFilter.SnowballFilter(TokenStream in, String name) Construct the named stemming filter.

Methods with return type org.apache.lucene.analysis.TokenStream
TokenStream	SnowballAnalyzer.tokenStream(String fieldName, Reader reader) Constructs a `StandardTokenizer` filtered by a `StandardFilter`, a `LowerCaseFilter` and a `StopFilter`.

Uses in package org.apache.lucene.analysis.standard

Classes derived from org.apache.lucene.analysis.TokenStream
class	StandardFilter Normalizes tokens extracted with `StandardTokenizer`.
class	StandardTokenizer A grammar-based tokenizer constructed with JavaCC.

Constructors with parameter type org.apache.lucene.analysis.TokenStream
StandardFilter.StandardFilter(TokenStream in) Construct filtering in.

Methods with return type org.apache.lucene.analysis.TokenStream
TokenStream	StandardAnalyzer.tokenStream(String fieldName, Reader reader) Constructs a `StandardTokenizer` filtered by a `StandardFilter`, a `LowerCaseFilter` and a `StopFilter`.

Uses in package org.apache.lucene.index.memory

Classes derived from org.apache.lucene.analysis.TokenStream
class	SynonymTokenFilter Injects additional tokens for synonyms of token terms fetched from the underlying child stream; the child stream must deliver lowercase tokens for synonyms to be found.

Constructors with parameter type org.apache.lucene.analysis.TokenStream
SynonymTokenFilter.SynonymTokenFilter(TokenStream input, SynonymMap synonyms, int maxSynonyms) Creates an instance for the given underlying stream and synonym table.

Methods with parameter type org.apache.lucene.analysis.TokenStream
void	MemoryIndex.addField(String fieldName, TokenStream stream) Iterates over the given token stream and adds the resulting terms to the index; Equivalent to adding a tokenized, indexed, termVectorStored, unstored, Lucene `Field`.

Methods with return type org.apache.lucene.analysis.TokenStream
TokenStream	MemoryIndex.keywordTokenStream(Collection keywords) Convenience method; Creates and returns a token stream that generates a token for each keyword in the given collection, "as is", without any transforming text analysis.
TokenStream	PatternAnalyzer.tokenStream(String fieldName, Reader reader) Creates a token stream that tokenizes all the text in the given Reader; This implementation forwards to `tokenStream(String, String)` and is less efficient than `tokenStream(String, String)`.
TokenStream	PatternAnalyzer.tokenStream(String fieldName, String text) Creates a token stream that tokenizes the given string into token terms (aka words).

Uses in package org.apache.lucene.search.highlight

Methods with parameter type org.apache.lucene.analysis.TokenStream
String	Highlighter.getBestFragment(TokenStream tokenStream, String text) Highlights chosen terms in a text, extracting the most relevant section.
String[]	Highlighter.getBestFragments(TokenStream tokenStream, String text, int maxNumFragments) Highlights chosen terms in a text, extracting the most relevant sections.
String	Highlighter.getBestFragments(TokenStream tokenStream, String text, int maxNumFragments, String separator) Highlights terms in the text , extracting the most relevant sections and concatenating the chosen fragments with a separator (typically "...").
TextFragment[]	Highlighter.getBestTextFragments(TokenStream tokenStream, String text, boolean mergeContiguousFragments, int maxNumFragments) Low level api to get the most relevant (formatted) sections of the document.

Methods with return type org.apache.lucene.analysis.TokenStream
TokenStream	TokenSources.getAnyTokenStream(IndexReader reader, int docId, String field, Analyzer analyzer) A convenience method that tries a number of approaches to getting a token stream.
TokenStream	TokenSources.getTokenStream(IndexReader reader, int docId, String field)
TokenStream	TokenSources.getTokenStream(IndexReader reader, int docId, String field, Analyzer analyzer)
TokenStream	TokenSources.getTokenStream(TermPositionVector tpv)
TokenStream	TokenSources.getTokenStream(TermPositionVector tpv, boolean tokenPositionsGuaranteedContiguous) Low level api.

Copyright © 2000-2007 Apache Software Foundation. All Rights Reserved.