Module org.apache.lucene.analysis.nori

Package org.apache.lucene.analysis.ko

Class Viterbi

java.lang.Object

org.apache.lucene.analysis.morph.Viterbi<Token,Viterbi.Position>

org.apache.lucene.analysis.ko.Viterbi

final class Viterbi
extends Viterbi<Token,Viterbi.Position>

Viterbi subclass for Korean morphological analysis.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.analysis.morph.Viterbi

Viterbi.Position, Viterbi.WrappedPositionArray<U extends Viterbi.Position>

Field Summary

Fields

Modifier and Type	Field	Description
private final CharacterDefinition	characterDefinition
private final EnumMap<TokenType,Dictionary<? extends KoMorphData>>	dictionaryMap
private final boolean	discardPunctuation
private GraphvizFormatter<KoMorphData>	dotOut
private final KoreanTokenizer.DecompoundMode	mode
private final boolean	outputUnknownUnigrams
private final UnknownDictionary	unkDictionary

Fields inherited from class org.apache.lucene.analysis.morph.Viterbi

buffer, costs, enableSpacePenaltyFactor, end, lastBackTracePos, MAX_UNKNOWN_WORD_LENGTH, outputLongestUserEntryOnly, outputNBest, pending, pos, positions, VERBOSE, wordIdRef

Constructor Summary

Constructors

Constructor	Description
Viterbi(TokenInfoFST fst, FST.BytesReader fstReader, TokenInfoDictionary dictionary, TokenInfoFST userFST, FST.BytesReader userFSTReader, UserDictionary userDictionary, ConnectionCosts costs, UnknownDictionary unkDictionary, CharacterDefinition characterDefinition, boolean discardPunctuation, KoreanTokenizer.DecompoundMode mode, boolean outputUnknownUnigrams)

Method Summary

Modifier and Type	Method	Description
protected void	backtrace(Viterbi.Position endPosData, int fromIDX)	Backtrace from the provided position, back to the last time we back-traced, accumulating the resulting tokens to the pending list.
protected int	computeSpacePenalty(MorphData morphData, int wordID, int numSpaces)	Returns the space penalty associated with the provided POS.Tag.
(package private) Dictionary<? extends KoMorphData>	getDict(TokenType type)
private static boolean	isCommonOrInherited(Character.UnicodeScript script)
private static boolean	isPunctuation(char ch)
private static boolean	isPunctuation(char ch, int cid)
private static boolean	isSameScript(Character.UnicodeScript scriptOne, Character.UnicodeScript scriptTwo)	Determine if two scripts are compatible.
protected int	processUnknownWord(boolean anyMatches, Viterbi.Position posData)	Add unknown words to the position graph.
(package private) void	setGraphvizFormatter(GraphvizFormatter<KoMorphData> dotOut)
private boolean	shouldFilterToken(Token token)

Methods inherited from class org.apache.lucene.analysis.morph.Viterbi

add, backtraceNBest, computePenalty, fixupPendingList, forward, getPending, getPos, isEnd, isOutputNBest, resetBuffer, resetState, shouldSkipProcessUnknownWord

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

dictionaryMap

private final EnumMap<TokenType,Dictionary<? extends KoMorphData>> dictionaryMap

unkDictionary

private final UnknownDictionary unkDictionary

characterDefinition

private final CharacterDefinition characterDefinition

discardPunctuation

private final boolean discardPunctuation

mode

private final KoreanTokenizer.DecompoundMode mode

outputUnknownUnigrams

private final boolean outputUnknownUnigrams

dotOut

private GraphvizFormatter<KoMorphData> dotOut

Constructor Details

Viterbi

Viterbi(TokenInfoFST fst,
 FST.BytesReader fstReader,
 TokenInfoDictionary dictionary,
 TokenInfoFST userFST,
 FST.BytesReader userFSTReader,
 UserDictionary userDictionary,
 ConnectionCosts costs,
 UnknownDictionary unkDictionary,
 CharacterDefinition characterDefinition,
 boolean discardPunctuation,
 KoreanTokenizer.DecompoundMode mode,
 boolean outputUnknownUnigrams)

Method Details

processUnknownWord

protected int processUnknownWord(boolean anyMatches,
 Viterbi.Position posData)
                          throws IOException

Description copied from class: Viterbi

Add unknown words to the position graph.

Specified by:

processUnknownWord in class Viterbi<Token,Viterbi.Position>

Returns:

word length

Throws:

IOException

setGraphvizFormatter

void setGraphvizFormatter(GraphvizFormatter<KoMorphData> dotOut)

backtrace

protected void backtrace(Viterbi.Position endPosData,
 int fromIDX)

Description copied from class: Viterbi

Backtrace from the provided position, back to the last time we back-traced, accumulating the resulting tokens to the pending list. The pending list is then in-reverse (last token should be returned first).

Specified by:

backtrace in class Viterbi<Token,Viterbi.Position>

computeSpacePenalty

protected int computeSpacePenalty(MorphData morphData,
 int wordID,
 int numSpaces)

Returns the space penalty associated with the provided POS.Tag.

Overrides:

computeSpacePenalty in class Viterbi<Token,Viterbi.Position>

getDict

Dictionary<? extends KoMorphData> getDict(TokenType type)

shouldFilterToken

private boolean shouldFilterToken(Token token)

isPunctuation

private static boolean isPunctuation(char ch)

isPunctuation

private static boolean isPunctuation(char ch,
 int cid)

isCommonOrInherited

private static boolean isCommonOrInherited(Character.UnicodeScript script)

isSameScript

private static boolean isSameScript(Character.UnicodeScript scriptOne,
 Character.UnicodeScript scriptTwo)

Determine if two scripts are compatible.