BigramSplitter (0.2.1.1)

Text character normalization process uses Python unicodedata. Convert full-width numeric and alphabet character into half-width equivalent. Convert half-width Katakana into full-width equivalent. Therefore all of above character variations can be recognized as same ones.

Language Specifications:

Chinese

    -- No space between words.

    -- There is only Kanji(Chinese) character

    -- Process with Bigram(2-gram) model

Japanese

    -- No space between words

    -- Combination 0f Kanji(Chinese), Katakana, and Hiragana character

    -- Discriminate Kanji, Hiragana, Katakana and processed with Bigram(2-gram) model

    -- Convert Katakana into Hiragana

Korean

    -- There are spaces between words, but it contains a particle

    -- Combination of Korean alphabet and Kanji(Chinese) character

    -- Discriminate Korean alphabet and Kanji(Chinese) character and processed with Bigram(2-gram) model

Thai

    -- No space between words

    -- It's very difficult to handle this language in a computer

    -- A vowel and a consonant are registered in Unicode separately so that it is difficult to recognize as one word.

    -- However, there is a possibility of dealing with Thai characters to use Bigram(2-gram) model.

Other languages (Including English)

    -- There is a space between words

    -- It is indexed each word

Source Code

Since no documents are available on how to develop 'word splitter', we refer to other splitter source code. But I still have a number of questions. If you have any more information, please feel free let us know.

Hotfix to Plone 3.0 source code

Because Plone 3.x catalog setting, catalog.xml, doesn't have existing index overwrite mechanism, we developed hotfix and added XML attribute. We believe Plone 3 XML define mechanism is simple and clear, so that we take this approach. We appreciate any comment.