Support for tokenization of languages without spaces #4
Description
Need to implement a smarter method of tokenization which takes into account languages that traditionally does not use spaces between words (currently resulting in full-sentence tokens not suitable for the current method of cosine similarity comparisons).
Some of these languages include:
- Chinese
- Japanese
- Thai
- Khmer
- Lao
- Burmese