In some languages like Chinese, a word of length not bigger than 2 is very common, so I suppose this is a very strong(sometimes wrong in other languages) assumption.

[Christophy]

classifier-reborn/lib/classifier-reborn/extensions/hasher.rb

Line 30 in 4e80749

next unless word.length > 2 && !STOPWORDS[language].include?(word)

[Ch4s3]

We could probably make this configurable. I’ll happily review a PR for this.

[Ch4s3]

@Christophy We just merged #162, which allows for custom tokenizers. Could you let us know if this helps?