SimVQ trains Failed!

Hi, i have tried SimVQ recently, but i failed. I used the default configuration, the W matrix is a single Linear layer. But i found the activation of codebook is very low, and also the perplexity is low. I set codebook_size is 32, the activation is about 7%. When the model is initialized, the initial codebook activation is high, nearly close 100%, but quickly fall down to zero. Why?