Fluctuate metrics with decreasing loss

[Jary-lrj]

Thanks for contributing the code. I have a question: when running the code with the beauty dataset, the loss keeps decreasing but the NDCG and HR fluctuate significantly. Is this normal?

[pmixer]

Thanks for contributing the code. I have a question: when running the code with the beauty dataset, the loss keeps decreasing but the NDCG and HR fluctuate significantly. Is this normal?

Seems not expected, actually, I haven't run on the beauty dataset yet, can you paste some logs showing how loss in training and metrics in inference changes?

[Jary-lrj]

Thanks for your reply. For the beauty dataset with 176520 interaction records, 12099 items and 22342 users:

epoch	loss	NDCG	HR
20	0.9931135448542509	0.23265257212442683	0.40654968373801265
40	0.9195027364925905	0.22999181973138	0.40654968373801265
60	0.8726524730974977	0.22371084577844116	0.3898564008554843
80	0.8370271264152094	0.22288970332628505	0.3933726367147794
100	0.8132828216661107	0.22256316719581565	0.3890531036591581

Some hyper parameters: bs=256, lr=1e-3, dropout=0.2, maxlen=50, hidden_units=384
As you can see, the loss keeps decreasing while the NDCG and HR has a fluctuation.

[pmixer]

Thank you, the beauty dataset is way too sparse which may require extra hyper-parameter tuning work, have you tried set dropout=0.5? As according to the original paper https://cseweb.ucsd.edu/~jmcauley/pdfs/icdm18.pdf :

The
dropout rate of turning off neurons is 0.2 for MovieLens-1m
and 0.5 for the other three datasets due to their sparsity. The
maximum sequence length n is set to 200 for MovieLens-1m
and 50 for the other three datasets.