Killed / Out of memory

[ajtao]

I'm curious if the full dataset must be fully memory resident for the training pipeline to work, is that how this works?
If so, that's really unfortunate because i'd like to continue to scale the data up.

Training CAMELTrack on a custom dataset with about 180 videos, resulting in a roughly 6GB pklz training state file.
When i try to train CAMELTrack with this, the training job dies pretty early in training, with "Killed" message, which i believe is the system running out of system/CPU memory. I have 64GB system RAM.

I ended up having to shrink my dataset by thresholding detections which allowed me to train.

Andrew

[VlSomers]

Hi @ajtao, could you please have a look at the data structures inside CAMELDataset (cameltrack/train/dataset.py) and tell me which one explodes when you use your full large dataset? And when is the training process killed exactly, could you share a stack trace? Best