Add a new mode in parse module to handle large trace file on a server with small CPU memory size #212
Description
When we have a large trace file but run parse on a server whose memory size is smaller, it'll trigger an OOM issue.
thoughts:
before we run the parse module, we should check system size and file size. if it is too large, we should have a mode to dump parsed lines during the process, rather than holding everything in the memory. the final output will not be a gzip file compressed the whole file. it becomes a similar format with .bin.ndjson but gzip rt mode should handle it automatically IIRC. the compress ratio is influenced a bit, it should be fine since the OOM is the biggest issue.