Clean up tmpdir correctly, cache task transformation result, and better notebook support.

[Logiquo]

Contributor: Yongda Fan (yongdaf2@illinois.edu)

Contribution Type: Dataset

Description

• Temporary directories will be cleared correctly after the processing (including dask-worker-space).
• Cached data will be cleared if the processing crashed, ensure no corrupted cache file.
• Better notebook support, always ensure single process when notebook is detected.
• Task tansformation result is now cached, allow different processors for .set_task without recomput the entire data.
• SampleDataset now need to .close after use to clear up with underlying data. It also support context manager.

e.g.

with dataset.set_task(...) as sample_dataset:
    ....

or

sample_dataset = dataset.set_task(...)
...
sample_dataset.close()

Benchmark
On a Ryzen 5950x with 64GB memory and NVME disk, using MIMIC4 with mortality prediction task

__init__(num_worker)	.set_task(num_worker)	memory usage	total runtime	task transform runtime
41d190a		16GB	180min+	180min+
1	1	16GB	167min	141min
2	2	16GB	77min	63min
4	4	32GB	41min	31min
8	8	64GB	26min	18min
8	16	64GB	21min	14min
8	32	64GB	16min	8min

[jhnwu3]

Tested, and lgtm! Will test more as I benchmark the process on different machines when I can.