Memory issues when running parallel walkers

[FrederikHeber]

There are some more issues when running things in parallel that I just stumbled upon. These are partly related to the new input pipeline, see #22.

• cache() is a memory hog when batch_size is smaller than the dataset size. For example, full MNIST with batch_size of 10,000 will use more and more memory, while batch_size of 55,000 has constant memory after initialization. This might be related to tf 1.4.1 and might have improved with tensorflow newer versions.
• Running multiple walkers (e.g., number_walkers of 2) on MNIST does not show the expected parallel load but fills the whole memory really quickly. This is independent of the above batch_size issue. In contrast to before, the memory filling stops sooner with smaller batch_size, e.g. 1,000. The DirectInputPipeline does not show this behavior. There, memory usage is roughly constant after initialization. In no case do we see parallel loading, there is just a single working thread.

[FrederikHeber]

This issue requires more testing and is related to fixing the ensemble quasi newton scheme.

I have had a short look at this again. I have notice that number_walkers seems to cause irregular memory use, e.g. 4 times as much memory for 2 walkers.

Hence, I will postpone this issue to after v0.9.