Trained the model using the example given. The intermediate results looked pretty good. However, when running the inference (render only mode), the results look appalling.
The validation output during training:

The inference result after training (using the trained weights)

What's the reason for this disparity ?