[OCTRL-998]: Sending kafka messages in wrong order

[justonedev1]

There was a goroutine in WriteEventWithTimestamp which can cause race condition resulting in problems mentioned in ticket OCTRL-998.

However, there can be performance problems caused by removing this concurrence, as every Kafka write call will wait until the message is send by the writer. This can be prevented by setting writer into async mode. But doing so we won't get any errors that might happen while sending messages. From kafka-go/writer.go Async comment:

	// Setting this flag to true causes the WriteMessages method to never block.
	// It also means that errors are ignored since the caller will not receive
	// the returned value. Use this only if you don't care about guarantees of
	// whether the messages were written to kafka.

[knopers8]

Indeed I am afraid this will have a performance hit and might block the core in case of any event writer hiccup. Could we use a buffered channel of a reasonable length (100-1000?) and do non-blocking sends in WriteEvent and WriteEventWithTimestamp, while there would be a goroutine on the other side to consume and perform writing?

[justonedev1]

Good idea, will do

[justonedev1]

I did it... but I guess that it is the same functionality as just Saying Async = true for the kafka-go library... except we can print errors if we encounter some during writing... it would be possible to create some error checker (eg. store encountered errors in kafkaWriter and check if he encountered some periodically), but I don't know if it is realy necessary and if error message in IL is enough, especially when there will be a lot of them if we cannot write into kafka

[knopers8]

except we can print errors if we encounter some during writing...

yes indeed, this is why i still opted for this solution. I assume with Async = true we would be blind.

[knopers8]

This will block if the buffer is full. Should we make it non-blocking perhaps?