After updating to gramine 1.4 MariaDB fails to start

[ckaratzas]

Hello,
After an attempt to update to gramine 1.4 based on the current patched MariaDB image:
(replacing [FROM enclaive/gramine-os:jammy-7e9d6925] with a similar image with gramine 1.4)
MariaDB fails to restart because some files have 0 length and thus cannot be decrypted properly.
Seems to be linked to this issue:
gramineproject/gramine#830 which is supposed to be merged.
and errors are similar:

(libos_fs_encrypted.c:203:encrypted_file_internal_open) [P1:T1:mariadbd] warning: pf_open failed: Callback failed
mariadb_1               | (libos_fs_encrypted.c:41:cb_read) [P1:T1:mariadbd] warning: EOF

Is there any hint that could help mitigate the issue?
Best Regards

[klassiker]

Hi,

if this is caused by a file in an encrypted mount with size zero (find /data -size 0 inside the container), this is probably related to the PR you referenced. I reported this issue back then with a more generic description (gramineproject/gramine#824) which resulted in the fix only handling clean shutdowns / thread exits.

When the application crashes or you terminate it with ctrl+c the encrypted buffer is not flushed to disk and the encrypted-file header is never written - assuming the data written to the file did not exceed the internal cache size of gramine, in which case only the contents still only in the cache would be lost but the application would restart.

If /data contains files with size zero, you can remove them and the application will probably start again. As a workaround, the SHUTDOWN statement can be used to cleanly exit and let gramine handle the cache flushing to prevent the corruption of a file inside an encrypted mount.

Ideally another issue should be opened for gramine to also handle process termination and recoverable crashes of the application.