Stale Cache Causing Truncated File Reads

[jamesstidard]

Hi,

I'm experiencing inconsistent state on reading a recently written to file from a separate container on another node. Initially, I thought this was a race condition in my codebase, but that doesn't appear to be the case.

NOTE: This also is only reproducible with two physically separate machines, and cannot be reproduced on with two containers on a single machine sharing the same volume mount.

Here's a setup, just so we have some reference names:

version: '3.2'
services:
  container_1:
    image: busybox:latest
    entrypoint: ["sh", "-c", "tail -f /dev/null"]
    restart: always
    volumes:
      - sshfs_test:/data
    deploy:
      placement:
        constraints:
          - "node.labels.role==node1"
  container_2:
    image: busybox:latest
    entrypoint: ["sh", "-c", "tail -f /dev/null"]
    restart: always
    volumes:
      - sshfs_test:/data
    deploy:
      placement:
        constraints:
          - "node.labels.role==node2"
volumes:
  sshfs_test:
    driver: "vieux/sshfs:latest"
    driver_opts:
      sshcmd: "username@hostname:/path/to/somewhere"
      password: "password"
      allow_other: ""

If you have these two containers running, and then sudo docker exec -it <container-id> sh into both. You can write in one, and read from another, with out issue:

$ # container_1
$ echo "example 1" > /data/example.txt

$ # container_2
$ cat /data/example.txt
example 1

Though if you go back and rewrite the file in container_1 to append more bytes to the file, the subsequent read will only read the number of bytes from the previous file size, but with the contents of the new file returned.

Note: You've got to be pretty quick, this will behave fine after ~3-5 seconds.

$ # container_1
$ echo "some new longer data" > /data/example.txt

$ # container_2
$ cat /data/example.txt
some new l
$ cat /data/example.txt
some new longer data

After that truncated read, the following attempts will work, and return the full files contents. Or if you wait long enough before that initial read from container_2 (approx 3-5seconds). If the write makes the file smaller, things also behave fine.

Hopefully this is enough to reproduce. I can see this not being a problem for log files, though my current use case is a *.json file, so am getting incomplete json objects.

Thanks.