Increase memory consumption 7.3.0

[Swizbiz]

Version

7.3.0

Operating System

Linux/Unix

Bug description

After upgrading jgit from 7.0.0.0.202409031743-r to 7.3.0.202506031305-r, the service starts consuming more memory than before.

Working with jgit version 7.0.0.0.202409031743-r the service uses ~2.35 GB (75% of existing memory) with a container memory limit of 3.15 GB, see first image.

After upgrading jgit to version 7.3.0.202506031305-r the service started consuming all the memory available in the container. Then, after adding an additional 1 GB of memory for the container(container memory limit = 4.19 GB), the service started consuming more memory than before update (almost 1.2 GB more at peak), see second image. The load from users on the service has not increased.
Since I had previously had memory issues with jgit, I decided to just downgrade jgit to the previous version 7.0.0.0.0.202409031743-r without any changes to the settings (just removed the extra 1 gigabyte of memory for the container that was added earlier after the service failed). After that the memory consumption was back to normal, see second image.
No change in java memory consumption, XMX has the same numbers when using both versions of jgit.

A year ago I found that jgit uses a lot of off-heap memory when cloning a project, almost 1.5 times the size of the repository. To solve this problem, I use ReentrantLock so that multiple repositories are not cloned at one time, but only 1 project can be cloned at a time.
Kotlin code:

private val cloningLock = ReentrantLock()

fun internalWithClonedProject(cloneUrl: String, startBranch: String? = null, spec: (git: Git) -> Unit) {
    try {
        cloningLock.lock()
        log.trace { "Lock taken for cloning $cloneUrl" }

        GitWorkingCopy().use { workingCopy ->
            Git.cloneRepository()
                .setURI(cloneUrl)
                .setBranch(startBranch)
                .setDirectory(workingCopy.dir)
                .call()
                .use { git -> // function use in kotlin closes resources, it will invoke close() method on Git
                    spec(git)
                }
        }
    } finally {
        cloningLock.unlock()
        log.trace { "Lock is released for $cloneUrl" }
    }

private class GitWorkingCopy : AutoCloseable {
    val dir: File = Files.createTempDirectory("temp-").toFile()
    override fun close() {
        dir.deleteRecursively()
    }

1. What could have caused a significant increase in memory?
2. Maybe there are some settings to limit the off-heap memory consumption?

Actual behavior

Significant increase in memory

Expected behavior

Relevant log output

Other information

[msohn]

• What is the unspecified service you are talking doing using JGit ? Does it clone repositories as shown in your code example ?
• I am not aware of JGit using off-heap memory explicitly.

[Swizbiz]

• What is the unspecified service you are talking doing using JGit ? Does it clone repositories as shown in your code example ?

• clone repo
• make an empty commit or merge branches
• push changes

[orange-buffalo]

I am not aware of JGit using off-heap memory explicitly.

Could it be related to file operations? Just as one example of possible scenarios, if JGIt keeps the handles open until the repo is closed, these could lead to excessive native memory usage by the OS. A possible solution would be to close the handles as soon as they are not longer needed (before the repo closure) and limit the number of concurrently open files.

[tomaswolf]

Direct memory usage is more complicated than that. AFAIK ByteBuffer.allocateDirect() (used in several NIO operations) under the hood keeps a per-thread cache of previously allocated buffers, and by default has no maximum size for cached buffers set.

See for instance https://dzone.com/articles/troubleshooting-problems-with-native-off-heap-memo .

I think tracking this down would need much more detailed monitoring, possibly using jemalloc, and including information about threads (How many are there? How many of them do have such cached direct buffers, and how large are these cached buffers?). If you want to try something blindly: try setting -Djdk.nio.maxCachedBufferSize=... (plain number, no k,M,G suffix).

Also: it would certainly help if you could narrow the version range. 7.0.0.202409031743-r to 7.3.0.202506031305-r is nine months of commits. Can you bisect this a little by trying 7.1.0 and 7.2.0 releases, and then possibly even some of the -m1/m2/m3 releases available at https://repo.eclipse.org/content/groups/releases/org/eclipse/jgit/org.eclipse.jgit/ ? Does the problem persist if you use 7.4.0.202508261455-rc1 (also available there)?

Also: is changing the JGit version the only variable here? Or did JVM versions or flags or OS in the container or anything else also change?

[orange-buffalo]

Also: is changing the JGit version the only variable here? Or did JVM versions or flags or OS in the container or anything else also change?

We localized it to only JGit version change, nothing else was changed.

AFAIK ByteBuffer.allocateDirect() (used in several NIO operations) under the hood keeps a per-thread cache of previously allocated buffers, and by default has no maximum size for cached buffers set. I think tracking this down would need much more detailed monitoring, possibly using jemalloc, and including information about threads (How many are there?

That sounds interesting, we indeed have many worker threads running concurrently. We observed previously that multiple JGit operations simultaneously would grow the memory uncontrollably (gigabytes). This was the reason to introduce the locking and allow a single thread only to operate with JGit. This solution allowed us to have stable max memory usage, up to the point of the upgrade where this no longer holds.

Many thanks for your suggestions. Testing it all on a live service is pretty hard. We will think if we can invest into a synthetic reproducer. I guess it should be possible to test the hypothesis of multi-threading being the culprit.