Fully Offline installer bundles

[madhephaestus]

At the moment the installer requires the internet for the first run of the application. It needs the internet because the Application JAR file, the JVM and all of the dependent Git repos are downloaded on first run, and each can be updated from the online sources as part of the auto-updater system.

The bundle would need to be included in each of the OS installers. This means the bundled installer would be VERY large, well over 2gb.

This bundeling will need to be done in the OS specific installer builder scripts. there is one each for the 3 OS's with configurations for the various architectures. The JVM and JAR files are straight-forward as to where they are and how they cna be included. The hard part is creating a snap-shot of the Git repos used by the application. the git repos are all of the image assets as well as all of the example models.

Step one would be to extract each of the git repos used in the application and make a list of them. This might be best done by adding temp code to ensureUpdated to capture each of the git repos as a list. This list can be cloned to the dist using the same statagy used by the git clone system inside the kernel. Alternatly one could use the bowler-scripting-kernel to clone the repos and that would result in each being cloned into a directory structure that matches the expectations of the application.

@rondlh had been asking questions about how to make bundles

[madhephaestus]

@rondlh A thought for how to do this involves accessing the location of the Jar file from a running application. I wonder if this works after the jpackage step...

If it does, we can bundle the jar, the zip of the github deps, the jvm and the jvm.json from the latest release into a fat installer. Those extra components can be gathered in CI and placed into the jar directory for the installers as an additional "fat" installer.

Ill try to make a script that clones all of the github depenancies into a directory and zips it. I will likely move the ensureUpdated call into a script in its own repo that CaDoodle will call, but also i can have the kernel call it in CI to clone each of the repos in a coherent location to where CaDoodle will expect them.

    public static String getJarPath(Class<?> clazz) {
        try {
            String path = new File(clazz.getProtectionDomain().getCodeSource().getLocation().toURI()).getAbsolutePath();
            return (System.getProperty("os.name").toLowerCase().contains("win") && path.startsWith("/") && path.contains(":")) 
                ? path.substring(1) : path;
        } catch (Exception e) {
            e.printStackTrace();
            return null;
        }
    }
    
    public static String getJarDirectory(Class<?> clazz) {
        try {
            String dir = new File(clazz.getProtectionDomain().getCodeSource().getLocation().toURI()).getParent();
            return (System.getProperty("os.name").toLowerCase().contains("win") && dir != null && dir.startsWith("/") && dir.contains(":")) 
                ? dir.substring(1) : dir;
        } catch (Exception e) {
            e.printStackTrace();
            return null;
        }
    }

[rondlh]

I'm just learning about this, thanks for your support in helping me to understand how things work.

Perhaps things could work a bit differently:

1. The Azul JVM used for the updater is the same as used by the application, so there is actually no need to download the JVM for the application, the JVM files could simply be copied, or even better, just reuse the same JVM.

2. The application also needs some github resources. "CaDoodle-workspace\gitcache\github.com", if I compress that folder it's about 40MB is size, only very little in there seems essential to get started offline.

3. The main application Jar is also essential, which is about 400MB. This brings the total offline installer size for "updater Jar with JVM" + Github resources + main application Jar to 100 + 40 + 400 = 540MB. Compress it with a 7z self extractor and the total package will be about 350MB and can run completely offline from first run.

4. The updater download and update should not block the main application start. The updater can download the main application Jar file in the background and report when it is done and indicate that a restart is required to update. The same can be done with the other github assets. The updater can check for updates in the background and keep them in a parallel directory, then copy the new files when restarting the application.

[madhephaestus]

1. jpackage doesnt bundle a full jvm, it analyses the classes used by the jar using it, and only bundles whats needed. it also compressed the jar and jvm into a single OS specific executable. I frankly do not know if its even possible to access, or where it would be after jpackage is done making the installer.
2. Yup! it just needs to all be in the right place for it to start. A zip with the current starter content should be possible. I can take responsibility for making that zip, since its a bit fiddely and i want it to be a single-source-of-truth for the main application and what the installer does
3. yup (but also the jvm at another 200mb)
4. Yeah, thats why it asks the user to update or not, if you click keep it the way it was (or setup a pinned version) it should startup with the current jar very quickly. (we might want to add something to the settings of the main application that also lets the user pin and unpin that version number) A pinned version can start without any network connections and would be fully secure and fast.

[rondlh]

1. jpackage doesnt bundle a full jvm, it analyses the classes used by the jar using it, and only bundles whats needed. it also compressed the jar and jvm into a single OS specific executable. I frankly do not know if its even possible to access, or where it would be after jpackage is done making the installer.

According to the documentation jpackage provides a full JVM, except for the java executable itself.

I frankly do not know if its even possible to access, or where it would be after jpackage is done making the installer.

I just did a proof of concept, here is the magic code:

`String[] finalCommand = {"D:\CaDoodle\CaDoodle-1.3.2\CaDoodle\runtime\bin\java.exe", "-jar", "D:\CaDoodle\CaDoodle-1.3.2\CaDoodle\app\CaDoodle-Application.jar"};

env.put("JAVA_HOME", "D:\CaDoodle\CaDoodle-1.3.2\CaDoodle\runtime");`

I put the CaDoodleUpdater.jar and CaDoodle-Application.jar in the same directory.

To fix the missing java.exe we need cp "$JAVA_HOME/bin/java.exe" "$NAME/runtime/bin"

This works fine, the Updater can start the application like this using the same JVM. In the task manager I confirmed that the correct JVM is used.

2. Yup! it just needs to all be in the right place for it to start. A zip with the current starter content should be possible. I can take responsibility for making that zip, since its a bit fiddely and i want it to be a single-source-of-truth for the main application and what the installer does

Great! I would love to have that zip.
Please check if this resource is needed: https://github.com/CommonWealthRobotics/CaDoodle-Application/blob/main/src/main/resources/com/commonwealthrobotics/BlueNoise470.png
It doesn't seem to be referenced anywhere, but it is 5MB in size and is packed in the Jar.

4. Yeah, thats why it asks the user to update or not, if you click keep it the way it was (or setup a pinned version) it should startup with the current jar very quickly. (we might want to add something to the settings of the main application that also lets the user pin and unpin that version number) A pinned version can start without any network connections and would be fully secure and fast.

With the approach above the updater can download an update while the application is running, and just launch the new Jar the next time the application is started. Several version can be available at the same time.

[madhephaestus]

I have updated the application release CI to collect and bundle the gitcache.zip this lets you have a complete and up to date copy of the cache. it will be updated with each release of the application and will be placed in the release alongside the jar and the jvm.json https://github.com/CommonWealthRobotics/CaDoodle-Application/releases/tag/0.25.21

[madhephaestus]

@rondlh with the newly packaged zip of the dependencies in each release, you could update this auto-updater to check to see if the gitcache directory exists and if it doesnt, download the zip and extract it into the gitcache directory. That would result in the main application starting with all of its resouces.

Pushed further, this repo could have an extra step in the CI that downloads the 3 release files into the installers and makes a fat installer that has the zip files in a known location and can deploy them from the disk on a networkless first boot. Combined with a version pinning, that should result in a fully offline install and first run.