reducing `nu_scripts` git repo size by removing images and some obsolete file and rewriting history

[maxim-uvarov]

Now the repo size seems huge to me:

> du | sort-by physical -r | first 4 | select path physical
╭─#─┬───────────────────────path────────────────────────┬─physical─╮
│ 0 │ /Users/user/git/nu_scripts_dev/.git               │ 297.5 MB │
│ 1 │ /Users/user/git/nu_scripts_dev/themes             │  58.6 MB │
│ 2 │ /Users/user/git/nu_scripts_dev/custom-completions │   5.5 MB │
│ 3 │ /Users/user/git/nu_scripts_dev/modules            │   1.1 MB │
╰─#─┴───────────────────────path────────────────────────┴─physical─╯

apparently because images are stored here.

Can we reduce the size of the repo by storing images using git-lfs?

I can do all the preparations and transfer the results to contributors with push rights.

[fdncred]

Sounds good to me. I don't know anything about git-lfs. Does it change the way you interact with the repo?

[maxim-uvarov]

@fdncred
Here is the project's site: https://git-lfs.com

Users who have cloned the 'nu_scripts' repository without installing 'git-lfs' will see text files in place of binary files, like this:

version https://hawser.github.com/spec/v1
oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
size 12345
(ending \n)

For those who have it installed, nothing will change.

On the GitHub site, images in markdown documents will be displayed as usual, and no changes are needed.

Here are instructions on how to move current files to 'git-lfs': https://docs.github.com/en/repositories/working-with-files/managing-large-files/moving-a-file-in-your-repository-to-git-large-file-storage

In short we will need to use 'filter-repo' to rewrite history and remove binary files. But before that we will need to accept or reject the current PRs

I've done it only once before, so I will review all the steps and let you know what's needed. I will rehearse everything on my clones to make sure that everything will go well and show you results.

Does that sound okay?

[fdncred]

What happens when someone doesn't have git lfs installed after we've made this change?

[maxim-uvarov]

@fdncred:

What happens when someone doesn't have git lfs installed after we've made this change?

In place of example_image.png, they will see a text file with content like:

version https://hawser.github.com/spec/v1
oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
size 12345
(ending \n)

Those, who have git-lfs - they will see regular png image.

[maxim-uvarov]

This operation has benefits and considerations that I propose to discuss and communicate with the community.

However, I'm still gathering information, and these are just intermediate results.

I have tested all the operations, and the results can be found here:
https://github.com/maxim-uvarov/nu_scripts_reduced_size

# Run git-filter-repo
python3 /Users/user/.local/bin/git-filter-repo --path-glob '**/*.png' --invert-paths --force

# Initialize Git LFS
git lfs install
git lfs track "*.png"

# From the unmodified repository directory
let original_repo_pngs = glob **/*.png | wrap orig | insert curr {|i| $i.orig | path relative-to (pwd)}

# Copy PNG images to the new repository
$original_repo_pngs | each {|i| cp $i.orig $i.curr}

Benefits:

• Faster Cloning and Operations: A smaller repository size leads to quicker clone times and improved overall performance.
• Clean History: Eliminates unwanted files or sensitive data from the repository's history.

Considerations and Caveats:

1. Merged Pull Requests:

   • Merged PRs will display old commits and diffs that no longer match the rewritten history, potentially causing confusion.

2. Open Pull Requests:

   • Open PRs may become unmergeable due to history divergence.
   • Contributors will need to rebase or update their branches to align with the new history.

3. Commit References:

   • References to old commit hashes in issues or comments will become invalid, leading to broken links.

4. Forks:

   • Forked repositories will contain the outdated history.
   • Fork owners must rebase or re-clone to sync with the updated repository.

[maxim-uvarov]

I've just found out that bandwidth from 'git lfs' is charged, and only 1 GB is free.

https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-storage-and-bandwidth-usage#tracking-storage-and-bandwidth-use

Plus, it has other downsides like losing signatures from signed commits, and PR history will point to the wrong commits.

Even though it would be much tidier to have a smaller repo size, the downsides are quite noticeable. I will stop here.

[maxim-uvarov]

Though for new binary objects, it would be better to use 'git lfs' to avoid increasing the repo size.

[fdncred]

Another thought is to put the screenshots in another repo and have a submodule in this repo. I think then you could clone and only get the scripts unless you wanted both. I don't really know how all that works though.

[maxim-uvarov]

I like this variant, and it is feasible to do with the workflow that I described.

But again, it will also involve rewriting history to reduce the size of this repo, which might be a bit painful for current contributors and their forks.

I think I should ask the community when I have the energy for that.

[NotTheDr01ds]

But again, it will also involve rewriting history to reduce the size of this repo, which might be a bit painful for current contributors and their forks.

Personally I think that's a hit we're ultimately going to have to take, so probably the sooner the better.

I think I'd recommend keeping the issue open, since it is an issue, even if we don't have a full solution just yet.

[maxim-uvarov]

Here is the rationale and the actions that I propose.

# Let's check the size of the `.git` folder
> du .git/ | get 0.physical
236.2 MB

let $a = git verify-pack -v .git/objects/pack/*.idx | lines | split column -r ' +' | rename SHA-1 type size size-in-packfile offset-in-packfile depth base-SHA-1 | where type == 'blob' | select SHA-1 size | into filesize size
let $b = git rev-list --objects --all | lines | split column -r ' +' | rename SHA-1 filename
let $sizes = $a | join $b SHA-1 | sort-by size -r

# Let's examine the largest files
> $sizes | first 5 | print
╭─#─┬──────────────────SHA-1───────────────────┬───size───┬──────────────────────filename───────────────────────╮
│ 0 │ 87cd314f73c3bd004ea52b060d8a8d6bf40fcbe7 │   1.3 MB │ custom-completions/auto-generate/completions/mvn.nu │
│ 1 │ 09e461721303572ea79c390feec763c498b68335 │ 369.0 KB │ assets/gh-emoji.json                                │
│ 2 │ c2dc98f8c9aa296affb97d2dbeb84f2c6dc09adf │ 241.6 KB │ themes/screenshots/github-light-colorblind.png      │
│ 3 │ 9a258215d6435fac43dbe125b8cf2d77a6cc5b82 │ 241.3 KB │ themes/screenshots/github-light-default.png         │
│ 4 │ e8557e9b8cbb79b0705792cb316e46bfd98c5559 │ 240.6 KB │ themes/screenshots/github-light.png                 │
╰─#─┴──────────────────SHA-1───────────────────┴───size───┴──────────────────────filename───────────────────────╯

# Notice that `custom-completions/auto-generate/completions/mvn.nu` seems broken and therefore useless. So I propose to delete it as well.

 Let's check how much space non-'.png' files occupy
> $sizes | where filename !~ '.*\.png' | get size | math sum | print
7.0 MB

# Let's check the size of .png files
> $sizes | where filename =~ '\.png' | get size | math sum | print
253.4 MB

# Now we execute the command that will remove broken completions in `mvn.nu` and all binary `.png` files
python3 /Users/user/.local/bin/git-filter-repo --path custom-completions/auto-generate/completions/mvn.nu --path-glob '**/*.png' --invert-paths

> du .git/ | get 0.physical
4.4 MB

# The .git folder size was reduced from 236.2 MB to 4.4 MB.

# And now I push the results to my home repo
> git remote add mu https://github.com/maxim-uvarov/nu_scripts_reduced_size
> git push mu main --force

[fdncred]

Is there a way to render the themes as text/html/markdown/ansi so we could store that instead of images?