Curated packages repository

[pepe]

As discussed in #554 (comment), there is the plan to hot up the game of packages for Janet language.

Here is the best place (for now) to come with your opinions on the issue.

And as always, EJIFG!

[sogaiu]

To curate, I presume we need something to act upon, thus...the following lists contain collections of Janet code repositories:

• https://github.com/ahungry/awesome-janet
• https://gist.github.com/felixr/5302f3f80b0510b402b8eaffa96dc77b
• https://gitlab.com/-/snippets/2054321

The first list is @ahungry's curated list that includes things related to Janet (so not just code repositories).

IIUC, the second list is created by @felixr in an (mostly?) automated fashion -- possibly via some GH API.

The last list is the result of manual collection. It also has some things in it that are not GH repositories (e.g. sr.ht and gitlab).

[pepe]

Great research as always and a very good start. Thank you.

[felixr]

Yes, my list came from a search via the github API. You need an API token, otherwise it wont allow you to search for filenames across all repos. You can generate a list with something like this (replace xxxTOKENxxx with your API token):

for page in $(seq 10); do
 curl -H'authorization: token xxxTOKENxxx' "https://api.github.com/search/code?q=filename%3Aproject.janet&page=${page}" > page${page}.json;
done  
jq -r '.items[] | select(.name="project.janet").repository | "https://github.com/\(.full_name)  \(.description)"' page*.json | sort -u

To get stars and created time you need to query each repo individually AFAIK.

[uvtc]

What aspects of the following should Janet's pkgs embody?

• a curated list of Janet libraries (like, for example, clojure-toolbox)
• a comprehensive janet package directory (like, eg, clojars)
• a comprehensive awesome-janet list
• include "ratings" (like, eg., Perl's cpanratings, or num of github stars)

Regarding that last one, as an example, this page of Raku's module directory displays badges with numbers of likes, and status of things like passing-tests, last-modified, current version, most-recent release, and so on.

Some other observations about what other projects have/use:

• I think Perl has something called a module's kwalitee, which is tested by its cpants service.
• Haxelib, luarocks, and Rust's package registry show numbers of downloads. One thing that seems like it would be useful would be to see number of downloads in the previous week.
• Julia appears to have an impressive packaging infrastructure.

[bakpakin]

The original purpose of pkgs is just to provide short names for jpm install [sompackage] so a user doesn't need to remember full package names, but we could do so much more. I think an important thing to separate is the registry data, and a site/frontend for it. The data in janet-lang/pkgs could be consumed by many different sites/programs.

The Julia package link is very interesting - they have a repository very similar in purpose to pkgs - https://github.com/JuliaRegistries/General - but much more complete. It is much how I had envisioned janet-lang/pkgs could work - a central repository that has lots of automated scripts for adding packages, updating packages, fetching package metadata, and so on.

[pepe]

I reviewed Julia Registrator and again found it a great place to steal the ideas from. I am not sure I would go with all of this structure (start letter - project dir - a couple of files for a project), but some more metadata will be good to have. This I would leave as one aspect, that needs to be discussed more.

[nektro]

yeah janet-lang/pkgs reminds me a lot of https://github.com/rust-lang/crates.io-index. this repo is automatically updated by some CI infrastructure and used to feed sites like https://crates.io and https://docs.rs.

[pepe]

Thank you all for your opinions and ideas!

I was thinking about the stuff written here, and after research done on the examples given, I got this plan for now:

• in the first phase, everything will stay the same from an infrastructure point of view. I will control the package's quality and correctness just manually or ask for a review by @bakpakin in the case I am not sure about something. I will add a more detailed summary of what the quality package looks like in my eyes to CONTRIBUTING.md. I want to add instructions about the information I would like to see in the PR. Examples can be platforms package can be used on, if and what native libraries package wraps, the caveats, and if the code is tested. PR can be searched if there is a need for that information.
• in the second phase, I would like to add metadata information, which will have to be present for the package. Metadata will be separated from the current table, which I would like to stay the same. I am not sure about the format now, but I will strive for simplicity. This metadata will be possible to transform into a static website hosted on the GitHub pages.
• in the third phase, I would love to have infrastructure similar to what Julia has but tailored to the second bullet's simple structure. All the scripts will be written in Janet ofc.

Please let me know if this makes sense to you.

[uvtc]

if and what native libraries package wraps

Ah. Forgot about that one. Yes, will be good to know that!

[pepe]

I found this idea by @uvtc very nice MikeBeller/janet-set#1 (comment). What do you think?

[pepe]

It came to my mind that we already have another metadata store in project.janet. The simple Mendoza site, which fetches and parses the project.janet, and builds static representation, would be a perfect start.

Then we can have simple rules about the content of the project.janet in the CONTRIBUTING.

But I am not sure if there is any limitation on fields in declare-project?

What do you think?

[ahungry]

@pepe - "I will control the package's quality and correctness just manually or ask for a review by @bakpakin in the case I am not sure about something."

This seems like it could end up becoming quite arbitrary if the "quality and correctness" is on a case by case basis at the discretion of one or more people.

Would it make more sense to come up with a set of rules/bare minimums to meet the "quality and correctness" case? Then code authors/package contributors will know if they meet all criteria on the list, their code would be fine for listing?

[jcmkk3]

In addition to Julia, it might we worth taking a look at the guidelines for CRAN (R’s primary repository) submission. https://cran.r-project.org/web/packages/policies.html

[pepe]

Sorry for the vagueness. I am off the computer shift today, but I will write more tomorrow.

[pepe]

@ahungry: I took it as the first kick-off. I am not sure that we ever discussed the GH issues, but I am hardly known as too arbitrary. It's written in the first person because it is my plan. I would be happy to take any help on all the phases. Even redefining the plan itself. And I am sure that you are one of the people which input will be precious.

About "quality and correctness." When I wrote this, I meant: the library should not crash on start and not fdisk /dev/sda. Not that it should have been formatted to my taste, and the names of the variables are too long. Sorry for not writing anything there, but I wanted the post not to be too convoluted.

Anyway, please can you give me examples of the rules/bare minimums to meet? Please reply to the original post: #563 (comment).

@jcmkk3: thank you. If you have any specific things on your mind, please comment on the original post too. At the end of the first phase, I would like to have something similar (but shorter, frankly).

EJIFG

[pepe]

In the janet-lang/pkgs#21 is the new CONTRIBUTING, please comment if you can.

[yumaikas]

So, one feature of a non-zero number of janet packages is that they don't necessarily work across operating systems. I don't think that they should be required to, but it would be nice if there was some package meta-data that let one know if a package was intended to work on windows vs linux vs osx. I could see arguments for supporting both a whitelist (for windows/osx specific packages, for example), or a blacklist (a package that relies heavily on posix apis, and isn't interested in the effort to make things work on windows. The sh package comes to mind).

[pepe]

Thanks, I am now thinking about reusing the project.janet as a metadata store for the pkgs (still a very early stage idea), and this could be part of it (or not) :)