Ensure OnDemand app installs during image build do not need a cluster to be deployed (#843)

* Ensure OOD app installs work for site image builds

* add ood app image build docs

* fix partition defaults and document

* Add detail to docs

* add newlines

* fix linting

* markdown prettier

* bump CI images

* Fix trivyscan action version

* fix cluster_compute_group group_var

* remove cluster compute group safe var indirection

* Bump new CI images

Author: bertiethorpe

ansible/roles/openondemand/README.md

@@ -69,10 +69,10 @@ This role enables SSL on the Open Ondemand server, using the following self-sign
   - `new_window`: Optional. Whether to open link in new window. Bool, default `false`.
   - `app_name`: Optional. Unique name for app appended to `/var/www/ood/apps/sys/`. Default is `name`, useful if that is not unique or not suitable as a path component.
 - `openondemand_dashboard_support_url`: Optional. URL or email etc to show as support contact under Help in dashboard. Default `(undefined)`.
-- `openondemand_desktop_partition`: Optional. Name of Slurm partition to use for remote desktops. Requires a corresponding group named "openondemand_desktop" and entry in openhpc_partitions.
+- `openondemand_desktop_partition`: Optional. Name of Slurm partition to use for remote desktops, by default supplied with `openhpc_partitions` entry. During open ondemand config the string is used to provide a default partition in the UX. During image build, with `openondemand` group, setting this partition as a boolean determines if app installed in image.
 - `openondemand_desktop_screensaver`: Optional. Whether to enable screen locking/screensaver. **NB:** Users must have passwords if this is enabled. Bool, default `false`.
 - `openondemand_filesapp_paths`: List of paths (in addition to $HOME, which is always added) to include shortcuts to within the Files dashboard app.
-- `openondemand_jupyter_partition`: Required. Name of Slurm partition to use for Jupyter Notebook servers. Requires a corresponding group named "openondemand_jupyter" and entry in openhpc_partitions.
+- `openondemand_jupyter_partition`: Required. Name of Slurm partition to use for Jupyter Notebook servers, by default supplied with `openhpc_partitions` entry. During open ondemand config the string is used to provide a default partition in the UX. During image build, with `openondemand` group, setting this partition as a boolean determines if app installed in image.
 - `openondemand_gres_options`: Optional. A list of `[label, value]` items used
   to provide a drop-down for resource/GRES selection in application forms. The
   default constructs a list from all GRES definitions in the cluster. See the

docs/openondemand.md

@@ -63,6 +63,14 @@ The appliance automatically configures Open OnDemand to proxy Grafana and adds a
 
 [^1]: Note that if `openondemand_auth` is `basic_pam` and anonymous Grafana login is enabled, the appliance will (by default) configure Open OnDemand's Apache server to remove the Authorisation header from proxying of all `node/` addresses. This is done as otherwise Grafana tries to use this header to authenticate, which fails with the default configuration where only the admin Grafana user `grafana` is created. Note that the removal of this header in this configuration means it cannot be used to authenticate proxied interactive applications - however the appliance-deployed remote desktop and Jupyter Notebook server applications use other authentication methods. An alternative if using `basic_pam` is not to enable anonymous Grafana login and to create Grafana users matching the local users (e.g. in `environments/<env>/hooks/post.yml`).
 
+## Image Build
+
+By default, most ondemand apps are installed in image builds when the build includes the inventory group `openondemand` (which is the default for "fatimage" builds). The apps installed are
+defined by the `openondemand_<app>_partition` variables in `environments/common/inventory/group_vars/all/builder/defaults.yml`. Note that in this case the values are not strings and are instead
+simply truthy, i.e. they do not describe cluster partition groups but just whether those apps will be installed in the image or not.
+
+For e.g. site-specific image builds where different app installs are required, due to precedence rules these must overriden in a `builder`-groupvars file e.g. `environments/site/inventory/group_vars/all/builder/defaults.yml`.
+
 ## Access
 
 By default the appliance authenticates against OOD with basic auth through PAM. When creating a new environment, a new user with username `demo_user` will be created.

environments/.stackhpc/tofu/cluster_image.auto.tfvars.json

@@ -1,6 +1,6 @@
 {
   "cluster_image": {
-    "RL8": "openhpc-RL8-251119-1202-332ac921",
-    "RL9": "openhpc-RL9-251119-1202-332ac921"
+    "RL8": "openhpc-RL8-251119-1833-cb477455",
+    "RL9": "openhpc-RL9-251119-1834-cb477455"
   }
 }

environments/common/inventory/group_vars/all/openhpc.yml

@@ -27,7 +27,7 @@ openhpc_rebuild_partition: # not a role var - could actually add more indirectio
     DisableRootJobs: false
     PreemptMode: "OFF"
     OverSubscribe: EXCLUSIVE
-openhpc_nodegroups: "{{ cluster_compute_groups | map('community.general.dict_kv', 'name') }}" # create nodegroup for each compute group
+openhpc_nodegroups: "{{ cluster_compute_groups | default([]) | map('community.general.dict_kv', 'name') }}" # create nodegroup for each compute group
 openhpc_user_partitions: "{{ openhpc_nodegroups }}" # create partition for each nodegroup (actually role default) - this is what we'd expect to be changed
 # yamllint disable-line rule:line-length
 openhpc_partitions: "{{ openhpc_user_partitions + ([openhpc_rebuild_partition] if groups['rebuild'] | length > 0 else []) }}" # auto-create rebuild partition if reqd.

environments/common/inventory/group_vars/all/openondemand.yml

@@ -12,11 +12,12 @@ openondemand_servername: "{{ hostvars[groups['openondemand'].0].ansible_host if
 
 openondemand_auth: basic_pam
 
-openondemand_jupyter_partition: "{{ openhpc_partitions[0]['name'] }}"
-openondemand_desktop_partition: "{{ openhpc_partitions[0]['name'] }}"
-openondemand_rstudio_partition: "{{ openhpc_partitions[0]['name'] }}"
+# Manages case where openhpc_partitions are not defined e.g. for site image builds
+openondemand_jupyter_partition: "{{ openhpc_partitions[0].name | default('') }}"
+openondemand_desktop_partition: "{{ openhpc_partitions[0].name | default('') }}"
+openondemand_rstudio_partition: "{{ openhpc_partitions[0].name | default('') }}"
 openondemand_matlab_partition: '' # Requires target site to already have MATLAB so set to empty
-openondemand_codeserver_partition: "{{ openhpc_partitions[0]['name'] }}"
+openondemand_codeserver_partition: "{{ openhpc_partitions[0].name | default('') }}"
 
 # Regex defining hosts which openondemand can proxy; the default regex is compute nodes (for apps) and grafana host,
 # e.g. if the group `compute` has hosts `compute-{0,1,2,..}` this will be '(compute-\d+)|(control)'.

environments/common/inventory/group_vars/builder/defaults.yml

@@ -27,3 +27,9 @@ sssd_enabled: false
 slurm_exporter_state: stopped
 appliances_mode: build
 proxy_remove: true
+# for image build ood partition var is just truthy as no cluster_groups defined
+openondemand_jupyter_partition: true
+openondemand_desktop_partition: true
+openondemand_rstudio_partition: false
+openondemand_matlab_partition: false
+openondemand_codeserver_partition: false