Merge branch 'master' into podAntiAffinity

Author: vartikashah

csc-overrides/partials/announcement.html

@@ -6,8 +6,8 @@
 #}
 {# import ".icons/material/coffee.svg" as icon #}
 {# import ".icons/material/chat.svg" as icon #}
-{# import ".icons/material/information.svg" as icon #}
-{% import ".icons/material/alert.svg" as icon %}
+{% import ".icons/material/information.svg" as icon %}
+{# import ".icons/material/alert.svg" as icon #}
 {# import ".icons/material/wrench.svg" as icon #}
 {# import ".icons/material/rocket-launch.svg" as icon #}
 {# import ".icons/material/test-tube.svg" as icon #}
@@ -18,7 +18,7 @@
 <div class="announcement-icon">{{ icon }}</div>
 {# Put the announcement, in HTML, right under this comment! #}
 <p>
-Due to a technical issue, some files cannot currently be imported from SD Connect to SD Desktop using the Data Gateway app. <a href="{{ base_url | url }}/support/wn/data-new/#sd-workaround">See temporary workaround here</a>.
+Note: Starting from June 11, 2025, using the MyCSC portal will require multi-factor authentication (MFA). <a href="{{ base_url | url }}/accounts/mfa/">Read our MFA guide here</a>.
 </p>
 {# ...and right above _this_ comment. #}
 <div class="announcement-icon announcement-icon-end">{{ icon }}</div>

docs/apps/geoconda.md

@@ -168,27 +168,15 @@ spyder
 * `os.environ["_JAVA_OPTIONS"] = "-Xmx4g"` in the beginning of your Python code.
 
 ### Google Earth Engine authentication set up in Puhti
-For using Google Earth Engine (GEE) API with `earthengine-api` package, one needs to have an account in GEE. Before first usage, also set up GEE authentication in Puhti:
-
-1. Open Puhti web interface
-2. Start Desktop app
-3. In the Desktop, open:
-    * Web Browser under Applications menu and
-    * Terminal from shortcuts
-4. In the Terminal:
+For using Google Earth Engine (GEE) API with `earthengine-api` package, one needs to have an account and project in GEE. Before first usage, also set up GEE authentication in Puhti:
 
 ```
 module load geoconda
-python
-
-import os
-os.environ['PATH'] = '/appl/opt/csc-cli-utils/google-cloud-sdk/bin:' + os.environ['PATH']
-
-import ee
-ee.Authenticate()
+export PATH=/appl/opt/csc-cli-utils/google-cloud-sdk/bin:$PATH
+earthengine authenticate
 ```
 
-This prints out a long link and asks for a code. Copy the link to the web browser and open the Google log-in page. Log-in and copy the created code back to Python.
+This prints out a long link and asks for a code. Copy the link to the web browser of your local laptop. Follow the instructions on the web page and finally copy the created code back to Terminal.
 
 ## Using Allas or LUMI-O from Python
 
@@ -224,20 +212,17 @@ Geoconda conda environment files and `download_wbt` and `start_wbt.py` needed fo
 
 ## References
 
--   [CSC Python parallelisation examples]
--   [Python spatial] libraries
--   [Geoprocessing with Python using Open Source GIS]
--   [GeoExamples], a lot of examples of using Python for spatial analysis
--   [Automating GIS processes course materials], where most of the exercises are done using Python (University of Helsinki)
--   [Geohack Week materials]
--   [Multiprocessing Basics]
--   [Geographic Data Science with Python]
--   [Aalto Spatial Analytics course material]
+-   [CSC Python parallelisation examples](https://github.com/csc-training/geocomputing/tree/master/python/puhti)
+-   [Multiprocessing Basics](https://pymotw.com/2/multiprocessing/basics.html)
+-   [Automating GIS processes course materials](https://automating-gis-processes.github.io) by University of Helsinki
+-   [Aalto Spatial Analytics course material](https://spatial-analytics.readthedocs.io/en/latest/course-info/course-info.html) by Henrikki Tenkanen / Aalto University
+-   [Introduction to GIS Programming](https://geog-312.gishub.org/index.html) by Dr. Qiusheng Wu / University of Tennessee
+-   [Geographic Data Science with Python](https://geographicdata.science/book/intro.html) by Sergio Rey, Dani Arribas-Bel, Levi Wolf
+-   [Python Foundation for Spatial Analysis](https://courses.spatialthoughts.com/python-foundation.html) by Ujaval Gandhi
 
 ------------------------------------------------------------------------
 
 
-  [Conda]: https://conda.io/docs/
   [cartopy]: http://scitools.org.uk/cartopy/
   [descartes]: https://pypi.python.org/pypi/descartes
   [fiona]: https://pypi.python.org/pypi/Fiona
@@ -261,16 +246,4 @@ Geoconda conda environment files and `download_wbt` and `start_wbt.py` needed fo
   [snappy]: https://senbox.atlassian.net/wiki/spaces/SNAP/pages/19300362/How+to+use+the+SNAP+API+from+Python
   [SNAP]: snap.md
   [spyder]: https://docs.spyder-ide.org/
-  [Conda environments]: https://conda.io/docs/user-guide/tasks/manage-environments.html#
-  [Bioconda]: bioconda.md
-  [Python spatial]: https://github.com/SpatialPython/spatial_python/blob/master/packages.md
-  [Geoprocessing with Python using Open Source GIS]: http://www.gis.usu.edu/%7Echrisg/python/2009/
-  [GeoExamples]: https://geoexamples.com/
-  [Automating GIS processes course materials]: https://automating-gis-processes.github.io
-  [Geohack Week materials]: https://geohackweek.github.io/schedule.html
-  [Multiprocessing Basics]: https://pymotw.com/2/multiprocessing/basics.html
   [sentinelsat]: https://sentinelsat.readthedocs.io/en/stable/index.html
-  [CSC Python parallelisation examples]: https://github.com/csc-training/geocomputing/tree/master/python/puhti
-  [Geographic Data Science with Python]: https://geographicdata.science/book/intro.html
-  [Aalto Spatial Analytics course material]: https://spatial-analytics.readthedocs.io/en/latest/course-info/course-info.html
-

docs/data/moving/copy_allas_ida.md

@@ -1,5 +1,6 @@
 # Copying data between Allas and IDA via Puhti
 
+
 ## Copying data from Allas to IDA via Puhti
 
 In order to be able to copy data from Allas to IDA with this procedure, you
@@ -114,15 +115,85 @@ for their username and password in IDA.
 If you do not need to continue working with the data on Puhti after uploading
 it to IDA, remove it from the scratch disk area of Puhti to free up disk space.
 
+
 ## Copying data from IDA to Allas via Puhti
 
-In order to copy data from IDA to Allas with this procedure, you need to be a
+In order to copy data from IDA to Allas, you need to be a
 member of a project that has Allas and Puhti services in use. On the IDA side,
 you either need to be a member of a project which has IDA service in use, or
 the data in question needs to be publicly available for downloading. Note that
 the projects in Allas, Puhti and IDA do not need to be the same.
 
-In short, there are four steps to follow:
+### Automatic copying with ida2allas tool
+
+If you wish to move a complete directory from IDA to Allas object storage service without
+any changes or rearrangements, you can use command line tool _ida2allas_. This tool is available 
+in Puhti server at CSC. 
+
+#### 1. Connect to Puhti
+
+We recommend that you run the data transfer process in one of the login nodes of the Puhti.
+(Login nodes can be used in this case as the data transfer process is not computationally heavy).
+The easiest way to open a login node session in Puhti is to use the WWW interface of Puhti:
+
+   * [Puhti Web interface](https://www.puhti.csc.fi)
+
+In the web interface, choose *Login node shell* from the *Tools* menu. This opens a
+terminal session where you can set up connections to IDA and Allas and where you can
+execute the data transport process.
+
+
+#### 2. Establishing connections
+
+First, open the Allas connection using the S3 protocol with the following commands:
+
+```text
+module load allas
+allas-conf -m S3
+```
+
+The _allas-conf_ command will prompt you to enter your CSC password. The Haka password is not accepted.
+After that you will choose the Allas project where the data will be copied.
+
+
+Next, establish the IDA connection with the commands:
+
+```text
+module load ida
+ida_configure
+```
+
+The configuration process will ask for your _IDA project number_ and  _app password_ that you can get from [IDA web interface](https://ida.fairdata.fi).
+If you already have a working IDA connection configured, you can choose not to overwrite the `.ida-config` and `.netrc` files. In that case, you won’t need to retrieve a new IDA key from the web interface.
+
+#### 3. Data transfer
+
+Data transfer is launched with the command:
+
+```text
+ida2allas
+```
+
+The program will first ask whether to fetch data from the IDA staging area or the frozen area.
+
+Then it will list the folders in the selected IDA area and prompt you to choose the folder to transfer.
+
+Finally, the program will list the buckets (storage folders) in Allas and ask you to select the one where you want to transfer the data. You can also create a new bucket. Do not use uppercase letters, spaces, or special characters in the name of the bucket. Further, note that bucket names must be unique compared to any Allas project. Thus it is a good habit to add some project specific part to the bucket name.  
+
+In case of large (over 100 GiB) data transfers you can start the transfer with the commands:
+
+```text
+screen
+ida2allas
+```
+
+In the example above, _screen_ command launches a virtual terminal session where the _ida2allas_ command will continue running even if the connection to Puhti would be lost.
+
+
+
+### Manual data transfer from IDA to Allas
+
+If you don't want to copy a complete IDA directory to Allas or if you want to rearrange the data, then you need to do the data transfer in four steps:
 
 1. Download the data from IDA to Puhti scratch disk
 2. Rearrange the data on the scratch disk, if necessary
@@ -190,7 +261,7 @@ cd /scratch/project_2000012/xferdir
 curl -fOJ "https://ida191.csc.fi:4430/download?token=18f6e5b7edae4f12a8a654ea22d57aa9.PA0p5PMqnzvgcXAU0Lw9SuVcyoQGgV8Ugnk3GEppU0b4UUhGWRLP8FRHB2MvyUTjPA0p5PMqnzvgcXAU0Lw9SuVcyoQGgV8Ugnk3GEppU0b4UUhGWRLP8FRHB2MvyUTjPA0p5PMqnzvgcXAU0Lw9SuVcyoQGgV8Ugnk3G_e3668097e34d437484e15d53624e7905=76679a7a-367c-474f-9e8c-c3869a106e2f_ehr3hd76&package=76679a7a-367c-474f-9e8c-c3869a106e2f_ehr3hd76.zip"
 ```
 
-### Step 2. Rearrange the data on scratch, if necessary
+#### Step 2. Rearrange the data on scratch, if necessary
 
 In case you would like to rearrange the data or remove parts of it, you can do
 it on the scratch disk before you upload it to Allas.
@@ -204,7 +275,7 @@ cd /scratch/project_2000012/xferdir
 unzip testi.zip
 ```
 
-### Step 3. Upload the data to Allas
+#### Step 3. Upload the data to Allas
 
 The easiest way to upload the data to Allas is using the `a-put` command.
 `a-put` uploads a directory as one archived object to Allas. It needs enough
@@ -229,7 +300,7 @@ cd /scratch/project_2000012/xferdir
 a-put experiment_data
 ```
 
-### Step 4. Clean the Puhti scratch
+#### Step 4. Clean the Puhti scratch
 
 If you do not need to continue working with the data on Puhti after uploading
 it to Allas, remove it from the scratch disk area of Puhti to free up disk

docs/data/moving/index.md

@@ -12,3 +12,4 @@ below the one that suits your needs best.
 * [Sharing and transporting files using Funet FileSender](funet.md)
 * [Moving data between IDA and CSC computing environment](../ida/using_ida.md)
 * [Remote disk mounts](disk_mount.md)
+* [Copying data between Allas and IDA via Puhti](copy_allas_ida.md)

docs/support/faq/index.md

@@ -71,6 +71,7 @@
 * [How to resize an instance or volume in Pouta?](how-to-resize-in-pouta.md)
 * [How to use cPouta for Remote Desktop](how-to-use-cpouta-for-remote-desktop.md)
 * [How to use cPouta GPU for rendering](how-to-use-cpouta-gpu-for-rendering.md)
+* [Is nested virtualization supported on Pouta?](is-nested-virtualization-supported.md)
 * [Is Pouta backed up?](is-pouta-backed-up.md)
 * [What does it cost to use Pouta?](what-does-it-cost-to-use-pouta.md)
 * [What are volumes? How do I use them?](what-are-volumes-and-how-to-use.md)

docs/support/faq/is-nested-virtualization-supported.md

@@ -0,0 +1,9 @@
+# Is nested virtualization supported on Pouta?
+
+No, for the time being nested virtualization is not enabled in Pouta. Nested virtualization disables live migrations, and at this moment we consider that Live migration is a much more useful feature for our user than nested virtualization.
+
+!!! Info "Nested virtualization"
+    Nested virtualization is a feature that allows you to run virtual machines (VMs) inside other VMs. This is useful for testing and development purposes
+
+!!! Info "Live migration"
+    Live migration is the process of moving a running virtual machine between different physical machines without rebooting or shuting down the virtual machine. There might be a short downtime during the migration, but the processes running on the virtual machine do not perceive this downtime.

scripts/report_changes.sh

@@ -1,6 +1,8 @@
 #!/bin/bash
 set -e
 
+GH_BASE="https://github.com/CSCfi/csc-user-guide/blob/master"
+
 usage="Usage: $(basename "$0") [-h] [-u]
 Print last update timestamp and most recent committer for each Docs page.
 
@@ -44,7 +46,7 @@ while read -r line ; do
     elif [[ "$line" && -z "${seen[$line]}" ]] ; then
         seen["$line"]="$date"
     fi
-done < <(git log --format="/%H %as %an" --name-only "$GIT_LOG_OPTS")
+done < <(git log --format="/%H %as %an" --name-only $GIT_LOG_OPTS)
 
 echo "|Updated|Filename|Head line|"
 echo "|:-:|:-:|:-:|"
@@ -60,7 +62,7 @@ git ls-files "$path*.md" | while read -r f ; do
         echo "##### $f"
       fi
 
-      echo "|${seen[$f]}|$f|$head_line|"\
+      echo "|${seen[$f]}|$GH_BASE/$f|$head_line|"\
       | iconv -f iso-8859-1 -t utf-8
     fi
 done | sort -r | grep "$name"