Skip to content

Commit b8a3fe2

Browse files
authored
note about PyMongo and update websocat (#1675)
* update PyMongo version to use * update lsof command So far, this command will only return the port of the most recently websocat process. If you have another localhost process for other purposes (i.e python3 -m http.server --bind localhost), this command won't return the python process.
1 parent 67289f7 commit b8a3fe2

File tree

1 file changed

+9
-4
lines changed

1 file changed

+9
-4
lines changed

docs/cloud/rahti/tutorials/connect-database-hpc.md

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
# Accessing databases on Rahti from CSC supercomputers
22

3+
!!! infos
4+
This tutorial can also be applied if you are running websocat locally or in a cPouta instance for other purposes.
5+
36
Many HPC workflows require a database. Running these on the login node poses several issues and running on Pouta brings administration overhead. Rahti is a good candidate, but one obstacle is that Rahti does not support non-HTTP traffic from external sources.
47

58
A workaround for this problem is to establish a TCP tunnel over an HTTP-compatible WebSocket connection. This can be achieved using a command-line client for connecting to and serving WebSockets called [WebSocat](https://github.com/vi/websocat). Here, a WebSocat instance running on Puhti/Mahti translates a database request coming from a workflow to an HTTP-compatible WebSocket protocol. Once the traffic enters Rahti we use another WebSocat instance running inside Rahti to translate back the WebSocket connection to a TCP connection over the original port the database is configured to receive traffic. A drawing of the process is shown below.
@@ -37,8 +40,7 @@ Configuring MongoDB and WebSocat on Rahti can be done either through the web int
3740
- An [OpenShift template](https://github.com/CSCfi/websocat-template/blob/main/websocat-template.yaml) is needed to configure WebSocat on Rahti. Download or copy this YAML file to your clipboard. Note that this is an unsupported beta template
3841
- On the Rahti web interface front page, select Import YAML/JSON and add the WebSocat template to the same project as the MongoDB (upload by dragging & dropping, selecting it, or pasting from the clipboard). Do not edit the template!
3942
- Select Create, process the template and input the above hostname and target port in the requested database service name and database port fields
40-
- Select the created project, navigate to `Applications > Services` and select the WebSocat service. Remember the
41-
- Route hostname (of the form `websocat-<project name>.rahtiapp.fi`)
43+
- Select the created project, navigate to `Applications > Services` and select the WebSocat service. Remember the route hostname (of the form `websocat-<project_name>.rahtiapp.fi`)
4244

4345
### Option 2: Using the `oc` command line tool
4446

@@ -78,7 +80,7 @@ MongoDB and WebSocat have now been set up on Rahti and you should have the follo
7880
- [Download `websocat` from GitHub](https://github.com/vi/websocat/releases) and add it to your `PATH`. For example:
7981

8082
```bash
81-
wget -O websocat https://github.com/vi/websocat/releases/download/v1.8.0/websocat_amd64-linux-static
83+
wget -O websocat https://github.com/vi/websocat/releases/download/v1.11.0/websocat.x86_64-unknown-linux-musl
8284
export PATH=$PATH:$PWD
8385
```
8486

@@ -88,7 +90,7 @@ export PATH=$PATH:$PWD
8890
websocat -b tcp-l:127.0.0.1:0 wss://websocat-<project name>.rahtiapp.fi -E &
8991
ws_pid=$! # $! contains the process ID of the most recently executed background command
9092
mkdir -p /tmp/$USER
91-
lsof -i -p $ws_pid 2>/dev/null | grep TCP | grep -oE "localhost:[0-9]*" | cut -d ":" -f2 > /tmp/$USER/${SLURM_JOB_ID}_rahtidb_port
93+
lsof -Pa -p $ws_pid -i -sTCP:LISTEN | grep -v ^COMMAND | sed -e 's#.*TCP\s*\S*:\([0-9]*\).*#\1#' > /tmp/$USER/${SLURM_JOB_ID}_rahtidb_port
9294
echo "Got target port $(cat /tmp/$USER/${SLURM_JOB_ID}_rahtidb_port)"
9395
```
9496

@@ -97,6 +99,9 @@ echo "Got target port $(cat /tmp/$USER/${SLURM_JOB_ID}_rahtidb_port)"
9799

98100
- Now `websocat` is running in the interactive session/batch job and you may connect to your MongoDB database on Rahti using the obtained target port. You can verify the connection with e.g. Python. Note that the username and password below refer to the created database service, not your CSC credentials
99101

102+
!!! info
103+
PyMongo version 3.13.0 must be installed otherwise the driver is too new for the MongoDB server. (More information: https://pymongo.readthedocs.io/en/stable/common-issues.html#server-reports-wire-version-x-pymongo-requires-y)
104+
100105
```python
101106
# module load python-data
102107

0 commit comments

Comments
 (0)