Jail does not start at boot -> vncport use on 2nd start / after reboot

[n-connect]

Hi @olevole

There's some hickups over xigmajail created jails. At first they do start & run plus provide the web based terminal via ttyd with the -c parameter.

After a reboot they won't start, it seems the reason beding the vnc_port file got lost (?) and a second file created as `vnc_port2

Listed in the other issue already, the cbsd jcounstruct-tui fails to run due to missing binary, you can check it in #1

I guess, there were some key changes with the release 13.3.05 :)

xigmanas: ~# cbsd jls
JNAME  JID  IP4_ADDR  HOST_HOSTNAME    PATH                    STATUS
jail1  0    10.0.0.2  jail1.my.domain  /data/cbsd/jails/jail1  Off
xigmanas: ~# cbsd jstart
List of offline jail
 0 .. CANCEL
 a .. jail1 on local

Default NIC automatically selected: re0
set resource limit: [ vmemoryuse:deny=256 ]
jail renice: 1
Starting jail: jail1, parallel timeout=5
jail1: created
ld-elf.so.1: morepages: cannot mmap anonymous memory: Cannot allocate memory
ld-elf.so.1: morepages: cannot mmap anonymous memory: Cannot allocate memory
Out of memory
jail: jail1: /bin/sh /etc/rc: failed
jail1: removed
Execute master script: expose.sh
 :: /data/cbsd/jails-system/jail1/master_poststart.d/expose.sh
head: /data/cbsd/jails-system/jail1/vnc_port: No such file or directory
CBSD Expose for jail1: 3000 -> 22 (tcp)
CBSD: Enable IP forwarding for NAT service
pfctl: pf already enabled
CBSD Expose for jail1: 3000 -> 22 (tcp)
CBSD: Enable IP forwarding for NAT service
pfctl: pf already enabled
CBSD Expose for jail1: 3000 -> 22 (tcp)
CBSD: Enable IP forwarding for NAT service
pfctl: pf already enabled
CBSD Expose for jail1: 3000 -> 22 (tcp)
CBSD: Enable IP forwarding for NAT service
pfctl: pf already enabled
jstart done in 9 seconds
xigmanas: ~# ls -la /data/cbsd/j
jails-data/   jails-fstab/  jails-rcconf/ jails-system/ jails/
xigmanas: ~# ls -la /data/cbsd/jails-system/jail1/
total 43
drwxrwxr-x  5 root  cbsd     21 May 20 09:29 .
drwxrwxr-x  3 root  cbsd      4 May 20 09:00 ..
drwxrwxr-x  3 root  cbsd      4 May 20 09:00 etcupdate
-rw-rw----  1 cbsd  cbsd   8192 May 20 09:29 expose.sqlite
-rw-rw----  1 cbsd  cbsd  32768 May 20 09:29 expose.sqlite-shm
-rw-rw----  1 cbsd  cbsd      0 May 20 09:29 expose.sqlite-wal
drwxrwxr-x  2 root  cbsd      5 May 20 09:00 helpers
-rw-r--r--  1 root  cbsd     20 May 20 09:29 jail.limits
-rw-r--r--  1 root  cbsd     74 May 20 09:00 jail.limits.extra
-rw-rw----  1 cbsd  cbsd  16384 May 20 09:00 local.sqlite
-rw-rw----  1 cbsd  cbsd  16384 May 20 09:00 local.sqlite-20240520090528.bkp
-rw-rw----  1 cbsd  cbsd  32768 May 20 09:29 local.sqlite-shm
-rw-rw----  1 cbsd  cbsd      0 May 20 09:05 local.sqlite-wal
drwxr-xr-x  2 cbsd  cbsd      3 May 20 08:54 master_poststart.d
-rw-rw-r--  1 root  cbsd      0 May 20 09:00 media.sql
-rw-rw-r--  1 root  cbsd    535 May 20 09:00 rc.conf_jail1
lrwxrwxr-x  1 root  cbsd     46 May 20 09:29 stats.conf -> /data/cbsd/jails-system/jail1/stats.conf.15082
-rw-rw-r--  1 cbsd  cbsd    504 May 20 09:29 stats.conf.15082
lrwxrwxr-x  1 root  cbsd     46 May 20 09:29 stats.json -> /data/cbsd/jails-system/jail1/stats.json.15082
-rw-rw-r--  1 cbsd  cbsd    500 May 20 09:29 stats.json.15082
-rw-rw-r--  1 root  cbsd     23 May 20 09:29 vnc_port2
xigmanas: ~# cbsd jstop
List of online jail
No jail specified

[n-connect]

@olevole

Is there any debug.log capability in either xigmajail or in CBSD so I can trace what happens with the original vnc_port file?

[olevole]

@n-connect I can confidently say that the vnc_port2 file is not served by the CBSD. At least, I do not remember (and did not find) that this file is controlled by the CBSD.

Moreover, it is very strange that you see this file in the jail container catalog. This file refers to virtual machines.

It is quite difficult for me to test some things with Xigma at the moment, because I am limited by the presence of equipment (Two of my production XigmaNAS instance are now far from me).

I tried to double -check the jails terminal (via ttyd) operation on a clean fresh system (virtual machine) and it works for me.

I apologize for this form of demonstration, but I decided to show all the way, starting with the installation of Xigma:

part 1, installation + partition: https://youtu.be/9XCbfPOdz2E ( My clumsy attempts to create ZFS pool for data for some reason failed, so I created the UFS )

part 2, CBSD jail extension: https://youtu.be/ymTjXOGSUxE ( There is a certain issue associated with the routing of my workstation, where I launched a virtual machine. So in a demo, the container did not have Internet access for pkg bootstrap. Also, I noticed the problem that the container did not initiate the IP address (aliases ) at the first start - this needs to be investigated. Nevertheless, I wanted to check only the graphic terminal and it seems to be a complete order with it. )

[n-connect]

@olevole

Thank you for the video output and check. Based on that, it seems with the UFS formatted data disk there is no issue with 3rd level of directories used for CBSD framework files (/mnt/mnt were the "dir" value in the plugin's ). So /mnt/mnt/cbsd/*` must be CBSD content. -> This was the first problem at #1

Based on your video, I'll recheck the first , original issue, now I have the feeling my first try making ZFS data disk over XigmaNAS' webUI was a wrong halfway somehow and be the root cause CBSD can't put its own files under "/mnt/data/cbsd/*". I need to find some reference how on Earth XigmaNAS people thought the ZFS format / vdev / mount process should be done sucessfully over their webUI :D

Please do a reboot and check again the two jails you have created if they are able to run/you can log into them -> that was the 2nd issue, this one #2 - (after I did made my ZFS formatted data disk mounted under /data)

About the XigmaNAS's ZFS format/vdev/mount, as I wrote at #1 , I could not make the ZFS data disk over XigmaNAS webui too :D - turned to SSH and with two commands it can be done:

---

1. Optional: remove the existing UFS partition, but only if you still have it

• gpart destroy -F /dev/vtbd1
• dd if=/dev/zero of=/dev/vtbd1 bs=512 count=8192
• gpart create -s gpt /dev/vtbd1
• gpart add -a 1m -t freebsd-zfs -i 1 -l 'data'

2. The only two commands needed if you have fresh data disk, quoted from my previous comment at xigmajail (& xismabhyve) fails to install specified directory #1 , disk device adapted based on your YT video.

• zpool create -f -O compress=zstd data /dev/vtbp1 (-f to overwrite the previously created vdev over the buggy Disks/ZFS/Pool/Virtual Devises webui)
• zfs set mountpoint=/data data
• the created zpool and its mount works, but the Disks/ZFS/Pools not showing the vdev/pool etc. Only the "configuration" has correct data: but for zroot and data pools too. It must be some XigmaNAS bug. From now on Truenas CORE and its webui looks better for me, by the way.

---

Last note: I'm doing these test on a Fujitsu Futro s920/s930 available from €20 on the eBay link for example.

[n-connect]

As I wrote back in the #1, using a fresh install the jail creation over webUI works. Jail settings editing works, after reboot the connected jail start.

But I can not "login" via the browser based terminal (backed with ttyd). It always asks for passwords in the browser. Here are the out put for processes, (fresh XigmaNAS install, still using the default user/pw yet):

root  15440   0.0  0.1  40744 21568  -  Ss   16:04    0:00.02 |-- ttyd -o -m 1 -t titleFixed=jail:jail1 -c 1c76b3fddbe68a9dabebdeb79713fece:1c76b3fddbe68a9dabebdeb79713fece --writable -p 7681 /usr/sbin/jexec

Decoding the -c parameter give back some garbage, even the newline disappears:

bash-3.2$ echo "1c76b3fddbe68a9dabebdeb79713fece" | base64 -D
���ow�u���]i��u���w}�bash-3.2$

The "cbsd expose" has 3 times the redirect rule from some reasons :D

cbsd expose
Expose for jail1:
3000 -> 22 (inaddr=0 -> 192.168.101.158 tcp)
3000 -> 22 (inaddr=0 -> 192.168.101.158 tcp)
3000 -> 22 (inaddr=0 -> 192.168.101.158 tcp)

So, now its reduced to the webUI console login issue. Should I rename the issue?

[olevole]

Thanks for the details. It is not necessary to rename - all the information here is described. As soon as I get access to Xigma, I will definitely check it

[n-connect]

@olevole

Good news :D

1. The ttyd usability is browser dependent! In Chrome, in Firefox it works, in Safari it is not asking for PW. Copying out manually from ps auxd gives a Reconnecting ttyd window. So its only not working in Safari, needs some research whatis the reason...
2. I could recreate the issue with the vncport. If you define a jail with memory limit, the jail do not start. Trying to start it gives you these console output:

xigmanas: /# cbsd jstart
List of offline jail
 0 .. CANCEL
 a .. jail2 on local

Default NIC automatically selected: cbsd0
Set CPU limit jail2: 1
set resource limit: [ vmemoryuse:deny=128 ]
jail renice: 1
Starting jail: jail2, parallel timeout=5
jail2: created
ld-elf.so.1: morepages: cannot mmap anonymous memory: Cannot allocate memory
ld-elf.so.1: morepages: cannot mmap anonymous memory: Cannot allocate memory
Out of memory
jail: jail2: /bin/sh /etc/rc: failed
jail2: removed
Execute master script: expose.sh
 :: /data/cbsd/jails-system/jail2/master_poststart.d/expose.sh
head: /data/cbsd/jails-system/jail2/vnc_port: No such file or directory
expose warning: port already in use on 192.168.101.158: tcp/3000
expose: fw_expose_add error: 1
jstart done in 3 seconds

Editing the properties of the very same jail jail2, the form do not shows the current values (1 cpu, 128Mb RAM), but the unlimited values. Plus can't change the vaules, either to unlimited or to another limit value (tested in Chrome)
3. Beside the vncport problem, the cbsd expose on the 2nd jail (jail2) can't add again port 3000 for SSH (port 22) redirect. Connected starter variable is at cbsd_manager_config.php line 216, cbsd_rdr

Some screenshots for the end with the same numbers up:

1. Ttyd works in Chrome
   
   Does not in Safari
   

2. Mem limited jail2
   
   Cbsd jstart for jail2 in CLI
   

3. Multiple port 3000 redirects for jail1 (4 pieces), jail2 can't have its own redirect, tries the same port (3000), I guess the port number in PHP needs and increment logic for external port numbers.
   

[olevole]

@n-connect Thanks for updating/info! I'm not ignoring you, unfortunately I don't have a XigmaNAS+CBSD right now ;-)

I currently have two XigmaNAS instances (without CBSD/jail/bhyve) in different cities/location. I understand that using virtual machines or running containers on NAS instances is not always a popular solution (even from the point of view of NAS developers). However, I don't like any resources being idle (CPU and memory - I have quite powerful hardware)

Running applications and virtual machines on NAS is a nice hyperconverged solution ( especially for home use when you are limited in space and want to save electricity ) that doesn't require you to install another hypervisor next to NAS.

Due to the fact that I have two NAS and they are not directly accessible from the Internet, it is not very convenient for me to access virtual environments, so before I return to development CBSD extenstion for Xigma NAS, I would like to implement some kind of router (service) on the Internet, through which I can connect my virtual environments from NAS appliance.

CBSD already supports API (+broker), In addition, a graphical QT6-based application for CBSD will be published soon, which will be available on all OS (MacOS, Linux, Windows, BSD) and mobile devices (tested on Android), demo: https://forums.freebsd.org/threads/welcome-bhyve-qt-alpha-0-3-first-look.91532/

Thus, I see the use of the CBSD XigmaNAS plugin + NAS for myself as follows (in addition to local use, as is done now):

Also, I was interested in this issue on the XigmaNAS forum, but so far no interest from the Xigma developers: https://www.xigmanas.com/forums/viewtopic.php?t=2111

For this reason (need to investigate before returning to development of the extension) I am currently delaying work on finalizing the extension, but I take your comments into account and really appreciate them ;-)

[olevole]

maybe there is some problem with ttyd + Safari ( https://github.com/tsl0922/ttyd/issues?q=is%3Aissue+safari+is%3Aclosed ), have you tried the Safari + ttyd without Xigma and cbsd? it is accessible from FreeBSD ports: /usr/ports/sysutils/ttyd

[n-connect]

@olevole

Will check the ttyd vs Safari at the weekend, thanks.

Your network landscape drawing pretty much the same idea I had (after you I guess), but I weren't tried the the API yet, not used the multinode capabilities of CBSD. I'll need to check them. From another angle, calling them a NAS it just a title we refer to it, it can be anything FreeBSD can provide. I need to be active on XigmaNAS forum to get know better their install type/logic/limitations.

What's the best link to check-out/using the CBSD API? The BSDstore web page seems sometimes outdated. Got your best answer in a CBSD bhyve/network topic at the main freebsd forum for example :)

I saw you are stretched thin with a lot of parallel development directions, seeing the patreon as well.
Seen how the QT client works, will try once you release it.

So, I'll take this thread easy than, replace Xigma with with a vanilla Freebsd 14/14.1 and finish the host pf stuff integration on that for CBSD.

Is there any direct way to communicate with you, maybe talk/chat IRC whatever, not public like here? There are some questions around CBSD I'd like to ask in general, or the direction. Also ran into some interesting parts (missing ZFS jailed property in running jails, compared to iocage for example), where I'm not sure if I made something wrong so maybe a quick talk would help.

[olevole]

The fastest and painless way to get acquainted with the CBSD API - just download & install (on bare metal if you want to run bhyve) MyBee distribution:

1. https://myb.convectix.com -> download
2. install MyBee (in fact it is FreeBSD + CBSD + API, without any modifications)
3. to test via curl, e.g. debian12.json file:

{
  "imgsize": "20g",
  "ram": "1g",
  "cpus": 2,
  "image": "debian12",
  "pubkey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPY9ozA89YZMGT9ZCZX3oZImqt+a0FO9qQmJA+1O7YU6 cloud_ssh-ed25519_cloud@localhost"
}

where pubkey is your ED25519/RSA ssh PUBKEY

or payload for jail:

{
  "image": "jail",
  "imgsize": "10g",
  "pubkey": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPY9ozA89YZMGT9ZCZX3oZImqt+a0FO9qQmJA+1O7YU6 cloud_ssh-ed25519_cloud@localhost"
}

Submit request:

curl --no-progress-meter -X POST -H "Content-Type: application/json" -d @debian12.json http://MyBEE-instance/api/v1/create/vm1

If you use Telegram IM, welcome to https://t.me/cbsdofficial (Telegram CBSD group) or ping me at https://t.me/olevole

Otherwise, I'm always available by email: olevole at ya.ru