[bug]: Fresh Install stuck in `Waiting for system cert key file...`

[storopoli]

Prerequisites

• I have searched for existing issues that already report this problem.

Device

Laptop/Desktop

Device OS

Other

Device OS Version

startos

Browser

Firefox

Browser Version

127

Current Behavior

Installed CLN from scratch and now it is stuck in

Waiting for system cert key file...

I did a search and found that I am hitting this line:

cln-startos/docker_entrypoint.sh

Lines 106 to 111 in 29f0fe7

	while ! [ -e /mnt/cert/rest.cert.pem ]; do
	echo "Waiting for system cert..."
	sleep 1
	done
	cp /mnt/cert/rest.cert.pem /usr/local/libexec/c-lightning/plugins/c-lightning-REST/certs/certificate.pem
	# fi

I have no idea what's going on. My start9 /mnt dir does not have anything.
Shouldn't CLN generate a system cert key file?

Cc @chrisguida since 48e6bbb introduce these lines.

Expected Behavior

CLN should start from a fresh install in StartOS and not be stuck in a infinite loop.

Steps to Reproduce

1. Install CLN from scratch
2. Run and see the logs

Anything else?

[storopoli]

Update, after fiddling around I've discovered that the cert directory in startos is:

pub const PACKAGE_CERT_PATH: &str = "/var/lib/embassy/ssl"; (https://github.com/Start9Labs/start-os/blob/fc8b1193de618efe3c9fe9f68ed9c7ce23cd562f/core/startos/src/net/mod.rs#L23).

Then:

cp /embassy-data/package-data/volumes/c-lightning/data/main/bitcoin/server-key.pem /var/lib/embassy/ssl/c-lightning/rest/rest.key.pem
cp /embassy-data/package-data/volumes/c-lightning/data/main/bitcoin/server.pem /var/lib/embassy/ssl/c-lightning/rest/rest.cert.pem

seems to fix.
Maybe the docker_entrypoint.sh should be made robust to that kind of scenario?

[chrisguida]

@storopoli I've stopped maintaining this package, but I'm sure @Dominion5254 would accept a PR to fix :)

[chrisguida]

FTR this is probably an edge case on your particular hardware, I haven't ever seen this on any of my installs. But yeah I'm sure my code is not watertight :)

[Dominion5254]

I don't believe I have seen this before. Is there anything unique about your setup that might give us some clues as to why /mnt/cert is empty? StartOS version? CLN version? Hardware running StartOS?

It is also worth mentioning that c-lightning-REST is slated to be deprecated in the near future in favor of clnrest, cln-grpc, commando, etc. So it might not make sense to add a find and patch a seemingly remote edge case for a soon to be deprecated connection interface.

[storopoli]

Start9 Server One 2023 with Celeron N4505
Version 0.3.5~1

This started happening after I've chrooted and created some systemd socat services following https://community.start9.com/t/core-lightning-with-tor-and-ipv4-clearnet/965

I've also added the following lines to config.main:

announce-addr=<CLEARNET_DNS_DOMAIN>:9735
announce-addr-dns=true

[chrisguida]

Ahh yeah that's definitely a hack. I can't immediately think of a reason that would break the container, but doing stuff like that definitely 'voids your warranty" :p

Have you tried just turning that off and seeing if it works? The container is not expecting the config.main file to be changed by the user.

[storopoli]

Yes tried restoring from a backup, wiping the /embassy-data/.../c-lightning folder. Tried from a brand new node.
Unfortunately I was forced to move to LND after 1 solid year in CLN.
I will probably wait for 0.3.6 that will arrive in $[14, \infty)$ days.

Feel free to close this if you cannot reproduce it.

[chrisguida]

It's doing the same thing from a fresh node with nothing in the datadir? That doesn't make any sense...

[storopoli]

Yes, because it does not have the /mnt/cert/ folder somehow

[Dominion5254]

The direct cause isn't immediately apparent to me either, but the clearnet hack almost certainly seems to be the culprit for the original issue. While it is great to see users doing cool DIY hacks, going under the hood to make changes such as this or resolving resulting issues is of course not something Start9 can officially support.

But I am confused by why this would occur with "a brand new node" - by this do you mean re-installing CLN on the same box that you had chrooted, or a completely fresh install of StartOS?

[chrisguida]

I mean this is an OS issue if the cert volume isn't getting mounted on a fresh new install of CLN on a brand new start-os server.

[Dominion5254]

Agreed, if that is the case it warrants opening an issue on the StartOS repo, but I want to make sure that is in fact what @storopoli meant by "a brand new node".