Juju/ops only reacts to public Pebble custom notices, breaking custom notices in rocks that run workloads as `_daemon_`

[skourta]

Juju (through ops) only emits pebble-custom-notice events for public Pebble notices — i.e. notices recorded by the root user. In rocks, the recommended practice is to run workloads as the unprivileged _daemon_ user. Custom notices emitted by a non-root user are scoped to that user's UID and are not public, so Juju never emits an event for them and the charm never wakes up.

Current behavior

• A workload running as _daemon_ (non-root) records a custom notice via pebble notify.
• Because the notice is emitted by a non-root UID, Pebble binds the notice to that user and it is not public.
• Juju/ops only surfaces public notices, so no pebble-custom-notice event is emitted.
• The charm never observes the notice and does not react.

Expected behavior

Charms should be able to react to custom notices emitted by workloads regardless of whether the workload runs as root or as the unprivileged _daemon_ user — the recommended way to run workloads in rocks.

Impact

This makes custom notices effectively unusable as an event mechanism for any rock that follows the best practice of running its workload as _daemon_. Charm authors are forced to either:

• run the workload as root (against rock best practices), or
• abandon custom notices as a signaling mechanism.

Environment

• Component: Juju / ops (Pebble custom notice handling)
• Workload user: _daemon_ (non-root)
• Notice visibility: user-scoped (not public)

[skourta]

I am unsure if this is the best place to create the issue. Please advise if otherwise.
Thank you

[tonyandrewmeyer]

I feel like Juju should probably be passing NoticesUsersAll to WaitNotices, because the charm should get notified about custom notices from all users. @hpidcock or @benhoyt would you agree?

[benhoyt]

Interesting gotcha. That makes sense to me (equivalent to pebble notices --users=all). I'm guessing Juju connects to the Pebble API as root/admin and hence would have access to all users' notices? @hpidcock What do you think?

[hpidcock]

I don't think we can introduce this without ops changes, since AFAIK the notice key when triggering the charm hook is not disambiguated by the user ID of the non-public notice. There is a security/DoS risk associated with running hooks from non-admin notices.

I'll need to consider this in the full context, so I don't think it is as straight forward as adding the functionality to the Juju uniter's pebble notice watching. My initial thoughts are that we might need to introduce a new charm hook type, then add support to Ops to handle this.

[benhoyt]

That makes sense, thanks. @skourta, we can see this is a real issue with the design of non-root users (or more specifically, the interaction between Pebble notices and non-root workloads). We will put this on our list, but I want to set expectations in terms of timeline: this will require more design and a Juju change, so don't hold your breath -- I'd recommend using a workaround in the meantime, which probably means root-workload charms for now.