Can't locate files in backup snapshot directory

[bshor]

I'm using httm 0.43.2 on a Debian 12 system, with a BTRFS file system and btrbk for creating snapshots. It works perfectly for showing the older versions living in my snapshots directory, which is located at @snapshots and mounted on /snapshots. My data is located at @data and mounted on /data. On a separate drive, I have @btrbk mounted on /btrbk. This is where I send "backup" snapshots.

For my local drive, these are both top-level subvolumes:

ID 3617 gen 147254 top level 5 path @data
ID 3619 gen 147245 top level 5 path @snapshots

For my backup drive it is as well:

ID 257 gen 16551 top level 5 path @btrbk

Here's an example of it working as expected using snapshots in /snapshots.

sudo httm /data/btrfs_free.sh
NOTICE: Falling back to detection of btrfs snapshot mounts perhaps defined by Snapper re: mount: "/extra"
────────────────────────────────────────────────────────────────────────────────
Sun Jun 09 02:54:59 2024  1.3 KiB  "/snapshots/data.20240804T0000/btrfs_free.sh"
Sat Aug 24 01:27:10 2024  1.3 KiB  "/snapshots/data.20241002T2351/btrfs_free.sh"
────────────────────────────────────────────────────────────────────────────────
Sat Aug 24 01:27:10 2024  1.3 KiB  "/data/btrfs_free.sh"
────────────────────────────────────────────────────────────────────────────────

But I can't seem to get httm to look at /btrbk on the other drive.

I tried map-aliases but no luck:

 sudo httm --map-aliases /data:/btrbk /data/btrfs_free.sh
NOTICE: Falling back to detection of btrfs snapshot mounts perhaps defined by Snapper re: mount: "/extra"
────────────────────────────────────────────────────────────────────────────────
Sun Jun 09 02:54:59 2024  1.3 KiB  "/snapshots/data.20240804T0000/btrfs_free.sh"
Sat Aug 24 01:27:10 2024  1.3 KiB  "/snapshots/data.20241002T2351/btrfs_free.sh"
────────────────────────────────────────────────────────────────────────────────
Sat Aug 24 01:27:10 2024  1.3 KiB  "/data/btrfs_free.sh"
────────────────────────────────────────────────────────────────────────────────

/snapshots and /btrbk look the same... a long list of snapshots.

I'm guessing that httm is trying to look in obvious places for snapshots, and doesn't think to look in some other place for them. Which is fine -- but is there a way to tell it to look there?

Here's the debug output. Interesting that it appears to believe /btrbk is a live subvolume that might have its own snapshots.

NOTICE: Falling back to detection of btrfs snapshot mounts perhaps defined by Snapper re: mount: "/more"
DEBUG: Base mount: "/data", Base subvol: "/@data", Snap Relative Path: "@snapshots/@data.20240307T0907"
DEBUG: Joined path: "/snapshots/@data.20240307T0907"
DEBUG: Base mount: "/data", Base subvol: "/@data", Snap Relative Path: "@snapshots/data.20241005T1315"
DEBUG: Joined path: "/snapshots/data.20241005T1315"
DEBUG: Base mount: "/data", Base subvol: "/@data", Snap Relative Path: "@snapshots/data.20241003T2215"
DEBUG: Joined path: "/snapshots/data.20241003T2215"
DEBUG: Base mount: "/data", Base subvol: "/@data", Snap Relative Path: "@snapshots/data.20241004T0015"
DEBUG: Joined path: "/snapshots/data.20241004T0015"
DEBUG: Base mount: "/data", Base subvol: "/@data", Snap Relative Path: "@snapshots/data.20241005T1245"
DEBUG: Joined path: "/snapshots/data.20241005T1245"
DEBUG: Base mount: "/data", Base subvol: "/@data", Snap Relative Path: "@snapshots/data.20241002T2351"
DEBUG: Joined path: "/snapshots/data.20241002T2351"
DEBUG: Base mount: "/data", Base subvol: "/@data", Snap Relative Path: "@snapshots/data.20241005T1215"
DEBUG: Joined path: "/snapshots/data.20241005T1215"
DEBUG: Base mount: "/data", Base subvol: "/@data", Snap Relative Path: "@snapshots/data.20241005T1345"
DEBUG: Joined path: "/snapshots/data.20241005T1345"
DEBUG: Base mount: "/data", Base subvol: "/@data", Snap Relative Path: "@snapshots/data.20241005T0015"
DEBUG: Joined path: "/snapshots/data.20241005T0015"
DEBUG: Base mount: "/data", Base subvol: "/@data", Snap Relative Path: "@snapshots/data.20240804T0000"
DEBUG: Joined path: "/snapshots/data.20240804T0000"
WARN: httm relies on the user (and/or the filesystem's auto-mounter) to mount snapshots.  Make certain any snapshots the user may want to view are mounted, or are able to be mounted, and/or the user has the correct permissions to view.
WARN: Mount "/backup" appears to have no snapshots available.
WARN: Mount "/btrbk" appears to have no snapshots available.
WARN: Mount "/extra" appears to have no snapshots available.
WARN: Mount "/more" appears to have no snapshots available.
WARN: Mount "/snapshots" appears to have no snapshots available.
Config {
    paths: [
        PathData {
            path_buf: "/data/btrfs_free.sh",
            metadata: Some(
                PathMetadata {
                    size: 1304,
                    modify_time: SystemTime {
                        tv_sec: 1724480830,
                        tv_nsec: 733481865,
                    },
                },
            ),
        },
    ],
    opt_recursive: false,
    opt_exact: false,
    opt_no_filter: false,
    opt_debug: true,
    opt_no_traverse: false,
    opt_omit_ditto: false,
    opt_no_hidden: false,
    opt_json: false,
    opt_one_filesystem: false,
    opt_no_clones: false,
    dedup_by: Metadata,
    opt_bulk_exclusion: None,
    opt_last_snap: None,
    opt_preview: None,
    opt_deleted_mode: None,
    opt_requested_dir: None,
    requested_utc_offset: -05:00:00,
    exec_mode: BasicDisplay,
    print_mode: Formatted(
        Default,
    ),
    dataset_collection: FilesystemInfo {
        map_of_datasets: MapOfDatasets {
            inner: {
                "/backup": DatasetMetadata {
                    source: "/dev/sdc1",
                    fs_type: Btrfs(
                        Some(
                            BtrfsAdditionalData {
                                base_subvol: "<FS_TREE>",
                                snap_names: OnceLock(
                                    <uninit>,
                                ),
                            },
                        ),
                    ),
                    link_type: Local,
                },
                "/btrbk": DatasetMetadata {
                    source: "/dev/sdc1",
                    fs_type: Btrfs(
                        Some(
                            BtrfsAdditionalData {
                                base_subvol: "/@btrbk",
                                snap_names: OnceLock(
                                    <uninit>,
                                ),
                            },
                        ),
                    ),
                    link_type: Local,
                },
                "/data": DatasetMetadata {
                    source: "/dev/sdb",
                    fs_type: Btrfs(
                        Some(
                            BtrfsAdditionalData {
                                base_subvol: "/@data",
                                snap_names: OnceLock(
                                    {
                                        "/snapshots/@data.20240307T0907": "@snapshots/@data.20240307T0907",
                                        "/snapshots/data.20240804T0000": "@snapshots/data.20240804T0000",
                                        "/snapshots/data.20241002T2351": "@snapshots/data.20241002T2351",
                                        "/snapshots/data.20241003T2215": "@snapshots/data.20241003T2215",
                                        "/snapshots/data.20241004T0015": "@snapshots/data.20241004T0015",
                                        "/snapshots/data.20241005T0015": "@snapshots/data.20241005T0015",
                                        "/snapshots/data.20241005T1215": "@snapshots/data.20241005T1215",
                                        "/snapshots/data.20241005T1245": "@snapshots/data.20241005T1245",
                                        "/snapshots/data.20241005T1315": "@snapshots/data.20241005T1315",
                                        "/snapshots/data.20241005T1345": "@snapshots/data.20241005T1345",
                                    },
                                ),
                            },
                        ),
                    ),
                    link_type: Local,
                },
                "/extra": DatasetMetadata {
                    source: "/dev/sdc3",
                    fs_type: Btrfs(
                        Some(
                            BtrfsAdditionalData {
                                base_subvol: "<FS_TREE>",
                                snap_names: OnceLock(
                                    <uninit>,
                                ),
                            },
                        ),
                    ),
                    link_type: Local,
                },
                "/more": DatasetMetadata {
                    source: "/dev/sde2",
                    fs_type: Btrfs(
                        Some(
                            BtrfsAdditionalData {
                                base_subvol: "/@more",
                                snap_names: OnceLock(
                                    <uninit>,
                                ),
                            },
                        ),
                    ),
                    link_type: Local,
                },
                "/snapshots": DatasetMetadata {
                    source: "/dev/sdb",
                    fs_type: Btrfs(
                        Some(
                            BtrfsAdditionalData {
                                base_subvol: "/@snapshots",
                                snap_names: OnceLock(
                                    <uninit>,
                                ),
                            },
                        ),
                    ),
                    link_type: Local,
                },
            },
        },
        map_of_snaps: MapOfSnaps {
            inner: {
                "/backup": [],
                "/btrbk": [],
                "/data": [
                    "/snapshots/@data.20240307T0907",
                    "/snapshots/data.20240804T0000",
                    "/snapshots/data.20241002T2351",
                    "/snapshots/data.20241003T2215",
                    "/snapshots/data.20241004T0015",
                    "/snapshots/data.20241005T0015",
                    "/snapshots/data.20241005T1215",
                    "/snapshots/data.20241005T1245",
                    "/snapshots/data.20241005T1315",
                    "/snapshots/data.20241005T1345",
                ],
                "/extra": [],
                "/more": [],
                "/snapshots": [],
            },
        },
        filter_dirs: FilterDirs {
            inner: {
                "/",
                "/dev",
                "/dev/.lxc/proc",
                "/dev/.lxc/sys",
                "/dev/console",
                "/dev/hugepages",
                "/dev/mqueue",
                "/dev/net/tun",
                "/dev/ptmx",
                "/dev/pts",
                "/dev/shm",
                "/dev/tty1",
                "/dev/tty2",
                "/proc",
                "/proc/cpuinfo",
                "/proc/diskstats",
                "/proc/fs/nfsd",
                "/proc/loadavg",
                "/proc/meminfo",
                "/proc/slabinfo",
                "/proc/stat",
                "/proc/swaps",
                "/proc/sys",
                "/proc/sys/kernel/random/boot_id",
                "/proc/sys/net",
                "/proc/sysrq-trigger",
                "/proc/uptime",
                "/run",
                "/run/credentials/systemd-sysctl.service",
                "/run/credentials/systemd-sysusers.service",
                "/run/credentials/systemd-tmpfiles-setup-dev.service",
                "/run/credentials/systemd-tmpfiles-setup.service",
                "/run/lock",
                "/run/rpc_pipefs",
                "/run/user/1000",
                "/sys",
                "/sys/devices/system/cpu",
                "/sys/devices/virtual/net",
                "/sys/fs/cgroup",
                "/sys/fs/fuse/connections",
            },
        },
        opt_map_of_alts: None,
        opt_map_of_aliases: None,
        opt_common_snap_dir: Some(
            "/snapshots",
        ),
        opt_alt_store: None,
    },
    pwd: "/home/xxxx",
}

[kimono-koans]

But I can't seem to get httm to look at /btrbk on the other drive.
/snapshots and /btrbk look the same... a long list of snapshots.

In general, re: ZFS, we have tried to do this via the -a, --alt-replicated flag, which will auto-discover snapshots of the type rpool/ROOT/ubuntu_w0ha6q when "backup-ed" to a local data pool, like so data/rpool/ROOT/ubuntu_w0ha6q or data/something/anotherthing/rpool/ROOT/ubuntu_w0ha6q.

Re: btrfs, auto-discovery of such pools may not be possible because of how you sent (or copied, which, if you see my next comment, is a no-no!) the datasets. If it were me, I'd first try to be as close as possible to the ZFS topology and see if you can't use -a, because it won't require any extra legwork on your part like --map-aliases.

WARN: Mount "/btrbk" appears to have no snapshots available.

Here, it seems httm is not detecting any snapshot subvolumes at /btrbk which means -- perhaps you're not doing a btrfs send to your data pool, but instead just copying new datasets? Note, httm will not discover snapshots which aren't snapshot subvolumes. My guess is that this is your problem.

See below, httm discovers the /btrbk dataset but does not see any snapshots because snapshots are just raw directories, not btrfs subvolumes (which you can see via btrfs subvolume list -s /path/to/btrfs/volume):

map_of_snaps: MapOfSnaps {
            inner: {
                "/backup": [],
                "/btrbk": [],
                "/data": [
                    "/snapshots/@data.20240307T0907",
                    "/snapshots/data.20240804T0000",
                    "/snapshots/data.20241002T2351",
                    "/snapshots/data.20241003T2215",
                    "/snapshots/data.20241004T0015",
                    "/snapshots/data.20241005T0015",
                    "/snapshots/data.20241005T1215",
                    "/snapshots/data.20241005T1245",
                    "/snapshots/data.20241005T1315",
                    "/snapshots/data.20241005T1345",
                ],
                "/extra": [],
                "/more": [],
                "/snapshots": [],
            },
        },

[kimono-koans]

@bshor very much appreciate you filing a bug report. Especially the included detail.

However, if I am to keep this issue open, I'm going to need you to confirm whether my supposition is correct, that is -- you copied rather than sent the sub-volumes to a new location, or for you to tell me whether perhaps something else is going on.

Again -- I'm very pleased to discuss further, but will need a response from you.

Thanks!

[bshor]

I apologize for the late response, work has been hectic. I will respond in detail in the next 12 hours (replacing this response).

[bshor]

Ok, back on! I'll be much quicker to respond now. I appreciate the very close attention you are giving me!

But I can't seem to get httm to look at /btrbk on the other drive.
/snapshots and /btrbk look the same... a long list of snapshots.

In general, re: ZFS, we have tried to do this via the -a, --alt-replicated flag, which will auto-discover snapshots of the type rpool/ROOT/ubuntu_w0ha6q when "backup-ed" to a local data pool, like so data/rpool/ROOT/ubuntu_w0ha6q or data/something/anotherthing/rpool/ROOT/ubuntu_w0ha6q.

Unfortunately the -a flag doesn't seem to work.

sudo httm -a /data/btrfs_free.sh
NOTICE: Falling back to detection of btrfs snapshot mounts perhaps defined by Snapper re: mount: "/extra"
────────────────────────────────────────────────────────────────────────────────
Sun Jun 09 02:54:59 2024  1.3 KiB  "/snapshots/data.20240804T0000/btrfs_free.sh"
Sat Aug 24 01:27:10 2024  1.3 KiB  "/snapshots/data.20241002T2351/btrfs_free.sh"
────────────────────────────────────────────────────────────────────────────────
Sat Aug 24 01:27:10 2024  1.3 KiB  "/data/btrfs_free.sh"
────────────────────────────────────────────────────────────────────────────────

Re: btrfs, auto-discovery of such pools may not be possible because of how you sent (or copied, which, if you see my next comment, is a no-no!) the datasets. If it were me, I'd first try to be as close as possible to the ZFS topology and see if you can't use -a, because it won't require any extra legwork on your part like --map-aliases.

WARN: Mount "/btrbk" appears to have no snapshots available.

Here, it seems httm is not detecting any snapshot subvolumes at /btrbk which means -- perhaps you're not doing a btrfs send to your data pool, but instead just copying new datasets? Note, httm will not discover snapshots which aren't snapshot subvolumes. My guess is that this is your problem.

Ok, so /btrbk is definitely the TARGET of send operations which is executed by btrbk. The whole point of btrbk is to unify taking snapshots and to send them to a backup location (here, /btrbk). Here's my btrbk configuration (/etc/btrbk/btrbk.conf):

timestamp_format        long
transaction_log         /var/log/btrbk.log
lockfile                /var/lock/btrbk.lock
send_compressed_data    yes

snapshot_preserve_min   2h
snapshot_preserve       3d 2w 3m

target_preserve_min     4h
target_preserve         7d 6w 9m

volume /data
    snapshot_dir /snapshots
    subvolume    /data
    target /btrbk

See below, httm discovers the /btrbk dataset but does not see any snapshots because snapshots are just raw directories, not btrfs subvolumes (which you can see via btrfs subvolume list -s /path/to/btrfs/volume):

map_of_snaps: MapOfSnaps {
            inner: {
                "/backup": [],
                "/btrbk": [],
                "/data": [
                    "/snapshots/@data.20240307T0907",
                    "/snapshots/data.20240804T0000",
                    "/snapshots/data.20241002T2351",
                    "/snapshots/data.20241003T2215",
                    "/snapshots/data.20241004T0015",
                    "/snapshots/data.20241005T0015",
                    "/snapshots/data.20241005T1215",
                    "/snapshots/data.20241005T1245",
                    "/snapshots/data.20241005T1315",
                    "/snapshots/data.20241005T1345",
                ],
                "/extra": [],
                "/more": [],
                "/snapshots": [],
            },
        },

Hmmm. While I don't fully understand this debug output, I'm trying to express that /data is my live data, and that /snapshots is where snapshots of /data go, while /btrbk is where backup snapshots are sent to. Those backup snapshots are very useful because they live a lot longer than in /snapshots.

I guess what I'm confused by is why don't we see entries that look like /btrbk/data.20240804T0000 underneath /data up above. Seems like httm is only detecting the snapshots in /snapshots.

Just to show you, here's the contents of /btrbk:

And the contents of /snapshots:

Thank you again for your attention and patience!

[kimono-koans]

In general, re: ZFS, we have tried to do this via the -a, --alt-replicated flag, which will auto-discover snapshots of the type rpool/ROOT/ubuntu_w0ha6q when "backup-ed" to a local data pool, like so data/rpool/ROOT/ubuntu_w0ha6q or data/something/anotherthing/rpool/ROOT/ubuntu_w0ha6q.

Unfortunately the -a flag doesn't seem to work.

Yes, as noted, you would first need to rearrange your snapshots to be in the form of a ZFS snapshot topology.

For now, I am supposing this is not possible for you (you're using btrbk, and this is how btrbk does things), and we will move on. You will need to, in this case, to use --map-aliases to view snapshots on a different dataset.

Ok, so /btrbk is definitely the TARGET of send operations which is executed by btrbk. The whole point of btrbk is to unify taking snapshots and to send them to a backup location (here, /btrbk). Here's my btrbk configuration (/etc/btrbk/btrbk.conf):

timestamp_format        long
transaction_log         /var/log/btrbk.log
lockfile                /var/lock/btrbk.lock
send_compressed_data    yes

snapshot_preserve_min   2h
snapshot_preserve       3d 2w 3m

target_preserve_min     4h
target_preserve         7d 6w 9m

volume /data
    snapshot_dir /snapshots
    subvolume    /data
    target /btrbk

Hmmm. While I don't fully understand this debug output, I'm trying to express that /data is my live data, and that /snapshots is where snapshots of /data go, while /btrbk is where backup snapshots are sent to. Those backup snapshots are very useful because they live a lot longer than in /snapshots.

I am no expert in btrbk but, from what I understand of your config file, it should be sending instead of copying your snapshots to the /btrbk dataset. So, it would seem that my first supposition/explanation was incorrect.

The question is then: Why doesn't httm see those snapshots? httm only includes sub-volumes known to btrfs. It does not concern itself with your btrbk configuration because it doesn't know about your btrbk configuration. Therefore, it can't know the directories in /btrbk are sub-volumes unless the btrfs command indicates as much (/brbk might contain other files and directories which aren't sub-volumes, etc.).

So -- I would check sudo btrfs subvolume list -s /btrbk and sudo btrfs subvolume show /btrbk to see what each says.

It is possible that even after btrfs sends a backup it still relates that backup with the same sub-volume, and never re-associates it with a new dataset, like ZFS. To the extent btrfs is just weird, and the behavior is not well-defined, it may be hard to help you, but maybe this is logical in btrfs world where a snapshots is sent to a new dataset on the same machine?

It's also possible your "backups" are privileged, whereas your "snapshots" are not, and you simply need to use sudo in conjunction with --map-aliases to view them, see: https://github.com/kimono-koans/httm?tab=readme-ov-file#example-usage

Note: Users may need to use sudo (or equivalent) to view versions on BTRFS or NILFS2 datasets, or Restic repositories, as BTRFS or NILFS2 snapshots or Restic repositories may require root permissions in order to be visible. Restic and Time Machine backups also require an additional flag, see further discussion of Restic --alt-store in the below.

[bshor]

Thank you again for writing. I feel bad for taking so much of your time for something that appears to be a BTRFS idiosyncracy -- but since you want your tool to work in BTRFS and btrbk is VERY common for BTRFS file system enthusiasts, maybe it's a useful exercise.

I'd love to try --map-aliases but as you can see above I tried my best and couldn't figure it out.

 sudo httm --map-aliases /data:/btrbk /data/btrfs_free.sh
NOTICE: Falling back to detection of btrfs snapshot mounts perhaps defined by Snapper re: mount: "/extra"
────────────────────────────────────────────────────────────────────────────────
Sun Jun 09 02:54:59 2024  1.3 KiB  "/snapshots/data.20240804T0000/btrfs_free.sh"
Sat Aug 24 01:27:10 2024  1.3 KiB  "/snapshots/data.20241002T2351/btrfs_free.sh"
────────────────────────────────────────────────────────────────────────────────
Sat Aug 24 01:27:10 2024  1.3 KiB  "/data/btrfs_free.sh"
────────────────────────────────────────────────────────────────────────────────

subvolume list -s /btrbk shows (clipping a bunch for being concise:

ID 988 gen 4928 cgen 4924 top level 257 otime 2024-04-07 00:00:03 path data.20240407T0000
ID 1610 gen 8634 cgen 8631 top level 257 otime 2024-05-09 16:00:09 path data.20240505T0001
ID 1940 gen 17038 cgen 17035 top level 257 otime 2024-10-11 09:20:03 path data.20241011T0920
ID 1941 gen 17041 cgen 17038 top level 257 otime 2024-10-11 09:40:02 path data.20241011T0940
ID 1942 gen 17045 cgen 17041 top level 257 otime 2024-10-11 10:00:04 path data.20241011T1000
ID 1943 gen 17045 cgen 17045 top level 257 otime 2024-10-11 10:20:04 path data.20241011T1020

sudo btrfs subvolume show /btrbk shows:

	Name: 			@btrbk
	UUID: 			e758aa42-c218-cd4d-8e9b-ddead514b91e
	Parent UUID: 		-
	Received UUID: 		-
	Creation time: 		2024-03-01 23:53:39 -0600
	Subvolume ID: 		257
	Generation: 		17045
	Gen at creation: 	8
	Parent ID: 		5
	Top level ID: 		5
	Flags: 			-
	Send transid: 		0
	Send time: 		2024-03-01 23:53:39 -0600
	Receive transid: 	0
	Receive time: 		-
	Snapshot(s):

Misc points:

• I always use sudo when I issue any btrbk, btrfs, or httm commands.
• For sure btrfs is ignorant of btrbk, but the author of the latter is using all native functionality (like snapshots and sending), so I gave you the config file just to underline that I'm not doing anything odd
• Given my technical expertise (or lack thereof) I'm not sure I fully understand your point about BTRFS idiosyncracy. Perhaps it might be useful to post a question on the subreddit, etc.

[kimono-koans]

subvolume list -s /btrbk shows (clipping a bunch for being concise:

This means it is possible to at least list those volumes! I'll start thinking about how best to allow with httm.

Given my technical expertise (or lack thereof) I'm not sure I fully understand your point about BTRFS idiosyncracy. Perhaps it might be useful to post a question on the subreddit, etc.

This is some btrfs weirdness where the snapshots are transferred but then not re-associated with any mount. It may just be that I'm more comfortable with ZFS (which is true!). But I also think the way btrfs handles snapshots is leaky implementation detail, where virtually all the functionality one could implement is permissible in btrfs. This is fine for users who want configurability, but it makes certain things impossible, like: Writing a snapshot tool, that works with all other snapshot tools, and which can write to custom btrfs snapshot locations safely.

This is just another layer of the btrfs onion and I will try to patch it up as best I know how, but it's simply btrfs wildness that makes it harder for httm to do the things it does. At some point I usually say: Eff this noise, I'm not screwing around with btrfs anymore, because of how impossible it makes everything.

[bshor]

Thank you for your amazing brainchild! I really, really hope you don't abandon BTRFS functionality -- and I'm unhappy I may have inadvertantly pushed you in that direction. httm is AMAZING! It's ridiculous to have to manually much about directory trees to restore an older version of a file which is what I did before httm. Now I can just quickly say which file I want to lookup and restore the desired snapshotted version in seconds. So cool.

This is merely a wished-for feature ... httm works really well AS IS, and if there were no easy support for backup snapshot restoring, I would 100% fine with what we have.

Really, really, please stay!

ps When I said the subreddit, I mean I'd try to do the legwork, but I think since I see you there sometimes, I'm afraid of creating more work for you, so I'm ok with status quo.