Too many runners registered

[kwngo]

We're running ARC version v0.22.3 and we're seeing an issue where after about 2000 runners are registered against Github the ListRunners endpoint fails and controller performance degredates. Our solution now is a cron job which runs a few times a day to delete all "offline" runners registered to Github. This appears to be an expected side-effect of ephemeral runners, or perhaps our runners aren't de-registering themselves correctly?

Has anyone else faced similar issues, and if so how are you tackling this issue?

[toast-gear]

what do your runner manifests look like? Your runners are likely using the --once flag which doesn't include logic to unregister them, If you are on GHES => 3.3 you can upgrade to latest ARC without any problems which will automatically set the new --ephemeral flag. Note we are removing support fo rthe --once flag entirely very very very soon #1196 so you will need to use the new flag soon to not have problems

[toast-gear]

so depending on what foundations-dev.artifactory.aws.venmo.biz/actions-runner-dind:82a20a4 is built from you need to different env vars to set the --ephemeral flag, do you know what runner container this is based on? If you build from latest as of today you should be getting the --ephemeral flag set as you now have to opt in to --once flag https://github.com/actions-runner-controller/actions-runner-controller#runner-entrypoint-features. Can you confirm what configuration is being set for one of your runners? (the runner logs will show this as part of the entrypoint)

[toast-gear]

--once was never fully implemented or supported by GitHub, you should upgrade to GHES => 3.3 as soon as possible so you can use the fully supported and fully implemented --ephemeral flag of which deregistering the runner is part fo the logic.

[kwngo]

Very interesting, good to know! That image Is built from runner version 2.285.1, is there a specific runner version we should be upgrading to?

[toast-gear]

I believe there isn't really a reason not to use latest, I'm not sure how GitHub manage support for runner versions across all the different GHES versions (pretty much most Action features require logic in the actions/runner software in addition to the server-side of things), you'd need to check their docs for what is supported. But basically, always run the latest runner version possible for your GHES environment, there is never really a reason to stick on older versions beyond waiting to see if the new release is bug free.

If you are on GHES => 3.3 then you just need to rebuild your container (please do check with GitHub how suppot works first) against latest and you should get the supported --ephemeral flag without having to set anything.

[toast-gear]

another reason for staying current with the runner version is we've had multiple reports, and apparently GitHub have confirmed, is that the self-update process can cause massive delays in jobs being allocated to runners https://github.com/actions-runner-controller/actions-runner-controller/blob/master/TROUBLESHOOTING.md#delay-in-jobs-being-allocated-to-runners

[Moser-ss]

Hey I have a similar issue, the number of offline runners is growing daily, in my case I am using the default images
https://gist.github.com/Moser-ss/c2de6b32859759023176f33418b71132

[kwngo]

Which version of the controller and runner are you using?

[Moser-ss]

I am using version 0.23. I am seeing a steady increase in the number of the offline runners

Does make sense to have a cron job in the controller to remove offline runners?

[kwngo]

This seems to have been the root cause. Thank you @toast-gear! I'll add a note to the FAQs/help docs :)

[toast-gear]

we've actually just removed support for the --once flag entirely so you can't even configure the --once flag anymore (#1196), it's been in the making for a long time. This problem will naturally disappear as people are now required to use GHES => 3.3 to use our runners at all. People running < 3.3 will start to disappear as their ARC system will stop being functional once they upgrade their base image unless they stop using ours as their base which seems like a bad idea and more effort than just upgrading their GHES instance (especially given GitHub will at some point remove support for the flag too)

[Moser-ss]

Good news about this topic.
GitHub change the behavior of removing offline runners
An ephemeral self-hosted Actions runner is automatically removed from GitHub if we have not seen it connect to GitHub for more than one day.

[samus-aran]

We're locking this discussion because it has not had recent activity and/or other members have asked for more information to assist you but received no response. Thank you for helping us maintain a productive and tidy community for all our members.