feat: improve retrying of the dynamic labels calls

[FUSAKLA]

Hi, just started to use promxy and have setup with up to hundreds of server groups and would like to use dynamic labels for filtering.

Unfortunately, if many of the server groups goes down, the amount of retries is huge, so I tried adding exponential back off

Also, initial fetch of the dynamic labels is now not necessary so if one of the server groups is not up, it won't stop promxy from starting up.

[jacksontj]

just started to use promxy

Welcome! Always glad to see more users :)

hundreds of server groups and would like to use dynamic labels for filtering.

I would definitely give some caution here; this requires loading all those filters into memory. So the cardinality of those can be a problem if they get high. For most setups this isn't necessary; but sometimes its just a lot easier for some subset of labels (e.g. region).

if many of the server groups goes down, the amount of retries is huge,

Just to clarify here; from the code it seems that in the case of discovery sync failure we never retry and in the case of the DynamicLabels we simply try again in the next interval? I wanted to confirm you weren't seeing some runaway retry (as that would DEFINITELY not be expected :) ).

initial fetch of the dynamic labels is now not necessary so if one of the server groups is not up, it won't stop promxy from starting up.

I think the necessity of the startup check here depends on your usecase. A common use-case I've seen here is when using a paid SAAS endpoint-- and in that situation you don't want to send queries there unless necessary (as they might be quite expensive). So if your use-case really wants to work without that initial sync I'm open to adding a config option to disable the initial sync -- but the change as is changes default behavior which is not great.