Split up and enhance retry mechanism

[ldang]

Currently, the HySDS retry mechanism allows us to
a) given a trigger rule, retry an job up to a maximum threshold (max retry count)
b) change the priority of the job

Given customer requests, it seems that it would be a better idea to redesign this and split them up into two actions.

1. Reprioritize and send job to a different queue

• there may be circumstances where we want a job to run with a higher priority or to run in a different queue to meet customer needs.

2. Retry, with incrementing priority, up to X number of retries
   • As part of our automated handling of anomalies, we would like to enable automatic retry of jobs up to X times, each time with increasing priority, to reduce the latency of when the retried job will run.

Will open up two issues in the appropriate repo to implement this