Update ingressClass deployment to newer spec

[gcarcassi]

As stated here:
https://kubernetes.io/docs/concepts/services-networking/ingress/#deprecated-annotation
the annotation "ingressclass.kubernetes.io/is-default-class" has been deprecated in favor of the field "spec/ingressClassName" field.

Ingress classes should be declared as ingressClass objects, as shown in:
https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-class

Our current deployment does not have a "slate" ingressClass, but it has a "nginx" ingress class, which does work. If one runs the command:
kubectl --kubeconfig uutah-prod.conf get ingressClass nginx --output=yaml
one can see the details of the object - there is no special configuration

We should create a "slate" ingressClass similar to the above. In principle, we could make it the default ingressClass as well:
https://kubernetes.io/docs/concepts/services-networking/ingress/#default-ingress-class
It may be a problem on other clusters since there can only be one default ingress per cluster, so it may clash with other settings.

[gcarcassi]

Created a slate ingressClass on umich using:

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: slate
spec:
controller: k8s.io/ingress-nginx

I was able to create an ingress that uses this ingressClass.

[gcarcassi]

The ingressClass name is defined in:
https://github.com/slateci/slate-client-server/blob/e7ed316d222d0a562586980c2881c6accaf4c2ba/resources/nginx-ingress.yaml#LL657C14-L657C14
and

slate-client-server/src/client/cluster_components/IngressController.cpp

Line 674 in e7ed316

name: nginx

[gcarcassi]

Example of an ingress that uses the old method

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-tls-test
  annotations:
    acme.cert-manager.io/http01-edit-in-place: "true"
    cert-manager.io/cluster-issuer: letsencrypt-prod
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: slate
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "Content-Security-Policy: frame-ancestors 'self' *.slateci.io *.slateci.net" "X-Frame-Options: allow-from *.slateci.io";
spec:
#  ingressClassName: slate
  rules:
  - host: ingress-tls-test.umich-prod.slateci.net
    http:
      paths:
      - backend:
          service:
            name: nginx-ingress-test
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - ingress-tls-test.umich-prod.slateci.net
    secretName: ingress-tls-test