Inference Extension support

[nitishkumar71]

PR is still in development phase and purpose of this PR is just to show the approach we are taking in to support inference extension. Few things which are still need to be implented here are

• experimental API support for inference extension which is not working right now due to error Error: CRD '' has no versions in version v1.0.2
• Inference extension will published as separate create which is not done yet
• Tests

PR adds support of the inference extension as independent crate which will add CRDs support based on upstream version. It's possible there will be some duplicate code between gateway-api and inference extension, which is accepted approach for now. There is also some code duplication in test code, which should be handled when more extension support is being added.

Supporting issue #175

[dawid-nowak]

I am not sure what the overall decision about different versioning for Inference Extension and Gateway API, but it looks like this change will use same version for both packages?

[dawid-nowak]

It is more of a nit than anything else...
I would prefer to rename extension folder to extensions assuming that there will be more of them in the future.
Otherwise, I think it might be more clear to have gateway-api and inference-extension-api folders at the top level.

[nitishkumar71]

I am not sure what the overall decision about different versioning for Inference Extension and Gateway API, but it looks like this change will use same version for both packages?

Ohh, we meant to have different release version for rust API. Sorry, I did not get that. I was under impression for different version of upstream CRDs. Should we start a new versioning like v0.0.1 or follow upstream?

[nitishkumar71]

It is more of a nit than anything else... I would prefer to rename extension folder to extensions assuming that there will be more of them in the future. Otherwise, I think it might be more clear to have gateway-api and inference-extension-api folders at the top level.

Agree, i missed it.

[dawid-nowak]

I am not sure what the overall decision about different versioning for Inference Extension and Gateway API, but it looks like this change will use same version for both packages?

Ohh, we meant to have different release version for rust API. Sorry, I did not get that. I was under impression for different version of upstream CRDs. Should we start a new versioning like v0.0.1 or follow upstream?

I think we need @shaneutt to make a call here.
Personally, I would prefer if gateway-api and inference-extension-api were somehow tracking the releases of Kubernetes standards in their versions. At the moment, in crates.io we have release 0.19 but it seems that release 0.16 is covering Kubvernetes version 1.2.1. So from the developer perspective you need to check the readme to find what you are looking for.

I have seen that some projects would use custom versioning. In this case for example 0.19+1.2.1 for inference-extension-api and 0.19-1.4 for gateway-api, would have all the information necessary.

[shaneutt]

I'm open to the suggestion of doing a version tag that tracks the upstream version. We can always have a couple of tags if needed.

[shaneutt]

Thanks @nitishkumar71!

Took a quick high-level review pass, and added a couple of comments. Also needs some updates to make CI happy.

Did you run into much trouble while working on this, or did it seem pretty straightforward?

[nitishkumar71]

Took a quick high-level review pass, and added a couple of comments. Also needs some updates to make CI happy.

It took me sometime but delay was mostly due to some other personal things at my end.

[shaneutt]

There's definitely a few things we could continue to tweak here, but most of that is reasonable to tweak in upcoming iterations, so I think we're mostly ready to go.

There are a couple things in the new script for safety we need to do now, and can't wait for future iterations, which I've added comments for.

After those are resolved, I think we're good to go and we can continue moving this forward with some smaller incremental PRs from here.

[nitishkumar71]

There's definitely a few things we could continue to tweak here, but most of that is reasonable to tweak in upcoming iterations, so I think we're mostly ready to go.

There are a couple things in the new script for safety we need to do now, and can't wait for future iterations, which I've added comments for.

After those are resolved, I think we're good to go and we can continue moving this forward with some smaller incremental PRs from here.

Thanks for pointing them out, changes are done

[nitishkumar71]

There's definitely a few things we could continue to tweak here, but most of that is reasonable to tweak in upcoming iterations, so I think we're mostly ready to go.

There are a couple things in the new script for safety we need to do now, and can't wait for future iterations, which I've added comments for.

After those are resolved, I think we're good to go and we can continue moving this forward with some smaller incremental PRs from here.

@shaneutt what were the additional things you were planning to tweak here? I can pick them up