Replace MinIO with Garage as S3-compatible storage backend by RitikaSen27 · Pull Request #25 · 52North/pygeoapi_k8s-manager

RitikaSen27 · 2026-03-29T10:27:07Z

Summary

Replaced MinIO with Garage as the S3-compatible object storage backend due to MinIO’s shift towards paid AIStor offerings.

Changes

Removed MinIO Kubernetes configuration (minio.yaml)
Added Garage deployment configuration (garage.yaml)
Updated storage endpoint from MinIO (localhost:30090) to Garage (localhost:30091)
Ensured compatibility with existing S3-based operations (log uploads via boto3)

Details

Garage is used as a self-hosted, open-source, S3-compatible alternative
No changes were required in application logic since existing code uses standard S3 APIs
Environment variables were updated to point to the new Garage endpoint

Notes

Current Garage setup uses emptyDir (non-persistent storage), similar to previous MinIO setup
Bucket initialization is not yet automated and can be added in future improvements

Future Improvements

Add a Kubernetes job for automated bucket creation (similar to previous MinIO setup)
Introduce persistent storage instead of emptyDir
Add health checks and readiness probes for Garage deployment

EHJ-52n · 2026-03-30T10:41:34Z

Provide missing bucket initialization and health checks, or this PR will be deleted.

RitikaSen27 · 2026-03-30T15:33:07Z

Thanks for the feedback! I’ll add bucket initialization and health checks to the Garage setup and update the PR shortly.

RitikaSen27 · 2026-03-30T15:48:31Z

Thanks for the feedback! I’ve added bucket initialization and health checks to the Garage setup and updated the PR accordingly. Please let me know if anything else needs improvement.

EHJ-52n · 2026-04-09T05:28:13Z

Could you please ensure, that the functionality of the garage set-up is matching the one of minio, thank you.

Use the same ports on node level for garage, so for the user and documentation, nothing needs to be changed.

RitikaSen27 · 2026-04-12T10:55:12Z

Thanks for your suggestion regarding matching the MinIO ports.

I’ve addressed this in a separate PR to keep the changes focused and clean:
#30

This update aligns Garage with MinIO’s external ports (9000/9001 and corresponding nodePorts), so no changes are required for users or documentation.

Please let me know if any further adjustments are needed.

EHJ-52n · 2026-04-13T06:28:07Z

Please do not create a new PR requested updates. Just update this branch and the PR will be updated automatically.

RitikaSen27 · 2026-04-13T15:35:45Z

Thank you for the feedback!
I have updated this PR as requested .Please let me know if i should close the new PR that i created for this.

EHJ-52n · 2026-04-14T06:16:35Z

Could you please ensure, that the functionality of the garage set-up is matching the one of minio, thank you.

Use the same ports on node level for garage, so for the user and documentation, nothing needs to be changed.

I am missing the fulfilling of these two.

So, please

add code to initialize the required bucket.
ensure some T/WUI is available to check/see the content in the bucket.

RitikaSen27 · 2026-04-14T09:55:00Z

I will surely work on this but could you please explain a bit what is missing in the previous PR and what should i add there ?
I am not being able to understand it clearly as previously i did change the ports so now do i need to make garage behave exactly like MinIO?

RitikaSen27 · 2026-04-14T10:17:18Z

I’ve updated this PR to address the missing parts:

Added a bucket initialization job to ensure the required bucket is created automatically on startup
Exposed the Garage admin API via port 9001 (nodePort 30091) to allow inspection of bucket contents

Along with aligning the ports to match MinIO, this ensures that no changes are required for users or documentation.

Please let me know if this is okay or do i need to do something else on this.

EHJ-52n · 2026-04-17T13:15:04Z

Please reply to my questions from the review, thank you.

RitikaSen27 · 2026-04-17T17:04:51Z

Thanks for your review!

I’m currently not able to see the specific review comments or questions on this PR (in either the Conversation or Files changed tabs). Could you please point me to them or re-share them here?

I’ll address them right away.

EHJ-52n · 2026-04-20T07:28:42Z

hope this helps.

RitikaSen27 · 2026-04-20T10:26:46Z

Review 1 -
Yes, the endpoint should remain unchanged to keep the replacement transparent.

I’ve ensured that the configuration uses the same endpoint as before, while the Garage service matches MinIO ports on the Kubernetes level.

RitikaSen27 · 2026-04-20T10:29:45Z

Review 2-
Yes, this change is unrelated to the MinIO replacement.I have a separate PR for this issue.

I’ve removed the delete_job implementation from this PR and will keep it in that separate PR dedicated to that functionality.

RitikaSen27 · 2026-04-20T10:32:37Z

Review 3-
You're right, these test changes are also related to the delete_job functionality.

I’ve removed them from this PR and will keep them only in the separate PR dedicated to that feature.

RitikaSen27 · 2026-04-20T10:39:48Z

Thank you for letting me the know the reviews. I have answered all of them and done the required changes. I hope this will be helpful.
Kindly let me know if something else needs to be done

EHJ-52n · 2026-04-24T12:44:54Z

@RitikaSen27 Thank you very much for the improvements. I added some new questions and I am looking forward to your responses!

RitikaSen27 · 2026-04-26T05:26:19Z

I apologise for the late response .

I’m still not able to see the new questions in the PR (I checked both the Conversation and Files changed tabs, including hidden/outdated comments). Could you please point me to them or re-share them?

I'll be working on them right away.

EHJ-52n

Dear @RitikaSen27,

I did not submit my review, hence you were not able to see my questions and comments.

Sorry for the delay.

EHJ-52n · 2026-04-14T06:12:23Z

                "PYGEOAPI_K8S_MANAGER_NAMESPACE": "default",
                "PYGEOAPI_K8S_MANAGER_API_TOKEN": "do_not_use_in_production",
-                "PYGEOAPI_K8S_MANAGER_FINALIZER_BUCKET_ENDPOINT": "http://localhost:30090",
+                "PYGEOAPI_K8S_MANAGER_FINALIZER_BUCKET_ENDPOINT": "http://localhost:30091",


Why is this changed?

Shouldn't the minio replacement happen under the hood, hence the endpoint for the manager is still the same?

EHJ-52n · 2026-04-14T06:12:59Z

        jobs = k8s_batch_api.list_namespaced_job(self.namespace)
        self.resource_version = jobs.metadata.resource_version
        LOGGER.debug(f"resource_version received: '{self.resource_version}'.")
+    def delete_job(self, job_id: str, k8s_batch_api: BatchV1Api | None = None):


This should not be part of the PR to replace minio AFAIK.

EHJ-52n · 2026-04-14T06:13:18Z


    mocked_get_logs_for_pod.assert_not_called()
    mocked_upload_logs_to_s3.assert_not_called()
+


this should not be part of the PR to replace minio.

EHJ-52n · 2026-04-24T12:42:16Z

+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              sleep 10;


done sleep for a fixed amount of time, but check, if the service is available. See the removed minio init job for an example.

EHJ-52n · 2026-04-24T12:43:17Z

+              sleep 10;
+              aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID;
+              aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY;
+              aws configure set default.region us-east-1;


Why us-east-1? Can't this be left out or ignored, because it's not required at all?

EHJ-52n · 2026-04-24T12:43:33Z

+              aws --endpoint-url=http://garage:9000 s3 mb s3://pygeoapi || true;
+          envFrom:
+            - secretRef:
+                name: garage-credentials


Please add empty line at EOF.

EHJ-52n · 2026-04-24T12:43:57Z

 import datetime
 import os
 from unittest.mock import MagicMock, patch
+from unittest.mock import MagicMock, ANY


Why is this still here?

EHJ-52n · 2026-04-24T12:44:01Z

        self.resource_version = jobs.metadata.resource_version
        LOGGER.debug(f"resource_version received: '{self.resource_version}'.")
-
+        


Why is this still here?

EHJ-52n · 2026-04-24T12:44:06Z

    V1Job,
    V1Pod,
 )
+from tests.test_manager import job_id


Why is this still here?

EHJ-52n · 2026-06-10T08:32:58Z

@RitikaSen27 are you willing to update this PR as requested? Looking forward to your updates.

RitikaSen27 · 2026-06-10T10:07:26Z

Hi,

Thank you for following up. I appreciate the review and the detailed feedback.

Unfortunately, I am currently in the middle of my university examinations and won't be able to work on the requested changes for a few weeks. Given this, I understand if you prefer to close the PR for now.

Once my exams are over, I'd be happy to revisit the issue and submit a new PR that addresses all the review comments and project requirements.

Thank you for your time and feedback.

RitikaSen27 · 2026-06-24T16:04:46Z

Hi,

My exams are over now, thanks for waiting. I'll address all the open review comments and push the fixes to this branch shortly.

…Garage init job

RitikaSen27 · 2026-06-24T17:03:42Z

Hi,

I've pushed updates addressing all the outstanding review comments:

Replaced the fixed sleep 10 in the Garage init job with a loop that checks if the service is actually available before proceeding
Removed the unnecessary aws configure set default.region us-east-1 line
Added the missing newline at the end of garage.yaml
Removed the leftover delete_job-related code in finalizer.py and test_finalizer.py (unused import and stray code)

All existing tests still pass. Please let me know if anything else needs adjusting.

Thanks again for your patience and the thorough review!

EHJ-52n · 2026-06-25T07:33:52Z

Please update the kind image to the latest version available for v1.33 and verify, if everything is working fine by providing the output of each command of the ./k8s-kind/README.md.
Fix the merge conflicts

Ensure this error is fixed, by testing your proposal:

(⎈|kind-pygeoapi-k8s-manager:N/A)~/Code/pygeoapi_k8s-manager/k8s-kind 
(RitikaSen27-replace-minio-with-garage|✚1) kubectl apply -k .

error: accumulating resources: accumulation err='accumulating resources from 
'minio.yaml': evalsymlink failure on '~/Code/pygeoapi_k8s-manager/k8s-kind/minio.yaml'
 : lstat ~/Code/pygeoapi_k8s-manager/k8s-kind/minio.yaml: no such file or directory': 
 must build at directory: not a valid directory: evalsymlink failure on 
 '~/Code/pygeoapi_k8s-manager/k8s-kind/minio.yaml' : lstat 
 ~/Code/pygeoapi_k8s-manager/k8s-kind/minio.yaml: no such file or directory

…lizer.py conflict

…nio.yaml

… leftover MinIO values

RitikaSen27 · 2026-06-29T11:01:23Z

Hi,

Pushed fixes for the latest review:

Resolved the merge conflict with upstream/main (tests/test_finalizer.py), all tests pass
Fixed kustomization.yaml to point at garage.yaml instead of the deleted minio.yaml — verified kubectl apply -k . now succeeds with no errors
Also fixed a leftover bug in manager.yaml where the bucket endpoint/credentials were still set to MinIO's values instead of Garage's
Bumped the kind node image to v1.33.12 (latest v1.33 patch)

Still working on full Garage bucket/key automation and the complete README verification with command output — will follow up with that shortly.

RitikaSen27 added 2 commits March 29, 2026 15:54

Replace MinIO with Garage as S3-compatible storage backend

df9ca42

Add delete_job method to support OGC API Processes Dismiss operation

2c8c3c6

Add bucket initialization job and health checks for Garage

e4a1ea3

Align Garage service ports with MinIO as requested

0126a7c

Add bucket initialization and expose admin interface for Garage

030b66f

Revert endpoint change to keep MinIO replacement transparent

5ac3788

Remove delete_job from this PR as it belongs to separate feature

4537681

Remove delete_job tests from this PR as they belong to separate feature

6317db6

EHJ-52n requested changes Jun 2, 2026

View reviewed changes

RitikaSen27 added 3 commits June 24, 2026 21:56

Replace fixed sleep with readiness check and drop unneeded region in …

f77aab4

…Garage init job

Remove stray trailing whitespace in finalizer.py

4411f6e

Remove duplicate unused ANY import in test_finalizer.py

748e372

RitikaSen27 added 4 commits June 27, 2026 11:01

Merge upstream/main into replace-minio-with-garage, resolve test_fina…

2d9e9ef

…lizer.py conflict

Fix kustomization.yaml to reference garage.yaml instead of removed mi…

c545ccf

…nio.yaml

Update manager.yaml to use Garage endpoint and credentials instead of…

ba45a76

… leftover MinIO values

Bump kind node image to v1.33.12 (latest patch for Kubernetes v1.33)

bb03027


		mocked_get_logs_for_pod.assert_not_called()
		mocked_upload_logs_to_s3.assert_not_called()

		self.resource_version = jobs.metadata.resource_version
		LOGGER.debug(f"resource_version received: '{self.resource_version}'.")

Uh oh!

Conversation

RitikaSen27 commented Mar 29, 2026

Summary

Changes

Details

Notes

Future Improvements

Uh oh!

EHJ-52n commented Mar 30, 2026

Uh oh!

RitikaSen27 commented Mar 30, 2026

Uh oh!

RitikaSen27 commented Mar 30, 2026

Uh oh!

EHJ-52n commented Apr 9, 2026

Uh oh!

RitikaSen27 commented Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EHJ-52n commented Apr 13, 2026

Uh oh!

RitikaSen27 commented Apr 13, 2026

Uh oh!

EHJ-52n commented Apr 14, 2026

Uh oh!

RitikaSen27 commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RitikaSen27 commented Apr 14, 2026

Uh oh!

EHJ-52n commented Apr 17, 2026

Uh oh!

RitikaSen27 commented Apr 17, 2026

Uh oh!

EHJ-52n commented Apr 20, 2026

Uh oh!

RitikaSen27 commented Apr 20, 2026

Uh oh!

RitikaSen27 commented Apr 20, 2026

Uh oh!

RitikaSen27 commented Apr 20, 2026

Uh oh!

RitikaSen27 commented Apr 20, 2026

Uh oh!

EHJ-52n commented Apr 24, 2026

Uh oh!

RitikaSen27 commented Apr 26, 2026

Uh oh!

EHJ-52n left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

EHJ-52n commented Jun 10, 2026

Uh oh!

RitikaSen27 commented Jun 10, 2026

Uh oh!

RitikaSen27 commented Jun 24, 2026

Uh oh!

RitikaSen27 commented Jun 24, 2026

Uh oh!

EHJ-52n commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

RitikaSen27 commented Apr 12, 2026 •

edited

Loading

RitikaSen27 commented Apr 14, 2026 •

edited

Loading

EHJ-52n commented Jun 25, 2026 •

edited

Loading