BCR Module Indexing

[jordan-bonser]

This is the discussion for the "BCR Module Indexing" proposal.

[meteorcloudy]

Discovering modules from a given registry is a very simple task that can be handled by coding agents nowadays, so I prefer not to add more complexity to BCR and Bazel itself.

[jordan-bonser]

Are you talking about querying the git repository and looking in the modules folder from there?

It might work for the public Github to access the BCR, although I'm not sure what Github's policy is of web requests from AI or whether they are likely to get rejected, rate limited or denied for lack of authentication.

For private registries I don't think this will work, there doesn't appear to be a mapping from the registry itself to the git repository for the registry. It would also likely require authenticating with the source control site before being able to read the repository.

Finally there isn't actually any guarantee that a registry has a git repo to query at all, the only requirement of the registry is that it hosts the files in a particular structure, with certain files like metadata.json and source.json.

Implementing this functionality would actually make it easier for an agent to query a registry, by being able to request a single file that is in JSON format that an agent can parse very easily.

Unless I'm missing another approach agents could use.

[meteorcloudy]

I see, yeah, the current Bazel registry setup allows checking if a module exists in BCR, but doesn't support discovering all available modules.

I'm open to this proposal. @Wyverald @fmeum WDYT?

[Wyverald]

Copying my comments over from bazelbuild/proposals#416:

I propose we add an index.json under the modules directory of the registry, which will list the modules that exist in the registry.

IMO we should put this file in the root directory -- it should be possible for a module named index.json to exist.

Handling Merge Conflicts

this is my biggest concern with a proposal like this, so thanks for devoting a sizeable chunk of text to it. I have just one question regarding the merge driver etc setup: would it allow multiple PRs touching index.json to be submitted without the PR authors needing to sync and resolve conflicts manually?

Say users A and B both submit a PR adding a new module, and both PRs change index.json accordingly. Both PRs are approved at around the same time. A's PR goes in first. When B's PR is about to be submitted, does B get an error, or does the PR submission process automatically run the merge driver? If B does get an error, can they somehow run the merge driver using GitHub's web UI, or will they actually have to resort to using the git CLI themselves?

[fmeum]

As we need to make this an optional feature anyway, could we make this endpoint dynamic and not represent it in git? Any kind of extra churn for contributors should be avoided.

[Wyverald]

Hmm. We could actually maybe synthesize the index.json file right before copying into GCS (somewhere here).

[meteorcloudy]

Ah, that's a great idea!