Perform 2 way sync

[dennyabrain]

Guiding Directions

• whatever anno.milli receives should be treated as source file
• anno.milli should erase what it receives a differently annotated file
• anno.milli is a decentralized platform, multiple anno.milli running in parallel to annotate same EAD

[maanasb01]

Hello, here is an overview of the existing flow and working of EAD import and Export functionalities. We can have a further discussion on the scoping and whether to change the existing flow or add a new functionality for syncing, based on this.

Importing an EAD

• During the upload of an EAD XML file, the Anno Milli app converts the XML to a map and extracts the relevant fields like the collection's metadata, series, sub-series, and file's metadata. Most of the extraction occurs for the File's properties. For Series and Sub-series, only the basic metadata, such as name and IDs, are extracted. These properties are then stored in the database.
  • All the other fields/ tags and properties are ignored. Therefore, any additional tags (such as annotations in the updated XML file) will not be able to get processed during uploading/ syncing in this current implementation.
• The "original" EAD.xml file is then stored as it is separately in the backend.

Exporting the EAD

• While exporting, the app takes the original EAD.xml file related to the collection, and then writes all the approved annotations linked to that collection to it, and then exports the updated file.
• This does not overwrite the original EAD.xml file. That file stays as it is. Every time during the export, the app collects all the annotations related to the collection and then writes them to a copy of the original EAD.xml contents, and then exports it.

CC: @dennyabrain @tarunima @dakuojas @vns2

[maanasb01]

Related to the syncing of annotations, it was discussed that in the future, there could be multiple apps like Anno Milli, and the syncing would happen between all of them and the NCBS Archival space (more like between each of them and the NCBS space).

An Approach Suggestion

One approach that I would like to suggest is that NCBS be the orchestrator of all the annotations, and the Anno Milli (and similar apps in the future) would just be there for annotating.

Syncing the metadata from the updated EAD:

• We also want to sync the metadata (any change in the properties of the relevant fields like collections, files, etc.).
• I think this functionality can be achieved with the existing flow (above comment), as we are only extracting specific details from an XML. So, for syncing an updated EAD, we can process it similarly and just update the properties (if any).
• Here, by updated EAD, I meant if there is any update on the properties of the objects (files, collection, etc.), and not whether there are new annotation tags or not.

Annotations on the Anno-Milli app

• The annotations that were created on the app remain on the app all the time. They will be displayed as annotations on the File page like they are now.

Syncing Annotations

• Here comes the part when "NCBS being an orchestrator" comes.
• We can provide an API endpoint for the NCBS to fetch all the annotations from Anno-Milli.
• We can probably do a versioning of the annotations, so the NCBS can just fetch the new annotations.
• In case someone deletes an annotation from Anno Milli (I currently cannot recall if we are allowing the users to delete their annotations after they are approved or not, but in case we want to allow them, this point is for that scenario), we can communicate that with the NCBS space as well.
• Now, the NCBS can add these annotations to their EAD however they want, with all the control, and display them on their archival space according to their preference.

New, Simpler flow on Anno-Milli

• As the current flow of Import EAD (above comment) is only concerned about extracting the specific files/collection-related info, that part could stay as it is to import a new EAD collection (and update the properties while syncing, in the future as well).
• For exporting EAD with annotations, we won't need this functionality as we can provide all the annotations to NCBS through an API.

Probable less complicated Interoperability and Ingestion flow

• This would also probably reduce the complexity of interoperability between Anno Milli and NCBS Archival Space because now the NCBS doesn't have to deal with the exported XMLs from the Anno-Milli. The NCBS Archival Space can edit its EAD within its own environment with all the annotations, so this could probably be a less complex flow while dealing with different EAD formats and standards, because I assume that the NCBS is already familiar with dealing with/processing all these formats.

[dakuojas]

@maanasb01 An API sounds like a neat solution to me!

OAI-PMH is an API developed for this specific reason. There are implementations of OAI-PMH that can serve EAD contructed directly from a database completely the problem of stale EADs
OAI-PMH actually exists to solve this specific problem of having to creating exports all the time. Now that we already know how to add annotations serving them in a standrds comliant way is just a matter of having an OAI-PMH enpoint for anno.milli
A good reference implementation OAI-PMH
in php: https://github.com/opencultureconsulting/oai-pmh2/
in python: https://github.com/infrae/pyoai

Central to the concept of anno.milli is its ability to get the data back to source archives in a standards compliant way -- it can be an EAD export (xml file export - clunky and complicated) or OAI-PMH (generate xml export from database -- providers and harvesters available). Providing only annotations is probably the most direct solution but getting archives to actually import and use them is the challenge. Having a non-standard API for annotations is adding another one to a list of many others (https://gitlab.com/Daniel-KM/Omeka-S-module-Annotate).
The onus is on the woefully inactive source archives to develop a client to harvest from the API and integrate annotations into the EAD. They will have no buy-in to use anno.milli if the data is siloed or the API access is non-standard (ie non OAI-PMH endpoint, EAD export or Dublin Core export).

To address the larger question of two-way sync and provide some courage with the fact that better minds have been at it longer than us :) --
There is the now-defunct OAI-PMH ResourceSync Framework (https://www.openarchives.org/rs/toc), which has the concept of change lists central to their whole architecture although I am not sure if there is a successful implementation of it. The only one I know of is https://github.com/DSpace/DSpaceResourceSync
Worth a quick read but not sure if its implementable in a sprint.

[dennyabrain]

@dakuojas I'm trying to think what is an acceptable solution to build during this sprint. It would ideally be great if

1. anno.milli could import data in all formats (ead, dublin core etc)
2. anno.milli could export data in all formats (ead, web-annotation, oai-pmh etc) and mechanisms (standard or non-standard API)

but in the interest of time, we will have to pick some of these to support at the moment. One factor to decide would be to prioritize the protocols, formats and mechanisms that are supported by our immediate imagined collaborators. I'm thinking of NCBS archive and Caltech archive as two entities I've heard from you. Maybe there's more that are on your immediate radar. What would be some choices to make that will increase our success with them?

For instance, if we were to support export/sync via the oai-pmh, would the Caltech team be able to use anno.milli immediately? If not, would you suggest a different choice for export/sync with them?

Are there any other factors you can think of that would help pick the format/mechanism for export/sync for this sprint?

From reading what you have written, I get the sense that oai-pmh, while a standard does require some buy in from the archives. To implement things on their end. Which I am guessing won't be a problem for NCBS but might not be a guarantee with other archives. So one way to plan the scope of work for this sprint could be that we demonstrate 2 way sync between milli archive and anno.milli using a standard like oai-pmh to advocate other archives to integrate oai-pmh. And then we don't worry about buy ins from other archives in this sprint. In which case maybe the decision is yours to choose the best standard that we can integrate that maybe guarantees future proofing and go ahead and implement it.

[dennyabrain]

In the documentation for oai-pmh standard, I see a lot of references to dublin-core as opposed to ead. Would that be an interporability issue that our 0.1 version supports ead for import but exports data in dublin-core format?

[dennyabrain]

We decided in the previous call to implement oai-pmh. Will use the documentation and this python implementation for reference,

I will document the various API endpoints and XML tree structure that @maanasb01 needs to implement. We need to figure out a way to test it with @dakuojas eventually.

[dennyabrain]

one thing to confirm from @dakuojas, it looks like this protocol requires the client app to make requests to the server (anno.milli). Does that mean the cat software needs to make recurring polling requests? Is this a norm with software in this space?

[dakuojas]

it does
there have been few examples of clients harvesting from an oai pmh server
its not ideal without showing a demonstration of a successful harvest that updates archival data on cat.milli but it allows us on anno.milli end that we provide the means to harvest -- we leave the archives client to deal with it however they please