Skip to content

Changing the DUG Data Model#393

Merged
YaphetKG merged 83 commits intoDugModel2.0from
data-model-update
Dec 19, 2025
Merged

Changing the DUG Data Model#393
YaphetKG merged 83 commits intoDugModel2.0from
data-model-update

Conversation

@hina-shah
Copy link

@hina-shah hina-shah commented Apr 15, 2025

This PR addresses the following changes

  • Introduce a new Data Model to represent studies, concepts, variables and questionnaires.
  • Make the data model to be pydantic
  • Update the parsers for HEAL
  • Update tests to support the new data model
  • Add new API endpoints to support this new data model
  • Make the APIs be backwards compatible to not break existing deployments.

@hina-shah hina-shah requested review from YaphetKG and gaurav April 15, 2025 13:43
@hina-shah hina-shah marked this pull request as draft April 23, 2025 20:26
@hina-shah hina-shah changed the base branch from develop to DugModel2.0 June 15, 2025 15:23
element.add_concept(concept)
elements.append(element)
return elements
# TODO; This function will be deprecated once CDEs are implemented.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this part i don't think we need it anymore.

@hina-shah hina-shah marked this pull request as ready for review December 9, 2025 15:07
@hina-shah hina-shah requested a review from YaphetKG December 11, 2025 18:39
@hina-shah hina-shah changed the title WIP: Changing the DUG Data Model Changing the DUG Data Model Dec 11, 2025
}


@APP.post('/concepts', tags=['v2.0'], response_model=ConceptsAPIResponse)
Copy link
Contributor

@frostyfan109 frostyfan109 Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How come /concepts was not updated to use Search.search_elements like the other endpoints?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: updated this endpoint to use search_elements on the filtering PR, so changes will come later.

Comment on lines +56 to +58
# Program names can be added to studies and sections as well?
if not isinstance(element, DugConcept) and self.program_name is not None:
element.add_program_name(self.program_name) ## Why?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comments are a bit puzzling to me here, what's the "why?" concerning?

Comment on lines +89 to 97
# dug_elements_from_graph += self.expand_to_dug_element(
# concept=concept,
# casting_config=casting_config,
# dug_element_type=dug_element_type,
# tranql_source=tranql_source
# )

# add new elements to parsed elements
self.elements += dug_elements_from_graph
Copy link
Contributor

@frostyfan109 frostyfan109 Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this whole dug_elements_from_graph part no longer does anything. probably should delete? seems like the associated method should also be deleted as per Yaphet's comment.

What replaces this functionality?

action = identifier.purl,
description=identifier.description
)
concept.concept_type = identifier.types
Copy link
Contributor

@frostyfan109 frostyfan109 Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line got me thinking, should concept_type shift towards being an array in the new model (e.g. concept_types), or has this been discussed already?

if filepath.is_file():
yield filepath
else:
print(filepath.glob("**/*"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be deleted/logged instead

for result in elastic_results:
item = result["_source"]
item["score"] = result["_score"]
item["explanation"] = result.get("_explanation", {})
Copy link
Contributor

@frostyfan109 frostyfan109 Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the moment, an explanation won't show up for variables. This field could be parametrized into the SearchElementQuery schema and added as a keyword arg to the search_elements method, though. I've done the kwarg part on my filtering branch, but not taken the extra step of parametrizing it across endpoints within the json payload.

@YaphetKG YaphetKG merged commit c4c081e into DugModel2.0 Dec 19, 2025
@YaphetKG YaphetKG deleted the data-model-update branch December 19, 2025 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants