Static lineage with OpenLineage in Marquez

[irua84]

Hi!
We are trying to do static modeling of a data virtualization system with many views that depend on each other using DatasetEvent. When we make the request in Marquez using those DatasetEvents we cannot see the lineage of the views, however if we use EventRun we do see the lineage.
It surprises us that despite being a static model, we need to add Jobs and Runs to be able to visualize the lineage we want in Marquez.
Is this normal from OpenLineage's point of view? Or is it something that Marquez demands because it is not completely adapted to a static lineage?

Thanks!

[thijs-s]

Hey! This is expected behavior from both the OpenLineage spec and Marquez, but you have a better option than RunEvent you can use JobEvent, which gives you static lineage without requiring synthetic runs.

A DatasetEvent describes a single dataset it has no inputs or outputs fields, so there's no way for Marquez to know how datasets relate to each other. It's designed for dataset-level metadata (schema changes, lifecycle events), not for expressing dependencies.

RunEvent carries inputs, outputs, a job, and a run it represents an executio that consumed and produced datasets. This is the standard OpenLineage way to express lineage, and Marquez builds its lineage graph from these relationships. But for a static model, creating synthetic runs is unnecessary overhead.

The better option: JobEvent

JobEvent was designed exactly for your use case, it declares a job's input/output interface without requiring a run. It has inputs and outputs fields like RunEvent, but no run field.

We verified this works end-to-end. For your view chain (e.g., raw_table → view_a → view_b → view_c), you would send:

{
  "eventTime": "2024-01-01T10:00:00.000Z",
  "job": { "namespace": "your-namespace", "name": "view_a" },
  "inputs": [{ "namespace": "your-namespace", "name": "raw_table" }],
  "outputs": [{ "namespace": "your-namespace", "name": "view_a_dataset" }],
  "producer": "https://your-system",
  "schemaURL": "https://openlineage.io/spec/2-0-0/OpenLineage.json#/definitions/JobEvent"
}

{
  "eventTime": "2024-01-01T10:01:00.000Z",
  "job": { "namespace": "your-namespace", "name": "view_b" },
  "inputs": [{ "namespace": "your-namespace", "name": "view_a_dataset" }],
  "outputs": [{ "namespace": "your-namespace", "name": "view_b_dataset" }],
  "producer": "https://your-system",
  "schemaURL": "https://openlineage.io/spec/2-0-0/OpenLineage.json#/definitions/JobEvent"
}

The key detail is the schemaURL it must end with #/definitions/JobEvent. This is how Marquez distinguishes it from a RunEvent.

The lineage is then queryable from any node:

GET /api/v1/lineage?nodeId=job:your-namespace:view_b&depth=20
GET /api/v1/lineage?nodeId=dataset:your-namespace:view_a_dataset&depth=20

Event Type	Has inputs/outputs?	Requires run?	Produces lineage?	Use case
DatasetEvent	No	No	No	Dataset metadata only
JobEvent	Yes	No	Yes	Static modeling (your case)
RunEvent	Yes	Yes	Yes	Execution tracking

Lineage in OpenLineage is always expressed through a job that connects inputs to outputs. But you don't need runs, JobEvent gives you the static lineage you want.

Looking ahead: first-class static lineage support

While JobEvent solves the immediate problem, we recognize that modeling views as "jobs" is a conceptual mismatch, views aren't jobs, they're datasets with dependencies. If you're interested in seeing first-class support for static lineage workloads (e.g., direct dataset-to-dataset lineage via DatasetEvent), we'd love to hear your use cases and requirements. Please open an issue(withh full explanation,description,ideas) at:

https://github.com/ilum-cloud/marquez/issues

[irua84]

Hey!

I've been doing several tests and I have not been able to obtain the columnlineage using JobEvents.
This is one of the JobEvents used:

{
"eventTime": "2026-02-24T09:18:15.8572708Z",
"producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/JobEvent",
"job": {
"namespace": "book05",
"name": "books_sales_details"
},
"inputs": [
{
"namespace": "book05",
"name": "books_details",
"facets": {
"schema": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://openlineage.io/spec/facets/1-2-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
"fields": [
{
"name": "book_id",
"type": "text"
},
{
"name": "publisher_id",
"type": "int"
},
{
"name": "title",
"type": "text"
},
{
"name": "price",
"type": "decimal"
},
{
"name": "category",
"type": "text"
},
{
"name": "quantity",
"type": "int"
},
{
"name": "book_format",
"type": "text"
},
{
"name": "book_year",
"type": "int"
},
{
"name": "author_id",
"type": "int"
},
{
"name": "author_name",
"type": "text"
},
{
"name": "publisher_name",
"type": "text"
},
{
"name": "city",
"type": "text"
},
{
"name": "zip",
"type": "text"
}
]
}
}
},
{
"namespace": "book05",
"name": "books",
"facets": {
"schema": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://openlineage.io/spec/facets/1-2-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
"fields": [
{
"name": "id",
"type": "text"
},
{
"name": "publisher_id",
"type": "int"
},
{
"name": "title",
"type": "text"
},
{
"name": "price",
"type": "decimal"
},
{
"name": "category",
"type": "text"
},
{
"name": "quantity",
"type": "int"
},
{
"name": "book_format",
"type": "text"
},
{
"name": "book_year",
"type": "int"
},
{
"name": "author_id",
"type": "int"
},
{
"name": "author_name",
"type": "text"
}
]
}
}
},
{
"namespace": "book05",
"name": "storeorders",
"facets": {
"schema": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://openlineage.io/spec/facets/1-2-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
"fields": [
{
"name": "id",
"type": "int"
},
{
"name": "customer_id",
"type": "int"
},
{
"name": "payment_type",
"type": "text"
},
{
"name": "order_date",
"type": "localdate"
}
]
}
}
},
{
"namespace": "book05",
"name": "publishers",
"facets": {
"schema": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://openlineage.io/spec/facets/1-2-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
"fields": [
{
"name": "id",
"type": "int"
},
{
"name": "publisher_name",
"type": "text"
},
{
"name": "city",
"type": "text"
},
{
"name": "zip",
"type": "text"
}
]
}
}
},
{
"namespace": "book05",
"name": "booksales",
"facets": {
"schema": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://openlineage.io/spec/facets/1-2-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
"fields": [
{
"name": "id",
"type": "int"
},
{
"name": "book_id",
"type": "text"
},
{
"name": "order_id",
"type": "int"
}
]
}
}
}
],
"outputs": [
{
"namespace": "book05",
"name": "books_sales_details",
"facets": {
"columnLineage": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://openlineage.io/spec/facets/1-2-0/ColumnLineageDatasetFacet.json#/$defs/ColumnLineageDatasetFacet",
"fields": {
"publisher_id": {
"inputFields": [
{
"namespace": "book05",
"name": "books_details",
"field": "publisher_id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book05",
"name": "books",
"field": "publisher_id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"book_format": {
"inputFields": [
{
"namespace": "book05",
"name": "books_details",
"field": "book_format",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book05",
"name": "books",
"field": "book_format",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"order_id": {
"inputFields": [
{
"namespace": "book05",
"name": "storeorders",
"field": "id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"quantity": {
"inputFields": [
{
"namespace": "book05",
"name": "books_details",
"field": "quantity",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book05",
"name": "books",
"field": "quantity",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"customer_id": {
"inputFields": [
{
"namespace": "book05",
"name": "storeorders",
"field": "customer_id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"category": {
"inputFields": [
{
"namespace": "book05",
"name": "books_details",
"field": "category",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book05",
"name": "books",
"field": "category",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"city": {
"inputFields": [
{
"namespace": "book05",
"name": "books_details",
"field": "city",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book05",
"name": "publishers",
"field": "city",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"zip": {
"inputFields": [
{
"namespace": "book05",
"name": "books_details",
"field": "zip",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book05",
"name": "publishers",
"field": "zip",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"book_id": {
"inputFields": [
{
"namespace": "book05",
"name": "books_details",
"field": "book_id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book05",
"name": "books",
"field": "id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"payment_type": {
"inputFields": [
{
"namespace": "book05",
"name": "storeorders",
"field": "payment_type",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"id": {
"inputFields": [
{
"namespace": "book05",
"name": "booksales",
"field": "id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"order_date": {
"inputFields": [
{
"namespace": "book05",
"name": "storeorders",
"field": "order_date",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"author_id": {
"inputFields": [
{
"namespace": "book05",
"name": "books_details",
"field": "author_id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book05",
"name": "books",
"field": "author_id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"title": {
"inputFields": [
{
"namespace": "book05",
"name": "books_details",
"field": "title",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book05",
"name": "books",
"field": "title",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"price": {
"inputFields": [
{
"namespace": "book05",
"name": "books_details",
"field": "price",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book05",
"name": "books",
"field": "price",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"author_name": {
"inputFields": [
{
"namespace": "book05",
"name": "books_details",
"field": "author_name",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book05",
"name": "books",
"field": "author_name",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"book_year": {
"inputFields": [
{
"namespace": "book05",
"name": "books_details",
"field": "book_year",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book05",
"name": "books",
"field": "book_year",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"publisher_name": {
"inputFields": [
{
"namespace": "book05",
"name": "books_details",
"field": "publisher_name",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book05",
"name": "publishers",
"field": "publisher_name",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
}
}
},
"schema": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://openlineage.io/spec/facets/1-2-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
"fields": [
{
"name": "publisher_id",
"type": "int"
},
{
"name": "book_format",
"type": "text"
},
{
"name": "order_id",
"type": "int"
},
{
"name": "quantity",
"type": "int"
},
{
"name": "customer_id",
"type": "int"
},
{
"name": "category",
"type": "text"
},
{
"name": "city",
"type": "text"
},
{
"name": "zip",
"type": "text"
},
{
"name": "book_id",
"type": "text"
},
{
"name": "payment_type",
"type": "text"
},
{
"name": "id",
"type": "int"
},
{
"name": "order_date",
"type": "localdate"
},
{
"name": "author_id",
"type": "int"
},
{
"name": "title",
"type": "text"
},
{
"name": "price",
"type": "decimal"
},
{
"name": "author_name",
"type": "text"
},
{
"name": "book_year",
"type": "int"
},
{
"name": "publisher_name",
"type": "text"
}
]
}
}
}
]
}

The JSON contains both the the input schemas and the output schema, and also the columnlineage but,, the following request in Marquez returns an empty graph:
http://localhost:3000/api/v1/column-lineage?nodeId=dataset:book05:books_sales_details

The view column lineage links do not appear:

Even though the event contains the input and output schemas, Marquez does not show them:

However, when we use this RunEvent:

{
"eventTime": "2026-02-23T13:46:12.8924979Z",
"producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent",
"eventType": "COMPLETE",
"run": {
"runId": "6856375b-f212-4e97-aabd-4b7e02a453eb"
},
"job": {
"namespace": "book04",
"name": "books_sales_details"
},
"inputs": [
{
"namespace": "book04",
"name": "books_details",
"facets": {
"schema": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://openlineage.io/spec/facets/1-2-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
"fields": [
{
"name": "book_id",
"type": "text"
},
{
"name": "publisher_id",
"type": "int"
},
{
"name": "title",
"type": "text"
},
{
"name": "price",
"type": "decimal"
},
{
"name": "category",
"type": "text"
},
{
"name": "quantity",
"type": "int"
},
{
"name": "book_format",
"type": "text"
},
{
"name": "book_year",
"type": "int"
},
{
"name": "author_id",
"type": "int"
},
{
"name": "author_name",
"type": "text"
},
{
"name": "publisher_name",
"type": "text"
},
{
"name": "city",
"type": "text"
},
{
"name": "zip",
"type": "text"
}
]
}
}
},
{
"namespace": "book04",
"name": "books",
"facets": {
"schema": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://openlineage.io/spec/facets/1-2-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
"fields": [
{
"name": "id",
"type": "text"
},
{
"name": "publisher_id",
"type": "int"
},
{
"name": "title",
"type": "text"
},
{
"name": "price",
"type": "decimal"
},
{
"name": "category",
"type": "text"
},
{
"name": "quantity",
"type": "int"
},
{
"name": "book_format",
"type": "text"
},
{
"name": "book_year",
"type": "int"
},
{
"name": "author_id",
"type": "int"
},
{
"name": "author_name",
"type": "text"
}
]
}
}
},
{
"namespace": "book04",
"name": "storeorders",
"facets": {
"schema": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://openlineage.io/spec/facets/1-2-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
"fields": [
{
"name": "id",
"type": "int"
},
{
"name": "customer_id",
"type": "int"
},
{
"name": "payment_type",
"type": "text"
},
{
"name": "order_date",
"type": "localdate"
}
]
}
}
},
{
"namespace": "book04",
"name": "publishers",
"facets": {
"schema": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://openlineage.io/spec/facets/1-2-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
"fields": [
{
"name": "id",
"type": "int"
},
{
"name": "publisher_name",
"type": "text"
},
{
"name": "city",
"type": "text"
},
{
"name": "zip",
"type": "text"
}
]
}
}
},
{
"namespace": "book04",
"name": "booksales",
"facets": {
"schema": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://openlineage.io/spec/facets/1-2-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
"fields": [
{
"name": "id",
"type": "int"
},
{
"name": "book_id",
"type": "text"
},
{
"name": "order_id",
"type": "int"
}
]
}
}
}
],
"outputs": [
{
"namespace": "book04",
"name": "books_sales_details",
"facets": {
"columnLineage": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://openlineage.io/spec/facets/1-2-0/ColumnLineageDatasetFacet.json#/$defs/ColumnLineageDatasetFacet",
"fields": {
"publisher_id": {
"inputFields": [
{
"namespace": "book04",
"name": "books_details",
"field": "publisher_id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book04",
"name": "books",
"field": "publisher_id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"book_format": {
"inputFields": [
{
"namespace": "book04",
"name": "books_details",
"field": "book_format",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book04",
"name": "books",
"field": "book_format",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"order_id": {
"inputFields": [
{
"namespace": "book04",
"name": "storeorders",
"field": "id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"quantity": {
"inputFields": [
{
"namespace": "book04",
"name": "books_details",
"field": "quantity",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book04",
"name": "books",
"field": "quantity",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"customer_id": {
"inputFields": [
{
"namespace": "book04",
"name": "storeorders",
"field": "customer_id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"category": {
"inputFields": [
{
"namespace": "book04",
"name": "books_details",
"field": "category",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book04",
"name": "books",
"field": "category",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"city": {
"inputFields": [
{
"namespace": "book04",
"name": "books_details",
"field": "city",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book04",
"name": "publishers",
"field": "city",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"zip": {
"inputFields": [
{
"namespace": "book04",
"name": "books_details",
"field": "zip",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book04",
"name": "publishers",
"field": "zip",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"book_id": {
"inputFields": [
{
"namespace": "book04",
"name": "books_details",
"field": "book_id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book04",
"name": "books",
"field": "id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"payment_type": {
"inputFields": [
{
"namespace": "book04",
"name": "storeorders",
"field": "payment_type",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"id": {
"inputFields": [
{
"namespace": "book04",
"name": "booksales",
"field": "id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"order_date": {
"inputFields": [
{
"namespace": "book04",
"name": "storeorders",
"field": "order_date",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"author_id": {
"inputFields": [
{
"namespace": "book04",
"name": "books_details",
"field": "author_id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book04",
"name": "books",
"field": "author_id",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"title": {
"inputFields": [
{
"namespace": "book04",
"name": "books_details",
"field": "title",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book04",
"name": "books",
"field": "title",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"price": {
"inputFields": [
{
"namespace": "book04",
"name": "books_details",
"field": "price",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book04",
"name": "books",
"field": "price",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"author_name": {
"inputFields": [
{
"namespace": "book04",
"name": "books_details",
"field": "author_name",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book04",
"name": "books",
"field": "author_name",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"book_year": {
"inputFields": [
{
"namespace": "book04",
"name": "books_details",
"field": "book_year",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book04",
"name": "books",
"field": "book_year",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
},
"publisher_name": {
"inputFields": [
{
"namespace": "book04",
"name": "books_details",
"field": "publisher_name",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
},
{
"namespace": "book04",
"name": "publishers",
"field": "publisher_name",
"transformations": [
{
"type": "DIRECT",
"subtype": "IDENTITY",
"description": ""
}
]
}
]
}
}
},
"schema": {
"_producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
"_schemaURL": "https://openlineage.io/spec/facets/1-2-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
"fields": [
{
"name": "publisher_id",
"type": "int"
},
{
"name": "book_format",
"type": "text"
},
{
"name": "order_id",
"type": "int"
},
{
"name": "quantity",
"type": "int"
},
{
"name": "customer_id",
"type": "int"
},
{
"name": "category",
"type": "text"
},
{
"name": "city",
"type": "text"
},
{
"name": "zip",
"type": "text"
},
{
"name": "book_id",
"type": "text"
},
{
"name": "payment_type",
"type": "text"
},
{
"name": "id",
"type": "int"
},
{
"name": "order_date",
"type": "localdate"
},
{
"name": "author_id",
"type": "int"
},
{
"name": "title",
"type": "text"
},
{
"name": "price",
"type": "decimal"
},
{
"name": "author_name",
"type": "text"
},
{
"name": "book_year",
"type": "int"
},
{
"name": "publisher_name",
"type": "text"
}
]
}
}
}
]
}

The Marquez http://localhost:3000/api/v1/column-lineage?nodeId=dataset:book04:books_sales_details request returns a graph:

In the Marquez UI the view column lineage links are available:

And all the datasets have the schema:

As you can see, the only difference between the JSON of the JobEvent and the JSON of the RunEvent is the schemaURL, the event type, run and namespace but the input and output schemas and the column lineage of the views are the same.

There is something wrong with the JobEvent I am creating?

Thanks!

[thijs-s]

good catch. I will do some tests.

[irua84]

Hi!

Any news on this?

Thanks.

[thijs-s]

yup, sorry. It might be a bug. Which version of marquez are you using ?

[irua84]

Hi,

I using the 0.51.1 version.

[irua84]

Hi,

I have tried version 0.50.0 and the results have been the same.

[thijs-s]

yup, I'm investigating this. I'll try to publish a fix together with the next version.

[thijs-s]

check this version https://github.com/ilum-cloud/marquez or that docker image ilum/marquez:0.54.0

[irua84]

This version works properly. Thanks!!!!