Incremental RestAPI pipeline fails when it has no new data

[vitorddf]

Mandatory informations:

Are there any customers directly impacted by this bug? Which ones?

Ativy

Bug Category

• Connection Manager
• Connection Test
• Creating Pipeline
• Executing Pipeline
• Cataloging Data Asset
• Other

Describe the bug

Ativy's pipelines were made through the RestAPI connector and configured to perform incremental loads.
For tables that don't have daily new data to increment, sync fails because the connector cannot identify the table's schema.

Example:
Pipeline: Stack_Cloud
Id: a8c0b8a7-4ce5-44ce-b8c1-ef05fae02cc4

Expected behavior

If you don't have new data, finish the process without pointing out errors

Does this bug impact any demo or a sale?

No

 

Other informations:

Any logs, error output, etc? …

ValueError: Could not detect type from empty type_dict. Did you forget to define a property in the stream schema?

What environment of software are you using?

• PRD
• Other

When the bug happened: 22 march 2023.

[rafaelsantanaep]

@beatrizaantunes pedi pro @vitorddf mapear esse bug porque a Ativy praticamente conseguiu implementar todos pipelines relacionados ao ERP deles, porém, como não tem dados novos, em alguns casos, vai existir falhas intermitentes. Pedi pro Vitor mapear pipelines que são one shot pra gente ajustar o scheduling pra diminuir esse número.