Feature/support cosmos #130
Conversation
hmoazam
force-pushed
the
feature/support-cosmos
branch
from
4a9abb2 to
5baee87
Compare
hmoazam
had a problem deploying
to
Integration
GitHub Actions
Failure
hmoazam
had a problem deploying
to
Integration
GitHub Actions
Failure
hmoazam
had a problem deploying
to
Integration
GitHub Actions
Failure
|
@hmoazam would you please also add a unit test to the QNParsers file too? |
| _log.LogError(ex, $"OlMessageConsolodation-JoinEventData: Error {ex.Message} when deleting entity"); | ||
| } | ||
| // Add Inputs to event if not already there (will only be done for DataSourceV2 sources) | ||
| if (olEvent.Inputs.Count == 0) { |
There was a problem hiding this comment.
Is there a case where there may be an input present initially and then more inputs later on?
What about a scenario where ABFSS is used and joined with a Data Source V2 source like Cosmos?
| /// from the start and complete events | ||
| /// </summary> | ||
| private bool isDataSourceV2Event(Event olEvent) { | ||
| string[] special_cases = {"azurecosmos://", "iceberg://"}; // todo: make this configurable? |
There was a problem hiding this comment.
Agreed, we should definitely make this part of the configuration instead. Maybe something like openLineageDataSourceV2Prefixes as the parameter and have it separated by semicolon (;)
Similar to the Spark_Entities config. Is there ever a case where "://" would not be part of the prefix?
|
|
||
| private bool IsJoinEvent(Event olEvent) | ||
| { | ||
| string[] special_cases = {"cosmos", "iceberg"}; |
There was a problem hiding this comment.
Does this get used in here?
| /// Performs initial validation of OpenLineage input | ||
| /// The tested criteria include: | ||
| /// 1. Events have both inputs and outputs | ||
| /// a. Except for special cases covered in isDataSourceV2Event |
There was a problem hiding this comment.
Could we add why this is necessary? Such as "OpenLineage does not emit a start event or complete event with all of the necessary information and so must be consolidated later on"?
|
A nitpicky thing :( function-app/adb-to-purview/src/Function.Domain/Helpers/parser/DatabricksToPurviewParser.cs Do not include any actual code changes, just whitespace. Any chance you'd be willing to undo the whitespace changes just to keep the commit clean and focused on the files necessary to change? |
|
Closing in favor of #145 |
2 participants
Implemented support for Azure Cosmos