Skip to content

Releases: bitol-io/open-data-contract-standard

Version 3.1.0

08 Dec 00:37
@pflooky pflooky
v3.1.0
b9d3ffc
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Version 3.1.0 Latest
Latest

Highlights

  • Splits Main specification document into several smaller documents.
  • Most sections have gained an optional id to enable easier linking as per RFC 26.
  • The team block is accepting both ODCS v3.0.x structure (now obsolete) or the updated RFC16 structure. The obsolete structure will be removed in ODCS v4.
  • Adds Relationships (Foreign Keys):
    • Add relationships array field to both SchemaObject and SchemaProperty to define foreign key relationships.
    • Support for property-level relationships where from field is implicit.
    • Support for schema-level relationships with explicit from and to fields.
    • Support for composite foreign keys using arrays in from and to fields.
    • Support for nested property references using dot shorthand notation (e.g., accounts.address_street).
    • Support for nested property references using fully qualified references (e.g /schema/schema_id/properties/my_property)
    • Add customProperties to relationships for metadata like cardinality, labels, and descriptions.
    • New Relationship definition in JSON schema with fields:
      • type: Type of relationship (defaults to foreignKey)
      • from: Source property reference (optional at property level)
      • to: Target property reference (required)
      • customProperties: Additional metadata
  • Breaking change to the JSON Schema (as a reminder the standard is not the JSON Schema but the textual document):
    • Alter exclusiveMaximum and exclusiveMinimum for integer/number logical data type to be number instead of boolean. Conforms with JSON Schema specification.
    • Alter exclusiveMaximum and exclusiveMinimum for date logical data type to be string instead of boolean.
    • No additional or unevaluated properties are allowed for the following sections of the schema:
      • authoritativeDefinitions
      • customProperties
      • dataQuality
      • dataQualityCheck
      • price
      • role
      • schemaElement
      • server
      • slaProperties
      • support
      • team
    • Alter team to be an object instead of an array.
      • Adds name, description, members, tags, customProperties, authoritativeDefinitions fields to team.
      • Adds tags, customProperties, authoritativeDefinitions fields to team.members.
  • Changes to logicalType and logicalTypeOptions:
    • Add timestamp and time to logicalType options.
    • Add timezone and defaultTimezone to logicalTypeOptions options for timestamp and time.
  • Changes to Quality
    • Add a maintained library of commonly used quality metrics rowCount, nullValues, invalidValues, duplicateValues, and missingValues.
    • Add schedule and scheduler to data quality properties.
  • Changes to SLA:
    • Add optional description field to SLA entries for human-readable context.
  • Changes to Support Channels:
    • Change url field to be optional.
    • Add customProperties field for additional metadata.
    • Add notifications as an example for scope
    • Add googlechat as an example for tool
  • Changes to Servers:
    • AzureServer format not longer an enum of parquet, delta, json, csv, but rather a string with the same examples.
    • AzureServer delimiter not longer an enum of new_line, array, but rather a string with the same examples.
    • S3Server format not longer an enum of parquet, delta, json, csv, but rather a string with the same examples.
    • S3Server delimiter not longer an enum of new_line, array, but rather a string with the same examples.
    • SftpServer format not longer an enum of parquet, delta, json, csv, but rather a string with the same examples.
    • SftpServer delimiter not longer an enum of new_line, array, but rather a string with the same examples.
    • Added HiveServer with type hive.
    • Added ImpalaServer with type impala
    • Duckdb schema was expecting an integer, but should expect a string.
    • Added support for Actian Zen Server.
    • Added missing stream property to CustomServer.
  • Deprecations:
    • slaDefaultElement is deprecated, and will be removed in ODCS v4.0.0 (see RFC 21).
    • The team structure has evolved. Both are valid, however the ODCS v3.0.x structure is deprecated (see RFC 16).
  • Changes to custom properties and authoritative definitions:
    • Add description field to both customProperties and authoritativeDefinitions.

What's Changed

Read more

Contributors

simonharrer, tomdw, and 8 other contributors
Assets 2
Loading

Version 3.0.2

31 Mar 11:28
@pflooky pflooky
v3.0.2
009d4e8
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Version 3.0.2

What's Changed

New Contributors

Full Changelog: v3.0.1...v3.0.2

Contributors

andrewrjones, simonharrer, and 6 other contributors
Loading

Version 3.0.1

08 Jan 04:36
@pflooky pflooky
v3.0.1
f313988
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Version 3.0.1

Highlights

  • Added field authoritativeDefinitions into JSON schema
  • Added field description.customProperties into JSON schema
  • Added field description.authoritativeDefinitions into JSON schema
  • Added field role.customProperties into JSON schema
  • Updated status field to include examples
  • Updated authoritativeDefinitions description to be vendor agnostic
  • Updated tags description and included examples

What's Changed

New Contributors

Full Changelog: v3.0.0...v3.0.1

Contributors

simonharrer, dirkvandepoel, and 4 other contributors
Loading

Version 3.0.0

20 Oct 15:47
@pflooky pflooky
v3.0.0
51bffe2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Version 3.0.0

Highlights

  • New section: Support & communication channels.
  • New section: Servers.
  • Changes to fundamentals :
    • Rename uuid to id.
    • Add name.
    • Rename quantumName to dataProduct and make it optional.
    • Rename datasetDomain to domain (we avoid the dataset prefix).
    • Drop datasetKind (example: virtualDataset, was optional, have not seen any usage).
    • Drop userConsumptionMode (examples: analytical, was optional, already deprecated in v2.).
    • Drop sourceSystem (example: bigQuery, information will be encoded in servers).
    • Drop sourcePlatform (example: googleCloudPlatform, information will be encoded in servers).
    • Drop productSlackChannel (will move to support channels).
    • Drop productFeedbackUrl (will move to support channels).
    • Drop productDl (will move to support channels).
    • Drop username (credentials should not be stored in the data contract).
    • Drop password (credentials should not be stored in the data contract).
    • Drop driverVersion (will move to servers if needed).
    • Drop driver (will move to servers if needed).
    • Drop server (will move to servers if needed).
    • Drop project (BigQuery-specific, will move to servers).
    • Drop datasetName (BigQuery-specific, will move to servers).
    • Drop database (BigQuery-specific, will move to servers).
    • Drop schedulerAppName (not part of the contract).
  • Changes to Schema:
    • Major changes, check spec.
    • Adds support for non table formats, hierarchies, and arrays.
    • name is a new field
    • items is a new field
    • priorTableName is not supported anymore, if needed, consider a custom property.
    • table is not supported anymore, if needed, consider using name.
    • columns is now properties
    • dataGranularity is now dataGranularityDescription.
    • encryptedColumnNameis now encryptedName.
    • partitionStatus is now partitioned.
    • clusterStatus is not supported anymore, if needed, consider a custom property.
    • clusterKeyPosition is not supported anymore, if needed, consider a custom property.
    • sampleValues is now examples.
    • isNullable is now required.
    • isUnique is now unique.
    • isPrimaryKey is now primaryKey.
    • criticalDataElementStatus is now criticalDataElement.
    • clusterKeyPosition is not supported anymore, if needed, consider a custom property.
    • transformSourceTables is now transformSourceObjects
    • Restrict schema.*.logicalType to be one of string, date, number, integer, object, array, boolean.
    • Add schema.*.logicalTypeOptions.
  • Changes to Data Quality:
    • Significant changes have been applied to support more tools and use cases. Please review the new section.
    • If needed, templateName is a custom property.
    • toolName is obsolete, replaced by type=custom; engine: <engine name>.
    • scheduleCronExpression is replaced by schedule and scheduler. scheduleCronExpression: 0 20 * * * becomes schedule: 0 20 * * * and scheduler: cron.
  • Pricing:
    • No changes.
  • Changes to Team (fka stakeholders):
    • Replaces stakeholders. Content stays the same.
  • Changes to Role:
    • Added description
    • Changed access is not required anymore
  • Security:
    • No changes.
  • Changes to SLA:
    • Starting with v3, the schema is not purely tables and columns, hence minor modifications: columns are now elements.
    • slaDefaultColumn is now slaDefaultElement.
    • column is now element.
    • Explicit reference to Data QoS.
  • Changes to custom and other properties:
    • systemInstance is not supported anymore, if needed, consider a custom property.

What's Changed

New Contributors

Full Changelog: v2.2.2...v3.0.0

Contributors

simonharrer, dirkvandepoel, and 5 other contributors
Loading

Version 2.2.2

23 May 02:48
@pflooky pflooky
v2.2.2
25a0698
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Version 2.2.2

Highlights

  • Changedataset.description data type from array to string
  • Change dataset.column.isPrimaryKey data type from string to boolean
  • Change price.priceAmount data type from string to number
  • Change slaProperties.value data type from string to oneOf[string, number]
  • Change slaProperties.valueExt data type from string to oneOf[string, number]
  • Update examples to adhere to JSON schema
  • Full example from README directs to full-example.yaml
  • Add in mkdocs for creating documentation website

What's Changed

New Contributors

Full Changelog: v2.2.1...v2.2.2

Contributors

jgperrin, caladogan, and 2 other contributors
Loading

Version 2.2.1

18 Dec 14:01
@jgperrin jgperrin
v2.2.1
edd82c1
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Version 2.2.1

Highlights:

  • Additional optional details for columns.
  • Clarification of authoritative definitions.
  • JSON Schema for validating contracts.

v2.2.1 is a non-breaking update of v2.2.0.

See CHANGELOG for a list of changes.

Loading

Version 2.2.0

27 Jul 15:00
@jgperrin jgperrin
v2.2.0
8ad4d16
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Version 2.2.0 
Merge pull request #3 from AIDAUserGroup/dev

v2.2 release
Loading