Skip to content

[ETL] Optimize schemaMigration to be updated only if there are changes #119

@himeshr

Description

@himeshr

Need

As part of every org/orgGroup ETL run, we invoke SchemaMetadata "Migrate" operation, which performs update of the SchemaMetadata for each table/ column / index metadata of the org/orgGroup, every single time.

Due to this, there exists following undesired effects

  • Migrate operation costs almost 30 seconds of run-time for each org.
  • Applies needless load on Avni database due to no change update operations

Recommendation (needs further analysis and prioritzation)

  • Compute diffs that include Create / Update as well as Delete changes in schemaMetadata
  • Only if there are changes, then perform the following snippet of code
public class SchemaMigrationService {
     public Organisation migrate(Organisation organisation) {

....

        schemaMetadataRepository.applyChanges(changes);
        organisation.applyNewSchema(newSchemaMetadata);
        schemaMetadataRepository.save(organisation.getSchemaMetadata());
....
    }
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    Focus Items

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions