Skip to content

Conversation

@vamshikolanu
Copy link
Collaborator

@vamshikolanu vamshikolanu commented Oct 9, 2023

Describe your changes

Currently inside list_relations_without_caching() macro, we are calling get_relation_type for each and every relation. Impala doesn't have direct way to fetch all the relationships along with the table type in one query. This was causing issues where the schemas had 1000s of tables in them.

As part of this change, fetching the table type dynamically as part of the getter method. Although it isn't a standard interface provided by DBT, solving it this way for impala will reduce the number of get_relation_type() metadata calls dramatically.

Internal Jira ticket number or external issue link:

https://jira.cloudera.com/browse/DBT-744

Testing procedure/screenshots(if appropriate):

Checklist before requesting a review

  • I have performed a self-review of my code
  • I have formatted my added/modified code to follow pep-8 standards
  • I have checked suggestions from python linter to make sure code is of good quality.

@vamshikolanu vamshikolanu changed the title impala__list_relations_without_caching Optimize list_relations_without_caching() macro to avoid calling get_relationship() on every object. Oct 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants