v0.9.3

• Added documentation for exclude flag (#265). A new exclude flag has been added to the configuration file for our lab tool, allowing users to specify a path to exclude from formatting during lab execution. This release also includes corrections to grammatical errors in the descriptions of existing flags related to catalog and database settings, such as updating seperated to "separate". Additionally, the flag descriptions for publish and open-browser have been updated for clarification: publish now clearly controls whether the dashboard is published after creation, while open-browser controls whether the dashboard is opened in a web browser. These changes are aimed at improving user experience and ease of use for our lab tool.
• Fixed dataclass field type in _row_to_sql (#266). In this release, we have addressed an issue related to #257 by fixing the dataclass field type in the _row_to_sql method of the backends.py file. Additionally, we have made updates to the _schema_for method to use a new _field_type class method. This change resolves a rare problem where the field.type is a string instead of a type and ensures compatibility with a pull request from an external repository (databrickslabs/ucx#2526). The new _field_type method attempts to load the type from __builtins__ if it's a string and logs a warning if it fails. The _row_to_sql method now consistently uses the _field_type method to get the field type. This ensures that the library functions seamlessly and consistently, avoiding any potential issues in the future.

Contributors: @ericvergnaud

v0.9.2

• Make hatch a prerequisite (#259). In this commit, Eric Vergnaud has introduced a change to make the installation of hatch version 1.9.4 a prerequisite for the project to avoid errors related to pip command recognition. The Makefile has been updated to handle the installation of hatch automatically, and the hatch env create command is now used instead of pip install hatch==1.7.0. This change ensures that the development environment is consistent and reliable by maintaining the correct version of hatch and automatically handling its installation. Additionally, the .venv/bin/python and dev targets have been updated accordingly to reflect these changes. This commit also formats all files using the make dev fmt command, which helps maintain consistent code formatting throughout the project.
• add support for exclusions in fmt command (#263). In this release, we have added support for exclusions to the fmt command in the 'databricks/labs/lsql/cli.py' module. This feature allows users to specify a list of directories or files to exclude while formatting SQL files, which is particularly useful when verifying SQL notebooks in ucx. The fmt command now accepts a new optional parameter 'exclude', which accepts an iterable of strings that specify the relative paths to exclude. Any sql_file that is a descendant of any exclusion is skipped during formatting. The exclusions are implemented by converting the relative paths into Path objects. This change addresses the issue where single line comments are converted into inlined comments, causing misinterpretation. The added unit test is manually verified, and this pull request fixes issue #261. This feature was authored and co-authored by Eric Vergnaud.

Contributors: @ericvergnaud

v0.9.1

• Fixed dataclass field types (#257). This PR introduces a workaround to a Python bug affecting the dataclasses.fields() function, which sometimes returns field types as string type names instead of types. This can cause the ORM to malfunction. The workaround involves checking if the returned f.type is a string, and if so, converting it to a type by looking it up in the __builtins__ dictionary. This change is global and affects the _schema_for function in the backends.py file, which is responsible for creating a schema for a given dataclass, taking into account any necessary type conversions. This change ensures consistent and accurate type handling in the face of the Python bug, improving the reliability of our ORM.
• Fixed missing EOL when formatting SQL files (#260). In this release, we have addressed an issue related to the inconsistent addition of end-of-line (EOL) characters in formatted SQL files. The QueryTile.format() method has been updated to ensure that an EOL character is always added, except when the input query already ends with a newline. This change enhances the reliability of the SQL formatting functionality, making the output format more predictable and improving the overall user experience. The new implementation is demonstrated in the test_query_format_preserves_eol() test case, and existing test cases have been updated to check for the presence of EOL characters, further ensuring consistent and correct formatting.
• Fixed normalize case input in cli (#258). In this release, we have updated the fmt command in the cli.py file to allow users to specify whether they want to normalize the case of SQL files when formatting. The normalize_case parameter now defaults to the string "true" and checks if it is in the STRING_AFFIRMATIVES list to determine whether to normalize the case of SQL files. Additionally, we have introduced a new optional normalize_case parameter in the format method of the dashboards.py file in the Databricks CLI, which normalizes the identifiers in the query to lower case when set to True. We have also added support for a new normalize_case parameter in the QueryTile.format() method, which prevents the automatic normalization of string input to uppercase when set to False. This change allows for more flexibility in handling string input and ensures that the input string is preserved as-is. These updates improve the functionality and usability of the open-source library, providing more control to users over formatting and handling of string input.

Contributors: @ericvergnaud, @nfx, @JCZuurmond

v0.9.0

• Added design for filter file (#251). A new feature has been added to enable the creation of filters for multiple widgets in a dashboard using a .filter.json file. This file allows users to specify columns to be filtered, the filter type, title, description, order, and a unique ID for each filter. Both the column and columns flags are supported, with the former taking a single string and the latter taking a list of strings. The filter type can be set to a drop-down menu or another type as desired. The .filter.json file schema also supports optional title and description strings, as well as order and ID flags. An example of a .filter.json file is provided in the commit message. Additionally, the dashboard.yml file documentation has been updated to include information on how to use the new .filter.json file.
• adding normalize-case option to databricks labs lsql fmt cmd (#254). In this open-source library release, the databricks labs lsql tool's fmt command now supports a new flag, normalize-case. This flag allows users to control the normalization of query text to lowercase, providing more flexibility when formatting SQL queries. By default, query text is still normalized to lowercase, but users can now prevent this behavior by setting the normalize-case flag to False. This change addresses an issue where some queries are case sensitive, such as those using map field keys in UCX dashboards. Additionally, a new parameter normalize_case has been added to the format method in the dashboards.py file, with updated method documentation. A new test function, test_query_formats_no_normalize(), has also been included to ensure consistent formatter behavior.

Contributors: @JCZuurmond, @HariGS-DB

v0.8.0

• Removed deploy_dashboard method (#240). In this release, the deploy_dashboard method has been removed from the dashboards.py file and the legacy deployment method has been deprecated. The deploy_dashboard method was previously used to deploy a dashboard to a workspace, but it has been replaced with the create method of the lakeview attribute of the WorkspaceClient object. Additionally, the test_dashboards_creates_dashboard_via_legacy_method method has been removed. A new test has been added to ensure that the deploy_dashboard method is no longer being used, utilizing the deprecated_call function from pytest to verify that calling the method raises a deprecation warning. This change simplifies the code and improves the overall design of the system, resolving issue #232. The _with_better_names method and create_dashboard method remain unchanged.
• Skip test that fails due to insufficient permission to create schema (#248). A new test function, test_dashboards_creates_dashboard_with_replace_database, has been added to the open-source library, but it is currently marked to be skipped due to missing permissions to create a schema. This function creates an instance of the Dashboards class with the ws parameter, creates a dashboard using the make_dashboard function, and performs various actions using the created dashboard, as well as functions such as tmp_path and sql_backend. This test function aims to ensure that the Dashboards class functions as expected when creating a dashboard with a replaced database. Once the necessary permissions for creating a schema are acquired, this test function can be enabled for further testing and validation.
• Updates to use the Databricks Python sdk 0.30.0 (#247). In this release, we have updated the project to use Databricks Python SDK version 0.30.0. This update includes changes to the execute and fetch_value functions, which now use the new StatementResponse type instead of ExecuteStatementResponse. A conditional import statement has been added to maintain compatibility with both Databricks SDK versions 0.30.0 and below. The execute function now raises TimeoutError when the specified timeout is greater than 50 seconds and the statement execution hasn't finished. Additionally, the fetch_value function has been updated to handle the case when the execute function returns None. The unit test file test_backends.py has also been updated to reflect these changes, with multiple test functions now using the StatementResponse class instead of ExecuteStatementResponse. These changes improve the system's compatibility with the latest version of the Databricks SDK, ensuring that the core functionality of the SDK continues to work as expected.

Contributors: @JCZuurmond

v0.7.5

• Fixed missing widget name suffixes (#243). In this release, we have addressed an issue related to missing widget name suffixes (#243) by adding a _widget suffix to the name of the widget object in the dashboards.py file. This change ensures consistency between the widget name and the id of the query, facilitating user understanding of the relationship between the two. A new method, _get_query_widget_spec, has also been added, although its specific functionality requires further investigation. Additionally, the unit tests in the tests/unit/test_dashboards.py file have been updated to check for the presence of the _widget suffix in widget names, ensuring that the tests accurately reflect the desired behavior. These changes improve the consistency of dashboard widget naming, thus benefiting software engineers utilizing or extending the project's widget-ordering functionalities.

Contributors: @nfx

v0.7.4

• Fixed dataset/widget name uniqueness requirement that was preventing dashboards being deployed (#241). A fix has been implemented to address a uniqueness requirement issue with the dataset/widget name that was preventing dashboard deployment. A new widget instance is now created with a unique name, generated by appending _widget to the metadata ID, in the get_layouts method. This ensures that multiple widgets with the same ID but different content can exist in a single dashboard, thereby meeting the name uniqueness requirement. In the save_to_folder method, the widget name is modified by removing the _widget suffix before writing the textbox specification to a markdown file, maintaining consistency between the widget ID and file name. These changes are localized to the get_layouts and save_to_folder methods, and no new methods have been added. The existing functionality related to the creation, validation, and saving of dashboard layouts remains unaltered.

Contributors: @nfx

v0.7.3

• Added publish flag to Dashboards.create_dashboard (#233). In this release, we have added a publish flag to the Dashboards.create_dashboard method, allowing users to publish the dashboard upon creation, thereby resolving issue #219. This flag is included in the labs.yml file with a description of its functionality. Additionally, the no-open flag's description has been updated to specify that it prevents the dashboard from opening in the browser after creation. The create_dashboard function in the cli.py and dashboards.py files has been updated to include the new publish flag, allowing for more flexibility in how users create and manage their dashboards. The Dashboards.create_dashboard method now calls the WorkspaceClient.lakeview.publish method when the publish flag is set to True, which publishes the created dashboard. This behavior is covered in the updated tests for the method.
• Fixed boolean cli flags (#235). In this release, we have improved the handling of command-line interface (CLI) flags in the databricks labs command. Specifically, we have addressed the limitation that pure boolean flags are not supported. Now, when using boolean flags, the user will be prompted to confirm with a y or 'yes'. We have modified the create_dashboard command to accept string inputs for the publish and no_open flags, which are then converted to boolean values for internal use. Additionally, we have introduced a new open-browser command, which will open the dashboard in the browser after creating when set to y or 'yes'. These changes have been tested manually to ensure correct behavior. This improvement provides a more flexible input experience and better handling of boolean flags in the CLI command for software engineers using the open-source library.
• Fixed format breaks widget (#238). In this release, we've made significant changes to the 'databricks/labs/lsql' directory's 'dashboards.py' file to address formatting breaks in the widget that could occur with Call to Action (CTA) presence in a query. These changes include the addition of new class variables, including _SQL_DIALECT and _DIALECT, and their integration into existing methods such as _parse_header, validate, format, _get_abstract_syntax_tree, and replace_catalog_and_database_in_query. Furthermore, we have developed new methods for creating and deleting schemas and getting the current test purge time. We have also implemented new integration tests to demonstrate the fix for the formatting issue and added new test cases for the query handler's header-splitting functionality, query formatting, and CTE handling. These enhancements improve the library's handling of SQL queries and query tiles in the context of dashboard creation, ensuring proper parsing, formatting, and metadata extraction for a wide range of query scenarios.
• Fixed replace database when catalog or database is None (#237). In this release, we have addressed an issue where system tables disappeared in ucx dashboards when replacing the placeholder database. To rectify this, we have developed a new method, replace_catalog_and_database_in_query, in the dashboards.py file's replace_database function. This method checks if the catalog or database in a query match the ones to be replaced and replaces them with new ones, ensuring that system tables are not lost during the replacement process. Additionally, we have introduced new unit tests in test_dashboards.py to verify that queries are correctly transformed when replacing the database or catalog in the query. These tests include various scenarios, using two parametrized test functions, to ensure the correct functioning of the feature. This change provides a more robust and reliable dashboard display when replacing the placeholder database in the system.

Contributors: @JCZuurmond

v0.7.2

• Fixed dashboard deployment/creation (#230). The recent changes to our open-source library address issues related to dashboard deployment and creation, enhancing their reliability and consistency. The deploy_dashboard function has been deprecated in favor of the more accurate create_dashboard function, which now includes a publish flag. A validate method has been added to the Tile, MarkdownTile, and QueryTile classes to raise an error if the dashboard is invalid. The test_dashboards.py file has been updated to reflect these changes. These enhancements address issues #222, #229, and partially resolve #220. The commit includes an image of a dashboard created through the deprecated deploy_dashboard method. These improvements ensure better dashboard creation, validation, and deployment, while also maintaining backward compatibility through the deprecation of deploy_dashboard.

Contributors: @JCZuurmond

v0.7.1

• Bump sigstore/gh-action-sigstore-python from 2.1.1 to 3.0.0 (#224). In version 3.0.0 of sigstore/gh-action-sigstore-python, several changes, additions, and removals have been implemented. Notably, certain settings such as fulcio-url, rekor-url, ctfe, and rekor-root-pubkey have been removed. Additionally, the output settings signature, certificate, and bundle have also been removed. The inputs are now parsed according to POSIX shell lexing rules for better consistency. The release-signing-artifacts setting no longer causes a hard error when used under the incorrect event. Furthermore, various deprecations present in sigstore-python's 2.x series have been resolved. The default suffix has been changed from .sigstore to .sigstore.json, in line with Sigstore's client specification. The release-signing-artifacts setting now defaults to true. This version also includes several bug fixes and improvements to support CI runners that use PEP 668 to constrain global package prefixes.
• Use default factory to create Tile._position (#226). In this change, the default value creation for the _position field in various classes including Tile, MarkdownTile, TableTile, and CounterTile has been updated. Previously, a new Position object was explicitly created for the default value. With this update, the default_factory argument of the dataclasses.field function is now used to create a new Position object. This change is made in anticipation of the Python 3.11 release, which modifies the field default mutability check behavior. By utilizing the default_factory approach, we ensure that a new Position object is generated during each instance creation, rather than reusing a single default instance. This guarantees the immutability of default values and aligns with best practices for forward-compatibility with future Python versions. It is important to note that this modification does not affect the functionality of the classes but enhances their initialization process.

Dependency updates:

• Bump sigstore/gh-action-sigstore-python from 2.1.1 to 3.0.0 (#224).

Contributors: @JCZuurmond, @dependabot[bot], @nfx