Added new checks for JSON validation

[cornzyblack]

Changes

Added new checks:

• is_valid_json to check whether the values in the input column are valid JSON strings.
• has_json_keys to check whether the values in the input column contain specific keys in the outermost JSON object.
• has_valid_json_schema to check whether the values in the specified column, which contain JSON strings, conform to the expected schema. This check is not strict. Extra fields in the JSON that are not defined in the schema are ignored.

Linked issues

Resolves #595

Tests

• manually tested
• added unit tests
• added integration tests
• added end-to-end tests
• added performance tests

[Copilot]

Pull Request Overview

Adds row-level JSON validation checks and integrates them into examples and tests.

• Introduces is_valid_json and has_json_keys row checks.
• Updates YAML examples, reference docs, and integration/unit tests to cover the new checks.

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/databricks/labs/dqx/check_funcs.py	Adds JSON validation/check functions; core logic for new checks.
tests/unit/test_build_rules.py	Extends metadata conversion tests to include new JSON checks.
tests/integration/test_apply_checks.py	Adds col_json_str to test schemas and values; exercises new checks in streaming and class-based tests.
tests/resources/all_row_checks.yaml	Includes is_valid_json check in the “all row checks” YAML.
src/databricks/labs/dqx/llm/resources/yaml_checks_examples.yml	Adds examples for is_valid_json and has_json_keys.
docs/dqx/docs/reference/quality_checks.mdx	Documents new checks and shows usage examples.

Comments suppressed due to low confidence (1)

docs/dqx/docs/reference/quality_checks.mdx:1

• Both examples use the same name 'col_json_str_has_json_keys', which is confusing and may collide in practice. Use distinct, descriptive names (e.g., 'col_json_str_has_no_json_key1' and 'col_json_str_has_no_json_key1_key2').

---

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[cornzyblack]

Updates
The current implementation of has_valid_json_schema is not strict but permissive. Take, for example, row 4 has an additional field c but still passes.

Making it strict will require UDFs for especially nested JSON

import pyspark.sql.functions as F

from databricks.labs.dqx.rule import DQRowRule
from pyspark.sql import types


from databricks.sdk import WorkspaceClient
from databricks.labs.dqx.engine import DQEngine
from pyspark.sql import functions as F
from databricks.labs.dqx import check_funcs

ws = WorkspaceClient()
dq_engine = DQEngine(ws)

json_schema = "STRUCT<a: BIGINT, b: BIGINT>"

checks = [

    DQRowRule(
        criticality="error",
        check_func=check_funcs.has_valid_json_schema,
        column="json",
        check_func_kwargs={"schema": json_schema},
    ),
    DQRowRule(
        criticality="error",
        check_func=check_funcs.has_valid_json_schema,
        column="json",
        check_func_kwargs={"schema": json_schema,},
    ),
]


df = spark.createDataFrame(
    [
        {"json": """{ "a" : 1 }"""},
        {"json": """{ "a" : 1, "b": 2}"""},
        {"json": """{ "a" : 1, "b": null}"""},
        {"json": """{ "a" : 1, "b": 1023455,  "c": null}}"""},
    ]
)

checked_df = dq_engine.apply_checks(df, checks)
display(checked_df)

will result in

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.

[mwojtyczka]

Mostly looking good, minor comments

[mwojtyczka]

Please put some protection, to limit the recursion calls.

[mwojtyczka]

Please also add passing example with nested fields

[mwojtyczka]

Please also add one passing example with valid schema