Merge pull request #112 from dlt-hub/feat/cli_fixes

remove init command word from cli, add better exceptions for broken specs

Author: sh-rp

Makefile

@@ -35,14 +35,14 @@ test-slow: update-rest-api
 
 # dev helpers
 create-pokemon-pipeline:
-	poetry run dlt-init-openapi init pokemon --url https://raw.githubusercontent.com/cliffano/pokeapi-clients/ec9a2707ef2a85f41b747d8df013e272ef650ec5/specification/pokeapi.yml --no-interactive
+	poetry run dlt-init-openapi pokemon --url https://raw.githubusercontent.com/cliffano/pokeapi-clients/ec9a2707ef2a85f41b747d8df013e272ef650ec5/specification/pokeapi.yml --no-interactive
 
 create-pokemon-pipeline-interactive:
-	poetry run dlt-init-openapi init pokemon --url https://raw.githubusercontent.com/cliffano/pokeapi-clients/ec9a2707ef2a85f41b747d8df013e272ef650ec5/specification/pokeapi.yml
+	poetry run dlt-init-openapi pokemon --url https://raw.githubusercontent.com/cliffano/pokeapi-clients/ec9a2707ef2a85f41b747d8df013e272ef650ec5/specification/pokeapi.yml
 
 # e2e test helpers
 create-e2e-pokemon-pipeline:
-	poetry run dlt-init-openapi init pokemon --path tests/cases/e2e_specs/pokeapi.yml --global-limit 2 --no-interactive
+	poetry run dlt-init-openapi pokemon --path tests/cases/e2e_specs/pokeapi.yml --global-limit 2 --no-interactive
 
 run-pokemon-pipeline:
 	cd pokemon_pipeline && poetry run python pokemon_pipeline.py

README.md

@@ -44,11 +44,11 @@ $ dlt-init-openapi --version
 Let's create an example pipeline from the [PokeAPI spec](https://raw.githubusercontent.com/cliffano/pokeapi-clients/ec9a2707ef2a85f41b747d8df013e272ef650ec5/specification/pokeapi.yml). You can point to any other OpenAPI Spec instead if you like.
 
 ```console
-# 1.a. Run the generator with the dlt-init-openapi init command:
-$ dlt-init-openapi init pokemon --url https://raw.githubusercontent.com/cliffano/pokeapi-clients/ec9a2707ef2a85f41b747d8df013e272ef650ec5/specification/pokeapi.yml
+# 1.a. Run the generator with an url:
+$ dlt-init-openapi pokemon --url https://raw.githubusercontent.com/cliffano/pokeapi-clients/ec9a2707ef2a85f41b747d8df013e272ef650ec5/specification/pokeapi.yml
 
 # 1.b. If you have a local file, you can use the --path flag:
-$ dlt-init-openapi init pokemon --path ./my_specs/pokeapi.yml
+$ dlt-init-openapi pokemon --path ./my_specs/pokeapi.yml
 
 # 2. You can now pick the endpoints you need from the popup
 
@@ -84,33 +84,33 @@ When you run the `init` command above, the following files will be generated:
 ## CLI commands
 
 ```console
-$ dlt-init [OPTIONS] COMMAND [ARGS]...
+$ dlt-init-openapi [OPTIONS] <source_name> [ARGS]
 # example:
-$ dlt-init init pokemon --path ./path/to/my_spec.yml
+$ dlt-init-openapi pokemon --path ./path/to/my_spec.yml
 ```
 
 **Options**:
 
-- `--version`: Print the version and exit [default: False]
+- `--version`: Print the version and exit
 - `--help`: Show this message and exit.
 
 **Commands**:
 
 - `init`: Generate a new `dlt` `rest_api` `source`
 
-### `dlt-init-openapi init`
+### `dlt-init-openapi`
 
 Generate a new `dlt` `rest_api` `source`
 
 **Usage**:
 
 ```console
-$ dlt-init-openapi init pokemon --path ./path/to/my_spec.yml
+$ dlt-init-openapi pokemon --path ./path/to/my_spec.yml
 ```
 
 **Options**:
 
-- `--url TEXT`: A url to read the OpenAPI JSON or YAML file from
+- `--url URL`: A url to read the OpenAPI JSON or YAML file from
 - `--path PATH`: A path to read the OpenAPI JSON or YAML file from locally
 - `--output-path PATH`: A path to render the output to
 - `--config PATH`: Path to the config file to use (see below)
@@ -119,7 +119,7 @@ $ dlt-init-openapi init pokemon --path ./path/to/my_spec.yml
 - `--help`: Show this message and exit.
 
 ## Config options
-You can pass a path to a config file with the `--config PATH` argument. To see available config values, go to https://github.com/dlt-hub/dlt-init-openapi/blob/master/dlt_init_openapi/config.py and read the information below each field on the `Config` class.
+You can pass a path to a config file with the `--config PATH` argument. To see available config values, go to https://github.com/dlt-hub/dlt-init-openapi/blob/devel/dlt_init_openapi/config.py and read the information below each field on the `Config` class.
 
 The config file can be supplied as json or yaml dictionary. For example to change the package name, you can create a yaml file:
 
@@ -131,7 +131,7 @@ package_name: "other_package_name"
 And use it with the config argument:
 
 ```console
-$ dlt-init-openapi init pokemon --url ... --config config.yml
+$ dlt-init-openapi pokemon --url ... --config config.yml
 ```
 
 ## Telemetry

dlt_init_openapi/cli/__init__.py

@@ -9,6 +9,7 @@
 
 from dlt_init_openapi.cli.cli_endpoint_selection import questionary_endpoint_selection
 from dlt_init_openapi.config import Config
+from dlt_init_openapi.exceptions import DltOpenAPITerminalException
 from dlt_init_openapi.utils import update_rest_api
 
 app = typer.Typer(add_completion=False)
@@ -33,34 +34,24 @@ def _load_config(path: Optional[pathlib.Path], config: Any) -> Config:
     return c
 
 
-# noinspection PyUnusedLocal
-# pylint: disable=unused-argument
-@app.callback(name="dlt-init-openapi")
-def cli(
-    version: bool = typer.Option(False, "--version", callback=_print_version, help="Print the version and exit"),
-) -> None:
-    """Generate a Python client from an OpenAPI JSON document"""
-
-
-CONFIG_OPTION = typer.Option(None, "--config", help="Path to the config file to use")
-
-
 # pylint: disable=too-many-arguments
 @app.command()
 def init(
     source: str = typer.Argument(None, help="A name of data source for which to generate a pipeline"),
     url: Optional[str] = typer.Option(None, help="A URL to read the JSON/YAML spec from"),
     path: Optional[pathlib.Path] = typer.Option(None, help="A path to the JSON/YAML spec file"),
     output_path: Optional[pathlib.Path] = typer.Option(None, help="A path to render the output to."),
-    config_path: Optional[pathlib.Path] = CONFIG_OPTION,
+    config_path: Optional[pathlib.Path] = typer.Option(None, "--config", help="Path to the config file to use"),
     interactive: bool = typer.Option(True, help="Wether to select needed endpoints interactively"),
     loglevel: int = typer.Option(20, help="Set logging level for stdout output, defaults to 20 (INFO)"),
     global_limit: int = typer.Option(0, help="Set a global limit on the generated source"),
     update_rest_api_source: bool = typer.Option(
         False, help="Wether to update the locally cached rest_api verified source"
     ),
+    version: bool = typer.Option(False, "--version", callback=_print_version, help="Print the version and exit"),
 ) -> None:
-    """Generate a new OpenAPI Client library"""
+    """Generate a new dlt pipeline"""
+
     _init_command_wrapped(
         source=source,
         url=url,
@@ -101,31 +92,38 @@ def _init_command_wrapped(
         typer.secho("Provide either --url or --path, not both", fg=typer.colors.RED)
         raise typer.Exit(code=1)
 
-    # synch rest api
-    update_rest_api.update_rest_api(force=update_rest_api_source)
-
-    config = _load_config(
-        path=config_path,
-        config={
-            "project_name": source,
-            "package_name": source,
-            "output_path": output_path,
-            "endpoint_filter": questionary_endpoint_selection if interactive else None,
-            "global_limit": global_limit,
-        },
-    )
-
-    if config.project_dir.exists():
-        if not questionary.confirm(
-            f"Directory {config.project_dir} exists, do you want to continue and update the generated files? "
-            + "This will overwrite your changes in those files."
-        ).ask():
-            logger.warning("Exiting...")
-            exit(0)
-
-    create_new_client(
-        url=url,
-        path=path,
-        config=config,
-    )
-    logger.success("Pipeline created. Learn more at https://dlthub.com/docs. See you next time :)")
+    try:
+
+        # synch rest api
+        update_rest_api.update_rest_api(force=update_rest_api_source)
+
+        config = _load_config(
+            path=config_path,
+            config={
+                "project_name": source,
+                "package_name": source,
+                "output_path": output_path,
+                "endpoint_filter": questionary_endpoint_selection if interactive else None,
+                "global_limit": global_limit,
+            },
+        )
+
+        if config.project_dir.exists():
+            if not questionary.confirm(
+                f"Directory {config.project_dir} exists, do you want to continue and update the generated files? "
+                + "This will overwrite your changes in those files."
+            ).ask():
+                logger.warning("Exiting...")
+                exit(0)
+
+        create_new_client(
+            url=url,
+            path=path,
+            config=config,
+        )
+        logger.success("Pipeline created. Learn more at https://dlthub.com/docs. See you next time :)")
+
+    except DltOpenAPITerminalException as exc:
+        logger.error("Encountered terminal exception:")
+        logger.error(exc)
+        logger.info("Exiting...")

dlt_init_openapi/exceptions.py

@@ -1,2 +1,38 @@
 class DltOpenAPIException(Exception):
     pass
+
+
+class DltOpenAPITerminalException(DltOpenAPIException):
+    pass
+
+
+class DltOpenAPINot30Exception(DltOpenAPITerminalException):
+    def __init__(self, swagger_detected: bool = False) -> None:
+
+        swagger_helper = "If this is a Swagger/OpenAPI 2.0 or earlier spec, "
+        if swagger_detected:
+            swagger_helper = "It looks like this is a Swagger/OpenAPI 2.0 spec, "
+
+        convert_helper = (
+            "you can convert it to an openapi 3.0 spec by going to https://editor.swagger.io/, "
+            + "pasting your spec and selecting 'Edit' -> 'Convert to OpenAPI 3.0' from the Menu "
+            + "and then retry with the converted file."
+        )
+
+        super().__init__(
+            "The spec you selected does not appear to be an OpenAPI 3.0 spec. " + swagger_helper + convert_helper
+        )
+
+
+class DltInvalidSpecException(DltOpenAPITerminalException):
+    def __init__(self) -> None:
+
+        super().__init__(
+            "Could not validate selected spec, please provide a valid YAML or JSON OpenAPI 3.0 or higher spec."
+        )
+
+
+class DltUnparseableSpecException(DltOpenAPITerminalException):
+    def __init__(self) -> None:
+
+        super().__init__("Could not parse selected spec, please provide a valid YAML or JSON document.")

dlt_init_openapi/parser/openapi_parser.py

@@ -7,7 +7,7 @@
 from loguru import logger
 from yaml import BaseLoader
 
-from dlt_init_openapi.exceptions import DltOpenAPIException
+from dlt_init_openapi.exceptions import DltInvalidSpecException, DltOpenAPINot30Exception, DltUnparseableSpecException
 from dlt_init_openapi.parser.config import Config
 from dlt_init_openapi.parser.context import OpenapiContext
 from dlt_init_openapi.parser.endpoints import EndpointCollection
@@ -39,30 +39,17 @@ def parse(self, data: bytes) -> None:
         try:
             spec = osp.OpenAPI.parse_obj(self.spec_raw)
         except Exception as e:
-            raise DltOpenAPIException("Could not Validate spec:\n" + str(e)) from e
+            raise DltInvalidSpecException() from e
         logger.success("Spec validation successful")
 
         # check if this is openapi 3.0
-        convert_info = (
-            "you can convert it to an openapi 3.0 spec by going to https://editor.swagger.io/, "
-            + "pasting your spec and selecting 'Edit' -> 'Convert to OpenAPI 3.0' from the Menu "
-            + "and then retry with the converted file."
-        )
         swagger_version = self.spec_raw.get("swagger")
         if swagger_version:
-            logger.error(
-                "The spec you selected appears to be a Swagger/OpenAPI version 2 spec or older, " + convert_info
-            )
-            exit(0)
+            raise DltOpenAPINot30Exception(swagger_detected=True)
 
         openapi_version = self.spec_raw.get("openapi")
         if not openapi_version or not openapi_version.startswith("3"):
-            logger.error(
-                "The spec you selected does not appear to be an OpenAPI 3.0 spec. "
-                + "If this is a a Swagger/OpenAPI version 2 spec or older, "
-                + convert_info
-            )
-            exit(1)
+            raise DltOpenAPINot30Exception(swagger_detected=False)
 
         logger.info("Extracting openapi metadata")
         self.context = OpenapiContext(self.config, spec, self.spec_raw)
@@ -88,19 +75,21 @@ def parse(self, data: bytes) -> None:
 
     def _load_yaml_or_json(self, data: bytes) -> Dict[str, Any]:
         logger.info("Trying to parse spec as JSON")
-
-        data_size = sys.getsizeof(data)
-        if data_size > 1000000:
-            mb = round(data_size / 1000000)
-            logger.warning(f"Spec is around {mb} mb, so parsing might take a while.")
         try:
-            result = json.loads(data.decode())
-            logger.success("Parsed spec as JSON")
+            data_size = sys.getsizeof(data)
+            if data_size > 1000000:
+                mb = round(data_size / 1000000)
+                logger.warning(f"Spec is around {mb} mb, so parsing might take a while.")
+            try:
+                result = json.loads(data.decode())
+                logger.success("Parsed spec as JSON")
+                return result
+            except ValueError:
+                logger.info("No valid JSON found")
+                pass
+            logger.info("Trying to parse spec as YAML")
+            result = yaml.load(data, Loader=BaseLoader)
+            logger.success("Parsed spec as YAML")
             return result
-        except ValueError:
-            logger.info("No valid JSON found")
-            pass
-        logger.info("Trying to parse spec as YAML")
-        result = yaml.load(data, Loader=BaseLoader)
-        logger.success("Parsed spec as YAML")
-        return result
+        except Exception as exc:
+            raise DltUnparseableSpecException() from exc