Skip to content

Mark AWSGlueETL 4.0.0 as "provided" using SBT on EMR Release label 6.9.1, however GlueContext class not found during runtime #171

@aganisatria

Description

@aganisatria

Hi team, i want to use AWSGlueETL 4.0.0 on EMR 6.9.1 via Spark Scala.

As your code said that we should mark the libs as "provided".
The local testing looks good, the library can be downloaded and success run.
However when trying to run on the EMR, the logs said like this:
"java.lang.NoClassDefFoundError: com/amazonaws/services/glue/GlueContext"

Does EMR 6.9.1 indeed not provide the AWSGlueETL? if it is provided, could you help to fix that?

Thanks.

SBT: 1.5.3
Scala: 2.12.15
Spark: 3.3.0
AWSGlueETL: 4.0.0

ThisBuild / resolvers ++= {
  Seq("AWS Glue Maven" at "https://aws-glue-etl-artifacts.s3.amazonaws.com/release/")}

val javaGlue = "com.amazonaws" % "AWSGlueETL" % "4.0.0" % "provided"

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions