Hi team, i want to use AWSGlueETL 4.0.0 on EMR 6.9.1 via Spark Scala.
As your code said that we should mark the libs as "provided".
The local testing looks good, the library can be downloaded and success run.
However when trying to run on the EMR, the logs said like this:
"java.lang.NoClassDefFoundError: com/amazonaws/services/glue/GlueContext"
Does EMR 6.9.1 indeed not provide the AWSGlueETL? if it is provided, could you help to fix that?
Thanks.
SBT: 1.5.3
Scala: 2.12.15
Spark: 3.3.0
AWSGlueETL: 4.0.0
ThisBuild / resolvers ++= {
Seq("AWS Glue Maven" at "https://aws-glue-etl-artifacts.s3.amazonaws.com/release/")}
val javaGlue = "com.amazonaws" % "AWSGlueETL" % "4.0.0" % "provided"