File tree Expand file tree Collapse file tree 1 file changed +11
-0
lines changed
Expand file tree Collapse file tree 1 file changed +11
-0
lines changed Original file line number Diff line number Diff line change @@ -18,6 +18,17 @@ If you prefer local access (not to have EC2 instance for Apache Spark history se
1818
1919
2020#### Start the Spark history server
21+ **Using AWS named profile **
22+
23+ 1. Run commands shown below
24+ - Set **LOG_DIR** by replacing **s3a://path_to_eventlog** with your event log directory
25+ - Set **PROFILE_NAME** with your AWS named profile
26+ ```
27+ $ LOG_DIR="s3a://path_to_eventlog/"
28+ $ PROFILE_NAME="profile_name"
29+ $ docker run -itd -v ~/.aws:/root/.aws -e AWS_PROFILE=$PROFILE_NAME -e SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=$LOG_DIR -Dspark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain" -p 18080:18080 glue/sparkui:latest "/opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer"
30+ ```
31+
2132**Using a pair of AWS access key and secret key**
22331. Run commands shown below
2334 - Set **LOG_DIR** by replacing **s3a://path_to_eventlog** with your event log directory
You can’t perform that action at this time.
0 commit comments