Skip to content

Commit a156a31

Browse files
authored
Added a new option to use named profile
Requested in the issue: #96
1 parent d002033 commit a156a31

File tree

1 file changed

+11
-0
lines changed

1 file changed

+11
-0
lines changed

utilities/Spark_UI/README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,17 @@ If you prefer local access (not to have EC2 instance for Apache Spark history se
1818
1919
2020
#### Start the Spark history server
21+
**Using AWS named profile **
22+
23+
1. Run commands shown below
24+
- Set **LOG_DIR** by replacing **s3a://path_to_eventlog** with your event log directory
25+
- Set **PROFILE_NAME** with your AWS named profile
26+
```
27+
$ LOG_DIR="s3a://path_to_eventlog/"
28+
$ PROFILE_NAME="profile_name"
29+
$ docker run -itd -v ~/.aws:/root/.aws -e AWS_PROFILE=$PROFILE_NAME -e SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=$LOG_DIR -Dspark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain" -p 18080:18080 glue/sparkui:latest "/opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer"
30+
```
31+
2132
**Using a pair of AWS access key and secret key**
2233
1. Run commands shown below
2334
- Set **LOG_DIR** by replacing **s3a://path_to_eventlog** with your event log directory

0 commit comments

Comments
 (0)