Skip to content

Commit 3a239c9

Browse files
vincentsaragovincentsarago
committed
init
0 parents  commit 3a239c9

30 files changed

+1538
-0
lines changed

‎.circleci/config.yml‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
version: 2
2+
common: &common
3+
machine:
4+
docker_layer_caching: false
5+
working_directory: ~/docker-lambda
6+
steps:
7+
- checkout
8+
- attach_workspace:
9+
at: ~/docker-lambda
10+
- run: echo "$DOCKER_PASS" | docker login --username $DOCKER_USER --password-stdin
11+
- run:
12+
name: create gdal docker image
13+
command: |
14+
docker build -f base/gdal${GDAL_VERSION}/Dockerfile -t gdal${GDAL_VERSION} .
15+
docker run --name lambda --volume $(pwd)/:/local -itd gdal${GDAL_VERSION} bash
16+
test=$(docker exec -it lambda bash -c 'cd /local/tests && sh tests.sh')
17+
docker cp ./scripts/create-lambda-layer.sh lambda:/tmp/create-lambda-layer.sh
18+
docker exec -it lambda bash -c '/tmp/create-lambda-layer.sh'
19+
docker stop lambda
20+
docker rm lambda
21+
if [[ "$test" =~ "OK" ]]; then echo "PASSED"; else exit 1; fi
22+
23+
- run:
24+
name: create gdal layer
25+
command: |
26+
docker run --name lambda --volume $(pwd)/:/local -itd gdal${GDAL_VERSION} bash
27+
docker cp ./scripts/create-lambda-layer.sh lambda:/tmp/create-lambda-layer.sh
28+
docker exec -it lambda bash -c '/tmp/create-lambda-layer.sh'
29+
docker stop lambda
30+
docker rm lambda
31+
32+
- run:
33+
name: push image
34+
command: |
35+
if [[ "$CIRCLE_BRANCH" == "master" ]]; then
36+
docker tag gdal${GDAL_VERSION} lambgeo/lambda:gdal${GDAL_VERSION}
37+
docker push lambgeo/lambda:gdal${GDAL_VERSION}
38+
fi
39+
40+
- run:
41+
name: create and push python images
42+
command: |
43+
for PYTHON_VERSION in ${PYTHON_VERSIONS}; do
44+
if [[ "$CIRCLE_BRANCH" == "master" ]]; then
45+
docker build --build-arg PYTHON_VERSION=${PYTHON_VERSION} --build-arg GDAL_VERSION=${GDAL_VERSION} -f base/python/Dockerfile -t lambgeo/lambda:gdal${GDAL_VERSION}-py${PYTHON_VERSION} .
46+
docker push lambgeo/lambda:gdal${GDAL_VERSION}-py${PYTHON_VERSION}
47+
fi
48+
done
49+
50+
- persist_to_workspace:
51+
root: .
52+
paths:
53+
- gdal*.zip
54+
55+
jobs:
56+
"gdal-3.1":
57+
<<: *common
58+
environment:
59+
- GDAL_VERSION=3.1
60+
- PYTHON_VERSIONS=3.7 3.8
61+
62+
"gdal-3.0":
63+
<<: *common
64+
environment:
65+
- GDAL_VERSION=3.0
66+
- PYTHON_VERSIONS=3.7 3.8
67+
68+
"gdal-2.4":
69+
<<: *common
70+
environment:
71+
- GDAL_VERSION=2.4
72+
- PYTHON_VERSIONS=3.7 3.8
73+
74+
"deploy":
75+
docker:
76+
- image: circleci/python:3.7.2
77+
environment:
78+
- GDAL_VERSIONS=2.4 3.0 3.1
79+
working_directory: ~/amazonlinux
80+
steps:
81+
- checkout
82+
- attach_workspace:
83+
at: ~/amazonlinux
84+
- run:
85+
name: install boto3
86+
command: sudo pip install boto3 click
87+
- run:
88+
name: Deploy Layer
89+
command: |
90+
for GDAL_VERSION in ${GDAL_VERSIONS}; do
91+
python scripts/deploy-layer.py $GDAL_VERSION
92+
done
93+
94+
workflows:
95+
version: 2
96+
build_and_push:
97+
jobs:
98+
- "gdal-3.1"
99+
- "gdal-3.0"
100+
- "gdal-2.4"
101+
- deploy:
102+
requires:
103+
- "gdal-3.1"
104+
- "gdal-3.0"
105+
- "gdal-2.4"
106+
filters:
107+
branches:
108+
only:
109+
- master

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
*.zip
2+
__pycache__/

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2020 LambGeo
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
# GDAL based docker-lambda
2+
3+
[![CircleCI](https://circleci.com/gh/lambgeo/docker-lambda.svg?style=svg)](https://circleci.com/gh/lambgeo/docker-lambda)
4+
5+
Create an **AWS lambda** like docker images and lambda layer with GDAL.
6+
7+
8+
## Images
9+
### GDAL - Based on lambci/lambda-base:build
10+
- 3.1.0 (April. 2020) - **lambgeo/lambda:gdal3.1** - Pre-release
11+
- 3.0.4 (April. 2020) - **lambgeo/lambda:gdal3.0**
12+
- 2.4.4 (April. 2020) - **lambgeo/lambda:gdal2.4**
13+
14+
## Lambda Layers
15+
16+
We are publishing each gdal version as lambda layer on the AWS infrastructure.
17+
Each layer are available for all runtimes.
18+
19+
**gdal${version}**
20+
21+
arn: **arn:aws:lambda:{REGION}:524387336408:layer:gdal${version}**
22+
23+
[Full list of version and ARN](https://github.com/RemotePixel/amazonlinux/blob/master/arns.json)
24+
25+
#### versions:
26+
27+
gdal | version | size (Mb)| unzipped size (Mb)
28+
---| ---| ---| ---
29+
3.1 | 1| 46.4| 136.9
30+
3.0 | 1| 46.4| 136.9
31+
2.4 | 1| 37.7| 126.2
32+
33+
#### Regions
34+
- ap-northeast-1
35+
- ap-northeast-2
36+
- ap-south-1
37+
- ap-southeast-1
38+
- ap-southeast-2
39+
- ca-central-1
40+
- eu-central-1
41+
- eu-north-1
42+
- eu-west-1
43+
- eu-west-2
44+
- eu-west-3
45+
- sa-east-1
46+
- us-east-1
47+
- us-east-2
48+
- us-west-1
49+
- us-west-2
50+
51+
### Python - Based on lambci/lambda:build-python*
52+
53+
Those images are here to help for the creation of lambda package or lambda layer.
54+
55+
- **3.1**
56+
- **lambgeo/lambda:gdal3.1-py3.7**
57+
- **lambgeo/lambda:gdal3.1-py3.8**
58+
59+
- **3.0**
60+
- **lambgeo/lambda:gdal3.0-py3.7**
61+
- **lambgeo/lambda:gdal3.0-py3.8**
62+
63+
- **2.4**
64+
- **lambgeo/lambda:gdal2.4-py3.7**
65+
- **lambgeo/lambda:gdal2.4-py3.8**
66+
67+
Content: GDAL Libs and python with numpy and cython
68+
69+
Checkout [/base/python/Dockerfile](/base/python/Dockerfile) to see how to create other runtime supported images.
70+
71+
# Create a Python Lambda package
72+
73+
You can use the docker container to either build a full package (you provide all the libraries)
74+
or adapt for the use of AWS Lambda layer.
75+
76+
## 1. Create full package (see [/examples/package](/examples/package))
77+
This is like we used to do before (with remotepixel/amazonlinux-gdal images)
78+
79+
- dockerfile
80+
```Dockerfile
81+
FROM lambgeo/lambda:gdal3.0-py3.7
82+
83+
ENV PACKAGE_PREFIX=/var/task
84+
85+
COPY handler.py ${PACKAGE_PREFIX}/handler.py
86+
RUN pip install numpy rasterio mercantile --no-binary :all: -t ${PACKAGE_PREFIX}/
87+
```
88+
89+
- package.sh
90+
```bash
91+
#!/bin/bash
92+
echo "-----------------------"
93+
echo "Creating lambda package"
94+
echo "-----------------------"
95+
echo "Remove lambda python packages"
96+
rm -rdf $PACKAGE_PREFIX/boto3/ \
97+
&& rm -rdf $PACKAGE_PREFIX/botocore/ \
98+
&& rm -rdf $PACKAGE_PREFIX/docutils/ \
99+
&& rm -rdf $PACKAGE_PREFIX/dateutil/ \
100+
&& rm -rdf $PACKAGE_PREFIX/jmespath/ \
101+
&& rm -rdf $PACKAGE_PREFIX/s3transfer/ \
102+
&& rm -rdf $PACKAGE_PREFIX/numpy/doc/
103+
104+
echo "Strip shared libraries"
105+
cd $PREFIX && find lib -name \*.so\* -exec strip {} \;
106+
107+
echo "Create archive"
108+
cd $PACKAGE_PREFIX && zip -r9q /tmp/package.zip *
109+
cd $PREFIX && zip -r9q --symlinks /tmp/package.zip lib/*.so* share bin
110+
cp /tmp/package.zip /local/package.zip
111+
```
112+
113+
- commands
114+
```bash
115+
docker build --tag package:latest .
116+
docker run --name lambda -w /var/task --volume $(shell pwd)/:/local -itd package:latest bash
117+
docker exec -it lambda bash '/local/package.sh'
118+
docker stop lambda
119+
docker rm lambda
120+
```
121+
122+
## 2. Use Lambda Layer (see [/examples/layer](/examples/layer))
123+
124+
- dockerfile
125+
126+
Here we install rasterio and we add our handler method.
127+
The final package structure should be
128+
129+
```
130+
package/
131+
|___ handler.py
132+
|___ rasterio/
133+
```
134+
135+
```Dockerfile
136+
FROM lambgeo/lambda:gdal3.0-py3.7
137+
138+
# Basically we don't want to replicated existant modules found in the layer ($PYTHONPATH)
139+
# So we use the $PYTHONUSERBASE trick to set the output directory
140+
ENV PYTHONUSERBASE=/var/task
141+
142+
# Create a package
143+
COPY handler.py $PYTHONUSERBASE/handler.py
144+
RUN pip install --user rasterio --no-binary rasterio
145+
```
146+
- layer.sh
147+
```bash
148+
# We move all the package to the root directory
149+
version=$(python -c 'import sys; print(f"{sys.version_info[0]}.{sys.version_info[1]}")')
150+
PYPATH=${PYTHONUSERBASE}/lib/python${version}/site-packages/
151+
mv ${PYPATH}/* ${PYTHONUSERBASE}/
152+
rm -rf ${PYTHONUSERBASE}/lib
153+
154+
echo "Create archive"
155+
cd $PYTHONUSERBASE && zip -r9q /tmp/layer.zip *
156+
157+
cp /tmp/layer.zip /local/layer.zip
158+
```
159+
- commands
160+
```bash
161+
docker build --tag package:latest .
162+
docker run --name lambda -w /var/task --volume $(shell pwd)/:/local -itd package:latest bash
163+
docker exec -it lambda bash '/local/layer.sh'
164+
docker stop lambda
165+
docker rm lambda
166+
167+
```
168+
169+
# AWS Lambda Layer architecture
170+
171+
The AWS Layer created within this repository have this architecture:
172+
173+
```
174+
layer.zip
175+
|
176+
|___ bin/ # Binaries
177+
|___ lib/ # Shared libraries (GDAL, PROJ, GEOS...)
178+
|___ share/ # GDAL/PROJ data directories
179+
```
180+
181+
You may want to extent this layer by adding runtime specific code
182+
183+
```
184+
layer.zip
185+
|
186+
...
187+
|___ python/ # Runtime
188+
|__ rasterio/
189+
|__ rio_tiler/
190+
|__ handler.py
191+
```
192+
193+
# AWS Lambda config
194+
- When using lambgeo gdal layer
195+
196+
- **GDAL_DATA:** /opt/share/gdal
197+
- **PROJ_LIB:** /opt/share/proj
198+
199+
- When creating full package
200+
- **GDAL_DATA:** /var/task/share/gdal
201+
- **PROJ_LIB:** /var/task/share/proj
202+
203+
### Other variable for optimal config
204+
- **GDAL_CACHEMAX:** 512
205+
- **VSI_CACHE:** TRUE
206+
- **VSI_CACHE_SIZE:** 536870912
207+
- **CPL_TMPDIR:** "/tmp"
208+
- **GDAL_HTTP_MERGE_CONSECUTIVE_RANGES:** YES
209+
- **GDAL_HTTP_MULTIPLEX:** YES
210+
- **GDAL_HTTP_VERSION:** 2
211+
- **GDAL_DISABLE_READDIR_ON_OPEN:** "EMPTY_DIR"
212+
- **CPL_VSIL_CURL_ALLOWED_EXTENSIONS:** ".tif"

0 commit comments

Comments
 (0)