Skip to content

Commit cba6d82

Browse files
chriskerychriskery
authored
Features/add demos (#9)
Added more Hadoop usage examples
1 parent e94e0e7 commit cba6d82

File tree

78 files changed

+3404
-661
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

78 files changed

+3404
-661
lines changed

‎.github/workflows/docker-publish.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Publish Hadoop Cluster Controller Images
1+
name: Publish Hadoop Controller Images
22

33
# This workflow uses actions that are not certified by GitHub.
44
# They are provided by a third-party and are governed by

‎.github/workflows/test-go.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,18 +12,18 @@ jobs:
1212
GOPATH: ${{ github.workspace }}/go
1313
defaults:
1414
run:
15-
working-directory: ${{ env.GOPATH }}/src/github.com/chriskery/hadoop-cluster-operator
15+
working-directory: ${{ env.GOPATH }}/src/github.com/chriskery/hadoop-operator
1616

1717
steps:
1818
- name: Check out code
1919
uses: actions/checkout@v3
2020
with:
21-
path: ${{ env.GOPATH }}/src/github.com/chriskery/hadoop-cluster-operator
21+
path: ${{ env.GOPATH }}/src/github.com/chriskery/hadoop-operator
2222

2323
- name: Setup Go
2424
uses: actions/setup-go@v3
2525
with:
26-
go-version-file: ${{ env.GOPATH }}/src/github.com/chriskery/hadoop-cluster-operator/go.mod
26+
go-version-file: ${{ env.GOPATH }}/src/github.com/chriskery/hadoop-operator/go.mod
2727

2828
- name: Check Go modules
2929
run: |

‎.github/workflows/unittest.yaml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ jobs:
1212
GOPATH: ${{ github.workspace }}/go
1313
defaults:
1414
run:
15-
working-directory: ${{ env.GOPATH }}/src/github.com/chriskery/hadoop-cluster-operator
15+
working-directory: ${{ env.GOPATH }}/src/github.com/chriskery/hadoop-operator
1616

1717
strategy:
1818
fail-fast: false
@@ -23,12 +23,12 @@ jobs:
2323
- name: Check out code
2424
uses: actions/checkout@v3
2525
with:
26-
path: ${{ env.GOPATH }}/src/github.com/chriskery/hadoop-cluster-operator
26+
path: ${{ env.GOPATH }}/src/github.com/chriskery/hadoop-operator
2727

2828
- name: Setup Go
2929
uses: actions/setup-go@v3
3030
with:
31-
go-version-file: ${{ env.GOPATH }}/src/github.com/chriskery/hadoop-cluster-operator/go.mod
31+
go-version-file: ${{ env.GOPATH }}/src/github.com/chriskery/hadoop-operator/go.mod
3232

3333
- name: Run Go test
3434
run: |
@@ -44,4 +44,4 @@ jobs:
4444
uses: shogo82148/actions-goveralls@v1
4545
with:
4646
path-to-profile: cover.out
47-
working-directory: ${{ env.GOPATH }}/src/github.com/chriskery/hadoop-cluster-operator
47+
working-directory: ${{ env.GOPATH }}/src/github.com/chriskery/hadoop-operator

‎.gitignore‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,4 +32,5 @@ vendor
3232

3333
.run
3434

35-
./kubecluster-controller
35+
target
36+

‎PROJECT‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ domain: kubecluster.org
66
layout:
77
- go.kubebuilder.io/v3
88
projectName: hadoop-cluster-operator
9-
repo: github.com/chriskery/hadoop-cluster-operator
9+
repo: github.com/chriskery/hadoop-operator
1010
resources:
1111
- api:
1212
crdVersion: v1
@@ -15,7 +15,7 @@ resources:
1515
domain: kubecluster.org
1616
group: hadoopcluster
1717
kind: HadoopCluster
18-
path: github.com/chriskery/hadoop-cluster-operator/api/v1alpha1
18+
path: github.com/chriskery/hadoop-operator/api/v1alpha1
1919
version: v1alpha1
2020
webhooks:
2121
defaulting: true

‎README.md‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,21 @@
1-
# HadoopCluster Operator
1+
# Hadoop Operator
22

3-
[![Go Report](https://goreportcard.com/badge/github.com/chriskery/hadoop-cluster-operator)](https:/goreportcard.com/report/github.com/chriskery/hadoop-cluster-operator)
4-
[![Build Status](https://github.com/chriskery/hadoop-cluster-operator/actions/workflows/test-go.yml/badge.svg?branch=master)](https://github.com/chriskery/hadoop-cluster-operator/actions/workflows/test-go.yaml?branch=master)
3+
[![Go Report](https://goreportcard.com/badge/github.com/chriskery/hadoop-operator)](https:/goreportcard.com/report/github.com/chriskery/hadoop-operator)
4+
[![Build Status](https://github.com/chriskery/hadoop-operator/actions/workflows/test-go.yml/badge.svg?branch=master)](https://github.com/chriskery/hadoop-operator/actions/workflows/test-go.yaml?branch=master)
55
[![Coverage Status](https://coveralls.io/repos/github/chriskery/hadoop-cluster-operator/badge.svg?branch=master)](https://coveralls.io/github/chriskery/hadoop-cluster-operator?branch=master)
6-
[![Version](https://img.shields.io/github/release/chriskery/hadoop-cluster-operator.svg)](https:/github.com/chriskery/hadoop-cluster-operator/releases)
6+
[![Version](https://img.shields.io/github/release/chriskery/hadoop-cluster-operator.svg)](https:/github.com/chriskery/hadoop-operator/releases)
77

88
## Overview
99

1010
[Hadoop](https://hadoop.apache.org/) is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, providing a highly-available service on top of a cluster of computers, each of which may be prone to failures
1111

12-
The hadoop-cluster Operator manages hadoop clusters deployed to Kubernetes.
12+
The hadoop Operator manages hadoop clusters deployed to Kubernetes, and provides a cloud-native way to submit Hadoop Yarn jobs.
13+
14+
![Architecture Diagram](docs/images/architecture.png)
1315

1416
## Install the Operator
1517
### Cert Manager
16-
hadoop cluster operator relies on [cert-manager](https://cert-manager.io/docs/installation/), you must have cert-manager installed and configured in your Kubernetes cluster. cert-manager is a Kubernetes add-on that automates the management of certificates and is a requirement for this Operator to function correctly.
18+
hadoop operator relies on [cert-manager](https://cert-manager.io/docs/installation/), you must have cert-manager installed and configured in your Kubernetes cluster. cert-manager is a Kubernetes add-on that automates the management of certificates and is a requirement for this Operator to function correctly.
1719

1820
Installing cert-manager:
1921

@@ -25,7 +27,7 @@ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/
2527

2628
### Master Branch
2729
```shell
28-
kubectl apply -k github.com/chriskery/hadoop-cluster-operator/manifests/default
30+
kubectl apply -k github.com/chriskery/hadoop-operator/manifests/default
2931
```
3032

3133
## Quick Start

‎demos/hdfsdemo/pom.xml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
2+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
3+
<modelVersion>4.0.0</modelVersion>
4+
5+
<groupId>kubecluster.org</groupId>
6+
<artifactId>hdfsdemo</artifactId>
7+
<version>1.0-SNAPSHOT</version>
8+
<packaging>jar</packaging>
9+
10+
<name>hdfsdemo</name>
11+
<url>http://maven.apache.org</url>
12+
13+
<properties>
14+
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
15+
</properties>
16+
17+
<dependencies>
18+
<dependency>
19+
<groupId>org.apache.hadoop</groupId>
20+
<artifactId>hadoop-client</artifactId>
21+
<version>3.3.6</version>
22+
</dependency>
23+
</dependencies>
24+
25+
<build>
26+
<plugins>
27+
<plugin>
28+
<artifactId>maven-assembly-plugin</artifactId>
29+
<configuration>
30+
<archive>
31+
<manifest>
32+
<mainClass>kubecluster.org.HdfsDemo</mainClass>
33+
</manifest>
34+
<manifestEntries>
35+
<Class-Path>.</Class-Path>
36+
</manifestEntries>
37+
</archive>
38+
<descriptorRefs>
39+
<descriptorRef>jar-with-dependencies</descriptorRef>
40+
</descriptorRefs>
41+
</configuration>
42+
<executions>
43+
<execution>
44+
<id>make-assembly</id>
45+
<phase>package</phase>
46+
<goals>
47+
<goal>single</goal>
48+
</goals>
49+
</execution>
50+
</executions>
51+
</plugin>
52+
</plugins>
53+
</build>
54+
</project>

‎demos/hdfsdemo/src/main/java/kubecluster/org/HdfsDemo.java‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
package kubecluster.org;
2+
3+
4+
import org.apache.hadoop.conf.Configuration;
5+
import org.apache.hadoop.fs.FileSystem;
6+
import org.apache.hadoop.fs.Path;
7+
8+
public class HdfsDemo {
9+
10+
public static void main(String[] args) {
11+
// Create a Hadoop configuration object, which will load the configuration files from the classpath
12+
Configuration conf = new Configuration();
13+
14+
// Optionally, set the HDFS namenode address directly
15+
// conf.set("fs.defaultFS", "hdfs://namenode_host:8020");
16+
17+
try {
18+
// Get the FileSystem object for HDFS
19+
FileSystem fs = FileSystem.get(conf);
20+
21+
// Define a path in HDFS to check for existence
22+
Path path = new Path("/path/to/test");
23+
24+
// Check if the path exists in HDFS
25+
if (fs.exists(path)) {
26+
System.out.println("File or directory exists: " + path);
27+
return;
28+
}
29+
System.out.println("File or directory does not exist: " + path);
30+
31+
// Try to create a new file to test the write operation
32+
boolean fileCreated = fs.createNewFile(path);
33+
if (fileCreated) {
34+
System.out.println("File created successfully: " + path);
35+
} else {
36+
System.out.println("Failed to create file: " + path);
37+
}
38+
} catch (Exception e) {
39+
// Print the stack trace to the console in case of an exception
40+
e.printStackTrace();
41+
}
42+
}
43+
}

0 commit comments

Comments
 (0)