Skip to content

chore: Refactor cache usage and code flow #60

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
adwk67 wants to merge 14 commits into
base: main
Choose a base branch
from
Draft

chore: Refactor cache usage and code flow #60

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
b83e70b
wip: refactoring
adwk67 Dec 9, 2025
a3cf6e8
refer to class in readme
adwk67 Dec 10, 2025
201717b
improved listener resolution
adwk67 Dec 10, 2025
a55e6be
added logging
adwk67 Dec 11, 2025
563edb1
more logging changes
adwk67 Dec 11, 2025
e96fbd7
code cleanup
adwk67 Dec 11, 2025
e2fb728
query listener version from crd
adwk67 Dec 11, 2025
a725e28
delay resolution of listener version
adwk67 Dec 11, 2025
e53831e
catch exception on listener fetch
adwk67 Dec 12, 2025
812033e
use full resource name
adwk67 Dec 12, 2025
0a53a65
more logging
adwk67 Dec 12, 2025
ba2dd44
bumped kubernetes client version, removed unused dependency
adwk67 Dec 12, 2025
46e9b3f
restored okio lib
adwk67 Dec 12, 2025
8628192
revert version
adwk67 Dec 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -701,7 +701,7 @@ In Kubernetes, the most commonly used mechanism for topology awareness are label
The most prevalent example for this is the node label [topology.kubernetes.io/zone](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone) which often refers to availability zones in cloud providers or similar things.

The purpose of this tool is to feed information from Kubernetes into the HDFS rack awareness functionality.
In order to do this, it implements the Hadoop interface `org.apache.hadoop.net.DNSToSwitchMapping` which then allows this tool to be configured on the NameNode via the parameter `net.topology.node.switch.mapping.impl`.
In order to do this, `tech.stackable.hadoop.StackableTopologyProvider` implements the Hadoop interface `org.apache.hadoop.net.DNSToSwitchMapping` which then allows this tool to be configured on the NameNode via the parameter `net.topology.node.switch.mapping.impl`.

The topology provider watches all HDFS pods deployed by Stackable and Kubernetes nodes and keeps an in memory cache of the current state of these objects.
From this state store the tool can then calculate rack IDs for nodes that HDFS asks for without needing to talk to the api-server and incurring an extra network round-trip.
Expand Down
Loading
Loading