k8s-ephemeral-storage-metrics

Helm Install

helm repo add k8s-ephemeral-storage-metrics https://jmcgrath207.github.io/k8s-ephemeral-storage-metrics/chart
helm repo update
helm upgrade --install my-deployment k8s-ephemeral-storage-metrics/k8s-ephemeral-storage-metrics

Values

Key Type Default Description
affinity object {}  
client_go_burst int 10 Maximum burst for throttle.
client_go_qps int 5 QPS indicates the maximum QPS to the master from this client.
containerSecurityContext.allowPrivilegeEscalation bool false  
containerSecurityContext.capabilities.drop[0] string "ALL"  
containerSecurityContext.privileged bool false  
containerSecurityContext.readOnlyRootFilesystem bool false  
containerSecurityContext.runAsNonRoot bool true  
deploy_labels object {} Set additional labels for the Deployment/Daemonset
deploy_type string "Deployment" Set as Deployment for single controller to query all nodes or Daemonset
dev object {"enabled":false,"grow":{"image":"ghcr.io/jmcgrath207/k8s-ephemeral-storage-grow-test:latest","imagePullPolicy":"IfNotPresent"},"shrink":{"image":"ghcr.io/jmcgrath207/k8s-ephemeral-storage-shrink-test:latest","imagePullPolicy":"IfNotPresent"}} For local development or testing that will deploy grow and shrink pods and debug service
fullnameOverride string "" Override the full name of the chart
image.imagePullPolicy string "IfNotPresent"  
image.imagePullSecrets list []  
image.repository string "ghcr.io/jmcgrath207/k8s-ephemeral-storage-metrics"  
image.tag string "1.21.3"  
interval int 15 Polling node rate for exporter
kubeconfig string "" Path to kubeconfig file; leave empty for in-cluster config
kubelet object {"insecure":false,"readOnlyPort":0,"scrape":false} Scrape metrics through kubelet instead of kube api
list_pods_with_cache bool false Use Kubernetes api server cache for pod list requests (reduces api server pressure at scale)
log_level string "info"  
max_node_concurrency int 10 Max number of concurrent query requests to the kubernetes API.
metrics object {"adjusted_polling_rate":false,"ephemeral_storage_container_limit_percentage":true,"ephemeral_storage_container_logs_usage":true,"ephemeral_storage_container_rootfs_usage":true,"ephemeral_storage_container_volume_limit_percentage":true,"ephemeral_storage_container_volume_usage":true,"ephemeral_storage_inodes":true,"ephemeral_storage_node_available":true,"ephemeral_storage_node_capacity":true,"ephemeral_storage_node_percentage":true,"ephemeral_storage_pod_usage":true,"port":9100,"scrape_miss_tolerance":2} Set metrics you want to enable
metrics.adjusted_polling_rate bool false Create the ephemeral_storage_adjusted_polling_rate metrics to report Adjusted Poll Rate in milliseconds. Typically used for testing.
metrics.ephemeral_storage_container_limit_percentage bool true Percentage of ephemeral storage used by a container in a pod
metrics.ephemeral_storage_container_logs_usage bool true Current logs bytes used/available/capacity for a container in a pod
metrics.ephemeral_storage_container_rootfs_usage bool true Current rootfs bytes used/available/capacity for a container in a pod
metrics.ephemeral_storage_container_volume_limit_percentage bool true Percentage of ephemeral storage used by a container’s volume in a pod
metrics.ephemeral_storage_container_volume_usage bool true Current ephemeral storage used by a container’s volume in a pod
metrics.ephemeral_storage_inodes bool true Current ephemeral inode usage of pod
metrics.ephemeral_storage_node_available bool true Available ephemeral storage for a node
metrics.ephemeral_storage_node_capacity bool true Capacity of ephemeral storage for a node
metrics.ephemeral_storage_node_percentage bool true Percentage of ephemeral storage used on a node
metrics.ephemeral_storage_pod_usage bool true Current ephemeral byte usage of pod
metrics.port int 9100 Adjust the metric port as needed (default 9100)
metrics.scrape_miss_tolerance int 2 Number of consecutive scrapes a pod can be missing from the stats summary before its metrics are evicted
nameOverride string "" Override the name of the chart
nodeSelector object {}  
node_label_selector string "" Label selector to filter watched nodes in Deployment mode (e.g. type=virtual-kubelet)
podAnnotations object {}  
podSecurityContext.runAsNonRoot bool true  
podSecurityContext.seccompProfile.type string "RuntimeDefault"  
pod_labels object {} Set additional labels for the Pods
pprof bool false Enable Pprof
priorityClassName string nil  
probes object {"liveness":{"failureThreshold":10,"initialDelaySeconds":10,"path":"/metrics","periodSeconds":10,"successThreshold":1,"timeoutSeconds":30},"readiness":{"failureThreshold":10,"path":"/health","periodSeconds":10,"successThreshold":1,"timeoutSeconds":1}} Liveness and Readiness probe configuration
probes.liveness object {"failureThreshold":10,"initialDelaySeconds":10,"path":"/metrics","periodSeconds":10,"successThreshold":1,"timeoutSeconds":30} Liveness probe configuration
probes.liveness.failureThreshold int 10 Number of consecutive failures required to consider the container as not ready
probes.liveness.initialDelaySeconds int 10 Delay before the first probe is initiated
probes.liveness.path string "/metrics" HTTP path used by the liveness probe
probes.liveness.periodSeconds int 10 How often to perform the probe
probes.liveness.successThreshold int 1 Minimum consecutive successes for the probe to be considered successful after having failed
probes.liveness.timeoutSeconds int 30 Number of seconds after which the probe times out
probes.readiness object {"failureThreshold":10,"path":"/health","periodSeconds":10,"successThreshold":1,"timeoutSeconds":1} Readiness probe configuration
probes.readiness.failureThreshold int 10 Number of consecutive failures required to consider the container as not ready
probes.readiness.path string "/health" HTTP path used by the readiness probe
probes.readiness.periodSeconds int 10 How often to perform the probe
probes.readiness.successThreshold int 1 Minimum consecutive successes for the probe to be considered successful after having failed
probes.readiness.timeoutSeconds int 1 Number of seconds after which the probe times out
prometheus.additionalLabels object {} Add labels to the PrometheusRule.Spec
prometheus.enable bool true  
prometheus.release string "kube-prometheus-stack"  
prometheus.rules.enable bool false Create PrometheusRules firing alerts when out of ephemeral storage
prometheus.rules.labels object {"severity":"warning"} What additional labels to set on alerts
prometheus.rules.predictFilledHours int 12 How many hours in the future to predict filling up of a volume
prometheus.rules.predictMinCurrentUsage float 33.3 What percentage of limit must be used right now to predict filling up of a volume
rbac object {"create":true} RBAC configuration
resources object {} Resource requests and limits for the container
revisionHistoryLimit int 10 Revision history limit for the Deployment
serviceAccount object {"create":true,"name":null} Service Account configuration
serviceAnnotations object {} Annotations to add to the metrics Service
serviceMonitor object {"additionalLabels":{},"enable":true,"metricRelabelings":[],"podTargetLabels":[],"relabelings":[],"targetLabels":[]} Configure the Service Monitor
serviceMonitor.additionalLabels object {} Add labels to the ServiceMonitor.Spec
serviceMonitor.metricRelabelings list [] Set metricRelabelings as per https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.RelabelConfig
serviceMonitor.podTargetLabels list [] Set podTargetLabels as per https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.ServiceMonitorSpec
serviceMonitor.relabelings list [] Set relabelings as per https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.RelabelConfig
serviceMonitor.targetLabels list [] Set targetLabels as per https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.ServiceMonitorSpec
tolerations list []  

Prometheus alert rules

To prevent from multiple kind of alerts being fired for a single container or emptyDir volume when both prometheus.enable and prometheus.rules.enable are on, add the following inhibition rules to your Alert Manager config:

- source_matchers:
    - alertname="EphemeralStorageVolumeFilledUp"
  target_matchers:
    - severity="warning"
    - alertname="EphemeralStorageVolumeFillingUp"
  equal:
    - pod_namespace
    - pod_name
    - volume_name
- source_matchers:
    - alertname="ContainerEphemeralStorageUsageAtLimit"
  target_matchers:
    - severity="warning"
    - alertname="ContainerEphemeralStorageUsageReachingLimit"
  equal:
    - pod_namespace
    - pod_name
    - exported_container

Prerequisites

For local development / testing, install:

bin/{ginkgo,crane,helm-docs,gosec,govulncheck} are auto-installed by the Makefile on first use.

ENV modes

scripts/deploy.sh keys behavior off ENV={local|debug|e2e|e2e-debug|test}. The make deploy_* targets set this for you:

Mode Command Behavior
local (default) make deploy_local Build → helm install → port-forward :9100 → tail logs
debug make deploy_debug Delve build → port-forward :30002 → connect dlv
e2e make deploy_e2e Build → helm install → ginkgo -v -r tests/e2e (~15-25 min)
e2e-debug make deploy_e2e_debug Build → helm install → hold for manual ginkgo/debug
test make deploy_test Helm install with chart/test-values.yaml overrides

Development tooling

Task Command
Format make fmt
Vet make vet
Security scan make gosec
Vulnerability scan make govulncheck
Helm render check (8 k8s versions) make test-helm-render
Regenerate docs (after values.yaml change) make helm-docs
Unit tests (~2s, no cluster) make test-unit

Run make fmt vet test-unit test-helm-render before opening a PR.

Contribute

Prereqs: Docker, minikube, helm, kubectl, Go 1.26.5.

Fork + branch from master.

Unit tests (no cluster)

make test-unit

Full e2e (~15-25 min)

make minikube_new_docker    # fresh minikube: calico CNI, registry addon, 3900MB
make deploy_e2e             # build 3 images, helm install, ginkgo tests

Local dev (no e2e)

make deploy_local           # build + helm install + port-forward :9100 + tail logs

Local dev with Delve

make deploy_debug           # port-forward :30002

Connect localhost:30002 with delve or your IDE.

Debug e2e

make deploy_e2e_debug       # build + helm install, then holds

Then run ginkgo manually against deployment_test.go.

Pre-PR gate

make fmt vet test-unit test-helm-render

If chart/values.yaml changed, also run make helm-docs and commit the regenerated README.md + chart/README.md.

kind alternative

scripts/create_kind.sh (no Makefile wrapper) brings up a kind cluster. e2e will fail because tests/e2e/deployment_test.go hardcodes node_name="minikube" in regexes. Use kind for dev iteration only.

Troubleshooting

Symptom Fix
Port conflict on :9100 / :30002 / :6060 sudo ss -aK '( sport = :PORT )' — trap_func cleans on normal exit only
“registry endpoint is not healthy” during minikube_new_docker create-minikube.sh retries every 5s; if stuck >2 min, minikube delete && make minikube_new_docker
e2e hang on “Waiting for k8s-ephemeral-storage-metrics pod” Minikube needs 3900MB memory floor. Or image push failed — check minikube registry
e2e hang on “Waiting for grow-test pod” grow-test image not pushed; check crane push output in deploy.sh
ginkgo panics on run Makefile pin out of sync with go.mod — rm bin/ginkgo && make ginkgo (pin must match github.com/onsi/ginkgo/v2 in go.mod, currently v2.32.0)
shrink-test e2e flakes Shrinking reflects slowly from Node API (up to 5 min). Test waits for currentPodSize ≥ 11MB first

License

This project is licensed under the MIT License. See the LICENSE file for more details.

This site is open source. Improve this page.