# Kubernetes Fixes Leaked Volumes with HonorPVReclaimPolicy
> Kubernetes 1.33 ensures PV reclaim policies are honored even if PVs are deleted before PVCs, preventing storage leaks across CSI and in-tree drivers.

Canonical: https://blog.abhimanyu-saharan.com/posts/kubernetes-fixes-leaked-volumes-with-honorpvreclaimpolicy
Published: 2025-06-10
Last updated: 2025-06-09
Authors: Abhimanyu Saharan
Categories: Kubernetes v1.33, Kubernetes

**In Kubernetes 1.33, a long-standing issue that caused persistent volumes to leak under certain deletion sequences has finally been resolved.** With the graduation of the `HonorPVReclaimPolicy` feature to GA, Kubernetes now reliably honors the reclaim policy of Persistent Volumes (PVs)—regardless of whether the PV or its associated Persistent Volume Claim (PVC) is deleted first.

## The Problem: PVs Were Ignored, Volumes Were Leaked

Until now, reclaim policies like `Delete` were only honored **if the PVC was deleted before the PV**. In reverse order—**deleting the PV first**—the reclaim policy would silently fail, leading to **orphaned volumes** in your cloud provider or storage backend.

This behavior was deeply tied to how Kubernetes controllers interpreted object states and finalizers. If the PV disappeared from the API server before the reclaim operation was completed, the backing storage was never cleaned up.

## What’s Changing: Finalizers to the Rescue

The fix revolves around **finalizers**—metadata markers that prevent deletion of a Kubernetes object until specific conditions are met.

### For CSI Driver Volumes:

- **`external-provisioner.volume.kubernetes.io/finalizer`** is now added not just to new PVs during provisioning, but also to **existing PVs**.
- This prevents premature deletion from the API server.
- Once the external CSI provisioner confirms the backing volume is deleted, the finalizer is removed, and the PV is fully deleted.

### For In-Tree Plugin Volumes:

- A new finalizer **`kubernetes.io/pv-controller`** has been introduced.
- It's added to newly created and existing PVs managed by in-tree plugins.
- The finalizer ensures the PV is only removed after a successful plugin-level volume deletion.
- It is dynamically added/removed based on CSI migration status.

## How It Works Now

With the `HonorPVReclaimPolicy` feature gate enabled:

1. **Delete PV first:** A `deletionTimestamp` is added to the PV, triggering a sync loop.
2. **Delete PVC next:** Finalizers ensure the PV isn’t purged from the API server.
3. **Controllers coordinate deletion:** Either the external provisioner or in-tree controller performs the actual volume deletion.
4. **Finalizer is removed:** Only once the backing storage is deleted.
5. **PV is finally deleted.**

This applies to both **dynamically and statically provisioned** volumes—provided a PVC is attached.

## Real-World Benefit: No More Leaked Resources

This enhancement:

- **Closes a long-standing bug** where storage resources were silently orphaned.
- **Improves consistency**, aligning the behavior of PVs regardless of deletion order.
- **Reduces ops overhead**, especially in production clusters where manual cleanup was previously required.

## Enabling the Feature

To use this feature:

- Ensure the `HonorPVReclaimPolicy` feature gate is enabled on both `kube-controller-manager` and the external CSI provisioner.
- Supported since **v1.26 (alpha)**, **v1.31 (beta)**, and **v1.33 (GA)**.

```yaml
--feature-gates=HonorPVReclaimPolicy=true
```

## Metrics and Monitoring

You can confirm it’s working by inspecting the PV object:

- For CSI: `metadata.finalizers` should contain `external-provisioner.volume.kubernetes.io/finalizer`.
- For in-tree: `kubernetes.io/pv-controller` should be present.

Metrics to monitor:

- `persistentvolume_delete_failed_total`
- `persistentvolume_delete_duration_seconds`

## Edge Cases and Safeguards

- Statically provisioned PVs without PVCs **won’t get finalizers**.
- Finalizers may block deletion if CSI drivers or controllers are not correctly upgraded.
- Manual intervention may be required if you disable the feature and have lingering finalizers.

## Summary

The `HonorPVReclaimPolicy` feature provides a **definitive fix** for volume leak issues in Kubernetes. By leveraging finalizers and refining controller coordination, it ensures that the reclaim policy is reliably honored—**no matter the deletion order**.

This is a major step forward for storage lifecycle management in Kubernetes, closing gaps that have persisted for years.

## FAQ

### What problem does the HonorPVReclaimPolicy feature solve in Kubernetes?

It fixes an issue where Persistent Volume (PV) reclaim policies (like `Delete`) were ignored if the PV was deleted before its associated PVC, causing orphaned volumes in cloud storage or on-prem environments.

### How does Kubernetes now ensure that reclaim policies are honored?

Through the use of **finalizers**. These metadata entries prevent premature PV deletion.

- For **CSI volumes**, the `external-provisioner.volume.kubernetes.io/finalizer` ensures external deletion completes.
- For **in-tree plugins**, the new `kubernetes.io/pv-controller` finalizer coordinates deletion through the in-tree volume controller.

### What Kubernetes versions support this feature, and is it enabled by default?

- Introduced as **alpha** in v1.26
- **Beta** in v1.31**GA (stable)** in v1.33  
To use it, the `HonorPVReclaimPolicy` feature gate must be enabled on the **kube-controller-manager** and **external CSI provisioners**.

### Does this work for statically provisioned PVs without a PVC?

No. Finalizers are only applied to PVs **with a bound PVC**. Statically created PVs without an associated PVC will not get the finalizer and are not covered by this behavior.

### How can I confirm the feature is working in my cluster?

Inspect the PV's metadata:

- For CSI volumes: look for `external-provisioner.volume.kubernetes.io/finalizer`
- For in-tree volumes: check for `kubernetes.io/pv-controller`  
You can also monitor deletion behavior using metrics like:
- `persistentvolume_delete_failed_total`
- `persistentvolume_delete_duration_seconds`