Proxmox Backup Hardening: Immutability and Ransomware

A standard backup strategy is no longer enough when modern ransomware actively hunts hypervisor infrastructure. When attackers compromise a Proxmox VE (PVE) host, their playbook rarely stops at encrypting running virtual machine disks. They look for /etc/pve/storage.cfg, find connected NFS, SMB, or local directory storages, and purge existing VZDump backup archives (.vma.zst) before executing the final payload.

If your backup storage shares the same administrative domain or grants unrestricted delete permissions to the hypervisor, your recovery window closes immediately. A hypervisor root compromise becomes an absolute business catastrophe.

Proxmox Backup Server (PBS) solves this structural vulnerability when designed correctly. Unlike legacy backup scripts, PBS uses content-defined chunking, client-side encryption, and a decoupled user permission model. By structuring PBS as an independent security boundary with pull-based replication and immutable datastore configurations, you ensure that even if an attacker gains root access to every PVE node in your cluster, your historical backup snapshots remain unalterable and fully recoverable.

Why Local Hypervisor Backups Fail During Ransomware Attacks

The traditional method of backing up Proxmox virtual machines relies on local storage or network shares mounted directly onto the PVE host. In this model, the PVE node acts as a privileged writer. The hypervisor creates snapshots, packages virtual disks into compressed tarballs, and writes them to disk using the root user credentials.

This setup presents three fatal architectural weaknesses during an incident:

1. Symmetric Trust Vulnerability: The storage target trusts the PVE host completely. Any process running as root on PVE can issue deletion commands, format mount points, or overwrite archive files.
2. Synchronous Destruction: Threat actors who gain access via vulnerable guest escape exploits, unpatched hypervisor services, or leaked root SSH keys immediately discover the backup paths. For detailed post-mortem steps on hypervisor isolation, refer to our analysis of investigating hypervisor attacks from compromised VMs.
3. Absence of Cryptographic Verification: Traditional dump files sitting on an SMB share can be quietly encrypted in place. Unless a test restore runs daily, sysadmins discover the corruption only when attempting disaster recovery.

Securing virtual machine environments requires treating the backup repository as an adversarial zone. The backup server must never trust the hypervisor it protects.

Proxmox Backup Server Architecture: Deduplication and Chunk Hashing

To implement effective immutability and encryption, you must understand how PBS manages data under the hood. Traditional backup utilities store monolithic image files. PBS breaks data into variable-size chunks using content-defined chunking.

When a virtual machine backs up to PBS:

• The hypervisor streams raw virtual disk blocks through a rolling hashing algorithm.
• Data divides into discrete chunks averaging 4MB in size.
• Each chunk is hashed using SHA-256. The resulting hash forms the unique filename and directory path for that chunk within the .chunks folder of the datastore.
• Identical chunks generated across different backup cycles, different virtual machines, or similar operating system templates are stored only once.

Datastore Root (/mnt/datastore/backup)
├── .chunks/
│   ├── 0000/
│   │   └── 0000a1b2c3d4... (Chunk 4MB)
│   ├── ffff/
│   │   └── ffffe5f6a7b8... (Chunk 4MB)
└── vm/
    └── 100/
        └── 2026-09-11T04:00:00Z/
            ├── qemu-server.conf.blob
            ├── drive-scsi0.img.fidx
            └── manifest.json

The backup snapshot itself does not contain disk images. It contains small index files:

• .fidx (Fixed Index): Maps virtual machine disk block offsets to chunk SHA-256 hashes.
• .didx (Dynamic Index): Used for variable-size file-level container backups.
• manifest.json: Stores snapshot metadata, hardware configurations, and cryptographic signatures.

Because chunks are named after their SHA-256 digest, any attempt by ransomware to alter the content of a chunk changes its hash. The verification engine immediately detects the discrepancy, flags the corrupted chunk, and alerts administrators before damaged blocks propagate into subsequent restore points.

Enforcing Client-Side AES-GCM 256-Bit Encryption

In high-security environments, you must assume that the backup storage hardware or remote network could be monitored. PBS addresses this through client-side encryption.

With client-side encryption enabled:

• Encryption occurs on the PVE hypervisor inside RAM before data enters the network interface.
• Data encrypts using AES-256 in Galois/Counter Mode (GCM), providing both confidentiality and cryptographic integrity verification.
• The PBS storage node receives and stores only encrypted ciphertext. PBS administrators and unauthorized parties cannot inspect the virtual disks or read database records stored within the chunks.

Generating the Encryption Key

Generate the encryption key on your PVE node using the command line:

proxmox-backup-client key create /etc/pve/priv/storage/pbs-enc-key.enc

This command generates an encrypted key file alongside a critical paper key. The paper key is a human-readable, printable representation of the secret key:

proxmox-backup-client key paperkey /etc/pve/priv/storage/pbs-enc-key.enc --output-format text > /root/pbs-paperkey-backup.txt

Print this paper key and place it in a physical safe or secure enterprise password vault. If your PVE cluster burns down, you cannot restore a single byte from your encrypted PBS datastore without this private key.

Configuring the PVE Storage Backend

Register the encrypted PBS storage into PVE by referencing the newly generated key:

pvesm add pbs PBS-Offsite \
    --server pbs.internal.example.com \
    --datastore production-vms \
    --username backup-pve@pbs \
    --password-file /etc/pve/priv/storage/pbs-password.cred \
    --fingerprint 2a:4f:8c:12:3e:7b:90:54:11:cd:ef:43:55:aa:bb:cc:dd:ee:ff:00:11:22:33:44:55:66:77:88:99:aa:bb:cc \
    --encryption-key /etc/pve/priv/storage/pbs-enc-key.enc

Verify that the fingerprint matches the TLS certificate generated by your PBS host during installation. Fingerprint pinning protects against man-in-the-middle attacks across WAN connections. Maintaining patched hypervisor components is just as important, so review our guide on Proxmox CVE patching and kernel maintenance to protect your storage tokens.

The Pull-Based Sync Model: Air-Gapping Secondary Datastores

The biggest mistake teams make when deploying Proxmox Backup Server is configuring direct push access with administrative privileges.

If the credentials stored in /etc/pve/priv/storage/pbs-password.cred hold root access on PBS, a compromised PVE host can send API requests to delete snapshots, destroy datastores, or alter retention schedules.

To neutralize this attack vector, implement role separation and a pull-based synchronization pipeline.

[ Production PVE Cluster ]
       │
       │ (1) Push Backup (Restricted: Datastore.Backup only)
       ▼
[ Primary PBS Node (DMZ / Local Rack) ]
       ▲
       │ (2) Pull Sync Job (Initiated from Remote Target)
       │     PVE has zero network route or credentials to this node
[ Secondary Hardened PBS Node (Offsite / Isolated VLAN) ]
       │
       └── (3) Write-Once Immutable Snapshot Verification

Step 1: Restrict PVE Permissions on Primary PBS

Never let PVE connect as root@pam. Create a dedicated service user on the primary PBS instance:

proxmox-backup-manager user create backup-pve@pbs --comment "PVE Cluster Production Backup User"
proxmox-backup-manager user update backup-pve@pbs --password "YourStrongGeneratedSecretPassphrase"

Create an API token for automated operations:

proxmox-backup-manager user generate-token backup-pve@pbs pve-token

Assign the most restrictive Access Control List (ACL) role possible. The hypervisor needs to create chunks and write manifests, but it should never possess permissions to prune historical records or destroy namespaces:

proxmox-backup-manager acl update /datastore/production-vms DatastoreBackup --auth-id backup-pve@pbs!pve-token

Under this configuration, the DatastoreBackup role allows writing new snapshots. If an attacker gains full control of PVE and attempts to prune old backups via the PBS API, the request returns HTTP 403 Forbidden.

Step 2: Establish a Secondary Pull-Based PBS Replica

To achieve true air-gap protection, deploy a second PBS server in an isolated network segment or remote facility.

The secondary PBS server initiates an outbound synchronization job to pull backups from the primary PBS instance:

proxmox-backup-manager sync-job create sync-from-primary \
    --store offsite-immutable \
    --remote primary-pbs \
    --remote-store production-vms \
    --schedule "daily" \
    --remove-vanished false

Setting --remove-vanished false is vital for ransomware protection. If an intruder manages to compromise the primary PBS node and deletes all local backups, the secondary offsite PBS node will ignore the deletions during its synchronization cycle. It retains all previously pulled chunks and manifests safely.

Furthermore, the production PVE cluster has no network path, firewall allowance, or authentication keys to communicate with the secondary PBS node. Even an adversary with physical console access to PVE cannot reach the air-gapped system.

Datastore Immutability and Retention Protection

Immutability ensures that once a backup chunk writes to physical media, no software command can delete or modify it until a defined retention policy expires.

PBS provides defensive mechanisms to safeguard snapshot retention schedules.

Prune and Garbage Collection Decoupling

In PBS, deleting a snapshot index does not remove the underlying disk data chunks immediately. Deletion occurs in two decoupled phases:

1. Pruning: Deletes snapshot index directories (/vm/100/2026-09-11T...) according to retention rules (keep-last, keep-daily, keep-monthly).
2. Garbage Collection (GC): A separate background process iterates through all .chunks on disk, marks unreferenced chunks, and purges them only if they have not been accessed within a 24-hour grace window.

Configure Garbage Collection to run on a predictable schedule managed strictly by internal system cron, not exposed to external triggers:

proxmox-backup-manager garbage-collection start production-vms

Filesystem-Level Protection: ZFS and Underlying Storage

If your PBS datastore rests on a ZFS zpool, configure automated read-only snapshots at the storage pool level. This provides an additional layer of isolation below the PBS application layer:

zfs snapshot -r tank/pbs-datastore@snap-immutable-$(date +%Y%m%d)
zfs set snapdir=visible tank/pbs-datastore

To prevent manual deletion of these ZFS snapshots even by an errant local script, apply a ZFS hold:

zfs hold ransomware-lock tank/pbs-datastore@snap-immutable-20260911

A held snapshot cannot be destroyed using zfs destroy until the hold tag is explicitly released. This prevents automated ransomware scripts from wiping underlying storage pools.

Automated Verification Jobs

Ransomware or bit rot can attempt to invalidate backup archives without deleting them. PBS provides automated verification jobs that compute SHA-256 checksums across every chunk in the datastore and compare them against the index manifests.

Create a verification job that scans the entire datastore weekly:

proxmox-backup-manager verify-job create verify-production \
    --store production-vms \
    --schedule "sat 02:00" \
    --outdated-after 30

If any chunk exhibits an invalid checksum, the job flags the exact snapshot, logs the event to syslog, and alerts your monitoring infrastructure.

Hardening the PBS Host: Network Isolation and Access Control

An encrypted, immutable architecture is only as resilient as the operating system hosting it. The PBS server must run on a minimal, hardened Debian footprint.

Restricting Access Ports

PBS operates primarily on TCP port 8007 for both web interface access and backup client API traffic.

Implement strict firewall rules using nftables or standard Linux firewall configuration to restrict access to port 8007 strictly to known PVE management IPs:

# /etc/nftables.conf snippet
table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;

        # Accept established connections
        ct state established,related accept

        # Loopback interface
        iif "lo" accept

        # Secure SSH access from Bastion host only
        ip saddr 10.10.10.50 tcp dport 22 accept

        # PVE Nodes allowed to reach PBS API port 8007
        ip saddr { 10.10.10.11, 10.10.10.12, 10.10.10.13 } tcp dport 8007 accept

        # ICMP echo requests (ping)
        icmp type echo-request accept
    }
}

For deeper systemic hardening strategies across production operating systems, consult our guide on Linux server hardening best practices.

Two-Factor Authentication for PBS Web Interface

Enforce Time-based One-Time Password (TOTP) or hardware WebAuthn security keys for all users accessing the PBS management dashboard:

• Navigate to Configuration -> Access Control -> Two-Factor Authentication.
• Add a new WebAuthn or TOTP record for root@pam.
• Disable standard password-only authentication for administrative access over remote networks.

Disaster Recovery Runbook: Restoring from Encrypted PBS Backups

When an outage strikes and production hypervisors are wiped, your recovery execution must be precise. Below is the emergency recovery runbook to restore virtual machines onto a clean Proxmox VE installation.

1. Re-install Clean PVE Nodes

Never restore backups onto a compromised, unformatted hypervisor. Install a clean Proxmox VE image from official verified media, apply network configurations, and verify hypervisor kernel integrity.

2. Import Encryption Keys

Retrieve your physical paper key from the safe. Reconstruct the private encryption key on the fresh PVE node:

proxmox-backup-client key create /etc/pve/priv/storage/pbs-recovery.enc

Enter the passphrase and paste the paper key string when prompted. Verify the key digest matches your recorded configuration:

proxmox-backup-client key show /etc/pve/priv/storage/pbs-recovery.enc

3. Mount the Hardened Backup Repository

Connect the clean PVE host to the secondary offsite PBS instance:

pvesm add pbs PBS-Recovery \
    --server pbs-offsite.internal.example.com \
    --datastore offsite-immutable \
    --username restore-auditor@pbs \
    --password-file /root/pbs-restore.cred \
    --fingerprint <offsite-pbs-fingerprint> \
    --encryption-key /etc/pve/priv/storage/pbs-recovery.enc

4. Query Available Snapshots via CLI

List available backup snapshots for the target virtual machine:

proxmox-backup-client snapshot list --repository restore-auditor@pbs@pbs-offsite.internal.example.com:offsite-immutable

You will see the historical timeline of uncorrupted snapshots:

┌──────────────────────────────┬──────────────┬────────────────────────────────────────┐
│ snapshot                     │ size         │ files                                  │
╞══════════════════════════════╪══════════════╪════════════════════════════════════════╡
│ vm/101/2026-09-10T02:00:00Z  │ 42.10 GiB    │ client.log manifest.json qemu-serv...  │
│ vm/101/2026-09-11T02:00:00Z  │ 42.15 GiB    │ client.log manifest.json qemu-serv...  │
└──────────────────────────────┴──────────────┴────────────────────────────────────────┘

5. Execute Full Virtual Machine Restoration

Restore the target virtual machine into clean local storage using the qmrestore utility:

qmrestore PBS-Recovery:backup/vm/101/2026-09-11T02:00:00Z 101 --storage local-zfs

PVE streams the encrypted 4MB chunks from the PBS repository, decrypts them in volatile memory on the hypervisor using the imported AES key, reconstructs the block device layout, and writes clean virtual disks to your local storage pool.

6. Boot in Isolated Quarantine Network

Do not connect restored virtual machines directly to production networks. Change the virtual network interface card (vNIC) bridge assignment to an isolated quarantine VLAN:

qm set 101 --net0 virtio,bridge=vmbr99
qm start 101

Inspect the guest operating system logs, verify database integrity, and run malware sweeps before routing external traffic back to the restored environment.

Summary Checklist for Proxmox Backup Hardening

Building an immutable backup defense requires disciplined separation of concerns across your virtualization cluster:

• No Root Credentials on PVE: Use restricted PBS API tokens with DatastoreBackup role only.
• Client-Side Encryption Active: Generate AES-GCM 256-bit keys and securely store physical paper keys offsite.
• Pull Replication Configured: Secondary PBS servers pull backups out of band with --remove-vanished false.
• Network Isolation: Restrict TCP port 8007 using firewall rules to authorized subnets only.
• Verification Jobs Scheduled: Run weekly automated chunk hashing jobs to detect bit rot and data tampering.
• Documented Disaster Recovery: Test the CLI restore runbook quarterly on bare-metal hardware.

When your backup architecture assumes zero trust between the hypervisor and the storage repository, ransomware loses its primary leverage. Even in a catastrophic hypervisor takeover, your systems remain recoverable within hours.