Proxmox Memory Optimisation: Squeeze More RAM Out of What You Have

RAM prices have crept back up, and if you're running a homelab on a fixed budget, buying more memory isn't always an option. The good news? You can probably do more with what you already have. With the right configuration across your Proxmox host, your VMs, and your containers, it's always possible squeeze a little more performance.

1. KSM: Kernel Samepage Merging

If you're running multiple VMs from the same base template — which, if you're not, you should be — they're all loading identical chunks of memory. The same kernel, the same libc libraries, the same base packages. Kernel Samepage Merging (KSM) scans for those duplicate memory pages across your VMs and merges them into a single shared read-only page.

Ten Debian VMs each loading the same 100MB library? Without KSM, that's 1GB of RAM. With KSM, it's 100MB — shared by all ten. If a VM writes to that page, KSM transparently forks a private copy (Copy-on-Write). The VMs never notice.

The trade-off is a small amount of CPU overhead for the scanning process. In practice, the RAM savings on a homelab running similar VMs vastly outweigh the cost.

Proxmox Host Side

KSM is usually already enabled in the Proxmox web UI by default — you'll find it under Datacenter → Options → KSM Sharing. If it's not enabled, switch it on. That's the host side done.

Step 1: Install and Enable KSM (Guest Side)

Inside each Ubuntu VM, you need the ksmtuned daemon installed and running. You may have already done this, but let's make sure it's set up correctly.

Install the daemon:

sudo apt update
sudo apt install ksmtuned

Enable and start the service:

sudo systemctl enable --now ksmtuned

Verify the service is running:

systemctl status ksmtuned

If it says "active (running)", you're good.

Step 2: Force KSM to Run

By default, ksmtuned might wait until your RAM is around 80% full before it actually turns KSM on. For testing — or if you want it active immediately — you can force it.

Switch to root:

sudo -i

Force the KSM kernel thread to run:

echo 1 > /sys/kernel/mm/ksm/run

Confirm it's on:

cat /sys/kernel/mm/ksm/run

Output 1: KSM is running. Output 0: it's stopped.

To see how much RAM KSM is actually saving:

echo "KSM saving: $(( $(cat /sys/kernel/mm/ksm/pages_sharing) * 4096 / 1024 / 1024 )) MB"

2. Memory Ballooning: Stop Locking RAM You're Not Using

This is one of the most common misconfigurations seen in homelabs. You allocate 4GB to a VM, and Proxmox locks that full 4GB away from every other VM on the host, even if the guest is sitting idle, using 400MB of it.

Memory Ballooning fixes this. You set a maximum (e.g., 4GB) and a minimum (e.g., 1GB) for the VM. Proxmox uses the virtio-balloon driver alongside the QEMU Guest Agent to dynamically adjust how much RAM the guest actually holds.

When your Proxmox host is under memory pressure, it inflates a virtual "balloon" inside the guest OS, which forces the guest to release its unused file caches and return that RAM to the hypervisor pool. When pressure eases, the balloon deflates and the guest gets its RAM back. The guest OS never notices anything unusual.

To configure it in the Proxmox web UI:

1. Navigate to your VM → Hardware → Memory
2. Check Ballooning Device
3. Set your Minimum Memory (the floor — the VM will never drop below this)
4. The existing Memory value becomes the maximum ceiling

For VMs running memory-hungry services (databases, media servers), be conservative with the minimum. For lightweight VMs (reverse proxies, DNS, monitoring agents), you can be much more aggressive.

3. ZRAM: Swap That Doesn't Kill Performance

Swap space is the traditional answer to memory pressure. It's also one of the worst experiences you can have on a server, the moment Linux starts swapping to a spinning disk or even an SSD, latency spikes, performance craters, and you know things have gone wrong.

ZRAM is a better solution. Instead of swapping to a physical drive, ZRAM creates a block device inside RAM itself that acts as a compressed swap pool. The Linux kernel compresses everything written to it, typically achieving 3:1 or 4:1 compression ratios.

A 1GB idle process gets compressed down to around 250MB in ZRAM, freeing 750MB of physical RAM — with orders-of-magnitude lower latency than disk swap, because it's still in RAM.

Setting it up inside a Debian or Ubuntu VM

apt install zram-tools

nano /etc/default/zramswap

# Recommended settings
ALGO=lz4      # Fast compression with a solid ratio
PERCENT=50    # Allocate 50% of physical RAM to the ZRAM pool

systemctl restart zramswap
systemctl enable zramswap

# Confirm it's active
swapon --show

You should see a ZRAM device listed. From this point, when the VM's memory pressure rises, the kernel compresses and packs idle processes into ZRAM rather than hammering your NVMe.

4. Docker Memory Limits: Hard Caps vs. Soft Reservations

Docker containers will consume every byte of RAM available to them if you let them. The instinctive fix is a hard memory limit — but that leaves performance on the table when the host has headroom to spare.

The smarter approach combines a hard limit with a soft reservation:

• Hard limit (--memory): The absolute ceiling. The container cannot exceed this, period.
• Soft reservation (--memory-reservation): The target Docker will try to shrink the container toward when the host is under pressure.

In practice the container bursts freely up to the hard limit when the host has spare RAM. The moment the host starts feeling memory pressure, Docker aggressively pushes the container back toward its reservation. Best of both worlds, full performance when capacity is available, guaranteed stability when it's not.

In Docker Compose:

services:
  jellyfin:
    image: jellyfin/jellyfin:latest
    deploy:
      resources:
        limits:
          memory: 4g          # Hard ceiling
        reservations:
          memory: 1g          # Soft floor under pressure

Or via CLI:

docker run -d   --memory=4g   --memory-reservation=1g   jellyfin/jellyfin:latest

Apply this pattern to anything that can realistically balloon: Jellyfin, Nextcloud, databases, build servers. Lightweight services like reverse proxies and DNS resolvers are fine with just a hard limit.

5. Base Image Optimisation: Stop Pulling What You Don't Need

The container memory conversation starts before the container even runs — it starts with the base image. Most people pull the default tag without thinking about it. That often means dragging in a full Debian or Ubuntu base that sits at 50–100MB+ of idle RAM, for a service that has no use for most of that.

Two leaner alternatives:

Alpine Linux — Uses musl libc instead of glibc, and BusyBox instead of standard GNU utilities. The base image is around 5MB and idle RAM usage is negligible. Perfect for: Nginx, Caddy, Redis, lightweight web apps, simple scripts and APIs. The trade-off: musl libc compatibility. Some apps built against glibc behave unexpectedly on Alpine. Always test before committing.

Debian Slim — If you are running Python, Node, or database containers, stick to Debian Slim. Alpine's musl libc can cause serious performance regressions and compilation bugs with certain programming languages — things that work fine in development and then silently misbehave in production. Debian Slim strips out documentation, locales, and optional packages, coming in at around 30MB while keeping the heavily optimised glibc underneath. You get the memory savings without the compatibility risk. It's the safer default for anything complex.

# Full Nginx image: ~140MB, ~8MB idle RAM
FROM nginx:latest

# Alpine Nginx: ~20MB, negligible idle RAM
FROM nginx:alpine

At scale, this adds up fast. Twenty containers at 100MB idle RAM each is 2GB. Swap them to Alpine or Slim equivalents and that headroom comes back immediately — no hardware changes, no restarts, just a different image tag.

Putting It All Together

These techniques aren't mutually exclusive - stack them. A well-optimised Proxmox homelab looks something like this:

• Proxmox host: ksmtuned running and active, ballooning enabled on every VM with a sensible minimum floor
• Inside each VM: ZRAM configured as swap, qemu-guest-agent installed and enabled
• Docker layer: Memory limits with soft reservations on resource-hungry containers, Alpine or Slim base images everywhere possible

Done right, this combination can effectively double the number of VMs and containers you can run on the same hardware. I went from struggling to fit 6 VMs on 32GB comfortably, to running 14 with headroom, without buying a single stick of RAM 😀.

If you're building out the rest of your Proxmox stack, check out Setting Up Ubuntu Server VMs on Proxmox and Docker Security: 10 Hardening Steps for the foundation that makes everything above work properly.