Timmy's picture

BLUF - Mediaserver VM usage not dropping as expected according to PVE GUI during idle periods.

Alright, first off, these images are awesome. The MediaServer got me off the ground and running way faster than the last one I tried building from instructions.

The data:
Server is Supermicro X10DRL-i with 2x E5-2670 and 256GB RAM, running Proxmox 5.4.124.
Hypervisor on a two-drive SSD ZFS mirror configuration.
The four spinning disk drives in hotswap bays are setup in a raidz2 configuration. Test VM's are then allocated storage from this pool.
No ZFS cache disk setup at this time.
Currently, two VMs setup. MediaServer v16.1 and Linux Mint 20.2.

 

Now the issue I'm observing.

Linux Mint (and other VMs in the past) seem to report accurate RAM usage. It goes up and down with demand.
MediaServer does not. I've allocated 48GB to the VM and during a file upload (using WinSCP), the usage will climb to 97% (if I'm moving more than ~50GB) and stay there, even well after the transfer is done.

I recall reading that ZFS will hold data to write to the pool in RAM, if there is no cache disk. But I would have figured once all data was uploaded that as the data is written, the usage would drop back down. It does not. 4hrs after transfer complete, the VM still shows 97% of allocated RAM in use, despite sitting at idle the entire time.

Possibly unrelated (but I'll mention anyway in case it is helpful) is the VM does not respect Proxmox command to shutdown/restart from the PVE gui. Shutdown/reboot at command line inside of VM works.

 

I've only been working in Proxmox for a few months (as I have time) and I've spun up a number of VMs. This is the first and only time I've observed this issue. I have kept Proxmox up-to-date but I can't point to any specific updates that might have triggered this behavior. And the fact it seems unique only to the Turnkey VM makes me think there is some issue between the two in their configuration.

Forum: 
Jeremy Davis's picture

To be honest, I'm not sure.

I have a Media Server instance running on my Proxmox server too. Mine is an LXC container and only has 4GB assigned to it and idles at 1GB usage. Mine isn't backed by ZFS. I have the OS filesystem running from a thin LVM volume on a SSD, with the media storage provided by a number of WD red HDDs in a singe logical volume, bind mounted into my Media Server instance at /srv/storage.

Neither my TurnKey Media Server nor Proxmox are the latest release, which may or may not have any relevance.

I'd be inclined to try to pin down what is actually chewing up the RAM. From within the VM/container run:

free -h

For example, here's my Media Server:

              total        used        free      shared  buff/cache   available
Mem:          4.0Gi       978Mi        56Mi        16Mi       3.0Gi       3.0Gi
Swap:         6.0Gi       349Mi       5.7Gi

Note that you need to read the 'available' column. As you can see buffer/cache is "using" most of my RAM leaving only 56MB 'free', but that would be cleared immediately if it's required, so there is actually 3GB available.

If yours is showing lots of free RAM internally, but not within Proxmox, then I'm not sure what's going on there. Personally, I'd be inclined to ask on the Proxmox forums. I suspect that they'd have more insight into what might cause that. If you do post over there, please feel free to post a link to that here.


As to not responding to the Promxox interface for shutdown/restart, it depends. If yours is also an LXC container, then I'm not clear what could cause that. If you have installed to a "proper" VM (i.e. KVM) then there is more that can go wrong there, but generally it should "just work". I vaguely recall someone else having an issue with that ages ago and it was some setting within Proxmox that they changed that fixed it. If yours is the same case, then IIRC that was a setting about using ACPI calls?! If Proxmox is sending ACPI calls, then TurnKey should respond to those. However, there is also a way it can be handled via the QEMU guest agent. And TurnKey doesn't include the agent by default, so installing that is a possible alternate route.

If you want to install the qemu-guest-agent package, do this (includes a reboot):

apt update && apt install -y qemu-guest-agent && reboot && exit

Please let me know if that helps.

Timmy's picture

Ran the free -h command prior to starting a transfer.

Then several times during a large transfer.

 

The internal count jives with proxmox on the way up. I waited and once the transfer completed, I watched the lights continue go (indicating the write is carrying on). Once the lights were off, I checked it again.

Then I looked into it and found a Debian command to clear the cache/buff.

Proxmox jives with the internal measurements after the cache clear. So Proxmox appears to be reading accurately.

root@mediaserver ~# free -h
              total        used        free      shared  buff/cache   available
Mem:           47Gi       365Mi        46Gi        10Mi       255Mi        46Gi
Swap:         5.9Gi          0B       5.9Gi
root@mediaserver ~# free -h
              total        used        free      shared  buff/cache   available
Mem:           47Gi       378Mi        42Gi        10Mi       4.6Gi        46Gi
Swap:         5.9Gi          0B       5.9Gi
root@mediaserver ~# free -h
              total        used        free      shared  buff/cache   available
Mem:           47Gi       379Mi        35Gi        10Mi        11Gi        46Gi
Swap:         5.9Gi          0B       5.9Gi
root@mediaserver ~# free -h
              total        used        free      shared  buff/cache   available
Mem:           47Gi       377Mi        32Gi        10Mi        14Gi        46Gi
Swap:         5.9Gi          0B       5.9Gi
root@mediaserver ~# free -h
              total        used        free      shared  buff/cache   available
Mem:           47Gi       429Mi        30Gi        10Mi        16Gi        46Gi
Swap:         5.9Gi          0B       5.9Gi
root@mediaserver ~# sync; echo 3 > /proc/sys/vm/drop_caches
root@mediaserver ~# free -h
              total        used        free      shared  buff/cache   available
Mem:           47Gi       425Mi        46Gi        10Mi       150Mi        46Gi
Swap:         5.9Gi          0B       5.9Gi

 

In so far as the power off/restart command issue: I'm not sure how to see what command Proxmox sends to the VM. I've got qemu agent disabled. I think I recall reading that thread yesterday. The VM is KVM. I have not dived into LXC at all and a quick look at the how-to, I've definitely not done that.

I've got ACPI enabled and verified packages installed on turnkey. Went ahead and installed QEMU on the mediaserver and enabled it on proxmox. QEMU agent works.

 

Jeremy Davis's picture

Whilst the RAM is still in use by buffer/cache and is not "free", if you look at the 'available' column, it is still available for use by other apps. So there isn't actually any issue there. If something needed to use the RAM, it would be instantly freed. Why the data remains cached in RAM after writing to disk I'm not sure, but what you've shown here shows that there isn't actually anything wrong or any issue here.

And it sounds like installing the qemu-guest-agent solved the PVE UI shutdown/restart issue?

Timmy's picture

Whilst the RAM is still in use by buffer/cache and is not "free", if you look at the 'available' column, it is still available for use by other apps.

I get it isn't bad; the memory is there and available.

But it does give me a misconception that the VM is under provisioned on memory when it isn't and makes it hard to judge performance when the main dashboard is, effectively, a lie. But reading around further last night, knowing what I know now, I see this is a long-standing complaint.

Guess I'll test and see how long it tries to hold that data. Maybe after an hour or two, it'll drop it. Otherwise, after doing uploads, I'll just clear it manually of clean data.

 

And it sounds like installing the qemu-guest-agent solved the PVE UI shutdown/restart issue?

Apologies. My final sentence wasn't clear. Yes. QEMU agent resolved the issue.

Jeremy Davis's picture

If you want to make it so Promxox sees (more closely) what the actual available RAM is, then you could write a script for that!? Something along the lines of checking the 'free' vs 'available' and if the difference is too high, runs the commands you posted before (i.e. 'sync; echo 3 > /proc/sys/vm/drop_caches'). Then run that script as a cron job.

It's a bit hacky, and as already discussed the issue is only cosmetic, but it should be fairly easy to achieve your desired outcome. If you're unfamiliar with bash, now is probably a great time to learn a bit! :)

As a hint, you can get the free and available RAM (in kb) like this:

free=$(grep ^MemFree /proc/meminfo | tr -s ' ' | cut -d' ' -f2)
avail=$(grep ^MemAvail /proc/meminfo | tr -s ' ' | cut -d' ' -f2)

And if your script is /usr/local/bin/clear-free-ram, then you can make a cron job that will run every 5 minutes like this:

cat > /etc/cron.d/clear-free-ram 
Timmy's picture

Yeah.. it is 'cosmetic' for the most part. Need to read up on Proxmox for scripting that sort of thing.

Put it on the list of things to do. After figuring out this VNC issue.

Thanks for the assist.

Jeremy Davis's picture

Good luck with it all and please don't hesitate to post back with any other questions and/or feedback you have.

Timmy's picture

Donation via paypal!

Add new comment