How to monitor RAM consumption on Red Hat 7.9

Question

Red Hat 7.9 server with 512 GB RAM.

We often have alerts about swap being full. Swap is often used 99%. Our server admin told us it is normal for linux to have swap used 100%. There is no way to check the real RAM consumption.

Ours is always around 99% on our server using SAR:

Memory & Swap
=============
                    kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
01/11/2024    9869583 517833049     98.13       977 285023790 319073740     58.60 340613912 162678784    614843
01/12/2024    4181004 523521628     99.21      1349 287757937 323767076     59.46 346046454 162168337   1115633
01/13/2024    2567787 525134845     99.51       844 285755715 327744180     60.19 352144911 157031086    654493
01/14/2024    2827695 524874937     99.46       844 285135562 328070493     60.25 352003644 156742082    742178
01/15/2024    3482087 524220545     99.34      1838 280133083 332837943     61.13 353676271 153986907    998373
01/16/2024    2152990 525549642     99.59       839 273578756 342252974     62.86 362157756 147013751   1136099
01/17/2024    4575639 523126993     99.13      2418 271393967 340334778     62.51 355987093 150884756   1033531
01/18/2024    2205445 525497187     99.58      2413 282078831 328216144     60.28 353148916 156770066    625021
01/19/2024    9451354 518251278     98.21      1542 293648210 305176716     56.05 352495156 150264497    680464

On the web I can see that it is not normal to have high swap usage. I also noticed that after a reboot, swap is very low during several days. When a process consumes all the RAM (mistake on our side), then swap usage increases to 100%, and then never decreases, even if the corresponding process is killed. This is why servers that are on for weeks have 100% swap used.

I was told to monitor the swap usage using sar (pswpin/s pswpout/s).

When swap usage is 100%, I could have no problem but when processes start being killed because of RAM issues, I can see high values for pswpin/s pswpout/s (sar -W).

During a week I can monitor this activity to check if I had RAM problems or not.

My problem is the following: How can I prevent the RAM problem from occurring? What can I use to check the % of RAM used (whereas it is always 99% in SAR...)? How to get the real value like in Windows OS? To be sure to kill the process that start taking all the RAM.

I would like to generate a warning when RAM usage is 80%.

I know that free -h could be used but I don't know how to interpret that. the same for "top".

For r instance, I compare sar output with free -h and top output and I don't see matching values... :-(

[XXXXX@YYYYYY ~]$ sar -r
Linux 3.10.0-1062.4.1.el7.x86_64 (XXXXXX)   02/02/2024  _x86_64_    (64 CPU)
10:30:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty

11:55:01 AM   7916396 519786232     98.50      4688 436720908 164442944     30.20 256757744 249531760      1548
12:00:01 PM   1106680 526595948     99.79      4688 436883120 173459020     31.86 263356476 249699552      4184
12:05:01 PM    742056 526960572     99.86      4688 436447380 173668428     31.90 264216776 249402376      5380
12:10:01 PM  11076780 516625848     97.90      4688 434763392 162944064     29.93 255501116 247835888      3732
12:15:01 PM   7891084 519811544     98.50      4688 434921220 165981024     30.48 258656448 247885164       600
Average:      2201518 525501110     99.58      5447 448667788 154099963     28.30 257069976 255388912      3431
[XXXXX@YYYYYY ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:           503G         81G        4.3G        1.3G        417G        419G
Swap:           15G         11G        4.4G
[XXXXX@YYYYYY ~]$ top
top - 12:22:09 up 9 days, 18:55, 48 users,  load average: 4.21, 4.69, 5.06
Tasks: 2645 total,   5 running, 2488 sleeping,   0 stopped, 152 zombie
%Cpu(s):  6.8 us,  3.4 sy,  0.0 ni, 89.8 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 52770262+total,  6888876 free, 83310656 used, 43750310+buff/cache
KiB Swap: 16777212 total,  4582892 free, 12194320 used. 44195168+avail Mem 

I don't know which values to check on top or free -h to know the real % of RAM consumption usage.

thanks a lot for helping

Answer 1

Your system does not seem to have any swap-related issue. However, as you mention high swap use and killed processes, you can have occasional low-memory issues.

By default, some swap is expected to be used even on system with plenty of free memory as the kernel proactively swap inactive memory to have more room for buffer/cache. You have 500+ GB of RAM and a 16 GB swap partition, so it is normal to fill it with inactive data.

If swap is causing you performance issues, you can limit or disable its use with the following:

• sysctl vm.swappiness=0 means the system should not use any swap unless free memory is dangerously very low. To persists across reboot, you need to put vm.swappiness=0 into /etc/sysctl.conf file;

• swapoff -a forcefully disable any swap. Please note that if you genuinely need some swap (ie: to absorb a memory consumption peak) your system could kill some process via OOM-killer. To persist across reboot, you need to remove the swap entry from /etc/fstab file;

• disable disk swap and use zram to create a compressed in-RAM device. zram can be very useful but I would not use this solution unless really needed and applicable to your workload.

Some more info:

When a process consumes all the RAM (mistake on our side), then swap usage increases to 100%, and then never decreases, even if the corresponding process is killed

When a swapped process exists, swap should be released. If swap is constantly filled it probably means that another non-swapped process was killed, rather than the swapped one. You can use dmesg to check for kernel OOM.

When swap usage is 100%, I could have no problem but when processes start being killed because of RAM issues, I can see high values for pswpin/s pswpout/s (sar -W).

This means the kernel is (unsuccessfully) trying to free some memory to prevent OOM.

I would like to generate a warning when RAM usage is 80%.

sar does not have alerting capability as far I know. You need to parse free output and/or use a network monitor tool as zabbix, netdata, etc.

I know that free -h could be used but I don't know how to interpret that. the same for "top".

Well, you should really read the relevant man pages. In short, you can check the used or available columns reported by free.

Answer 2

I would suggest to install atop

https://www.redhat.com/sysadmin/analyzing-linux-server-performance-atop

This creates performance snapshots in a pre-defined interval, so you can see what leads up to the memory consumption.

---

Edit regarding the follow up question for network

The Linux kernel does not maintain counters for the number of network accesses issued per process or per thread. As a consequence, it is not possible to analyze which process and/or thread causes most load in case that atop shows a high utilization on one of your network interfaces. The optional kernel module netatop can be loaded to gather statistics about the TCP and UDP packets that have been transmitted/received per process and per thread. As soon as atop discovers that this module is active, it shows the columns SNET and RNET in the generic screen for the number of transmitted and received packets per process. When the 'n' key is pressed, it shows detailed counters about the number packets transmitted/received via TCP and UDP, the average sizes of these packets, and the total bandwidth consumed for input and output per process/thread. https://www.atoptool.nl/netatop.php

Answer 3

This is mostly comment.

Our server admin told us it is normal for linux to have swap used 100%

Erm, no. I hope this story lost some of its original meaning in the re-telling. Apart from some very rare edge cases, actively using swap on a host acting as a server is BAD.

What's also very peculiar is that the swap space is sized at around 3% of the RAM. I am struggling to imagine why this would ever make sense.

I am also left wondering what your role in this story is, if you are not the system administrator.

when processes start being killed because of RAM issues

This should NOT happen on a properly configured/managed server host.

Also, the tools you are trying to use for this task are useful for diagnostics, but poor for monitoring. There are lots of excellent tools available, many of them as free software, for monitoring; Zabbix, Nagios, Icinga, LibreNMS, Check_MK to name but a few. Don't try to write your own unless you have 1) a good knowledge of system admin 2) a very good reason for not using an available tool.

Answer 4

The Linux memory utilisation can be confusing and the output of different tools hard to interpret. You may want to read something like https://www.linuxatemyram.com/ as introductory material first.

The output of top and free shows that your system underutilizes the RAM - only 81GB is used after 9 days of uptime and everything else is buffers and cache which can be discarded if the system needs more memory for programs. But if top and free were run right after a large program was OOM-killed and memory was freed then these metrics are irrelevant.

The first sar output shows more committed memory (~60%) and more active memory (360GB) vs the second one (~30% and 260 GB respectively). These values are far from the situation of running out of memory and processes being killed. The first sar shows measurements over multiple days without any significant change, so there are no slow memory leaks.

The swap usage in your case is insignificant (16 GB out of 512GB) and has little effect. Your system can run pretty fine with 100% swap usage as long as you have enough memory in available column in top or free output. You can increase the swap size to keep its utilisation below 100% so it improves the system performance by freeing the real RAM. Keep an eye on vmstat swap-in (si) and swap-out (so) metrics - they should be close to zero most of the time. Keep in mind that swap usage percentage is a second-order effect of memory consumption caused by a runaway process.

Considering all the available details and the absence of slow memory leaks I would suspect that you sometimes have a runaway process which consumes all the memory fast, forcing swap usage to 100% and then this process gets OOM-killed. You need to check your logs for OOM messages, they should contain information about the killed process and its memory consumption. Once you identified the runaway process you can set its memory limits in its systemd unit so it gets killed before consuming all system resources.

You can also inspect memory utilisation by the individual processes at any moment by running top -o RES which would sort the processes by their resident memory consumption.

You can install something simple like monit to monitor both system memory/swap consumption and per-process memory consumption for selected processes.