1

Red Hat 7.9 server with 512 GB RAM.

We often have alerts about swap being full. Swap is often used 99%. Our server admin told us it is normal for linux to have swap used 100%. There is no way to check the real RAM consumption.

Ours is always around 99% on our server using SAR:

Memory & Swap
=============
                    kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
01/11/2024    9869583 517833049     98.13       977 285023790 319073740     58.60 340613912 162678784    614843
01/12/2024    4181004 523521628     99.21      1349 287757937 323767076     59.46 346046454 162168337   1115633
01/13/2024    2567787 525134845     99.51       844 285755715 327744180     60.19 352144911 157031086    654493
01/14/2024    2827695 524874937     99.46       844 285135562 328070493     60.25 352003644 156742082    742178
01/15/2024    3482087 524220545     99.34      1838 280133083 332837943     61.13 353676271 153986907    998373
01/16/2024    2152990 525549642     99.59       839 273578756 342252974     62.86 362157756 147013751   1136099
01/17/2024    4575639 523126993     99.13      2418 271393967 340334778     62.51 355987093 150884756   1033531
01/18/2024    2205445 525497187     99.58      2413 282078831 328216144     60.28 353148916 156770066    625021
01/19/2024    9451354 518251278     98.21      1542 293648210 305176716     56.05 352495156 150264497    680464

On the web I can see that it is not normal to have high swap usage. I also noticed that after a reboot, swap is very low during several days. When a process consumes all the RAM (mistake on our side), then swap usage increases to 100%, and then never decreases, even if the corresponding process is killed. This is why servers that are on for weeks have 100% swap used.

I was told to monitor the swap usage using sar (pswpin/s pswpout/s).

When swap usage is 100%, I could have no problem but when processes start being killed because of RAM issues, I can see high values for pswpin/s pswpout/s (sar -W).

During a week I can monitor this activity to check if I had RAM problems or not.

My problem is the following: How can I prevent the RAM problem from occurring? What can I use to check the % of RAM used (whereas it is always 99% in SAR...)? How to get the real value like in Windows OS? To be sure to kill the process that start taking all the RAM.

I would like to generate a warning when RAM usage is 80%.

I know that free -h could be used but I don't know how to interpret that. the same for "top".

For r instance, I compare sar output with free -h and top output and I don't see matching values... :-(

[XXXXX@YYYYYY ~]$ sar -r
Linux 3.10.0-1062.4.1.el7.x86_64 (XXXXXX)   02/02/2024  _x86_64_    (64 CPU)
10:30:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty

11:55:01 AM   7916396 519786232     98.50      4688 436720908 164442944     30.20 256757744 249531760      1548
12:00:01 PM   1106680 526595948     99.79      4688 436883120 173459020     31.86 263356476 249699552      4184
12:05:01 PM    742056 526960572     99.86      4688 436447380 173668428     31.90 264216776 249402376      5380
12:10:01 PM  11076780 516625848     97.90      4688 434763392 162944064     29.93 255501116 247835888      3732
12:15:01 PM   7891084 519811544     98.50      4688 434921220 165981024     30.48 258656448 247885164       600
Average:      2201518 525501110     99.58      5447 448667788 154099963     28.30 257069976 255388912      3431
[XXXXX@YYYYYY ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:           503G         81G        4.3G        1.3G        417G        419G
Swap:           15G         11G        4.4G
[XXXXX@YYYYYY ~]$ top
top - 12:22:09 up 9 days, 18:55, 48 users,  load average: 4.21, 4.69, 5.06
Tasks: 2645 total,   5 running, 2488 sleeping,   0 stopped, 152 zombie
%Cpu(s):  6.8 us,  3.4 sy,  0.0 ni, 89.8 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 52770262+total,  6888876 free, 83310656 used, 43750310+buff/cache
KiB Swap: 16777212 total,  4582892 free, 12194320 used. 44195168+avail Mem 

I don't know which values to check on top or free -h to know the real % of RAM consumption usage.

thanks a lot for helping

4
  • 1
    The process that used all of the memory is not necessarily the only one that was swapped out and killing it will not automagically bring all other processes out of swap. Check for the string VmSwap in /proc/${pid}/status to see what processes are still using swap. When each process needs its memory it will pull it from swap; until then there is no performance hit on swapped & idle processes.
    – doneal24
    Commented Feb 2 at 17:47
  • you mean that having a lot of process in swap may not be a problem because it could be idle process that are not used so there is no performance impact, right ? if the process wake up then it will swap and come back in RAM memory ? So it mean that because of my memory problem a lot of idle process were swapped but then don't automatically come back in RAM memory because there are still idle process. am I correct ?
    – iostrym
    Commented Feb 6 at 8:40
  • I tried "pidof memcached" and I got no pid displayed whereas swap is 100% used. Do I have process using the swap or not ? also I tried pgrep "vmswap" and I got no pid. So I don't undertand how can I have swap 100% used. I can see around 100% is used here : Tasks: 2749 total, 6 running, 2318 sleeping, 0 stopped, 425 zombie %Cpu(s): 14.5 us, 5.1 sy, 0.0 ni, 80.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 52770748+total, 34250076 free, 22634787+used, 26710953+buff/cache KiB Swap: 16777212 total, 8208 free, 16769004 used. 29533993+avail Mem
    – iostrym
    Commented Feb 6 at 10:31
  • Most likely you have some processes leaking memory but your stats show only total memory usage on the system and no details about processes. Check the logs for OOM messages and add them to the question. Run top -b -o %MEM -n 1 | head -30 when you get your swap full and add the output to the question.
    – AlexD
    Commented Feb 7 at 11:37

4 Answers 4

2

Your system does not seem to have any swap-related issue. However, as you mention high swap use and killed processes, you can have occasional low-memory issues.

By default, some swap is expected to be used even on system with plenty of free memory as the kernel proactively swap inactive memory to have more room for buffer/cache. You have 500+ GB of RAM and a 16 GB swap partition, so it is normal to fill it with inactive data.

If swap is causing you performance issues, you can limit or disable its use with the following:

  • sysctl vm.swappiness=0 means the system should not use any swap unless free memory is dangerously very low. To persists across reboot, you need to put vm.swappiness=0 into /etc/sysctl.conf file;

  • swapoff -a forcefully disable any swap. Please note that if you genuinely need some swap (ie: to absorb a memory consumption peak) your system could kill some process via OOM-killer. To persist across reboot, you need to remove the swap entry from /etc/fstab file;

  • disable disk swap and use zram to create a compressed in-RAM device. zram can be very useful but I would not use this solution unless really needed and applicable to your workload.

Some more info:

When a process consumes all the RAM (mistake on our side), then swap usage increases to 100%, and then never decreases, even if the corresponding process is killed

When a swapped process exists, swap should be released. If swap is constantly filled it probably means that another non-swapped process was killed, rather than the swapped one. You can use dmesg to check for kernel OOM.

When swap usage is 100%, I could have no problem but when processes start being killed because of RAM issues, I can see high values for pswpin/s pswpout/s (sar -W).

This means the kernel is (unsuccessfully) trying to free some memory to prevent OOM.

I would like to generate a warning when RAM usage is 80%.

sar does not have alerting capability as far I know. You need to parse free output and/or use a network monitor tool as zabbix, netdata, etc.

I know that free -h could be used but I don't know how to interpret that. the same for "top".

Well, you should really read the relevant man pages. In short, you can check the used or available columns reported by free.

4
  • thanks for your detailled answer. about using free command, yes I use it now and to know the real free RAM I need to calculate free RAM + buff/cached RAM. Then I have the real RAM not used by process. buff/cached ram is free ram that was filled by the OS to be used as cache, right ?
    – iostrym
    Commented Feb 7 at 13:08
  • Yes, you can consider buffer/cache as free memory. Anyway, you can simply look at available column.
    – shodanshok
    Commented Feb 7 at 14:06
  • thanks, it is simpler using this available column : available = free + cache - shared. right ?
    – iostrym
    Commented Feb 13 at 8:11
  • You already have the available columns in free output. No need to do any other calculation. For more info on how it is calculated, give a look at free man page.
    – shodanshok
    Commented Feb 13 at 10:49
1

I would suggest to install atop

https://www.redhat.com/sysadmin/analyzing-linux-server-performance-atop

This creates performance snapshots in a pre-defined interval, so you can see what leads up to the memory consumption.


Edit regarding the follow up question for network

The Linux kernel does not maintain counters for the number of network accesses issued per process or per thread. As a consequence, it is not possible to analyze which process and/or thread causes most load in case that atop shows a high utilization on one of your network interfaces. The optional kernel module netatop can be loaded to gather statistics about the TCP and UDP packets that have been transmitted/received per process and per thread. As soon as atop discovers that this module is active, it shows the columns SNET and RNET in the generic screen for the number of transmitted and received packets per process. When the 'n' key is pressed, it shows detailed counters about the number packets transmitted/received via TCP and UDP, the average sizes of these packets, and the total bandwidth consumed for input and output per process/thread. https://www.atoptool.nl/netatop.php

4
  • thanks. I already have nmon and SAR activated on the server but I don't have the "process" view with these tools except for nmon that show top processes that use CPU but there is no detailled information about process to detect the one that take all RAM or all network... is it possible with ATOP to detect the process that use the most network ?
    – iostrym
    Commented Feb 6 at 8:46
  • You mean memory instead of network, because network would be a new question?
    – Turdie
    Commented Feb 6 at 9:00
  • I mean network, i violated the forum rule :/ because I have also this kind of need (to be able to monitor NFS network), I will ask a separated question for this. as atop was a proposal I wanted to know if it could be also used for network monitoring
    – iostrym
    Commented Feb 6 at 10:19
  • DANGER! DANGER! Asking atop to map per-process memory usage causes a pause in execution of tasks on the machine. For a host with 0.5t of RAM, this is going to last a significantly long time.
    – symcbean
    Commented Feb 7 at 9:51
1

This is mostly comment.

Our server admin told us it is normal for linux to have swap used 100%

Erm, no. I hope this story lost some of its original meaning in the re-telling. Apart from some very rare edge cases, actively using swap on a host acting as a server is BAD.

What's also very peculiar is that the swap space is sized at around 3% of the RAM. I am struggling to imagine why this would ever make sense.

I am also left wondering what your role in this story is, if you are not the system administrator.

when processes start being killed because of RAM issues

This should NOT happen on a properly configured/managed server host.

Also, the tools you are trying to use for this task are useful for diagnostics, but poor for monitoring. There are lots of excellent tools available, many of them as free software, for monitoring; Zabbix, Nagios, Icinga, LibreNMS, Check_MK to name but a few. Don't try to write your own unless you have 1) a good knowledge of system admin 2) a very good reason for not using an available tool.

1
  • The part that interests me is "When a process consumes all the RAM". Even if it has 500 GB, doing that probably creates other performance related issues, if fragmentation occurs.
    – Greg Askew
    Commented Feb 7 at 10:24
1

The Linux memory utilisation can be confusing and the output of different tools hard to interpret. You may want to read something like https://www.linuxatemyram.com/ as introductory material first.

The output of top and free shows that your system underutilizes the RAM - only 81GB is used after 9 days of uptime and everything else is buffers and cache which can be discarded if the system needs more memory for programs. But if top and free were run right after a large program was OOM-killed and memory was freed then these metrics are irrelevant.

The first sar output shows more committed memory (~60%) and more active memory (360GB) vs the second one (~30% and 260 GB respectively). These values are far from the situation of running out of memory and processes being killed. The first sar shows measurements over multiple days without any significant change, so there are no slow memory leaks.

The swap usage in your case is insignificant (16 GB out of 512GB) and has little effect. Your system can run pretty fine with 100% swap usage as long as you have enough memory in available column in top or free output. You can increase the swap size to keep its utilisation below 100% so it improves the system performance by freeing the real RAM. Keep an eye on vmstat swap-in (si) and swap-out (so) metrics - they should be close to zero most of the time. Keep in mind that swap usage percentage is a second-order effect of memory consumption caused by a runaway process.

Considering all the available details and the absence of slow memory leaks I would suspect that you sometimes have a runaway process which consumes all the memory fast, forcing swap usage to 100% and then this process gets OOM-killed. You need to check your logs for OOM messages, they should contain information about the killed process and its memory consumption. Once you identified the runaway process you can set its memory limits in its systemd unit so it gets killed before consuming all system resources.

You can also inspect memory utilisation by the individual processes at any moment by running top -o RES which would sort the processes by their resident memory consumption.

You can install something simple like monit to monitor both system memory/swap consumption and per-process memory consumption for selected processes.

2
  • thanks for your detailled answer. I think this is what happen. my question is the following, when the process are moved in swap because of one process that use all the RAM, is it normal that the swap is kept filled at 100% even if the RAM is now free ? maybe moved process in the swap were "sleeping process" ? also about your proposl for limiting ram for process, can I configure a limit for all process called "vsim" or is the limit only for one process (using its pid)
    – iostrym
    Commented Feb 13 at 8:16
  • The least used memory goes to the swap first and if it is still unused then it remains in the swap. It is normal to have the swap fill up under memory pressure and remain filled up after the pressure is gone. Your system has a lot of inactive (potentially swappable) memory - 150-250 GB in different reports. You can't limit memory for a process by a name but you can limit memory for a group of processes if they are all spawned from a single process within a systemd unit. It is better to ask this as a separate question or search existing answers.
    – AlexD
    Commented Feb 14 at 6:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .