Who is leaking memory in my Linux Jessie/KDE4 installation?

[Omer Zak]

I have a 8GB PC which runs Linux Debian Jessie with KDE 4.4.
My problem is to find out who is occupying almost 4GB memory some time
after rebooting, even when nothing heavy is running.

The heaviest applications that I run are:
- A VirtualBox virtual machine occupying 3GB memory
- Google Chrome browser (version 37.0.2062.94, 64-bit)
- Evolution 3.12.2.

However, even when they are closed, a lot memory is still reported to be
in use.

My question is: besides top, what tools can be used to find who is using
all this memory?
The next question, of course, is how to get rid of those memory hogs
without destabilizing the system.

--- Omer
--
More proof the End of the World has started. Just saw this online:
I think it's beginning! Ten minutes ago there was a group of people
waiting at the bus stop outside my house. Now, they're all gone!
My own blog is at http://www.zak.co.il/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html

[shimi]

First things first: What is "Used Memory" in Linux?

Most people think that "Used Memory" means "used by my running
applications".

When Linux says "used", it means "used for any purpose", and furthermore,
"used" does NOT mean "total - available for applications". So what one sees
as "free" does not mean that's the amount of memory available for
applications. It can, and usually is, a much higher number.

In Linux, everything in use, including by the kernel, for purposes of
caches and buffers, i*s* ALSO "used", but, when memory is needed by
applications, these buffers and caches can, and are, being discarded (after
all, they're cache, they don't contain anything not written elsewhere, e.g.
on disk). Usually very little buffers would not be on the disk - such ones
cannot be reclaimed as free memory before they're flushed.

So, one should add 'cached' and 'buffers' from 'top' to be considered as
the 'free memory pool', and not take the 'free' number as the real free
memory. Linux, in its way of operation, will always have a growing and
growing 'cached' value. This is OK, by design, and part of the thing that
makes it so fast. When RAM is needed, cache is evicted. There are a few
examples for this on www.linuxatemyram.com

Now, there are a few other places where RAM can be taken, which do not
count towards 'cached', even though they're cache. There's the SLAB. You
can examine it by running cat /proc/slabinfo (as root). There's even a
top-like utility for it: slabtop(1).

Some of the Slab is reclaimable for use (you can 'grep Reclaim
/proc/meminfo '), some is not. Likely lots of Slab would be for dentry
cache, especially if you're opening many many files. Some buggy-designed
software does this (for example nss... which is unfortunately used by
default in cURL SSL connections if you've not compiled cURL to use OpenSSL
instead...). See:
https://www.splyt.com/blog/2014-05-16-optimizing-aws-nss-softoken

I would also appreciate others insights on the subject :)

HTH,

-- Shimi

Post by Omer Zak
I have a 8GB PC which runs Linux Debian Jessie with KDE 4.4.
My problem is to find out who is occupying almost 4GB memory some time
after rebooting, even when nothing heavy is running.
- A VirtualBox virtual machine occupying 3GB memory
- Google Chrome browser (version 37.0.2062.94, 64-bit)
- Evolution 3.12.2.
However, even when they are closed, a lot memory is still reported to be
in use.
My question is: besides top, what tools can be used to find who is using
all this memory?
The next question, of course, is how to get rid of those memory hogs
without destabilizing the system.
--- Omer
--
I think it's beginning! Ten minutes ago there was a group of people
waiting at the bus stop outside my house. Now, they're all gone!
My own blog is at http://www.zak.co.il/tddpirate/
My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html
_______________________________________________
Linux-il mailing list
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il

[Oleg Goldshmidt]

Post by Omer Zak
I have a 8GB PC which runs Linux Debian Jessie with KDE 4.4.
My problem is to find out who is occupying almost 4GB memory some time
after rebooting, even when nothing heavy is running.

What is your question? The subject mentions "leaking", while the body of
your post seems to indicate (I am guessing a bit) that what bothers you
is that most of your memory is reported as "used" rather than as
"free" or "available".

If an application really leaks memory you should normally be able to
observe it by running top or similar and seeing, say, RES increasing
with time.

If your problem is "too much memory is reported as used" then you should
know that most of memory is "used" by something. This does not mean it
is not available. The classic case is memory that is - or was - used for
buffers or cache.

Post by Omer Zak
However, even when they are closed, a lot memory is still reported to
be in use.

Operating systems do not free memory until someone requests it - again,
your buffers and cached stuff are primary examples, as explained
above. If you look at the "Mem" line of your top(1) or free(1) it will
*always* report most of your memory as "used". Here is my free(1) output
on an old laptop with 4G of RAM:

$ free
total used free shared buffers cached
Mem: 3941664 3522680 418984 0 218636 2208160
-/+ buffers/cache: 1095884 2845780
Swap: 6029308 113872 5915436

How much memory is used? About 1GB, actually, with ~3GB available. See
the 2nd line, which takes into account that whatever is listed as
buffers+cache are available for applications if/when they request it.

Check your free(1), if you see anything suspicious, post it.

Post by Omer Zak
My question is: besides top, what tools can be used to find who is using
all this memory?

Find out the PIDs of the processes you suspect (make top sort by RES by
typing Fq in top?) and do

egrep "^Vm" /proc/<pid>/status

for each - it will give you a lot of detailed info. If you suspect a
leak repeat a few times while the suspect process is running (and doing
something).

Relevant details have been discussed here before (guilty). Check

http://www.mail-archive.com/linux-il-NSemkxREmS1YZAO8hgG6+***@public.gmane.org/msg31797.html
http://mailman.cs.huji.ac.il/pipermail/linux-il/2011-December/008322.html

- you will either find that useful or it will help you sleep (well).

Post by Omer Zak
The next question, of course, is how to get rid of those memory hogs
without destabilizing the system.

First, find out if there is a problem, and what the problem is.

[Omer Zak]

Hello Shimi and Oleg,
Thanks for helping me clarify what I mean by "memory leak" in this
context.

I routinely monitor memory usage by means of gnome-system-monitor, and
it appears to display memory consumption by watching the
used -/+ buffers/cache number.

As the time since last reboot lengthens, the above memory usage number
creeps upwards. I cannot get it back to the value it had shortly after
reboot by closing all heavy applications. This is why I am referring to
it as "memory leak".

This kind of memory usage concerns me, because when more than 90% of
memory is being used, the system slows down. It is not because swap
memory is used (swap usage remains 0% at all times) but apparently
because there is not enough free memory to buffer/cache the entire
working set of frequently-used files.

When the system slows down, I reboot it and few more days pass before
memory usage (-/+ buffers/cache) again nears 90%.

So I am interested in finding who is hogging memory at expense of the
buffer/cache memory. Furthermore, I'd like to find out who is hogging
more and more memory as time passes on.

--- Omer

Post by Oleg Goldshmidt

Post by Omer Zak
I have a 8GB PC which runs Linux Debian Jessie with KDE 4.4.
My problem is to find out who is occupying almost 4GB memory some time
after rebooting, even when nothing heavy is running.

What is your question? The subject mentions "leaking", while the body of
your post seems to indicate (I am guessing a bit) that what bothers you
is that most of your memory is reported as "used" rather than as
"free" or "available".
If an application really leaks memory you should normally be able to
observe it by running top or similar and seeing, say, RES increasing
with time.
If your problem is "too much memory is reported as used" then you should
know that most of memory is "used" by something. This does not mean it
is not available. The classic case is memory that is - or was - used for
buffers or cache.

Post by Omer Zak
However, even when they are closed, a lot memory is still reported to
be in use.

Operating systems do not free memory until someone requests it - again,
your buffers and cached stuff are primary examples, as explained
above. If you look at the "Mem" line of your top(1) or free(1) it will
*always* report most of your memory as "used". Here is my free(1) output
$ free
total used free shared buffers cached
Mem: 3941664 3522680 418984 0 218636 2208160
-/+ buffers/cache: 1095884 2845780
Swap: 6029308 113872 5915436
How much memory is used? About 1GB, actually, with ~3GB available. See
the 2nd line, which takes into account that whatever is listed as
buffers+cache are available for applications if/when they request it.
Check your free(1), if you see anything suspicious, post it.

Post by Omer Zak
My question is: besides top, what tools can be used to find who is using
all this memory?

Find out the PIDs of the processes you suspect (make top sort by RES by
typing Fq in top?) and do
egrep "^Vm" /proc/<pid>/status
for each - it will give you a lot of detailed info. If you suspect a
leak repeat a few times while the suspect process is running (and doing
something).
Relevant details have been discussed here before (guilty). Check
http://mailman.cs.huji.ac.il/pipermail/linux-il/2011-December/008322.html
- you will either find that useful or it will help you sleep (well).

Post by Omer Zak
The next question, of course, is how to get rid of those memory hogs
without destabilizing the system.

First, find out if there is a problem, and what the problem is.

--
Your liberty to swing your fist ends just where my nose begins.
Your freedom of expression ends where my freedom of expression begins.
Your freedom of religion ends where my rights for equality and
accessibility begin.
My own blog is at http://www.zak.co.il/tddpirate/

My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html

[Oleg Goldshmidt]

Post by Omer Zak
This kind of memory usage concerns me, because when more than 90% of
memory is being used, the system slows down.

Every system I use is at or above 90% memory consumption. Always. It is
normal. The laptop I am typing on is at present at 88.5%, running KDE,
firefox, emacs, a konsole, and nothing else of any significance.

Here is how you investigate it.

Please post the output of your free command, and the top 4-5 lines of
top (excluding the header), sorted by RSS. For the latter, start top,
press F (to select the sort field), then q (to sort by the resident set
size - top knows it as RES). The result will look like

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24689 oleg 20 0 1745m 450m 37m S 1.7 11.7 99:48.48 firefox
1632 oleg 20 0 3324m 143m 31m S 0.0 3.7 16:44.52 plasma-desktop
1845 oleg 20 0 706m 70m 13m S 0.3 1.8 9:38.23 emacs

Then please also post the memory usage details of the top "hog". In my
case above it is firefox, with PID 24689:

$ egrep "^Vm" /proc/24689/status
VmPeak: 1990664 kB
VmSize: 1787052 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 723388 kB
VmRSS: 464648 kB
VmData: 954040 kB
VmStk: 1048 kB
VmExe: 80 kB
VmLib: 95980 kB
VmPTE: 3324 kB
VmSwap: 0 kB

The RSS (resident set size, this is the memory mapped into the process's
virtual memory space) is ~465kB, close to the 450M that top reports (it
may fluctuate a bit, and there may be 1024/1000 involved in random cases
- I can never remember -> 450*1.024=460.8; NB: this corresponds to 11.7%
of my 4GB RAM, as top reports). The high watermark of RSS, VmHWM, is
~720MB - FF never used more memory than that. This is also a hint that
it does not leak memory - it uses much less now than HWM. Note that the
heap (VmData) is more than twice as large as RSS, but it is not (not all
of it, anyway) mapped into VM. It may be (pre-)allocated but it does not
mean that the memory is not available to other processes on demand. The
stack is ~1MB. 96MB is in shared libs - the part used by FF and mapped
into its process space, so while it is used by FF it does not belng
exclusively to it. I mention this because the shared part may be large
compared to RSS, depending on application. [Use "man 5 proc" for
reference.]

Think of it this way: applications (and the OS) may be smart, and they
may pre-load, pre-map, pre-fetch into RAM all sorts of stuff that is not
really in use but may be used at some point. If there is memory
available, why not do it? On top of it, memory that was allocated and
used and is no longer in use will not be freed unless there is
demand. If no one requests it, why bother? If more processes are started
and demand memory the memory will be re-allocated. This is why most
memory is always marked as used. Only if the sum of the resident sizes of
the running processes exceeds the physical memory size will you run into
trouble, trouble being either performance problems associated with swap
usage or OOM killer springing into action.

I hope the above will help you in your investigations. Frankly, I
suspect you do not have a real problem.

[Dolev Farhi]

_______________________________________________
Linux-il mailing list
Linux-il-NSemkxREmS1YZAO8hgG6+***@public.gmane.org
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il