memory usage per process

[Sierp]

Hello,

I'm thinking about some very simple thing - I was assuming it would be
simple. But it's not that simple.
I'm talking about memory usage, real memory usage for PID and for all
machines PIDs.

Let's say I'm looking at nmon output, and see something like this:

Physical PageSpace
MB Used 15156 2256
MB Free 1740 3887
Total(MB) 16896 6144

Is there any command I could use to check, what exactly is consuming
this 17412MB of RAM?
I know, that some part of memory is used for filesystemcache - in this
case, 10%.
I was thinking about svmon -Pu and taking "Inuse" field, for example I have

svmon -Pu

Pid Command Inuse Pin Pgsp Virtual .....
11111 oracle 755797 65584 303066 746923

but when I add all Inuse/4/1024=something in MB, I recive something more
than 22GB. As I assumed from manual for svmon, this might be becouse of
shared memory between processes, which is every time counted separatly.

In the other hand, when I try to do something like this:

for PID in `ps -ef | awk '{ print $2}'`; do
ps v $PID | grep -v COMMAND | awk '{ print $7}';
done

which means I have all RSS outputs for all PIDs on this machine - and
when I sum all this RSS outputs - I receive value of 4835808, which
means about 4722MB used. About 1700MB is used for filesystem cache.
So this is all I know - I know what is happening with 6,5GB of my RAM,
but what about 10GB of it?
What is utilizing it?

I would be gratefull for good answer, I was googling for it and to be
honest, I couldn't find the answer except what I wrote above.

regards,

Sierp

[Andreas Schulze]

                Physical        PageSpace
MB Used         15156           2256
MB Free         1740            3887
Total(MB)       16896           6144
Is there any command I could use to check, what exactly is consuming
this 17412MB of RAM?
I know, that some part of memory is used for filesystemcache - in this
case, 10%.
I was thinking about svmon -Pu and taking "Inuse" field,[...]

You are on the right track by using svmon to identify memory usage.
I'd like to suggest to you that you use the system wide view with
svmon -S instead of the process view with svmon -P. With the system
wiede view every memory segment will show up once and you can also see
memory segments that don't belong to a process.
If you want to know which process uses a certain memory segment use
svmon -P <PID> -l and/or svmon -Pns. Look out for same virtual segment
IDs vsid and shmat/mmap description.
A good starting point for reading IMHO would be the AIX documentation
at the Information Center:
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.prftungd/doc/prftungd/mem_usage.htm

[Sierp]

Post by Andreas Schulze
You are on the right track by using svmon to identify memory usage.
I'd like to suggest to you that you use the system wide view with
svmon -S instead of the process view with svmon -P. With the system
wiede view every memory segment will show up once and you can also see
memory segments that don't belong to a process.
If you want to know which process uses a certain memory segment use
svmon -P <PID> -l and/or svmon -Pns. Look out for same virtual segment
IDs vsid and shmat/mmap description.

ok.. is there any little bit more simple way to do it? :-)
svmon -P PID -l is.. well.. long ;-) and very hard to script it

Post by Andreas Schulze
A good starting point for reading IMHO would be the AIX documentation
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.prftungd/doc/prftungd/mem_usage.htm

yes, I know, I've read it already and this is why I'm asking this question :)

regards,

Sierp

[Mark Taylor]

You probably just was RSS don't you ? i.e. SZ from ps -leaf .. or "ps -
efF vsz,comm,pid | sort -nr" which excludes all of the other crap
like shared libs, persistient blah ..

HTH
Mark Taylor

[Sierp]

Post by Mark Taylor
You probably just was RSS don't you ? i.e. SZ from ps -leaf .. or "ps -
efF vsz,comm,pid | sort -nr" which excludes all of the other crap
like shared libs, persistient blah ..

but if you sum SZ from all processes you will not receive all your
memory consumption, am I right?

Sierp

[Henry]

Post by Sierp

Post by Mark Taylor
You probably just was RSS don't you ? i.e. SZ from ps -leaf .. or "ps -
efF vsz,comm,pid | sort -nr" which excludes all of the other crap
like shared libs, persistient blah ..

but if you sum SZ from all processes you will not receive all your
memory consumption, am I right?
Sierp

my numbers still don't add up from the svmon -G and svmon -S
sudo ksh svmon-G.sh

mem_size mem_inuse mem_free mem_pin mem_virtual pg_space_size
pg_space_inuse pin_work pin_pers pin_clnt in_use_work in_use_pers
in_use_clnt
9216 8831 385 3903 7680
2048 1121 3534 0 0 7376
0 1454

sudo ksh sum_svon-S.sh

total clnt 1175.1
total work 1067.91
total work kernel heap 1180.05
total work kernel segment 31.918
total work mbuf pool 2328.38
total work other kernel segments 129.297
total work mmap source 2208.07
Grand total 8120.72

[Andreas Schulze]

Post by Henry

Post by Sierp

Post by Mark Taylor
You probably just was RSS don't you ? i.e. SZ from ps -leaf .. or "ps -
efF vsz,comm,pid | sort -nr"  which excludes all of the other crap
like shared libs, persistient blah ..

but if you sum SZ from all processes you will not receive all your
memory consumption, am I right?

Yes.

Post by Henry

Post by Sierp
Sierp

[...]
my numbers still don't add up from the svmon -G and svmon -S
sudo ksh svmon-G.sh
[...]

svmon -G and svmon -S should return an almost identical number for
working storage AVM. It is not identical because of some differences
in handling pinned memory. The output of these commands is useful for
a global view at AIX memory usage.

With the svmon -P command you get a lot of duplicate entries for inuse
usually. E.g. a work kernel and a work shared library text segment. At
the same time some memory used will not show up at al with the process
view, e.g. files that were opened before (by a process that stopped/
ended already) and are kept in cache for possible reuse later.

[Sierp]

Post by Andreas Schulze
With the svmon -P command you get a lot of duplicate entries for inuse
usually. E.g. a work kernel and a work shared library text segment. At
the same time some memory used will not show up at al with the process
view, e.g. files that were opened before (by a process that stopped/
ended already) and are kept in cache for possible reuse later.

in my case there was some additional issue
I had a lot of memory reserved in shared memory segment, which could be
seen by ipcs -ap
The problem is (I think) that the PID which reserved this memory wasn't
alive (there was no such PID in OS at all). I could remove this shared
memory segment with ipcrm, but to be honest, I don't know what are the
implications of this issue for system (when there is no PID)

I wonder, why this memory was still reserved (there were oracle user's),
and did I broke anything ;-)

regards,

Sierp

[Thomas Braunbeck]

Post by Sierp

Post by Andreas Schulze
With the svmon -P command you get a lot of duplicate entries for inuse
usually. E.g. a work kernel and a work shared library text segment. At
the same time some memory used will not show up at al with the process
view, e.g. files that were opened before (by a process that stopped/
ended already) and are kept in cache for possible reuse later.

in my case there was some additional issue
I had a lot of memory reserved in shared memory segment, which could be
seen by ipcs -ap
The problem is (I think) that the PID which reserved this memory wasn't
alive (there was no such PID in OS at all). I could remove this shared
memory segment with ipcrm, but to be honest, I don't know what are the
implications of this issue for system (when there is no PID)
I wonder, why this memory was still reserved (there were oracle user's),
and did I broke anything ;-)
regards,
Sierp

Why 'reserved"? This is shared memory, one process creates it, others
can use it (the idea behind shared memory). And why should the system
remove this shared memory in case the creator terminaes? There can be
still users (other processes) needing it.

[Sierp]

Post by Thomas Braunbeck
Why 'reserved"? This is shared memory, one process creates it, others
can use it (the idea behind shared memory).

ok, the problem is, all oracle processes were stopped - this memory was
still reserved, I don't think something was using it

Post by Thomas Braunbeck
And why should the system
remove this shared memory in case the creator terminaes? There can be
still users (other processes) needing it.

I'm not writing that system should remove this shared memory, I just
wonder why oracle didn't do it

regards

Sierp

[Thomas Braunbeck]

Post by Sierp
I'm not writing that system should remove this shared memory, I just
wonder why oracle didn't do it

Some type of failure (memory fault - core dump) or a kill -9 of
the process supposed to cleanup. Check the oracle logs and the
errpt -a.

[Henry]

Post by Sierp
Hello,
I'm thinking about some very simple thing - I was assuming it would be
simple. But it's not that simple.
I'm talking about memory usage, real memory usage for PID and for all
machines PIDs.
Physical PageSpace
MB Used 15156 2256
MB Free 1740 3887
Total(MB) 16896 6144
Is there any command I could use to check, what exactly is consuming
this 17412MB of RAM?
I know, that some part of memory is used for filesystemcache - in this
case, 10%.
I was thinking about svmon -Pu and taking "Inuse" field, for example I have
svmon -Pu
Pid Command Inuse Pin Pgsp Virtual .....
11111 oracle 755797 65584 303066 746923
but when I add all Inuse/4/1024=something in MB, I recive something more
than 22GB. As I assumed from manual for svmon, this might be becouse of
shared memory between processes, which is every time counted separatly.
for PID in `ps -ef | awk '{ print $2}'`; do
ps v $PID | grep -v COMMAND | awk '{ print $7}';
done
which means I have all RSS outputs for all PIDs on this machine - and
when I sum all this RSS outputs - I receive value of 4835808, which
means about 4722MB used. About 1700MB is used for filesystem cache.
So this is all I know - I know what is happening with 6,5GB of my RAM,
but what about 10GB of it?
What is utilizing it?
I would be gratefull for good answer, I was googling for it and to be
honest, I couldn't find the answer except what I wrote above.
regards,
Sierp

I'll be interested because I too have done a lot of work with svmon,
ps (and awk).

I wrote this to translate the svmon -P output to MB and have tried all
sorts with running totals of all procs

cat /opt/local/bin/svmon-P.awk
#!/usr/bin/awk
BEGIN{ printf("%-10s%20s%10s%10s%10s%10s\n",
"Pid","Command","Inuse","Pin","Pgsp","Virtual")}
{
if ( /Pid Command/ )
{
#print
getline
printf("%-10s%20s%10.0f%10.0f%10.0f%10.0f\n", $1, $2, ($3*4/1024),
($4*4/1024), ($5*4/1024), ($6*4/1024))
sum_Inuse+=($3*4/1024)
sum_Pin+=($4*4/1024)
sum_Pgsp+=($5*4/1024)
sum_Virtual+=($6*4/1024)
}
}
END{
printf("%-30s%10.0f%10.0f%10.0f%10.0f\n", "Totals in MB", sum_Inuse,
sum_Pin, sum_Pgsp, sum_Virtual)


so I run this to see if DB2 can be calculated - I don't believe its
correct

#!/bin/ksh
# $Id$

PATH=/usr/bin

function svmon_pid {
# check svmon memory mapping

# see if there's any DB2 processes running
# the ps "comm" and "args" produce different output for the "db2sysc"
command
list_of_db2_pids=$( ps -e -o user,pid,ppid,etime,time,pcpu,args | awk
'$7 ~ "db2sysc|db2agent|db2ckpwd|db2fmp|db2acd" && $5 !=
"00:00:00" {print $2}')
if [[ -n $list_of_db2_pids ]] ; then

# need root for svmon
if [[ $(whoami) != "root" ]]; then
print have to be root user, exiting
exit
fi

echo "svmon stats"
for db2_pid in $list_of_db2_pids ; do
# some output modifications
svmon -P $db2_pid | awk -f /opt/local/bin/svmon-P.awk
done | awk '{print}/Totals in MB/ {sum_Totals+=$4;+
+num_db2_hits}END {print "Totals of all DB2 " sum_Totals " MB - ",
num_db2_hits " PIDS\n"}'
else
echo no DB2 running
exit 0
fi
}

function db2_by_user {
# look through process table for DB2
echo "only show processes using TIME and CPU"
ps -e -o user,pid,ppid,etime,time,pcpu,args| awk '
BEGIN { printf("\n%-10s%-10s%-10s%-15s%-15s%-10s\n", "UID", "PID",
"PPID", "ETIME", "TIME", "CPU%") }
$7 ~ "db2sysc|db2agent|db2ckpwd|db2fmp|db2acd" && $5 != "00:00:00" &&
$6 != "0.0" {
UID=$1
PID=$2
PPID=$3
ETIME=$4
TIME=$5
PCPU=$6
printf("%-10s%-10s%-10s%-15s%-15s%-10s\n" , UID, PID , PPID, ETIME,
TIME, PCPU)
}'

}
# entry point
svmon_pid
db2_by_user

and running it

sudo show_db2_size.sh
svmon stats
Pid Command Inuse Pin Pgsp Virtual
262182 db2sysc 135 31 2 144
Totals in MB 135 31 2 144
Pid Command Inuse Pin Pgsp Virtual
266340 db2sysc 136 31 2 144
Totals in MB 136 31 2 144
Pid Command Inuse Pin Pgsp Virtual
323726 db2sysc 136 31 2 145
Totals in MB 136 31 2 145
Pid Command Inuse Pin Pgsp Virtual
372760 db2sysc 232 31 3 241
Totals in MB 232 31 3 241
Pid Command Inuse Pin Pgsp Virtual
377068 db2fmp 125 31 2 133
Totals in MB 125 31 2 133
Pid Command Inuse Pin Pgsp Virtual
434226 db2sysc 135 31 2 144
Totals in MB 135 31 2 144
Pid Command Inuse Pin Pgsp Virtual
462916 db2sysc 136 31 2 145
Totals in MB 136 31 2 145
Pid Command Inuse Pin Pgsp Virtual
475318 db2sysc 136 31 2 145
Totals in MB 136 31 2 145
Pid Command Inuse Pin Pgsp Virtual
479232 db2fmp 181 31 5 192
Totals in MB 181 31 5 192
Pid Command Inuse Pin Pgsp Virtual
528400 db2sysc 162 31 2 171
Totals in MB 162 31 2 171
Pid Command Inuse Pin Pgsp Virtual
561222 db2sysc 321 31 3 329
Totals in MB 321 31 3 329
Pid Command Inuse Pin Pgsp Virtual
618522 db2fmp 125 31 2 133
Totals in MB 125 31 2 133
Pid Command Inuse Pin Pgsp Virtual
630910 db2fmp 131 31 2 139
Totals in MB 131 31 2 139
Pid Command Inuse Pin Pgsp Virtual
721076 db2sysc 162 31 2 171
Totals in MB 162 31 2 171
Pid Command Inuse Pin Pgsp Virtual
802824 db2sysc 162 31 2 171
Totals in MB 162 31 2 171
Pid Command Inuse Pin Pgsp Virtual
843922 db2fmp 125 31 2 133
Totals in MB 125 31 2 133
Pid Command Inuse Pin Pgsp Virtual
929960 db2fmp 133 31 2 141
Totals in MB 133 31 2 141
Pid Command Inuse Pin Pgsp Virtual
970770 db2sysc 733 31 3 722
Totals in MB 733 31 3 722
Totals of all DB2 3406 MB - 18 PIDS

only show processes using TIME and CPU

UID PID PPID ETIME TIME CPU%
toltt1in 970770 938066 57-15:12:03 2-23:53:08 1.3