How to check and analyze memory usage on a Solaris system

Very often, gentlemen, system administrators are faced with the problem of viewing the operating system load, and if most are familiar with monitoring systems on standard operating systems, then in Solaris, this often raises many questions and doubts, especially in the conclusions of commands that are frightening with their numbers. Today we will try to understand this a bit.

Initially, we are interested in three questions, consider them in order:
1. What affects system performance?

• Memory
• CPU
• Disk
• Network

2. General cases of performance problems

• Insufficient load performance tests lead to uncalculated loads on the existing system.
• The emergence of a lack of system resources as a result of increasing the load on an existing application
• Problems of software and hardware complex

3. What concepts are used during data collection?
It is necessary to have two types of data, the first collected at the time when the system is under critical loads, the second during the stable operation of the system.
4. Finally, what kind of means do we have in our system, and what are they responsible for?

• vmstat - memory, processor, I / O
• mpstat - CPU
• iostat - I / O
• netstat - network
• lockstat - system calls, locks
• sar - a multi-purpose utility, it all depends on the options
• other.

I will try to gradually tell about these and some other utilities, variants of conclusions, and reading of these conclusions. Let's start with the most necessary, analyzing the use of RAM.
We analyze memory usage: first, we need to find out how much memory is occupied by the kernel, and how much by the user, our next steps will depend on it:
root@T1000-spare # echo "::memstat" | mdb -k
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 54267 423 5%
Anon 25457 198 3%
Exec and libs 4346 33 0%
Page cache 9650 75 1%
Free (cachelist) 10775 84 1%
Free (freelist) 908455 7097 90%

Total 1012950 7913
Physical 1007634 7872
User memory usage: shows the processes using the memory most
root@T1000-spare # prstat -s rss
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
1238 noaccess 155M 117M sleep 59 0 0:01:01 0.0% java/18
662 root 18M 15M sleep 59 0 0:00:05 0.0% fmd/26
7 root 16M 13M sleep 59 0 0:00:09 0.0% svc.startd/12
9 root 12M 11M sleep 59 0 0:00:34 0.0% svc.configd/17
697 root 10M 8304K sleep 59 0 0:00:00 0.0% snmpd/1
489 root 50M 8048K sleep 59 0 0:00:00 0.0% gdm-binary/1
446 root 9744K 6352K sleep 59 0 0:00:01 0.0% inetd/4
1773 root 7464K 6152K sleep 59 0 0:00:00 0.0% sshd/1
184 root 6704K 4728K sleep 59 0 0:00:00 0.0% nscd/31
187 root 5536K 4720K sleep 59 0 0:00:00 0.0% picld/9
1772 root 5680K 4312K sleep 59 0 0:00:00 0.0% sshd/1
667 root 9608K 4184K sleep 59 0 0:00:00 0.0% sendmail/1
142 root 6832K 3784K sleep 59 0 0:00:00 0.0% syseventd/15
664 smmsp 9672K 3744K sleep 59 0 0:00:00 0.0% sendmail/1
202 daemon 4984K 3728K sleep 59 0 0:00:00 0.0% kcfd/5
2019 oracle 3880K 3600K cpu17 59 0 0:00:00 0.0% prstat/1
==================================================
The following command shows the shared memory and “semaphore” output:
root @ T1000-spare # ipcs -a (unfortunately this server is test and only just raised, I can't show the load)
The following command will show us the use of user memory by all processes, except (PID 0,2,3):
root@T1000-spare # egrep "[0-9]:|^total" /var/tmp/pmap-x
1: /sbin/init
total Kb 3024 2712 312 -
1112: /usr/dt/bin/dtlogin -daemon
total Kb 6576 4536 304 -
1238: /usr/java/bin/java -server -Xmx128m -XX:+UseParallelGC -XX:ParallelGCT
total Kb 158552 120040 103304 -
142: /usr/lib/sysevent/syseventd
total Kb 6832 5832 1096 -
146: /usr/lib/ldoms/drd
total Kb 2600 2320 408 -
1539: /home/OracleDB/11.2/ccr/bin/nmz -cron -silent
total Kb 3936 3152 264 -
1772: /usr/lib/ssh/sshd
total Kb 5680 5352 728 -
1773: /usr/lib/ssh/sshd
total Kb 7464 6928 408 -
===================================================
The next step is to check the use of the / tmp file system:
root@T1000-spare # df -kl /tmp/
Filesystem kbytes used avail capacity Mounted on
swap 14380920 104 14380816 1% /tmp
===================================================
root @ T1000-spare # du -akd / tmp | sort -n | tail -5 - and this command will display 5 large files inside this file system.
Total system memory usage:
root@T1000-spare # vmstat -p 3
memory page executable anonymous filesystem
swap free re mf fr de sr epi epo epf api apo apf fpi fpo fpf
14314224 7375848 26 67 1 0 13 23 0 0 0 0 0 40 1 1
14381608 7361752 3 6 0 0 0 0 0 0 0 0 0 0 0 0
14381288 7361616 0 0 0 0 0 0 0 0 0 0 0 0 0 0
=============================================
This command will display information every three seconds.
Of course, you must check the use of SWAP system:
root@T1000-spare # swap -s
total: 193424k bytes allocated + 47176k reserved = 240600k used, 14381936k available
root@T1000-spare # swap -l
swapfile dev swaplo blocks free
/dev/dsk/c0t0d0s1 32,9 16 16387600 16387600
===================================================
In this section, the final command is to check the statistics of the kernel: a very good utility kstat, has a large variety of additional keys, you can also use the command I indicated below
root@T1000-spare # echo "::kmastat"| mdb -k > /var/tmp/kmastat
root@T1000-spare # more /var/tmp/kmastat
cache buf buf buf memory alloc alloc
name size in use total in use succeed fail
------------------------- ------ ------ ------ --------- --------- -----
kmem_magazine_1 16 1161 1524 24576 1161 0
kmem_magazine_3 32 2184 2540 81920 2805 0
kmem_magazine_7 64 1730 2413 155648 3043 0
kmem_magazine_15 128 2701 2772 360448 3401 0
kmem_magazine_31 256 337 341 90112 337 0
kmem_magazine_47 384 0 0 0 0 0

In this table, you need to look at the “memory in use” column and at any line in which “alloc fail” is greater than zero.
Well, I think for one article so far enough, I don’t want to load you with a large canvas of text I hope soon I can continue my review on monitoring pets on Solaris.