Performance Monitoring – Identifying Memory Bottlenecks
Memory bottlenecks are evidenced by two different things happening on the system — paging and swapping. Paging refers to pages of memory being reclaimed by the page daemon when the system starts to get low on free memory. Swapping is more extreme, and refers to entire processes being swapped out.
To determine if you are only paging, or also swapping, examine two columns in the vmstat output. The first column is the sr column. If the value in this column is greater than zero then the page scanner is scanning memory pages to put them back on the free list to be reused.
The page scanner runs when memory falls under the value of a system parameter known as lostfree – default value is 1/64th of physical memory – or cachefree if priority_paging is enabled default value is 1/128th of physical memory.
You should not worry about high scan rates if you are using the file system heavily. High scan rates can be normal in many circumstances. If priority_paging is enable, the page scanner steals the pages more effectively so the file system I/O does not cause unnecessary paging of applications. priority_paging causes sr rate to be higher for its own good. Solaris 8 introduces the cyclic cache. With cyclic cache, the scanner is not used to reclaim pages during file system I/O therefore if sr is greater than 0 then it’s a indication that the system is running low in memory.
To see if you are swapping, refer to the w column. It is the third column of the output, and refers to entire processes which are swapped out. You can determine what these processes are by running the command ‘ /usr/bin/ ps -e -o pid,rss,args ‘ and looking for a RSS of 0 (sched, pageout and fsflush processes should always have a RSS of 0).
If you have anything in the w column, you are either low on memory right now, or you have been in the past. If your system gets low on memory and processes are swapped out, it may take a long time for them to get back into memory. This is especially true if they are daemons which are not run often, because they have to receive an event in order to try to run again. This is not necessarily bad, as long as when they need to run, they will have the memory to do so.
If, over time, you see swapping, you should probably consider adding memory to the system or devising a strategy to low overall memory usage on the system.
Just made a quick 5 mins demo video to explain sample scenario related to memory bottleneck. The video is not a high quality video, but good enough to understand the simple scenario.