A Closer Look at Processes on the Linux Operating System

On Linux there are a number of utilities which allow the System Administrator to examine the status of processes and to adjust their relative priorities or change their operational status. Some of these capabilities can be demonstrated by examining one programme available for the Linux OS : the Apache httpd daemon : a freely available web server. In the following prints of screen outputs some columns have been ommitted where they do not affect the demonstration being given.

From the following it can be seen that the size of the http daemon is 142699 bytes on the disk. Within the ELF file there are 108786 bytes of code, 4796 bytes of initialised data and 19015 bytes of uninitialiased data.

orion-1:# ls -l httpd

-rwxr-x--- 1 root root 142699 Oct 5 1996 httpd*

orion-1:# size httpd

text | Data | bss | dec | hex | filename
108786 | 4796 | 19015 | 132597 | 205f5 | httpd

Performing an object dump of the file shows that the code section is offset within the file and starts at hexadecimal address 0x1ce0, read only data starts at 0x16238, other data at 0x1a9e8 and uninitialised data at 0x1bcb0. There is a table of debugging symbols at 0x1bcb0 :

orion-1:# objdump --headers httpd

httpd: file format elf32-i386

Sections: < Some sections excluded for clarity > Idx Name Size VMA LMA File off Algn

0 .interp 00000013 080000d0 080000d0 000000d4 2**0

CONTENTS, ALLOC, LOAD, READONLY, DATA

8 .text 00014544 08001ce0 08001ce0 00001ce0 2**4

CONTENTS, ALLOC, LOAD, READONLY, CODE

10 .rodata 000047ac 08016238 08016238 00016238 2**2

CONTENTS, ALLOC, LOAD, READONLY, DATA

11 .data 00001050 0801b9e8 0801b9e8 0001a9e8 2**2

CONTENTS, ALLOC, LOAD, DATA

16 .bss 00004a47 0801ccb0 0801ccb0 0001bcb0 2**4

ALLOC

17 .stab 000004f8 00000000 00000000 0001bcb0 2**2

CONTENTS, READONLY, DEBUGGING

The http daemon utilises functions from the standard 'C' library, linked as a dynamic linked library (the 'so' stands for 'shared object') :

orion-1:# ldd -r httpd

libc.so.5 => /usr/local/lib/libc.so.5 (0x4000a0)

The daemon is started at system boot time and was sleeping (S) when its status was checked, it is consuming very little CPU time :

orion-1:# ps -fc | grep httpd

PID TTY STAT TIME COMMAND

90 ? S 0:00 httpd

A printout of the tree of all processes running on the computer and their interrelationships shows that the httpd has init as its parent process, this is typical of a Unix daemon process. It has also created three child processes which it now controls, presumably to listen for http connections.

orion-1:# pstree init-+-4*[agetty]

|-bash--script--script--bash--script--script--bash--pstree

|-bash

|-crond

|-gpm

|-httpd--3*[httpd]

|-inetd

|-kflushd

|-klogd

|-kswapd

|-lpd

|-rpc.mountd

|-rpc.nfsd

|-rpc.portmap

|-sendmail

|-syslogd

`-update

This is confirmed by examining a few of the processes running in more detail. The init process has a Process ID (PID) of 1. It can be seen that the parent PID of the httpd is 1 as expected.

It can be seen that the resident memory footprint (Resident Set Size (RSS)) of the httpd process is 528 kilobytes whereas its overall size in virtual memory is 1012 Mb. It can be seen that the process runs with a User ID of 0 (owned by the administrator (root)) and that it is running at the default priority of 0 (-20 being maximum real-time and 20 being the lowest priority). The SW< status for the kernel swap daemon (kswapd) process shows that it is sleeping, has no resident pages and has a priority less than 0.

orion-1:# ps -cl1,2,3,90

F UID PID PPID PRI NI SIZE RSS WCHAN STAT TTY TIME COMMAND

100 0 1 0 0 0 844 328 c01115c9 S ? 0:03 init [5]

40 0 2 1 0 0 0 0 c0111a38 SW ? 0:00 (kflushd)

40 0 3 1 -12 -12 0 0 c0111a38 SW< ? 0:00 (kswapd)

140 0 90 1 0 0 1012 528 c0119272 S ? 0:00 httpd

Looking at the virtual memory performances of the process it can be seen that the httpd has had 23 major page faults where a page fault represents an action to load a page of text from disk or buffer cache. The text (code) resident set size (TRS) is 24 kilobytes which suggests a code page size of 1024 bytes when viewed in association with the page fault number. The Data Resident Size is 108 kilobytes, giving a total SIZE of 132 kilobytes. The process shares 115 kilobytes with other processes (this may represent the standard 'C' shared library).

orion-1:# ps -cmp1,2,3,90

PID | MAJFLT | MINFLT | TRS | DRS | SIZE | SWAP | RSS | SHRD | COMMAND
2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | kflushd
3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | kswapd
1 | 206 | 49 | 5 | 77 | 82 | 0 | 82 | 65 | init
90 | 23 | 51 | 24 | 108 | 132 | 0 | 132 | 115 | httpd

The system administrator can re-prioritise the httpd process by changing it's NICE level. Here the priority of process 90 (httpd) is made higher by 10. Only the administrator can change priorities in a negative direction as shown. This prevents inexperienced users from overloading the CPU and also protects the system from hackers.

orion-1:# renice -10 90

90: old priority 0, new priority -10

The new priority level can now be seen, although on this system the httpd is idle, receiving no web access requests so it would not consume much more CPU.

orion-1:# ps -cl

UID | PID | PRI | PPID | NI | SIZE | RSS | STAT | TIME | COMMAND
0 | 1 | 0 | 0 | 0 | 844 | 328 | S | 0:03 | init [5]
0 | 2 | 0 | 1 | 0 | 0 | 0 | SW | 0:00 | (kflushd)
0 | 3 | -12 | 1 | -12 | 0 | 0 | SW< | 0:00 | (kswapd)
0 | 90 | -10 | 1 | -10 | 1012 | 528 | S < | 0:00 | httpd

The system is now loaded somewhat by running some CPU intensive shell scripts. The virtual memory statistics of the computer are examined every 5 seconds for five iterations.

The procs field shows that on average 2 processes are waiting for CPU run time (r) and on iteration 3 two processes are in uninterruptable sleep. There is no swapping activity going on as the processes are small enough to be fully resident in memory.

There is however I/O activity as blocks of data are brought in (bi) and sent out (bo) to I/O devices. The 'system' fields show that many interrupts are occurring per second (in) as well as many context switches (cs). The first iteration gives the average history since the last system reboot, whereas the subsequent four readings are for time now. It can be seen that the CPU is never idle (id) and is spending around 35% of its time in user mode and 65% of its time performing privileged operations in system mode.

This report is consistent with the known operations being performed : I/O intensive disk reads : the Linux 'find' command searching the disk.

orion-1:# vmstat 5 5

procs | memory | swap | io | system | cpu
r | b | w | swpd | free | buf | si | so | bi | bo | in | cs | us | sy | id
2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 10 | 8 | 133 | 34 | 6 | 8 | 86
2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 614 | 11 | 737 | 1234 | 32 | 68 | 0
1 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 100 | 129 | 435 | 366 | 40 | 60 | 0
2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 202 | 26 | 375 | 455 | 33 | 67 | 0
2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 230 | 0 | 331 | 472 | 31 | 69 | 0

It is possible to probe the Linux kernel and find which operations are consuming the most CPU load. When this is done it is seen that the kernel is spending most of it's time dealing with the directory cache, file system and writing to the console. The system scheduling process is only the twentieth most active process on the list.

To be able to read the profiling information from the kernel it must first be compiled with profiling enabled.

orion-1:# readprofile | sort -nr | head -20

CLK_TCK | Function | Normalised load
67649 | total | 0.0858
7796 | d_lookup | 54.1389
5425 | ext2_readdir | 3.5319
5034 | scrup | 7.8168
4573 | filldir | 14.8474
4481 | find_inode | 86.1731
2849 | ext2_check_dir_entry | 15.8278
2581 | getname | 7.9660
1885 | getblk | 2.2440
1665 | sys_newlstat | 6.8238
1546 | lookup_dentry | 5.1533
1542 | do_con_write | 0.3119
1425 | get_hash_table | 9.6284
1422 | cp_new_stat | 4.5577
1323 | __namei | 10.3359
1270 | system_call | 19.8438
1231 | ext2_getblk | 2.2140
1084 | raw_scan_sector | 1.8951
1077 | sys_getdents | 3.0597
973 | schedule | 1.2102

The Linux 'ps' command takes its information from the /proc virtual-file system (VFS) which is a disk image of the process structures as controlled by the kernel. Note that the VFS is a direct mirror of the kernel data structures and does not actually reside on disk, although it appears to do so when acessed with normal Linux commands.

Looking directly at the httpd process in the /proc file system, all of the information given by the administrative tools can be obtained, though often in a less user friendly format. The following is a direct dump of the httpd status table, with comments added for explanation :

orion-1:# cat /proc/90/status Name: httpd

State: S (sleeping)

Pid: 90 # Process ID

PPid: 1 # Parent Process (init)

Uid: 0 0 0 0 # User ID (root)

Gid: 65535 65535 65535 65535 # Group ID

VmSize: 1012 kB # Total virtual memory

VmLck: 0 kB # Total locked

VmRSS: 512 kB # Text Resident Set Size

VmData: 276 kB # Virtual Memory Data size

VmStk: 20 kB # Stack size

VmExe: 108 kB # Executable

VmLib: 576 kB # Library

SigPnd: 00000000 # Signals pending

SigBlk: 00000000 # Signals blocked

SigIgn: 80000000 # Signals ignored

SigCgt: 00006441 # Signals caught

The following are all the signals recognised by the Linux operating system, they are enumerated starting at zero (i.e. SIGHUP is 0, SIGKILL is 9, etc) :

orion-1:# fuser -l HUP INT QUIT ILL TRAP ABRT IOT BUS FPE KILL USR1 SEGV USR2 PIPE ALRM TERM STKFLT CHLD CONT STOP TSTP TTIN TTOU URG XCPU XFSZ VTALRM PROF WINCH IO PWR UNUSED