System Monitoring strace Tool for kernel space and non kernel space | Troubleshooting Performance Analysis

Table Of Contents

• 1. Monitoring
• 2. How can we monitor User space(non kernel) and system space(kernel)
• 3. Which Key Metrics we need to monitor the most on the applicationt
• 4. Testing 1 Firefox
• 5. Testing 2 dd
• 6. Testing 3 OBS
• 7. Testing 4 Virtualbox
• 8. conclusion

Monitoring 

Monitoring is a very vital important job. (All the hacking/disaster happens through  low standards/careless system maintains). Which may cause disaster in the setup. Good architecture of the system/platform make system standard reliable services. we can monitor system metric through lot of commands, in this article I am going to focus only on the user space and kernel space.what is command to monitor this user space and kernel space, it is strace.it will help to analyse the cpu systems call statistics.if you come know Your Linux operating system running very slow and you want to improve this article would be helpful.

Kernel Space	User Space
Run the Code in High privilege mode	Run the code unprivileged mode

How can we monitor User space(non kernel) and system space(kernel) ? 

Using strace command we can analyse the particular application  

1. system call(userspace / system space) and their respective counts  

we can get summarise details. The same information we can get it from the top command also. However strace will get clear summary.

Which Key Metrics we need to monitor the most on the application? 

 Every program will definitely utilize the memory +CPU. Because 

1. memory -> Loading program into memory 
2. CPU -> run the arithmetic operating system. 
3. 
   

CPU vs memory  

which is utilized most this information we can get it from the top command by sorting memory(M)/cpu(P) sort. 

To check Kernel space Vs User Space Please refer below. using strace command I have done some of the basic testing 

Testing 1 Firefox  

mostly used memory not CPU 

1. recvmsg – Receive message from a socket. Mostly rely on the memory rather than a CPU. 
2. 
   

Testing 2 dd  

 dd command used most Memory + CPU less ( with Kernel SPACE rather user space) I have done dd command to create a 10GB file. 

1. Read() - Read from a file descriptor 
2. write() - send a message to another user 

Reason : Since huge IO happens(Rely on Hardware) kernel space, it is not depend on the user space(Arithmetic Operation). 

Testing 3:  OBS  

while I am recording obs software Linux OBS, it is used most on CPU – USER SPACE rather kernel space. While I am checking system call it is used following system calls. 

1. Futex – fast user space locking. 
2. lseek - Repositing read/write file offset. 
3. Poll – wait for some event on a file descriptor 
4. writev() - read/write data into multiple buffers 
5. write() - send the message to another user. 
6. Fcntl – Manipulate file descriptor 

Reason : OBS Recording software will perform above operation most it depends on the arithmetic operating so User(non kernel space ) will increase. #strace <command> we can get this details 

Testing 4: Virtualbox  

while using Virtual box. It is used CPU System space rather user space. 

% time     seconds  usecs/call     calls    errors syscall

------ ----------- ----------- --------- --------- ----------------
 92.28   85.148852       18705      4552         3 poll
  3.03    2.799959         321      8720       702 futex

  1.44    1.327339         501      2646         2 read
Poll - Wait for some even in a file descriptor

Application and Their Performance Utilization

Firfox/Chrome	User Space
Snap - onenote	User Space
OBS - Software	User Space
Virtualbox	Kernel Space
Linux dd	Kernel space

Above strace summary we can get it from #strace -c -w ls(command)
Kernel Space -> Program rely on the mostly on Hardware Part 

User Space -> program rely on the Arithmetic operation(software operation) 

if the program depends more on the kernel operating like (memory ,process,device handling,File system,Network) will come under Kernel space else user space which perform most of the operation related to the unprivileged mode.

Linux is a very interesting Operating System, we can drive deeper and learn 

• every program instruction what is doing.
• why it is occur.

How can we improve the performance depends on our requirement. 

We have other performance tools also. For your information

1. Perf- Performance Analysis tool – kernel on CPU and code path  
2. Ksar - CPU/Memory Paging/utilization/Network IO  
3. Strace -> #strace -c -w ls  
4. #mpstat - Report the processor(CPU) related statistics 
5. #atop - Monitor cpu, memory, disk and network. 
6. nmon - Monitoring tool (CPU,MEMORY, Network,DISK)
7. iotop - simple top like I/O monitor for storage 
8. Galances - simple windows, we will get lot of informationCPU, load average, memory, network, disks, file systems, processeshttps://www.youtube.com/watch?v=E3Ioopzt8ko 
9. Iptraf-ng - IP LAN Monitor Interactive tool 
10. Netdata - CPU/RAM/DISK IO/Network Traffic 
11. nethogs - Monitor which process use the network bandwidth
12. iftop -i wlp18s0 (Monitor the particular interface and list of applicaiton network bandwidth)

1. (base) jaikumar@jaikumar-Inspiron-N5010:~$ mpstat -P ALL Linux 5.0.0-38-generic (jaikumar-Inspiron-N5010) 24/03/20 _x86_64_ (4 CPU) 10:10:18 AM IST  CPU %usr %nice    %sys %iowait %irq %soft  %steal %guest %gnice %idle 10:10:18 AM IST  all 7.64 0.09    2.23 11.55 0.00 0.71    0.00 0.00 0.00 77.79 10:10:18 AM IST    0 7.55 0.08   2.28 11.84 0.00   1.04 0.00 0.00 0.00   77.20 10:10:18 AM IST    1 7.72 0.06   2.22 12.04 0.00   0.67 0.00 0.00 0.00   77.29 10:10:18 AM IST    2 7.63 0.15   2.21 11.64 0.00   0.80 0.00 0.00 0.00   77.57 10:10:18 AM IST    3 7.65 0.07   2.21 10.65 0.00   0.32 0.00 0.00 0.00   79.10

conclusion: 

Just like if we want to monitor any program we can monitor using top/strace and other commands and find which metrics are utilised most. Depends on this we can come to the conclusion about the program behaviour and improve the performance. Developer write better code for development.