Using KernelShark to analyze the real-time scheduler【转】

转自:https://lwn.net/Articles/425583/

This article brought to you by LWN subscribers

Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

 

February 2, 2011

This article was contributed by Steven Rostedt

The last LWN article on Ftrace described trace-cmd, which is a front end tool to interface with Ftrace. trace-cmd is all command line based, and works well for embedded devices or tracing on a remote system. But reading the output in a text format may be a bit overwhelming, and make it hard to see the bigger picture. To be able to understand how processes interact, a GUI can help humans see what is happening at a global scale. KernelShark has been written to fulfill this requirement. It is a GUI front end to trace-cmd.

KernelShark is distributed in the same repository that trace-cmd is:

  git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git

To build it, you just need to type make gui, as just typing make will only build trace-cmd. These two tools have been kept separate since a lot of embedded devices do not have the libraries needed to build KernelShark. A full HTML help is included in the repository and is installed with make install_doc. After installing the documentation, you can access the help directly from KernelShark from the "Help" menu.

This article is not a tutorial on using KernelShark, as everything you need to know about the tool is kept up-to-date in the KernelShark repository. Instead, this article will describe a use case that KernelShark was instrumental in helping to solve.

Analyzing the real-time scheduler

Some time ago, when the push/pull algorithm of the real-time scheduler in Linux was being developed, a decision had to be made about what to do when a high priority process wakes up on a CPU running a lower real-time priority process, where both processes have multiple CPU affinity, and both can be migrated to a CPU running a non-real-time task. One would think that the proper thing to do would be to simply wake up the high priority process on that CPU which would cause the lower priority process to be pushed off the running CPU. But a theory was that by doing so, we move a cache hot real-time process onto a cache cold CPU and possibly replace it with a cache cold process.

After some debate, the decision was made to migrate the high priority process to the CPU running the lowest priority task (or no task at all) and wake it there. Some time later, after the code was incorporated into mainline, I started to question this decision even though I was the one that fought for it. With the introduction of Ftrace, we now have a utility to truly examine the impact that decision has made.

The decision to move the higher priority task was based on an assumption that if the task was waking up, that it is more likely to be cache cold than a task that is already running. Thinking more about this case, one must think about what would cause a high priority task to wake up in the first place. If it is woken up periodically to do some work, then it can very well be the case that it will be cache cold. Any task that was scheduled in between can easily push out the cache of this high priority task. But what if the high priority task was blocked on a mutex? If the task was blocked on a mutex and another RT task was scheduled in its place then when the high priority task wakes up again, there is a good chance that the task will be cache hot.

A mutex in most real-time programs will usually be held for a short period of time. The PREEMPT_RT patch, which this code was developed from, converts spinlocks into mutexes, and those mutexes are held for very small code segments, as all spinlocks should be. Migrating a task simply because it blocked on a mutex increases the impact these locks have on the throughput. Why punish the high priority task even more because it blocked and had to wait for another task to run?

Before making any decision to change the code, I needed to have a test case that can show that the moving of a high priority task instead of preempting the lower priority task will cause the high priority task to ping pong around the CPUs when there is lock contention. A high priority task should not be punished (migrated) if it simply encounters lock contention with lower priority real-time tasks. It would also be helpful to know how changing this decision affects the total number of migrations for all the tasks under lock contention.

First try

Having a 4 processor box to play with, I started writing a test case that would possibly cause this scenario, and use Ftrace to analyze the result. The first test case to try was to create five threads (one more than CPUs) and four pthread mutex locks. Have all threads wake up from a barrier wait and then loop 50 times grabbing each lock in sequence and do a small busy loop. The name of this test is called migrate.c.

The test application uses trace_marker as explained in previous articles to write what is happening inside the application to synchronize with kernel tracing.

Running the following with trace-cmd:

   # trace-cmd record -e 'sched_wakeup*' -e sched_switch -e 'sched_migrate*' migrate# kernelshark

[KernelShark] Like trace-cmd report, KernelShark will, by default, read the file trace.dat. You can specify another file by naming it as the first argument to KernelShark. While the KernelShark display images may be difficult to read fully in the article, clicking any of them will bring up a full-resolution version.

Since all tasks have been recorded, even trace-cmd itself, we want to filter out any tasks that we do not care about. Selecting Filter->Tasks from the KernelShark menu, and then choosing only the migrate threads will remove the extraneous tasks. Note that events that involve two tasks, like sched_switch or sched_wakeup, will not be filtered out if one of the tasks should be displayed.

[KernelShark post-filtering]

In the default graph view, each on-line CPU is represented by a plot line. Each task is represented by a different color. The color is determined by running the process ID through a hash function and then parsing that number into a RGB format.

  • The purple (   ) colored bar represents thread 4, the highest priority task.
  • The orange(ish) (   ) colored bar represents thread 3.
  • The turquoise (   ) colored bar represents thread 2.
  • The brown (   ) colored bar represents thread 1.
  • The light blue (   ) colored bar represents thread 0, the lowest priority task.

The lines sticking out of the top of the bars represent events that appear in the list below the graph.

By examining the graph we can see that the test case was quite naive. The lowest priority task, thread 0, never got to run until the other four tasks were finished. This makes sense as the machine only had four CPUs and there were four higher priority tasks running. The four running tasks were running in lock step, taking the locks in sequence. From this view it looks like the tasks went out of sequence, but if we zoom in to where the migrations happened, we see something different.

[KernelShark zoom]

To zoom into the graph, press and hold the left mouse button. A line will appear, then drag the mouse to the right. As the mouse moves off the line, another line will appear that follows the mouse. When you let go of the mouse button, the view will zoom in making the locations of the two lines the width of the new window.

Repeating the above procedure, we can get down to the details of the migration of thread 3. Double clicking on the graph brings the list view to the event that was clicked on. A green line appears at the location of that was clicked.

[KernelShark event selection]

On CPU 0, thread 3 was preempted by the watchdog/0 kernel thread. Because we filtered out all threads but the migration test tasks, we see a small blank on the CPU 0 line. This would have been filled in with a colored bar representing the watchdog/0 thread if the filters were not enabled. The watchdog/0 thread runs at priority 99, which we can see from the sched_switch event as the priority of the tasks is between the two colons. The priority shown is represented by the kernel's view of priority, which is inverse to what user-space uses (user space priority 99 is kernel priority zero).

When the watchdog/0 thread preempted thread 3, the push/pull algorithm of the scheduler pushed it off to CPU 3, which had the lowest priority running task. Zooming into the other migrations that happened on the other CPUs, show that the watchdog kernel thread was responsible for them as well. If it wasn't for the watchdog kernel threads, this test would not have had any migrations.

Test two, second failure

The first test took the naive approach of just setting up four locks and having the tasks grab them in order. But this just kept the tasks in sync. The next approach is to try to mix things up a little more. The concern about the real-time scheduler is how it affects the highest priority task. The next test creates the four locks again (as there are four CPUs) and five tasks each of increasing priority. This time, only the highest priority task grabs all the locks in sequence. The other four tasks will grab a single lock. Each lock will have a single task and the highest priority task grabbing that lock. To try to force contention, pthread barriers are used. For those unfamiliar with pthread barriers, they are synchronization methods to serialize threads. A barrier is first initialized with a number and all threads that hit the barrier will block until that number of threads have hit the barrier, then all the threads are released.

This test case creates two barriers for each lock (lock_wait and lock_go) each initialized with the number 2, for the two tasks (the unique low priority task and the high priority task) that will take the lock. The low priority task will take the lock and wait on a barrier (lock_wait). The high priority task will hit that barrier before it takes the corresponding lock. Because the low priority task is already waiting on the barrier, the high priority task will trigger the barrier to release both tasks because the barrier has a task limit of two. The high priority task will most likely try to take the mutex while the low priority task aleady has it. The low priority task will release the mutex and then wait on the other barrier (lock_go) letting the high priority task take the mutex.

Running this test under trace-cmd yields the following from KernelShark after filtering out all but the migrate test tasks.

As the colors of the tasks are determined by their process IDs, this run has the following: [KernelShark second try]

  • The initial green (   ) bar is the main thread that is setting up all the locks and barriers.
  • The light purple (   ) bar is the lowest priority thread 0.
  • The red (   ) bar is the next-higher priority thread 1.
  • The yellow(ish) (   ) bar is the next-higher priority thread 2.
  • The blue (   ) bar is the next-higher priority thread 3.
  • The light turquoise (   ) bar is the highest priority thread 4.

Looking at the graph it seems that the highest priority thread stayed on the same CPU, and was not affected by the contention. Considering that the scheduler is set to migrate a waking real-time task if it is woken on a CPU that is running [KernelShark zoom view] another real-time task, regardless of the priorities, one would think the high priority task would have migrated a bit more. Zooming in on the graph brings to light a bit more details to what is occurring.

What we can see from the graph, and from the list, is that the high priority thread did have contention on the lock. But because all threads are waiting for the high priority process to come around to its lock, the other threads are sleeping when the high priority process wakes up. The high priority process is only contending with a single thread at a time. Threads 0 and 2 share CPU 2 without issue, while threads 1 and 3 each still have a CPU for themselves.

The test to force migration

The second test was on the right track. It was able to produce a contention but failed to have the CPUs busy enough to cause the highest priority task to wake up on a CPU running another real-time task. What is needed is to have more tasks. The final test adds twice as many running threads as there are CPUs.

This test goes back to all tasks grabbing all locks in sequence. To prevent the synchronization that has happened before, each thread will hold a lock a different amount of time. The higher the priority of a thread, the shorter time it will hold the lock. Not only that, but the threads will now sleep after they release a lock. The higher the priority of a task, the longer it will sleep:

   lock_held  = 1 ms * ((nr_threads - thread_id) + 1)sleep_time = 1 ms * thread_id

The lowest priority thread will never sleep and it will hold the lock for the longest time. To make things even more interesting, the mutexes have been given the PTHREAD_PRIO_INHERIT attribute. When a higher priority thread blocks on a mutex held by a lower priority thread, the lower priority thread will inherit the priority of the thread it blocks.

The test records the number of times each task voluntarily schedules, the number of times it is preempted, the number of times it migrates, and the number of times it successfully acquired all locks. When the test finishes, it gives an output of these for each thread. The higher the task number the higher the priority of the thread it represents.

Taskvolnonvolmigratediterations
04330071571108
162113341247108
27777691072108
377517701108
478350699108
57882610109
680189680109
78130693110
Total540152687273868

[KernelShark success] Running this test under trace-cmd and viewing it with KernelShark yields a graph with lots of pretty colors, which means we likely succeeded in our goal. To prove that the highest priority thread did indeed migrate, we can plot the thread itself.

Using the "Plots" menu and choosing "Tasks" brings up the same type of dialog as the task filter that was described earlier. I selected the highest priority thread (migrate-2158), and zoomed in to get a better view. The colors on a task plot are determined by the CPU number it was running on. When a task migrates, the colors of the plot changes.

[KernelShark task plot]

This test now demonstrates how a high priority task can migrate substantially when other RT tasks are running on the system. Changes to the real-time scheduler can now be tested. The commit changes the decision on which thread migrates when an real-time task wakes up on a CPU running another real-time task. The original way was to always move the task that is waking up if there is a CPU available that is running a task that is lower in priority than both tasks. Instead, the commit changes this to just wake up the real-time task on its CPU if it is higher priority than the real-time task that is currently running.

The migrate test now shows:

Taskvolnonvolmigratediterations
0:5229232268108
1:56915291457109
2:80119612194109
3:8087891274109
4:81061155109
5:8131057109
6:8273581110
7:82404110
total:550473087490873

The total number of migrations has stayed around the same (several runs will yield a fluctuation of a few hundred), but the number of migrations for the highest priority task has dropped substantially, as it will not migrate simply because it woke up on a CPU running another real-time task. Note, the reason that the highest priority task migrated at all was because it woke up on a CPU that was running the task that owned the mutex that it was blocked on. As these are priority inheritance mutexes, the owner would have the same priority as the highest priority process that it is blocking. The wake up will not preempt a real-time task of equal priority. Perhaps that can be the next change to the real-time scheduler: have the wake up be aware of priority mutexes.

[KernelShark after change]

The highest priority thread (migrate-21412) was woken on CPU 3, which was running thread 1 (migrate-21406) which is the task that thread 7 originally blocked on. CPU 2 happened to be running thread 0 (migrate-21405) which was the lowest priority thread running at the time. Note that the empty green box that is at the start of the task plot represents the time between when a task was woken and the time it actually was scheduled in.

Using KernelShark allowed me to analyze each of my tests to see if they were doing what I expected them to do. The final test was able to force a common scenario where a high priority process is woken on a CPU running another real-time task, and cause the decision to be made, whether to migrate the waking task or not. This test allowed me to see how the changes to that decision affected the results.

This article demonstrates a simple use case for KernelShark, but there are a lot more features that aren't explained here. To find out more, download KernelShark and try it out. It is still in beta and is constantly being worked on. Soon there will be plugins that will allow it to read other file formats and even change the way it displays the graph. All the code is available and under the GPL, so you can add your own features as well (hint hint).


(Log in to post comments)

 

Using KernelShark to analyze the real-time scheduler

Posted Feb 3, 2011 2:32 UTC (Thu) by mhelsley (guest, #11324) [Link]

Interesting. Looking at the second table of 8 tasks it's clear there are 4 CPUs. The 4 highest priority threads have much fewer than 1000 migrations -- 150 or less -- while the remaining threads have 1000 or more. The number of non-voluntary pre-emptions is similarly distributed (presumably because those are involved in the push/pull migrations).

 

This looks excellent

Posted Feb 3, 2011 15:03 UTC (Thu) by robert_s (subscriber, #42402) [Link]

I've been looking for something like kernelshark ever since I started doing realtime programming and I'm so glad to know it exists.

 

This looks excellent

Posted Feb 10, 2011 21:56 UTC (Thu) by oak (guest, #2786) [Link]

Did you have some problem with LTT:
http://en.wikipedia.org/wiki/LTTng
?

(It's UI has some usability issues, but otherwise LTT has been used for this kind of stuff on Linux for the past decade.)

 

 

RT bandwidth causing gaps in the test

Posted Feb 4, 2011 3:36 UTC (Fri) by nevets (subscriber, #11875) [Link]

An interesting note:

Those gaps you see in the last test run are caused by the RT bandwidth kicking in. If I perform the following:

  # echo -1 > /proc/sys/kernel/sched_rt_runtime_us

Those gaps disappear.

 

Using KernelShark to analyze the real-time scheduler

Posted Feb 5, 2011 11:58 UTC (Sat) by tardyp (subscriber, #58715) [Link]

Looks great!
I wish we could more easily put our efforts in common and put the best of pytimechart (great graphical display), and kernelshark (great textual display and filtering capabilities) into the same tool.

I guess the choice of toolkit and programming language does not help.

Pierre

 

Using KernelShark to analyze the real-time scheduler

Posted Feb 6, 2011 20:55 UTC (Sun) by nevets (subscriber, #11875) [Link]

I thought gtk and python work well together. There's even python plugins for kernelshark. The filter capability is separated out and would be easy to make into a library that python could read. As the parse-events library is already separate, and I'm working on getting perf to use it too.
I'm all for reuse of code, and would love to help in any effort to share capabilities between the tools.

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.mzph.cn/news/255209.shtml

如若内容造成侵权/违法违规/事实不符,请联系多彩编程网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

3、时间和随机数

一、时间 1.1 使用Calendar/[ˈkləndɚ]/类获取时间 1.1.1 常用方法 (1)public static Calendar getInstance(): 使用默认时区和语言环境获取一个基于当前时间的Calendar对象。 (2)public int get(int field) 返回给定日历字段表示的日历部分的数字…

哥尼斯堡的“七桥问题” (欧拉回路,并查集)

哥尼斯堡的“七桥问题” (25分) 哥尼斯堡是位于普累格河上的一座城市,它包含两个岛屿及连接它们的七座桥,如下图所示。 可否走过这样的七座桥,而且每桥只走过一次?瑞士数学家欧拉(Leonhard Euler,1707—1783)最终解决…

无人驾驶汽车之争本田为何未战先败

摘要 : 本田汽车的研发部门对于汽车虽然理解深刻,但从整体而言,本田的造车理念还停留在上个时代,在未来的无人驾驶竞争中,本田已经有未战先啊败的苗头。 百度百家The BIG Talk硅谷站连续5小时的高密度头脑风暴,果然让人…

理解git结构与简单操作(四)合并分支的方法与策略

接上节,此时的dev分支与master分支的进度就不一样了,所以需要将dev分支与master分支同步。这里需要的就是合并分支的操作,大家应该都知道用git merge或者git rebase。 git merge merge,即「合并」。 fast-forward 当出现我们上面图…

摄像机标定

利用摄像机所拍摄到的图像来还原空间中的物体。在这里,不妨假设摄像机所拍摄到的图像与三维空间中的物体之间存在以下一种简单的线性关系:[像]M[物],这里,矩阵M可以看成是摄像机成像的几何模型。 M中的参数就是摄像机参数。通常,这…

Linux下Tomcat重新启动

在Linux系统下,重启Tomcat使用命令操作的! 首先,进入Tomcat下的bin目录 cd /usr/local/tomcat/bin 使用Tomcat关闭命令 ./shutdown.sh 查看Tomcat是否以关闭 ps -ef|grep java 如果显示以下相似信息,说明Tomcat还没有关闭 root …

大数据和人工智能的关系是什么?

何为大数据?何为人工智能? 大数据,百度百科上是这么定义的,指无法在一定时间范围内用常规软件工具进行捕捉、管理和处理的数据集合,是需要新处理模式才能具有更强的决策力、洞察发现力和流程优化能力的海量、高增长率…

张正友标定法 【计算机视觉学习笔记--双目视觉几何框架系列】

三、致敬“张正友标定” 此处“张正友标定”又称“张氏标定”,是指张正友教授于1998年提出的单平面棋盘格的摄像机标定方法。张氏标定法已经作为工具箱或封装好的函数被广泛应用。张氏标定的原文为“A Flexible New Technique forCamera Calibration”。此文中所提到…

软工网络15个人阅读作业2——提问题

提出问题 快速通读教材《构建之法》,并参照提问模板,提出5个问题。 问题一: p83有一段话: 两人在一起合作,自然会出现不同意见,每个人都有自己的想法,在两个人平等合作的情况下,不存…

绝对路径VS相对路径

绝对路径:不必赘述,就是从盘符开始写直到找到你所需要的文件为止,把所有的目录写完整即可。但是在做网站的时候绝对不推荐用绝对路径,因为不可能服务器中的路径和在做设计时候所用的电脑的路径一致,也不可能说在服务器…

四、极大似然参数估计

四、极大似然参数估计 此篇博文,玉米将和大家分享一下“张氏标定”除几何推导外的另外一大精髓:参数估计。 张教授在大作“A Flexible New Technique for Camera Calibration”中的原话如下:” The above solution is obtained through minim…

xtrabackup安装使用

2019独角兽企业重金招聘Python工程师标准>>> 【MySQL】xtrabackup安装使用 拾荒者charley 关注 2018.03.19 09:19 字数 186 阅读 17评论 0喜欢 0 前言:说到MySQL备份,主要采用的方法无非就是mysqldump/mysqldumper以及我们今天要说的热备利器…

2017/3/10 morning

转载于:https://www.cnblogs.com/bgd140206325/p/6529497.html

六、张正友标定法小结

六、张正友标定法小结 这一博文,是玉米后补上的。因为觉得前面用了三篇博文来描述张氏标定法,略显散乱。在这里总结一下,使条理清晰一点。另外关于张氏标定所得参数也还有两点需要澄清。下面这个总结,其实也是在“A Flexible New …

SqlServer图形数据库初体验

SQL Server2017新增了一个新功能叫做图形数据库。图形指的拓扑图形,是一些Node表和Edge表的合集,Node对应关系数据库中的实体,比如一个人、一个岗位等,Edge表指示Node之前的关系,比如张三在经理岗位。图形表比较适合用…

Oracle CASE WHEN 用法介绍

1. CASE WHEN 表达式有两种形式 --简单Case函数 CASE sex WHEN 1 THEN 男 WHEN 2 THEN 女 ELSE 其他 END --Case搜索函数 CASEWHEN sex 1 THEN 男 WHEN sex 2 THEN 女 ELSE 其他 END 2. CASE WHEN 在语句中不同位置的用法 2.1 SELECT CASE WHEN 用法 SELECT grad…

五、畸变矫正—让世界不在扭曲

五、畸变矫正—让世界不在扭曲 这篇博文所要讲述的内容,是标定的主要用途之一:矫正摄像机的畸变。对于图像畸变矫正的方法,张正友教授也在其大作“A Flexible New Technique forCamera Calibration”中给出。 玉米在这里先为大家介绍一下&…

《MySQL必知必会》[01] 基本查询

《MySQL必知必会》(点击查看详情)1、写在前面的话这本书是一本MySQL的经典入门书籍,小小的一本,也受到众多网友推荐。之前自己学习的时候是啃的清华大学出版社的计算机系列教材《数据库系统概论》,基础也算是半罐水&am…

(七)立体标定与立体校正 【计算机视觉学习笔记--双目视觉几何框架系列】

七、立体标定与立体校正 这篇博文中,让玉米和大家一起了解一下,张氏标定是怎样过渡到立体标定的?在这里主要以双目立体视觉进行分析。对于双目立体视觉,我们有两个摄像头。它们就像人的一双眼睛一样,从不同的方向看世界…

让 jQuery UI draggable 适配移动端

背景: 在移动端,本人要实现对某个元素的拖动,想到使用 jQuery UI 的 draggable 功能。但是发现此插件的拖动只支持PC端,不支持移动端。 原因: 原始的 jQuery UI 里,都是mousedown、mousemove、mouseup来描述…