[Music]
welcome back everyone and to our uh
second talk of the day at the lca
colonel miniconf
uh next up we've got paul mckinney
who has been a programmer for a very
long time and has been working on
parallel hardware for approximately my
entire life
um he's a software engineer at facebook
where he
works as the maintainer of the rcu or
read copy update
implementation uh
synchronization primitive um within the
linux kernel
and today he'll be talking to us uh
about the rcu torture test suite
um paul has asked uh if uh will be he'll
be taking questions during the talk so
if you want to um get your questions in
via the questions tab in venulis
as is going along we'll um we'll do our
best to
get them to paul
during the course of the talk and
there'll be a q a session at the end as
well
please welcome paul
thank you andrew
so why would anyone want to torture rcu
i suppose is a
starting question and let's face it
synchronization primitives can be
somewhat obnoxious to use rsu is a
synchronization primitive when you're
using them sometimes it feels like
they're torturing you so why not get
back
but before we start torturing rcu i
would like to call for a round of
appreciation for these projects and the
people behind them
these guys don't always get the respect
and appreciation they deserve but uh
their efforts are really important
to making all our software work as
robustly reliably as it does so
big thanks to them okay so some of you
are probably thinking hey paul shut up
and let me start torturing
uh before we get to that point though uh
the url at the bottom is uh
an overview of a bunch of blog posts
covering this material and there'll be
urls at the bottom some of the slides
that will have more information on the
topic in that slide
for the people who want me to shut up so
they start torturing i'm sorry but i'm
not going to shut up immediately maybe
never
but if you want to start torturing this
slide is for you all right if you have a
system that can run qmu and kvm and any
modern x86 system will do just fine and
there's a bunch of other ones as well
uh you set your path up as shown there
and if you have a lot of time all you
have to do is just type kvm.sh enter
what that's going to do
is it's going to torture rsu with 19
different scenarios
how that's going to work is it's going
to build a kernel for one of the
scenarios it'll then do a 30 minute run
by creating a guest os and then running
rcu torture within that guest os
when he gets done with that it'll build
the next kernel and so on through all 19
scenarios
as a result this is going to take like
10 or maybe even 11 hours depending on
how quickly your system builds kernels
if you have a system with a lot of cpus
this is a really wasteful process
because a lot of those scenarios use
only one or two or four cpus
so if you have 64 cpus the next line is
more appropriate kvm.s h cpus 64.
and what that's going to do
is going to run the scenarios in two
batches so it's going to build a whole
pile of kernels one at a time build them
and then when it's got them all built
it's going to create a guest awaits for
each of the kernels that just built and
run them concurrently for 30 minutes
when it gets done with that it'll build
the rest of the kernels one at a time
and then it'll run all of the
kernels produced as guest os's again
then it'll summarize the results
so that allows you to get done a lot
faster that's going to be oh i don't
know
maybe one two hours depending on how
fast again your current
kernels are built on your system
but if you're running different systems
keeping track of which cpu has how many
put how much system has so many cpus for
that matter can be a little obnoxious so
you can just say kvm
kvm.sh
all cpus
and that'll just look and see how many
cpu is there and use all of them which
makes everybody happy well aside from
anybody who might be trying to use that
system for something else but yeah you
can't have everything
and you can also tell it how long to run
so dash dash duration
1d says one day 24 hours
what this will do then
is it if you have 64 cpus it'll act just
as it did before but it'll take
two days kind of a weekend run to run
through all of the scenarios if you have
128 cpus or more you can run all the
scenarios in one batch build all the
kernels run everything in parallel be
happy
in this case we're setting the duration
to 12 hours
one important safety tip
you've got your kernel source tree
you're doing this in
if it has things like tags files or c
scopes databases you'll want this trust
make thing that's going to do two things
for you well one
it's not going to do a make mr proper
make this clean at the beginning of the
bill each build
and that means that you'll be able to
reuse build products of course there's
some risk in there you may have a
problem that's caused by an earlier
build
but the nice thing about trust make is
that it's not going to destroy all of
your tags files or c scopes databases
if you just want to torture one aspect
of rcu for example i modified srcu some
last week so i'd want to torture srcu
and this is one way to do it on the next
line
given 28 cpus if i say configs and then
list all of the srcu scenarios there are
four of them there i can also put two
star in front of it to say run each of
them twice
so what's going to happen with this
thing is it's going to do four kernel
builds
and then it's gonna
run eight
guest os's
two for each kernel and do that by
default we haven't said so do it for 30
minutes
if you have a boot bug
that happens rarely
it's going to be really annoying waiting
for the kernel build each time
and so kvm again allows you to reuse the
build products from a previous run so if
you just run kvm.sh like the previous
five lines
in this output there'll be a results
directory and a big path name you take
that path name and drop it in as a
second argument to kvm.again.sh
and in this case we can set the duration
of the runs to 45 seconds 45 s 45
seconds and that allows us to do a huge
number of boots really quickly
and hopefully be able to reproduce the
bug we are chasing
if you have a large number of systems
then kvm remote is for you you can give
it a list of systems the names as you
used ssh into them
and uh you'll usually need to give it
the configs the cpu's 80 each system has
at least 80 cpus configs you want to
usually run a lot of them cf list is a
keyword that says all of the 19 normal
scenarios so we're going to run 19 times
15 scenarios plus 310 so it's going to
do a lot of torturing in a short period
of time that's something i normally do
something like that for my testing
okay so that's how we start testing
and
so one question is what success look
like and if we just run the stock
thing without specifying configurations
we get those 19 scenarios
and the important thing is that all of
these scripts all three of them will
give you an x code of zero if there are
no problems detected by the test and
this can be really useful you can take
any of those scripts and hand it to get
bisect run for example
and yes i really have done get bisect
run with kvm remote
it helps you
reproduce and test things involving uh
difficult to reduce bugs much more
readily
okay uh there are other kvm.sage
parameters i'm not going to go through
these in violent detail k config allows
you to vacation arguments you can
specify kernel boot arguments as well
you can tell it to run either ka san or
kcsan you'll get to run both at the same
time
both of these guys generate big binaries
big vm linux files so be a little
careful of your disk space
and kcsan requires recent compilers
clang 11 works for me but you'll need to
check for you
you can torture different things if you
don't want if you're not upset enough
that are seating torture locks for
example
uh and uh if you're
running a special purpose thing if you
have a lot of cpus you can do dash dash
dry runs scenario and uh
tell us scenarios and what that'll do is
it'll won't actually run the tests but
it'll tell you what the batches are
going to be in other words what
scenarios can run in what order and that
can allow you to
choose the dash configs or the cpus
argument
appropriate for whatever you're trying
to test
all right uh maybe you can't decide what
you want or torture well there's the
torture.sh script and it tortures a
little of everything i use it as kind of
an acceptance test before i do a pull
request occasionally
i also use it for dash next testing
now this does take some time like about
11 hours on a heavy duty laptop it does
not run kc sand by default if you do
that you're looking at 14. it's got a
lot of arguments and controls torturing
but we'll leave those off
due to time
you maybe you want to run a debugger
well dash gb does that for you you only
get to run one scenario at a time i'm
sorry i didn't want to mess with having
multiple parallel debugging sessions and
well if you want to do that send me a
patch okay
one
safety tip
you don't get to use the gdb brake
command
the reason is the linux kernel really
doesn't like gdb rewriting as binary
so use h-break which uses the hardware
breakpoint registers use that instead
and life will be a lot better
you also have to be fast once you hit a
breakpoint otherwise all sorts of things
the linux kernel complain about the
delay but
either be fast or just hit the
breakpoint look around
so you found a bug what do you do now
well you know if you fixed it sent me a
patch that'd be great
uh of course fixing it can be a little
bit of an issue because with rcu
heisenbugs are the common case
and you'd like to dehyze and bug them
and that can be hard but here's a few
tricks
the overall
approach is to make risky operations
happen more frequently in my testing cpu
hot plug is often
what's causing the problem or related to
the problem and that's because it
doesn't really get tested that much
so i end up doing some of the testing
early on for it and you can give a boot
parameter to tell
the scripts to do the hot plug more
quickly by default you'll get a cpu hot
plug operation every few seconds
with this boot args in red there it's
going to wait only 200 milliseconds
between
hot plug operations so you'll be getting
the hot plug operations about an order
of magnitude more quickly
and thus putting a lot more stress on
whatever might be causing the problem if
it's related to cpu hot plug
long-lived readers rcu torture does this
automatically
so you shouldn't have to worry about
this but you can always modify the
source code if you do
sometimes there are bugs that happen
when the system goes from really really
busy down to idle or back up
and there's a module parametered rcu
torture called stutter that controls how
quickly and often that happens
some other bugs happen when you have
huge numbers of callbacks and there's a
fwd underbar progress
module parameter that controls the
intensity and how how frequently those
happen
another thing that can make things
happen more reasonably more quickly is
preemption and the neat thing is is that
if you're running on a hypervisor which
these scripts are
you can preempt the guest os even if the
guest os thinks it has interrupts
disabled
okay and there's a dash dash jitter
argument to kvm.h to make that happen
by default it just makes a jitter script
for every cpu that it's using
and each of these things goes and
randomly grabs a cpu spins on it for a
while lets go and waits for a little bit
and then randomly grabs another cpu
you can control how long it sleeps and
how long it spins and you can also
control the number you can make more or
fewer of them
so these can help accelerate there are
some other things you can do as well but
that'll do for the for right now
another thing that it does we could try
torturing rcu from user space you know
we could have a user space thing a big
test script that went and hammered
various system calls and io ctls and
whatever else to try to make the kernel
do rcu work and try to force things to
happen but
there's a lot of overhead involved in
all that and so you're not going to be
putting a whole lot of stress on rcu
when you do that so we don't do that
what we do instead is these tests are
kernel modules and that allows us to
directly invoke the rc apis and thus
allows us to put a lot more stress on
our cu
alright so you know i torture rsu a lot
is it really worthwhile torturing it
more a legitimate question to ask i
suppose
and uh unfortunately the answer is yeah
there's a lot of reason to torture it
more
turns out rcu
as of 5 15
as in before
just the beginning of this week contains
a little over 18 000 lines of code and
that's just wc-l lines
there's a lot of statistics on bug
density and usually get somewhere
between one and three bugs per thousand
lines of code if you work the math
that's somewhere between 18 and 55 bugs
i'd like to think i'm better than that
but
you know
i know better i've
been around for a while and i've seen
what kind of code i generate
plus the median age of a line of code in
rc was less than four years
okay in contrast rc was introduced in
the kernel
almost 20 years ago
all right so
there's a lot of new code in there and
new code tends to be bigger than all the
code so yeah there's more bugs in here
and they need to be tortured so we can
find them
sad but true
another thing working against this is
the installed base and i'm going to
illustrate this by kind of running
quickly through my career
my very first professional
program
was written in 1975
pro bono for the national honor society
it was i kid you not a computer dating
program
they used it as a fundraiser
well
if we had a million-year bug in that
program a sequential year program
millionaire bug but whatever
that bug would happen once every million
years so yeah you know murphy is
actually kind of a nice guy sure
everything can happen will
eventually you know maybe in geologic
time
as i've moved through my career i've had
the good fortune and privilege
to have increasing numbers of people use
my software
in 2017 a gentleman from arm told me
that there were at least 20 billion
that's billion with a b
you know 10 to the ninth power 20 times
10 to the ninth power
linux instance is running on arm at that
time about four years ago
and uh that's a lot of systems if we
have a million year bug it's happening
several times per hour across the
installed base
so
you know infrequent bug
i'm sorry it's still happening
frequently
and it could get a lot worse
we got this thing called internet of
things uh there was a fact an lca about
a little while ago of internet of
internet of things lca
and we could easily end up with
trillions even if linux doesn't get that
much
of a share
of the iot devices
and my concern
is that over a period of
you know 45 years
murphy might have transitioned
from a nice guy to a homicidal maniac
and i don't know about you guys but i'd
feel really really bad if my software
happened to kill somebody so you know
this is this is even worse so what can
we do here
i mean uh
my current employer is quite generous
with systems i really thank them for
providing a good environment for me to
test and develop rcu it's wonderful
but uh they don't provide me with with
20 billion of them let alone a trillion
of them
nor nor do i ever expect them to
so what can we do about this
well one
one reason for hope is that there are
living things like ourselves
and this
evolutionary process has generated us
over a period of a fair amount of time
and
we are subject to all sorts of ills
witness the fact that i'm presenting to
you virtually rather than being in
australia with you okay i'd rather be
there but you know if if i can't this is
at least second best so yes there are
some problems with living things but
still we seem to operate fairly well
we certainly live longer than software
does generally
so maybe we can use natural selection
and of course we're going to discuss
natural selection we need a picture of
the great man there charles jarwin who
observed natural selection happening in
living things
what charles darwin didn't know
was he was watching natural selection in
software of a sort because we have this
dna
and
the dna and codes proteins and things
like that and it resembles a software of
a sort
well if we can do natural selection on
living software we should be able to do
that on non-living software as well
and uh you know that shouldn't be too
hard to do and so let's give the great
man something to look at
and here we've got a process for doing
this now some of you may object to my
bit about software being randomly
generated
and
i'm sorry but
if you want me to change my description
of software in general in the world
today
change the software okay
but that's okay it's randomly generated
we can still do validation on it run it
through tests formal verification where
that's applicable whatever else we can
do and this will act as a selection
function
so we'll get bugs out of that we'll fix
those bugs hopefully injecting fewer
bugs than we fix along the process and
if we do that we go around this cycle
around and around and eventually we have
robot software dropping out the bottom
wonderful stuff right
well almost
you see we're assuming here the natural
selection applies to software with good
reason after all living things have this
dna based software right
and there's a particularly annoying form
of software that is called
a bug
all right
so if we go through this cycle we're not
going to have robust software dropping
out the bottom unless we're luckier than
anyone deserves to be instead we're
going to have software and bugs that
have adapted to the validation
this by the way is one reason that test
suites tend to become less effective
over time because the bugs adapt to the
test suite and it doesn't find them
anymore
what this means is that we have to do
more okay so one additional thing would
be put a loop on the bottom as we get
bug reports from the field escapes
uh yeah we fixed the bug in the software
sure but let's also fix the validation
where feasible
so that will catch similar bugs in the
future
if we do this we should have some
reasonable chance of getting
better
software at the bottom on fewer adapted
bugs
still you have to be a little bit
careful
you see normally what you do is you
write
if you write tests at all hopefully you
do okay you do write tests right
when you do that you're going to
validate the use cases you're aware of i
mean you write a test case for a use
case i'm not aware of what would i write
and then what happens is we randomly
generate our software which has lots of
bugs we run the tests and fix the bugs
and if things work perfectly we get rid
of all the bugs that the test finds
and then we do it again right and then
we end up something like that
okay well you know our users aren't
inconvenienced by the bug so what's the
problem
well the problem is of course that the
world changes
and so we're likely to get some new use
cases and what we'll find if we've been
doing this
is those new use cases will have been
protected from our software by walls of
bugs
and the poor guys that are trying to
make those use cases happen might
quite understandably decide you know
this is a real pain i'm going to write
my own software thank you this isn't
working
now don't get me wrong
life is sometimes hard and sometimes
hard choices have to be made and if
you're in a situation where you can only
fix
one of two bugs
yeah you're going to fix the one that's
hurting your users right now
but
understand this is where that leads to
so
what we'd like to do then if we have the
luxury of doing that and
sometimes we do
instead of just bug reports we'd like to
add paranoia and uh make the validation
a little bit more aggressive and try to
clear the way for future
use cases
and the fuzzers actually seem to do a
reasonable job of that at least so far
so we are already doing this to some
extent and hopefully that will continue
okay
the another thing is that natural
selection is a euphemism for a rather
ugly process actually
but sticking just to software and
avoiding living things for the rest of
the slide
the key point is if your tests are not
failing they are not helping to improve
your software
that actually bears repeating if your
tests are not failing at least from time
to time they are not helping to improve
your software
they might be maybe prevent preventing
bugs from coming in but they aren't
actually improving it as it is
so
what's the price than a robust
concurrent software
as you know internal bug fixing i mean
we all know that we've done it but also
eternal validation development
as
especially in the popular things like
the linux kernel and any number of other
popular software projects
new use cases appear all the time and we
need to prepare the software for them
so we've gone through these topics
torturing rcu tracking down heisenbugs
installed base causing trouble uh
natural selection good and bad ugly and
the price of current software
uh this next slide uh is where you can
find some more information as is the
slide after that
and with that uh if you've got questions
i'm happy to answer them or at least say
something in response to them
so any questions
hearing none uh thank you all very much
for your time and attention
and uh here we are
oh we do have one okay i guess i spoke
too soon
okay should we view the rcu torture as a
pass fail only or other statistics that
can flag performance aggressions that's
actually a good question and then that
gives me excuse to revisit an early
slide
let's get back up to it
eventually here
yeah there we go
okay that slide
um there is our statistics
uh the the uh column after all the
dashes is the grace periods that were
executed for the full test
and uh
and then that there's also per seconds
as you can see the different types of
rcu have rather wildly different
uh statistics on the number of grace
bridges per second and that's that's
life i mean they're doing very different
things
um
the type of rcub tested so tasks rude is
uh one that uses uh
work cues to interrupt all the cpus kind
of the old style approach and so on
uh and so it'll it'll give some state
information pertaining to that type of
rcu uh there uh the grace period number
and the flags thing that says whether
there's a grace period being asked for
there are a few types of rcu that are
robust enough to be able to stand having
piles of callbacks jammed down their
throats
and for those and if you've selected the
test that's the the
if
the progress uh uh
kernel
parameter we talked about earlier
it'll tell you how many the maximum
queue depth of of callbacks
and so uh the tiny ones which are single
cpu keep up fairly well uh the worst
case is tree zero three which
uh it makes things hard on its
configuration many things hard on rcu
and there was a point in time when there
were in this particular run 10 million
callbacks excuse me one million
callbacks waiting to be
uh invoked
so there are some statistics let's see
i've gotten behind on questions um
and uh the statistics some of the six is
really noisy for example the nmax cbs is
quite noisy so you have to be a little
bit careful relying too much on that one
what's failure look like you know i
you're right i should have a slide on
that i don't what'll do what'll happen
is between these you'll see these lines
and then if there was a failure
after the line corresponding to the test
that failed there'd be stuff printed out
which couldn't vary really wildly
depending on exactly what failed
it'll give a summary of i got this many
warnings i got this many stalls it'll
print out the occasional thing about
what's going on there's quite a bit of
variety on the failure reports
can more testing be done with
emulated timing hit entries case and
currency bugs are lighting on relying on
large numbers of machines and uh doing
things randomly
uh there is a little bit of that in rsu
as it is now
so
there are places in the grace period
k-thread where some of the tests will
insert delays
and if i remember correct that's one of
the reasons 303 has a hard time
and
what will happen is that it'll randomly
insert delays as you ask it to in
addition uh there's random delays
inserted in the in the readers that are
being done
uh you could of course do that quite a
bit more there have been at eth zurich i
believe it was there were about 10 years
ago there was a project that would
insert those automatically based on
various things
perhaps another sanitizer sometime in
the kernel's future we'll do that
automatically throughout the kernel
how can we identify which torture is the
most relevant to an employer's use cases
okay
okay let's uh
take a look at that one one thing to do
is to look at the kernel you build
and see what rc related k config options
it uses
and so one thing you could do
there's a directory
what you can do in fact to find that
directory is just for example do a find
uh dot dash name root zero one dash
print okay and that'll that'll show you
the directory where the root zero one
file exists and that root zero one file
defines the k config options for the
root zero one scenario there also is a
root zero one dot boot file the boot
file is optional and if it exists it'll
have boot arguments that help define
that so what you could do is just make
your own scenario for your employer
just go and drop the file into that
directory
it's tools testing self-test rcu torture
configs rcu and then the names of the
files
uh and so you can put a you know
my company
uh all it has to be all caps you have
dashes but it has to be all caps this is
an old things that's why there's
lowercase d and u there
and put in the k config parameters that
match the kernel you guys use and get
exactly what you want
and then also the boot
dot boot file if you have boot arguments
that you care about and then you can run
whatever you want otherwise what you can
do is just go through the existing files
and see if there's one that's a close
match and there might well be
there is a another file in that bin
directory called
config.csv.sh and that will make a
spreadsheet of the existing ones
the ones that are run by default and you
can look at that and and get a quicker
view
how often are you finding this is
catching bugs in new patches
uh if i write a patch i probably have an
rc torture failure in my future
i sometimes find uh problems in other
parts of the colonel
uh the there was a rsu torture failure
that showed up just before thanksgiving
that and i forget where it was uh
frederick weisberg took pity on me and
chased it down for me so i could
actually could celebrate uh usa
thanksgiving holiday he's in france so
he didn't have that one
i hope to return the favor someday
we had found some things in the
scheduler uh we occasionally find things
and timers
and so uh
but mostly uh i'll write a patch i'll
run oversea torture and it'll explode
anyway i think we're at the end of the
session thank you all very much for your
time and attention and for all the good
questions and have a great rest of the
conference