First things first, lets define what a memory model is. The memory consistency model defines what values a read operation can return. The simplest memory model is sequential consistency, in which the execution behaves as if there were a single global interleaving of memory operations and the operations of a given thread appear in the same order as they appear in the program. It is the most natural model for normal humans to think about because the execution behaves as a multitasking uniprocessor. For example, consider the following example:

The question is, what value the read from data in P2 can return? The most obvious answer here is 42. Now what would happen if P2 observed the writes to data and flag in the opposite order? P2 could actually read data as “0″, which is surprising and not allowed by the sequential consistency memory model.

The main problem with sequential consistency is that systems like to reorder memory operations to hide long latency operations and consequently improve performance. For example, when a cache miss is being serviced, the processor may execute another memory access that comes after that in program order, which may hit in the cache and therefore complete earlier than the missing access. However, processors are not the only source of memory operation reordering. Many compiler optimizations effectively reorder code, e.g., loop-invariant code motion, common sub-expression elimination, etc. Furthermore, the memory models of languages and the hardware they run on need not be the same. The compiler and synchronization libraries need to insert fences in the code to map the language memory model to the hardware model. For example, Java and C++0x (the upcoming C++ standard) support memory models that guarantee sequential consistency for programs free of data races.

Due the difficulty of improving performance under sequential consistency, a variety of “relaxed” memory models were conceived. For example, in the Weak Ordering memory model, there is no guarantee that a processor will observe another processor’s memory operations in program order. This is where a “memory fence” (a.k.a. “memory barrier”) comes into play. When a fence instruction is executed, it guarantees that all memory operations prior to it in program order are completed (and visible to other processors) before any operation after the fence in program order is allowed to proceed.  You would be bored and stop reading if I described the multitude of consistency models in this post. However, I do encourage you to read more about memory models in this very nice tutorial by Sarita Adve and Kourosh Gharachorloo.  Also, Paul McKenney’s paper has a nice table summarizing the ordering relaxations in modern microprocessors.

Now let’s talk about some of highlights of the x86 memory model. A big disclaimer first. This can change and probably does change between models, so it is always a good idea to check the manuals before endeavoring in sensitive code (8-8 Vol. 3 in this manual for Intel and Section 7.2, page 164 in this manual for AMD).

In a nutshell, recent implementations of the X86 ISA (P6 and on), implement, roughly, what is normally termed processor consistency. Its key ordering properties are:

1. reads are not reordered with respect to reads;
2. writes are not reordered with respect to reads that come earlier in the program order;
3. writes are not reordered with respect to most writes (e.g., it excludes multiple writes implicitly caused by string operations);
4. reads may be reordered with respect to writes that come earlier in program order as long as those writes are to a different memory location;
5. reads are not reordered with respect to I/O instructions, locked instructions and other serializing instructions.

There are no guarantees whatsoever of ordering between writes of different processors, the outcome of concurrent writes to the same memory location is non-deterministic. Increment instructions have no atomicity guarantees, moreover, even some write operations that update multiple bytes are not guaranteed to be atomic. For example, if a write operation to multiple bytes happen to cross a cache line boundary, the operation is not guaranteed to be atomic.

Here is an example of how the x86 memory model can get you in trouble:

An execution whose final state is t1 == 0 and t2 == 0 is allowed. Such an outcome is unintuitive (therefore non-sequentially consistent) because there is no serialized execution that leads to this state. In any serialized execution, there will be an assignment in one processor (A = 1 or B = 1) prior to a read  (t1 = B or t2 = A) in the other processor. Another way to look at the problem is to build a happens-before graph of the execution. In this representation, a node is an executed instruction and a directed edge exists from instruction P to instruction Q if Q has observed the effects of P, and P has not observed the effects of Q. Here is the happens before graph for the example above when the outcome is t1 == 0 and t2 == 0:

Edge (1) in the graph exists because the read t1 = B in P1 did not observe the write B = 1 in P2. Same applies to edge (2). Edges (3) and (4) are there because of program order. Since there is a cycle in the happens-before graph, there is no serialized order that would satisfy the happens-before relationship, therefore, the execution is non-sequentially consistent. This happened in this example because  the read operation t1 = B in P1 can proceed before the write operation in A = 1 is completed and visible to P2.

Here is another example of how the x86 memory model leads to surprising results:

The snippet of execution above might lead to a state where t2 == t4 == 0. Lets look at this from the perspective of P1. This can happen because the processor can forward the value from the pending write to A (A = 1) to the read of A (t1 = A), which can complete and then allow the read of B to proceed (t2 = B). Note that this does not characterize a reordering of t2 = B and t1 = A! Intuitive, huh?

One final example for you to noodle about. Consider a boiled-down version of Dekkers’ mutual exclusion

algorithm:

The gist of the algorithm is to use two flag variables, one for each processor, flag1 and flag2. P1 sets flag1 when it is attempting to enter the critical section, it then checks if flag2 is set; if it is not set, it means P2 has not attempted to enter the critical section, so P1 can safely enter it. Because the x86 memory model allows reordering loads with respect to earlier stores, the read of flag2 can proceed before setting flag1 is completed, which can lead to both processors entering critical sections, since P2 might have just set flag2!

That is it! I hope this helped you get a better grasp of what a memory consistency model is and understand a few of the key aspects of the x86 model. And, if you come across something that looks like a memory consistency bug, try building that happens-before graph to find cycles and remember to look at the manual . Have fun!

The state of the art with regard to debugging concurrency errors has improved somewhat, but for multi-threaded programs it still typically involves visiting a pile of threads to see which may be involved with the problem.  It’s a breadcrumb party… great for bonding with your fellow developers and producing war stories.  Less great for shipping your code.  Various things make the bug harder to understand in this scenario.

What I’ll call “overshoot” is primary among these.  Overshoot is a phenomenon where one thread or task in your program, corresponding to a single CPU at the point of failure, runs for a long time after the error has been detected.  The detecting thread traps into the operating system, which sends out messages to other CPUs saying “stop any thread in the following process immediately”.  A nice clean operating system might be able to deliver a stop within a few tens of microseconds, increasing as the number of processing cores in your computer increases.  Sadly, that few microseconds corresponds to a few tens of thousands of machine instructions.  That’s a lot of overshoot.  To understand how that kills your ability to find bugs, let’s look at a few examples.  We’ll start with an example where a normal debugger doesn’t do a horrible job.  It’s the example from last time, and you may remember that it corresponds to code that looks like this:

    // Thread 1 (corrupter)
    float myradius = 1;
    lock();
    circle->radius = myradius;
    unlock();
    lock();
    circle->area = PI * myradius * myradius;
    unlock();

    // Thread 2 (verifier)
    lock();
    assert(circle->radius * circle->radius * PI == circle->area);
    unlock();

An atomicity violation as seen by the machine.

Because we have now added the assert, the verifier thread now will never unlock, and consequently the corrupter will never perform the update of the area.  From the perspective of a developer, your program crashes, one thread is complaining about an inconsistent structure, and one or more threads are trying to acquire the lock on the structure.  You may notice or it may be obvious that one of these threads has just updated the structure in a non-atomic way.  Because the proximity is probably pretty good, this might not be a hard-to-understand bug.

Things get more interesting, and unpleasant, when we consider a low-level data race.  We take a similar example, but rather than showing an atomicity violation, we make it into a data race, by eliminating the locking in the corrupter altogether.

    // Thread 1 (corrupter)
    float myradius = 1;
    circle->radius = myradius;
    circle->area = PI * myradius * myradius;

Instead of an atomicity violation, a low-level data race. The corrupter thread never takes a lock at all.

What is the developer’s experience in this case?  Well, after the assert is detected, an operating system call is made, which stops all threads currently running inside the program in question.  This is if you’re lucky.  On more primitive OSes you might have to wait until your timer tick is up.  Regardless, it’s pretty plausible that the corrupter thread in this case has done two horrible things:

• It has advanced thousands of instructions so that it’s nowhere near the problem location.  You probably won’t even know what the culprit thread is.
• It has repaired the state of the circle structure so that the invariant has been repaired.  This is that second STORE in the diagram above.  That’s the corrupter thread cleaning up its mess before going to hide in some other far flung piece of code.

In this case, the developer has one useful piece of information.  That the invariant has failed once.  Since we started Petra, we’ve talked to many people who have seen invariants fail and never figured out why.

One more example before we discuss solutions to this mess.  A wakeup-race is what we call an ordering violation wherein one thread is the cause of the start of activity on another thread. Learning from mistakes: a comprehensive study on real world concurrency bug characteristics describes a common bug pattern found a few times in firefox, which involves the creation of a new thread, and the assignment of its ID to a global.

    // Thread 1
    g_newThread = CreateThread(myfunc);

    // Thread 2
    myfunc() {
        assert(g_newThread != INVALID_THREAD);
    }

Sometimes we get away with it, as below.  In the case of Firefox, this was presumably a very rare bug.

This latent ordering violation isn't revealed in this example, as the creator thread runs first.

But, sometimes, we don’t get away with it.  As you would expect, Jinx is good at making this sort of bug happen.

The unlucky case, where the awoken thread runs first, observing inconsistent state.

The unlucky case is, again, very bad for the programmer.  After the bug is detected, there is a long window until the thread that stored the thread id late gets stopped.  During that time it can return to its event loop, enter some other unrelated code, and, in the case shown, set g_newThread to be valid, thus obfuscating the state of the program at fault time.  The basic problem here is that in overshoot siutations

1. The threads keep running.
2. They overwrite bits of global memory state as they go, hiding the bug or further obfuscating it.

So, for the three major classes of bugs, the experience of debugging a crash varies from bad to horrible.  How can we make this better?

The obvious answer is that we have to stop threads earlier.  One thing we could do is just sequentialize the execution, and stop all threads as soon as one of them detects a problem.  This would be an improvement, but it’s not that great.  There are a couple of reasons for this.

1. The gap between the load of the wrong data value and the detection of the bug may be long.  Say, for example, instead of computing πr² we were computing sin(acos(log²(n)))… you’re thousands of cycles in on all threads before you know there’s a problem.
2. The gap between the application detecting an error and Jinx picking it up may be large, also.  For example, the application may call into the OS as a part of its abort sequence.

Fortunately, there are better ways to solve this problem.  Jinx takes the (patent-pending) approach of advancing each of the threads involved in the crash the minimum computational distance forward before stopping them.  That is, we only want each thread to run far enough forward to allow the crash to happen.  One approximation of this point is the last communication point between threads.  What does it mean to compute the last communication point?  For two threads, the simple definition is that it’s the last time before the crash that a non-crashing thread wrote something that was read by the crashing thread before the crash.  An example is probably easiest.

There is no reason to allow other threads in the process to advance beyond the last time they communicated with the crashing thread.

First, we take the example of the low-level data race.  The last thing that was written by the corrupter thread that was still read by the crashing thread before the crash is radius, in the course of the “STORE radius” instruction.  Simply by terminating execution of the corrupter thread at its last communication point with the verifier thread (immediately after this instruction), we provide the following user experience:

1. The verifier thread is stopped on the assert line, as before.
2. The corrupter thread is stopped on the next instruction after writing circle->radius = 1;

That’s a pretty cool result.  You’re stopped at the debugger, and all of the participating threads are in a meaningful location.  What about the other really thorny case, the ordering violation?

Stopping on last communication, applied to an ordering violation.

Here, the last communication is virtual:  it’s where the one thread creates the other, which is a virtual communication.  In Jinx’s case this is trapped at the IPI (interprocessor interrupt) sent from one CPU to another.  Since no memory communications occur after that, it’s not necessary to allow the thread to run past that point.  Again, the user experience is pretty good:  one thread is stopped creating another, and that thread is referencing the illegal global.  The value of g_newThread is still INVALID_THREAD at the point of the crash.

You’ll note that it’s only possible to do any of this with Jinx’s simulation approach.  Only by running executions multiple times can you perform the kind of analysis necessary to do this sort of thing, and you can only do that with Jinx.  You might imagine we’re pretty excited about SmartStop.  It will be present in our beta in late January or early February, and we’ll release a video of it in action as a part of the beta launch.  We hope you’ll sign up for the beta and let us know what you think of this feature and the rest of Jinx!

First off, how does Jinx find crashing bugs?  It runs simulations of system state in parallel, and chooses interesting ones to “retire.”  The following diagram illustrates this process.

At time A, Jinx takes a checkpoint, and conducts simulations until finding a bug in a simulation ending at point B. It chooses this simulation to retire into reality. This process is completed at C, and normal execution resumes.

After taking a checkpoint at A, Jinx starts by conducting a first exploratory simulation, and based on the result of this preliminary simulation conducts zero or more follow-on simulations, scoring these scenarios as they complete.  Our life is straightforward in the case where we find a crash, as in B.  We assign a high score to the crashing simulation, and later on we choose this simulation to retire (our jargon for “commit” or “turn into reality”).  Retirement completes at C.  Typically we only retire bugs that don’t happen in the kernel or device drivers…  most developers don’t want their boxes bluescreened!

So you run your program once, and Jinx runs parts of it tens or hundreds of times behind the scenes, choosing for retirement the simulations that are most likely to result in bugs.  The performance impact is mitigated by running the simulations in parallel across all the tens (soon hundreds) of mostly idle cores in that teraflop multi-core machine you just bought.

We have a pretty elaborate scoring mechanism to encourage the retirement of bugs (when we retire a bug, we make it really happen, which is what we want).  At the machine layer, INT3, #DB, #BP, #PF, and explicit hypercalls like jinx_assert() can all suggest to Jinx that a simulation is a good thing to replay.  We don’t get this data for correctness bugs, where the bug isn’t caught during execution, but pollutes the output with incorrect data.  So, which simulation to make into reality is not so obvious.  However, Jinx is great at making correctness bugs happen more often.  Why?

The first thing to understand is the scales of time that we’re talking about. If we were to divide time into seconds, then in some one-second intervals both sides of a bug would occur…  for concurrency errors, you have to have two parties involved, by definition.  So, for example, not only does someone update a structure non-atomically, but someone else chances to read it within the same second.  For structures updated seldom, only in a fraction of the one-second intervals will the potential bug occur.  For the sake of example, let’s say a bug’s two parties occur in the same second, in one of every ten seconds.

During each one second interval, either one or both parties to a bug may run. The two parties occur only in second 6 in this example.

Now, within the second’s duration, let’s say we’ve committed the most original of atomicity violation sins:  the non-atomic update of radius and area for the canonical circle.

    float myradius = 1;
    lock();
    circle->radius = myradius;
    unlock();
    lock();
    circle->area = PI * myradius * myradius;
    unlock();

In diagram form, with another thread potentially observing the state of the circle.

The corrupter thread introduces an atomicity violation by updating radius and area in separate transactions. The observer thread is likely to observe the inconsistency if it attempts to take the lock anywhere during the race window.

In order for the second thread to observe the problem, it has to begin its attempt to lock the structure during time period A or B.  As the critical section A becomes smaller, the likelihood of the second thread attempting to lock during that period approaches zero.  The more optimized the critical section in this case, the more rare the bug becomes.  There’s a lower bound on how small the critical section can become (all instructions take time, and a locked instruction takes quite a bit of time).  Let’s say that the critical section takes 1000 cycles, which is pretty typical for a lock and unlock, and you’re on a 1Ghz CPU.  If you know that cause and detection occur in the space of one second, you have an approximately one in one-million chance of observing the bug.

The time between the two parties in a bug may be glacial in scope relative to the size of the race window, which is vanishingly small in this one-second example.

You should expect that a one in one-million bug that could occur once every ten seconds actually does occur at a rate of about one every ten million seconds, or 2700 machine hours.

So, how does Jinx make this better? Remember when I talked about the simulation approach?  Well, after every simulation is done, Jinx creates a graph that looks like this:

When Jinx considers how to reorder events, it only considers reordering A with respect to 1, 2, 3, or 4. Reordering with respect to 2 or 3 force the bug to occur.

Or, in a representation which is closer to the way that Jinx sees it:

Jinx ignores all noncommunicating instructions. What is left is the set of accesses that could have meaningful reorderings. For example, access A has meaningful reorderings with respect to 1, 2, 3, and 4.

Jinx uses this representation to decide what to change in subsequent simulations.  Jinx wants to reorder memory accesses with respect for each other.  For example, if Jinx decides to reorder memory access A, the only interesting points to reorder with respect to are 1, 2, 3, and 4 If we rearrange with respect to 1, then the observer runs before the corrupter, so no bug.  If we rearrange with respect to 4, we get a little lock contention but no difference in behavior.  But, if we rearrange with respect to 2 or 3, then we force the rare circumstance to occur.  Jinx has turned a one-in-a-million event into a two-in-four event! There are so few cases, we can enumerate all of them:

Of four interleavings suggested by Jinx, two reveal bugs, as they result in accesses to radius and area between the updates to radius and area.

This is Jinx’s notion of communication logical time.  Instead of rearranging events in wallclock time, we rearrange events in logical time.  This cuts out the huge amount of temporal noise involved in hard-to-reproduce bugs.  In this example, this transition to logical time changes a 2700 machine hour repro to a 20 second repro.  Give Jinx a factor of 100 slowdown during simulation, and it’s still reproducing the bug 10,000x faster than reality would.  Debugging a corruption bug that occurs in an hour on one machine is a completely different story than debugging one that takes 2700 machine hours.

So by transforming execution into logical communication time, Jinx forces even non-crashing correctness bugs to occur more frequently than they would in the wild, allowing you to find your non-deterministic bugs in development rather than test, and in test rather than deployment.

Next time I’ll talk about how Jinx makes it easy to understand a bug when it has occurred.

Corruption is worse than crashing

If a program stops and complains, that’s one thing: an automated system or a human can just start it again.  There might be real-time or reliability goals for the program that aren’t being met, but at least you know something went wrong.  If it keeps on going, but corrupts the output in some way, there’s a much bigger problem.  Most software is based on an implicit assumption of the fail-stop model.  This just means that errors in the code stop the program’s forward progress before erroneous data can become nonvolatile: written to disk or communicated to another process.  You can see examples of applications that make this assumption all the time, when they trashed their registry entries or made their tables inconsistent, and consequently can’t run any more.  At least one study (http://www.computer.org/portal/web/csdl/doi/10.1109/TDSC.2007.70208) finds that 7% of bugs aren’t fail-stop bugs… that is they produce bad outputs rather than crashing or stopping.

Correctness bugs are hard to fix, hard to detect, and costly

In my acquaintance’s domain, correctness bugs result in subtly wrong graphics.  There, the fault tends to get detected, but the error-resolution process is lengthy, as much more of the codebase is suspect.  These bugs are detected later in the development process, because earlier  stages lack sufficient verification to detect corruption, but crashes are always obvious.  In other domains, correctness bugs could mean incorrect chances of success when drilling a test well, an incorrect diagnosis, a wrongly sequenced chromosome, a corrupted database, poor search results, or an ill-advised financial trade.  In some of the worst cases, humans or machines take the data out of computation and start committing real-world resources to the results of the computation.  Things get really expensive when errors in computation get into nonvolatile storage inside people’s brains.

Some of these bugs are eventually detected, as when you compute an incorrect launch window for a rocket.  Others, like choosing not to drill a test well in a fruitful spot, will likely never be detected.  The hidden and obvious economic drags of correctness bugs include unhappy customers, missed opportunities, and wasted efforts.

Correctness bugs are more horrible as your software gets more non-deterministic

These bugs get more insidious when you make the leap to shared-memory parallel programming.  Whereas in the single-threaded, synchronous world, a given input to a program always yielded the same output, you’re now faced with programs that may give a correct output only a certain percentage of the time.  You expect 100%, but it’s not generally possible to prove that a shared-memory parallel program produces deterministic results.  This causes trouble for crashing bugs because it makes repros rarer, and allows bugs to creep further through the development process before detection.  It also renders debuggers less useful…  You can’t just step through until you hit the bug, because sometimes the bug happens, and sometimes it doesn’t, and often the debugger stops it from happening.

The problem is much worse in the case of correctness bugs: both correctness and non-deterministic bugs tend to get detected late in development, and combined, they appear very late.  Users of your code can act upon the results because they’re not crashing bugs, and the bug can slip through your test because it’s non-deterministic, even if you have a test case that could have caught the bug!  Input coverage isn’t enough any more.  How do we deal with this problem?

Dealing with the Problem

The forces of non-determinism and corruption!

There are three big parts to the answer to this problem.  They don’t make the problem go away.  They just stop your life from becoming a hell of customer escalations and expensive round-trips to QA.

Choose a simple parallel programming model

First, the way you use multiple cores will change the amount of non-determinism in your software.  All the parts of your program that can

avoid shared-mutable memory should probably avoid it.  For most applications, you’ll never eliminate non-determinism altogether.  After all, what is input if not an access of shared-mutable state?  But, you can try to guarantee that like inputs yield like outputs by being careful about the sorts of shared-memory parallel programming you do.

Turn your correctness bugs into crashing bugs

Next, increase your likelihood of bugs being crashing bugs, rather than correctness bugs.  The trusty assert() is your friend here.  Ship your software with your asserts enabled, and don’t tolerate failing asserts.  They tell you where the bug was, and what it was.  When you get that bug report from the field, you know exactly what you need to fix.  Programs that need to continue to function in the face of failure can use software fault-tolerance approaches like running every computation twice and comparing results, with an optional tie-breaker.  Some problems are intrinsically easy to verify, like factoring an integer.  Most are not.  During test you can use late-stage output verification to achieve fail-stop.  Make sure everything does this.  By doing this, you’ll change more bugs into being crashing bugs, rather than correctness bugs.

Amplify your testing

Lastly, amplify your non-deterministic bug-find rate.  If you expect to deploy software that will run at a rate of tens of thousands of machine-hours per hour, that is, you have tens of thousands of users, or a few users using thousands of computers, you won’t match their ability to find non-deterministic bugs in software without amplifying your bug-find rate. This is a potential success disaster:  shipping lots of software will make you realize how buggy it is.

Our goal with Jinx is to let the developer use one computer to test as thoroughly as thousands would, and to let QA engineers make one test pass count for a thousand.  To learn more about what we’re doing with Jinx, check out the rest of our website!

While the intent of the products is similar, Jinx implementation as a hypervisor device driver and its checkpoint/simulation approach is very different from Cuzz. We haven’t gotten our hands on Cuzz yet, so our remarks here are in response to the presentation above, and we look forward to its public release. The Jinx pre-release product is available to the public with the completion of a short form. Take a look!

• A significant difference between Jinx and just about everything else is that we operate at the hardware/memory level instead of the threads/API level. Cuzz instruments synchronization libraries to manipulate thread schedules. In contrast, Jinx naturally handles native and CLR code, and exotic synchronization libraries such as Intel’s thread building blocks or custom-implemented wait-free data structures.This means that Jinx easily handles 32-bit native, 64-bit native, and CLR code, and can debug synchronization libraries too!

• Jinx takes checkpoints and Cuzz doesn’t. It appears Cuzz works by periodically introducing pauses into the mainline execution path. One big benefit from checkpoints is deep testing of code regions. We test a given code region N times, whereas Cuzz tests it once. This is particularly important for rare code paths. Interestingly, Microsoft’s Featherweight race detector relies on a “cold-region” hypothesis, which states that concurrency bugs tend to occur in rarely executed code. Jinx is well-suited to giving these rare regions good coverage.

• Cuzz has less control over target applications. As a user-mode DLL, Cuzz has limited insight into memory accesses inside the target application. Cuzz seems limited to inserting pauses at synchronization points. Jinx can insert pauses at arbitrary points — in particular, we can interleave at arbitrary memory communication boundaries. This means we can detect bugs that involve old-fashioned data races — for example, a multi-threaded program that uses no locks. It also means we can detect bugs inside synchronization primitives themselves.

• Cuzz is limited to a single process. Jinx can simulate a set of processes or the entire system. This is important because thread interleaving can be affected (and thus bugs be made manifest) by factors outside of your application

.

If you’re thinking of evaluating Cuzz when it’s available, take a look at Jinx in the meantime and let us know what you think.

Existing race detection tools have difficulty distinguishing benign races from true concurrency errors. This imposes a burden on the end user, who must manually inspect the output to remove false positives. Jinx eliminates most false positives by coupling race detection with simulation. First, Jinx identifies a program race, which may or may not be a bug. Then, Jinx exercises this race in the background. If the race is benign, Jinx discards the simulation. If the program execution results in a crash, Jinx moves the simulation to the foreground to make its effects visible to the user. Because of its simulation-driven nature, Jinx does not require a separate user interface; bugs simply appear more often on Jinx than they would on an ordinary system.

Internally, Jinx’s simulation engine uses a race detection algorithm in order to choose “interesting” thread schedules, which are likely to be involved in concurrency errors. Jinx’s race detector differs from happens-before detection in several key respects. Conventional race detectors suffer from false negatives because they focus on a limited set of program events (reads and writes of ordinary program variables). Jinx suffers from fewer false negatives because it considers any access to any memory location. Crucially, this includes synchronization primitives such as locks and atomic variables. As a result, Jinx will reorder critical sections in their entirety. This capability is necessary to find important classes of bugs, such as the increment race we considered in the previous post.

Jinx’s race detection algorithm is so general that it can even find bugs in low-level synchronization primitives such as locks and condition variables. Jinx can find bugs in the lock-free data structures built using atomic compare-and-swap. Conventional race detectors cannot find these types of program errors.

Jinx captures a larger set of concurrency errors than traditional data race detectors such as happens-before. Interestingly, Jinx can also reveal more data races than traditional tools. We do this by leveraging a capability called follow-on simulations. The output of each simulation is fed back into the race detector, which can then propose additional simulations.

Consider the following program execution (first considered by Savage et al.). The unsynchronized accesses to the variable y are a data race (and a bug!). However, a traditional happens-before race detector cannot reveal this error because all accesses to y are totally ordered by the intervening lock operations. By contrast, Jinx can find this bug by using a sequence of simulations. First, Jinx would reorder the critical sections (which are captured by the generalized race detector). Then, Jinx would reorder the accesses to y to expose the race.

Race Detectors Miss this one!

In summary, Jinx overcomes many of the issues associated with happens-before race detectors. Jinx’s generalized race detection algorithm means we expose a wider range of concurrency errors. At the same time, Jinx’s simulation engine hides uninteresting races from the end user. The result? Jinx is more likely to find bugs in your code, more quickly, before you ship your product to the market.

The goal of HB race detectors (and most race detectors) is to reveal a specific class of concurrency errors called data races. A data race occurs when two threads access a shared variable and the following conditions hold:

• At least one access is a write
• No mechanism ensures that the variable is only accessed by one thread at a time.

Many basic concurrency errors are data races. An example is when two threads increment a shared variable without using a lock or atomic operation:

int x = 0;

void incrementWithDataRace() {

  x++;

}

Before describing how HB detectors work, I need to describe the underlying shared memory execution model they presume. On a uniprocessor, all memory operations occur in a fixed order that is defined by the flow of execution. The first memory operation occurs, then the second, then the third, etc. If the program is re-executed with the same same inputs, it will execute the same sequence of memory operations and produce the same output. Life is good.

On a multiprocessor, life is not so simple. The underlying system (the hardware, OS, and language runtime) is free to schedule threads/processors in any order it sees fit. If the program is re-executed, it can (and typically will) run with a different thread schedule, potentially producing a different result. Even worse, different threads may disagree on the order in which memory operations occurred (the exact details depend on the underlying memory consistency model, which is a topic beyond the scope of this post. Stay tuned!). This multiplicity of different outcomes is what makes parallel programming so demanding and error-prone.

If memory operations could occur in any order, then programmers would be helpless to write multi-threaded programs that shared data in any way. Fortunately, there is a lifeline: synchronization operations such as locks and condition variables place constraints on the set of permissible memory reorderings. In the academic lingo, they say that one synchronization operation happens-before another. This means that both threads agree on the order these operations occurred, and that the underlying system will not reorder memory accesses across this pair of operations.

Consider the simple example shown below. In this case, two threads correctly increment a shared variable using a lock. We say that the unlock operation happens-before any subsequent lock operation. Crucially, the happens-before relation is transitive, so all operations in thread A before the unlock happen-before all operations in thread B after the lock. In this case, this means that thread A’s increment of x happens-before thread B’s increment of x.

Given these concepts, it is straightforward to describe the behavior of HB race detectors. Suppose two threads access a shared variable, and one of those accesses is a write. In a correct program, all such accesses should be totally ordered by the happens-before relation. A data race occurs when one of those accesses does not happen-before the other. In the example above, we could create a data race by removing the lock operations. In fact, it would suffice to remove the lock operations from one of the two threads.

As this example suggets, HB detectors are a powerful technology for detecting certain classes of concurrency errors. However, they suffer from two basic problems. First, not all data races are concurrency bugs; we refer to these as false positives. Second, not all concurrency bugs are data races; we refer to these as false negatives. As a result, a HB race race detector is both incomplete and inaccurate in its ability to detect the complete spectrum of concurrency errors. These concepts are illustrated in the Venn diagram below.

The code snippet below shows a simple concurrency bug that is not a data race. In this example, the programmer is incrementing a counter variable that is accessed by multiple threads. The programmer is incorrectly using a temporary variable to make modifications to the shared variable. The program will produce an incorrect result if a thread is preempted between the two critical sections. However, this bug will not be found by a HB race detector, or any tool that focuses exclusively on data races. This is because all accesses to shared, mutable state are properly protected by a lock.

int x = 0;

void incrementWithoutDataRace() {

  int tmp;

  lock(L);

  temp = x;

  unlock(L);

  temp++;

  lock(L);

  x = temp;

  unlock(L);

}

What about the other side of the coin? Are there benign data races that do not result incorrect program behavior. Yes. One example is the use of a flag variable. A common paradigm is to construct a pool of worker threads that periodically poll a flag variable to determine whether they should terminate. Clearly, such a flag variable is shared, mutable state. However, flag variables are typically not locked, yet this does not result in incorrect program behavior. Programs written in this fashion are immune to such data minor races.

So, HB race detectors are not a perfect solution for detecting concurrency errors. In a follow-up post, I will discuss some of the technology we are developing at PetraVM that addresses these limitations.