The phrase “You are Not Expected to Understand This” is probably the most famous comment in the history of Unix.
And last month, at the Systems We Love conference in San Francisco, systems researcher Arun Thomas explained to an audience exactly what it was that they weren’t supposed to understand.
Computer science teacher Ozan Onay, who was in the audience, called it “one of my favorite talks of the day,” writing on his blog that “Nothing should be a black box, even when Dennis Ritchie says it’s ok!”
Thomas opened his talk by saying “I love systems — operating systems in particular.” But even he had to marvel at the popularity of this 42-year-old Unix source code comment: “You started seeing people wearing sweatshirts, t-shirts, and baby onesies with ‘You are not expected to understand this’ It took off in the hacker community.” And geeks have been riffing on it ever since.
The code comment originally appeared in the Sixth Edition Unix operating system, describing context switching — or, as Thomas put it, “the mechanism that allows for time-sharing and multi-tasking … essentially how a computer is allowed to be shared by multiple concurrent users and concurrent applications.”
Basically, a context is a mix of stack pointers and registers that allows Unix processes to be paused and resumed. “It’s a neat trick, and Unix’s time-sharing support was a big reason why it took off. It was a big leap forward in usability,” Thomas noted. “Some things have changed over the years, but the overall scheme has fundamentally remained the same.”
Thomas reminded the audience of Unix co-creator Dennis Ritchie‘s own “Comment about the comment” web page on the subject:
“It’s often quoted as a slur on the quantity or quality of the comments in the Bell Labs research releases of Unix. Not an unfair observation in general, I fear, but in this case unjustified… we tried to explain what was going on. ‘You are not expected to understand this’ was intended as a remark in the spirit of ‘This won’t be on the exam,’ rather than as an impudent challenge.”
Ritchie saw it as just another fun piece of geek history. (On the same page, he also explained why the mv command sometimes gave the diagnostic message “values of β will give rise to dom!”) But Ritchie was aware that his 1975 comment had attracted a cult following:
“I’ve even been given two sweatshirts that quote it,” he wrote.
Thomas picks up the story from there. “Another amusing bit that Ritchie shared was it turns out the code was actually buggy, and that the Unix authors didn’t completely understand it either.”
“So the moral of the story is that context switching can be tricky, even for the creators of Unix.”
Unix v6 was released in 1975. “It was roughly 9,000 lines of code,” Thomas points out, “so, small enough that it could be studied in a university course.” He remembers how in 1979 AT&T changed the licensing terms for Unix to allow only official licensees to the study the code.
But the entire source code for version 6 had already been published in a book by John Lions, a professor at the University of New South Wales, along with his commentary, making the book unusually popular. “People were making photocopies of photocopies,” Thomas tells the audience — noting it’s a phenomenon that was ultimately even commemorated in the book’s cover:
The book’s Wikipedia entry states “It was commonly held to be the most copied book in computer science.”
“I gave an early version of this talk at Papers We Love Boston, and one of the attendees showed up with his unauthorized copy of Lions. It’s kind of cool that he still had it, and really kind of treasured it.”
But it’s this book’s popularity that also helped publicize Ritchie’s throw-away comment.
So what was it that the source code didn’t expect us to understand? Thomas quickly delved into how Unix defines a process — with a virtual processor and its associated memory address space — and explained to the audience the hardware involved. It’s the CPU that isolates memory for the processes, while also keeping track of whether privileged operations are allowed and supports interrupts and exceptions.
Inevitably, this leads to a point where a CPU context needs to be saved (for restoring later) whenever processes are switched. “This is constantly happening. Processes are getting context-switched out, and then context-switched back in,” Thomas told the audience. When a time slice of CPU time gets used up, or when there’s an interrupt signal, a page fault, or a system call, the context gets switched. “Processes keep getting juggled,” Thomas said.
“You can picture the OS as the bouncer at club CPU: if you get too rowdy, you get kicked out. And if a VIP buys out the place, you get kicked out as well, but you can probably come back the next day.”
How this all happens is a little trickier: the CPU and kernel save the first process’s context in a data structure, identify the next highest-priority process to make a scheduling decision, and then restore the context for the next process.
But to fully appreciate Ritchie’s comment, Thomas pulled up snippets of the actual Unix source code for his audience, giving them a time-travelling peek back to 1975.
There they saw how a swtch() routine calls a context-saving savu() subroutine, whose responsibility was to save your context in a kernel data structure named u_rsav(). More of the context is saved elsewhere. Then a retu() subroutine restores some of the next context, while the sureg() subroutine helps prepare its virtual address space. “The full details are in Lions — and it’s actually fairly complicated — but it’s an interesting read.”
Though remember — you’re not expected to understand it.
The original Unix V6 code for swtch() also had some other more straight-forward comments:
/* Remember stack of caller /* Search for highest-priority runnable process /* Switch to stack of the new process and /* set up his segmentation registers.
But alas, that most beloved comment of all soon disappeared in just a few years, when the context-switching code was re-written for Unix v7 in 1979. But fast forward to today, and you’ll still find a switching subroutine in FreeBSD, Thomas explained — though it’s called cpu_switch() because it’s now CPU-specific — and the scheduling code has been moved to sched_switch(), which is machine-independent.
So in the end, we’re all building on the work of the geeks who came before us. Thomas reminded the audience that a modern Unix, like today’s FreeBSD, is more complicated — there are threads and dynamic linking, for example — but “the process abstraction is fundamentally the same.” Scheduling algorithms have also become more advanced, and the hardware landscape has obviously changed too since the days of the PDP-11. But in the end, “Modern operating systems fundamentally do timesharing the same way, by using the process abstraction and context switching.”
Thomas’s talk gave me a new respect for all the heavy lifting that was done in 1975, and he seemed to recognize it too. “Timesharing is not easy to get going — the hardware is complicated — and I hope I gave you a flavor of the complications,” he told the San Francisco audience, adding one last context switch of his own.
“Even if I failed at the task, and I didn’t get across the importance of the process abstraction and context switching, you’ll at least understand my t-shirt now.”
Joyent is a sponsor of The New Stack.
Feature image: Notes taken by Sarah Huffman, during Arun Thomas talk, at Systems We Love.