Quantcast
Channel: Hacker News
Viewing all 25817 articles
Browse latest View live

Trying to corner the market on legal marijuana

$
0
0

Some very powerful people are trying to corner the market on legal weed and turn their company into the Monsanto of marijuana. Who are they? And can they be stopped?

The search for the hidden forces that might soon control the marijuana industry began, as many wild journeys do, in Las Vegas. It was last November, and I was party-hopping at the biggest weed-business gathering of the year, a week of overlapping conferences and decadent soirees. I was a few blocks off the Strip, celebrating a new line of bongs and pipes in a penthouse with chandeliers and dark-wood furniture, when I happened to meet a faunlike 40-something man named after a character from The Jungle Book: Mowgli Holmes.

Holmes had something he needed to get off his chest—a quagmire that had been keeping him up at night for the better part of a year. He was soft-spoken but had an earnest intensity that made me lean in to hear him. Little did I know that he was about to set me off on a months-long quest that would involve an obscure company potentially worth hundreds of millions of dollars, the most prominent cannabis scientists in the world, and the former talk-show host Montel Williams—all to uncover an audacious plot with profound implications for one of the country’s biggest agricultural products.

All around us, former drug dealers chatted with finance types, passing vape pens as they pressed one another for information, boasting all the while about how much smarter and better positioned they were than everyone else there. The people in this room were vying to become major players in a marijuana industry that was getting larger every day. A sense of inevitability around legalization had left everyone giddy, but few understood that the greatest obstacles were yet to come. Holmes pulled out his phone and showed me what he saw when he looked at this surge of people working in weed: a 3-D visualization he’d created to illustrate the thousands of kinds of pot now on the market.

Turns out he had a Ph.D. from Columbia and ran a lab in Portland, Oregon, where he’d been mapping the genetics of every marijuana strain he could get his hands on. He pointed to the cluster of strains that taste like tangerine, and then the ones that provide a calming high with none of the mind-racing anxiety so many casual users despise. There were the famous strains, like Sour Diesel and Blue Dream, as well as little-known varieties bred as folk medicine by underground botanists. But this database—a cornucopia of cannabis DNA—had captured a very specific moment in time, a moment Holmes believed would be over soon. The age of artisanal marijuana might have already peaked, and the era of corporate pot was just beginning.

Intrigued, I met up with him the next day to hear more. The first thing he said was: “Be nervous.”

According to Holmes, a secretive company called BioTech Institute LLC had begun registering patents on the cannabis plant. Three have already been granted, and several more are in the pipeline, both in the U.S. and internationally. And these are not narrow patents on individual strains like Sour Diesel. These are utility patents, the strongest intellectual-property protection available for crops. Utility patents are so strict that almost everyone who comes in contact with the plant could be hit with a licensing fee: growers and shops, of course, but also anyone looking to breed new varieties or conduct research. Even after someone pays a royalty, they can’t use the seeds produced by the plants they grow. They can only buy more patented seeds.

“Utility patents are big. Scary,” Holmes said. “All of cannabis could be locked up. They could sue people for growing in their own backyards.”

Pot is an industry worth over $40 billion, which makes it the second-most-valuable crop in the U.S. after corn. And even though weed is still federally forbidden, it sounded like whoever was behind BioTech Institute had spent the past several years surreptitiously maneuvering to grab every marijuana farmer, vendor, and scientist in the country by the balls, so that once the drug became legal, all they’d have to do to collect payment is squeeze.

Very High Stakes

“The craziest part is no one knows who’s behind this or when they plan to enforce what they have,” Holmes said. He’d heard that BioTech Institute was owned by a mysterious billionaire with an unclear agenda.

Even worse, he believed, the patents might affect our ability to unlock the therapeutic potential of cannabis. With all other crops, from soybeans to carrots, the introduction of utility patents has created obstacles that threaten to reduce genetic diversity. Right now, the word “cannabis” is similar to the word “dog”—covering everything from Chihuahuas to huskies. Each plant contains its own mix of compounds. Some strains suppress your appetite, and others give you the munchies. Some get you high, and some don’t. Many seem to have incredible healing properties. Research in mice has shown that the compounds we’re most familiar with can do everything from increase bone density to shrink tumors, but we’ve hardly scratched the surface on the less common compounds.

“The medical gold mines that we’re looking for are buried in random varieties that are spread around,” Holmes said. Like many people, he hoped legalization would lead to clinical trials, to prove the benefits of marijuana are more than anecdotal. But he worried that one company controlling access to so many kinds of pot would drastically restrict research. And considering how long weed had been illegal and underground, he felt it was ludicrous to claim all these strains were truly novel, as patent law requires.

“I want someone to challenge the patents by proving all of these strains existed before.” He blinked and met my eyes with a solemn zeal. Here came the sales pitch. Questioning the patents’ validity would require cooperation from all corners of the marijuana world. He had a database of strains, but he didn’t think that would be enough. Would I help him get the word out, so more people knew about BioTech Institute?

I didn’t say anything, and then, just like that, Holmes was gone. This was crazy. Wasn’t it? I alternated between feeling paranoid that he might be right about what a big deal these patents were and considering the possibility he’d been exaggerating or making it all up. I felt stoned and bewitched. But if what Holmes was saying was true, it was explosive.

I staggered down the hall and ran into a trusted source who does serious medical work. When I asked her about BioTech Institute, she gasped and forwarded me a memo that had been circulating among the most influential people in marijuana for two months. This formal explanation of BioTech Institute’s entirely legal annexation of weed’s intellectual property had been written by another pot geneticist, Reggie Gaudino. He estimated that together, all of the company’s desired patents would be worth hundreds of millions of dollars and could eventually affect every marijuana strain currently being bred. Every. Single. Strain.

After Vegas, I brought up this story with every contact I encountered. Most had no idea what I was talking about. But it was the people who did who really alarmed me. Eyes grew wide. Voices became either hushed or loud and angry. Texting buddies suddenly needed to confirm we were off the record. In four years of observing the marijuana game, I’d never encountered a situation that struck fear and fury in so many people’s hearts.

The Invisible Company

Outside of these patents, BioTech Institute barely exists. The company has no website, manufactures no products, and owns no pot shops. Public records for BioTech Institute turned up two Los Angeles addresses—a leafy office park an hour northwest of downtown and a suite in a Westside skyscraper—both of which led to lawyers who didn’t want to talk.

A source familiar with BioTech Institute’s patenting process estimated that the company had spent at least $250,000 in research and legal fees on each of its patents. I knew that if I could figure out who was paying for the patents, I might learn who held the keys to the future of the marijuana industry. But I hardly knew where to start.

Broadly speaking, the people in pot can be divided into three groups, based on when they decided to start working with weed. First there are the Outlaws. These are the drug dealers and the backwoods contrarians: the folks who knowingly chose a so-called life of crime, often because they dislike mainstream society or believe in the spiritual and physical benefits of cannabis.

Then there are the Green Rushers: the financiers and entrepreneurs who only got involved once it looked like the country was moving inexorably toward legalization. Many Green Rushers like to get high, but most are primarily concerned with making money. They typically try to take as little legal risk as possible, which often means starting businesses that relate to marijuana but do not involve the plant itself. Banks fall under federal law, so anyone who opens a dispensary could have their assets seized.

Finally, there are the people I think of as Code-Switchers. Most Code-Switchers turned to weed back when the economy was faltering and there was an even greater overlap between the black market and the medical-marijuana market than there is now. Code-Switchers saw regulated pot on the horizon and perhaps had experience in a legal industry but still had to contend with shady situations and police raids. They learned to speak the language of the Outlaws, but they also knew how to appear professional in front of the Green Rushers, giving them the powerful (and lucrative) ability to toggle between worlds, or code-switch.

The most notable inventor on BioTech Institute’s patents is a Code-Switcher named Michael Backes, a renowned figure in California marijuana circles. Some call him Morpheus, his username on weed message boards, and he does look a bit like a white version of the Matrix character: bald and defiant, with piercing eyes. Backes co-founded one of the most medically legitimate marijuana dispensaries in L.A. and is familiar with rare strains from around the world. He’s also consulted on Michael Crichton novels and Hollywood blockbusters.

I e-mailed Backes. He refused to talk and said he was no longer working with BioTech Institute.

Further poking around led me to a handful of companies run by people associated with BioTech Institute, based out of the same lawyers’ offices. There was BHC Consultants. There was NaPro Research. And then, most interestingly, there was Phytecs.

Phytecs has a website but does not yet sell anything and never uses the words “marijuana” or “cannabis”—a classic Green Rusher move to minimize legal risk. Instead, Phytecs talks about “products and therapies targeting the endocannabinoid system,” meaning the receptors in the body that respond to pot’s primary compounds, called cannabinoids. Working with cannabinoids is generally more legal than working with cannabis because they can be synthesized in a lab or extracted from other plants.

I was stunned to see that the Phytecs team included two of the greatest authorities on cannabis in the world: the legendary biochemist who discovered that THC is the thing that gets you high and a neurologist named Ethan Russo, who helped develop the first modern marijuana-derived prescription drugs. Over the phone, Russo told me that Phytecs is “a very small, circumscribed R&D company” with plans to make products both with and without the cannabis plant. (Phytecs later clarified that the company works with compounds from “a range of sources, both natural and semi-synthetic.”)

“We have a very active program in dermatology,” Russo said, potentially involving both prescription products and over-the-counter cosmetics.

So were Phytecs and BioTech Institute part of a single, larger undertaking? Russo directed all of my questions about the business operations, and the patents, to the lawyers.

“Some of this stuff is delicate. Obviously we’re engaged in a business, the legality of which, well… We’ve got a new administration...” Russo said, and wondered aloud what would happen with cannabis under Trump, concluding: “You have to be circumspect.”

The Greed of the Outlaws

It was around this time that I began to rule out the possibility that an Outlaw was behind BioTech Institute. Everything seemed too grounded in conventional business logic. BioTech Institute appeared to be following in the footsteps of Big Ag companies like Pioneer and Monsanto, which began accumulating utility patents on conventional crops in the 1980s, when patents on plants became legal.

At first, Big Ag sought patents to protect the genetically modified organisms that took years and millions of dollars to develop, like pesticide-laced corn. But over time, acquiring plant-utility patents turned into a race to call dibs. In one case, a seed company got a utility patent on a centuries-old yellow bean widely available in Mexico, and it took almost ten years to definitively overturn the claim.

Monsanto in particular became shorthand for rapacious capitalism and corporate influence over government policy—exactly the things that most hippie Outlaws say they were trying to escape by growing weed off-grid.

Considering how expensive patent battles have become in Silicon Valley, an experienced tech investor would know to lock down pot’s intellectual property early.

In February, I went to the mountains of Northern California, where tens of thousands of marijuana farmers have lived since the 1970s. One Sunday, I had breakfast at the Eel River Café with two men in their 50s who have been growing pot for their entire adult lives: Kevin Jodrey, the proprietor of a cannabis nursery, and his friend Ziggy.

Jodrey is a fast-talking community leader who says his nursery’s clones produce $150 million to $200 million worth of weed each year. Recently, he’s been encouraging farmers to get their strains genetically sequenced by Mowgli Holmes, the scientist from Portland. In his fight against utility patents, Holmes had created an option for growers to put their strain data in the public domain.

“I’m trying to get everybody I know to tag every fucking thing they have,” Jodrey said between bites of scrambled eggs and hash browns. “This way, no one’s gonna be able to come to my farm and hit me with a ‘You need to pay me’ kind of thing. That’s just usury.”

Only a quarter of the farmers he’d spoken to had made their genetics public.

Many were suspicious of Holmes. After all, he too was running a business. His company planned to use genetic data to help breed better cannabis. So Holmes might also end up selling marijuana seeds, and the patents could eventually affect his bottom line.

Other Outlaws told Jodrey they hoped to get their own patents.

I began to see the holes in the nostalgic notion that before the suits came in, everyone who grew pot was an altruistic bohemian out to spread the love. Drug dealing is profitable: That’s why people take huge risks to do it.

Consider Ziggy and Jodrey. They want to stop BioTech Institute, sure, but they also respect the hustle.

“If I saw the dollars that were available in cannabis, and I had the funding to do a clandestine project that gave me a monopoly? Absolutely I’d take that,” Ziggy said.

“That’s good business,” Jodrey conceded, “but get ready. We’re coming back at you, buddy.” He wagged his finger, addressing the patent inventors. “If you don’t think we’re gonna brawl it out, you’re crazy.”

Chasing Millionaires and Billionaires

In all likelihood, the people behind BioTech Institute were Green Rushers who stayed anonymous to avoid being associated with a quasi-legal endeavor.

If what Holmes had heard was true, and the investor was indeed a billionaire, one obvious candidate would be Sean Parker. You might remember him as the guy who co-founded Napster and was played by Justin Timberlake in The Social Network. The maverick tech magnate gave over $7 million to the 2016 weed-legalization campaign in California. Parker’s insistence that his donations were purely philanthropic merely fueled speculation that he had covert plans to invest in pot. Considering how expensive patent battles have become in Silicon Valley, an experienced tech investor would know to lock down pot’s intellectual property early.

Parker’s publicist stated that Parker is not involved in BioTech Institute, nor does he have any investments in the cannabis industry.

The investor wasn’t necessarily a billionaire. There are tons of Green Rushers worth merely tens or hundreds of millions of dollars, any one of whom could afford to pull off a feat like BioTech Institute. Plenty of Silicon Valley types, lured by the relatively permissive and professional marijuana economy of the Bay Area. Scores of pothead Hollywood producers, hip-hop greats, and retired athletes. The kind of Wall Street guys who played roulette with our economy and lost but never experienced any consequences for that, so don’t really mind violating federal law. Titans of the alcohol and pharmaceutical industries who saw a shift coming and thought, If you can’t beat ’em, join ’em. Really, the investor could be anyone.

By chance, right when I began trying to track down potential pot plutocrats, I was invited to a dinner for cannabis investors, organized by a former Bear Stearns trader named Gregg Schreiber.

“This is extremely high-end shit,” Schreiber told me. “These are not millionaires. These are billionaires!”

Billionaires! He rattled off names of high-net-worth individuals who would be there. So on a Wednesday in February, I made my way to a boutique hotel near the ocean for the first Green Table dinner party.

The crowd was mostly gray-haired white men in suit jackets or cashmere. There was champagne and wine and a weed-infused meal—a series of four-bite courses plated with jubilant swirls of sauce—served at two long tables with candles and centerpieces of twigs and tulips. A guy soloed on a cello in the corner. Becky with the Good Hair was there, and several former football players. There was a handful of pot entrepreneurs and a rep from High Times. There was a guy with long blond hair wearing a suit with a pocket square and flip-flops he claimed were Gucci and worth $500.

There were so many Green Rushers, I figured someone must know something. I met an investor in a suit and a backward baseball cap who played professional polo. When I brought up pot patents, he thought I meant trademarks, like logos and brand names. I approached person after person, asking with practiced nonchalance whether anyone knew anyone investing in marijuana patents, but no one did.

Disappointed, I decided most Green Rushers were too, well, green to put together something like BioTech Institute. So if the Outlaws knew too much and the Green Rushers knew too little, perhaps the people I was searching for fell somewhere in the middle. And sure enough, there was one promising lead I needed to follow up on, and it involved a very famous Code-Switcher.

A Clue from Montel Williams

The investigation took a surreal turn on another day in February, when I spent a nervous afternoon hour in the lobby of the Ritz-Carlton in downtown L.A., waiting for a celebrity who was running late. My snooping had led to a rather unexpected person: former daytime-talk-show host Montel Williams. Unlike Whoopi Goldberg or Willie Nelson, Green Rushers who jumped into the weed game only recently, Williams was a pot entrepreneur long before it was trendy—a true Code-Switcher.

I suspected Williams might be involved in the patents because, in 2011, he had helped start a Sacramento marijuana dispensary called Abatin Wellness Center, which shared key personnel with BioTech Institute. In 2012, Backes was described in an interview as the “head of R&D for Abatin,” and in 2016, the attorney who runs Phytecs and BioTech Institute was cited as the president of Abatin. On a hunch, I called the current face of the Sacramento dispensary, Aundre Speciale, who said she still worked with Backes, though she herself wasn’t part of Phytecs or BioTech Institute.

“It’s separate groups, and we have some overlapping things,” she said. “We have a lot of crossover.”

Williams had mentioned in a 2013 interview that he was no longer working with Abatin, but it seemed totally possible that he had the money and the know-how to fund BioTech Institute. After all, the man was on TV for 17 years and has been paid to hawk everything from blenders to payday loans.

Williams finally arrived at the Ritz, wearing a lilac button-down with a white collar, as well as several pieces of shiny, perhaps diamond-encrusted jewelry, and immediately announced, “I gotta go wash off some makeup.” He disappeared, and I was left with Jonathan Franks, his right-hand man, the LeFou to Williams’s Gaston. Franks sipped a Monster energy drink and name-dropped his client at the front desk to finagle some space for us to talk in the Wolfgang Puck restaurant on the 24th floor, where we settled into a booth with waist-to-ceiling views of the 110 Freeway and several choice parking lots.

Williams has been preaching the benefits of medical marijuana since he was diagnosed with MS in 1999. Now he was in town to promote his new cannabis-oil line, Lenitiv Labs, at an event that night called the CannaCool Lounge.

Unless the hustlers in the rest of the cannabis industry take a break from shoving one another aside to work together against the patents, BioTech Institute could have the market cornered in a few years.

Soon after the star plopped down across from me, it became clear that he treats much of his daily life as if he’s conducting an IRL infomercial. He hardly breaks eye contact, and I was quickly won over by his Bill Clinton-level charm and his fluidity with pot jargon. He certainly knew more about weed than any of the Green Rushers at that dinner party.

Williams kept mentioning his “proprietary” formulations. I asked whether he had acquired any patents.

At another point, Phytecs’ medical director, Ethan Russo, came up in conversation, and I asked Williams if they were working together. He said they had in the past, back when he was with Abatin.

I was surprised to hear Williams associate Russo with his time at Abatin. Russo denied working with Abatin.

I asked Williams if he had worked with Michael Backes at Abatin.

“Yes, yes, and then I kind of broke away from that group of guys completely, and I got out of the business for a while,” he said. I asked why, but he wouldn’t say.

So Williams hadn’t funded the patents, but it really was starting to sound like the people behind Abatin, Phytecs, and BioTech Institute had been working together for a while.

After the interview, I was waiting for the valet when, suddenly, Franks started telling me why Williams had left Abatin.

“It was keeping him up at night,” he said. “And quite frankly, it was keeping the investors up at night, too. Because the investor in that room has two children at home. A wife. There were times where he was, like, he looked like he had aged ten years in 18 months.”

Aha! The investor. As in, a single investor. Who has two children. Now we were getting somewhere. This could very well be the same person who was financing BioTech Institute.

Just then, my Honda pulled up.

I arrived at the CannaCool Lounge a few hours later, determined to get Franks to tell me more. Several hundred Green Rushers mingled in a ballroom encircled by faux-gilded columns, sampling pot-laced polenta and pork sliders and ice pops as they passed by on silver platters. Most partygoers were black and wearing red-carpet-ready ensembles. A lesser-known member of the Wu-Tang Clan was there.

Williams, in a black turtleneck and black jeans, was doing what he does best: connecting with fans and selling the hell out of his product.

“Push this button right there,” he instructed an older woman, handing her a vape pen. She took a hit. “Lady, you got a big draw,” he said. “She’s a hose! A vacuum!”

By the time I found Franks, sweat was rolling down his face. I asked about the investor, and he smirked. “You’ll never find him,” he said. “There’s no trace of him on the Internet.”

So Parker was definitively out. His every move, from Napster to Facebook to his lavish wedding in the woods, had been public. I asked about the relationship between Abatin and the other companies.

“They’ve had so many names,” Franks said. “One guy has the money.”

“Well, how did he make his money?” I asked.

“Cosmetics,” he said. Russo’s words about Phytecs developing skin-care products came back to me.

“I worked for him for a while,” Franks added. “He lives in Beverly Hills. They’re not bad guys.”

I asked for more specifics, but Franks demurred. Okay, I said. If he couldn’t tell me the investor’s name, maybe he could tell me where to find it on my own?

Franks thought for a minute. Another drop of sweat ran down his cheek. “Abatin tried to open a dispensary in Washington, D.C.,” he said. “Look there.”

Uncovering the Pot Plot

A month later, I got a series of e-mails from the Washington, D.C., Department of Health. It was the results of my Freedom of Information Act request for every application to their medical-marijuana program. Abatin hadn’t received a dispensary license, but it was approved for a cultivation center that is currently growing cannabis for D.C. patients. I found Abatin’s documents and began scrolling. On the first full page, in a list of “each member of the partnership or limited liability company,” just beneath Montel B. Williams, was the name of a man who lived in Beverly Hills and had made his money in cosmetics: Shawn Sedaghat.

After all that, was it just some random guy?

In one respect, Franks had been wrong. Once I had his name, Sedaghat was not impossible to find online. According to the L.A. Times, Sedaghat sold his family’s cosmetics-packaging company for $255 million in 1997. Since then, he’d founded at least two more cosmetics businesses. And his $10.3 million mansion in Beverly Hills was featured in the music video for Britney Spears’s “Slumber Party.”

I also discovered that, in the back of Backes’s 2014 book, Cannabis Pharmacy, one of the first people he thanked was “Shawn for his peerless vision.” A source familiar with Backes’s work independently mentioned that his research was being bankrolled by a Middle Eastern man who ran a cosmetics-packaging company. And Fred Gardner, editor of the medical-marijuana journal O’Shaughnessy’s, said that one of the other inventors on BioTech Institute’s patents had introduced him to an Iranian man named Shawn at a cannabis festival in 2014 or 2015.

“I could tell from how they were talking, like, ‘Oh, Shawn is here.’ Like Bob Dylan had walked into the room.” Gardner recalled. “I was definitely left with the impression that he was the decision maker in their business.”

But was Sedaghat really a billionaire? In an unsuccessful 2011 Abatin application to open a dispensary in New Jersey, a letter from a bank read: “Shawn Sedaghat and affiliates maintain balances in the high eight figures”—that is, nearly $100 million.

I soon learned that Sedaghat’s current cosmetics-packaging concern, Yonwoo/PKG, has an office in the same skyscraper as Phytecs and BioTech Institute. Yonwoo/PKG is in suite 2240. Suite 2250 belongs to Gary Hiller, the lawyer who runs Phytecs and BioTech Institute.

Each time I went there, I was told that Hiller and Sedaghat were out of town. Once, a well-known marijuana activist named Don Duncan answered the door to Hiller’s suite and said he worked for BHC Consultants. Who wasn’t part of this murky network of companies? I wondered. Duncan referred to the two suites as “separate businesses but all the same family.” (He later denied knowledge of Sedaghat’s involvement.)

It was certainly starting to look like Hiller and Sedaghat were behind a stealthy conglomerate. I later found at least seven businesses with both Hiller’s and Sedaghat’s names or addresses attached, six of which have filed renewal forms in the past two years. And a cannabis researcher who was approached about a job by someone involved with BioTech Institute told me, “One hundred percent, this is a single group of people working together under several names.”

Hiller said that Sedaghat was not the primary investor behind BioTech Institute and that BioTech Institute and Phytecs and NaPro and BHC “do not have a relationship, other than the fact that I am involved with each of these companies,” although “individuals from some of these companies have collaborated with one another from time to time.”

Sedaghat’s attorney confirmed that he is an investor in Phytecs and “is an investor and has business relationships with a number of non-cannabis-based, multi-national public companies that may have had, or currently have, some involvement with securing intellectual property related to cannabis.” However, other than that, Sedaghat has not “been an investor in, or in any sense been professionally affiliated with, any cannabis-related business” since 2015.

The whole thing seemed more confounding than ever. What “multi-national public companies” was Sedaghat’s attorney referring to? He seemed to be allowing for the possibility that a broader network of investors and companies could be occupying this space.

Either way, with hardly anyone paying attention, whoever was behind BioTech Institute had pulled off a brilliant and ostensibly legal coup. I still didn’t know whether they intended to start charging licensing fees and, if so, whether they would allow research to be done at an affordable rate. It suddenly felt possible that even if the U.S. legalized marijuana, the anticipated boom in clinical trials and medical discoveries would never come to pass. A potential revolution in health and wellness might become a whimper, with innovation hemmed in by the restrictive clauses of utility patents.

But BioTech Institute didn’t see it that way. Hiller said that the company’s work “will directly facilitate research in the cannabis field that was not previously possible.” And it’s true that pharmaceutical companies use the profits from their patented drugs to fund research and development for new therapies. Perhaps BioTech Institute would become a one-stop shop for every possible permutation of medically beneficial marijuana products.

Still, the all-encompassing ramifications of what had been accomplished by tight-lipped people working in secret were overwhelming. If the rest of its patents get approved, and they hold up legally, BioTech Institute might have a near-monopoly over crucial intellectual property for a commodity whose soothing, mind-altering effects make it more valuable than wheat. Unless the hustlers in the rest of the cannabis industry take a break from shoving one another aside to work together against the patents, BioTech Institute could have the market cornered in a few years. The company could control access to a plant that has been shown to reduce tremors in Parkinson’s patients, to help veterans stop dreaming in flashbacks, to relieve the nausea of anyone undergoing chemotherapy. They could charge whatever they wanted.

And there would be nothing anyone else could do about it.

Amanda Chicago Lewisis a writer based in Los Angeles. This is her first article for GQ.

This story originally appeared in the September 2017 issue with the title “The Great Pot Monopoly Mystery”


Why Are So Many Wingsuit BASE Jumpers Dying? (2016)

$
0
0

In researching 2016’s dramatic rise in BASE jumping deaths, I was almost unable to keep up with the pace with which people were dying.

I even watched one of these deaths in real time, live-streamed on Facebook.

A guy with the handle of “Sat Dex” popped up in my feed on the morning of August 26, broadcasting himself via Facebook Live.

The video opens with him stepping into a wingsuit. He has dark hair, a sleeve tattoo, and a Hollywood-style beard. He speaks in German. He gives the finger to the camera and grins. He zips up his suit, flashing more smiles with a sort of nervous or excited energy, the kind you might associate with a child opening a birthday gift.

“Today you fly with me,” he says in German.

He waves into his outstretched phone. The video goes dark as the phone is now inside his wingsuit, ostensibly in his hand, and still live-streaming to Facebook.

Now I hear a whoosh of air. The sound of airflow grows, reaching a ferocious decibel. The turbulent din lasts no more than a few seconds.

Suddenly I hear the man emit an acute bellow. Then, pandemonium. He’s tumbling and tumbling. I hear cowbells. The tumbling stops. It’s silent. Cows continue milling around in what I’ve imagined to be idyllic mountain scenery somewhere in Europe, with giant blue limestone cliffs towering over rolling green pastures.

Then long, low, soft moans.

Then all that is left is cowbells.

I later learned this man’s real name was Armin Schmieder. He was Italian, but lived in Freiburg, Germany. He was a father to a young child. He was 28.

Facebook did not immediately take down the video, despite outrage from many commenters, including his family. The social media site slapped on a disclaimer: “Warning - Graphic Video.” Yet the views continued to pile up.

Meanwhile, a rather grim irony remained, as per Facebook’s design protocol for these types of posts. It said: “Sat Dex was live.”

After 36 hours, the video was finally removed.

The Killing Season

The end of summer can’t come fast enough for many in the speed-fueled world of wingsuit BASE jumping (BASE is an acronym standing for the types of objects participants may leap from: Buildings, Antennas, bridges—aka, “Spans”—and the Earth itself, in the form of cliffs or promontories). This has already become the deadliest year on record for BASE jumping, with at least 31 deaths thus far. Twenty-three of those fatalities occurred this summer—six deaths in June, two in July, and 15 in August. The fatal spree has spurred practitioners to dub summer as “Wingsuit BASE Killing Season.”

“It’s been a horrific last couple of months,” says Richard Webb, a former fighter pilot for the U.S. Navy, current private pilot, and active wingsuit BASE jumper from Moab, Utah. (I profiled Webb last year for his unique, scientific approach to opening new wingsuit jumps around Moab.)

“This is easily the worst season I can remember,” he says. “And, honestly, I haven’t even been keeping up with who’s been going in. I’m tired of the carnage.”

Summer is a time when the European Alps—with its myriad locations that feature highly accessible, legal, and very large cliffs from which to jump—are devoid of snow and are considered to be in good condition for flying. Like moths to a flame, wingsuit BASE jumpers from around the world descend on the Alps each summer to get their fix.

Aside from the record fatality count, the BASE world hit another morbid milestone in August. The BASE Fatality List (BFL), an unofficial and non-comprehensive wiki that records BASE fatalities dating back to 1981 for educational purposes within the community, surpassed 300 names. The greater cause for concern, though, is that the BFL appears to be trending at an accelerated rate—more than 260 of those names have been recorded since 2000.

So, why are so many BASE jumpers dying?

“The simplest answer is wingsuits,” says Webb. “Right now, wingsuit BASE jumping is, globally, the hottest thing going for the impressionable, 18- to 35-year-old single-male demographic.”

BASE jumping has no organizing bodies that keep track of participation numbers. Anecdotally, the sport is growing, perhaps as evidenced by the increasing numbers of people who are dying.

This year’s constant, gruesome news has spurred some BASE jumpers to rebuke their wingsuiting counterparts. “Sketchy Andy” Lewis, one of the world’s most accomplished BASE jumpers (and hardly a model of prudence himself, as evinced by his nickname), wrote a scathing post on Facebook that, among other things, called for the BFL to be split into two separate lists: one for wingsuiting and one for regular BASE jumping.

“I called out wingsuiting as not being BASE jumping anymore,” he says. “I wanted the death list split. I also just asked everyone who wingsuits to just go die, so we can get it over with. Obviously, it was a horrible thing to say.”

Some context: Lewis wrote the post in the wake of a friend’s death. John Van Horne, an extremely experienced wingsuiter, had just crashed in the Alps at the end of June. “JVH,” as he was called, was “one of my best friends and last idols,” says Lewis. “He died with his family there.”

Lewis was upset, angry, and just sick and tired of waking up to learn about yet another friend’s death. Most dismissed Lewis as a pot calling the kettle black. Yet in some way, what he wrote in July was prescient. Summer’s big death wave was just about to begin.

“The post got deleted by Facebook, and I pissed off the entire community,” says Lewis. “My message was followed up by the most fatal month in BASE. I just painfully sat back and watched my friends die one after the other.”

Matt Gerdes is the chief test pilot and co-designer at Squirrel, a U.S.-based wingsuit manufacturer. He’s logged about 1,200 safe BASE jumps to date, many of them being wingsuit flights from alpine walls. On August 14, he wrote a Facebook post that stated:

“There are a lot of people saying a lot of things about wingsuit BASE deaths. There is no single factor, and there is truth in every statement about ego, video, complacency, access, summer vacations, etc. But if we were to work on just one thing, it would be education … The simple truth is that wingsuit BASE jumpers don’t know what they are getting into, don’t know how to practice the sport safely, and don’t even know enough to know how little they know. “

Of Wings and Men

BASE jumping, without the wingsuit qualifier, is simply the act of parachuting from fixed objects, as opposed to planes. Wingsuit BASE jumping takes the act of BASE jumping to an entirely different level, one in which the stakes are as high, and the margins are as thin, as the rewards are unforgettable.

And beautiful. And primal. And extremely addictive.

In 2009, Dean Potter set the world record for wingsuit BASE jumping off the Eiger in Switzerland. Potter flew over 9,000 vertical feet and 3.4 miles forward in horizontal distance. He died, along with Graham Hunt, while wingsuit BASE jumping in Yosemite in May 2015.


We’re talking about real, human-powered flight—or, at least as close as humanity has ever come to it. It’s the stuff of ancient myths and archetypal dreams. It’s also the focus of the many viral YouTube videos, which feature costume-wearing superhumans carving through the air, through trees, over rivers and glaciers and picturesque farmlands.

Skydiving and BASE jumping are, essentially, just free-falling. Wingsuit flying—especially when it’s performed close to terrain such as cliffs and trees, in a style known as “proximity flying”—simulates what it might feel like to be a bird carving through the air in a long, fast swoop.

A wingsuit is a baffled full-body nylon costume that resembles the stretched membrane of a flying squirrel. The “pilot” of the wingsuit leaps off a cliff and within a split second, an intake fills the suit’s baffled chambers with air, turning them rigid. By holding a proper body position, the wingsuit pilot is able to glide forward at a ratio of 3:1, meaning that he is hurtling forward three feet for every foot of descent.

Subtle shifts in the shoulder and arm positions allow the wingsuit pilot to steer; his speeds, meanwhile, are peaking at over a hundred miles an hour. Compared to the initial, crude wingsuit designs circa 2000, today’s suits, when worn by skilled and experienced pilots, yield mind-blowing accuracy. Wingsuiters are striking apple-size targets from a mile away. And they’re flying mere inches above, around, and sometimes even through terrain that’s barely wider than their own outstretched arms.

If wingsuits are to blame for the growing death toll, as Webb speculates, the numbers certainly support this thesis. In 2016, 20 of the 31 BASE deaths involved a wingsuit. If you add in deaths that involved a tracking suit, which you may think of as an entry-level training wingsuit, that number jumps to 27 out of 31.

Throughout the 1980s and 1990s—prior to the advent of wingsuit BASE jumping in 1999—the number of BASE jumping fatalities remained fairly consistent and never got higher than five per year.

In September 2002, the Russian jumper Krill Kiselev became the first recorded wingsuit BASE death.

The next one took place two weeks later.

What’s more worrying about wingsuit BASE fatalities, though, is that there appears to be no consistent bias toward experience. For every beginner who rushes out to buy a wingsuit before he—and wingsuit-BASE deaths are overwhelmingly male, perhaps a reflection of the sport’s demographics—has acquired the skill and experience, there also seems to be an alarming number of highly experienced, veteran wingsuiters crashing.

For example, August saw the deaths of Alexander Polli and Uli Emanuele, two of the more experienced and current wingsuit pilots on Earth. (“Currency,” a term borrowed from the aeronautical world, reflects a pilot’s log of recent airtime.)

Polli and Emanuele’s deaths join a disturbing tradition in which the sport loses, every year, at least a few of its biggest, best, and most current stars —perhaps no longer even to the shock of the community. Other big names that come to mind are Shane McConkey (2009), Mario Richard (2013), Sean Leary (2014), Dean Potter and Graham Hunt (2015), Jhonathan Florez (2015), Chris Labounty (2016), and John Van Horne (2016).

Polli, 31, died when he crashed into a tree near Chamonix, France, on August 22. He was widely beloved in the wingsuit community for his charismatic personality, and deeply revered for his skill. In 2012, he became the first wingsuit BASE jumper to strike a wingsuit target—the target was made of foam and around 10 feet tall. He was also known for his extreme proximity flights. One of his most memorable stunts included flying through the “Batman Cave” in Montserrat, Spain.

Five days earlier, the GoPro star Uli Emanuele, 29, died when he crashed in the Dolomites of his native country, Italy.

It’s like stealing the keys to your dad’s Corvette. They think getting the keys was the hard part. Dude, the hard part has only just begun.

Richard WebbPilot and BASE jumper

In June, Emanuele released a video of himself flying through a burning ring of fire. The stunt was broadcast on GoPro’s YouTube channel, and demonstrated Emanuele’s precision and mastery. In 2015, he demonstrated his aerial accuracy in arguably even more spectacular fashion, when he bulls-eyed a six-and-a-half-foot opening in a freestanding rock formation. Flying inches to the left or right would have easily resulted in his death. That video, titled “Wingsuit Flight Through 2 Meter Cave,” now has 7.5 million views.

What’s little known about this particular stunt was that Emanuele actually flew through the hole not just once, but at least four times simply to amass enough footage for the video edit.

“I did it because I was able to do it,” he stated then. “I don't use luck in jumping. We need enough luck in our normal life.”

The question facing the BASE community now is whether luck could’ve helped guys like Polli and Emanuele, and hundreds of others before them. And if not luck, then what, exactly?

Smarter training? More experience? More education? Increased regulation?

Or, perhaps, is the lure of death just the inevitable outcome of this addictive, dangerous, and still nascent sport? If so, why are so many people flocking to it?

Perhaps the most confounding question, though, is why so many people seem to believe that the unthinkable won’t happen to them.

History is stacked with tales of humankind’s myriad experiments with attaching ersatz wings made of feathers or cloth to our arms in an attempt to fly.

From as early as I can remember, I had flying dreams. Once, at age 10, I cut up a cardboard box and fashioned wings, which I then strapped to my arms with pipe cleaners. I took my creation to a porch on the second story of my house in Dobbs Ferry, New York. As I stood atop the railing, cardboard wings lashed to my arms, I recall my seven-year-old sister standing in the grass beneath, pleading and crying at the sight of what I was about to do. I vividly recall a sense of confidence that my design was foolproof; I was certain that when I jumped, flight would ensue, just as in my dreams.

I jumped. I landed so hard that my knee went through my jaw and knocked me out. I haven’t forgotten that vivid (painful) memory, nor have I forgotten those early flying dreams, nor have I ever stopped longing for them to one day return.

This stuff is deep within us all.

In February 1912, a tailor named Franz Reichelt stitched himself a burlap wingsuit/parachute hybrid costume and jumped from the top of the 984-foot Eiffel tower. He crashed so hard that he put a hole into the frozen turf below.

From the 1930s to the 1960s, crowds gathered at local air shows around the world to watch stuntmen called “batmen,” who performed skydiving maneuvers in their crude wingsuits. During this era, 72 of the 75 batmen died.

The dream of raw human flight—not the kind that’s confined to a vessel—has been thousands of years in the making. And until 1999, virtually all of humankind’s attempts to achieve that dream have failed.

In 1999, Croatian skydiver and BASE jumper Robert Pecnik and Finnish BASE jumper Jari Kuosma essentially invented today’s iteration of wingsuit BASE jumping.

Taking inspiration from a jumpsuit worn by the late Patrick de Gayardon, a famous French skydiver who perished in 1998 when the rigging of his prototype “ram-air” wingsuit failed during a skydive, Pecnik designed a new suit that he dubbed, simply, The Original. Pecnik and Kuosma tested their creation on May 10, 1999, an experience Pecnik describes as “very exciting.”

The two launched a company called BirdMan, and The Original became the first commercial wingsuit available to the skydiving public.

Kuosma created an instructor program that offered wingsuit education to experienced skydivers with more than 200 jumps. At first, most skydivers thought wingsuits were really just death suits, but The Original soon proved otherwise.

In comparison to wingsuit BASE jumping, flying a wingsuit in a skydiving scenario has virtually zero fatalities.

Wingsuiters get into their gear.


Flying a wingsuit out of a plane at 20,000 feet offers little perspective. The sensation of speed and flight are only mildly augmented at that altitude. If the dream was the true and raw perception of flight, then somehow wingsuits needed to be flown closer to objects.

By 2003, wingsuit technology had advanced enough to encourage early practitioners to dare flying closer and closer to the cliffs from which they leapt. Perhaps the first man to achieve a true proximity flight was a 23-year-old French jumper named Loïc Jean-Albert, who jumped out of a helicopter hovering over a snow-covered mountain in Verbier, Switzerland, and took a flight path that saw him paralleling the mountain’s slope, never more than 20 feet above the surface. The stunt instantly rocketed him to fame in the skydiving and BASE jumping worlds.

“When wingsuit BASE was introduced in early 1999, this was a new discipline,” explains Pecnik. “People had a lot of respect for those of us who tried first and started to explore the possibility of such flying.”

Today, Pecnik is manufacturing wingsuits for his new company, Phoenix-Fly. He describes how BASE jumping was, at one point, much more under the radar. Finding mentors was more challenging, and most BASE jumpers who got into the sport had a vast skydiving background.

“In the last five to six years, that’s all changed,” he says.

A Shortcut to Death

“When I started BASE jumping wingsuits about nine years ago, there weren’t as many deaths,” wrote Steph Davis, a professional rock climber who lives in Moab, Utah, and is one of the top female wingsuit BASE jumpers in the world. “There has been an explosion of popularity of BASE jumping and wingsuit flying in the last few years. So there are more people doing it, and there are more people getting into it very quickly without a lot of experience. That accounts for a lot of the accidents.”

Pecnik agrees. “Wingsuiting actually looks very easy in videos,” he says. “People are now getting into skydiving only as a way to reach their ultimate goal: wingsuit BASE. Now we have a population who barely knows how to fall or how to fly a canopy [parachute], but they are already skimming rocks in wingsuits.”

Perhaps one example of this approach, according to Pecnik, is Armin Schmieder—the guy who inadvertently broadcast his death on Facebook.

“This guy had barely over 200 skydives, and just a hundred jumps in BASE,” says Pecnik. “He knew so little, and was pushing so hard for his level that it’s no surprise that he’s dead.”

“Gear is stupidly easy to get,” adds Rich Webb. “People are educated enough to know how to say the right things to get someone to sell them the gear. After that, it’s a ticket to go wild. It’s like stealing the keys to your dad’s Corvette. They think getting the keys was the hard part. Dude, the hard part has only just begun.”

Webb says that he sees a growing community of wingsuit BASE novices who are shortcutting every step of the process, which begins by becoming an expert skydiver, then becoming an expert BASE jumper, then going back to a skydiving scenario and training to become an expert wingsuit flyer within that relatively safer environment. It’s a process that might involve tens of thousands of dollars in airplane time, hundreds of jumps at each stage, and ultimately many years of consistent, full-time practice.

“There's a lack of education, a lack of awareness of what they're getting into, a lack of how real the consequences are,” says Webb. “I’m really getting tired and disgusted and annoyed at people dying the same way over and over again.”

Most beginners who die appear to be making variations of the exact same error, according to Webb. “They jump off a cliff, get flying, and for some reason there's just this human reaction to try to hug the air like a big, gigantic beach ball,” he explains. “By hugging air you feel as if you're creating or catching more lift than you actually are. What ends up happening is your suit can only grab so much air, and it starts to stall. When it starts to stall, it loses lift, starts to drag, and then, splat.

“It would be like a skier showing up on a big mountain and trying to snow plow down a big line. Beginners tend to fly slow, and slow speed and flying is just a bad combination.”

But if this newbie error is obvious to Webb and to many other experts, what isn’t as obvious is why so many of the very best are going down, too.

“Some people do not really get the fact that they might die,” says Pecnik. “They’re doing really stupid stuff. Then people are screaming, But he was experienced! Well, he was, but not in all aspects of wingsuit BASE. And that even goes for the two most recent guys: Polli and Emanuele.”

Davis observes one trend among beginners and pros. “Seeing accidents with very inexperienced jumpers and very experienced jumpers, one thing we have in common is that the person is pushing it.”

Media and Progression

If it seems like extreme sports are growing in popularity, it’s probably because we see them everywhere—in movies, in online media, and in advertising. It’s not just wingsuit BASE jumping, but extreme big-mountain skiing and snowboarding, free-solo rock climbing, big-wave surfing, and a host of other death-defying sports involving bikes, boards, and planes.

To varying degrees, many adventure sports today teeter at the margin between what’s possible and what’s crazy; between life and death.

Wingsuit skydiving has virtually zero fatalities and is a safer way to learn how to fly a wingsuit. It’s also a process many newbies are shortcutting in their rush to start wingsuit BASE jumping.


The skiing world has been hit hard in recent years, as several of the best free skiers and snowboarders have been lost to avalanches and injury while pushing the limits of their sport. Sebastien Haag, Andrea Zambaldi, JP Auclair, Andreas Fransson, Dave Rosenbarger, and Liz Daley were all considered world-class, conscientious, experienced athletes. To further complicate the issue, many died while working to create “content” in the form of films and Instagram photos for their sponsors.

There is no Super Bowl of wingsuit flying, skiing, or climbing. Athletes are free to define—and push the limits in—their pursuits however they want. This freedom is a double-edged sword. On one hand, it fosters a creative, artistic approach to these amazing objectives. On the other hand, it has resulted in a culture that constantly needs to take bigger risks, make bolder decisions, and ride narrower margins of safety.

For some athletes, the fame, attention, and corporate sponsorships that come from pushing the boundaries help eke out a living—often a modest one. Reaching a certain level of notoriety or significance in most of adventure/extreme sports, however, has gotten exponentially harder, if also more dangerous.

“I think that as soon as you make being an ‘action-sports athlete’ your profession, there’s a huge risk factor,” says Robb Gaffney, a psychiatrist based in Squaw Valley, California, arguably the definitive action-sports capital of the U.S. The Lake Tahoe region, including Squaw, has been hit hard in recent years as many of its local athletes have died pushing the limits.

“I think that the biggest factor for action-sports athletes is the fantasy of their identity as an action-sports athlete. Most of these athletes aren’t making any money. In fact they might not even have health insurance,” Gaffney adds. “And yet they’re still doing something that could potentially kill them.”

“Progression” is a loaded buzz word that you’ll hear every top action-sports star use. Often, progression means gaining recognition by completing an objective that’s so dangerous that no one else would ever be willing to attempt it.

Until someone else does—or dies trying.

What role, if any, sponsors play in encouraging their athletes to take outsize risks, while heavily promoting and benefiting fromthose risks, became an issue in 2014 when the energy bar company Clif Bar declined to renew the contracts of five climbers, including free-soloist Alex Honnold and climber/BASE jumper/highliner Dean Potter.

Clif Bar’s official statement read that it was no longer comfortable “benefitting from the amount of risk certain athletes are taking in areas of the sport where there is no margin for error; where there is no safety net.”

At the opposite end of the spectrum are companies like Red Bull, GoPro, and, most recently, Stride Gum—which backed a skydiving stunt this summer in which Luke Aikens jumped out of a plane without a parachute and landed in a large net.

What message is being sent when GoPro’s “Be a Hero” slogan flashes at the start of an Uli Emanuele wingsuit BASE video? Emanuele’s fearless wingsuit flight through a hole in a rock or a burning ring of fire is an incredibly cool, if not heroic, achievement. But the activity that we celebrate with millions of YouTube views is the same pursuit that cut his life short.

“I don’t believe that GoPro telling you to ‘Be a Hero’ as they show a wingsuit video is any different than Mountain Dew’s ‘Doing the Dew’ campaign back in the 1990s," says Robert Rose, a chief strategy officer at the Content Marketing Institute. “But I do think it’s important to question. Ultimately, it will be the groundswell that will make that judgement. And we currently live in a world where double standards are plentiful. It’s managing both sides of that risk that every brand lives in today.”

Pecnik, for one, agrees. “To blame GoPro, or any company, is a shallow excuse for what is happening,” he says. “Every person must be responsible for his own actions.”

I think we may be using the wrong definition of 'progression' in wingsuit BASE. Perhaps it really means refining the experience, becoming safer, more elegant, and more aware.

Steph DavisBASE jumper and climber

The public honors athletes when they die in accidents, and even calls them heroes, Gaffney says, “but at the same time, what that’s doing is sending a message to all of us that that’s a great path to follow.

“Glamorization of athletes who have passed creates a positive reinforcement for all of us as spectators in the audience to continue to follow in their path,” Gaffney adds. “Whether it’s conscious or not doesn’t even matter. It’s a positive reinforcement mechanism that has dramatic impacts on how we think about things and all the decisions that we make.”

The underlying question, though, is whether these sports, and especially wingsuit BASE jumping, can “progress” without also becoming deadlier. Davis says folks in the wingsuit BASE jumping communities are thinking about this question from entirely the wrong paradigm.

“There is a limit with how close you can fly, especially with having terrain under you,” she says. “At a certain point, the next step is impacting the terrain—this is why I don’t see this type of flying as progression. We’ve had a constant discussion about progression in wingsuit BASE. I think we may be using the wrong definition. Perhaps progression means something very different. Perhaps it means refining the experience, becoming safer, more elegant, and more aware. Perhaps it means sustainability.”

When Heroes Die

Figuring out why the best are dying confounds, saddens, and even irritates just about everyone I’ve spoken to in the wingsuit BASE world.

“It’s really puzzling to me,” says Rich Webb. “I wish I knew, but I think it has to do with complacency.”

A BASE jumper exits from a cliff into the deep valley below.


Webb points out that many of the best are dying on flight lines that might be considered either “easy” for them, or they’re lines that they’ve done before.

“It comes down to the fact that they're so comfortable in a stupidly high-risk environment,” says Webb. “We just don't have the luxury of margins in our sport, so at some point it catches up to you if you're not on it all the time.”

Pecnik agrees. “They’re not dying because they’re pushing the limit. They’re dying because of ignorance and complacency. It’s always easier to screw up on an easy cliff and an easy flight than on the hard one that you’ve trained for. It’s a typical trap in many outdoor sports.”

Andy Lewis speculates, “I would say so many experienced wingsuiters are dying because they are trying to execute jumps with very low margin for error. Eventually when you make a mistake you hope you have room. And now low margins are so standard in the sport. It just kills people.”

Chris McNamara, a big-wall rock climber from California, took up wingsuit BASE jumping in the early 2000s when it was still a new sport. After a few years, he gave it up when he realized he was probably going to die if he continued, a feeling that hit home when Dean Potter died in a 2015 wingsuit accident. “Dean Potter was part of a small group of active wingsuit jumpers who had been jumping more for than a decade. I really thought he had cracked the code on being able to survive wingsuit BASE,” McNamara said.

After years of narrow misses and seeing friends die who were better and more experienced than he, McNamara concluded, “BASE jumping does not get safer with experience.”

The reason so many wingsuit BASE jumpers believe that accidents won’t happen to them is easy to see: that mentality is a prerequisite to taking up the sport. It’s like marriage. No one gets married thinking that it’s going to end, even though the statistics show it might.

To reduce fatalities, the BASE community has volleyed around a number of ideas, from increased regulation to better education. It’s easy to draw parallels from other worlds that might also benefit from these ideas.

Motorcycles, like the standards of wingsuit BASE jumping, have become bigger, faster, and more daring since 2000. In that time, the number of crash fatalities has increased from 2,829 to 4,295 in 2014, the last year for which data is available. The National Highway Transportation Safety Administration estimates that motorcycle helmets reduce the likelihood of a crash fatality by 37 percent, but only 19 states require their use. It’s likely that more states requiring helmets, and more safety education, might help reduce motorcycle deaths. But equally as important is the need for culture to change—and the same principle can be applied to action sports.

“There’s evolution within sports,” says Chris McNamara. “It starts out a little more risky until, collectively, everyone decides that it just won’t be that risky. There was a point in time when you were an idiot if you wore a helmet on the ski slopes. Fast-forward five years, and you’re now an idiot if you don’t wear a helmet. That’s a collective thing that people decide together. It doesn’t happen overnight.”

Matt Gerdes, the owner of Squirrel wingsuits, argued for more education on Facebook. “Right now, in 2016, all of the information and knowledge that we need to fly safely exists. Unfortunately, what the sport also lacks is a desire by its participants to learn. People have to understand that education is a huge factor, and paying for information that may save your life is worth it.”

Recently, in the skiing world, there has been a huge educational outreach to improve backcountry safety and reduce the number of avalanche deaths. Ian McCammon, a researcher at University of Utah, identified six “heuristic traps” that identify common errors made in backcountry situations. Interestingly, a lot of these traps could easily be applied to the wingsuit BASE jumping world.

There’s even evidence that this education has worked. The 2015-2016 ski season in Colorado saw a record low in avalanche fatalities. Only one person died last year.

“That's an amazingly safe winter considering the number of backcountry skiers has doubled at least twice in the last 15 years or so,” says Lou Dawson, a Colorado-based backcountry skiing legend.

Says Matt Hansen, an editor at large for Powder magazine, “One of the things that I think doesn't get enough attention is the fact that many people are making better decisions in the backcountry. Yes, we have a long way to go—there were several people who nearly got smoked week after week here in the Tetons last winter, including a few fatalities—but you'd hope that backcountry users are starting to get better at avoiding avalanches. I do think that awareness is growing, thanks to many different organizations like the Colorado Avalanche Information Center.”

Earlier this year Rich Webb launched a new side project called “Top Gun Base,” a website that aims to bridge a gap in knowledge with regards to how aeronautical concepts might apply to the wingsuit BASE world. “What I’m writing about is baseline stuff that people need to figure out before they even put on their gear,” he says. “I hope it’s going to gather momentum and people are going to start recognizing that there’s something to this idea of education and learning from mistakes.”

Still, Webb remains skeptical much will change—at least in the short term. “I think we're going to see the fatality numbers continuing in the same trend. There’s going to be a lot more wingsuit fatalities every year. Even if the gear gets better, and we regulate the hell out of the sport, there’s still going to be a niche demographic of people who find an even riskier way to push harder.”

Fly or Die—Or, Perhaps, Don’t Fly at All

A BASE jumper does a backflip near a cliff.


“I’m not sure why our culture at large seems to think that people will live forever or that people should live forever, and that it might somehow be possible to erase anything that might inadvertently cause death to happen,” wrote Davis. “Death is part of life.”

Death has especially been a part of Davis’s life. Not only has she lost many friends to BASE jumping, but she lost both her husband, Mario Richard, and ex-husband, Dean Potter, to wingsuit BASE jumping.

“Flying has given me the best things in my life, and it has also taken them away,” she wrote. “Flying has brought me more pain than anything else. It has also saved my life and brought me more happiness than anything else. Everything has to balance. This degree of trade-off is not worth it for everyone, but we are not all the same person.”

McNamara is one of the few wingsuit jumpers who left the sport. It wasn’t easy, though. He says giving up wingsuit BASE jumping was one of the hardest and scariest decisions he’s ever made.

“I don’t want to tell people to stop BASE jumping,” he says. “But if people are on the fence of whether to quit or not, I think they should know there’s another 60 years of adventures out there.”

“I’m now seven years into quitting, and I could not be more happy,” McNamara says. “Nothing releases quite as much adrenaline as a wingsuit BASE jump, but at the same time, there’s so many more activities that are more enduring and filled with a more lasting kind of happiness.”

What makes a good REPL?

$
0
0
  1. What does a good REPL give you?
    1. A smooth transition from manual to automated
    2. A REPL lets you improvise
    3. A REPL lets you write fewer tests, faster
    4. A REPL makes you write accessible code
  2. What makes a good REPL?
  3. What makes a programming language REPL-friendly?
  4. Conclusion

Dear Reader: although this post mentions Clojure as an example, it is not specifically about Clojure; please do not make it part of a language war. If you know other configurations which allow for a productive REPL experience, please describe them in the comments!

Most comparisons I see of Clojure to other programming languages are in terms of its programming language semantics: immutability, homoiconicity, data-orientation, dynamic typing, first-class functions, polymorphism 'à la carte'... All of these are interesting and valuable features, but what actually gets me to choose Clojure for projects is its interactive development story, enabled by the REPL (Read-Eval-Print Loop), which lets you evaluate Clojure expressions in an interactive shell (including expressions which let you modify the state or behaviour of a running program).

If you're not familiar with Clojure, you may be surprised that I describe the REPL as Clojure's most differentiating feature: after all, most industrial programming languages come with REPLs or 'shells' these days (including Python, Ruby, Javascript, PHP, Scala, Haskell, ...). However, I've never managed to reproduced the productive REPL workflow I had in Clojure with those languages; the truth is that not all REPLs are created equal.

In this post, I'll try to describe what a 'good' REPL gives you, then list some technical characteristics which make some REPLs qualify as 'good'. Finally, I'll try to reflect on what programming language features give REPLs the most leverage.

What does a good REPL give you?

The short answer is: by providing a tight feedback loop, and making your programs tangible, a REPL helps you deliver programs with significantly higher productivity and quality. If you're wondering why a tight feedback loop is important for creative activities such as programming, I recommend you watch this talk by Bret Victor.

If you have no idea what REPL-based development looks like, I suggest you watch a few minutes of the following video:

Now, here's the long answer: A good REPL gives you...

A smooth transition from manual to automated

The vast majority of the programs we write essentially automate tasks that humans can do themselves. Ideally, to automate a complex task, we should be able to break it down into smaller sub-tasks, then gradually automate each of the subtasks until reaching a fully-automated solution. If you were to build a sophisticated machine like a computer from scratch, you would want to make sure you understand how the individual components work before putting them together, right? Unfortunately, this is not what we get with the typical write/(compile)/run/watch-stdout workflow, in which we essentially put all the pieces together blindly and pray it works the first time we hit 'run'. The story is different with a REPL: you will have played with each piece of code in isolation before running the whole program, which makes you quite confident that each of the sub-tasks is well implemented.

This is also true in the other direction: when a fully-automated program breaks, in order to debug it, you will want to re-play some of the sub-tasks manually.

Finally, not all programs need be fully automated - sometimes the middle ground between manual and automated is exactly what you want. For instance, a REPL is a great environment to run ad hoc queries to your database, or perform ad hoc data analysis, while leveraging all of the automated code you have already written for your project - much better than working with database clients, especially when you need to query several data stores or reproduce advanced business logic to access the data.

How's life without a REPL? Here's a list of things that we do to cope with these issues when we don't have a REPL:

  • Experiment with interactive tools such as cURL or database clients, then reproduce what we did in code. Problem: you can't connect these in any way with your existing codebase. These tools are good at experimenting manually, but then you have to code all the way to bridge the gap between making it work with these tools and having it work in your project.
  • Run scripts which call our codebase to print to standard output our files. Problem: you need to know exactly what to output before writing the script; you can't hold on to program state and improvise from there, as we'll discuss in the next section.
  • Use unit tests (possibly with auto-reloading), which have a number of limitations in this regard, as we'll see later in this post.

A REPL lets you improvise

Software programming is primarily and exploratory activity. If we had a precise idea of how our programs should work before writing them, we'd be using code, not writing it.

Therefore, we should be able to write our programs incrementally, one expression at a time, figuring out what to do next at each step, walking the machine through our current thinking. This is simply not what the compile/run-the-whole-thing/look-at-the-logs workflow gives you.

In particular, one situation where this ability is critical is fixing bugs in an emergency. When you have to reproduce the problem, isolate the cause, simulate the fix and finally apply it, a REPL is often the difference between minutes and hours.

Fun fact: maybe the most spectacular occurrence of this situation was the fixing of a bug of the Deep Space 1 probe in 1999, which fortunately happened to run a Common Lisp REPL while drifting off course several light-minutes away from Earth.

A REPL lets you write fewer tests, faster

Automated tests are very useful for expressing what your code is supposed to do, and giving you confidence that it works and keeps working correctly.

However, when I see some TDD codebases, it seems to me that a lot of unit tests are mostly here to make the code more tangible while developing, which is the same value proposition as using a REPL. However, using unit tests for this purpose comes with its lot of issues:

  1. Having too many unit tests makes your codebase harder to evolve. You ideally want to have as few tests as possible capture as many properties of your domain as possible.
  2. Tests can only ever answer close-ended questions: "does this work?", but not "how does this work?", "what does this look like?" etc.
  3. Tests typically won't run in real-world conditions: they'll use simple, artificial data and mocks of services such as databases or API clients. As a result, they don't typically help you understand a problem that only happens on real-life data, nor do they give you confidence that the real-life implementations of the services they emulate do work.

So it seems to me a lot of unit tests get written for lack of a better solution for interactivity, even though they don't really pull their weight as unit tests. When you have a REPL, you can make the choice to only write the tests that matter.

What's more, the REPL helps you write these tests. Once you have explored from the REPL, you can just copy and paste some of the REPL history to get both example data and expected output. You can even use the REPL to assist you in writing the fixture data for your tests by generating it programmatically (everyone who has written comprehensive fixture datasets by hand knows how tedious this can get). Finally, when writing the tests require implementing some non-trivial logic (as is the case when doing Property-Based Testing), the productivity benefits of the REPL for writing code applies to writing tests as well.

Again, do not take from this that a REPL is a replacements for tests. Please do write tests, and let the REPL help you write the right tests effectively.

A REPL makes you write accessible code

A REPL-based workflow encourages you to write programs which manipulate values that are easy to fabricate. If you need to set up a complex graph of objects before you can make a single method call, you won't be very inclined to use the REPL.

As a result, you'll tend to write accessible code - with few dependencies, little environmental coupling, high modularity, and tangible inputs and outputs. This is likely to make your code more clear, easy to test, and easy to debug.

To be clear, this is an additional constraint on your code (it requires some upfront thinking to make your code REPL-friendly, just likes it requires some upfront thinking to make your code easy to test) - but I believe it's a very beneficial constraint. When my car engine breaks, I'm glad I can just lift the hood and access all the parts - and making this possible has certainly put more work on the plate of car designers.

Another way a REPL makes code more accessible is that it makes it easier to learn, by providing a rich playground for beginners to experiment. This applies to both learning languages and onboarding existing projects.

What makes a good REPL?

As I said above, not all REPLs give you the same power. Having experimented with REPLs in various configurations of language and tooling, this is the list of the main things I believe a REPL should enable you to do to give you the most leverage:

  1. Defining new behaviour / modify existing behaviour. For instance, in a procedural language, this means defining new functions, and modify the implementation of existing functions.
  2. Saving state in-memory. If you can't hold on to the data you manipulate, you will waster a ton of effort re-obtaining it - it's like doing your paperwork without a desk.
  3. Outputting values which can easily be translated to code. This means that the textual representation the REPL outputs is suitable for being embedded in code.
  4. Giving you access to your whole project code. You should be able to call any piece of code written in your project of its dependencies. As an execution platform, the REPL should reproduce the conditions of running code in production as much as possible.
  5. Putting you in the shoes of your code. Given any piece of code in one of your project files, the REPL should let you put yourself in the same 'context' as that piece of code - e.g write some new code as if it was in the same line of the same source file, with the same lexical scope, runtime environment, etc. (in Clojure, this is provided by the (in-ns ...) - 'in namespace' - function).
  6. Interacting with a running program. For instance, if you're developing a web server, you want to be able to both run the webserver and interact with it from the REPL at the same time, e.g changing the implementation of a route and seing the change in your web browser, or sending a request from your web browser and intercepting it in your REPL. This implies some form of concurrency support, as the program state needs to be accessed by at least 2 independent logical processes (machine events and REPL interactions).
  7. Synchronizing REPL state with source code files. This means, for instance, 'loading' a source code file in the REPL, and then seeing all behaviour and state it defines effected in the REPL.
  8. Being editor-friendly. That is, exposing a communication interface which can be leveraged programmatically by an editor Desirable features include syntax highlighting, pretty-printing, code completion, sending code from editor buffers to the REPL, pasting editor output to editor buffers, and offering data visualization tools. (To be fair, this depends at least as much on the tooling around the REPL than on the REPL itself)

What makes a programming language REPL-friendly?

I said earlier that Clojure's semantics were less valuable to me than its REPL; however, these two issues are not completely separate. Some languages, because their semantics, are more or less compatible with REPL-based development. Here is my attempt at listing the main programming language features which make a proficient REPL workflow possible:

  1. Data literals. That is, the values manipulated in the programs have a textual representation which is both readable for humans and executable as code. The most famous form of data literals is the JavaScript object Notation (JSON). Ideally, the programming language should make it idiomatic to write programs in which most of the values can be represented by data literals.
  2. Immutability. When programming in a REPL, you're both holding on to evaluation results and viewing them in a serialized form (text in the output); what's more, since most of the work you're doing is experimental, you want to be able confine the effects of evaluating code (most of the time, to no other effect than showing the result and saving it in memory). This means you'll tend to program with values, not side-effects. As such, programming languages which make it practical to program with immutable data structures are more REPL-friendly.
  3. Top-level definitions. Working at the REPL consists of (re-)defining data and behaviour globally. Some languages provide limited support for this (especially some class-based languages); sometimes they ship with REPLs that 'patch' some additional features to the language for this sole purpose, but in practice this results in an impedance mismatch between the REPL and an existing codebase - you should really be able to seamlessly transfer code from one to the other. More generally, the language should have semantics for re-defining code while the program is running - interactivity should not be an afterthought in language design!
  4. Expressive power. You may think it's a bit silly to mention this one, but it's not a given. For the levels of sophistication we are aiming for, we need our languages to have clear and concise syntax which can express powerful abstractions that we know how to run efficiently, and there is no level of interactivity that can make up for those needs. This is why we don't write most of our programs as Bash scripts.

Conclusion

If you've ever played live music on stage without being able to hear your own instrument, then you have a good idea of how I feel when I program without a REPL - powerless and unconfident.

We like to discuss the merits of programming languages and libraries in terms of the abstractions they provide - yet we have to acknowledge that tooling plays an equally significant role. Most of us have experienced it with advanced editors, debuggers, and version control to name a few, but very few of us have had the chance to experience it with full-featured REPLs. Hopefully this blog post will contribute to righting that wrong :).

The Most Officialest SkiFree Home Page

$
0
0

SkiFree screenshot
Microsoft SkiFree 1.0

History

In 1991 I was working at Microsoft as a programmer, writing programming utilities for use by other programmers, such as a dialog editor used in the development of Word and Excel. I programmed mostly in C for OS/2 (back then that was a Microsoft product, and supposedly the wave of the future). Deciding it was time to learn Windows programming (Windows 3.0 had just come out) I jumped right in and did a graphical version of my old VAX/VMS skiing game for VT100 terminals.

The VAX version had looked something like this:

                          ^        420m  01:33 
              ^ 
                                           ^ 
                                ^ 
    ^ 
              ^        // 
                    ^ 
         ^ 
                           ^ 
                                      ^ 
                   ^ 
  ^ 
           ^ 
                                    ^ 
                                          ^ 
                      ^ 
Ski for VAX/VMS

The "^" signs are trees, and "//" is the skier turning slightly right. The program was written in Fortran, and used a combination of VT100 escape sequences and ordinary text scrolling to achieve its animation. I made several text-terminal video games like this in college (at the University of Puget Sound in Tacoma, Washington) and they were fairly popular among the computer nerds.

VAX Ski was itself inspired by an Activision game for the Atari 2600 console, which I enjoyed playing in my youth. I remember very little of the Activision game, but I think it looked pretty much like SkiFree.

I wrote SkiFree in C on my home computer, entirely for my own education and entertainment. One day while I was playing with it at work, the program manager for Windows Entertainment Pack happened to look over my shoulder and immediately decided he had to have this game. I called it WinSki, but the Microsoft marketroids hated that and decided, for inscrutible marketroidal reasons, to call it SkiFree. After some token resistance I let them have their way. Since the program was not originally a Microsoft product, Microsoft licensed it from me and paid me some trivial one-time fee (something like 100 shares of MSFT stock, no royalties) for its use.

SkiFree was intended to run on a 386 PC with VGA display. Such computers were not very powerful, nothing like modern PCs that can do 3-D rendering at millions of textured polygons per second.... No, in those days there wasn't even any such thing as a "video accelerator" -- the VGA was just a dumb pixel buffer hanging off the excruciatingly slow ISA bus. This made it pretty challenging to get good performance out of even simple sprite-oriented animation! Windows didn't help matters any by introducing several layers of abstraction between the program and the video hardware.... I discovered that it was worth almost any amount of preprocessing (on the "fast" 386 CPU) to reduce the amount of video I/O (over the slow ISA), so I designed a fairly clever algorithm to combine overlapping objects/erasures and blt minimal regions in each frame. The result was perfectly flicker-free transparent sprite animation at reasonable speed even on very slow computers, such as an old 286/EGA machine I found in the testing lab. Nowadays one would probably just render the sprites back-to-front in a memory buffer and blt the entire window on each frame.

In 1993 I started working on Version 2 of SkiFree, which would have slightly more realistic physics, multi-player, network play, robot opponents, and sounds. I got about half of those things done (split screen/keyboard multiplayer, very crude robots, and sound), but managed to get the physics completely screwed up to the point where it was no longer playable. At about the same time I also lost the original source code and got distracted by other projects, so SkiFree sort of went into permanent stasis at version 1.0.

In April 2005 I found the source code for SkiFree 1.03 and compiled it, so now we have a real 32-bit version that should run on any Windows XP system, even the new 64-bit XP. It also is more CPU-friendly (uses 1% of the CPU instead of 100%) so it won't drain your notebook battery. There are a few other changes from 1.0 -- see if you can spot them!

In October 2005 I fixed a few bugs and released version 1.04. (Some of the bitmap colors were wrong, and ski slope didn't "wrap around" like in 1.0.)


Sights and Sounds

This is the voice of the Abominable Snow Monster.monster
This is what he sounds like at dinner.eating
This is where baby snow monsters come from:monster sex
Here is the crappy Windows icon I made:my icon
Here is the nice icon that some graphic artist at Microsoft made:nice icon


SkiFree on the Web

This Google search actually returns a lot of pages about SkiFree now, and not so many promotional "buy one lift pass get one free" adverts any more. Most of them are warez download sites and "cheat code" lists, but there are a few gems like this SkiFree clone for the TI 92 calculator (!!). There's an amazingly long article on Everything2 and a Wikipedia entry (though some of the more philosophical passages have been deleted). There's even a SkiFree Fan Fiction site!

I've received plenty of fan mail over the years, but this one is my favorite.

I even got a fan message on my answering machine! (Sorry about the lousy sound quality. Here's my best guess at what he's saying. Let me know if I got it wrong.)

On February 10, 2010, fragments of the lost diaries of my cousin SigFried were discovered in a curio shop in Istanbul. SigFried went missing sometime in 1991 while doing field research for SkiFree. We still haven't found SigFried himself, but now we have some further clues about what happened to him....


Download

Get SkiFree here (39409-byte ZIP containing one 118784-byte Windows 32-bit EXE).
This is the latest version (1.04) compiled for 32-bit Windows, which should also work on Wine and 64-bit Windows.
If for some reason that ZIP file doesn't work for you, try downloading the uncompressed 118784-byte EXE file instead.

For historical interest, you could download the original 16-bit SkiFree 1.0 that shipped as part of the Windows Entertainment Pack in 1991. It does work on most 32-bit Windows systems, but not very well. If you can't get it to work in Windows XP, try this: Configure Windows XP to run 16-bit Windows programs. That has fixed the problem for a few people so far. Most likely you want to get the latest 32-bit version instead.


If you feel like blowing some money today, why not buy aT-shirt?


Contact

One of the most complex wristwatches ever made

$
0
0

Vacheron Constantin

In 2016, Vacheron Constantin created the most complicated timepiece ever, the 57260 pocket watch, featuring 57 complications. It caused a stir in the horological industry.

This year, the company has doubled down, introducing its most complicated wristwatch, the Les Cabinotiers Celestia Astronomical Grand Complication 3600, with 23 complications.

One would assume these two developments had to have been connected - symbiotic, even. But the truth is, they were both developed in secret, with neither development team even aware of the other's existence.

The Celestia was made to honour the way the grand watchmaking masters worked, according to Christian Selmoni, Vacheron Constantin's artistic director. "They were developing something that reflected their new ideas, without regard for the time required," he says. "In almost all other watchmaking brands, this kind of spirit doesn't exist - we always tend to reuse something to save time. Developing movements just for one piece is unique. This timepiece was created in the spirit of the historical astronomical clocks with extraordinary levels of precision, and that makes the Celestia very important."

The Celestia, the brainchild of a project head, master watchmaker and conceptor (of whom Vacheron Constantin would not release his name for fear of being poached - he will from here on be referred to as "MW"), is part of a relatively recent programme inside the Les Cabinotiers ("the Artisans") department. Its aim is to develop unique "blue sky" timepieces while undertaking its raison d'etre, bespoke work. MW has had a lifelong interest in astronomy, and astronomical indications are at the Celestia's core.

"Astronomy is a very serious science as is watchmaking. Without astronomy, we wouldn't be able to measure time - it's all dependent," MW explains. "Our ancestors were reading the time with the stars, the Sun and the Moon, and the origins of watchmaking come from astronomy."

According to MW, mathematics was at the very heart of the Celestia's development. "The initial idea is to think about the concept of the watch, then to work out the equations to make it a reality," he explains. "I spent a year conceiving the watch and working on the mathematical equations, then I had to transform the mathematics into gear trains to provide the indications.

The Celestia's rear display has a celestial chart indicating sidereal time

Vacheron Constantin

"For example, if I want to make a wheel turn, I have to calculate the number of teeth on the wheel. All the other wheels depend on this equation, so I need to be able to understand the principles, even though I have the software," he says. "No software can give me the answers. I rely on these equations and the powerful software to find a solution with our input. It's a question of using these tools to find new solutions."

The Celestia's development took five years - three of which were dedicated to conceptualisation, including mathematical work, then it took two years to fabricate the watch itself. Not every company would allow this kind of development, but Vacheron Constantin, as the oldest continuous-watchmaking maison in the world, feels a responsibility to push horology forward.

"We have an obligation to advance watchmaking with our developments," MW says. "For example, we do an astronomical Moon phase that also incorporates a coaxial day/night indicator, based on the solar gear train. For the Moon phase display, a 3D Moon is engraved on a superimposed sapphire crystal disc, which is also linked to the solar gear train. We have a figurative perception of the Moon in the sky, and it is much more precise than most normal Moon phases. We could do something very simple, but that's not what our department's mission is."

The most challenging piece of the Celestia's puzzle was the running equation of time, a display that shows the difference between apparent solar time (true solar time, the actual solar day) and mean solar time (for all intents and purposes, civil time). Usually, the equation of time is displayed using plus or minus relative to normal civil time (this can be ahead by more than 15 minutes or behind by as much as 14 minutes throughout the year).

"The running equation of time in the Celestia is not linked to the perpetual calendar because that would not be precise enough," MW explains. "We developed one gear train that's linked to the solar indications, and the equation of time is driven by this dedicated tropical gear train, with a precision of approximately 150 years. This is much more precise than the perpetual calendar gear train."

The important specs


Price
$1 million
Size
45mm
Movement
Mechanical manual-winding

Equipping this watch with an extra-long power reserve was also important to MW and Vacheron Constantin. The standard power reserve for watches is 48 hours or so, but the company realised this would not be enough for such a complicated timepiece that is very intricate to set correctly should it wind down. As a result, MW incorporated an almost five-metre long mainspring, in six barrels, to obtain a 21-day power reserve in keeping with the spirit of those early astronomical clocks, which needed to be wound every two weeks or so.

The Celestia is the most complicated wristwatch Vacheron Constantin has ever made, but from a distance, one would never know - which was the point from the very beginning. MW wanted to create a wearable watch, so he built the integrated movement from scratch, aiming for a timepiece thin enough to wear every single day.

In fact, making it as thin as possible was so important that when MW thought of a way to save 0.2mm, it required a rethinking of the entire gear train. "This is usually never done - once you have the gear train, the technical department would prefer not to touch it again, but we decided to completely rethink it," he says. "The flexibility to recreate the entire movement ourselves gives us the freedom to do it any way we decide is best."

Though the initial Celestia is a unique piece, and has already been sold for an estimated $1 million (£806,000), the brand plans to use this as a base for future customer orders - as long as each iteration remains unique. And some of the developments within the Celestia could quite possibly find their way into other timepieces in the future. After all, the secrecy is over, so Les Cabinotiers are happy to share it all with the rest of the company.

VOID: The Void (Linux) distribution

$
0
0

Not a fork!

Void Linux is an independent distribution, developed entirely by volunteers.

Unlike trillions of other existing distros, Void is not a modification of an existing distribution. Void's package manager and build system have been written from scratch.

Rolling release

Install once, update daily. Your system will always be up-to-date.

Thanks to our continuous build system, new software is built into binary packages as soon as the changes are pushed to the void-packages repository.

runit

We use runit as the init system and service supervisor.

runit is a simple and effective approach to initialize the system with reliable service supervision. See the usage page for a brief introduction.

LibreSSL

We were the first distribution to switch to LibreSSL by default, replacing OpenSSL.

Due to the Heartbleed fiasco we believe that the OpenBSD project has qualified and pro-active developers to provide a more secure alternative.

xbps

xbps is the native system package manager, written from scratch with a 2-clause BSD license.

xbps allows you to quickly install/update/remove software in your system and features detection of incompatible shared libraries and dependencies while updating or removing packages (among others). See the usage page for a brief introduction.

xbps-src

xbps-src is the xbps package builder, written from scratch with a 2-clause BSD license.

This builds the software in containers through the use of Linux namespaces, providing isolation of processes and bind mounts (among others). No root required!

Additionally xbps-src can build natively or cross compile for the target machine, and supports multiple C libraries (glibc and musl currently).

July 13, 2017

Wiki Cleanup

I don't like spam

We’re going to clean up our wiki from spammers. Please contact us via twitter or the forum if you’re accidentally blocked.

April 14, 2017

Flatpak

Today we merged Flatpak into the repository for all supported architectures and both libc implementations (musl and glibc).

Flatpak provides a simple and user-friendly way to run, update and create self-contained desktop applications.

It is possible to run proprietary, big and bloated software like Skype or Spotify on a lightning fast and sleek musl based Void Linux system.

Flatpak

Teach Yourself Logic 2017: A Study Guide [pdf]

MISSILEMAP

$
0
0
FAQ : MISSILEMAP by Alex Wellerstein

What is MISSILEMAP? Who made it?

MISSILEMAP is an interactive web visualization meant to aid in the understanding of information about the capabilities and consequences of missile launches, in particular nuclear-armed ballistic missiles. It allows for the graphical representation of ranges, great-circle paths, accuracy (Circular Error Probable), blast damage, and probabilities of kill (the chance that a given weapon will put a particular amount of blast damage on a target). It was made to aid in discussions about missile development, since the technical nature of honest-to-god “rocket science” can make it rather impenetrable from the perspective of laymen, yet many of the fundamental questions are key to local understanding of geopolitical questions (e.g., “could North Korea hit my city with their latest missile?”).

It was created by Alex Wellerstein, a historian of science and technology at the College of Arts and Letters at the Stevens Institute of Technology, in Hoboken, New Jersey, USA. The site’s hosting is paid for by the College of Arts and Letters. It is programmed in Javascript, making extensive use of JQuery and the D3.js libraries, as well as the Google Maps Web API. Professor Wellerstein is a historian of nuclear weapons, the creator of the NUKEMAP, the author of the Restricted Data Blog, and developed this application using Cold War-era algorithms that have long since been declassified.

The Single Shot Probability of Kill (SSPK) algorithm for area targets was based on equations and concepts from two publications:

But also contains substantial original programming by Professor Wellerstein (the “Monte Carlo” approach to the area target is essentially original, for better or worse).

The underlying nuclear blast model (which allows for arbitrary height of bursts) is the same one used by NUKEMAP and is derived from Samuel Glasstone and Philip J. Dolan’s The Effects of Nuclear Weapons, 3rd edn. (Washington, DC: Department of Defense and Department of Energy, 1977), via the method described here.

Return to top

What are the limitations of MISSILEMAP?

MISSILEMAP deliberately uses somewhat simplified models for understanding missile accuracy, damage, and so on. When specifying a range, for example, it reads that as essentially “ground range” and does not take into account the full ballistic trajectory. It uses (as the CEP description below explains) the most simplified and basic sort of accuracy considerations. It should not be used by actual rocket scientists; it is meant as a simplified graphical representation that can be used by laymen, but nonetheless it is meant to give considerable flexibility in its operation (so one can “experiment” with how much changing various parameters changes the overall results). As noted below, the choice of values from the “presets” is based on public estimates and thus may not be entirely accurate.

Return to top

Where does the data for the “presets” come from?

Missile range, yield, and accuracy data can be tricky to come by, as these numbers tend to be closely-kept secrets for reasons that are fairly obvious. However there are many places that estimate these numbers, and these estimates have been used for this application. In most cases, missile data comes from the Center for Strategic and International Studies Missile Threat Project, which contains a large database of international missile profiles, including range, yield, and CEP estimates. The CSIS database was chosen primarily because its estimates seemed plausible, and because it contained a lot of useful information in one place; experts can, and do, dispute individual estimates.

For missiles not in their CSIS database, Wikipedia has sometimes been a source of estimates, as have discussions with missile experts. Estimates can, of course, can vary a lot— MISSILEMAP’s values are fairly conservative, but certainly can be disputed. Users are invited to enter in their own values to see how they change matters. Experts who desperately want to dispute existing values in the preset list should contact Alex Wellerstein.

For launch sites, the data is taken from a lot of different places. Some of it is quite speculative! They are provided just as illustrations. Don't plan your strike plans based on the data here...

Return to top

How should I use MISSILEMAP? Is there a tutorial?

What follows in this FAQ is something of a tutorial — in explaining the concepts (particularly CEP and SSPK), it will walk through potential use situations. So continue reading, especially the section on CEP.

Return to top

What do the various icons and rings shown in MISSILEMAP indicate?

There are a lot of possible rings and icons shown on the NUKEMAP screen. When zoomed out fairly far, this is something you might see (this example is a Polaris A-3 missile in a hypothetical attack against the Kremlin). In all of these examples, high contrast mode is enabled, because it allows for easier visualization of these rings even while it can make finding actual locations a little difficult (it is good for screenshots).

Screenshot of the MISSILEMAP icons visible from a zoomed out view, with labels

The launch site is where the missile is being launched from and can be dragged wherever, or positioned by clicking the magnifying glass icon (Magnifying glass icon) next to the “Launch site” text in the top menu. The target aim point shows where the missile is aimed, and can also be dragged or positioned using the magnifying glass icon next to the “Target ” text in the top menu.

The weapon range is a circle that shows the range of the weapon, as indicated by the “Range” input box. The target aim point is constrained to the range. The missile path shows the shortest ground distance between the launch site and the target aim point. (It does not attempt to model the altitude of an actual missile, and does not model all possible paths of maneuverable weapons like cruise missiles.)

It is worth noting that the standard Google Maps projection (Web Mercator) can produce considerable distortion, making weapon ranges (and missile paths) look very “curved” out of alignment. They are, in fact, perfect circles or great circle paths, as indicated by the globe display in the top right corner of the screen. The globe display will move so that the center of the display corresponds with the center of the Google Maps display, and similarly the globe can itself be dragged to change the Google Maps view.

Zooming in on the target aim point, additional “rings” are visible:

Screenshot of the MISSILEMAP icons visible from a zoomed in view, with labels

The target aim point and missile path are the same as before. In blue, one can see three rings indicating different aspects of the Circular Error Probable (CEP), which is set via the CEP input box. The smallest blue ring is the circle described by the CEP (which defines its radius, in meters). The second largest ring is a circle with the radius of CEP × 2. The third largest ring is a circle with the radius of CEP × 3. (See the section on CEP below for the relevance of these circles.)

The blast damage display shows the area that would be subjected to the blast pressure selected from the “Blast damage display” dropdown menu (in this example, it is 600 psi, for a surface detonation). The “ghosted” blast ranges indicated above always are shown, for the 1 psi blast range (breaks windows), 5 psi blast range (destroys houses), and 20 psi blast range (destroys most buildings). (This is so that when looking at high intensity blast damage, one does not lose focus of the fact that there will be other major effects spreading a great deal further.) Note that in the image above, the 5 psi blast range and CEP × 2 are close enough to the same size that the 5 psi boundary is hard to see.

The target area size depicts the size of the target as calculated for an area target Single Shot Probability of Kill (SSPK), and is set in the “Single Shot Probability of Kill calculator” accessible by clicking the magnifying glass icon (Magnifying glass icon) next to the SSPK label in the top window. It is an optional setting that is not enabled by default. Please read the section on SSPK below to understand what this parameter indicates.

What does “Animate CEP” and “shadow” do?

If you click the checkbox next to “Animate CEP” when you have a CEP, yield, and blast pressure from the “Blast damage display” selector, then it will move the blast pressure ring around, “launching” new missiles and landing them according to the accuracy specified. This is meant as a useful way to think about what CEP “really means” — how it would affect actual placement of blast damage.

If you have “shadow” checked, it will retain a ghostly image of each blast ring “hit” in its place. The accumulation of these shows how multiple, repeated launches might land. Unchecking the “shadow” button will clear the accumulated shadows. (If you want to see them without the target reticule, uncheck “Animate CEP” and then move the target reticule.)

Return to top

What is CEP (Circular Error Probable) and why does it matter?

There are many ways to talk about how accurate a projectile can be. In the American context, historically it became common to compress all of these various ways into a single number, the CEP or Circular Error Probable. (For the technically-minded, you can think of this as how a “rocket scientist” conveys information about Gaussian probabilities to a senior military officer or Congressman in a single number.)

For any given point aiming point of a target, the CEP describes the radius of a circle in which there is a 50% chance that the projectile will land. So if our CEP is, say, 1,000 meters (1 km), then we can say that if we aim our missile at a given target (say, the Kremlin, just to pick a Cold War target), there is a 50% chance that the missile will fall within 1,000 meters of it. Based on Gaussian probabilities, we can also say that there is a 43% chance of the target falling between the range of 1 × CEP and 2 × CEP (so between 1 km and 2 km in our example), and a 7% chance of it falling between 2 × CEP and 3 × CEP (between 2 km and 3 km in our example). See the illustration below:

Illustration of the concept of CEP

(The arrows are staggered just so they each stand out from one another — the point to radii of the specified distances from the aiming point.)

In real life, missile accuracy and CEP is not always so perfectly spherical or evenly distributed. Real-life errors can be tilted in one direction or another, or biased towards centers other than the aimed-at target. See this page on CEP for a long discussion of this. MISSILEMAP uses a simplified model for the sake of visualization and understanding (a “Rayleigh” approach, per the terminology on the linked to page, but even more simplified in that it is impossible for any shot to land outside of 3 × CEP); it should not be understood as the end of the discussion.

When CEP is combined with the explosive radius of a weapon, it becomes a very useful number. For example: if you selected the “preset” for a Nazi V-2 rocket, a non-nuclear ballistic missile used during World War II, you can see it has a relatively large CEP (12 km). If you then select the “Blast damage display” of a “1 psi surface burst,” it will show you the approximate expected range of very light blast damage from the weapon’s explosive components, which are only equivalent to about 0.8 tons of TNT (0.0008 kt). You will have to zoom in very far to see the blast ring, because the range of the blast is much smaller than the size of the CEP.

Example showing a 1 psi blast pressure from a V-2 rocket aimed at London.

Position the missile site so that it is within range of London, United Kingdom. (You can do this very easily by selecting one of the WWII V-2 launch sites from the “Launch preset” window, which will locate the missile at an actual launch site from the war, and then clicking the magnifying glass icon (Magnifying glass icon) next to the “Target” line in the top window, and then typing “London” into the lower search box.) You can see that with a 12 km CEP, the V-2 had nearly a 93% chance of destroying some part of greater London (the CEP × 3 radius is outside of it, but the CEP × 2 radius is not), but it couldn’t pick which part. Which is to say, the V-2 missile had about as much explosive as you’d need to destroy part of a city block (which is attested to in photos of its damage). But it lacked accuracy to target any particular city block in particular: the Germans were shooting it “blind” in the direction of London, knowing it would hit something, but they were unable to specify which “something” they were hitting. (In actual historical reality, they were fooled by double-agents into aiming somewhat to the east of London, so their actual hit rate was even lower than it would have been.)

Let’s look at another preset: choose a Titan II launch site from the “Launch preset” menu. The Titan II was an American Cold War thermonuclear-armed intercontinental ballistic missile (ICBM). Which is to say, it was a very big rocket with a very big warhead, and could easily hit sites as far away as the Soviet Union. Position the target onto Moscow (which probably had several Titan II’s aimed at it during the Cold War). Let’s imagine that we are interested in finding out how much heavy damage to houses would be caused, and selected the “5 psi airburst” option from the “Blast damage display” menu. You can see that the 5 psi blast range is several times larger than the CEP, and covers the entirety of the city: if your goal is to put 5 psi of blast pressure onto a target the size of Moscow, you literally cannot miss with a Titan II. (If you want to confirm this, click “Animate CEP” and watch how successive missiles “fired” at Moscow always hit it, even if their exact ground zeroes move by a little bit.)

"Shadowed" blast rings from repeated "launched" of a Titan II missile on Moscow, showing areas hit by 5 psi of blast pressure.

If you change the blast pressure range to 1000 psi, you can see that the area covered is not only smaller, but there are significant distances between some of the shots (e.g. those at the very top or bottom of the shadows), even though they cluster around the aiming point:

"Shadowed" blast rings from repeated "launched" of a Titan II missile on Moscow, showing areas hit by 100 psi of blast pressure.

One last example. Let’s select a Soviet R-7 missile site, their first generation ICBM. Set the Pentagon (just outside Washington, DC) as its target, and zoom in. Let’s imagine that we are a Soviet war planner, and we want to see if the R-7 could destroy a deep, hardened bunker underneath center of the Pentagon. Let’s say that it would require 10,000 psi (a very high number!) to destroy this bunker (so set the “Blast damage display” to 10,000 psi). We can see, if we zoom in very close, that the 10,000 psi blast radius of the R-7’s 5 megaton warhead would certainly destroy the Pentagon complete — if it hit it. But that particular blast radius is very small compared to the size of the CEP. If we click “Animate CEP,” we can see that the odds of actually putting that much blast pressure on the Pentagon is very low — so low, in fact, that it’s unlikely that this missile gives us the capability to do that. As seen below, with the “shadows” turned on we can see that the 10,000 psi blast pressure range, while centered on the Pentagon, also misses a huge proportion of the time.

Blast shadows from a 10,000 psi surface burst, 5 megaton warhead, aimed at the Pentagon

How low, exactly? See the section on SSPK below to see how to calculate that probability. On the other hand, if our goal is to put 5 psi of blast damage on Washington, DC, itself, then it can’t miss. This illustrates very nicely how accuracy and blast damage work together to indicate what kinds of uses a missile can be put to: the R-7 would be a useful weapon for killing cities, but not for killing bunkers.

Is this of more than academic interest? Yes. If we believe, for example, that North Korea has a missile capable of hitting Los Angeles, but that it has a low yield (e.g., 30 kt) and large CEP (e.g., 2 km, which is close to what early Chinese ICBMs were), then we can use MISSILEMAP to ask the question: What could the North Koreans actually do with that missile? The answer is fairly clear: they could take out a chunk of Los Angeles, but not specify which chunk — the accuracy would be too low to be very specific, given that blast range.

Map showing several "launches" of a hypothetical North Korean nuclear weapon agains Los Angeles, with the 5 psi blast radius "shadows"

We can in turn easily ask other questions: if we imagined North Korea wanted to destroy a specific military base, such as the Naval Base at San Diego, what CEP would they need to put 20 psi onto that target? Conversely, how would increasing the yield (without increasing their CEP) change that possibility? MISSILEMAP thus gives some useful insights into key policy questions, even if the relevant variables (like the CEP and warhead size) are unknown.

Return to top

What is SSPK (Single Shot Probability of Kill) and why does it matter?

The MISSILEMAP allows for the calculation of a number called the Single Shot Probability of Kill, or SSPK. Roughly speaking, this is the odds that any single missile launch will put an appropriate amount of blast pressure onto a given target to destroy it. There are a variety of ways to calculate SSPK. MISSILEMAP uses the following variables: the CEP (missile accuracy), the weapon yield, the blast damage required to destroy the target (which is input through the “Blast damage display” menu), the reliability of the weapon system (default is 100%, but real-life reliabilities can vary a lot, because a lot can go wrong with a missile), and the size of the target (by default it is a target with a diameter of 0 meters, which is to say, a “point target,” but this can be changed, as will be described).

For point targets with 100% reliability, the SSPK is defined as:

SSPK = 1 − 0.5(RL/CEP)2

Where RL is the “lethal radius” (that is, the range of the blast damage necessary to destroy the target) and the CEP is the Circular Error Probable. But we don’t have to do this math ourselves; MISSILEMAP will calculate it from the settings we choose.

So let’s pick a weapon — let’s choose a Chinese DF-5 from the “Missile preset” menu, and position it so that the Pentagon is the target. (No offense meant to the Pentagon — it’s just a prominent target of a useful size, and easy to see in Google Maps.) Let’s say we want to just leave a crater where it used to be, and so let’s choose 600 psi from the “Blast damage display” menu (that is probably overkill, but we want to make sure it’s really gone, right?).

The SSPK indicator at the top left should be updated now, and say that the chances of a single DF-5 putting on an exact (0 km diameter) point at its aiming point are “only” about 55%. That’s kind of interesting by itself, but we can also do several other operations with that number if we want to. By opening the “Single Shot Probability of Kill calculator” window, by clicking the little icon of knobs next to the SSPK indicator in the top left (Knobs icon), it tells us that we would need to target 4 missiles at the target to have a 95% confidence that the destruction would be total. On the other hand, if we only require 85% confidence (we can change the confidence number from the drop-down menu on that window), we only require 3 bombs.

Screenshot of the SSPK menu for a DF-5 weapon

What if you don’t want to assume that missiles are 100% reliable, or that your target is larger than a 0 km point? On the “Single Shot Probability of Kill calculator” window, we can set those options.

The weapon reliability percentage is a total system reliability. Real-life examples are hard to come by, but you can play with the number to see how “sensitive” it is; the difference between 60% reliability, 80% reliability, 90% reliability, and 95% reliability makes for interesting comparison (in many cases, adding 5%-10% more reliability does not result in much overall difference, which raises questions about how reliable a system has to be to be a “credible” deterrent, especially given that adding more reliability often comes at great expense).

You can also set the target diameter, which is equivalent to saying, what are the odds that the entire target of a given size will have that blast pressure applied to it? It should encompass about 95% of the target. The Pentagon is about 400 m in diameter. When you set the target diameter option, it will superimpose a green ring onto the target reticule, so you can eyeball the size:

A zoomed-in map showing the Pentagon surrounded by the green ring of a 400 m diameter target area

Once you’ve set a target diameter, the SSPK will be automatically recalculated. When the SSPK is calculated for a point target (e.g., a target of 0 m diameter), it does it analytically from the formula indicated above. This is quick and easy. For area targets, the analytical solutions are not as easy to do, so, taking advantage of the fact that modern computers (and web browsers) are pretty speedy, the SSPK is calculated via a numerical (or “Monte Carlo”) method: it “simulates” a large number of missile launches and checks how many of them either entirely encompass the target (total destruction) or partially overlap the target (partial destruction). These two numbers are then displayed as part of the SSPK readout in the top window, and also on the Single Shot Probability of Kill calculator window. Because this is a simulation based on a finite number of “launches,” depending on how many launches are simulated, the amount of precision can vary, particularly on slower computers/browsers (if the SSPK function detects it is taking too long, it will “downgrade” the number of launches to a number that it detects will run quickly; for more information, see here).

So with the same Chinese DF-5 missile as before, with the Pentagon as our target, and a target radius of 400 m, and a blast destruction requirement of 600 psi, we can see that with a 100% reliable weapon our chances of destroying the Pentagon with a single missile is 41%, with a 23% chance of partially destroying it. Our required missiles for 95% confidence in destruction thus rises to 6 missiles.

You can see, perhaps, how calculations like this can lead to war planners targeting what seem like hugely redundant missiles at a single target: it depends on how much confidence is desired. If a Chinese war planner only required 50% confidence in destruction of a target to feel “safe,” they would only require 2 missiles aimed at the target. (The confidence calculation only takes into account total destruction, not partial; summing up partial destructions would require keeping track of how “destroyed” a target is through the simulations, and the SSPK algorithm does not try to do that, because the math gets sticky.)

Return to top

How does the precision of the SSPK calculation work?

As noted in the section above, the SSPK calculation for area targets (not point targets) is numerical, which is to say, it is derived from a statistical “simulation” of many launches. The default maximum number of launches run is 100,000, a fairly large number.

If, while running, the SSPK function detects that over 100 milliseconds (1/10th of a second) has passed, it will immediately stop the function and set the maximum number of launches to the number it had gotten to up to that point (which is to say, it is aiming for the function to take no more than 1/10th of a second to run). If you view your browser’s Javascript console, it will give a message to this effect if it does this downgrading. If the number of launches is between 10,000 and 5,000, it will give a warning message that indicates that it is low enough that there may be 1-3% variance between subsequent runs. If it falls beneath 5,000, it will warn that the precision is very low indeed.

If you are the kind of person who is interested in running an arbitrary number of simulations, or turning off the dynamic behavior of the SSPK precision, you can do this through your browser’s Javascript console. To disable dynamic SSPK precision adjustment, set sspk_dynamic = false. To set the number of launches arbitrarily, set sspk_max = n, where n is the number of iterations to run. To re-run the SSPK calculation, type updateSSPK() into the console and press enter (it should return “true” if it updated).

Return to top

Can I use screenshots of this website, or embed it within my own site?

This website was designed to be used and shared widely. Please feel free to link to it, post screenshots of it (though note that Google Maps has its own screenshot guidelines), quote and cite it, etc. If possible, please credit it or its information as “MISSILEMAP by Alex Wellerstein,” and provide a link to the website (either as a hyperlink or verbose link, depending on the media). For clarification, you may e-mail Professor Wellerstein (wellerstein@gmail.com), but he gets a lot of e-mails and is pretty busy and may be slow to respond or may not respond (and if he does, he will essentially say what is written here).

If you are interested in having a working copy of MISSILEMAP on your own website, the best and easiest way to do this is to embed it as an IFRAME rather than trying to host the code yourself (there are many pieces of code involved, and they are occasionally updated, and the Google Maps API will not work correctly on another server without some code modification). You can use the permalink URL structure indicated below to link to specific settings or even to modify the appearance (within some limits) of the MISSILEMAP. You are welcome to embed it as such, though Professor Wellerstein reserves the right to modify his code at any time, including in ways that would prevent websites he judges to be malicious or harmful from embedding it.

Return to top

What is the permalink URL structure?

When you click on the “Permalink to settings” link, it will create a stable, permanent link to the current set of settings, which you can use to save work, share settings with others, or embed settings into a webpage as an IFRAME source. There are considerable permalink options that are supported but are not obvious, the sort of thing that might be of interest to advanced users (i.e., missile wonks) or people desiring to embed MISSILEMAP into another web page. What follows below is a set of URL variables that can be passed to the MISSILEMAP. Note that there are probably combinations of manually-set variables that will cause the MISSILEMAP to function in ways that you don’t desire — change variables with care. At least one variable (indicated below) could lock up your computer if set in a haphazard way, depending on the speed of your browser — change it with care if you change it at all.

To make a permalink, one or more of the following variables must follow the main MISSILEMAP url (http://nuclearsecrecy.com/missilemap/) which itself must be initially followed by a question mark (http://nuclearsecrecy.com/missilemap/?). Each variable must be preceded by an ampersand (&) and followed by an equal sign, plus a value (e.g., &c=800 would set variable c to 800). What follows is a table of variables, possible values (including an example value in brackets), and a description.

Note that each setting is processed sequentially; if you set all of the variables required for SSPK calculation (CEP, Blast Damage Display, Yield), the SSPK calculation will start to run automatically based on that data, and so if you have variables that affect that calculation (e.g. Target Diameter) they may not be taken into account unless they are passed earlier on in the sequence. You may have to experiment if constructing a permalink manually.

In addition to the above, passing “hide” as a parameter, followed by one of the following text codes (which should be separated by commas), will hide various aspects of the interface. There are two things to note here: the MISSILEMAP logo will always be visible, and clicking it will always move the user to the original website; the version number of MISSILEMAP will have an “L” for “Limited” displayed whenever one of the hide parameters are used. Example usage: &hide=globe,top

These changes are entirely aesthetic; the various bits will just be “hidden” with CSS, still technically there, just not visible. Hiding the SSPK will not stop it from being calculated behind the scenes, in other words, and hiding a control does not mean you cannot set its value with another variable. It is imagined that this would allow for easier embedding within a larger website, to show the results of the MISSILEMAP without all of its complexity.

Note that if you hide a container (e.g., col1) you do not need to hide all of the controls within it separately; they are hidden as well automatically.

Below is an example of an embedded version of MISSILEMAP with many settings, with the IFRAME source corresponding to this permalink:

Return to top

Acknowledgments

Thank you to the College of Arts and Letters at the Stevens Institute of Technology for hosting MISSILEMAP and materially enabling its development.

This application has been improved through discussions with Edward Geist, Jeffrey Lewis, and others...

Return to top

The MISSILEMAP FAQ was written by Alex Wellerstein. Last updated August 27, 2017.


6502cloud – Bringing the 80's to the cloud

$
0
0

In the beginning...

Once upon a time, the 6502 CPU ruled the world. Atari 2600, Atari 400, Atari 800, PET, Vic-20, Commodore 64, Apple II and even the NES were all driven by the 6502 processor by MOS. We've since moved on. Alpha, Pentium, PowerPC, ARM, and so on. Now cloud computing has come around. Cloud computing is just "Someone else's computer." Why shouldn't that computer be a 6502 based one? Or to bring an old Slashdot joke, why not a Beowolf cluster of 6502's?

What do we have here?

We have a AWS Lambda function, fronted by an API Gateway. The Lambda runs a 6502 emulated, with memory loaded from the caller. It then starts executing from a specified address, and when it times out, or hits a BRK op-code. It will then return the specified section of memory.

Why?!

Because I can. Because we can. Imagine a Beowolf cluster of 6502's. Distributed 6502 processing. Cloud 6502. 6502 WEBSCALE!

Contact

Should you have any questions, you can email me. Or get me on reddit, u/6502cloud.

The ocean is a strange place after dark

$
0
0

Moonlight triggers the world’s biggest orgy, strange creatures emerge from the depths, and waves glow blue. Some phenomena in the ocean can only be witnessed after dark.

1. Bioluminescence makes the sea shimmer

You may have seen the pictures.

It’s night-time in an impossibly exotic location. Waves are breaking on the beach. The water is sparkling with electric blue lights. 

The internet loves an image of a magical-looking bioluminescent bay. You may also have seen travel bloggers bemoaning the real event as not quite living up the hype.

Even if the latter is true, bioluminescence (in this case usually caused by planktonic organisms called dinoflagellates) is a pretty amazing natural phenomenon.

Dinoflagellates emit blue light when disturbed, which is why they can be seen sparkling over wave crests, around boats or when a hand or paddle runs through them.

These tiny creatures are the most common source of bioluminescence at the ocean’s surface.

So-called bioluminescent bays such as in Puerto Rico and Jamaica are among the best-known places to witness the glow. However, the ephemeral phenomenon can be found throughout the ocean where there are dense gatherings of dinoflagellates.

Sometimes dinoflagellates’ population increases rapidly causing blooms, which by day are coloured a less attractive red-brown, sometimes known as red tides. And some, but not all, of these red tides are poisonous. 

Even stranger and rarer than bioluminescent bays are “milky seas”, where continually glowing water stretches for as far as the eye can see.

Milky seas have only been seen a few hundred times since 1915, mainly concentrated around north-western Indian Ocean and near Java, Indonesia.

They are not caused by dinoflagellates, but are thought to be the result of “bioluminescent bacteria that have accumulated in large numbers near the surface”, explains to Dr Matt Davis, Assistant Professor of Biology, St. Cloud State University in the US, who specialises in bioluminescence.

Reports by sailors over the centuries have described milky seas as a nocturnal whitish glow like a field of snow, but scientists have had little chance to investigate the phenomenon first-hand.

In 2005, researchers analysing archived satellite images found that milky seas could be seen from space and that one satellite had captured images of a huge area of ocean that had displayed the strange glow for three consecutive nights a decade earlier. 

2. Animals glow in the dark

Bioluminescence, the emission of visible light by an organism as the result of a natural chemical reaction, is common among marine life such as fishes, squid and molluscs. In the deep sea most species are bioluminescent, where it is the main source of light.

In shallower waters, most bioluminescent fish display their lights at night.

“Flashlight fishes have a specialized pouch under their eye that they can rotate to expose the light emitted from these bacteria, and they use this glow at night to hunt for food and communicate,” says Dr Matt Davis.

Ponyfish emit light from the bioluminescent bacteria housed in a pouch using transparent muscular shutters, to communicate, he explains.

Camouflage, defence and predation are among the variety of reasons fishes are thought to emit light. 

For example, bobtail squid have an ingenious way of using lights. These nocturnal animals have a mutually beneficial relationship with luminescing bacteria that live in a mantel cavity on its underside. At night the squid control the intensity of this light to match the moonlight, and can reduce their silhouette to camouflage themselves from predators.

3. Moonlight triggers the planet’s biggest orgy

There is nothing more romantic than a moonlit night, especially if you are a coral on the Great Barrier Reef off Australia.

One night a year in spring, the biggest orgy on earth is triggered by lunar light.

Over 130 coral species simultaneously release their eggs and sperm into the water during a window of just 30-60 minutes.

This mass spawning event might be the most extraordinary example of synchronised behaviour in the natural world.

When the gametes – eggs and sperm cells - are released they hover for a moment, forming a ghostly replica of the reef’s shape, before dispersing into an underwater blizzard as the sperm fertilise the eggs.

Dr Oren Levy, a marine biologist and ecologist and Professor of Life Sciences at Bar-Ilan University, Israel, has studied this extraordinary event.

“This is really fascinating phenomena…we know this event is going to happen a few nights after November's full moon each year, three to five [days] post full moon,” he says.

“[It is] always amazing, in particular I am so amazed how each of the coral species year after year spawn at the same hour of the night.”

He adds: ”Once it happens it is always so exciting to see how everything is becoming so live and synchronised. It is almost [a] spiritual event and you understand the power of nature in its best.”

Moonlight triggers the phenomenon by acting as a synchroniser or “alarm” probably with other environmental signals such as sunset timings, water temperature and tides to cue the time of the gamete [egg and sperm cells] release, explains Dr Levy.

He adds that corals seem to possess photoreceptors that detect the phases of the moon, which helps with the “fine tuning” of the gamete release.

4. Sharks and seals rely on celestial light

For some seals, moonlit nights spell danger.

During winter months, the 60,000 cape fur seals on Sea Island in False Bay, South Africa run the gauntlet of being picked off by great white sharks patrolling the seas when they enter and exit the water.

One study in 2016 hypothesised seals swimming at night during a full moon are at more risk of being eaten by a shark since bright moonlight silhouetting them against the surface makes them an easy meal for predators lurking below.

However, most shark attacks on seals happen just after sunrise. Researchers behind the study, which measured shark attacks at dawn, were surprised to find seals were much less likely to be predated at this time of day if there was a full moon. 

The researchers theorised that lunar illumination combined with emerging sunlight may decrease the stealth ability of the sharks and that the advantage switched from sharks to seals as night turned to day.

And seals may rely on another celestial feature to navigate - the stars.

Captive harbour seals (Phoca vitulina) are able to locate a single lodestar and steer by it, researchers have shown.

During a test using a simulated night sky, seals swam towards the brightest star and could orientate themselves when the stars were swivelled around.

In the wild, seals need to navigate the open ocean to find foraging grounds that may be separated by hundreds of kilometres.

Researcher Dr Bjorn Mauck said at the time: "Seals might learn the position of the stars relative to foraging grounds during dawn and dusk when they can see both the stars and landmarks at the coast."

5. Strange animals come to the surface every night

Under the cover of darkness rarely seen creatures migrate to the ocean’s surface to feed.

The Humboldt squid, also known as the jumbo squid, is one of the most eye-catching marine animals you can see lurking in surface waters.
By day the squid lurk in the deep waters of the Eastern Pacific Ocean along the deep shelf that runs off the west coast of the Americas and every night they are one of the many ocean animals to migrate upwards to find dinner.

Vertical (or diel) migration - when ocean animals swim to the surface at dusk and disappear down again at dawn – is extremely common.

“What [Humbioldt squid are] doing largely is following their main food item, which is the so-called lantern fish,” explains Professor Paul Rodhouse, an Emeritus Fellow for the British Antarctic Survey (BAS) and former head of the organisation’s biological sciences division.

In turn, lantern fish follow vertically migrating zooplankton.

Since zooplankton are depended on by so many ocean animals, “the rest of the food chain will be following on after it,” says Prof Rodhouse.

“It is a huge movement of biomass every day,” says Prof Rodhouse. “More than a thousand metres. Some of the oceanic squid probably migrate over 1000m every day.”

He adds that almost all pelagic species (animals that live in the water column not near the bottom or shore) that can swim make the journey.

Humboldt squid are among the most striking creatures to surface every night. Their ability to change colour and flash bright red when agitated has earned them the nickname “red devils”. Although much smaller than their cousin, the 13m-giant squid, they can reach a length of about 1.5m (almost 5ft). Highly aggressive predators, they capture prey with strong tentacles and suckers and tear into it with powerful beaks, and have reportedly occasionally attacked humans.

But even ferocious Humboldts are preyed upon by bigger predators such as billfish, swordfish and sharks.

“Of course what they are all doing [by being active at night] is avoiding predation by the top predators,” says Prof Rodhouse. "The big predators that are visual predators and which stay in the surface waters and see their prey.”

“So they’re all… reducing the risk of being preyed on by going down into deep, dark waters at night.”

For more incredible ocean stories, follow OurBluePlanet on Twitter. Get in touch with to share your most magical ocean moments, and for inspiration watch the launch video on the BBC Earth YouTube Channel.

#OurBluePlanet is a collaboration between BBC Earth and Alucia Productions.

Join the #EarthCapture underwater film and photo challenge by uploading your shots here:

Never miss a moment. Sign-up now for the BBC Earth newsletter.

Uber’s board wants Expedia CEO Dara Khosrowshahi to be its next CEO

$
0
0

The board of Uber has voted and wants Expedia Dara Khosrowshahi to be its next CEO. But here is a shocking twist for those who have had to endure this awful, messy and convoluted process: He has not been officially offered the job as of 15 minutes ago, said sources.

Still, most expect him to take it and he appears to be the one person dueling factions of the board can agree on. Unknown until now, Khosrowshahi was the third candidate — after Hewlett Packard Enterprise CEO Meg Whitman and former General Electric CEO Jeff Immelt.

Khosrowshahi is considered the “truce” choice for the board, which has been riven by ugly infighting between ousted CEO Travis Kalanick and one of its major investors, Benchmark. Benchmark had backed Whitman, while Kalanick had backed Immelt.

Sources said that going into this morning, after Immelt withdrew his name from contention after it was clear he would not win the job, Whitman had the upper hand in the race for the job. But she also wanted a number of things — including less involvement by ousted Uber CEO Travis Kalanick and more board control — that became too problematic for the directors, said sources.

As he left, sources close to Immelt’s thinking called the search process totally “dysfunctional” (and worse). Cue sources close to Whitman to say that very soon (and more). Both are quite accurate.

And, in keeping with the cup full of crazy modus operandi, about 30 minutes ago, in a statement, an Uber board spokesperson said: “The Board has voted and will announce the decision to the employees first.”

Well, it’s Khosrowshahi, if he says yes, that is— so now we can all get back to the season finale of Game of Thrones.

Sources close to Meg Whitman said she has not been informed of any choice nor had the board agreed to some the the things she was asking for to take the job.

More to come, obvi!


Mino Games Is Hiring Programmers in Montreal

$
0
0

Mino Games is a mobile gaming studio that strives to build the best possible games. We produced the hit mobile game Mino Monsters, with over 15 million downloads. Our development studio is in Montreal, and we are building one of the top teams in the industry.

We are funded by the leading angel, institutional investors, and gaming companies from across the world. (Andreessen Horowitz, Y Combinator, Sybo Games).

We are rapidly growing our Montreal studio, and looking for world class talent to come join us.

6 months of working remotely taught me a thing or ten

$
0
0

Six months ago, I started as a Senior Front-End Developer for Helix Education. Most of what I do now is identical to what I did for my past employer—making websites, writing code, collaborating with writers and designers, and all while navigating the choppy waters of higher education.

The only difference is that now I’m working remotely. In fact, I’m not even in the same time zone as most of my coworkers.

I wasn’t sure what to expect going into my first remote job. I did a lot of research before taking the plunge. Ultimately, though, I wasn’t sure if it would be as glamorous as others made it out to be, or if I would find it counterproductive to working, learning, and performing to the best of my abilities.

As it turns out, it’s a mixed bag. A lot of positives… and some negatives. If you’re a remote employee (or considering a remote job), here are my top ten tips that I’ve gleaned in the last six months.

1. Be as available as possible

This is probably the best piece of advice I received from a friend prior to starting my new job. Things move quickly back at the home office, and since you’re not there, there’s a built-in delay for someone wishing to communicate with you.

If it always takes hours for someone to get your ear for just a minute, you’ll build a reputation as someone who’s “never around”—even if only subconsciously—among your coworkers. 

As a remote employee, you have to work harder at being available to avoid the impression that you’re never available.

Strive to build a reputation as someone who is quick to respond. Be eager to hear the latest developments on projects so that you’re never out of sync with everyone else in your office.

Sometimes there are time zones that separate you. I’m two hours ahead of the rest of my team. If it’s not possible to align your working hours with those of your team, make sure there’s a clear expectation for when you will be at your desk, and be there. Use your alone hour(s) to be as productive as you can be so that you can spend time communicating with others later. Speaking of communication…

2. Communicate clearly

You may find a majority of your communication happens over chat, Slack, HipChat, text messages, (GitHub commit messages?) and the like as opposed to face-to-face conversation. With non-verbals and facial expressions removed from the equation, miscommunication is just waiting to happen. 

Until your coworkers get a chance to really know you, avoid sarcasm to prevent misunderstandings. Read what you write before you hit “Send” to make sure it can’t be easily misread in a different mood or tone of voice.

When talking via video conference, exercise brevity and conciseness with your words. The occasional “glitch” in the video can cause words to get cut off and can leave people confused or asking for clarification. Also, smile. It improves your face value.

3. Go out of your way to be human

Spend time getting to know your coworkers—especially if you’re coming into a new job with new people you’ve never met before. Don’t make every conversation just about work. 

Ask your coworkers how they’re doing, what the weather is like, or what the mood in the office is like lately (without being too nosey). You won’t be around the office to hear the banter and water cooler conversations. 

Unless you make a concerted effort to be a part of what’s going on, your coworkers’ birthdays will come and go, people will leave the company, important announcements will be made… and you’ll miss it all.

4. Offer praise and positive sentiments early and often

You will also need to go out of your way to foster a sense of teamwork. You won’t be around to share the little daily “wins” with the rest of the team, but having positive relationships with your coworkers is essential to being able to do your job well. 

Some of your coworkers may not communicate with you very often; in those cases make sure what little they do hear from you is positive and optimistic. This all goes back to the fact that you need to work actively to shape others’ impressions of yourself. And no one like a Negative Nellie.

5. Create a comfortable space conducive to productivity

The idea of being able to work from your favorite coffee shop everyday is tempting to wannabe remote employees… but it’s also extremely impractical. I am three times as productive when I’m sitting at my normal desk at home (which incidentally lets me work on three times as many screens—laptop + 2 external). Get out every now and then for a change of scenery if it’s needed, but ultimately you’ll work best if you shun the nomadic life. 

Also, don’t work out of your bedroom. You absolutely need to separate your work and personal lives (and taking a nap on your bed is just too tempting). If possible, have a dedicated space in your home where you can be productive, work undisturbed (especially important if you have kids… or an attention-starved kitten like I do), and leave at the end of the work day.

To keep from being too sedentary, I recommend getting an adjustable sitting/standing desk. I’ve also invested in a ball chair which helps me burn more calories, have better posture, and just keeps things more interesting overall.

6. Put your pants on

Another good piece of advice from another friend who has worked remotely for a long time (bonus tip: you can learn a lot by asking people their advice) is to get completely ready in the morning. Make your bed. Fix your hair. Get dressed (completely—yes, that means pants). Just because they might only see your face via video chat doesn’t mean you should get that comfortable. 

You might consider holding yourself to the same dress standards as the others in the office — you don’t want to be known as “that guy who works in his pajamas all day.” Putting yourself together at the start of the day will put you in a productive mood and help you fight lazy tendencies from the get-go.

7. Go outside

Even if you enjoy the solitary life (I include myself here), resist the inevitable tendency to become a hermit. I realized there were some days I didn’t even go outside the house—not even once. 

Go check your mail (I mean the physical mailbox… outside). Walk around the block on your lunch hour. And on the weekends, try to do something away from home, even if it’s running errands. 

A quick, brisk walk and some sunshine will do wonders for your productivity, boost your mental health, and maybe get the creative juices flowing again when you’ve hit a wall.

On a related note, when you’re done working, leave work (and don’t come back if you can help it). It’s harder to “leave work at work” when “work” is just in the other room. But try not to return to your desk. At the end of the day, make a list of things you want to accomplish tomorrow, and then wait until tomorrow to do it. This is another reason why it’s ideal to have a dedicated space in your home just for your “office.”

8. Turn on your camera

You can’t foster the right kinds of relationships if you seclude yourself and only ever talk over the phone. People need to see your face to connect with you. A friend told me once that he can never get one of his remote employees to turn on his webcam for their team meetings. As a result, he has very little insight into this guy’s life.

9. Work on more than one project at a time

You read that right. It might sound counter-intuitive to “burn the candle from both ends,” so to speak, but when you work remotely, you can’t always get someone’s attention right away when you need a roadblock cleared. Always have something else you can work on while you’re waiting for a response from a teammate. 

Communication is almost always slower when you’re remote — be forewarned and plan your work accordingly.

10. Take advantage of the perks… and be responsible

I won’t lie—there are a lot of perks to working from home. This isn’t a right, it’s a privilege. You’re saving time and money in ways that your commuting coworkers aren’t. So, don’t flaunt it (no one likes working with that person). 

Use your extra time to your benefit and that of your company and coworkers. Since you don’t have to fight traffic for an hour every morning, you could choose to get that extra sleep if you need it—but then show up and work hard when you’re at your desk. You could also choose to use at least part of that time reading, listening to podcasts, or learning a new skill.

In conclusion…

Working remotely is a privilege. It isn’t always ideal, but it can be incredibly rewarding. Know the potential pitfalls and make a concerted effort to compensate for them. You will be more productive, have better relationships with your coworkers, and make your employer look great.

Back to work.

And for goodness sakes, put your pants on.


Thanks for reading! What are your best remote working tips? Leave a comment or follow me on Twitter at @peteranglea and start a conversation.

Maybe today’s Navy is just not very good at driving ships

$
0
0

“They didn’t run aground, they hit another moving object,” he said, referring to the two major collisions this summer. “If I am staring at my radar, nav chart, phone or other watch standers, then I am not looking out the window,” Hoffman said.

Bubble Sort: An Archaeological Algorithmic Analysis

$
0
0

Owen Astrachan
Computer Science Department
Duke University

This work was supported by NSF grants CAREER 9702550 and CRCD 0088078.

Bubble Sort: An Archaeological Algorithmic Analysis

Abstract

Text books, including books for general audiences, invariably mention bubble sort in discussions of elementary sorting algorithms. We trace the history of bubble sort, its popularity, and its endurance in the face of pedagogical assertions that code and algorithmic examples used in early courses should be of high quality and adhere to established best practices. This paper is more an historical analysis than a philosophical treatise for the exclusion of bubble sort from books and courses. However, sentiments for exclusion are supported by Knuth [17], ``In short, the bubble sort seems to have nothing to recommend it, except a catchy name and the fact that it leads to some interesting theoretical problems.'' Although bubble sort may not be a best practice sort, perhaps the weight of history is more than enough to compensate and provide for its longevity.

Categories and Subject Descriptors K.3.2 [Computers & Education]: Computer & Information Science Education -Computer Science Education

General Terms Algorithms, Measurement, Theory

Keywords Analysis, Performance, Bubble sort

1  Introduction

What do students remember from their first programming courses after one, five, and ten years? Most students will take only a few memories of what they have studied. As teachers of these students we should ensure that what they remember will serve them well. More specifically, if students take only a few memories about sorting from a first course what do we want these memories to be? Should the phraseBubble Sort be the first that springs to mind at the end of a course or several years later? There are compelling reasons for excluding discussion of bubble sort2, but many texts continue to include discussion of the algorithm after years of warnings from scientists and educators. For example, in a popular new breadth-first text [6] bubble sort is given equal footing with selection sort and quicksort in online student exercises.

Starting with Knuth's premise that ``bubble sort seems to have nothing to recommend it''[17], we trace the origins and continued popularity of the algorithm from its earliest days as an unnamed sort to its current status as perhaps the most popular O(n2) sort (see below) despite wide-spread ridicule. We began this study with the intent to document (and ridicule) the continued popularity of bubble sort, but the algorithmic archaeological investigation has proven more interesting than casting aspersions.

Our initial premise that bubble sort should not be studied is reflected in [23] with a warning for potential misuse.

For N < 50, roughly, the method of straight insertion is concise and fast enough. We include it with some trepidation: it is an N2 algorithm, whose potential for misuse (by using it for too large an N) is great. The resultant waste of computer time is so awesome, that we were tempted not to include any N2 routine at all. Wewill draw the line, however, at the inefficient N2 algorithm bubble sort. If you know what bubble sort is, wipe it from your mind; if you don't know, make a point of never finding out!

This sentiment is similar to the reference to bubble sort found in [1], where it says of bogo sort, ``The archetypical perversely awful algorithm (as opposed to bubble sort, which is merely the generic bad algorithm).''

In Section 2 we trace the origin of the algorithm, both in name and in code. In Section 3 we analyze the performance of the algorithm and the simplicity of the code. In Section 4 we summarize our study.

2  Origins of Bubble Sort

In an effort to determine why bubble sort is popular we traced its origins. Knuth [17] does not provide information on the origin of the name, though he does provide a 1956 reference [10] to an analysis of the algorithm. That paper refers to ``sorting by exchange'', but not to bubble sort. An extensive bibliography and sequence of articles from the 1962 ACM Conference on Sorting [11] do not use the term bubble sort, although the ``sorting by exchange'' algorithm is mentioned. With no obvious definitive origin of the name ``bubble sort'', we investigated its origins by consulting early journal articles as well as professional and pedagogical texts.

An early (1959) book on programming [21] devotes a chapter to sorting, but uses the term exchange sorting rather than bubble sort. The same term is used in a 1962 [4] JACM article as well as in the earlier (1961, submitted 1959) [9] JACM article referenced as the definitive source. Iverson uses the name ``bubble sort'' in 1962 [13]; this appears to be the first use of the term in print. As we note below, each work cited in [13] uses a phrase other than ``bubble sort'' to describe the algorithm we describe in Section 2.1. This reinforces the claim that Iverson is the first to use the term, though obviously not conclusively. However, we could find no work published after 1962 with a reference to bubble sort in an earlier publication than Iverson's.

Despite these earlier publications the algorithm officially enters the ACM algorithm repository as Algorithm 175 [25] in 1963 where it is namedShuttle Sort. Soon thereafter [14] the published algorithm is found to be ``not free from errors'', a gentle way of saying the published code is wrong. There a modified version of the code (that stops early when no swaps are made) is given and the author says that in this form the code was ``studied in this form on the ORDVAC computer, Aberdeen Proving Ground, in 1955.''

2.1  Bubble Sort, The Code

Taking the description of bubble sort in [17] as definitive the code below is bubble sort.3 This version ``bubbles'' the largest elements to the end of the vector.

void BubbleSort(Vector a, int n)
{
    for(int j=n-1; j > 0; j--)
        for(int k=0; k < j; k++)
            if (a[k+1] < a[k])
                Swap(a,k,k+1);
}
Nearly every description of bubble sort describes how to terminate the sort early if the vector becomes sorted. This optimization requires checking if any swaps are made and terminating if no swaps are made after j iterations of the inner loop.

Another optimization is to alternate the direction of bubbling each time the inner loop iterates. This isshaker sort or cocktail shaker sort (see, e.g., [17,13]).

2.2  A Sort by Any Other Name ...

Nomenclature is interesting. An early (1963) Fortran textbook [22] refers to the following code as ``jump-down'' sort.

void JumpDownSort(Vector a, int n)
{
    for(int j=n-1; j > 0; j--)
        for(int k=0; k < j; k++)
            if (a[j] < a[k])
                Swap(a,k,j);
}
Bubble sort as we've identified it is called a ``push-down'' sort. In another early (1962) work [19] the ``jump-down'' version is presented first in the book, with no name. The bubble sort follows also with no name. In two early works [3,10] the jump-down sort is referred to as selection sort. Bubble sort is also covered, but referred to as sorting by repeated comparison andexchanging, respectively. In the latter paper, one of the earliest works comparing algorithms, the exchange/bubble sort is described thus: ``Exchanging requires at least twice as many non-housekeeping comparisons as Inserting and for most computers will be inferior''.

In moving from bubble sort to jump-down sort the only change to the code above is that one array index has been changed from k+1 to j. How does this differ from bubble sort? After m iterations of the outer loop the last m elements are in their final position-the same invariant as bubble sort. However, items are not ``bubbled'' to the top; this code implements what is essentially a selection sort, but the current maximal element is stored/swapped into location a[j] on each pass. We see why early works sometimes refer to jump-down as selection sort.

2.3  Origins of Popularity

In a survey article on sorting appearing in 1971 [20] the first sort discussed is bubble sort. It is endorsed with the following qualities.

However, the bubble sort is easy to remember and to program, and little time is required to complete a single step.

As mentioned above, Knuth [17] belittles bubble sort. The first edition of this book appeared in 1973, but perhaps did not have the same influence as did early textbooks of the same period.4

In [2] (arguably the first ``real'' textbook on algorithms and algorithm analysis) we find the following endorsement.

However, we assume that the number of items to be sorted is moderately large. If one is going to sort only a handful of items, a simple strategy such as the O(n2) ``bubble sort'' is far more expedient.

Perhaps a generation of computer scientists and teachers used this book and the acceptability of bubble sort began. In the influential 1976 work Software Tools [15] the first sort mentioned is bubble sort (the second is Shell sort).

Why is bubble sort popular? In [26] we get an endorsement.

... Nevertheless, this sorting algorithm is commonly used where the value of n is not too large and programming effort is to be kept to a minimum. ... The bubble sort has the additional virtue that it requires almost no space in addition to that used to contain the input vector.

Perhaps early concerns with allocating registers and using memory led to the adoption of bubble sort which in its primitive form (no early stopping) requires no storage other than the array and two index variables. This conclusion is supported by the early explanations of selection sort (e.g., [3,]) in which only the version described in Section 2.1 as ``jump-down'' sort is described, no separate minimum index variable is kept, and the minimal element selected in each pass of selection sort is replaced by ``a series of 9's ... so that said item will never again be selected.'' [10].

Another early work [7] is referenced in [18] with the following warning.

From a mathematical standpoint, Demuth's thesis was a beautiful piece of work. ... But from a practical standpoint, the thesis was of no help. In fact, one of Demuth's main results was that in a certain sense ``bubble sorting'' is the optimum way to sort. ... It turns out that [all other sorting methods studied] are always better in practice, in spite of the fact that Demuth has proved the optimality of bubble sorting on a certain peculiar type of machine.

Demuth's work was published 29 years after he wrote it, and he does not recall using the term ``bubble sort'' in his studies. [8].

2.4  Measures of Popularity

A 1988 SIGCSE paper [16] notes that bubble sort ``while once the best known sort, seems relegated to the status of an example of inefficiency''. Fifteen years later this sort is still with us, and students continue to use it.

Bubble sort is covered in many texts, occasionally as the only O(n2) sort, but often compared to another sort like insertion or selection. Rarely are its bad features emphasized. This may lead students to think that bubble sort is acceptable when it has provably bad performance and is arguably not the simplest sort to learn.

As unscientific albeit interesting evidence of popularity, on August 1, 2000 we searched the Internet for different sorts using the search engine Google. We repeated this search in August of 2002. For each method we used the name with and without a space, e.g., ``bubblesort'' and ``bubble sort''. Table 1 shows the results.

sort# hits 2000# hits 2002
Quick 26,780 80,200
Merge 13,330 33,500
Heap 9,830 22,960
Bubble 12,400 33,800
Insertion 8,450 21,870
Selection 6,720 20,600
Shell 4,540 8,620
Table 1: Web-based popularity of sorts

The good news is that Quicksort is by far the most-referenced sort on the web. The bad news is that bubble sort is the second or third most popular sort and is by far the most cited O(n2) sort.5 Hopefully the situation is somewhat mitigated by the greater rate of increase in hits for selection sort compared to bubble sort.

3  Performance Characteristics

Here we show that by several measures bubble sort is not simpler to code than other sorts and that its performance is terrible. We worry about bubble sort because of its inexplicable popularity. We view the use of bubble sort as an instance of a larger problem: choosing examples that are not exemplars of accepted best-practices. Such examples invariably must be unlearned later which is often an impossible task.

3.1  Ease of Coding

Conventional wisdom holds that bubble sort is simple to program. For example, in [5] we find: ``The bubble sort is worse than selection sort for a jumbled array-it will require many more component exchanges-but it's just as good as insertion sort for a pretty well-ordered array. More important, it's usually the easiest one to write correctly.''

However, in [24] we find a weak rebuttal.

Bubble sort's prime virtue is that it is easy to implement, but whether it is actually easier to implement than insertion or selection sort is arguable.

This rebuttal is stated more forcefully in [27].

The bubble sort algorithm is not very useful in practice, since it runs more slowly than insertion sort and selection sort, yet is more complicated to program.

3.1.1  Measuring Ease of Coding

Software metrics are controversial. However, perhaps the most-cited metrics based on a static analysis of code (no control-flow paths) are the Halstead Metrics [12]. These metrics are based on counts of operators and operands, though symbols such as a semi-colon are interpreted as operators. Table 2 gives the Difficulty andEffort measures for several sorts.6

Difficulty purports to measure how hard it is to create the program. Effort purports to measure the effort required to convert an algorithm into a program.

sortDE
Bubble 17.25 4165
Jump-down 14.38 3828
Select 15.95 4242
Insert 23.20 6652
Quick 7.88 3157
Partition 12.07 1522
Table 2: Halstead Complexity of Sorts

For some perspective on these numbers consider an implementation of selection divided into two parts: a function to return the index of the minimal element and the sorting function below.

void SelectSort(Vector a, int n)
{
    for(int j=0; j< n-1;j++) {
        Swap(a, minIndex(a,j,n), j);
    }    
}

This version of selection sort has a difficulty of 7.88 and an effort of 966; the minimal index function has D=14 and E=2247.

Note that the implementation of quick sort is divided into two functions, one recursive (Quick in Table 2) and one with a single loop (Partition).

3.2  Performance

Although popular, bubble sort is nearly universally derided for its poor performance on random data. This derision is justified as shown in Figure 1 where bubble sort is nearly three times as slow as insertion sort.

*plot*
Figure 1: Sorting Strings in Java

3.3  Good on Nearly-Sorted Data

Some books laud bubble sort because it runs in O(n) time on sorted data and works well on ``nearly sorted'' data. In [24] this is qualified in slightly more detail with the conclusion that insertion sort is better than bubble sort, is stable, and is the basis for the more efficient Shell sort.

This leaves little to recommend bubble sort. In any situation in which it does well insertion sort does as well and is better by other criteria. Insertion sort is used to sort small (sub) arrays in standard Java and C++ libraries.

4  Conclusion

In most practical situations the best sort to use is the one provided by the standard (e.g., Java, C, or C++) libraries. However, we study sorts because general sorts do not work in all situations and because sorting is a simple illustration of algorithmic techniques. Although examples used in first year courses must be simple enough to be understandable and complex enough to be useful in a variety of situations, they should also exemplify best practices so that these practices endure after many of the details of a course have been forgotten. In this paper we have investigated the origins of bubble sort and its enduring popularity despite warnings against its use by many experts. We confirm the warnings by analyzing its complexity both in coding and runtime.

References

[1]
The jargon file. .
[2]
Aho, A. V., Hopcroft, J. E., and Ullman, J. D. The Design and Analysis of Computer Algorithms. Addison-Wesley, 1974.
[3]
Bell, D. The principles of sorting.The Computer Journal 1 (1958), 71-77.
[4]
Bose, R. C., and Nelson, R. J. A sorting problem. Journal of the ACM (JACM) 9, 2 (1962), 282-296.
[5]
Cooper, D.Oh My! Modula-2! W.W. Norton, 1990.
[6]
Dale, N., and Lewis, J.Computer Science Illuminated. Jones and Bartlett, 2002.
[7]
Demuth, H.Electronic Data Sorting. PhD thesis, Stanford University, 1956.
[8]
Demuth, H. personal communication. 2000.
[9]
Flores, I. Analysis of internal computer sorting.Journal of the ACM (JACM) 8, 1.
[10]
Friend, E. Sorting on electronic computer systems.J. ACM 3 (1956), 134-168.
[11]
Gotlieb, C. Sorting on computers.Communications of the ACM 6, 5 (May 1963), 194-201.
[12]
Halstead, M. H. Elements of Software Science, Operating, and Programming Systems Series, vol. 7. Elsevier, 1977.
[13]
Iverson, K.A Programming Language. John Wiley, 1962.
[14]
Juelich, O. Remark on algoirthm 175 shuttle sort.Communications of the ACM 6, 12 (December 1963), 739.
[15]
Kernighan, B. W., and Plauger, P. Software Tools. Addison-Wesley, 1976.
[16]
Klerlein, J. B., and Fullbright, C. A transition from bubble sort to shell sort. In The Papers of the Nineteenth Technical Symposium on Computer Science Eduction (February 1988), ACM Press, pp. 183-184. SIGCSE Bulletin V. 20 N. 1.
[17]
Knuth, D.The Art of Computer Programming: Sorting and Searching, 2 ed., vol. 3. Addison-Wesley, 1998.
[18]
Knuth, D. E. The dangers of computer science theory. Logic, Methodology and Philosophy of Science 4 (1973). Also in Selected Papers on Analysis of Algorithms, CLSI, 2000.
[19]
Ledley, R.Programming and Utilizing Digital Computers. McGraw-Hill, 1962.
[20]
Martin, W. A. Sorting. ACM Computing Surveys 3, 4 (1971), 147-174.
[21]
McCracken, D., Weiss, H., and Lee, T.Programming Business Computers. John Wiley, 1959.
[22]
Organick, E. I. A Fortran Primer. Addison-Wesley, 1963.
[23]
Press, W. H., Flannery, B. P., Teukolsky, S. A., and Vetterling, W. T. Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press, 1988.
[24]
Sedgewick, R.Algorithms in C++, 3 ed. Addison-Wesley, 1998.
[25]
Shaw, C., and Trimble, T. Algorithm 175: Shuttle sort.Communications of the ACM 6, 6 (June 1963), 312-313.
[26]
Stanat, D. F., and McAllister, D. F. Discrete Mathematics in Computer Science. Prentice-Hall, 1977.
[27]
Vitter, J., and Flajolet, P. Average-case analysis of algorithms and data structures. In Handbook of Theoretical Computer Science. Volume A, J. van Leeuwen, Ed. MIT Press, 1990.

Footnotes:

2 A discussion of bubble sort with warnings that the performance is bad and the code isn't simple (arguably) is like telling someone ``don't think about pink elephants.''

3 This code is close to legal in both C++ and Java and should be readable by anyone with a working knowledge of Algol-like languages.

4 We are not arguing that Knuth's work is less substantial, but that it may have had less of a curricular impact than books specifically designed as textbooks rather than as works of reference.

5 It is possible though unlikely that every web page referring to bubble sorts extols its bad qualities.

6 These metrics were calculated automatically from the code given in this paper using a tool from and verified as accurate using the Unix program npath.


File translated from TEX by TTH, version 1.50.

A Tour of Acme (2012) [video]

$
0
0

People I work with recognize my computer easily: it's the one with nothing but yellow windows and blue bars on the screen. That's the text editor acme, written by Rob Pike for Plan 9 in the early 1990s. Acme focuses entirely on the idea of text as user interface. It's difficult to explain acme without seeing it, though, so I've put together a screencast explaining the basics of acme and showing a brief programming session. Remember as you watch the video that the 854x480 screen is quite cramped. Usually you'd run acme on a larger screen: even my MacBook Air has almost four times as much screen real estate.

The video doesn't show everything acme can do, nor does it show all the ways you can use it. Even small idioms like where you type text to be loaded or executed vary from user to user. To learn more about acme, read Rob Pike's paper “Acme: A User Interface for Programmers” and then try it.

Acme runs on most operating systems. If you use Plan 9 from Bell Labs, you already have it. If you use FreeBSD, Linux, OS X, or most other Unix clones, you can get it as part of Plan 9 from User Space. If you use Windows, I suggest trying acme as packaged in acme stand alone complex, which is based on the Inferno programming environment.

Mini-FAQ:

  • Q. Can I use scalable fonts? A. On the Mac, yes. If you run acme -f /mnt/font/Monaco/16a/font you get 16-point anti-aliased Monaco as your font, served via fontsrv. If you'd like to add X11 support to fontsrv, I'd be happy to apply the patch.
  • Q. Do I need X11 to build on the Mac? A. No. The build will complain that it cannot build ‘snarfer’ but it should complete otherwise. You probably don't need snarfer.

If you're interested in history, the predecessor to acme was called help. Rob Pike's paper “A Minimalist Global User Interface” describes it. See also “The Text Editor sam

Correction: the smiley program in the video was written by Ken Thompson. I got it from Dennis Ritchie, the more meticulous archivist of the pair.

The Limits of Copy-On-write: How Ruby Allocates Memory

$
0
0

Why Ruby's scheme for memory allocation doesn't play nicely with copy-on-write, and how a compacting garbage collector will help.

Anyone who’s run Unicorn (or Puma, or Einhorn) may have noticed a curious phenomena. Worker processes that have been forked from a master start with low memory usage, but before too long will bloat to a similar size as their parent. In a big production installation, each worker can be 100s of MBs or more, and before long memory is far and away the most constrained resource on servers. CPUs sit idle.

Modern operating systems have virtual memory management systems that provide copy-on-write facilities designed to prevent this exact situation. A process’s virtual memory is segmented into 4k pages. When it forks, its child initially shares all those pages with its parent. Only when the child starts to modify one of them does the kernel intercept the call, copy the page, and reassign it to the new process.

Child processes transitioning from mostly shared memory to mostly copied as they mature.

So why aren’t Unicorn workers sharing more memory? Most software has a sizeable collection of static objects that are initialized once, sit in memory unmodified throughout a program’s entire lifetime, and would be prime candidates for staying shared across all workers. Apparently though, practically nothing is reused, and to understand why, we’ll have to venture into how Ruby allocates memory.

Let’s start with a very brief overview of object allocation. Ruby requests memory from the operating system in chunks that it refers to internally as heap pages. The naming is a little unfortunate because these aren’t the same thing as the 4k page that the OS will hand out (which I will refer to hereafter as OS pages), but a heap page is mapped to a number of OS pages in virtual memory. Ruby sizes its heap pages so that they’ll maximize use of OS pages by occupying an even multiple of them (usually 4x4k OS pages = 1x16k heap page).

A heap, its heap pages, and slots within each page.

You might also hear a heap page referred to as a “heap” (plural “heaps”), “slab”, or “arena”. I’d prefer one of the last two for less ambiguity, but I’m going to stick with heap page for a single chunk and heap for a collection of heap pages because that’s what they’re called everywhere in Ruby’s source.

A heap page consists of a header and a number ofslots. Each slot can hold an RVALUE, which is an in-memory Ruby object (more on this in a moment). A heap points to a page, and from there heap pages point to each other, forming a linked list that allows the entire collection to be iterated.

Harnessing the heap

Ruby’s heap is initialized by Init_heap (ingc.c), called from ruby_setup (ineval.c), which is the core entry point for a Ruby process. Along with the heap, ruby_setup also initializes the stack and VM.

void
Init_heap(void)
{
    heap_add_pages(objspace, heap_eden,
        gc_params.heap_init_slots / HEAP_PAGE_OBJ_LIMIT);

    ...
}

Init_heap decides on an initial number of pages based on a target number of slots. This defaults to 10,000, but can be tweaked through configuration or environmental variable.

#define GC_HEAP_INIT_SLOTS 10000

The number of slots in a page is calculated roughly how you’d expect (in gc.c). We start with a target size of 16k (also 2^14 or 1 << 14), shave a few bytes off for what malloc will need for bookkeeping , subtract a few more bytes for a header, and then divide by the known size of an RVALUE struct:

/* default tiny heap size: 16KB */
#define HEAP_PAGE_ALIGN_LOG 14
enum {
    HEAP_PAGE_ALIGN = (1UL << HEAP_PAGE_ALIGN_LOG),
    REQUIRED_SIZE_BY_MALLOC = (sizeof(size_t) * 5),
    HEAP_PAGE_SIZE = (HEAP_PAGE_ALIGN - REQUIRED_SIZE_BY_MALLOC),
    HEAP_PAGE_OBJ_LIMIT = (unsigned int)(
        (HEAP_PAGE_SIZE - sizeof(struct heap_page_header))/sizeof(struct RVALUE)
    ),
}

On a 64-bit system, an RVALUE occupies 40 bytes. I’ll save you some calculations, and just tell you that with its defaults Ruby initially allocates 24 pages at 408 slots each . That heap is grown if more memory is needed.

RVALUE: An object in a memory slot

A single slot in a heap page holds an RVALUE, which is a representation of an in-memory Ruby object. Here’s its definition (from gc.c):

typedef struct RVALUE {
    union {
        struct RBasic  basic;
        struct RObject object;
        struct RClass  klass;
        struct RFloat  flonum;
        struct RString string;
        struct RArray  array;
        struct RRegexp regexp;
        struct RHash   hash;
        struct RData   data;
        struct RTypedData   typeddata;
        struct RStruct rstruct;
        struct RBignum bignum;
        struct RFile   file;
        struct RNode   node;
        struct RMatch  match;
        struct RRational rational;
        struct RComplex complex;
    } as;

    ...
} RVALUE;

For me this is where the mystique around how Ruby can generically assign any type to any variable finally starts to fall away; we immediately see that an RVALUE is just a big list of all the possible types that Ruby might hold in memory. These types are compacted with a C union so that all the possibilities can share the same memory. Only one can be set at a time, but the union’s total size is only as big as the largest individual type in the list.

To help concrete our understanding of a slot, lets look at one of the possible types it can hold. Here’s the common Ruby string (from ruby.h):

struct RString {
    struct RBasic basic;
    union {
        struct {
            long len;
            char *ptr;
            union {
                long capa;
                VALUE shared;
            } aux;
        } heap;
        char ary[RSTRING_EMBED_LEN_MAX + 1];
    } as;
};

Looking at RString’s structure yields a few points of interests:

  • It internalizes RBasic, which is struct that’s common to all in-memory Ruby types that helps distinguish between them.

  • A union with char ary[RSTRING_EMBED_LEN_MAX + 1] shows that while the contents of a string might be stored in the OS heap, a short string will be inlined right into anRString value. Its entire value can fit into a slot without allocating additional memory.

  • A string can reference another string (VALUE shared in the above) and share its allocated memory.

VALUE: Both pointer and scalar

RVALUE holds many of Ruby’s standard types, but it doesn’t hold all of them. Anyone who’s looked at a Ruby C extension will be familiar with the similarly named VALUE, which is the general purpose type that’s used to pass around all Ruby values. Its implementation is quite a bit simpler than RVALUE’s; it’s just a pointer (fromruby.h):

typedef uintptr_t VALUE;

This is where Ruby’s implementation gets clever (or gross, depending on how you think about these things). While VALUE is often a pointer to an RVALUE, by comparing one to constants or using various bit-shifting techniques, it may also hold some scalar types that will fit into the pointer’s size.

true, false, and nil are the easiest to reason about; they’re all predefined as values in ruby.h:

enum ruby_special_consts {
    RUBY_Qfalse = 0x00,		/* ...0000 0000 */
    RUBY_Qtrue  = 0x14,		/* ...0001 0100 */
    RUBY_Qnil   = 0x08,		/* ...0000 1000 */

    ...
}

A fixnum (i.e. very roughly a number that fits in 64 bits) is a little more complicated. One is stored by left-shifting a VALUE by one bit, then setting a flag in the rightmost position:

enum ruby_special_consts {
    RUBY_FIXNUM_FLAG    = 0x01,	/* ...xxxx xxx1 */

    ...
}

#define RB_INT2FIX(i) (((VALUE)(i))<<1 | RUBY_FIXNUM_FLAG)

Similar techniques are used to store “flonums” (i.e. floating point numbers) and symbols. When the time comes to identify what type is occupying a VALUE, Ruby compares pointer values to a list of flags that it knows about for these stack-bound types; if none match, it goes to heap (from ruby.h):

static inline VALUE
rb_class_of(VALUE obj)
{
    if (RB_IMMEDIATE_P(obj)) {
        if (RB_FIXNUM_P(obj)) return rb_cInteger;
        if (RB_FLONUM_P(obj)) return rb_cFloat;
        if (obj == RUBY_Qtrue)  return rb_cTrueClass;
        if (RB_STATIC_SYM_P(obj)) return rb_cSymbol;
    }
    else if (!RB_TEST(obj)) {
        if (obj == RUBY_Qnil)   return rb_cNilClass;
        if (obj == RUBY_Qfalse) return rb_cFalseClass;
    }
    return RBASIC(obj)->klass;
}

Keeping certain types of values on the stack has the advantage that they don’t need to occupy a slot in the heap. It’s also useful for speed. “Flonum” was a relatively recent addition to the language, and its author estimated that it sped up simple floating point calculations by ~2x.

Avoiding collision

The VALUE scheme is clever, but how can we be sure that the value of a scalar will never collide with a pointer? This is where the cleverness gets kicked up a notch. Remember how we talked about how an RVALUE is 40 bytes in size? That sized combined with the use of an alignedmalloc means that every address for an RVALUE that Ruby needs to put into a VALUE will be divisible by 40.

In binary, a number that’s divisible by 40 will always have three 0s in its rightmost bits (...xxxx x000). All the flags that Ruby uses to identify stack-bound types like fixnums, flonums, or symbols involve one of those three bits, therefore guaranteeing perfect exclusivity between them and an RVALUE pointer.

Now that we’ve seen some basics of the heap, we’re getting closer to understanding why our mature Unicorn processes can’t share anything with their parent (some readers may have guessed already). Let’s get the rest of the way by walking through how Ruby initializes an object; in this case a string.

The entry point is str_new0 (from string.c):

static VALUE
str_new0(VALUE klass, const char *ptr, long len, int termlen)
{
    VALUE str;

    ...

    str = str_alloc(klass);
    if (!STR_EMBEDDABLE_P(len, termlen)) {
        RSTRING(str)->as.heap.aux.capa = len;
        RSTRING(str)->as.heap.ptr = ALLOC_N(char, (size_t)len + termlen);
        STR_SET_NOEMBED(str);
    }

    if (ptr) {
        memcpy(RSTRING_PTR(str), ptr, len);
    }

    ...

    return str;
}

Just like we speculated when examining RString earlier, we can see that Ruby embeds the new value into the slot if it’s short enough. Otherwise it uses ALLOC_N to allocate new space for the string in the operating system’s heap, and sets a pointer internal to the slot (as.heap.ptr) to reference it.

Initializing a slot

After a few layers of indirection, str_alloc calls intonewobj_of back in gc.c:

static inline VALUE
newobj_of(VALUE klass, VALUE flags, VALUE v1, VALUE v2, VALUE v3, int wb_protected)
{
    rb_objspace_t *objspace = &rb_objspace;
    VALUE obj;

    ...

    if (!(during_gc ||
          ruby_gc_stressful ||
          gc_event_hook_available_p(objspace)) &&
        (obj = heap_get_freeobj_head(objspace, heap_eden)) != Qfalse) {
        return newobj_init(klass, flags, v1, v2, v3, wb_protected, objspace, obj);
    }

    ...
}

Ruby asks the heap for a free slot withheap_get_freeobj_head (in gc.c):

static inline VALUE
heap_get_freeobj_head(rb_objspace_t *objspace, rb_heap_t *heap)
{
    RVALUE *p = heap->freelist;
    if (LIKELY(p != NULL)) {
        heap->freelist = p->as.free.next;
    }
    return (VALUE)p;
}

Ruby has a global lock (the GIL) that ensures that Ruby code can only be running in one place across any number of threads, so it’s safe to simply pull the next available RVALUE off the heap’s freelist and repoint it to the next free slot in line. No finer grain locks are required.

After procuring a free slot, newobj_init runs some generic initialization on it before it’s returned to str_new0 for string-specific setup (like copying in the actual string).

Eden, the tomb, and the freelist

You may have noticed above that Ruby asked for a free slot from heap_eden. Eden, named for the biblical garden , is the heap where Ruby knows that it can find live objects. It’s one of two heaps tracked by the language.

The other is the tomb. If the garbage collector notices after a run that a heap page has no more live objects, it moves that page from eden to the tomb. If at some point Ruby needs to allocate a new heap page, it’ll prefer to resurrect one from the tomb before asking the OS for more memory. Conversely, if heap pages in the tomb stay dead for long enough, Ruby may release them back to the OS (in practice, this probably doesn’t happen very often, which we’ll get into in just a moment).

We talked a little about how Ruby allocates new pages above. After being assigned new memory by the OS, Ruby will traverse a new page and do some initialization (fromgc.c):

static struct heap_page *
heap_page_allocate(rb_objspace_t *objspace)
{
    RVALUE *start, *end, *p;

    ...

    for (p = start; p != end; p++) {
        heap_page_add_freeobj(objspace, page, (VALUE)p);
    }
    page->free_slots = limit;

    return page;
}

Ruby calculates a memory offset for the page’s start andend slots, then proceeds to walk from one end of it to the other and invoke heap_page_add_freeobj (fromgc.c) on each slot along the way:

static inline void
heap_page_add_freeobj(rb_objspace_t *objspace, struct heap_page *page, VALUE obj)
{
    RVALUE *p = (RVALUE *)obj;
    p->as.free.flags = 0;
    p->as.free.next = page->freelist;
    page->freelist = p;

    ...
}

The heap itself tracks a single freelist pointer to a slot that it knows is free, but from there new free slots are found by following a free.next on the RVALUE itself. All known free slots are chained together by a long linked list that heap_page_add_freeobj has constructed.

A heap's freelist pointer to a free RVALUE, and the continuing linked list.

heap_page_add_freeobj is called initializing a page. It’s also called by the garbage collector when it frees an object. In this way, slots get added back to freelist so that they can be reused.

Ruby has an elaborate scheme for memory management, but reading between the lines, you may also have noticed that something that’s not going to mesh well with an operating system’s copy-on-write. Ruby allocates expansive heap pages in memory, stores objects to them, and GCs slots when able. Free slots are tracked carefully, and the runtime has an efficient way of finding them. However, despite all this sophistication, a live slot will never change position within or between heap pages.

In a real program where objects are being allocated and deallocated all the time, pages quickly become a mix of objects that are alive and dead. This gets us back to Unicorn: the parent process sets itself up, and by the time it’s ready to fork, its memory looks like that of a typical Ruby process with live objects fragmented across available heap pages.

Workers kick off with the entirety of their memory shared with their parent. Unfortunately, the first time child initializes or GCs even a single slot, the operating system intercepts the call and copies the underlying OS page. Before long this has happened on every page allocated to the program, and child workers are running with a copy of memory that’s completely divergent from their parent’s.

Copy-on-write is a powerful feature, but one that’s not of much practical use to a forking Ruby process.

The Ruby team is well-acquainted with copy-on-write, and has been writing optimizations for it for some time. As an example, Ruby 2.0 introduced heap “bitmaps”. Ruby uses a mark-and-sweep garbage collector which traverses all of object space and “marks” live objects it finds before going through and “sweeping” all the dead ones. Marks used to be a flag directly on each slot in a heap page, which had the effect of GCs running on any fork and performing their mark pass to cause every OS page to be copied from the parent process.

The change in Ruby 2.0 moved those mark flags to a heap-level “bitmap” which is a big sequence of single bits mapping back slots on the heap. The GC performing a pass on a fork would only copy the OS pages needed for bitmaps, allowing more memory to be shared for longer.

The future with compaction

Upcoming changes are even more exciting. For some time Aaron Patterson has been talking publicly about implementing compaction in Ruby’s GC, and has suggested that it’s spent time in production at GitHub with some success. Practically, this would look like a method named GC.compact that’s called before workers fork:

# Called before a parent forks any workers
before_fork do
  GC.compact
end

The parent would get a chance to finish churning objects as part of its initialization, then take the objects that are still living and move them into slots on a minimal set of pages that are likely to be stable for a long time. Forked workers can share memory with their parent for longer.

A fragmented heap before and after GC compaction.

For anyone running big Ruby installations (GitHub, Heroku, or like we are at Stripe), this is really exciting work. Even deploying to high-memory instances, memory is still usually the limiting resource on the number of runnable workers. GC compaction has the potential to shave off a big chunk of the memory that every worker needs. With the savings we can run more workers per box and fewer total boxes – with very few caveats, the fleet is immediately cheaper to run.

The Limits of Copy-on-write: How Ruby Allocates Memory was published on August 28, 2017 from .

Find me on Twitter at @brandur.

Did I make a mistake? Please consider sending a pull request.

Eclipses and Decibels

$
0
0

The world in which IPv6 was a good design

Last November I went to an IETF meeting for the first time. The IETF is an interesting place; it seems to be about 1/3 maintenance grunt work, 1/3 extending existing stuff, and 1/3 blue sky insanity. I attended mostly because I wanted to see how people would react to TCP BBR, which was being presented there for the first time. (Answer: mostly positively, but with suspicion. It kinda seemed too good to be true.)

Anyway, the IETF meetings contain lots and lots of presentations about IPv6, the thing that was supposed to replace IPv4, which is what the Internet runs on. (Some would say IPv4 is already being replaced; some would say it has already happened.) Along with those presentations about IPv6, there were lots of people who think it's great, the greatest thing ever, and they're pretty sure it will finally catch on Any Day Now, and IPv4 is just a giant pile of hacks that really needs to die so that the Internet can be elegant again.

I thought this would be a great chance to really try to figure out what was going on. Why is IPv6 such a complicated mess compared to IPv4? Wouldn't it be better if it had just been IPv4 with more address bits? But it's not, oh goodness, is it ever not. So I started asking around. Here's what I found.

Buses ruined everything

Once upon a time, there was the telephone network, which used physical circuit switching. Essentially, that meant moving connectors around so that your phone connection was literally just a very long wire ("OSI layer 1"). A "leased line" was a very long wire that you leased from the phone company. You would put bits in one end of the wire, and they'd come out the other end, a fixed amount of time later. You didn't need addresses because there was exactly one machine at each end.

Eventually the phone company optimized that a bit. Time-division multiplexing (TDM) and "virtual circuit switching" was born. The phone company could transparently take the bits at a slower bit rate from multiple lines, group them together with multiplexers and demultiplexers, and let them pass through the middle of the phone system using fewer wires than before. Making that work was a little complicated, but as far as we modem users were concerned, you still put bits in one end and they came out the other end. No addresses needed.

The Internet (not called the Internet at the time) was built on top of this circuit switching concept. You had a bunch of wires that you could put bits into and have them come out the other side. If one computer had two or three interfaces, then it could, if given the right instructions, forward bits from one line to another, and you could do something a lot more efficient than a separate line between each pair of computers. And so IP addresses ("layer 3"), subnets, and routing were born. Even then, with these point-to-point links, you didn't need MAC addresses, because once a packet went into the wire, there was only one place it could come out. You used IP addresses to decide where it should go after that.

Meanwhile, LANs got invented as an alternative. If you wanted to connect computers (or terminals and a mainframe) together at your local site, it was pretty inconvenient to need multiple interfaces, one for each wire to each satellite computer, arranged in a star configuration. To save on electronics, people wanted to have a "bus" network (also known as a "broadcast domain," a name that will be important later) where multiple stations could just be plugged into a single wire, and talk to any other station plugged into the same wire. These were not the same people as the ones building the Internet, so they didn't use IP addresses for this. They all invented their own scheme ("layer 2").

One of the early local bus networks was arcnet, which is dear to my heart (I wrote the first Linux arcnet driver and arcnet poetry way back in the 1990s, long after arcnet was obsolete). Arcnet layer 2 addresses were very simplistic: just 8 bits, set by jumpers or DIP switches on the back of the network card. As the network owner, it was your job to configure the addresses and make sure you didn't have any duplicates, or all heck would ensue. This was kind of a pain, but arcnet networks were usually pretty small, so it was only kind of a pain.

A few years later, ethernet came along and solved that problem once and for all, by using many more bits (48, in fact) in the layer 2 address. That's enough bits that you can assign a different (sharded-sequential) address to every device that has ever been manufactured, and not have any overlaps. And that's exactly what they did! Thus the ethernet MAC address was born.

Various LAN technologies came and went, including one of my favourites, IPX (Internetwork Packet Exchange, though it had nothing to do with the "real" Internet) and Netware, which worked great as long as all the clients and servers were on a single bus network. You never had to configure any addresses, ever. It was beautiful, and reliable, and worked. The golden age of networking, basically.

Of course, someone had to ruin it: big company/university networks. They wanted to have so many computers that sharing 10 Mbps of a single bus network between them all became a huge bottleneck, so they needed a way to have multiple buses, and then interconnect - "internetwork," if you will - those buses together. You're probably thinking, of course! Use the Internet Protocol for that, right? Ha ha, no. The Internet protocol, still not called that, wasn't mature or popular back then, and nobody took it seriously. Netware-over-IPX (and the many other LAN protocols at the time) were serious business, so as serious businesses do, they invented their own thing(s) to extend the already-popular thing, ethernet. Devices on ethernet already had addresses, MAC addresses, which were about the only thing the various LAN protocol people could agree on, so they decided to use ethernet addresses as the keys for their routing mechanisms. (Actually they called it bridging and switching instead of routing.)

The problem with ethernet addresses is they're assigned sequentially at the factory, so they can't be hierarchical. That means the "bridging table" is not as nice as a modern IP routing table, which can talk about the route for a whole subnet at a time. In order to do efficient bridging, you had to remember which network bus each MAC address could be found on. And humans didn't want to configure each of those by hand, so it needed to figure itself out automatically. If you had a complex internetwork of bridges, this could get a little complicated. As I understand it, that's what led to the spanning tree poem, and I think I'll just leave it at that. Poetry is very important in networking.

Anyway, it mostly worked, but it was a bit of a mess, and you got broadcast floods every now and then, and the routes weren't always optimal, and it was pretty much impossible to debug. (You definitely couldn't write something like traceroute for bridging, because none of the tools you need to make it work - such as the ability for an intermediate bridge to even have an address - exist in plain ethernet.)

On the other hand, all these bridges were hardware-optimized. The whole system was invented by hardware people, basically, as a way of fooling the software, which had no idea about multiple buses and bridging between them, into working better on large networks. Hardware bridging means the bridging could go really really fast - as fast as the ethernet could go. Nowadays that doesn't sound very special, but at the time, it was a big deal. Ethernet was 10 Mbps, because you could maybe saturate it by putting a bunch of computers on the network all at once, not because any one computer could saturate 10 Mbps. That was crazy talk.

Anyway, the point is, bridging was a mess, and impossible to debug, but it was fast.

Internet over buses

While all that was happening, those Internet people were getting busy, and were of course not blind to the invention of cool cheap LAN technologies. I think it might have been around this time that the ARPANET got actually renamed to the Internet, but I'm not sure. Let's say it was, because the story is better if I sound confident.

At some point, things progressed from connecting individual Internet computers over point-to-point long distance links, to the desire to connect whole LANs together, over point-to-point links. Basically, you wanted a long-distance bridge.

You might be thinking, hey, no big deal, why not just build a long distance bridge and be done with it? Sounds good, doesn't work. I won't go into the details right now, but basically the problem is congestion control. The deep dark secret of ethernet bridging is that it assumes all your links are about the same speed, and/or completely uncongested, because they have no way to slow down. You just blast data as fast as you can, and expect it to arrive. But when your ethernet is 10 Mbps and your point-to-point link is 0.128 Mbps, that's completely hopeless. Separately, the idea of figuring out your routes by flooding all the links to see which one is right - this is the actual way bridging typically works - is hugely wasteful for slow links. And sub-optimal routing, an annoyance on local networks with low latency and high throughput, is nasty on slow, expensive long-distance links. It just doesn't scale.

Luckily, those Internet people (if it was called the Internet yet) had been working on that exact set of problems. If we could just use Internet stuff to connect ethernet buses together, we'd be in great shape.

And so they designed a "frame format" for Internet packets over ethernet (and arcnet, for that matter, and every other kind of LAN).

And that's when everything started to go wrong.

The first problem that needed solving was that now, when you put an Internet packet onto a wire, it was no longer clear which machine was supposed to "hear" it and maybe forward it along. If multiple Internet routers were on the same ethernet segment, you couldn't have them all picking it up and trying to forward it; that way lies packet storms and routing loops. No, you had to choose which router on the ethernet bus is supposed to pick it up. We can't just use the IP destination field for that, because we're already using that for the final destination, not the router destination. Instead, we identify the desired router using its MAC address in the ethernet frame.

So basically, to set up your local IP routing table, you want to be able to say something like, "send packets to IP address 10.1.1.1 via the router at MAC address 11:22:33:44:55:66." That's the actual thing you want to express. This is important! Your destination is an IP address, but your router is a MAC address. But if you've ever configured a routing table, you might have noticed that nobody writes it like that. Instead, because the writers of your operating system's TCP/IP stack are stubborn, you write something like "send packets to IP address 10.1.1.1 via the router at IP address 192.168.1.1."

In truth, that really is just complicating things. Now your operating system has to first look up the ethernet address of 192.168.1.1, find out it's 11:22:33:44:55:66, and finally generate a packet with destination ethernet address 11:22:33:44:55:66 and destination IP address 10.1.1.1. 192.168.1.1 shows up nowhere in the packet; it's just an abstraction at the human level.

To do that pointless intermediate step, you need to add ARP (address resolution protocol), a simple non-IP protocol whose job it is to convert IP addresses to ethernet addresses. It does this by broadcasting to everyone on the local ethernet bus, asking them all to answer if they own that particular IP address. If you have bridges, they all have to forward all the ARP packets to all their interfaces, because they're ethernet broadcast packets, and that's what broadcasting means. On a big, busy ethernet with lots of interconnected LANs, excessive broadcasts start becoming one of your biggest nightmares. It's especially bad on wifi. As time went on, people started making bridges/switches with special hacks to avoid forwarding ARP as far as it's technically supposed to go, to try to cut down on this problem. Some devices (especially wifi access points) just make fake ARP answers to try to help. But doing any of that is a hack, albeit sometimes a necessary hack.

Death by legacy

Time passed. Eventually (and this actually took quite a while), people pretty much stopped using non-IP protocols on ethernet at all. So basically all networks became a physical wire (layer 1), with multiple stations on a bus (layer 2), with multiple buses connected over bridges (gotcha! still layer 2!), and those inter-buses connected over IP routers (layer 3).

After a while, people got tired of manually configuring IP addresses, arcnet style, and wanted them to auto-configure, ethernet style, except it was too late to literally do it ethernet style, because a) the devices had already been manufactured with ethernet addresses, not IP addresses, and b) IP addresses were only 32 bits, which is not enough to just manufacture them forever with no overlaps, and c) just assigning IP addresses sequentially instead of using subnets would bring us back to square one: it would just be ethernet over again, and we already have ethernet.

So that's where bootp and DHCP came from. Those protocols, by the way, are special kinda like ARP is special (except they pretend not to be special, by technically being IP packets). They have to be special, because an IP node has to be able to transmit them before it has an IP address, which is of course impossible, so it just fills the IP headers with essentially nonsense (albeit nonsense specified by an RFC), so the headers might as well have been left out. (You know these "IP" headers are nonsense because the DHCP server has to open a raw socket and fill them in by hand; the kernel IP layer can't do it.) But nobody would feel nice if they were inventing a whole new protocol that wasn't IP, so they pretended it was IP, and then they felt nice. Well, as nice as one can feel when one is inventing DHCP.

Anyway, I digress. The salient detail here is that unlike real IP services, bootp and DHCP need to know about ethernet addresses, because after all, it's their job to hear your ethernet address and assign you an IP address to go with it. They're basically the reverse of ARP, except we can't say that, because there's a protocol called RARP that is literally the reverse of ARP. Actually, RARP worked quite fine and did the same thing as bootp and DHCP while being much simpler, but we don't talk about that.

The point of all this is that ethernet and IP were getting further and further intertwined. They're nowadays almost inseparable. It's hard to imagine a network interface (except ppp0) without a 48-bit MAC address, and it's hard to imagine that network interface working without an IP address. You write your IP routing table using IP addresses, but of course you know you're lying when you name the router by IP address; you're just indirectly saying that you want to route via a MAC address. And you have ARP, which gets bridged but not really, and DHCP, which is an IP packet but is really an ethernet protocol, and so on.

Moreover, we still have both bridging and routing, and they both get more and more complicated as the LANs and the Internet get more and more complicated, respectively. Bridging is still, mostly, hardware based and defined by IEEE, the people who control the ethernet standards. Routing is still, mostly, software based and defined by the IETF, the people who control the Internet standards. Both groups still try to pretend the other group doesn't exist. Network operators basically choose bridging vs routing based on how fast they want it to go and how much they hate configuring DHCP servers, which they really hate very much, which means they use bridging as much as possible and routing when they have to.

In fact, bridging has gotten so completely out of control that people decided to extract the layer 2 bridging decisions out completely to a higher level (with configuration exchanged between bridges using a protocol layered over IP, of course!) so it can be centrally managed. That's called software-defined networking (SDN). It helps a lot, compared to letting your switches and bridges just do whatever they want, but it's also fundamentally silly, because you know what's software defined networking? IP. It is literally and has always been the software-defined network you use for interconnecting networks that have gotten too big. But the problem is, IPv4 was initially too hard to hardware accelerate, and anyway, it didn't get hardware accelerated, and configuring DHCP really is a huge pain, so network operators just learned how to bridge bigger and bigger things. And nowadays big data centers are basically just SDNed, and you might as well not be using IP in the data center at all, because nobody's routing the packets. It's all just one big virtual bus network.

It is, in short, a mess.

Now forget I said all that...

Great story, right? Right. Now pretend none of that happened, and we're back in the early 1990s, when most of that had in fact already happened, but people at the IETF were anyway pretending that it hadn't happened and that the "upcoming" disaster could all be avoided. This is the good part!

There's one thing I forgot to mention in that big long story above: somewhere in that whole chain of events, we completely stopped using bus networks. Ethernet is not actually a bus anymore. It justpretends to be a bus. Basically, we couldn't get ethernet's famousCSMA/CD to keep working as speeds increased, so we went back to the good old star topology. We run bundles of cables from the switch, so that we can run one cable from each station all the way back to the center point. Walls and ceilings and floors are filled with big, thick, expensive bundles of ethernet, because we couldn't figure out how to make buses work well... at layer 1. It's kinda funny actually when you think about it. If you find sad things funny.

In fact, in a bonus fit of insanity, even wifi - the ultimate bus network, right, where literally everybody is sharing the same open-air "bus" - we almost universally use wifi in a mode, called "infrastructure mode," which simulates a giant star topology. If you have two wifi stations connected to the same access point, they don't talk to each other directly, even when they can hear each other just fine. They send a packet to the access point, but addressed to the MAC address of the other node. The access point then bounces it back out to the destination node.

HOLD THE HORSES LET ME JUST REVIEW THAT FOR YOU. There's a little catch there. When node X wants to send to Internet node Z, via IP router Y, via wifi access point A, what does the packet look like? Just to draw a picture, here's what we want to happen:

X -> [wifi] -> A -> [wifi] -> Y -> [internet] -> Z

Z is the IP destination, so obviously the IP destination field has to be Z. Y is the router, which we learned above that we specify by using its ethernet MAC address in the ethernet destination field. But in wifi, X can't just send out a packet to Y, for various reasons (including that they don't know each other's encryption keys). We have to send to A. Where do we put A's address, you might ask?

No problem! 802.11 has a thing called 3-address mode. They add a third ethernet MAC address to every frame, so they can talk about the real ethernet destination, and the intermediate ethernet destination. On top of that, there are bit fields called "to-AP" and "from-AP," which tell you if the packet is going from a station to an AP, or from an AP to a station, respectively. But actually they can both be true at the same time, because that's how you make wifi repeaters (APs send packets to APs).

Speaking of wifi repeaters! If A is a repeater, it has to send back to the base station, B, along the way, which looks like this:

X -> [wifi] -> A -> [wifi-repeater] -> B -> [wifi] -> Y -> [internet] -> Z

X->A uses three-address mode, but A->B has a problem: the ethernet source address is X, and the ethernet destination address is Y, but the packet on the air is actually being sent from A to B; X and Y aren't involved at all. Suffice it to say that there's a thing called 4-address mode, and it works pretty much like you think.

(In 802.11s mesh networks, there's a 6-address mode, and that's about where I gave up trying to understand.)

Avery, I was promised IPv6, and you haven't even mentioned IPv6

Oh, oops. This post went a bit off the rails, didn't it?

Here's the point of the whole thing. The IETF people, when they were thinking about IPv6, saw this mess getting made - and maybe predicted some of the additional mess that would happen, though I doubt they could have predicted SDN and wifi repeater modes - and they said, hey wait a minute, stop right there. We don't need any of this crap! What if instead the world worked like this?

  • No more physical bus networks (already done!)
  • No more layer 2 internetworks (that's what layer 3 is for)
  • No more broadcasts (layer 2 is always point-to-point, so where would you send the broadcast to? replace it with multicast instead)
  • No more MAC addresses (on a point-to-point network, it's obvious who the sender and receiver are, and you can do multicast using IP addresses)
  • No more ARP and DHCP (no MAC addresses, no so mapping IP addresses to MAC addresses)
  • No more complexity in IP headers (so you can hardware accelerate IP routing)
  • No more IP address shortages (so we can go back to routing big subnets again)
  • No more manual IP address configuration except at the core (and there are so many IP addresses that we can recursively hand out subnets down the tree from there)

Imagine that we lived in such a world: wifi repeaters would just be IPv6 routers. So would wifi access points. So would ethernet switches. So would SDN. ARP storms would be gone. "IGMP snooping bridges" would be gone. Bridging loops would be gone. Every routing problem would be traceroute-able. And best of all, we could drop 12 bytes (source/dest ethernet addresses) from every ethernet packet, and 18 bytes (source/dest/AP addresses) from every wifi packet. Sure, IPv6 adds an extra 24 bytes of address (vs IPv4), but you're dropping 12 bytes of ethernet, so the added overhead is only 12 bytes - pretty comparable to using two 64-bit IP addresses but having to keep the ethernet header. The idea that we could someday drop ethernet addresses helped to justify the oversized IPv6 addresses.

It would have been beautiful. Except for one problem: it never happened.

Requiem for a dream

One person at work put it best: "layers are only ever added, never removed."

All this wonderfulness depended on the ability to start over and throw away the legacy cruft we had built up. And that is, unfortunately, pretty much impossible. Even if IPv6 hits 99% penetration, that doesn't mean we'll be rid of IPv4. And if we're not rid of IPv4, we won't be rid of ethernet addresses, or wifi addresses. And if we have to keep the IEEE 802.3 and 802.11 framing standards, we're never going to save those bytes. So we will always need the "IPv6 neighbour discovery" protocol, which is just a more complicated ARP. Even though we no longer have bus networks, we'll always need some kind of simulator for broadcasts, because that's how ARP works. We'll need to keep running a local DHCP server at home so that our obsolete IPv4 light bulbs keep working. We'll keep needing NAT so that our obsolete IPv4 light bulbs can keep reaching the Internet.

And that's not the worst of it. The worst of it is we still need the infinite abomination that is layer 2 bridging, because of one more mistake the IPv6 team forgot to fix. Unfortunately, while they were blue-skying IPv6 back in the 1990s, they neglected to solve the "mobile IP" problem. As I understand it, the idea was to get IPv6 deployed first - it should only take a few years - and then work on it after IPv4 and MAC addresses had been eliminated, at which time it should be much easier to solve, and meanwhile, nobody really has a "mobile IP" device yet anyway. I mean, what would that even mean, like carrying your laptop around and plugging into a series of one ethernet port after another while you ftp a file? Sounds dumb.

The killer app: mobile IP

Of course, with a couple more decades of history behind us, now we know a few use cases for carrying around a computer - your phone - and letting it plug into one ethernet port wireless access point after another. We do it all the time. And with LTE, it even mostly works! With wifi, it works sometimes. Good, right?

Not really, because of the Internet's secret shame: all that stuff only works because of layer 2 bridging. Internet routing can't handle mobility - at all. If you move around on an IP network, your IP address changes, and that breaks any connections you have open.

Corporate wifi networks fake it for you, bridging their whole LAN together at layer 2, so that the giant central DHCP server always hands you the same IP address no matter which corporate wifi access point you join, and then gets your packets to you, with at most a few seconds of confusion while the bridge reconfigures. Those newfangled home wifi systems with multiple extenders/repeaters do the same trick. But if you switch from one wifi network to another as you walk down the street - like if there's a "Public Wifi" service in a series of stores - well, too bad. Each of those gives you a new IP address, and each time your IP address changes, you kill all your connections.

LTE tries even harder. You keep your IP address (usually an IPv6 address in the case of mobile networks), even if you travel miles and miles and hop between numerous cell towers. How? Well... they typically just tunnel all your traffic back to a central location, where it all gets bridged together (albeit with lots of firewalling) into one super-gigantic virtual layer 2 LAN. And your connections keep going. At the expense of a ton of complexity, and a truly embarrassing amount of extra latency, which they would really like to fix, but it's almost impossible.

Making mobile IP actually work1

So okay, this has been a long story, but I managed to extract it from those IETF people eventually. When we got to this point - the problem of mobile IP - I couldn't help but ask. What went wrong? Why can't we make it work?

The answer, it turns out, is surprisingly simple. The great design flaw was in how the famous "4-tuple" (source ip, source port, destination ip, destination port) was defined. We use the 4-tuple to identify a given TCP or UDP session; if a packet has those four fields the same, then it belongs to a given session, and we can deliver it to whatever socket is handling that session. But the 4-tuple crosses two layers: internetwork (layer 3) and transport (layer 4). If, instead, we had identified sessions usingonly layer 4 data, then mobile IP would have worked perfectly.

Let's do a quick example. X port 1111 is talking to Y port 80, so it sends a packet with 4-tuple (X,1111,Y,80). The response comes back with (Y,80,X,1111), and the kernel delivers it to the socket that generated the original packet. When X sends more packets tagged (X,1111,Y,80), then Y delivers them all to the same server socket, and so on.

Then, if X hops IP addresses, it gets a new name, say Q. Now it'll start sending packets with (Q,1111,Y,80). Y has no idea what that means, and throws it away. Meanwhile, if Y sends packets tagged (Y,80,X,1111), they get lost, because there is no longer an X to receive them.

Imagine now that we tagged sockets without reference to their IP address. For that to work, we'd need much bigger port numbers (which are currently 16 bits). Let's make them, say, 128 or 256 bits, some kind of unique hash.

Now X sends out packets to Y with tag (uuid,80). Note, the packets themselves still contain the (X,Y) addressing information, down at layer 3 - that's how they get routed to the right machine in the first place. But the kernel doesn't use the layer 3 information to decide which socket to deliver to; it just uses the uuid. The destination port (80 in this case) is only needed to initiate a new session, to identify what service you want to connect to, and can be ignored or left out after that.

For the return direction, Y's kernel caches the fact that packets for (uuid) go to IP address X, which is the address it most recently received (uuid) packets from.

Now imagine that X changes addresses to Q. It still sends out packets tagged with (uuid,80), to IP address Y, but now those packets come from address Q. On machine Y, it receives the packet and matches it to the socket associated with (uuid), notes that the packets for that socket are now coming from address Q, and updates its cache. Its return packets can now be sent, tagged as (uuid), back to Q instead of X. Everything works! (Modulo some care to prevent connection hijacking by impostors.2)

There's only one catch: that's not how UDP and TCP work, and it's too late to update them. Updating UDP and TCP would be like updating IPv4 to IPv6; a project that sounded simple, back in the 1990s, but decades later, is less than half accomplished (and the first half was the easy part; the long tail is much harder).

The positive news is we may be able to hack around it with yet another layering violation. If we throw away TCP - it's getting rather old anyway - and instead use QUIC over UDP, then we can just stop using the UDP 4-tuple as a connection identifier at all. Instead, if the UDP port number is the "special mobility layer" port, we unwrap the content, which can be another packet with a proper uuid tag, match it to the right session, and deliver those packets to the right socket.

There's even more good news: the experimental QUIC protocol already, at least in theory, has the right packet structure to work like this. It turns out you need unique session identifiers (keys) anyhow if you want to use stateless packet encryption and authentication, which QUIC does. So, perhaps with not much work, QUIC could support transparent roaming. What a world that would be!

At that point, all we'd have to do is eliminate all remaining UDP and TCP from the Internet, and then we would definitely not need layer 2 bridging anymore, for real this time, and then we could get rid of broadcasts and MAC addresses and SDN and DHCP and all that stuff.

And then the Internet would be elegant again.

1 Edit 2017-08-16: It turns out that nothing in this section requires IPv6. It would work fine with IPv4 and NAT, even roaming across multiple NATs.

2 Edit 2017-08-15: Some people asked what "some care to prevent connection hijacking" might look like. There are various ways to do it, but the simplest would be to do something like the SYN-ACK-SYNACK exchange TCP does at connection startup. If Y just trusts the first packet from the new host Q, then it's too easy for any attacker to take over the X->Y connection by simply sending a packet to Y from anywhere on the Internet. (Although it's a bit hard to guess which 256-bit uuid to fill in.) But if Y sends back a cookie that Q must receive and process and send back to Y, that ensures that Q is at least a man-in-the-middle and not just an outside attacker (which is all TCP would guarantee anyway). If you're using an encrypted protocol (like QUIC), the handshake can also be protected by your session key.

August 16, 2017 19:11

What I learnt from coding a text editor in C

$
0
0

As a modern JavaScript developer, I have worked on numerous single page applications with various technology stacks. I know enough of those technologies to be productive. I learned the best practices and learned when to apply them. I tried to stay on the cutting-edge and adopt new programming patterns early. However, I was never confident to say that I understand [insert state mangement library name here]. I know how to use them but I did not know why to use them. Until I started to work on viw.

For those of you who have never heard of viw, make sure to check it out,github.com/lpan/viw. It is a VI-like, terminal-based text editor written in C. I implemented Undo & Redo with an event-sourcing inspired algorithm and I applied the data-driven programming pattern. Feel free to read the source. A star would be greatly appreciated as well. :)

In this blog post, I want to discuss the lessons I learnt from implementingviw and how are they related to modern front end development. I also want to briefly talk about the Unidirectional UI pattern.

It is all about working with constraints. In C, it is hard to find recipes for stuff that you want to do. Libraries and frameworks like Redux that force you to employ a particular design pattern simply do not exist. As a result, when I was working on viw, I was forced to make many seemingly trivial decisions on my own: when do I trigger an update to the UI?, how should I categorize those functions?, how should I name this file?. As I am adding more features to the project, I have to constantly refactor, change internal APIs and move stuff around.

Here are two big refactorings I have done:

Interestingly, as the project grows bigger and as I make more incremental adjustments, the application architecture ends up becoming something that is very similar to the modern unidirectional UI architecture. The app has an infinite loop that captures all the keyboard events. Depending on the current state of the editor, these keyboard events are mapped to a series of functions (I call them mutations) that make changes to the application state—the single source of truth. After all the mutations are done, It recalculates all the computed properties (eg. cursor position) based on the new state, and pass them to ncurses to render the new output on the terminal. Then, the application waits for the next keyboard event. Recently I added Undo & Redo functionality. Inspired by event sourcing, I refactored mutations such that each of them is modelled as a command that can be stored in a log. I can pop the log to accomplish undo and re-add the command back to the log to accomplish redo (the actual implementation is slightly more complex).

viw helped me understand what UI development is really about. It is not like programming a compiler which “simply” takes an input and spits out an output. When you are programming an UI, your app has to react to events. The events can be initiated from a user, a websocket subscription or a returning AJAX request. Then, according to the event as well as the current state of the UI, your app will produce a series of resulting effects to address the incoming event. In other words, UI programming is about mapping incoming events to a series of effects.

Sounds confusing? Let’s walk through a concrete example.

Consider a simple Todo application. Our user is able to see a list of all the active todo entries as well as their total count. In addition, she is able to add new entries.

og todo app

As front end developers, the first thing we should do when given a problem like this is to identify what the incoming events are.

  1. User clicks on the “add todo” button.
  2. User enters a character into the input field

Now we have identified all the incoming events, what’s next?

Remember

UI programming is about mapping incoming events to a series of effects.

To implement this simple Todo app, our goal is to map #1 and #2 toa series of effects that responds to them!

According to the specifications of our todo app, #1 should be mapped to

if the body of the input field is not empty

  • increment the total todo count.
  • draw the new todo entry on the UI.

if the body of the input field is empty

  • draw “PLEASE AT LEAST ENTER SOMETHING” with an angry emoji in red right below the input box.

See the conditional statement ;)? This is why I said

According to the event as well as the current state of the UI…

#2 should be mapped to

if the key pressed is a backspace and the input field is not empty

  • erase the last character in the input field

if the key pressed is a backspace and the input field is empty

if the key pressed is a valid character

  • draw the character on the input field

As long as you have those two mappings implemented, you will get a working todo application.

If you want to implement the todo app with imperative programming, you will model each of the programmatic effects as an impure function. In pseudo code, it will look something like this:

function handleAddTodo(todoText: String) {
  if (!todoText) {
    renderError!
  }
  else {
    incrementCounter!
    renderNewTodo!(todoText)
  }
}

function handleKeyboardEvent(c: char) {
  if (c == backspace) {
    deleteChar!
  }

  else {
    renderChar!(c)
  }
}

This is bad because:

  • Multiple sources of truth => will result in invalid UI states.
  • Hard to implement computed properties.
  • Hard to test (functions are not pure).

Unidirectional UI pattern is an attempt of data-driven programming in the UI world. Instead of making a series of effects right away in respond to the incoming events, we do data transformation first. Then we emit all the effects based on the newly transformed data (push effects to the edges). In other words, instead of do incrementCounter! and then do renderNewTodo!, we “mutate” a data structure (let’s call it the application state), then according to the new state, we emit all the effects.

// without a persistent data structure

state = {
  todos: ['eat pizza', 'drink water']
  error: nil
  newTodoField: 'I am a new todo'
}

function handleAddTodo(state) {
  if (!newTodoField) {
    state.error = 'Empty field!'
  }

  state.todos.push(state.newTodoField)
  state.newTodoField = nil

  return state;
}

function handleKeyboardEvent(state, c: char) {
  if (c == backspace) {
    if (!state.newTodoField) {
      return
    }
    state.newTodoField.pop()
  }

  else {
    state.newTodoField.push(c)
  }

  return state;
}

// effects
renderTodo!(state) {
  renderCount!(state.todos.length)
  renderTodos!(state.todos) // react is gonna take care of it LOL
  renderError!(state.error)
}

while (true) {
  event = getEvent // blocking
  if (event.type == AddTodoButtonClicked) {
    state = handleAddTodo(state)
  }

  else if (event.type == KeyPressed) {
    state = handleKeyboardEvent(state, event.payload)
  }

  renderTodo!(state)
}

With this pattern, we get the benefits of functional programming:

  • Pure functions (with persistent data structures) => easy unit tests.
  • Predictable states.
  • And more!

Unidirectional UI is an attempt to bring data-driven programming to the UI world. Disagree with me? Feel free to leave a comment below!

Microsoft’s ‘mixed reality’ headsets are a bit of a mixed bag

$
0
0

About ten months ago, Microsoft promised the VR industry a shot in the arm and an update to PC-tethered systems by way of their “mixed reality” platform and OEM partner-crafted headsets which would boast inside-out tracking, lower system spec requirements, 1440p per eye displays and a much lower price point, with headsets starting at just $299.

Many of the headsets will be arriving this fall with full details being shared at IFA this week for some of the partners. I had a chance to test out a couple of the consumer-ready headsets from Acer and Dell earlier this month and while these “mixed reality” headsets boast a couple pretty clear advantages over the hardware offerings from HTC and Oculus, I have some doubts that they will do much to open up the VR market’s appeal to new users.

When the Windows MR headsets were unveiled, one of the main selling points of the devices were their low price-point. The promise of a $299 VR headset running on rigs powered by integrated graphics seemed to be a game-changer.

Facebook appears to have effectively neutered this impact significantly through their aggressive price cuts on the Oculus Rift. In the last six months, Oculus has sliced the all-in price of its headset and motion controllers to $399 (the price will return to $499 after a temporary sale ends), additionally, last week brought the price of the Vive to $599.

Prices of Windows MR headsets vary from $299 to $399, with some slight differences in build quality and ergonomics while technically they remain pretty much identical to each other. If you want the motion controllers it’s an extra hundred bucks. So, all-in it’s around $399-$499 to get into Windows Mixed Reality. For these prices you unfortunately don’t seem to get nearly the level of refinement offered by the Rift or Vive in terms of build quality or overall precision.

The area where price would seem to be called into question most is on what’s powering the headset. Windows Mixed Reality has a pair of modes: regular and “Ultra,” the latter of which relies on a PC with specs similar to what you need to operate a Rift or Vive, i.e. a several hundred dollar dedicated graphics card and a decent CPU. More interestingly, the regular spec Windows MR platform is capable of running on a cheap computer running integrated graphics, which I also tried. How is that experience? Odd.

When playing on systems with integrated graphics, Microsoft has made the poor choice of downgrading the headset’s frame rate from 90fps to 60fps, a move which I can guarantee will make a lot of people’s first VR experience a nauseating one. I wasn’t in the headset for more than two minutes before the warning signs availed themselves. The fact that the headset is positionally tracked helps, but extensive movement is not recommended and performance felt on-par with smartphone platforms like Daydream or the Gear VR.

For “Ultra,” things were much smoother. I demoed a number of VR experiences and on-average things felt comparable to an experience on the Gear VR while the higher-resolution displays offered crisp views. Notably during a 30-minute play session with the headset on a Mixed Reality Ultra-specced PC (with a 1080 GPU no less), it was difficult to notice any obvious issues with the headset’s lower-quality LCD displays.

There’s a lot to play. I demoed Superhot VR, Rec Room, Arizona Sunshine and a few other familiar titles. One of today’s big announcements is that Steam games will be playable on the Windows Mixed Reality platform, a big, if not expected and necessary, win.

On the topic of clear wins, the platform’s overall tracking is indeed a game-changer without qualification and an evolution that will undoubtedly be coming other headsets soon. It’s based on HoloLens tracking tech which is regarded quite highly so it’s not a huge surprise. Of course, the Windows MR headsets also have the key benefit of being tethered and not facing the same power restrictions for tracking that some future mobile headsets undoubtedly will.

Getting rid of external sensors has been something that other VR companies have expressed clear interest in but few have delivered the quality needed for high-quality immersive experiences. Microsoft has, and the ease of use that results is probably the clearest differentiating factor it has from the broader competition. It’s lacking in quality at times, but it seems to remain about as reliable as the outside-in tracking on the PlayStation VR from my brief experience.

Setup is pretty simple for the play space and basically just involves grabbing the headset and walking around the borders of your VR area so that its cameras can pick up on the space’s 3D mesh to make its job of tracking the room a bit easier. It will feel familiar to users on the Vive and Rift but unlike those sets processes, that’s where it ends. From there you jump into the Windows Cliff House and get to playing.

The performance of the controllers was much better than expected given that Microsoft has foregone the complex sensor-based infrared technology Valve and Oculus have implemented in favor of computer vision-driven solutions. Microsoft motion controllers give off a bit of a disco ball vibe as they’re tracked by a number of shining embedded LEDs that convey the controllers’ exact positioning to the headset.

It was a short, but telling demo that highlighted where the headsets are and where they will likely be at launch this holiday.

The Microsoft Mixed Reality hardware has some key strengths and pitfalls that may make this a great option for enterprise users looking for hardware that can quickly adapt to different locations and system specs, but for consumers, the price drops from HTC and Oculus make the platform a tough one to recommend at this point in time, even though there’s clearly a lot the other hardware manufacturers can learn from Microsoft’s first serious foray into virtual reality.

Viewing all 25817 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>