Quantcast
Channel: Hacker News
Viewing all 25817 articles
Browse latest View live

XChat and HexChat: When distributions get it wrong

0
0

This is a brief rant about a poor packaging decision in Debian that was made recently that just caught me by surprise and I felt like sharing.

For context for those who do not know; I am the maintainer of HexChat which is a fork of the popular IRC client XChat. This fork is roughly 8 years old in which time XChat has had 0 releases. As you can guess the fork exists in its current form solely because XChat is dead. In those years the majority of distributions have removed XChat from their repositories since it is an unmaintained project and HexChat is a logical replacement.

So this story starts with Debian removing XChat from its repo on 2016-01-30 which is not terrible in comparison to other distros but the problem arises when on 2017-08-08 it was accepted back into the repository to my surprise. Since then the maintainer has backported a few patches from HexChat including some CVE fixes and making UI changes to the input box totaling up to 44 patches as of today. Since no other upstream exists this project is no longer XChat really it is a Debian specific fork and due to timing this will land in Ubuntu 18.04 meaning this is theoretically “supported” (by the community) until 2023.

I contacted the new maintainer just to get some sort of reasoning for this and his response honestly confused me more:

“I don’t really have an answer for my adoption of xchat in Debian, it has been my first irc client, back in the days irc was really used, I loved it, I didn’t love the hexchat necessary switch, but now I’m used to the new graphic, and I find it even superior. that said, I like to have a B plan in case hexchat stops working because of some new features requiring new systems, new libraries not available maybe on older pc (e.g.I maintain an Ubuntu ppa that builds xchat back to Ubuntu 14.04, I don’t think hexchat can run on such older systems without patching, mainly due to the necessary switch to new libraries.”

This package was added to Debian Unstable and Testing so any arguments about requiring libraries don’t make sense and surely he doesn’t maintain a fork for every application he uses just as a fallback.

“I was confused about the reintroduction, as well as you, but since the first upload, I got a lot of emails, thanking me bringing it back, and a backport request really minutes after it has hit unstable again. Other Debian Developers asked me to comaintain backports on older Debian distributions, so I think I wasn’t the only one feeling nostalgic of the old days, and old graphics :).”

It is indeed popular with the community as they love choices…

“BTW, xchat is fixed for CVEs, and stable wrt libraries, I could even say that developing something increases the possibility to introduce new bugs :) Anyhow, unless you really find bugs / vulnerabilities in xchat/Debian, I would like to keep it in the archive for some more years, maybe until Ubuntu 14.04 goes End Of Life, or maybe until I find another good replacement for hexchat in case of breakages :)”

Now this reasoning is just completely wrong. I have touched nearly every corner of that codebase and stared into its soul for many years and I probably know it better than anyone else (the original author would need to refresh his memory). This codebase is terrible and it is full of security problems and bugs. Yes there were only 2 new CVEs because I don’t spend a third of my time hacking on the project reporting new CVEs as more than a few of the thousands of bugs I have fixed are potentially exploitable. The idea that only CVEs are real bugs is ridiculous and is not how software development actually works.

I have no real conclusion for this story as I cannot solve it but I hope users of these distros don’t just accept that software in the repos is maintained or safe and I hope members of the Debian and Ubuntu community can recognize that pulling in completely dead software into their repositories is a bad idea.

EDIT: As a clarification I have nothing against forks of XChat or HexChat but they need to be done responsibly as a clear new project with a dedicated developer and not a distro specific package under the defunct projects name.


25 years ago: first digital photograph in Tulsa World

0
0

Not really sure what to call it, some editor decided on "Electronic Art." It was the headline placed on the first digital photo ever published in the Tulsa World on Feb. 26, 1993.

The photo not only made history 25 years ago, but it also foreshadowed another historic event. Just a year later, the Tulsa World would be the first American newspaper to switch to digital photography completely from film.

The photo captured two kids playing at Third Street and Delaware Avenue. The caption said the photo was taken with a digital system from Kodak. "The Tulsa World is testing the system, which takes pictures electronically instead of film," the caption read. "The system includes a device that fits on Nikon N8008 cameras. The photographer takes a picture, then dumps the electronic image directly into a computer. An artist sizes the picture on a computer screen, then prints it out. The resulting print is suitable for publication. The system is faster than traditional film-processing methods."

"I was so nervous," said Brandi Simons, who took that first digital photo under the byline of Brandi Stafford. "I was 20, the only female on staff and had never shot with a digital camera before. Yeah, no pressure at all. The camera was huge, heavy and the battery was awful."

Unlike cameras today, Simons didn’t have a screen to see if her photo looked good. "I wasn’t even sure if I had anything when I went back to the newsroom."

Despite all of its burdens, the digital camera didn't require a dark room to publish a photograph. Soon after, the Tulsa World invested in 10 cameras at about $15,000 each. Without having to purchase chemicals, film and photo paper, the cameras paid for themselves quickly.

"I was very proud to be a part of something so groundbreaking," said Simons, who owns her own photography business in Owasso. "I do remember being extremely proud of the Tulsa World, and I was very proud to shoot that photo. I wasn't sure why I was given the opportunity, but I was thankful."

The Tulsa World was her first job out of Oklahoma State University’s Institute of Technology in Okmulgee. She had just been hired before given the digital camera to try out.

Simons came across a copy of the photo while helping going through her aunt’s belongings recently.

"I thought it was sweet that she had clipped it out," Simons said. "And then I looked at the date and it was 25 years ago. For me, the caption was the best thing."

Unprecedentedly wide and sharp dark matter map

0
0
Figure 1: 2 dimensional dark matter map estimated by weak lensing technique. The dark matter is concentrated in dense clumps. We can identify massive dark matter halos (indicated by oranges circles). The area shown in this figure is approximately 30 square degrees (a total of 160 square degrees were observed this time). The distribution map without the orange circles is available here. Credit: NAOJ/University of Tokyo

A research team of multiple institutes, including the National Astronomical Observatory of Japan and University of Tokyo, released an unprecedentedly wide and sharp dark matter map based on the newly obtained imaging data by Hyper Suprime-Cam on the Subaru Telescope. The dark matter distribution is estimated by the weak gravitational lensing technique (Figure 1, Movie). The team located the positions and lensing signals of the dark matter halos and found indications that the number of halos could be inconsistent with what the simplest cosmological model suggests. This could be a new clue to understanding why the expansion of the Universe is accelerating.

Mystery of the accelerated Universe

In the 1930's, Edwin Hubble and his colleagues discovered the of the Universe. This was a big surprise to most of the people who believed that the Universe stayed the same throughout eternity. A formula relating matter and the geometry of space-time was required in order to express the expansion of the Universe mathematically. Coincidentally, Einstein had already developed just such a formula. Modern cosmology is based on Einstein's theory for gravity.

It had been thought that the expansion is decelerating over time (blue and red lines in Figure 2) because the contents of the Universe (matter) attract each other. But in the late 1990's, it was found that the expansion has been accelerating since about 8 Giga years ago. This was another big surprise which earned the astronomers who found the expansion a Nobel Prize in 2011. To explain the acceleration, we have to consider something new in the Universe which repels the space.

The simplest resolution is to put the cosmological constant back into Einstein's equation. The cosmological constant was originally introduced by Einstein to realize a static universe, but was abandoned after the discovery of the expansion of the Universe. The standard cosmological model (called LCDM) incorporates the cosmological constant. The expansion history using LCDM is shown by the green line in Figure 2. LCDM is supported by many observations, but the question of what causes the acceleration still remains. This is one of the biggest problems in modern cosmology.

Wide and deep imaging survey using Hyper Suprime-Cam

Figure 2: Expansion history of the Universe. The blue line shows what was believed to be likely in the early days of cosmology. Later this cosmological model fell out of favor because it predicts a higher growth rate and more structures, inconsistent with the observed galaxy distribution. Thus a much lighter Universe model was proposed which is shown by the red line. This light model also solved the so called "age problem," the existence of globular clusters older than the age of the Universe predicted by the blue track. But both the blue and red lines conflict with the inflation cosmology. Later when the acceleration of the Universe was discovered, LCDM represented by the green track, was adopted as the most likely model. Thanks to the addition of the cosmological constant, LCDM becomes consistent with the inflation model. Credit: NAOJ

The team is leading a large scale imaging survey using Hyper Suprime-Cam (HSC) to probe the mystery of the accelerating Universe. The key here is to examine the expansion history of the Universe very carefully.

In the early Universe, matter was distributed almost but not quite uniformly. There were slight fluctuations in the density which can now be observed through the temperature fluctuations of the . These slight matter fluctuations evolved over cosmic time because of the mutual gravitational attraction of matter, and eventually the large scale structure of the present day Universe become visible. It is known that the of the structure strongly depends on how the Universe expands. For example, if the expansion rate is high, it is hard for matter to contract and the growth rate is suppressed. This means that the expansion history can be probed inversely through the observation of the growth rate.

It is important to note that growth rate cannot be probed well if we only observe visible matter (stars and galaxies). This is because we now know that nearly 80 % of the matter is an invisible substance called dark matter. The team adopted the 'weak gravitation lensing technique." The images of distant galaxies are slightly distorted by the gravitational field generated by the foreground dark matter distribution. Analysis of the systematic distortion enables us to reconstruct the foreground dark matter distribution.

Figure 3: Hyper Suprime-Cam image of a location with a highly significant dark matter halo detected through the weak gravitational lensing technique. This halo is so massive that some of the background (blue) galaxies are stretched tangentially around the center of the halo. This is called strong lensing. (Credit: NAOJ

This technique is observationally very demanding because the distortion of each galaxy is generally very subtle. Precise shape measurements of faint and apparently small galaxies are required. This motivated the team to develop Hyper Suprime-Cam. They have been carrying out a wide field imaging survey using Hyper Suprime-Cam since March 2014. At this writing in February 2018, 60 % of the survey has been completed.

Unprecedentedly wide and sharp dark matter map

In this release, the team presents the dark matter map based on the imaging data taken by April 2016 (Figure 1). This is only 11 % of the planned final map, but it is already unprecedentedly wide. There has never been such a sharp dark matter map covering such a wide area.

Imaging observations are made through five different color filters. By combining these color data, it is possible to make a crude estimate of the distances to the faint background galaxies (called photometric redshift). At the same time, the lensing efficiency becomes most prominent when the lens is located directly between the distant galaxy and the observer. Using the photometric redshift information, galaxies are grouped into redshift bins. Using this grouped galaxy sample, dark matter distribution is reconstructed using tomographic methods and thus the 3-D distribution can be obtained. Figure 4 shows one such example. Data for 30 square degrees are used to reconstruct the redshift range between 0.1 (~1.3 G light-years) and 1.0 (~8 G light-years). At the redshift of 1.0, the angular span corresponds to 1.0 G x 0.25 G light-years. This 3-D dark matter mass map is also quite new. This is the first time the increase in the number of dark matter halos over time can be seen observationally.

What the dark matter halo count suggests and future prospects

Figure 4: An example of 3D distribution of dark matter reconstructed via tomographic methods using the weak lensing technique combined with the redshift estimates of the background galaxies. All of the 3D maps are available here. Credit: University of Tokyo/NAOJ

The team counted the number of dark matter halos whose lensing signal is above a certain threshold. This is one of the simplest measurements of the growth rate. The histogram (black line) in Figure 5 shows the observed lensing signal strength versus the number of observed halos whereas the model prediction is shown by the solid red line. The model is based on the standard LCDM model using the observation of cosmic microwave background as the seed of the fluctuations. The figure suggests that the number count of the halos is less than what is expected from LCDM. This could indicate there is a flaw in LCDM and that we might have to consider an alternative rather than the simple cosmological constant.

The statistical significance is, however, still limited as the large error bars (vertical line on the histogram in Figure 5) suggest. There has been no conclusive evidence to reject LCDM, but many astronomers are interested in testing LCDM because discrepancies can be a useful probe to unlock the mystery of the accelerating Universe. Further observation and analysis are needed to confirm the discrepancy with higher significance. There are some other probes of the growth rate and such analysis are also underway (e.g. angular correlation of galaxy shapes) in the team to check the validity of standard LCDM.

These results were published on January 1, 2018 in the HSC special issue of the Publications of the Astronomical Society of Japan. The report is titled "A large sample of shear-selected clusters from the Hyper Suprime-Cam Subaru Strategic Program S16A Wide field mass maps."

Figure 5: Number of dark matter halos versus their lensing signal strength (black histogram) and number count expected from LCDM and the most recent CMB observation by the Planck satellite. Credit: NAOJ/University of Tokyo

Two-dimensional dark matter map estimated by weak lensing technique. The dark matter is concentrated in dense clumps. Credit: NAOJ

Explore further:Mapping dark matter

More information: A large sample of shear selected clusters from the Hyper Suprime-Cam Subaru Strategic Program S16A wide field mass maps. arxiv.org/abs/1802.10290v1

Probabilistic thinking can lead to improved decision-making

0
0

When making hard decisions, do you go with your gut or try to calculate the risks? In many cases going with your gut is fine, but the answers to our February puzzle problems show how explicit probabilistic thinking can outperform intuitive estimates. They also highlight the differences between situations where an intuitive approach succeeds and ones where it fails.

Our first problem was an excerpt from a whimsical children’s story.

Problem 1

A man went on an airplane ride.

Unfortunately, he fell out.

Fortunately, he had a parachute on.

Unfortunately, the parachute did not open.

Fortunately, there was a haystack below him, directly in the path of his fall.

Unfortunately, there was a pitchfork sticking out of the top of the haystack, directly in the path of his fall.

Fortunately, he missed the pitchfork.

Unfortunately, he missed the haystack.

There have indeed been alleged instances of people surviving falls from planes by landing on haystacks, or even by falling into trees or thick shrubbery, as a quick online search will reveal. So the alternating screams inside this imaginary man’s head — “I’m dead!”/“I’m saved!” — are not conclusive until the sad finale. (Though our story seems to end tragically, in the original the protagonist survives with many more abrupt reversals of fortune!) Can principled methods of risk estimation be applied here? Given the available information, estimate his odds of survival at the end of each line above.

Douglas Felix made a very good attempt at answering this question. As his answers indicate, the probability of survival at each step is either very close to zero or very close to one, so it is very hard to express the absolute and relative risks in a way that’s easy to understand. The Stanford scientist Ronald Howard, pioneer of risk analysis, invented a measure for just this purpose — the micromort. The micromort is a one-in-a-million chance of death. You can use the usual metric prefixes to express risks even closer to zero, such as nanomorts and picomorts. For the purposes of our puzzle, we can name a flipped unit, the microvivé, for situations in which the chances of survival are very close to zero — a risk of one microvivé would mean a one in a million chance of surviving. The sum of the number of micromorts and microvivés is always one million. Armed with these measures, let’s take a look at the risks in the story above, assuming it happened around the present time in history. For reference, the average risk of death by an unnatural cause for a middle-aged person per day is about 1 micromort.

A man went on an airplane ride.

In the absence of any more information, it is overwhelmingly likely that a random plane ride is on a commercial air carrier. The rate of fatal accidents worldwide for this today is about 0.2 to 0.4 per million flights, giving a risk of death of 0.2 to 0.4 micromorts, not adding very much to one’s normal everyday risk.  However, if we knew that the man had a parachute strapped on, as is revealed in the third line, we would have to conclude that this was probably no ordinary ride. Perhaps it was a skydiving trip or military aircraft mission. In that case, we would need to use the figures reported for general aviation fatalities, not the much-better figures for commercial air carriers. One published estimate based on official data puts this risk at something on the order of 1 in 100,000, which is 10 micromorts.

Unfortunately, he fell out.

The chances of survival after a fall from an airplane are hard to estimate, but probably not as high as we think. When we hear this sentence, we assume he didn’t have a parachute, but that is not explicitly denied yet. Even if he didn’t have a parachute, there are documented cases of a handful of people who survived a fall from an airplane. The number of people who have fallen out of airplanes without parachutes throughout history is probably not more than 100,000 to 300,000, based on published air fatality numbers. Based on these two facts, the chances of survival without a parachute may range from 1 in 10,000 to 1 in 30,000, which is 30 to 100 microvivés (a chance of dying of 999,900 to 999,970 micromorts).

Fortunately, he had a parachute on.

The official figures for skydiving accidents put the risk of dying in a skydiving accident at about 8 to 9 micromorts. The chance of dying has dropped considerably.

Unfortunately, the parachute did not open.

The chances of survival plummet to 30 to 100 microvivés, as above.

Fortunately, there was a haystack below him …

Considering that people have survived falls into shrubbery, a haystack could possibly improve the chances of survival, perhaps allowing an escape with some broken bones. It is hard to estimate, but the odds are perhaps somewhere between one in a hundred to one in a thousand of surviving —  1,000 to 10,000 microvivés, many times better than before.

Unfortunately, there was a pitchfork sticking out … 

Ouch! But with a surrounding haystack, the odds would still be better than if the pitchfork and haystack were absent. I’d put the odds somewhere between 50 and 500 microvivés.

Fortunately, he missed the pitchfork.

Back to 1,000 to 10,000 microvivés.

Unfortunately, he missed the haystack.

As after line two, back to 30 to 100 microvivés, or a 99.990 to 99.997 percent chance of dying.

As is obvious, we need hard data to give tight probabilities. Nevertheless, we can give reasonable, albeit broad estimates. The units, once you get used to them, make things clearer and give more information rather than simply saying, “very, very close to zero/one.”

(It is tempting to invent similar units to add clarity in other situations: For instance, a one-in-a-million chance of an utterance being true could be called a microvérité or a microvér for short — a handy unit in this era of “fake news.”)

Problem 2

The number of deaths on commercial airlines is about 0.2 deaths per 10 billion flight-miles. For driving, the rate is 150 deaths per 10 billion vehicle-miles. While this rate is about 750 times higher than for air travel, we still take long road trips because the absolute risks are small. But let us pursue a thought experiment using two hypothetical and admittedly unrealistic assumptions — first, that your expected life span is 1 million years (and you enjoy every year of it!), and second, that the above risks remain the same during those million years. Now imagine that every year you could either fly 10,000 miles or cover that distance by car over multiple road trips. The time spent is not a concern at all — after all, you have a million years to live! Under these conditions, by how many years and by what proportion would your life be shortened if you lived a million years and drove every time instead of flying? How would your answer differ for a more normal life span of 100 years?

Alexandre answered this problem correctly. To figure out how much your life would be shortened on average by flying, consider that, in this problem, a flying death occurs every 50 billion flight-miles on average. In a life span of 1 million years, flying 10,000 miles a year, you cover only a fifth of this distance, a total of just 10 billion miles. So you have an 80 percent chance of living a full million years. This means that out of a group of five randomly selected people, four would be expected to live one million years, while the fifth would be expected to have a fatal accident sometime in his or her life, for an average life span of half a million years. So the average life expectancy for the entire group would be 4.5 million/5 or 900,000 years, or 90 percent of normal, as Alexandre correctly calculated. On the other hand, for driving, a death occurs on average every 66.67 million miles (10 billion/150), a distance you’d expect to have driven by the time you were 6,667 years old. So the average 1-million-year lifetime would be reduced to 6,667 years, or about 0.67 percent of its expected length. Using the same method of calculation for a 100-year life span, we obtain the following results: For flying, the average life span will be shortened by 0.001 years or about eight hours, whereas for driving it will be shortened by 0.075 years or about a month.

What this demonstrates is that people who lived a million years would tend to be very risk-averse! But more seriously, it shows that choosing just slightly riskier options repeatedly can increase risk significantly over time.

Problem 3

Here are two similar scenarios in which you have to make probability judgments. Before you make an exact calculation, hazard an intuitive guess and jot it down.

Variation A: A certain town has two ethnic groups, the Ones and the Twos. Ones make up 80 percent of the population. A hospital clinic conducts a standard, unbiased screening test for a rare disease that is equally common in both groups. It results in the collection of 100 blood samples, and sure enough, 80 of the samples come from Ones. On more rigorous testing, just one of the 100 samples is found to be positive for the disease. A researcher who is not privy to the ethnicity data because of HIPAA laws runs a test on this sample, which determines that it comes from a Two. However, this ethnicity-determining test is known to be only 75 percent accurate. What is the probability that the sample actually came from a Two?

Variation B: In this variation, Ones and Twos both make up 50 percent of the population, but Ones are more likely to have the rare disease. The same screening procedure as above collects 100 blood samples, again yielding 80 from Ones and 20 from Twos. The rest of the problem is exactly the same. Now what is the probability that the diseased sample actually came from a Two?

In which of the two cases was your intuition more accurate?

The right answer for the probability, in the case of both variations, is 3/7 or 42.9 percent, which was obtained by Douglas Felix and Alexandre. This is a standard question in probability theory, and can be solved by the standard Bayesian conditional probability formula P(A|B) = P(B|A) × P(A) / P(B), or for those who like visual thinking, by using the elegant geometric approach illustrated by Alexandre. Below I give a simple arithmetic approach that can always be applied to such problems and offers an understanding of how the answer follows logically.

Let’s start with the 100 samples in variation A. Of these, 80 belong to Ones. The ethnicity test is accurate three-quarters of the time, so it will label only 60 of these samples as belonging to Ones, and mislabel the other 20 as belonging to Twos. Similarly, out of the 20 samples from Twos, the ethnicity test will label 15 correctly as belonging to Twos, and mislabel 5 as belonging to Ones. Thus, the ethnicity test will label a total of 20 + 15 = 35 samples as Twos, out of which only 15 are truly Twos. Since the screening test is unbiased, any one of these is equally likely to have the rare disease. So the chance that the sample actually came from a Two is 15/35 or 42.9 percent. What this means is that even though the ethnicity test is reasonably accurate, and has identified the disease sample as that of a Two, that sample is still more likely to have come from a One! When asked to guess in situations like these, most people estimate the chance that the disease sample came from a Two to be around 75 percent.

Note that when we say that the screening test is unbiased, we mean that all individuals whose screening test is positive have an equal chance of having the disease, regardless of their ethnicity. A couple of readers wrote that the test could not be unbiased in variation B, because 80 percent of the samples were from Ones, even though there are equal numbers of Ones and Twos in the population. Here’s how this could happen. Let’s say that high blood pressure predisposes a person to the disease, and the screening test catches everyone with a blood pressure above a certain threshold. Now, in Variation B, Ones may be genetically predisposed to get high blood pressure. So the screening test collects more samples from Ones. And therefore, as mentioned, Ones are more likely to get the disease. Once the screening test is done, however, its ethnically unbiased nature ensures that all samples collected are equally likely to have the rare disease. Thus variation B also gives the same answer: 42.9 percent. In this case, however, our intuitions kick in correctly, and our estimated probability comes out closer to the correct answer.

Why should this be? Such errors in our intuitive reasoning about probabilities and the reasons behind them have been fascinatingly classified and described by the Nobel Prize-winning psychologist Daniel Kahneman in his best-selling book Thinking, Fast and Slow. As Kahneman notes, the kind of false Bayesian reasoning demonstrated in our problem was elucidated by the psychologist Icek Ajzen. Kahnemann describes this false reasoning as “causes trump statistics.” Our brains reason better with causes, rather than with numbers. In variation B, the idea that Ones are more prone to the disease is a cause we can latch on to, and it helps us discount the results of the ethnicity test as we should; in variation A, however, we don’t have such a causal story to tell ourselves — we just have numbers, which are not as compelling. Hence we give greater importance to the results of the ethnicity test, and discount the numbers, even though we should not. Mathematically, both variations are exactly the same.

Thank you to all those who commented. The Quanta T-shirt goes to Alexandre. Congratulations!

Insights will take a break in March and continue on a bimonthly basis. See you all in April.

Updated on March 2, 2018: This article was updated to clarify that the average risk of death “by an unnatural cause” for a middle-aged person per day is about 1 micromort.

American using Britain’s NHS (2015)

0
0
I've spent half my life in the US and half of it in the UK, so I'm used to both countries' healthcare systems.I recently returned to London after 20 years in America , and after a few doctors' appointments I've come to see the NHS through American eyes.

The National Health Service is, as all Americans know and fear, a completely public "socialized medicine" system. It's dramatically different from the US's patchwork system of private providers and insurance companies.

My story isn't representative, of course. Healthcare delivery is different in the UK depending on where you live and which doctors and hospitals you use - just as it is in the US. But I've now used both systems for about two decades each, so I feel I have a pretty good handle on the main contrasts.

'THIS ROLLS ROYCE ISN'T MOVING FAST ENOUGH!'

The context here is that the NHS just released its most recent stats on accident and emergency room waiting times. The headline number is that84% of patients are seen within four hours . In the UK, this is regarded as a huge failure - the standard the NHS is supposed to meet is 95% of patients in four hours. The UK media went into a fury about it, and some hospitals have begun postponing and rescheduling some non-emergency procedures in order to get those waiting times down.

In the US, having sat in many an ER waiting room for hours at a stretch, the idea of a hospital seeing nearly 9 out of 10 patients in four hours would be regarded as a miracle. Bear in mind that within that four-hour period the NHS doctors are triaging patients: If you get hit by a bus, you're going to see someone instantly. If you broke a finger because you fell over while drunk at the pub, you're probably going to wait at the back of the line. It's not like people are literally bleeding to death while they wait for attention (although the British media loves it when it finds individual cases where that has happened).

So my overall impression is that currently, the Brits' complaints that the NHS isn't hitting that 95% mark is akin to saying, "This Rolls Royce isn't moving fast enough!"

SHOW UP WHEN YOU'RE TOLD TO - OR ELSE

The first major difference from the patient's point of view is what happens when you call your doctor for a routine appointment. My specific health issue was that I thought I was going slightly deaf, and wanted it checked out.

I'm a dual US/UK citizen, so I qualify for NHS treatment. Here's what happened to me.

In America, you call your doctor and request an appointment when it's convenient for you. They might ask you what's wrong with you, presumably to make sure it's not something that requires immediate treatment. But basically, it's first come, first served, regardless of how important it is. Usually, you can pick an appointment time that's convenient for you if it is not an emergency.

In the UK, I was given an appointment whether I liked it or not. I called and leave a message. Within an hour or two a nurse practitioner called me back and asked me a few questions about my problem over the phone. (You've got to take the call in a private place if you don't want your office mates to hear.) Then they said: Come in at 9am on Thursday. There was no choice over appointment times - the assumption is that if you're ill, you're going to come to the doctor when they say.

At first I found this jarring. In America,I get to choose whenI see the doctor! In Britain, I better show up when I'm told. But the appointment came quickly, as the local health authority in London has targets it needs to meet. Ultimately, I saw the logic of it: This is a public health system. It needs to manage its costs and services. If you're really sick, you'll show up. If you only want to show up when it's convenient for your schedule, then how sick are you, really?

AMERICA IS WORSE AT ON-THE-DAY CARE

In America, I've always had a long wait to see my doctor. I have read many a back issue of Newsweek in my primary care / general practitioner (GP) doctor's office. I've sat there for an hour playing with my phone while the doc sees patients in the order they were booked.

In the UK, I showed up at 9am and was seen instantly , at the Waterloo Health Centre. For an American, this was bizarre: My butt barely touched the seat in the waiting room before my name was called. Turns out my doc and her staff are serious about patient scheduling.

This was one reason I became convinced that the NHS way of scheduling is superior: You might not get the time or date that you want, but once you're in, you get seen super-quick.

THE NHS ACTIVELY DISCOURAGES SOME PATIENTS - FOR GOOD REASON

The NHS actively discourages some types of patients: Interestingly, NHS offices and hospitals have posters up all over the place warning you not to show up at the emergency room if you have a cold or the flu. They're actively discouraging patients with minor ailments from seeking emergency treatment, and trying to get them to see their regular doctors instead. It's sensible - everyone knows that a vast amount of hospital time and money is wasted treating people who are not an emergency. And hospitals and doctor's surgery waiting rooms are a hotbed of germs. But still, it's a culture shock to see a medical institution put up signs that basically say, "go home, you idiot!" in every waiting room.

The US never discourages patients from doing anything. I've never seen any kind of public campaign to persuade patients to apply some common sense before dropping themselves off at an emergency room. The entire US pharmaceutical industry is also dedicated to running ads encouraging people to "go see your doctor" for even the most trivial of conditions.

The treatment from my primary care GP was the same in the UK as it was in the US. I've had great care from 95% of doctors I've ever seen in both the US and the UK. Doctors are doctors. They're mostly really nice and good at what they do. The system that pays them doesn't seem to make them better or worse.

THERE IS BASICALLY NO PAPERWORK WITH THE NHS

There is a load of paperwork for patients in the US. This is easily the worst aspect of US healthcare - the billing paperwork. If you've ever had any health issue that required more than a simple doctor visit, you will know that it precipitates a seemingly never-ending series of forms, bills, and letters. You can be paying bills months, years later. And it's almost impossible to correct a billing error. It's stressful. I developed an intense hatred for health insurance companies in the US because of this.

There was close to zero paperwork in the NHS. I filled in a form telling my doc who I was and where I lived, and that was pretty much it. The only other paperwork I got was a letter in the mail reminding me of my next appointment. They sent me a text reminder, too, which no American doc has ever done. It was incredibly refreshing.

THE STANDARD OF CARE IS THE SAME

So, was I going deaf? Maybe. Maybe not. I'd lost my sense of balance in summer 2014, which an American doctor had diagnosed asBenign Paroxysmal Positional Vertigo . It's a condition of the inner ear. It made my body feel slightly drunk and clumsy even though I am completely sober.

The US doc told me there is no treatment and it goes away on its own, mostly. A lot of people get it, apparently. I was managing fine and it doesn't bother me, anyway.

The UK doc told me the same thing, but also suggestedI might have Meniere's Disease , and wanted to send me to a specialist to get it checked out. Meniere's isn't really a disease, it's just a collection of symptoms: dizziness, hearing loss and a ringing in the ears. Again, there is no treatment. But it's rare.

This freaked me out a little bit. I was used to the US system which is heavily "defensive." Doctors tend to over-treat patients because they get sued if they miss something. They also get paid more money for doing more work. Yet it was the NHS doctor that suggested extra treatment.

It was going to be free - so I said yes!

A LONG WAIT FOR NHS TREATMENT ...

I then made an appointment with a specialist at the Guy's and St Thomas' Hospital in London.

In the US, I've always been able to see a specialist within a few days. Score one for America.

In the UK, they said "we'll see you in January." It was late November, six weeks or more away. This was a shock. I was going deafnow - not in six weeks! What the hell?!

NHS waiting times are a real thing, it turns out. I comforted myself with the assumption that the staff had made a decision that my condition was likely not life- or health-threatening, and had moved me to the back of the line. It was frustrating. Ultimately, I also needed to change my appointment because I had to leave the country on business, and this was quite difficult to do. I had to call a few times, basically to catch the hospital booking staff at the right time of day, in order to do it. I wished Guy's and St. Thomas' had an online system for this, but they don't - just a bunch of people answering phones, most of whom don't have access to the right appointment schedule.

It was that appointment system again: You're booked in according totheir priority, notyours. The big lesson with the NHS is, it's a lot easier to just show up when you're told.

OLD PEOPLE IN BRITAIN ARE REALLY RUDE

In the US, I expect to wait up to an hour in the specialist's waiting room on the day of my appointment. I often wonder if Time and Newsweek were such big magazines in the US because they're needed for bored patients in American doctors' waiting rooms. Nothing ever happens promptly on the day in US healthcare, as far as I can tell.

In the NHS, again, I waited only a couple of minutes. Credit to the staff at St. Thomas, they arecranking through their patients.

On two occasions I noticed old people complaining angrily (and rudely) to the office staff that they had been made to wait 15 or 20 minutes to see their doctor. As an American, I almost laughed out loud. Fifteen minutes to see a free doctor!This Rolls Royce isn't moving fast enough! I asked a British friend - someone who has ongoing health issues and sees a lot of doctors - if old people complaining like this was common. Turns out, it is. Old British people love to complain to NHS staff if they wait more than 1o minutes. Everyone just expects their appointments to be exactly on time.

Again, the NHS care was great. I saw two different doctors within an hour, one for testing and one for diagnosing. A third admin staffer was coordinating the lists so there was no doctor downtime. It was like being in a highly efficient factory. It looked like hard work. I could tell that one of my doctors was not interested in chatting. She treated me, and wanted me out the door. There was a bunch of patients behind me, after all. In America, docs seem to be happy to chat as long as you want - and you can tell that extra couple of minutes with each patient creates long delays as the day wears on.

The good news: I am not going deaf! I havegreat hearing, it turns out. They even showed me a chart of it. But the tinnitus - ringing in my ears that started years ago because I used to go to a lot of punk rock gigs in my youth - has gotten worse, making mefeel more deaf.

The UK NHS specialist said she was 99% sure there was nothing wrong with me, or at least nothing that could be treated, but she recommended an MRI to see what the condition of my inner ears is like. This was reassuring. In no way was my treatment rationed or denied, the way Americans fear. It was just the same as in the US, with the same number of docs and the same level of high-tech equipment.

THE COST TO THE PATIENT IS MUCH CHEAPER IN THE UK, OBVIOUSLY

So how much did all this NHS care cost me? £0. Nothing. Zero. I paid not a penny for some top-notch healthcare. There is no such thing as a "free," of course, but the per-capita cost of healthcare in the UK (paid by the government via tax collections) is generally lower than the US, according to the World Health Organization . Americans spend $8,362 per capita on healthcare annually, the Brits spend $3,480. Here is a breakdown:

NHS prices

  • Doctor visit: £0
  • Specialist: £0
  • Diagnostic test: £0
  • MRI: £0
  • Total: £0

Typical US prices*

  • Doctor visit: $100
  • Specialist: $150
  • Hearing test: $72
  • MRI: $1,000
  • Total: $1,372 (Total payable by the patient in cash, or typically 90% from insurance and 10% as a patient copay.Prices taken from Healthcare Bluebook. )

SORRY AMERICA, BUT NHS TREATMENT REALLY IS BETTER OVER ALL

The bottom line: I prefer the NHS to the American private system. It's a little more inconvenient in terms of appointment times, but due to the fact that it is free, has no paperwork, and the treatment on the day is super-fast, the NHS wins. That Rolls Royce is moving at a pretty decent clip.

And, of course, there is the small matter of the fact that the NHS covers everyone equally, whereas Americans get care based on their ability to pay, leavingtens of millions with only minimal access to care. (Obamacare is changing that, but it's leagues behind the NHS if you're comparing them by the standard of universal full-service coverage.)

Americansthink they have the best healthcare in the world. Take it from me, a fellow American: They don't.

Clojure Don’ts: Lazy Effects

0
0

This is probably my number one Clojure Don’t.

Laziness is often useful. It allows you to express “infinite” computations, and only pay for as much of the computation as you need.

Laziness also allows you to express computations without specifying when they should happen. And that’s a problem when you add side-effects.

By definition, a side-effect is something that changes the world outside your program. You almost certainly want it to happen at a specific time. Laziness takes away your control of when things happen.

So the rule is simple: Never mix side effects with lazy operations.

For example, if you need to do something to every element in a collection, you might reach for map. If thing you’re doing is a pure function, that’s fine. But if the thing you’re doing has side effects, map can lead to very unexpected results.

For example, this is a common new-to-Clojure mistake:

(take 5 (map prn (range 10)))

which prints

0
1
2
3
4
5
6
7
8
9

This is the old “chunked sequence” conundrum. Like many other lazy sequence functions, map has an optimization which allows it to evaluate batches of 32 elements at a time.

Then there’s the issue of lazy sequences not being evaluated at all. For example:

(do(map prn [0 1 2 3 4 5 6 7 8 9 10])
(println "Hello, world!"))

which prints only:

Hello, world!

You might get the advice that you can “force” a lazy sequence to be evaluated with doall or dorun. There are also snippets floating around that purport to “unchunk” a sequence.

In my opinion, the presence of doall, dorun, or even “unchunk” is almost always a sign that something never should have been a lazy sequence in the first place.

Only use pure functions with the lazy sequence operations like map, filter, take-while, etc. When you need side effects, use one of these alternatives:

  • doseq: good default choice, clearly indicates side effects
  • run!: new in Clojure 1.7, can take the place of (dorun (map ...))
  • reduce, transduce, or something built on them

The last requires some more explanation. reduce and transduce are both non-lazy ways of consuming sequences or collections. As such, they are technically safe to use with side-effecting operations.

For example, this composition of take and map:

(transduce (comp (take 5)
(map prn))
           conj
[]
(range 10))

only prints 5 elements of the sequence, as requested:

0
1
2
3
4

The single-argument version of map returns a transducer which calls its function once for each element. The map transducer can’t control when the function gets evaluated — that’s in the hands of transduce, which is eager (non-lazy). The single-argument take limits the reduction to the first five elements.

As a general rule, I would not recommend using side-effecting operations in transducers. But if you know that the transducer will be used only in non-lazy operations — such as transduce, run!, or into— then it may be convenient.

(defnoperation[input]
;; do something with input, return result
(str "Result for " input))

(prn (into #{}
(comp (take 3)
(map operation))
(range 100)))

reduce, transduce, and into are useful when you need to collect the return value of the side-effecting operation.

We X-Rayed Some MLB Baseballs. Here’s What We Found

0
0

On 6,105 occasions last season, a major leaguer walked to the plate and hammered a baseball over the outfield wall. The 2017 season broke the home run record that was set in 2000 — the peak of the steroid era — when players hit 5,693 homers, and it built upon the remarkable 5,610 that were hit in 2016. It was a stunning display of power that played out in every MLB park almost every night. And with spring training underway in Florida and Arizona, MLB’s power surge is showing no sign of letting up.

But while we now know what caused the spike in home runs at the turn of the century — even if we didn’t at the time — the reason for the most recent flurry of long balls remains an unsolved mystery. Any number of factors might have contributed to the home run surge, including bigger, stronger players or a new emphasis on hitting fly balls. But none of those possibilities looms larger than the ball itself.

MLB and its commissioner, Rob Manfred, have repeatedly denied rumors that the ball has been altered in any way — or “juiced” — to generate more homers. But a large and growing body of research shows that, beginning in the middle of the 2015 season, the MLB baseball began to fly further. And new research commissioned by “ESPN Sport Science,” a show that breaks down the science of sports,1 suggests that MLB baseballs used after the 2015 All-Star Game were subtly but consistently different than older baseballs. The research, performed by the Keck School of Medicine at the University of Southern California and Kent State University’s Department of Chemistry and Biochemistry, reveals changes in the density and chemical composition of the baseball’s core — and provides our first glimpse inside the newer baseballs.

Looking inside the balls and testing their chemical composition revealed that the cores of recent balls were somewhat less dense than the cores of balls used before the 2015 All-Star Game. The newer cores weigh about a half a gram less than the older ones, which might be enough to cause baseballs hit on a typical home run trajectory to fly about 6 inches farther. That alone is hardly enough to explain the home run surge of recent seasons, but when combined with previousresearchfinding that baseballs began to change in other small ways starting around the same time, it suggests that a number of minor differences may have combined to contribute to the remarkable upswing in home run power we’ve witnessed since 2015.

Asked about these findings, MLB noted that it had commissioned a group of scientists and statisticians to investigate any changes to the ball, and that the committee would issue a report on its research soon. According to Alan Nathan, one of the physicists on the commission, the task force found that all the characteristics that MLB regularly measures, including the weight, circumference, seam height and bounciness of the ball, were within ranges that meant variations in the baseballs were unlikely to significantly affect home run rates. MLB declined to provide the data supporting these assertions.

Independent investigations byFiveThirtyEight, publications like The Ringer, and Nathan himself have shown differences in the characteristics of the ball and the way it performs. Research has shown that balls used in games after the 2015 All-Star Game were bouncier and less air resistant compared with baseballs from the 2014 season, when players hit a relatively modest 4,186 homers, the fewest since 1995. (Nathan noted that MLB does not regularly measure air resistance.) Taken together, these changes would result in a ball that would come off the bat at a higher speed and carry farther. While investigations have been able to show that the baseball behaves differently in recent years, no one had looked inside the ball to see if there was evidence of changes to the way the baseball is constructed.

So far, these investigations have primarily looked at the exterior of the baseball.Unassuming photo of a baseball

Broadly, MLB baseballs — which are produced by Rawlings in Costa Rica— are made of three components: an exterior shell of cowhide, a winding of several layers of yarn, and a core of rubber-coated cork, also known as a “pill.”Cross section of a baseball

To analyze possible changes to the inside of the ball, particularly the core, “ESPN Sport Science” purchased one new ball from Rawlings and seven game-used baseballs from eBay, confirming their authenticity through MLB’s authenticator program.2

The eight baseballs we tested were split into two groups: an “old group” of four balls used in games played between August 2014 and May 2015, and a “new group” of three balls used in games played between August 2016 and July 2017, plus the brand-new ball. The aim was to see if the internal composition of the baseballs had changed in ways that would affect the ball’s performance.3

The balls were first analyzed by Dr. Meng Law, Dr. Jay Acharya and Darryl Hwang at the Keck School of Medicine at USC using a computerized tomography, or CT, scan. This test is typically used to look inside a human head or body, but in this case, it allowed Dr. Law’s team to examine the interior of the baseballs without cracking them open and destroying them.CT scans of four baseballs from 2014 -- 2015 and four from 2016 -- 2017

Initial CT imaging showed that baseballs in the same group had a negligible variation in internal properties.Closer look at 2014 -- 2015 baseball CT scans

When comparing the new and old groups, however, there was a clear difference in the density of the core.Comparison of CT scans from a 2014 -- 2015 baseball to its 2016 -- 2017 counterpart

In an MLB baseball, the core consists of four parts: a cork pellet at the center, surrounded by a layer of black rubber held together by a rubber ring where the halves meet, all of which is then molded together in a layer of pink rubber.Labels on a cross section of the CT scan

Dr. Law’s team isolated the density difference to the outer (pink) layer of the core, which was, on average, about 40 percent less dense in the new group of balls.Scale showing the density of 2014 -- 2015 baseball CT scans vs. 2016 -- 2017 scans, where the newer ball is less dense

While other parts of the ball showed slight differences in density and volume, none were as noteworthy as the changes to the core.

It’s not just that the inside of the ball looks different — the chemical composition of the cores appears to have changed as well. After being tested at the Keck School, the same set of balls were sent to Kent State University. There, researchers at Soumitra Basu’s lab in the Chemistry and Biochemistry department cut open the balls to examine the cores using a thermogravimetric analysis (TGA). This test essentially cooks a material to see which parts parts of it vaporize at which temperatures. Using that information, researchers can create a molecular profile of a given material.

This test showed that the pink layer of the core in baseballs from the new group was, on average, composed of about 7 percent more polymer than the same area in baseballs from the old group. Additionally, an analysis with a scanning electron microscope showed that the same layer in the new balls contained, on average, 10 percent less silicon, relative to the amount of other ingredients in the pill. According to the Kent State researchers, these chemical changes produced a more porous, less dense layer of rubber — which explains the results seen in the CT scan at the Keck School.

It may not seem obvious, but these slight changes in the chemical composition of the core could have an impact on how the balls played once they were sewn up and shipped to major league teams. Less dense cores could mean lighter baseballs. The cores of the new balls weighed, on average, about 0.5 grams less than the cores from the old group. This difference was statistically significant, which means it’s highly unlikely that it was due to sampling error. The overall weight of the balls also dropped by an average of about a 0.5 grams between groups, but, unlike with the cores, this difference was not statistically significant.4

Half a gram isn’t much — it’s only about the weight of a paperclip. A tiny change like this might add only about 6 inches to flight of a baseball hit on a typical home run trajectory, according to Nathan’s calculations. But the timing of these changes to the weight and density of the core coincides with a much larger boost to the bounciness of the baseball. According to a previous analysis performed by The Ringer, that increase in bounciness alone would add around 0.6 mph to the speed of the ball as it leaves the bat and add roughly 3 feet to the travel distance of a fly ball — enough to make the difference between the warning track and the stands.

On top of the fact that the balls became bouncier as the core itself changed, previous research at FiveThirtyEight showed that they also became less air resistant. The decrease in drag is probably a result of a smaller, slicker baseball with lower seams. The change in air resistance could add an additional 5 feet to the travel distance of a fly ball. Combine all these factors together — a lighter, more compact baseball with tighter seams and more bounce — and the ball could fly as much as 8.6 feet farther. According to Nathan’s calculations, this would lead to a more than 25 percent increase in the number of home runs. Asked whether these changes in combination could have significantly affected the home run rate, MLB declined to comment.

In actuality, home runs spiked by about 46 percent between 2014 and 2017, which means that the changes to the ball could account for more than half of the increase. The remainder could be reasonably chalked up to a philosophical shift among MLB hitters, who are likely swinging upward to maximize the number of balls they hit in the air and are not shy about the increase in strikeouts that may come with that approach.

MLB Commissioner Rob Manfred has repeatedlydenied that the baseball is juiced. On numerous occasions, he has said league testing found that baseballs continue to fall within the range that MLB designates as acceptable, and he recently said that MLB testing showed the balls to be fundamentally the same. But even if the baseballs still meet the league’s manufacturing guidelines, their performance could change enough to double (or, theoretically, halve) the number of home runs hit in a year.

In fact, in January of 2015, Rawlings filed a patent application for a manufacturing process that would allow it to produce softballs and non-MLB baseballs5 that were as bouncy as possible while still falling within the manufacturing specifications set by the league. This type of ball is constructed quite differently from MLB baseballs, so there’s no indication that this patent means Rawlings is deliberately manipulating major league baseballs in this way, but it demonstrates that it’s at least theoretically possible for balls to be “fundamentally the same” while also performing differently than they have in the past.

Kathy Smith-Stephens, senior director of quality and compliance at Rawlings, said that no change had been made to the baseballs but that “natural variation” occurs in the manufacturing process. She noted that they “continuously tweak” — though later in the interview she asked that we say “continuously refine” — the manufacturing process in an effort to reduce variations, but said that Rawlings’ internal testing had shown no difference in the ball’s weight or bounciness.

Evidence that the baseball is at least partially responsible for the last few years’ spike in the home run rate mounted throughout the summer of 2017 and reached a peak during October’s World Series. In those seven games, the Houston Astros and Los Angeles Dodgers smashed 24 homers, including eight in one game. In the wake of this power display, Manfred asked all 30 teams to start storing baseballs in a climate-controlled room and commissioned a task force of scientists and statisticians to investigate whether the ball was juiced in 2017. Our own research, combined with controlled tests from three separate academic laboratories, strongly suggests that the physical properties of the ball have changed. Taken together, all these studies give us a lot of evidence to suggest that today’s baseballs differ in meaningful ways from those of a few years ago. In other words, there are many questions for Manfred’s committee to address.

Special thanks to Sean O’Rourke, Dr. Cynthia Bir and Nathan Beals for additional research assistance.

Amazon won't sell you a Chromecast, but they will sell a counterfeit

0
0

Fulfillment by Amazon (FBA) is a service we offer sellers that lets them store their products in Amazon's fulfillment centers, and we directly pack, ship, and provide customer service for these products. Something we hope you'll especially enjoy: FBA items qualify for FREE Shipping and Amazon Prime.

If you're a seller, Fulfillment by Amazon can help you increase your sales. We invite you to learn more about Fulfillment by Amazon.


Ethereum fixes serious “eclipse” flaw that could be exploited by any kid

0
0

Developers of Ethereum, the world's No. 2 digital currency by market capitalization, have closed a serious security hole that allowed virtually anyone with an Internet connection to manipulate individual users' access to the publicly accessible ledger.

So-called eclipse attacks work by preventing a cryptocurrency user from connecting to honest peers. Attacker-controlled peers then feed the target a manipulated version of the blockchain the entire currency community relies on to reconcile transactions and enforce contractual obligations. Eclipse attacks can be used to trick targets into paying for a good or service more than once and to co-opt the target's computing power to manipulate algorithms that establish crucial user consensus. Because Ethereum supports "smart contracts" that automatically execute transactions when certain conditions in the blockchain are present, Ethereum eclipse attacks can also be used to interfere with those self-enforcing agreements.

Like most cryptocurrencies, Ethereum uses a peer-to-peer mechanism that compiles input from individual users into an authoritative blockchain. In 2015 and again in 2016, separate research teams devised eclipse attacks against Bitcoin that exploited P2P weaknesses. Both were relatively hard to pull off. The 2015 attack required a botnet or a small ISP that controlled thousands of devices, while the 2016 attack relied on the control of huge chunks of Internet addresses through a technique known as border gateway protocol hijacking. The demands made it likely that both attacks could be carried out only by sophisticated and well-resourced hackers.

Attention script kiddies

Many researchers believed that the resources necessary for a successful eclipse attack against Ethereum would considerably higher than the Bitcoin attacks. After all, Ethereum's P2P network includes a robust mechanism for cryptographically authenticating messages and by default peers establish 13 outgoing connections, compared with eight for Bitcoin. Now, some of the same researchers who devised the 2015 Bitcoin attack are back to set the record straight. In a paper published Thursday, they wrote:

We demonstrate that the conventional wisdom is false. We present new eclipse attacks showing that, prior to the disclosure of this work in January 2018, Ethereum's peer-to-peer network was significantly less secure than that of Bitcoin. Our eclipse attackers need only control two machines, each with only a single IP address. The attacks are off-path-the attacker controls endhosts only and does not occupy a privileged position between the victim and the rest of the Ethereum network. By contrast, the best known off-path eclipse attacks on Bitcoin require the attacker to control hundreds of host machines, each with a distinct IP address. For most Internet users, it is far from trivial to obtain hundreds (or thousands) of IP addresses. This is why the Bitcoin eclipse attacker envisioned [in the 2015 research] was a full-fledged botnet or Internet Service Provider, while the BGP-hijacker Bitcoin eclipse attacker envisioned [in the 2016 paper] needed access to a BGP-speaking core Internet router. By contrast, our attacks can be run by any kid with a machine and a script.

Raising the bar

In January, the researchers reported their findings to Ethereum developers. They responded by making changes to geth, the most popular application supporting the Ethereum protocol. Ethereum users who rely on geth should ensure they've installed version 1.8 or higher. The researchers didn't attempt the same attacks against other Ethereum clients. In an email, Ethereum developer Felix Lange wrote:

"We have done our best to mitigate the attacks within the limits of the protocol. The paper is concerned with 'low-resource' eclipse attacks. As far as we know, the bar has been raised high enough that eclipse attacks are not feasible without more substantial resources, with the patches that have been implemented in geth v1.8.0." Lange went on to say he didn't believe another popular Ethereum app called Parity is vulnerable to the same attacks.

The paper, titled Low-Resource Eclipse Attacks on Ethereum's Peer-to-Peer Network, described two separate attacks. The simplest one relied on two IP addresses, which each generate large numbers of cryptographic keys that the Ethereum protocol uses to designate peer-to-peer nodes. The attacker then waits for a target to reboot the computer, either in the due course of time, or after the hacker sends various malicious packets that cause a system crash. As the target is rejoining the Ethereum network, the attacker uses the pool of nodes to establish incoming connections before the target can establish any outgoing ones.

The second technique works by creating a large number of attacker-controlled nodes and sending a special packet that effectively poisons the target's database with the fraudulent nodes. When the target reboots, all of the peers it connects to will belong to the attacker. In both cases, once the target is isolated from legitimate nodes, the attacker can present a false version of the blockchain. With no peers challenging that version, the target will assume the manipulated version is the official blockchain.

It's about time

The researchers presented a third technique that makes eclipse attacks easier to carry out. In a nutshell, it works by setting the target's computer clock 20 or more seconds ahead of the other nodes in the Ethereum network. To prevent so-called replay attacks—in which a hacker resends an old authenticated message in an attempt to get it executed more than once—the Ethereum protocol rejects messages that are more than 20 seconds old. By setting a target's clock ahead, attackers can cause the target to lose touch with all legitimate users. The attackers use malicious nodes with the same clock time to connect to the target. Some of the same researchers behind the Ethereum eclipse technique described a variety of timing attacks in a separate paper published in 2015.

Ethereum developers put a countermeasure in place against the first attack that ensures each node will always make outgoing connections to other peers. The fix for the second attack involved limiting the number of outgoing connections a target can make to the same /24 chunk of IP address to 10. The changes are designed to make it significantly harder to completely isolate a user from other legitimate users. When even a single node presents users with a different version of the blockchain, they will be warned of an error that effectively defeats the attack.

Ethereum developers haven't implemented a fix for the time-based attack. Since it generally requires an attacker to manipulate traffic over the target's Internet connection or to exploit non-Ethereum vulnerabilities on the target's computer, it likely poses less of a threat than the other two attacks.

The researchers, from Boston University and the University of Pittsburgh, warned users to protect themselves against the eclipse threat.

"Given the increasing importance of Ethereum to the global blockchain ecosystem, we think it's imperative that countermeasures preventing them be adopted as soon as possible," they wrote. "Ethereum node operators should immediately upgrade to geth v1.8."

GDPR – A Practical Guide for Developers

0
0

You’ve probably heard about GDPR. The new European data protection regulation that applies practically to everyone. Especially if you are working in a big company, it’s most likely that there’s already a process for getting your systems in compliance with the regulation.

The regulation is basically a law that must be followed in all European countries (but also applies to non-EU companies that have users in the EU). In this particular case, it applies to companies that are not registered in Europe, but are having European customers. So that’s most companies. I will not go into yet another “12 facts about GDPR” or “7 myths about GDPR” posts/whitepapers, as they are often aimed at managers or legal people. Instead, I’ll focus on what GDPR means for developers.

Why am I qualified to do that? A few reasons – I was advisor to the deputy prime minister of a EU country, and because of that I’ve been both exposed and myself wrote some legislation. I’m familiar with the “legalese” and how the regulatory framework operates in general. I’m also a privacy advocate and I’ve been writing about GDPR-related stuff in the past, i.e. “before it was cool” (protecting sensitive data, the right to be forgotten). And finally, I’m currently working on a project that (among other things) aims to help with covering some GDPR aspects.

I’ll try to be a bit more comprehensive this time and cover as many aspects of the regulation that concern developers as I can. And while developers will mostly be concerned about how the systems they are working on have to change, it’s not unlikely that a less informed manager storms in in late spring, realizing GDPR is going to be in force tomorrow, asking “what should we do to get our system/website compliant”.

The rights of the user/client (referred to as “data subject” in the regulation) that I think are relevant for developers are: the right to erasure (the right to be forgotten/deleted from the system), right to restriction of processing (you still keep the data, but mark it as “restricted” and don’t touch it without further consent by the user), the right to data portability (the ability to export one’s data in a machine-readable format), the right to rectification (the ability to get personal data fixed), the right to be informed (getting human-readable information, rather than long terms and conditions), the right of access (the user should be able to see all the data you have about them).

Additionally, the relevant basic principles are: data minimization (one should not collect more data than necessary), integrity and confidentiality (all security measures to protect data that you can think of + measures to guarantee that the data has not been inappropriately modified).

Even further, the regulation requires certain processes to be in place within an organization (of more than 250 employees or if a significant amount of data is processed), and those include keeping a record of all types of processing activities carried out, including transfers to processors (3rd parties), which includes cloud service providers. None of the other requirements of the regulation have an exception depending on the organization size, so “I’m small, GDPR does not concern me” is a myth.

It is important to know what “personal data” is. Basically, it’s every piece of data that can be used to uniquely identify a person or data that is about an already identified person. It’s data that the user has explicitly provided, but also data that you have collected about them from either 3rd parties or based on their activities on the site (what they’ve been looking at, what they’ve purchased, etc.)

Having said that, I’ll list a number of features that will have to be implemented and some hints on how to do that, followed by some do’s and don’t’s.

  • “Forget me”– you should have a method that takes a userId and deletes all personal data about that user (in case they have been collected on the basis of consent or based on the legitimate interests of the controller (see more below), and not due to contract enforcement or legal obligation). It is actually useful for integration tests to have that feature (to cleanup after the test), but it may be hard to implement depending on the data model. In a regular data model, deleting a record may be easy, but some foreign keys may be violated. That means you have two options – either make sure you allow nullable foreign keys (for example an order usually has a reference to the user that made it, but when the user requests his data be deleted, you can set the userId to null), or make sure you delete all related data (e.g. via cascades). This may not be desirable, e.g. if the order is used to track available quantities or for accounting purposes. It’s a bit trickier for event-sourcing data models, or in extreme cases, ones that include some sort of blockchain/hash chain/tamper-evident data structure. With event sourcing you should be able to remove a past event and re-generate intermediate snapshots. For blockchain-like structures – be careful what you put in there and avoid putting personal data of users. There is an option to use a chameleon hash function, but that’s suboptimal. Overall, you must constantly think of how you can delete the personal data. And “our data model doesn’t allow it” isn’t an excuse. What about backups? Ideally, you should keep a separate table of forgotten user IDs, so that each time you restore a backup, you re-forget the forgotten users. This means the table should be in a separate database or have a separate backup/restore process.
  • Notify 3rd parties for erasure– deleting things from your system may be one thing, but you are also obligated to inform all third parties that you have pushed that data to. So if you have sent personal data to, say, Salesforce, Hubspot, twitter, or any cloud service provider, you should call an API of theirs that allows for the deletion of personal data. If you are such a provider, obviously, your “forget me” endpoint should be exposed. Calling the 3rd party APIs to remove data is not the full story, though. You also have to make sure the information does not appear in search results. Now, that’s tricky, as Google doesn’t have an API for removal, only a manual process. Fortunately, it’s only about public profile pages that are crawlable by Google (and other search engines, okay…), but you still have to take measures. Ideally, you should make the personal data page return a 404 HTTP status, so that it can be removed.
  • Restrict processing– in your admin panel where there’s a list of users, there should be a button “restrict processing”. The user settings page should also have that button. When clicked (after reading the appropriate information), it should mark the profile as restricted. That means it should no longer be visible to the backoffice staff, or publicly. You can implement that with a simple “restricted” flag in the users table and a few if-clasues here and there.
  • Export data– there should be another button – “export data”. When clicked, the user should receive all the data that you hold about them. What exactly is that data – depends on the particular usecase. Usually it’s at least the data that you delete with the “forget me” functionality, but may include additional data (e.g. the orders the user has made may not be delete, but should be included in the dump). The structure of the dump is not strictly defined, but my recommendation would be to reuse schema.org definitions as much as possible, for either JSON or XML. If the data is simple enough, a CSV/XLS export would also be fine. Sometimes data export can take a long time, so the button can trigger a background process, which would then notify the user via email when his data is ready (twitter, for example, does that already – you can request all your tweets and you get them after a while). You don’t need to implement an automated export, although it would be nice. It’s sufficient to have a process in place to allow users to request their data, which can be a manual database-querying process.
  • Allow users to edit their profile– this seems an obvious rule, but it isn’t always followed. Users must be able to fix all data about them, including data that you have collected from other sources (e.g. using a “login with facebook” you may have fetched their name and address). Rule of thumb – all the fields in your “users” table should be editable via the UI. Technically, rectification can be done via a manual support process, but that’s normally more expensive for a business than just having the form to do it. There is one other scenario, however, when you’ve obtained the data from other sources (i.e. the user hasn’t provided their details to you directly). In that case there should still be a page where they can identify somehow (via email and/or sms confirmation) and get access to the data about them.
  • Consent checkboxes– “I accept the terms and conditions” would no longer be sufficient to claim that the user has given their consent for processing their data. So, for each particular processing activity there should be a separate checkbox on the registration (or user profile) screen. You should keep these consent checkboxes in separate columns in the database, and let the users withdraw their consent (by unchecking these checkboxes from their profile page – see the previous point). Ideally, these checkboxes should come directly from the register of processing activities (if you keep one). Note that the checkboxes should not be preselected, as this does not count as “consent”. Another important thing here is machine learning/AI. If you are going to use the user’s data to train your ML models, you should get consent for that as well (unless it’s for scientific purposes, which have special treatment in the regulation). Note here the so called “legitimate interest”. It is for the legal team to decide what a legitimate interest is, but direct marketing is included in that category, as well as any common sense processing relating to the business activity – e.g. if you collect addresses for shipping, it’s obviously a legitimate interest. So not all processing activities need consent checkboxes.
  • Re-request consent– if the consent users have given was not clear (e.g. if they simply agreed to terms & conditions), you’d have to re-obtain that consent. So prepare a functionality for mass-emailing your users to ask them to go to their profile page and check all the checkboxes for the personal data processing activities that you have.
  • “See all my data”– this is very similar to the “Export” button, except data should be displayed in the regular UI of the application rather than an XML/JSON format. I wouldn’t say this is mandatory, and you can leave it as a “desirable” feature – for example, Google Maps shows you your location history – all the places that you’ve been to. It is a good implementation of the right to access. (Though Google is very far from perfect when privacy is concerned). This is not all about the right to access – you have to let unregistered users ask whether you have data about them, but that would be a more manual process. The ideal minimum would be to have a feature “check by email”, where you check if you have data about a particular email. You also need to tell the user in what ways you are processing their data. You can simply print all the records in your data process register for which the user has consented to.
  • Age checks– you should ask for the user’s age, and if the user is a child (below 16), you should ask for parent permission. There’s no clear way how to do that, but my suggestion is to introduce a flow, where the child should specify the email of a parent, who can then confirm. Obviously, children will just cheat with their birthdate, or provide a fake parent email, but you will most likely have done your job according to the regulation (this is one of the “wishful thinking” aspects of the regulation).
  • Keeping data for no longer than necessary– if you’ve collected the data for a specific purpose (e.g. shipping a product), you have to delete it/anonymize it as soon as you don’t need it. Many e-commerce sites offer “purchase without registration”, in which case the consent goes only for the particular order. So you need a scheduled job/cron to periodically go through the data and anonymize it (delete names and addresses), but only after a certain condition is met – e.g. the product is confirmed as delivered. You can have a database field for storing the deadline after which the data should be gone, and that deadline can be extended in case of a delivery problem.

Now some “do’s”, which are mostly about the technical measures needed to protect personal data (outlined in article 32). They may be more “ops” than “dev”, but often the application also has to be extended to support them. I’ve listed most of what I could think of in a previous post. An important note here is that this is not mandated by the regulation, but it’s a good practice anyway and helps with protecting personal data.

  • Encrypt the data in transit. That means that communication between your application layer and your database (or your message queue, or whatever component you have) should be over TLS. The certificates could be self-signed (and possibly pinned), or you could have an internal CA. Different databases have different configurations, just google “X encrypted connections. Some databases need gossiping among the nodes – that should also be configured to use encryption
  • Encrypt the data at rest– this again depends on the database (some offer table-level encryption), but can also be done on machine-level. E.g. using LUKS. The private key can be stored in your infrastructure, or in some cloud service like AWS KMS.
  • Encrypt your backups– kind of obvious
  • Implement pseudonymisation– the most obvious use-case is when you want to use production data for the test/staging servers. You should change the personal data to some “pseudonym”, so that the people cannot be identified. When you push data for machine learning purposes (to third parties or not), you can also do that. Technically, that could mean that your User object can have a “pseudonymize” method which applies hash+salt/bcrypt/PBKDF2 for some of the data that can be used to identify a person. Pseudonyms could be reversible or not, depending on the usecase (the definition in the regulation implies reversibility based on a secret information, but in the case of test/staging data it might not be). Some databases have such features built-in, e.g. Orale.
  • Protect data integrity– this is a very broad thing, and could simply mean “have authentication mechanisms for modifying data”. But you can do something more, even as simple as a checksum, or a more complicated solution (like the one I’m working on). It depends on the stakes, on the way data is accessed, on the particular system, etc. The checksum can be in the form of a hash of all the data in a given database record, which should be updated each time the record is updated through the application. It isn’t a strong guarantee, but it is at least something.
  • Have your GDPR register of processing activities in something other than ExcelArticle 30 says that you should keep a record of all the types of activities that you use personal data for. That sounds like bureaucracy, but it may be useful – you will be able to link certain aspects of your application with that register (e.g. the consent checkboxes, or your audit trail records). It wouldn’t take much time to implement a simple register, but the business requirements for that should come from whoever is responsible for the GDPR compliance. But you can advise them that having it in Excel won’t make it easy for you as a developer (imagine having to fetch the excel file internally, so that you can parse it and implement a feature). Such a register could be a microservice/small application deployed separately in your infrastructure.
  • Log access to personal data– every read operation on a personal data record should be logged, so that you know who accessed what and for what purpose. This does not follow directly from the provisions of the regulation, but it is kinda implied from the accountability principles. What about search results (or lists) that contain personal data about multiple subjects? My hunch is that simply logging “user X did a search for criteria Y” would suffice. But don’t display too many personal data in lists – for example see how facebook makes you go through some hoops to get a person’s birthday. Note: some have treated article 30 as a requirement to keep an audit log. I don’t think it is saying that – instead it requires 250+ companies (or companies processing data regularly) to keep a register of the types of processing activities (i.e. what you use the data for). There are other articles in the regulation that imply that keeping an audit log is a best practice (for protecting the integrity of the data as well as to make sure it hasn’t been processed without a valid reason)
  • Register all API consumers– you shouldn’t allow anonymous API access to personal data. I’d say you should request the organization name and contact person for each API user upon registration, and add those to the data processing register.

Finally, some “don’t’s”.

  • Don’t use data for purposes that the user hasn’t agreed with– that’s supposed to be the spirit of the regulation. If you want to expose a new API to a new type of clients, or you want to use the data for some machine learning, or you decide to add ads to your site based on users’ behaviour, or sell your database to a 3rd party – think twice. I would imagine your register of processing activities could have a button to send notification emails to users to ask them for permission when a new processing activity is added (or if you use a 3rd party register, it should probably give you an API). So upon adding a new processing activity (and adding that to your register), mass email all users from whom you’d like consent. Note here that additional legitimate interests of the controller might be added dynamically.
  • Don’t log personal data– getting rid of the personal data from log files (especially if they are shipped to a 3rd party service) can be tedious or even impossible. So log just identifiers if needed. And make sure old logs files are cleaned up, just in case
  • Don’t put fields on the registration/profile form that you don’t need– it’s always tempting to just throw as many fields as the usability person/designer agrees on, but unless you absolutely need the data for delivering your service, you shouldn’t collect it. Names you should probably always collect, but unless you are delivering something, a home address or phone is unnecessary.
  • Don’t assume 3rd parties are compliant– you are responsible if there’s a data breach in one of the 3rd parties (e.g. “processors”) to which you send personal data. So before you send data via an API to another service, make sure they have at least a basic level of data protection. If they don’t, raise a flag with management.
  • Don’t assume having ISO XXX makes you compliant– information security standards and even personal data standards are a good start and they will probably 70% of what the regulation requires, but they are not sufficient – most of the things listed above are not covered in any of those standards

Overall, the purpose of the regulation is to make you take conscious decisions when processing personal data. It imposes best practices in a legal way. If you follow the above advice and design your data model, storage, data flow , API calls with data protection in mind, then you shouldn’t worry about the huge fines that the regulation prescribes – they are for extreme cases, like Equifax for example. Regulators (data protection authorities) will most likely have some checklists into which you’d have to somehow fit, but if you follow best practices, that shouldn’t be an issue.

I think all of the above features can be implemented in a few weeks by a small team. Be suspicious when a big vendor offers you a generic plug-and-play “GDPR compliance” solution. GDPR is not just about the technical aspects listed above – it does have organizational/process implications. But also be suspicious if a consultant claims GDPR is complicated. It’s not – it relies on a few basic principles that are in fact best practices anyway. Just don’t ignore them.

Satan Comes to Dinner (1997)

0
0
Satan Comes to Dinner
in E
©1997 Electric Communities
All Rights Reserved
A solution to Dijkstra's Dining Philosophers Problem which provides for deadlock avoidance and fairness. It does not require that the philosopher classes be trusted. Satan is invited to supply a class. An example of the failure of class-signing. The classes are written in E, an extendedJava subset that provides message passing, orthogonal persistence, and capability-based security. The Dining Philosophers Problem is one of the standard exercises in the teaching of concurrent programming. It is instructive in the design of things like operating systems and distributed systems. It is an interesting problem because it introduces management of scarce, shared resources. It is tricky because most naive implementations will result in deadlock. The statement of the problem usually goes like this:
There is a group of philosophers (usually 5) who eat together at a round table. There are forks placed between the philosophers. Philosophers spend their time either thinking or eating. In order to eat, a philosopher must pick up exactly two forks, one on his immediate left, and the other on his immediate right. When he is done eating, he will put his forks down so that his neighbors may use them, and he thinks again.
The Dining Philosophers Problem (aka The Dining Quintuple Problem) was designed in 1965 by Edsger W. Dijkstra to demonstrate the horror that is deadlock. In some versions of the problem, the forks are replaced with chopsticks. This change does not substantively alter the problem, although it can simplify the graphics. Some versions have the philosophers eating spaghetti or rice. In this version, they are dining on shrimp.

The programming problem is to construct a simulation which will allow philosophers to move between their eating and thinking states while properly controlling the forks.

A typical solution has each philosopher doing something like this:

    forever {
        busy thinking
        wait for left fork
        wait for right fork
        busy eating
        drop left fork
        drop right fork
    }
The busy statements will consume time. The length of time is often random. The wait statements are often implemented using semaphores or other mutual exclusion gadgets, causing the program to block until the fork becomes available.

If every philosopher picks up his left fork and then waits for the right fork to become available, then the system will deadlock. Computation cannot go forward because the philosophers will never get their second forks. The fact that the busy periods are of random duration may reduce the occurrence of deadlock, but it does not prevent it. Occasional deadlock is actually a bigger problem, because it is much more difficult to replicate, diagnose, and fix. Deadlock is one of the greatest programming hazards in concurrent systems.

Some solutions modify some of the philosophers, so that they always pick up their right forks first. This does avoid deadlock, but it is a special solution. It cannot be generalized to avoid deadlock in other situations. Other implementations introduce waiters, rooms, chairs, tokens, and other objects to help manage the deadlock problem. These solutions are complex, and also not sufficiently generalizable.

Most solutions depend on the philosophers behaving correctly. For example, if a philosopher picks up his forks and never puts them down, the philosophers sitting beside him will die of starvation. If either happens to be holding a fork (which is likely), then all of the other philosophers will starve, too.

So we will redefine the problem. We still want to avoid deadlock, but we also want to be able to invite anyone to code their own philosopher class, and to have it run fairly with others.

Suppose that you invite Satan to contribute one of the philosopher classes. How can you be confident that his code will not contaminate the shared environment, will not cause starvation among the other philosophers, and will not spoil everyone else's fun? Even more important, how can you be sure that Satan's code will not run amok and drain your bank account, reveal your private key, and trash your hard disk?

It is not sufficient to simply stuff Satan's code into a sandbox. He is not giving you an applet. He is giving you a class which needs to work cooperatively, even under the cloud of mutual suspicion, with other objects.

The goal is not to keep Satan out. You want to invite Satan in, but without compromising your security. You don't want to choose between safety and the benefits of openness. You want both.

Satan offers to prove to you that his class is safe by presenting an Authenticode certificate which will certify that Satan signed the class. You could verify that the signature is truly Satan's, but proof that it came from Satan is not proof that it is safe.

Satan, pretending to be offended by your skepticism, asks "Whose signature do you trust? Just say the name, and I will have them sign my class. Would you like Microsoft to sign it? There's a guy over there who owes me a favor."

You come to the conclusion that a signature cannot by itself transform a suspect class into a fully trusted class, so you decide that you need a more credible security mechanism.

That mechanism is capabilities. A capability is the unforgable, transferable, non-revocable right to communicate with an object. We will create a framework which will support anyone's Philosopher code, certified or not. By relying on good capability-based design, rather than certification, we have less overhead and a better system.
(fig. 1)

In fig. 1 we see an example of a capability relationship. The ovals represent objects. The arrow indicates an object reference. Object C has a reference to Object A, and therefore Object C has the capability to communicate with Object A. An object can receive a reference to an object by having the reference passed to it, or by instantiating a new object.

Assuming that Object C is not trusted, we cannot be certain that it will destroy its reference to Object A when requested to. Capabilities are not directly revokable. But they can be indirectly revocable, as can be seen in fig. 2:

(fig. 2)

Object C is given a reference to Object T. Normally, Object T will pass Object C's messages on to Object A. Object B can send a message to Object T which will cause Object T to destroy its own reference to Object A. Object C will be left with a reference to the now useless Object T. Object C's capability to communicate with Object A has been revoked. Object B doesn't need to trust Object C if it can trust Object T.

We will now introduce a new object to the Dining Philosophers: The Fork Dispenser. It is not safe to share forks, so the Fork Dispenser will dispense new disposable forks. As fig. 3 shows, each philosopher must interact with two fork dispensers in order to interact with the plate of shrimp. Each fork dispenser is shared by two philosophers.
(fig. 3)

Each philosopher is assigned two fork dispensers. Philosophers do not have the capabilities necessary to interact directly with the other philosophers, the plates of shrimp, or with the system in general.

A philosopher sends requests to its fork dispensers, which respond by sending fork capabilities. The forks contain capabilities to the plate of shrimp, which the fork dispenser can revoke.

(fig. 4)

Joining the fork dispensers in the cast are forks, plates, and philosophers, using these interfaces:

public einterface ForkDispenser {
    emethod service (Object, Philosopher, Plate);
    emethod forkPlease (Philosopher);
    emethod forkReturn (Philosopher);
}
 
public einterface Fork {
    emethod shrimpPlease (int);
    emethod hereIsYourShrimp (int);
    emethod revoke ();
}
 
public einterface Plate {
    emethod shrimpPlease (Fork theFork, int serialNr);
}
 
public einterface Philosopher {
    emethod hereIsYourForkDispenser (ForkDispenser);
    emethod hereIsYourFork (Fork);
    emethod hereIsYourShrimp (int);
}
 
public eclass OurForkDispenser implements ForkDispenser {
    private Object       trusted = null;
    private Philosopher  nowServing = null;
    private Philosopher  firstPhilosopher = null;
    private Philosopher  secondPhilosopher = null;
    private Plate        firstPlate = null;
    private Plate        secondPlate = null;
    private Fork         theFork = null;
    private Timer        theTimer = new Timer();
    private int          timerSerialNr;
    private boolean      otherIsWaiting = false;
    
    public ForkDispenser (eobject trusted) {
        this.trusted = trusted;
    }
The service message transmits a philosopher and plate pair. The philosopher must never get a direct capability to the plate. This information allows the fork dispenser to determine which plate to use. The service message should only come from the trusted source that created the fork dispenser. Checking this prevents Satan from trying to confuse the fork dispenser by sending his own service messages.

The fork dispenser sends its own reference to the philosopher. This gives the philosopher the capability to interact with the fork dispenser.

    emethod service (Object theSource, Philosopher thePhilosopher, 
            Plate thePlate) {
        if (theSource == trusted) {
            thePhilosopher <- hereIsYourForkDispenser (this);
            if (firstPhilosopher == null) {
                firstPhilosopher = thePhilosopher;
                firstPlate = thePlate;
            } else {
                secondPhilosopher = thePhilosopher;
                secondPlate = thePlate;
            }
        }
    }
A philosopher requests a fork. Determine which plate he uses. If the requester is not associated with either plate, then ignore the request. (This will prevent Satan from pretending that he is both philosophers.)

If a fork is not in use, then dispense one. If the requester is the same as current fork holder, then remind him that he already has a fork. (This can occur in some recovery strategies.)

Otherwise, the situation is that a request was made for a fork while a fork is in use by another philosopher. Make a note that the other philosopher is waiting and set a watchdog timer which will start the revocation of the current fork in 10,000 milliseconds.

    emethod forkPlease (Philosopher who) {
        int thePlate;
        if (who == firstPhilosopher) {
            thePlate = firstPlate;
        } else if (who == secondPhilosopher) {
            thePlate = secondPlate;
        } else {
            return;
        }
        
        if (nowServing == null) {
            nowServing = who;
            theFork = new OurFork (nowServing, thePlate);
            nowServing <- hereIsYourFork (theFork);
        } else if (nowServing == who) {
            nowServing <- hereIsYourFork (theFork);
        } else {
            otherIsWaiting = true;
            timerSerialNr += 1;
            theTimer.in (10000, (invocation 
                    (this <- timeOut (theTimer, timerSerialNr)));
        }
    }
A philosopher returns a fork, allowing someone else to eat. The protocol does not require this kind of politeness, but it is nice, isn't it?

First, check that the person returning the fork is the recorded holder of the fork. (This prevents Satan from pretending to be the other philosopher.) Then revoke the fork. (This prevents Satan from holding on to the capability.) If the other philosopher was waiting, then issue him a new fork and cancel the timer.

    emethod forkReturn (Philosopher thePhilosopher) {
        if (nowServing == thePhilosopher) {
            theFork <- revoke ();
            if (otherIsWaiting) {
                timerSerialNr += 1;
                serveOther();
            } else {
                nowServing = null;
                theFork = null;
            }
        }
    }
The watchdog timer has expired. If the event was not canceled, then revoke the fork and issue a new one to the other philosopher. (All timer messages except the one with a serial number matching timerSerialNr have been canceled.)
    emethod timeOut (Timer theSource, int messageSerialNr) {
        if (theSource == theTimer &&
                messageSerialNr == timerSerialNr) {
            theFork <- revoke ();
            serveOther();
        }
    }
Issue a fork to the other philosopher. Because a fork dispenser serves two philosophers, each with its own plate, figure out which plate to use.
    private void serveOther () {
        if (nowServing == firstPhilosopher) {
            nowServing = secondPhilosopher;
            thePlate = secondPlate;
        } else {
            nowServing = firstPhilosopher;
            thePlate = firstPlate;
        }
        theFork = new OurFork(nowServing, thePlate);
        nowServing <- hereIsYourFork (theFork);
        otherIsWaiting = false;
    }
}
The ForkDispenser has three facets, or sets of interfaces:
Sender E Methods 
The trusted source service 
Timer timeOut 
Philosopher forkPlease
forkReturn 
In particular, OurForkDispenser does not want a Philosopher to be able to send the service or timeOut messages. This is accomplished by having the senders of these messages include the sender's reference as the first parameter, which OurForkDispenser can easily validate. Forks are disposable objects which give a philosopher indirect access to a plate of shrimp.
public eclass OurFork implements Fork {
    private int          previousSerialNr = Integer.MIN_VALUE;
    private Philosopher  thePhilosopher = null;
    private Plate        thePlate = null;
 
    public Fork (Philosopher thePhilosopher, Plate thePlate) {
        this.thePhilosopher = thePhilosopher;
        this.thePlate = thePlate;
    }
The philosopher requested a shrimp. To assist the plate in pairing the messages, the fork verifies that the message contains a serial number that is larger than the previous serial number. If the number is ok, then a message is sent to the plate. If not, disconnect from the plate. (This prevents the sending of consecutive messages with the same serial number.)

The fork will be unable to forward the message if its reference to the plate has been revoked.

(The serial number test will fail after 4 billion shrimp requests to a fork due to an int rollover. That could fixed by adding a rollover test here, or by changing serial numbers to long. A well formed philosopher will easily recover. It seems that this warning is requiring more effort than the fix. It is unlikely that the fix would ever be needed.)

    emethod shrimpPlease (int serialNr) {
        if (previousSerialNr < serialNr) {
            previousSerialNr = serialNr;
            if (thePlate != null) {
                thePlate <- shrimpPlease (this, serialNr);
            } else {
                thePhilosopher <- noShrimp();
            }
        } else {
            thePlate = null;
            thePhilosopher <- noShrimp();
        }
    }
A shrimp is delivered successfully from the plate of shrimp. If this fork has not been revoked, pass the message on to the philosopher. This indirection assures that the philosopher was holding two valid forks at the same time.
    emethod hereIsYourShrimp (int serialNr) {
        if (thePlate != null) {
            thePhilosopher <- hereIsYourShrimp (serialNr);
        } else {
            thePhilosopher <- noShrimp ();
        }
    }
Revoke the fork by making it useless. By erasing the fork's reference to the plate, it is no longer able to send the messages that make shrimp for the philosopher. This emethod is intended to be used by the fork dispenser.
    emethod revoke () {
        thePlate = null;
    }
}
 
public eclass OurPlate implements Plate {
    private int   firstSerialNr = Integer.MIN_VALUE;
    private Fork  firstFork = null;
Respond to a request for a shrimp. The philosopher sends a message through each of his forks.

An E object receives one message at a time. After the plate has seen two matching serial numbers, it sends a piece of shrimp. Because a fork is unable to send the same serial number twice in succession, the plate can determine that two forks were used. Further, by confirming that the ForkDispenser references do not match, forks from two different fork dispensers were used. We prove that the first fork is still valid by using it to return the shrimp.

    emethod shrimpPlease (Fork theFork, int serialNr) {
        if (firstSerialNr != serialNr) {
            firstSerialNr = serialNr;
            firstFork = theFork;
        } else {
            firstFork <- hereIsYourShrimp (serialNr);
            firstFork = null;
            firstSerialNr = Integer.MIN_VALUE;
        }
    }
}
It might seem that a hereIsYourShrimp message is not a suitable payoff. A better payoff would be to have the plate send a reference to a shrimp object. The Nice Philosopher puts down his forks and spends at least part of his time thinking.
public eclass NicePhilosopher implements Philosopher {
    private ForkDispenser  firstForkDispenser  = null;
    private ForkDispenser  secondForkDispenser = null;
    private Fork           firstFork  = null;
    private Fork           secondFork = null;
    private int            serialNr = Integer.MIN_VALUE;
    private int            timerSerialNr = 0;
    private Timer          theTimer = new Timer;
    private boolean        eatingMode = false;
A philosopher is not necessarily made by a fork dispenser, but it is initialized by a fork dispenser. (This avoids a race condition on starting up.) Once the fork dispensers arrive, leap into action and begin to think.
    emethod hereIsYourForkDispenser (ForkDispenser theForkDispenser) {
        if (firstForkDispenser == null) {
            firstForkDispenser = theForkDispenser;
        } else {
            secondForkDispenser = theForkDispenser;
            busyThinking();
        }
    }
To think, set up a timer to send startEating in 10 seconds.
    private void busyThinking() {
        eatingMode = false;
        timerSerialNr += 1;
        theTimer.in (10 * 1000, (invocation 
                (this <- startEating (timerSerialNr)));
    }
Request a fork from each of the fork dispensers. Set up a timer to stop eating in 20 seconds.
    emethod startEating () {
        firstFork  = null;
        secondFork = null;
        firstForkDispenser  <- forkPlease (this);
        secondForkDispenser <- forkPlease (this);
        serialNr = Integer.MIN_VALUE;
        timerSerialNr += 1;
        theTimer.in (20 * 1000, (invocation 
                (this <- stopEating (timerSerialNr)));
    }
Receive the forks. After both have arrived, serious eating can begin.
    emethod hereIsYourFork (Fork theFork) {
        if (firstFork == null) {
            firstFork = theFork;
        } else {
            secondFork = theFork;
            eatingMode = true;
            eat();
        }
    }
Eat by requesting shrimp from the forks. Dijkstra requires that two forks be used. A serial number is added to the request to help the plate pair up the messages.
    void eat() {
        serialNr += 1;
        firstFork  <- shrimpPlease (serialNr);
        secondFork <- shrimpPlease (serialNr);
    }
A shrimp is delivered successfully from the plate of shrimp. What to do next? Get more shrimp!
    emethod hereIsYourShrimp (int serialNr);
        if (eatingMode) {
            eat();
        }
    }
A noShrimp message will can occur if a fork was revoked. If that happens, cancel the timer and stop eating.
    emethod noShrimp () {
        timerSerialNr += 1;
        this <- stopEating();
    }
The time has come to put down the forks and stop eating. It isn't really necessary to put down the forks, because the fork dispenser will revoke them anyway. But returning the forks allows the other philosophers to begin eating a little sooner, so it is a polite thing to do. Revocation may already have occurred. That's ok.
    emethod stopEating () {
        firstForkDispenser  <- forkReturn (this);
        secondForkDispenser <- forkReturn (this);
        busyThinking();
    }            
}
This is Satan's own implementation of Philosopher. It is so constrained by capability security that it is not really very evil. About the worst you can say about it is that it is very impolite. It is an infernal eating machine. It never rests. It never sets down its forks. Even so, the fork dispensers can guarantee fairness (starvation avoidance). As nasty as Satan is, he cannot prevent the other philosophers from getting their turns.
public eclass EvilPhilosopher implements Philosopher {
    private ForkDispenser  firstForkDispenser = null;
    private ForkDispenser  secondForkDispenser = null;
    private Fork           firstFork = null;
    private Fork           secondFork = null;
    private int            serialNr = Integer.MIN_VALUE;
First collect two fork dispensers. Then begin to eat. Without the fork dispensers, he can't get plate-aware forks, and without those forks, he cannot get shrimp.
    emethod hereIsYourForkDispenser (ForkDispenser theForkDispenser) {
        if (firstForkDispenser == null) {
            firstForkDispenser = theForkDispenser;
        } else {
            secondForkDispenser = theForkDispenser;
            startEating();
        }
    }
Request a fork from each of the fork dispensers. It does no good to request two forks from a single dispenser, because the plate of shrimp can tell the difference.
    void startEating() {
        firstFork  = null;
        secondFork = null;
        firstForkDispenser  <- forkPlease (this);
        secondForkDispenser <- forkPlease (this);
        serialNr = Integer.MIN_VALUE;
    }
Receive the forks. After both have arrived, serious eating can begin.
    emethod hereIsYourFork (Fork theFork) {
        if (firstFork == null) {
            firstFork = theFork;
        } else {
            secondFork = theFork;
            eat();
        }
    }
Eat by requesting shrimp from the forks.
    void eat () {
        serialNr += 1;
        firstFork  <- shrimpPlease (serialNr);
        secondFork <- shrimpPlease (serialNr);
    }
This philosopher has no thinking mode, so eventually the noShrimp message will arrive when one of the fork dispensers revokes a fork. When that happens, startEating again.
    emethod noShrimp () {
        startEating();
    }
A shrimp is delivered successfully from the plate of shrimp. What to do next? More shrimp!
    emethod hereIsYourShrimp (int serialNr);
        eat();
    }
}
  This Main is sufficient to get everything running on a single machine. In the next chapter, we will distribute this over a number of machines.
public class Main {
    public final static int NR_DINERS = 5;
    public final static int NR_FORKS  = NR_DINERS;
 
    public static void main (String argv[ ]) {
        ForkDispenser  forkDispensers[ ] = new ForkDispenser[NR_FORKS];
        Philosopher    thePhilosopher = null;
        Plate          thePlate = null;
 
        for (int i = 0; i < NR_FORKS; i += 1) {
            forkDispensers [i] = new OurForkDispenser(this);
        }
 
        for (int i = 0; i < NR_DINERS; i += 1) {
            if (i == 0) {
                thePhilosopher = new EvilPhilosopher();
            } else {
                thePhilosopher = new NicePhilosopher();
            }
            thePlate = new OurPlate();
            forkDispensers [i]
                    <-  service (this, thePhilosopher, thePlate);
            forkDispensers [(i + 1) % NR_FORKS] 
                    <-  service (this, thePhilosopher, thePlate);
        }
    }
}
Fig. 5 shows the flow of messages as a philosopher engages in the main part of the dining protocol.
(fig. 5)
  1. The philosopher sends forkPlease requests to both fork dispensers.
  2. The fork dispenser creates a new fork object with capabilities to the philosopher and the plate of shrimp.
  3. The fork dispenser sends a capability to the fork in a hereIsYourFork message to the philosopher.
  4. The philosopher sends a shrimpPlease message to each fork.
  5. The fork determines that it has not been revoked, and that it has never seen the current serial number. If all is ok, it sends a shrimpPlease message containing a capability to itself to the plate of shrimp.
  6. The plate of shrimp will receive two messages. If the serial numbers on the messages match, then it knows that two different forks were used. It sends a hereIsYourShrimp message to the fork that sent the first of the two messages to prove that the first fork is still valid.
  7. If the fork has not been revoked, then send a hereIsYourShrimp message to the philosopher.
The fork dispenser is able to revoke a fork at any time by sending a revoke message. In this way, the fork dispenser can impose a scheduling policy which resembles time slicing. This assures fairness, in that all philosophers will get chances to eat.

The only capabilities granted to a philosopher are the fork dispensers and forks. This prevents Satan's philosopher class from doing anything harmful.

Java is vulnerable to Consumption of Resources Attacks, in which Satan's class attempts to consume all of the available memory or compute time. This can degrade performance, and in many cases cause your machine crash. This can be minimized if Satan is required to host this philosopher on his own machine. Then the worst thing he can do is attempt to flood the network with messages.

A later version of E will provide a remedy to Consumption of Resources Attacks.

Suppose that Satan is allowed to provide two philosophers. Can they collude to prevent the nice philosophers from eating?

If the philosophers are run in a machine which is not under Satan's control, then the answer is no. E's containment mechanism assures that the philosophers cannot communicate directly with each other. They can only communicate with fork dispensers (and the forks they dispense), which do not support collusion.

The answer is less clear if Satan is allowed to execute his philosopher code remotely. Because Satan can hack his own machine, he can locally break containment, allowing the evil philosophers to communicate with each other. Suppose there is a nice philosopher sitting between the two evil philosophers. By sharing information about the condition of their forks, they can play a synchronized game in which the one on the left puts his fork down then immediately requests another when the one on the right gets a fork. When it is issued, the one on the right puts down his fork and immediately requests another. In this way, they can prevent the philosopher in the middle from getting two forks at the same time.

A similar kind of collusion can be found in an online poker game. While we can have confidence that the cards are dealt fairly, we cannot be confident that some of the players will not collude. The cheaters may be communicating by telephone, or may even be a single person pretending to be multiple people. By colluding, they have more information about the distribution of cards and the state of the deck, and so may be able to shave the odds in their favor. This kind of collusion could occur without the hacking of software.

Similarly, collusion in the Dining Philosopher example could occur without the hacking of software. The conspirators are able to speak to each other, and so can say things like "I just received my fork. Put down yours and request a new one."

ForkDispenser can be modified to make such collusion less effective. If every dispenser used a different time out value, or random time out values, then the attack will fail at least occasionally, and starvation will be prevented.

Campione, Mary and Walrath, Kathy. Deadlock and the Dining Philosophers. <URL: http://www.javasoft.com/docs/books/tutorial/java/threads/deadlock.html> from The Java Tutorial: Object-Oriented Programming for the Internet. Addison-Wesley.

Englin, Jessica. Edsger W. Dijkstra. <URL: http://cda.mrs.umn.edu/~englinjm/EWD.html >

Electric Communities. Dicing with the Devil. <URL: http://www.communities.com/products/tools/e/techpapers/devil.html>

Electric Communities. Introduction To Capability Based Security. <URL: http://www.communities.com/company/papers/security/index.html>

Falk, Bennett. Table Manners. <URL: http://www.dnai.com/~bfalk/dining.html>

Feldman, M.B. Introduction to Concurrent Programming <URL: http://www.seas.gwu.edu/faculty/mfeldman/cs2-book/chap15.html> from Software Construction and Data Structures with Ada 95. Addison-Wesley.

Gloyer, Brian. The Dining Philosophers. <URL: http://www.javasoft.com/applets/contest/DiningPhilosophers/>

Hardy, Norm. Capability Theory. <URL: http://www.MediaCity.com/~norm/CapTheory/>

Herring, Charles. Implementing Discrete-Event Simulation in Java. <URL: http://csl.ncsa.uiuc.edu/~herring/java.html>

Holdsworth, D. Dijkstra's Dining Philosophers. <URL: http://www.leeds.ac.uk/ucs/people/DHoldsworth/diners/diners.html>

Miller, Mark. Taxonomy of Computer Security Problems. <URL: http://www.caplet.com/security/taxonomy/index.html>

McGregor, Tony. The Dining Philosophers Problem <URL: http://byerley.cs.waikato.ac.nz/~tonym/201/sync/node15.html> from 201 - Computer Systems.

Why Women Choose Differently at Work

0
0

If you were to predict the future on the basis of school achievement alone, the world would be a matriarchy.”

Susan Pinker, who wrote that sentence almost a decade ago in her book The Sexual Paradox: Extreme Men, Gifted Women, and the Real Gender Gap, could make the same claim today: A 2014 meta-analysis found that the “female advantage in school marks has remained stable in the data retrieved (from 1914 to 2011).”1 Given that there’s an overall female advantage in school—one largest in language courses, smallest in math courses, according to the paper—why isn’t there a corresponding advantage, in terms of pay and high-powered positions, for females later, at work?

The developmental psychologist’s answer hasn’t changed since she wrote The Sexual Paradox: The female advantage at school doesn’t translate to work partly because of the different career and lifestyle preferences men and women go on to develop, ones that aren’t entirely culturally determined. These preferences, Pinker says, are a key part of the story about why women and men follow different career trajectories and schedules. Out-and-out discrimination, she argues, is still a problem and a significant part of the story—just not the whole story. For this, The New York Times called Pinker’s book a “ringing salvo in the sex-difference wars” and Pinker, because of the psychological cast she gives to sex differences, a “difference” feminist. The Telegraph, meanwhile, found the book interesting (“much food for thought”) but its case unconvincing (“more journalistic than scientific”).

Nautilus recently caught up with Pinker so we could see for ourselves.

Susan Pinker

What does the most recent data on women’s preferences regarding work tell us?

Studies show that, among other differences, educated women’s career trajectories often have a different shape—based on their wider interests and a common desire to have flexible or part-time work when their families are young—and an accelerated pace later on. And women’s careers tend to persist longer than men’s do. After all, women live 5 to 8 years longer, and their careers tend to have more variety over their lifespans.

Do these differences persist among the gifted?

Results from a 2014 long-term study of gifted young adults show that women who have exactly the same level of ability as men often choose different life paths.2 The study was conducted by David Lubinski, Camilla P. Benbow, and Harrison J. Kell from Vanderbilt University. They followed up on teenagers who tested at the top 1 percent in mathematical reasoning ability (they also followed another cohort that had high levels of verbal abilities) four decades later to find out what were they doing. Many of the women and men had different career outcomes, but they also had different preferences and values, which had a big role to play in their career choices. For example, most of the men were more concerned with outward, personal success and felt that society should invest in them because their ideas were better than other people’s. Whereas the women, on average, were more concerned with fairness: that members of society should not go without what they needed, and that they had a role to play in ensuring that fairness. The men, on average, were more focused on their own achievements and on creating concrete things. Women, on average, were more focused on investing in a society that was healthy and that provided for all. There were more men in STEM careers than there were women. But there were more women in medicine, law, health, and education.

Are these differences the result of biological or cultural differences?

I think that both play a role. It’s simplistic and scientifically untrue to say it’s one or the other. The two sexes are not blank slates, even if a gender-continuum exists. The weight of the evidence shows that most women and men are pretty similar on average, and that the differences are more evident at the extremes. And many technology positions may well attract male extremes. It may be a certain type of guy who is drawn to IT jobs, a particularly systematic and single-minded male. In my recent book, The Village Effect: How Face-to-face Contact Can Make Us Healthier and Happier, I present evidence that some of the choices that women make, such as devoting a lot of time grooming their close relationships, help to extend their lifespans. It’s essentially a question of where we shift our telescope. If we focus our telescope on what men have traditionally valued, which is high-income and STEM jobs and long work-hours, say over 60 or 70 hours a week, then yes, women are not there, at 50/50, and I don’t think they will ever be—even if we had the most gender-neutral society possible. But if we aim our lens on lifespan, career satisfaction, and close personal relationships nurtured over decades—relationships that have been shown to protect cognition, resilience, and immunity—then men, on average, trail well behind women. If men want to live longer, happier lives, they have a lot of catching up to do. It all depends on the outcomes you value most.

Women, on average, were more focused on investing in a society that was healthy and that provided for all.

Do women and men regard part-time careers differently?

In the Netherlands, for example, where part-time work is legally protected for both men and women—you can’t lose your job if you’re full-time and then want to work three days a week—they found the vast majority of people who choose to work part-time are women. Indeed, 62 percent of the women who work part-time don’t even have children at home. They work part-time because they have other priorities in their lives—their relationships, their friends, their outside interests. When we look at the North American data, we find that people who choose flexible jobs or schedules in industries that offer those options, two-thirds of them are women. Most, but not all of them are mothers.

Doesn’t this put them at a disadvantage?

Many women put their eggs in a number of baskets—their families, their friends, their interests—as opposed to putting them in just one: their careers. In fact, this more diverse set of interests and skills protected women after the financial crisis of 2008. Fewer women than men were unemployed, and fewer women committed suicide due to financial losses and a complete loss of identity. When you ask women to describe their ideal job, most women—especially highly educated women—say they want to work with people they respect; they want to do a job that’s meaningful, that provides social contact, and that is flexible. Most tech jobs do not satisfy those criteria.

How does government support for part-time and flexible career choices affect outcomes?

In the United States, a woman can’t even take off six months with a new baby without losing face, or losing her job. It’s not like this everywhere. A senior newspaper editor who interviewed me in the Netherlands when I was on a book tour for The Sexual Paradox had a very high-level job, and she worked part-time, which surprised me. I said, “Why do you work part time?” She answered, “Well, Wednesday afternoons is for my aging parents and for my friends. And Friday is for piano. I’m a very serious piano player, and I want to reserve time in my life for something that gives me joy.” Protected part-time work might be one of the reasons why Dutch women are happier than American women: They have more career options than we do.

Math Shall Set You Free—From Envy

Maegan Ayers and her then-boyfriend, Nathan Socha, faced a dilemma in the fall of 2009. They had found the perfect little condo for sale in the Boston neighborhood of Jamaica Plain: on the ground floor, just a mile from the...READ MORE

Would gender disparity in career outcomes decrease or increase as a society becomes more egalitarian?

I can’t predict the future, but several excellent recent studies show that, paradoxically, as a society becomes more egalitarian, the gender gap in occupational choice becomes wider, not narrower. A case in point: A study published last month in Psychological Science, by the psychologists David Geary and Gijsbert Stoet, looked at the academic performance of nearly half a million adolescents from 67 countries.3 What they found was that the more gender equal a country was, as determined by the World Economic Forum’s Global Gender Gap Report, the fewer women ultimately took up STEM paths in college. Countries with the most robust legal and cultural protections for gender equality—along with the strongest social safety nets—such as Sweden, Switzerland, Norway, and Finland, have the fewest female STEM graduates, weighing in at about 20 percent of the total (the U.S. has 24 percent). In contrast, countries with almost no protections, with few guarantees for women and where life satisfaction is low—such as Algeria, Tunisia, the United Arab Emirates, Turkey, and Albania—had by far the highest representation of women in STEM, approaching the researchers’ estimates of 41 percent, based on how well girls do in math and science in high school, without considering their other skills. Another study showing this paradoxical effect, from 2008, was led by David Schmitt.4 He and his colleagues found that gender differences in personality are way larger in cultures that offer more egalitarian gender roles and opportunities. This is not what one would predict if men’s and women’s preferences were exclusively constrained by cultural forces.

What do you think accounts for this paradox?

The researchers in the Psychological Science study also examined the range of strengths and weaknesses within each student. Girls were capable of college level STEM studies nearly everywhere they looked, but they had even higher scores in reading—reading was more likely to be their personal strength—whereas boys’ strengths (relative to themselves) were more likely to be in science. If the boys chose a career based on their personal strengths, it would most often be STEM, whereas girls, especially those in countries where they felt free to make their own choices, could choose either, but often followed the lead of their own strengths and interests.

Do other psychologists agree with this interpretation?

I checked this interpretation out with Wendy Williams, Cornell psychologist and co-author, with Stephen Ceci, of the seminal book, Why Aren’t More Women in Science? She concurred. “If the environment offers options for a good life in multiple domains of work, then girls choose to pursue what they are best at relative to their other abilities. This might be STEM, or it might be law, for example. However, if the environment offers limited options and if the best options are in STEM careers, girls tend to focus more on their skills in STEM. The key is that girls and women are making choices that maximize their success, and these choices are not always for careers in STEM.” In places where girls and women feel they have the freedom to make their own choices, in other words, they are more likely to act on their personal strengths and interests. But in places where they feel constrained by cultural or financial strictures, they are more likely to go for what they consider a sure thing, which is a STEM career. I absolutely agree with and promote equal access to opportunities and education. But equal access to opportunities and education does not determine an equal result. Assuming that women are simply a tamped down, smothered version of men—and would always choose what men choose if they only had the chance—is neither respectful of women’s autonomy nor supported by the data.

Brian Gallagher is the blog editor at Nautilus.

References

1. Voyer, D. & Voyer, S.D. Gender differences in scholastic achievement: A meta-analysis. Psychological Bulletin140, 1174-1204 (2014).

2. Lubinsky, D., Benbow, C.P., & Kell, H.J. Life paths and accomplishments of mathematically precocious males and females four decades later. Psychological Science25, 2217-2232 (2014).

3. Stoet, G. & Geary, D.C. The gender-equality paradox in science, technology, engineering, and mathematics education. Psychological Science (2018). Retrieved from DOI:10.1177/0956797617741719

4. Schmitt, D.P., Realo, A., Voracek, M., & Allik, J. Why can’t a man be more like a woman? Sex difference in big five personality traits across 55 cultures. Journal of Personality and Social Psychology94, 168-182 (2008).

Lead Art: Compassionate Eye Foundation / Getty Images

The Lottery Hackers

0
0

Chapter 1

Gerald Selbee broke the code of the American breakfast cereal industry because he was bored at work one day, because it was a fun mental challenge, because most things at his job were not fun and because he could—because he happened to be the kind of person who saw puzzles all around him, puzzles that other people don’t realize are puzzles: the little ciphers and patterns that float through the world and stick to the surfaces of everyday things.

This was back in 1966, when Jerry, as he is known, worked for Kellogg’s in Battle Creek, Michigan. He was a materials analyst who designed boxes to increase the shelf life of freeze-dried foods and cereals. “You ever buy a cereal that had a foil liner on the inside?” Jerry asked not long ago. “That was one of my projects.”

He worked in the same factory where the cereals were cooked, the smells wafting into his office—an aroma like animal feed at first, and then, as the grains got rolled and flaked and dried, like oatmeal. Near his desk, he kept a stash of cereal boxes made by Kellogg’s competitors: Cheerios from General Mills, Honeycomb from Post. Sales reps brought these back from around the country, and Jerry would dry, heat and weigh their contents in the factory’s lab, comparing their moisture levels to that of a Kellogg’s cereal like Froot Loops. It wasn’t the most interesting job, but both of Jerry’s parents had been factory workers, his father at a hose-fitting plant and his mother at the same Kellogg’s factory, and he wasn’t raised to complain about manual labor.

One day Jerry found himself studying a string of letters and numbers stamped near the bottom of a General Mills box. Companies like Kellogg’s and Post stamped their boxes too, usually with a cereal’s time and place of production, allowing its shelf life to be tracked. But General Mills’ figures were garbled, as if in secret code. Jerry wondered if he could make sense of them. After locating a few boxes of General Mills and Kellogg’s cereals that had sat on store shelves in the same locations, he decided to test their contents, reasoning that cereals with similar moisture must have been cooked around the same time. Scribbling on a piece of scratch paper, he set up a few ratios.

All of a sudden, he experienced the puzzle-solver’s dopamine hit of seeing a solution shine through the fog: He had worked out how to trace any General Mills box of cereal back to the exact plant, shift, date and time of its creation. “It was pretty easy,” Jerry would recall decades later, chuckling at the memory. In a more ruthless industry, cracking a competitor’s trade secrets might have generated millions in profits. This, however, was the cereal business. Discovering the adversary’s production schedule didn’t make anyone rich, and so when Jerry shared his findings with his managers, his discovery was swallowed and digested without fuss.

He didn’t mind. To him, the fun was in figuring it out—understanding how this small piece of the world worked. He’d always had a knack for seeing patterns in what struck other people as noise. As a kid, Jerry had been dyslexic, fumbling with his reading assignments, and he hadn’t realized he possessed academic gifts until a standardized test in eighth grade showed he could solve math problems at the level of a college junior. His senior year of high school, he’d married his sweetheart, a bright, blue-eyed classmate named Marjorie, and after graduation he took a job as a Kellogg’s factory worker. As their family grew over the next decade—with six kids in all—Jerry worked a series of factory and corporate jobs: chemist at a sewage-treatment plant, pharmaceutical salesman, computer operator, cereal packaging designer and, eventually, shift manager.

Still, he remained intellectually restless, and he enrolled in night classes at Kellogg Community College, known around town as “Cornflake U.” It wasn’t easy to squeeze in a life of the mind between the demands of a growing brood, so Jerry invited his kids into his obsessions with various hidden layers of the world: When he got interested in mushrooms, he took them hunting for morels in the forests; when he became captivated by geology, he brought them to gravel pits in search of fossilized spheres called Petoskey stones. Around the time his oldest son, Doug, was in high school, Jerry asked Doug for help counting rolls of coins he’d collected. Knowing that people rolled up their spare change and cashed it at the bank, it had occurred to Jerry to buy these rolls at face value, hoping that the bank hadn’t opened and checked them. Jerry’s idea was that maybe bank customers, by mistake, had included certain rare and valuable coins along with the normal ones. Father and son would sit in front of the TV at night and rip open the rolls, searching for buffalo nickels and silver Mercury head dimes; they made about $6,000. “Anything he jumps into, he jumps into one hundred percent,” Doug explained later. “He gets interested in string theory, and black holes, and all of a sudden you’re surrounded by all these Stephen Hawking books.”

As the years passed, Jerry earned a pile of diplomas: an associate’s degree from Kellogg, a bachelor’s in mathematics and business from Western Michigan University and an MBA from WMU. He also started a master’s in mathematics, though eventually family duties got in the way and he didn’t finish. Even then, he couldn’t stop thinking about numbers. One year, when he and Marge went to a used-book sale at a library to find gifts for their family, Jerry’s main purchase was a stack of college math textbooks. When their daughter Dawn asked why, he replied, “To keep my skills sharp.”

So perhaps it was only fitting that at age 64, Jerry found himself contemplating that most alluring of puzzles: the lottery. He was recently retired by then, living with Marge in a tiny town called Evart and wondering what to do with his time. After stopping in one morning at a convenience store he knew well, he picked up a brochure for a brand-new state lottery game. Studying the flyer later at his kitchen table, Jerry saw that it listed the odds of winning certain amounts of money by picking certain combinations of numbers.

That’s when it hit him. Right there, in the numbers on the page, he noticed a flaw—a strange and surprising pattern, like the cereal-box code, written into the fundamental machinery of the game. A loophole that would eventually make Jerry and Marge millionaires, spark an investigation by a Boston Globe Spotlight reporter, unleash a statewide political scandal and expose more than a few hypocrisies at the heart of America’s favorite form of legalized gambling.

The Corner Store, in downtown Evart, where Jerry first got interested in the lottery.

Chapter 2

Evart, Michigan: 1,903 residents, three banks, one McDonald’s, no Starbucks, a single stoplight on Main Street, a combination Subway/gas station where locals drink coffee in the morning, a diner with the stuffed heads of elk mounted on wood-paneled walls. Historically an auto industry town, sustained by two factories that provided parts to General Motors and Chrysler. Four months of winter and rutted, ice-glazed roads. People endure the cold and the economy and vote for Republicans. Summer brings a shuffleboard tournament and a musical festival billed as “The World’s Largest Hammered Dulcimer Gathering.”

In other words, a perfect town—at least as far as Jerry and Marge were concerned, in 1984, when Jerry decided that he was tired of working for other people and wanted to run something himself: a convenience store. With typical analytic intensity, he had gathered data for 32 “party stores” available for sale across Michigan, places that sold mainly cigarettes and liquor. He studied their financial histories, the demographics of their towns, the traffic patterns on surrounding roads, and found exactly the place to move his family. Though Evart, 120 miles north of Battle Creek, was remote and cold, the town’s auto plants provided a steady customer base, and the store, simply called the Corner Store, was located on Main Street. He and Marge and the kids moved into a two-story house with white siding less than a mile away, on the edge of a forest and the Muskegon River.

Before long, everyone knew the Selbees. Marge, who for years had devoted herself to the role of supportive housewife, joined Jerry at the store. A practical woman who could clear a fallen tree with a chainsaw and sew a men’s suit from scratch without a pattern, Marge did the books, stocked the shelves, and handled impulse items like candy. Jerry purchased the liquor and cigarettes. They opened at 7 a.m. and didn’t leave until midnight, even opening on Christmas morning, when Evart’s only grocery store was closed. Everyone in town passed through the Corner Store—factory workers, lawyers, bankers—and if Jerry didn’t know a customer by name, he knew him by his order. Pall Mall and a Mountain Dew came in a lot. Six-Pack of Strohs was also a regular. Jerry figured out that if he put his beer cooler on defrost late in the evening, the bottles would develop a layer of frost by morning that made them irresistible to factory workers coming off the night shift. “Oh God, did they love that. A lot of 40-ouncers went out of that store. And they said, ‘Oh my God, coldest beer in town,’” Jerry recalled, laughing. “Never told ’em.”

Jerry was happiest when he was trying to solve the puzzle of the store like this, dreaming up ways to squeeze every last penny of profit out of a fixed space. He knew, for instance, that cigarette companies paid store owners for shelf space by discounting the price of cigarettes to the tune of $2 a carton. Jerry figured out that if he bought cigarettes wholesale at this discounted rate, then marked them up by $1 and sold them to smaller retailers who didn’t get the discount, he could undercut cigarette wholesalers. It wasn’t exactly fair to the cigarette companies, but it wasn’t exactly illegal, either.

A year after taking over the Corner Store, Jerry thought to install a lottery machine, a maroon box the size of a cash register that printed tickets for Michigan’s state lottery. The machine was the only one in Evart and one of the few in the county. Word got around fast. “All of our customers that came into our store would play—every one of ’em,” Jerry recalled. The loyal customer known as Six-Pack of Strohs became Six-Pack of Strohs and Five Quick Picks. Jerry offered 16 or 18 different instant games, earning a 6 percent commission from the state on every ticket sold and 2 percent of winning tickets cashed at his store. He advertised in the local paper, and when sales fell on a particular game, he took the unsold tickets and taped brand-new pennies to them. “Those are lucky pennies,” he’d tell his customers, who would then buy the tickets. Soon he was selling $300,000 in lottery tickets per year, pocketing about $20,000 of that in profit. (The biggest prize a customer ever won at his store was $100,000.)

Despite running a vice depot, the Selbees were teetotalers. They didn’t smoke or drink—Jerry permitted himself a single dark beer at Christmas—and Marge avoided the lottery entirely, disliking the sense of risk. Jerry bought a couple of tickets from time to time, but to him, the lottery was only interesting as a phenomenon with order, a set of rules mediated by math and a marketplace. The machine was so successful, however, that he and Marge were able to build a small addition to the store, and he hired an extra clerk to run the machine on the days of the weekly drawings, when business was especially brisk. Eventually, their profits helped pay for the educations of their six children, all of whom earned advanced degrees. “It was like free money,” said Jerry.

At his old convenience store, Jerry liked to dream up new ways of squeezing out a profit from his business, like when he made his 40-ounce beer bottles look frosty for Evart’s factory workers. After a day of work, he and Marge would close up at midnight and head home to their house on the edge of the woods.

And for more than 15 years, this is how it went. The store opened, the sun rose, the sun set, the store closed. Cigarettes, liquor, tickets, tickets, tickets. The Selbee children grew up, left home and started families of their own. Finally, in 2000, Jerry and Marge decided it was time to retire. Jerry began hanging out at the Subway/gas station, arriving each day at 6 to drink coffee and read The Detroit News. Sometimes he’d stop by the Corner Store too, chatting with the new owners to see how they were getting along.

It was on one of these mornings at the Corner Store, in 2003, that Jerry saw the brochure for the new lottery game. Though he’d spent tens of thousands of hours watching his old customers hope for the break that might alter their fortunes, he knew better than to believe the lottery was ruled by chance. “People have been conditioned to think it is luck,” he would later reflect. “They don’t look at the structure of games.”

This particular game was called Winfall. A ticket cost $1. You picked six numbers, 1 through 49, and the Michigan Lottery drew six numbers. Six correct guesses won you the jackpot, guaranteed to be at least $2 million and often higher. If you guessed five, four, three, or two of the six numbers, you won lesser amounts. What intrigued Jerry was the game’s unusual gimmick, known as a roll-down: If nobody won the jackpot for a while, and the jackpot climbed above $5 million, there was a roll-down, which meant that on the next drawing, as long as there was no six-number winner, the jackpot cash flowed to the lesser tiers of winners, like water spilling over from the highest basin in a fountain to lower basins. There were lottery games in other states that offered roll-downs, but none structured quite like Winfall’s. A roll-down happened every six weeks or so, and it was a big deal, announced by the Michigan Lottery ahead of time as a marketing hook, a way to bring bettors into the game, and sure enough, players increased their bets on roll-down weeks, hoping to snag a piece of the jackpot.

The brochure listed the odds of various correct guesses. Jerry saw that you had a 1-in-54 chance to pick three out of the six numbers in a drawing, winning $5, and a 1-in-1,500 chance to pick four numbers, winning $100. What he now realized, doing some mental arithmetic, was that a player who waited until the roll-down stood to win more than he lost, on average, as long as no player that week picked all six numbers. With the jackpot spilling over, each winning three-number combination would put $50 in the player’s pocket instead of $5, and the four-number winners would pay out $1,000 in prize money instead of $100, and all of a sudden, the odds were in your favor. If no one won the jackpot, Jerry realized, a $1 lottery ticket was worth more than $1 on a roll-down week—statistically speaking.

“I just multiplied it out,” Jerry recalled, “and then I said, ‘Hell, you got a positive return here.’”

Chapter 3

The lottery as an American pastime stretches back to the Colonial era, when churches, universities and Congress itself hawked lottery tickets to the public, keeping a cut of the sales and plowing those funds back into the community to pay for roads, or schools, or churches, or armies. This is the basic contract of the lottery: The player accepts a sucker’s bet, a fantastically tiny shot at getting rich, and the organizer accepts the player’s money and does something socially constructive with it.

Lotteries have always been popular with players. Psychological research suggests that we do it for a variety of negative or desperate reasons: a desire to escape poverty, coercion by advertising, gambling addiction, ignorance of probability. Yet there’s also the fun of it. Even when we understand on some level that the odds are ridiculous, that the government is the casino that always wins, we play anyway, because we enjoy the illusion, the surge of risk and hope.

Jerry wasn't thinking about socioeconomics. He was thinking about how he would hide his new hobby from his wife.

This demand for the lottery has made it deathless in America, a vampire institution that hides and sleeps during certain ages but always comes back to life. In 1762, lawmakers in Pennsylvania noticed that poor people bought more tickets than rich people and argued that the lottery functioned as a sort of tax on the poor. They fined operators of these “mischievous and unlawful games” for causing the “ruin and impoverishment of many poor families.” Toward the end of the 19th century, after a corruption scandal in Louisiana—criminal syndicates gained control of the state lottery by bribing elected officials—many states banned lotteries altogether. But Americans continued to play the game underground, with bookies siphoning off the cash that would have otherwise flowed into public coffers, and in 1964, when New Hampshire launched the first legal, government-sponsored lottery in the continental U.S. in 70 years, other states followed.

Today 44 states, Washington, D.C., the U.S. Virgin Islands and Puerto Rico run their own lotteries; they also collaborate to offer Mega Millions and Powerball jackpots, controlled by a nonprofit called the Multi-State Lottery Association. The modern lottery industry is highly complex, offering a zoo of products that are designed and administered with the aid of computers (cash games with a drawing, instant scratch-off games, video lottery games, keno), and the sales of all of these tickets add up to a staggering yearly figure: $80 billion. For comparison, the entire U.S. film industry sells only about $11 billion in tickets.

As for the payouts: More than $50 billion goes to players in prizes, while $22 billion flows to public programs like education, senior assistance, land conservation, veteran support and pension funds. This is why lotteries don’t have a lot of political enemies: the money is impossible for elected officials of both parties to resist. At the same time, as the lottery has grown stronger, so has the fundamental case against it: that the lottery is regressive, taking from the poor and giving to the rich. One review in the Journal of Gambling Studies in 2011 concluded that the poor are “still the leading patron of the lottery”; another study, conducted by the State University of New York at Buffalo in 2012, found that men, black people, Native Americans and those in disadvantaged neighborhoods play the game at higher rates than others. Over the past 40 years, the lottery has played a key role in the broader shift of the American tax burden away from the wealthy; it’s far easier, politically, for states to raise money through a lottery than through more progressive means like corporate or property taxes. According to the investigative reporter David Cay Johnston, who won a Pulitzer for his work on the inequalities in the American tax code, 11 states made more from the lottery in 2009 than they did from corporate income tax.

Jerry was thinking about none of this at his kitchen table. He was thinking about how he would hide his lottery playing from Marge. She had always been the pragmatic one in the relationship, disliking uncertainty and valuing old-fashioned elbow grease over entrepreneurial brainstorms. Even now, in retirement, she was finding it difficult to relax; while her husband watched science shows on TV, she could often be found painting the barn or moving a fallen tree in the yard.

Marge would have questions, Jerry knew, and he might not have bulletproof answers. He didn’t quite believe the numbers himself. How likely was it that the hundreds of employees at the state lottery had overlooked a math loophole obvious enough that Jerry could find it within minutes? Could it be that easy? He decided to test his theory in secret, simulating the game with a pencil and yellow pad first. He picked numbers during a roll-down week, waited for the drawing, and counted his theoretical winnings. On paper, he made money.

Michigan The first time Jerry played, at a convenience store in Mesick, he lost money.$2,200PLAYED
($1 per ticket)
$2,150WINNINGS-$50
But he realized his problem was that he hadn’t wagered enough. Three months later, after buying more tickets, he confirmed his suspicion that big paydays were ahead.$8,000PLAYED$15,700WINNINGS+$7,700

The next time the Winfall jackpot crept north of $5 million and the state announced a roll-down, Jerry drove to a convenience store in Mesick, 47 miles northwest of Evart, so that no one would ask him questions. Standing at the machine, he spent $2,200, letting the computer pick all the numbers for him. A few days later, after the lottery drew six winning numbers, Jerry sorted through his 2,200 tickets and circled all the two-, three- and four-number matches (there were zero five-number matches). His winnings added up to $2,150, slightly less than he had spent on the tickets.

A less confident person might have stopped there. But Jerry figured it was mere bad luck. Odds are just odds, not guarantees. Flip a quarter six times and you might get six heads even though you have better odds of getting three heads and three tails. But flip it 5,000 times and you’ll approach 2,500 heads and 2,500 tails. Jerry’s mistake had been risking too little money. To align his own results with the statistical odds, he just needed to buy more lottery tickets.

This was an uncomfortable leap for a guy with no experience in gambling, but if he stopped now, he would never know if his theory was correct. During the next roll-down week, he returned to Mesick and made a larger bet, purchasing $3,400 in Winfall tickets. Sorting 3,400 tickets by hand took hours and strained his eyes, but Jerry counted them all right there at the convenience store so that Marge would not discover him. This time he won $6,300—an impressive 46 percent profit margin. Emboldened, he bet even more on the next roll-down, $8,000, and won $15,700, a 49 percent margin.

The Selbees then went on vacation, camping at a state park in Alabama with some friends, and while sitting at the campfire one evening, Jerry decided to let his wife in on the secret. He was playing the lottery. He knew how to beat it. He had a system. He’d already won five figures.

Marge didn’t react. The logs cracked in the dusk. She mulled his words over for a long moment. Then, at last, she smiled. She had seen her husband solve so many different kinds of puzzles over the years. Certainly he was capable of doing so again. And who could argue with $15,700? “Oh, I knew it would work,” Marge would later say. “I knew it would work.”

Jerry would eventually buy hundreds of thousands of tickets every roll-down week.

Chapter 4

The American heist master Willie Sutton was famously said to have robbed banks because that’s where the money was. The lottery is like a bank vault with walls made of math instead of steel; cracking it is a heist for squares. And yet a surprising number of Americans have pulled it off. A 2017 investigation by the Columbia Journalism Review found widespread anomalies in lottery results, difficult to explain by luck alone. According to CJR’s analysis, nearly 1,700 Americans have claimed winning tickets of $600 or more at least 50 times in the last seven years, including the country’s most frequent winner, a 79-year-old man from Massachusetts named Clarance W. Jones, who has redeemed more than 10,000 tickets for prizes exceeding $18 million.

It’s possible, as some lottery officials have speculated, that a few of these improbably lucky individuals are simply cashing tickets on behalf of others who don’t want to report the income. There are also cases in which players have colluded with lottery employees to cheat the game from the inside; last August, a director of a multistate lottery association was sentenced to 25 years in prison after using his computer programming skills to rig jackpots in Colorado, Iowa, Kansas, Oklahoma and Wisconsin, funneling $2.2 million to himself and his brother.

But it’s also possible that math whizzes like Jerry Selbee are finding and exploiting flaws that lottery officials haven’t noticed yet. In 2011, Harper’s wrote about “The Luckiest Woman on Earth,” Joan Ginther, who has won multimillion-dollar jackpots in the Texas lottery four times. Her professional background as a PhD statistician raised suspicions that Ginther had discovered an anomaly in Texas’ system. In a similar vein, a Stanford- and MIT-trained statistician named Mohan Srivastava proved in 2003 that he could predict patterns in certain kinds of scratch-off tickets in Canada, guessing the correct numbers around 90 percent of the time. Srivastava alerted authorities as soon as he found the flaw. If he could have exploited it, he later explained to a reporter at Wired, he would have, but he had calculated that it wasn’t worth his time. It would take too many hours to buy the tickets in bulk, count the winners, redeem them for prizes, file the tax forms. He already had a full-time job.

It never occurred to Jerry to alert the Michigan Lottery that Winfall was vulnerable to exploitation. For all he knew, the state was perfectly aware of the flaw already. Maybe the flaw was intentional, to encourage players to spend lots of money on lottery tickets, since the state took a cut of each ticket sold, about 35 cents on the dollar. (In 2003, the year that Jerry began playing, the state lottery would sell $1.68 billion in tickets and send $586 million of that revenue into a state fund to support K-12 public education.) In Jerry’s opinion, if he was purchasing large quantities of tickets at certain opportune moments, he wouldn’t be manipulating the game; he would be playing it as it was meant to be played. His tickets would have the same odds of winning as anyone else’s. He would just be buying a lot more of them.

Jerry founded an American company that sold nothing, created nothing, had no inventory, no payroll. Its one and only business was to play the lottery.

And, unlike Srivastava, he and Marge were willing to do the grunt work, which, as it turned out, was no small challenge. Lottery terminals in convenience stores could print only 10 slips of paper at a time, with up to 10 lines of numbers on each slip (at $1 per line), which meant that if you wanted to bet $100,000 on Winfall, you had to stand at a machine for hours upon hours, waiting for the machine to print 10,000 tickets. Code in the purchase. Push the “Print” button. Wait at least a full minute for the 10 slips to emerge. Code in the next purchase. Hit “Print.” Wait again. Jerry and Marge knew all the convenience store owners in town, so no one gave them a hard time when they showed up in the morning to print tickets literally all day. If customers wondered why the unassuming couple had suddenly developed an obsession with gambling, they didn’t ask. Sometimes the tickets jammed, or the cartridges ran out of ink. “You just have to set there,” Jerry said.

The Selbees stacked their tickets in piles of $5,000, rubber-banded them into bundles and then, after a drawing, convened in their living room in front of the TV, sorting through tens or even hundreds of thousands of tickets, separating them into piles according to their value (zero correct numbers, two, three, four, five). Once they counted all the tickets, they counted them again, just to make sure they hadn’t missed anything. If Jerry had the remote, they’d watch golf or the History Channel, and if Marge had it, “House Hunters” on HGTV. “It looked extremely tedious and boring, but they didn’t view it that way,” recalled their daughter Dawn. “They trained their minds. Literally, they’d pick one up, look at it, put it down. Pick one up, put it down.” Dawn tried to help but couldn’t keep pace; for each ticket she completed, Jerry or Marge did 10.

In the beginning, his children didn’t understand Jerry’s new passion. “I thought he was crazy,” Dawn said. “He starts to explain it to you, and your eyes glaze over.” Doug couldn’t make sense of it either. “He always said, this is just sixth-grade math. I was like, ‘Yeah, did you see what I got in math in sixth grade?’” Jerry and Marge insisted that they were enjoying themselves. They had the time. It was a game. Marge even seemed to like the manual labor. (“I’m just the grunt,” she explained, with a mix of self-deprecation and pride.) In the weeks between roll-downs, they got antsy.

Jerry and Marge placed the losing numbers in large plastic tubs that they stored in a barn out back. That way, there would always be a paper trail for the IRS.

And they were happy to share their good fortune. Like lotteries in other states, the Michigan Lottery welcomed large betting groups; after all, the more people who played, the more money the state got to play with. Jerry saw that office pools and other large bettors were allowed to play as corporations instead of individuals, and it seemed to him that the state was practically inviting groups to play Winfall for big stakes. So in the summer of 2003, about six months after Jerry bought his first tickets, the Selbees asked their six children if they wanted in. The kids ponied up varying amounts for Jerry to wager; on their first try together, the family bet $18,000 and lost most of it, because another player hit the six-number jackpot. When Jerry insisted this was just bad luck, Marge and the kids decided to believe him. They let him risk their money again, and within two more plays, everyone was in the black.

That June, Jerry created a corporation to manage the group. He gave it an intentionally boring name, GS Investment Strategies LLC, and started selling shares, at $500 apiece, first to the kids and then to friends and colleagues in Evart. Jerry would eventually expand the roster to 25 members, including a state trooper, a parole officer, a bank vice president, three lawyers and even his personal accountant, a longtime local with a smoker’s scratchy voice named Steve Wood. Jerry would visit Wood’s storefront office downtown, twist the “Open” sign to “Closed,” and seek his advice on how to manage the group.

The corporation itself was nearly weightless. It existed purely on paper, in a series of thick three-ring binders that Jerry kept in his basement, a ream of information about the members, the shares, the amounts wagered on roll-down weeks, the subsequent winnings and losses, the profits and the taxes paid. It was an American company that sold nothing, created nothing, had no inventory, no payroll. Its one and only business was to play the lottery.

And business was good. By the spring of 2005, GS Investment Strategies LLC had played Winfall on 12 different roll-down weeks, the size of the bets increasing along with the winnings. First $40,000 in profits. Then $80,000. Then $160,000. Marge squirreled her share away in a savings account. Jerry bought a new truck, a Ford F350, and a camping trailer that hooked onto the back of it. He also started buying coins from the U.S. Mint as a hedge against inflation, hoping to protect his family from any future catastrophe. He eventually filled five safe deposit boxes with coins of silver and gold.

Then, in May 2005, the Michigan Lottery shut down the game with no warning, replacing it with a new one called Classic Lotto 47. Officials claimed that sales of Winfall tickets had been decreasing. Jerry was offended. He’d found something he loved, something to order his days that felt constructive and rewarding and didn’t hurt anyone. He didn’t want to stop. “You gotta realize, I was 68 years old. So it just—it gave me a sense of purpose.” His fellow players were just as disappointed, including Marge. “I like to have something to do, especially in the wintertime,” she explained.

The following month, Jerry received an email from a member of the lottery group. The player, a plant manager at a Minute Maid juice factory in Paw Paw Township, had noticed that Massachusetts was promoting a brand-new lottery game called Cash WinFall. There were a few differences between it and the now-defunct Michigan game: a Cash WinFall ticket cost $2 instead of $1; you picked six numbers from 1 to 46 instead of 1 to 49; and the jackpot rolled down when it hit $2 million, not $5 million. But otherwise, it appeared to be the same. “Do you think we could play that?” the plant manager asked.

Jerry did a few brisk pencil-and-paper calculations. The odds were good. He wondered about the logistics: Lottery tickets had to be purchased in person, and the western edge of Massachusetts was more than 700 miles from Evart. He had no connections to store owners in Massachusetts, either. Who would ever let him and Marge stand in one spot for hours, printing ticket after ticket?

Still, he couldn’t resist. Jerry emailed the plant manager back, asking if he knew anyone who ran a party store in the state. The player gave him a name: Paul Mardas, the owner of Billy’s Beverages, in Sunderland, about 50 miles from the western border of Massachusetts. Disliking the hassle of airports, Jerry climbed into his gray Ford Five Hundred one day in August 2005 and began the 12-hour drive to the East Coast. What he didn't know was that, for the first time in his gambling career, he was about to encounter some ruthless adversaries.

Chapter 5

Seven months earlier, a student at the Massachusetts Institute of Technology named James Harvey was knocking on doors in his dorm, trying to get people excited about two personal projects. One was a Super Bowl party—the New England Patriots were looking for a back-to-back championship. The other was a lottery betting pool he wanted to start.

The dorm, a four-story building known as Random Hall, was packed with computer science and engineering majors. It had a custom lab in the basement and a student-coded website that tracked when the dorm’s washing machines and bathrooms were in use. Harvey’s Super Bowl party had little appeal in Random Hall, but people sparked to his lottery idea. A mathematics major in his final semester, Harvey had been researching lottery games for an independent study project, comparing the popular multistate games Powerball and MegaMillions to see which offered players a better shot at winning. He’d also analyzed different state games, including Cash WinFall, and it hadn’t taken him long to spot its flaw: On a roll-down week, a $2 lottery ticket was worth more than $2, mathematically.

Within days, Harvey had recruited some 50 people to pony up $20 each, for a total of $1,000, enough to buy 500 Cash WinFall tickets for the February 7 roll-down drawing. The Patriots won the Super Bowl on February 6, and the following day, the MIT group took home $3,000, for a $2,000 profit.

Counting $70,000 in tickets took them a full 10 days, working 10 hours a day. They never left the room except to get lunch.

Curiously enough, the MIT students weren’t the only ones playing Cash WinFall for high stakes that day. A biomedical researcher at Boston University, Ying Zhang, had also discovered the flaw, after an argument with friends about the nature of the lottery. Believing it to be exploitative, Zhang had researched the Massachusetts State Lottery to bolster his point. Then he found the glitch in Cash WinFall, and as happens so often in America, a skeptic of capitalism became a capitalist. Zhang encouraged friends to play and formed his own betting club, Doctor Zhang Lottery Club Limited Partnership. His group began wagering between $300,000 and $500,000 on individual roll-down weeks, and eventually Zhang quit his job as a biomedical researcher to focus on the lottery full time. He bought tickets in bulk at a convenience store near his home, in the Boston suburb of Quincy, and stored the losing tickets in boxes in his attic until the weight made his ceiling crack.

As energetically as Zhang played the game, however, he couldn’t match the budding lottery moguls at MIT. After the first roll-down, Harvey assembled 40 to 50 regular players—some of them professors with substantial resources—and recruited his classmate, Yuran Lu, to help manage the group. Lu was an electrical engineering, computer science and math major with a mischievous streak: one time, to make a point about security, he’d stolen 620 passwords from students and professors. Now he helped Harvey form a corporation, named Random Strategies LLC, after their dorm. Their standard wager on a roll-down week was $600,000—300,000 tickets. Unlike the Selbees, who allowed the computer to pick numbers for them (“Quic Pics”), the MIT students preferred to choose their own, which avoided duplicates but also meant that the students had to spend weeks filling in hundreds of thousands of tiny ovals on paper betting slips.

Of course, it would have been a lot easier for the MIT students to print their lottery slips in bulk, using their own computers, and then hand the slips over to a convenience store owner when it was time to play. But Cash WinFall rules didn’t allow this. It was one of several safeguards put in place by the Massachusetts State Lottery to monitor betting activity and prevent manipulation of the game. Officials at lottery headquarters, in Braintree, were hardly in the dark; sales information went straight to them in real time, or close to real time, tracking the number of tickets sold at each store in the state. Any agent who sold more than $5,000 in tickets per day was also required to get a special waiver, which meant that lottery officials could detect unusually heavy betting well in advance.

As a result, the Massachusetts State Lottery was perfectly aware of several anomalies in Cash WinFall ticket-buying, unusual patterns over the months that signaled that something was up. One day in July, a store manager in Cambridge called headquarters because a kid from MIT had walked in and asked to buy $28,000 in tickets. The manager was stunned and wanted to know: Was that legal? (A compliance officer replied that yes, it was legal.) That same week, a dozen stores suddenly requested waivers to increase their Cash WinFall betting limits. Three of the stores were clustered in the town of Quincy, where Zhang lived, and the fourth was in the next town over. When lottery compliance officers visited the stores, they found two clear violations: a player had been scanning stacks of computerized betting slips, and the store where he operated had been extending him credit, allowing the slips to be scanned before they’d been paid for. Later, officials discovered that a whopping 23 stores across the state were violating a different rule involving a “free bet” feature of the game.

Though the Massachusetts State Lottery was within its rights to suspend or revoke the licenses of all these stores, it instead let them off with warnings. This lax approach to rule enforcement is perhaps why, when Jerry showed up at the party store in Sunderland, Paul Mardas was more intrigued than concerned by the Michigan retiree’s proposition. Jerry reckoned that, for starters, he aimed to buy about $100,000 in lottery tickets. Mardas laughed. Billy’s Beverages was one smallish room with a wood-paneled ceiling; he had no frame of reference for bets that large. But Jerry, wearing rubber bands around his left wrist, offered a deal: If Mardas allowed him to print tickets in bulk at his store, he would give him a stake in GS Investment Strategies LLC.

Mardas agreed, and a few weeks later, Jerry returned with Marge. As in Michigan, the two would need to split the work of printing tickets, and so they sought out a second terminal. They found it at Jerry’s Place, a diner in South Deerfield, whose owner was also willing to join their lottery corporation. That taken care of, the Selbees quickly developed a routine around Cash WinFall. About a week before a roll-down drawing, they would drive the 700 miles from Michigan, cutting across Canada to save time, listening to James Patterson novels on tape. They’d book a room at a Red Roof Inn in South Deerfield, and in the mornings, they’d go to work: Jerry to Jerry’s Place; Marge to Billy’s. They started at 5:30 a.m., before the stores opened to the public, and went straight through to 6 p.m., printing as many tickets as the terminals would handle, rubber-banding them in stacks of $5,000, and throwing the stacks into duffel bags.

Billy’s Beverages, in Sunderland.Jerry’s Place, in South Deerfield.

After a drawing, they retreated to the Red Roof Inn and searched for winning numbers, piling tickets on the double beds and the tables and the air conditioner and the floor. Counting $70,000 in tickets took a full 10 days, working 10 hours a day. They never left the room except to get lunch. Then they claimed their winning tickets and drove the 12 hours back to Michigan with the tens of thousands of losing tickets, storing them in plastic tubs in a barn, behind a door that kept the raccoons out, in case an IRS auditor ever wanted to see the paper trail.

The first time they played Cash WinFall, on August 29, Jerry and Marge ended up spending $120,000 on 60,000 lottery tickets. After that they increased their wager to 312,000 individual tickets per roll-down, ultimately going as high as 360,000 tickets—a $720,000 bet on a single drawing. At first, Marge found these figures terrifying—it was more than they had ever risked in Michigan—but after a while she got used to it. “You know, you think of this as money,” Marge recalled, “but pretty soon you never really look. It’s just numbers. It’s just numbers on a piece of paper.” She grew friendly with other customers, chatting about her kids and the weather as if she had lived in Massachusetts all her life. Mardas came to think of her and Jerry as part of his family. “They’re salt-of-the-earth kind of people,” he said. “Genuine.” He was also amazed by their frugality. “I said to Marge, ‘You guys should go on a cruise or something.’ She said, ‘I’d rather go pick rocks in a quarry.’”

Massachusetts The first time Jerry and Marge played, at convenience stores in Sunderland and South Deerfield, they made money.$120KPLAYED
($2 per ticket)
$178KWINNINGS+$58K
But not nearly as much as they made in one drawing near the end of their run, six years later.$712KPLAYED$998KWINNINGS+$286K

According to lottery regulations, customers weren’t allowed to operate terminals themselves—that was the store owner’s job—and the terminals weren’t supposed to be used outside normal business hours. Jerry got around the first rule by having the corporation, of which the store owners were members, “hire” the Selbees to print the tickets. As for printing tickets within posted store hours—well, yes, that was a violation. But Jerry saw it as a minor sin, no different than what millions of American businesses do every day to get by. He didn’t mind the funny looks he sometimes got. One day, a woman at the diner stared as Jerry printed tickets, then asked the store owner to tell Jerry to “stop doing that.” The owner shook his head. “No,” he replied.

More important to Jerry was that the Massachusetts State Lottery didn’t seem to have a problem with anything that he and Marge were doing. And his comfort level increased when he learned through the grapevine, in 2008, that there were other large betting groups playing Cash WinFall using strategies similar to his own. Over five years, the couple would return to Massachusetts six to nine times per year, never deviating from their system: printing tickets, counting them at the Red Roof Inn, redeeming the winners for a giant check, and driving back to Evart with the losers in the trunk. The lottery checked in on them as they printed tickets at least once, in April 2010, when a compliance officer was sent to Billy’s Beverages and Jerry’s Place. After observing the Selbees at work, the officer reported that he found nothing out of the ordinary. “I spent some time observing the wagering routine,” he wrote to his superiors in an email. “Everything is very organized and runs smoothly.”

One lottery employee replied to the email with a joke: “How do I become a member of the [Selbees’] club when I retire?”

Jerry and Marge would drive the 700 miles to Massachusetts about a week before a roll-down drawing.

Chapter 6

Meanwhile, around them, the larger American economy was imploding. The housing bubble, the bank bailouts, the executive bonus scandals, the automotive bankruptcies—panic, panic, panic, panic. In Evart, an auto glass plant that had supplied Chrysler closed down, throwing 120 people out of work. American corporations had been playing a lot of games, noted Jerry, and their ways had finally caught up. “They were taking far more risks than I was, based on their rewards. That’s why I did a risk-reward analysis after every game, to make sure I was still on track.”

Compared with Bear Stearns or Goldman Sachs, the Selbees were downright conservative. By 2009 they had grossed more than $20 million in winning tickets—a net profit of $5 million after expenses and taxes—but their lifestyle didn’t change. Jerry and Marge remained in the same house, hosting a family gathering each Christmas as they always had. Though she could have chartered a private jet and taken everyone to Ibiza, Marge still ran the kitchen, made her famous toffee candy and washed dishes by hand. It didn’t occur to her to buy a dishwasher.

Instead, the Selbees' lottery playing helped cushion their friends and family, as well as a few people they had never met whom they’d allowed to join the betting group. (One such couple was confronted by their accountant after their tax returns listed winnings and losses in the six figures. “Do you have a gambling problem?” he wanted to know.) Jerry and Marge’s kids socked the winnings away for their children’s educations. A few players paid down debts. Wood, the Selbees’ accountant, took four cruises and renovated his house. Mardas filed for divorce. Meeting the Selbees had given him the financial freedom to “make some changes in my life,” as he put it. “I fell in love again, and remarried, and I’ve got three stepkids that I never thought I would have.”

From time to time, players in the group asked Jerry if he had a plan for stopping. How many more bets were they going to make, for how many years? Weren’t they pushing their luck? “I mean, if I were running a lottery game and somebody spotted a flaw, I would shut it down immediately,” said Jerry. The group had lost money only three times, and even after the biggest loss—$360,000 in a drawing in 2007, when another player correctly chose all six numbers and took the jackpot—the group had made the money back. As long as they kept playing conservatively, Jerry felt, they would not attract undue attention, and there was no reason not to continue. “I’m going to milk this cow as long as it’ll stand,” he’d reply.

Unbeknownst to him, however, the MIT students were preparing to attack the game with a new and unprecedented level of aggression. Though it would later be estimated that their group made at least $3.5 million by playing Cash WinFall, they had noticed that their profit margins were declining, for a simple reason: competition. With MIT, Zhang and the Selbees pushing huge pots of money into each roll-down drawing, they were all having to split the payouts. This had gotten the students thinking. Might there be a way to freeze out the other groups? They hit on an idea: Instead of waiting for a roll-down, perhaps they could force one to happen, by making an insanely large bet.

Jerry was enraged. It was one thing to make large bets, like he had been doing, and it was another thing entirely to manipulate the game.

In the week leading up to the Cash WinFall drawing of August 16, 2010, the state had not announced a roll-down, because the jackpot was only $1.6 million; it didn’t seem that it would reach the required $2 million. Harvey and his MIT friends saw their opening. Over three and a half days, they bought an astonishing 700,000 lottery tickets, costing $1.4 million. This was more than enough to tip the jackpot over $2 million before lottery officials knew what was happening—and before they could announce the roll-down. No one else knew that the money was going to roll down, so the other bettors, including Jerry and Marge, did not buy tickets. The MIT group hoovered up a $700,000 cash profit.

Surprised by the jackpot’s extremely rapid inflation, lottery employees reviewed their data to see what had gone wrong. One technical manager guessed, correctly, that one of the large betting groups had triggered the roll-down, though he misidentified the culprits. “FYI,” he wrote in an email to a colleague. “Michigan guys decided last Friday to push [Cash WinFall] jackpot over $2 mill.” Rather than impose penalties, however, lottery technicians instead installed a new software script to notify them of especially high sales, so that in the future, Braintree could alert all players to an imminent roll-down and give everyone a fair shot.

Jerry was enraged. It was one thing to make large bets based on a certain system, like he had been doing, and it was another thing entirely to manipulate the mechanics of the game to crowd other bettors out. “They took us out of the game,” Jerry said. “Intentionally.” The next time MIT tried to force a roll-down, he decided, he was going to be ready.

Marge and Jerry in 2017 with five of the Selbee children and their spouses (Doug is standing behind Marge; Dawn, holding one of the couple's great-grandchildren, is behind Jerry.) COURTESY OF DAWN TOMLINSON

He suspected something would happen around Christmas. There was a drawing scheduled for December 27, when a lot of convenience stores would be closed for the holiday; with betting activity slow, it made for a perfect time for MIT to strike. On high alert for any shenanigans, Jerry asked Mardas to call lottery headquarters to see if stores were reporting spikes in sales. When Mardas was told that, yes, five stores were seeing a surge, Jerry hopped in his car. Leaving Marge behind, he drove on Christmas Day to Jerry’s Place, where he spent hours printing 45,000 tickets, long after the sun went down.

He was printing the last of them by the pale light of the lotto terminal when he heard a knock on the door. The store was closed—it was just Jerry behind the counter—so he opened the door a crack to talk to the visitor, a polite young man who said his name was Yuran Lu.

“I’m from the other club, and I think it would be mutually beneficial if we knew how much money each of us were playing,” Jerry would later claim Lu told him. Jerry gathered that the MIT kids were proposing to collude; instead of all groups pushing into every pot, it might make sense to take turns. This was unethical in Jerry’s mind, so he shook his head and closed the door. Lu walked away. (Lu did not respond to interview requests for this story.)

Despite its new alert software, lottery officials were slow to react once again, and sure enough, the large bets of the Selbees and the MIT group triggered a roll-down. Jerry had no idea how much went to the MIT kids, but his group made about $200,000 in profit. Driving back to Michigan, he felt vindicated. Maybe this would teach his rivals something about playing by the rules.

Chapter 7

Turning web traffic into a Super Computer

0
0

The subject matter of this post is controversial as it discusses extracting computing resources from the visitors of a website. There are a lot of discussions at the moment centered around web-browser based crypto currency mining. Most paint a deplorable picture of the practice; please keep in mind that there are very desirable paths alongside which these practices can develop. I am not elaborating on these arguments here, I am only describing a method to harness the resources.

Web browsers are becoming quite powerful for code execution. Between Javascript’s increase in capability, WebAssembly, access to GPU & threading, a web browser today is almost as desirable for computing as the machine it’s running on. Ever since the rise of web-based crypto currency miners, I’ve been thinking of harnessing all that computing power as a single entity: a super computer made of your visitor’s web browsers.

Just like a regular computer cluster, the nodes all participate in a coordinated fashion to solving a single problem. Unlike a regular computer cluster, the nodes are very ephemeral (as website visitors come and go) and can’t talk to each other (no cross site requests).

Here’s a demo of what I came up with:

Right: the super computer control server
Left: one of the web clients contributing to the super computer simply by being connected to a website (& CPU metrics)

The problem being solved here is the hashing of 380,204,032 string permutations to find the reverse of a given hash. Problem parameters were chosen to make heavy processing quick for the clients.

At the core of the idea is the websocket technology. It creates a persistent connection between a server and all of the nodes (the visitors of your website). This connection can be used to orchestrate actions between the nodes so that they can act as a concerted entity. From delivering the code to passing messages for coordination, websockets are what make everything possible.

Having a websocket connection to clients dramatically changes what you can do with web clients. They are fully addressable for the duration of their visit. They may show up on a website and be served some pre-established javascript; but with websockets, any javascript can materialize at any time.

Right: the super computer control server
Left: a web client being given an instruction on the fly

Slightly tangential but still worth considering, using a web view app, Javascript can pass execution to the app itself. This means code showing up on the websocket can escape the webview bubble and go into app land.

Right: the super computer control server
Left: a web app being given an instruction which percolates to the app layer

Now this is nothing new in a lot of ways; apps can be made to get instructions from C&Cs, and websites can get Javascript after the initial page load from dynamic sources. The websocket technique though is as dynamic as it gets (no Ajax pull), it is portable to many browsers and many devices, it is hard to catch looking at a web inspector; lastly, it executes with full access to the context it materialized in.

So we’ve established that websockets can be used to dynamically deliver code to be ran by the nodes. It can also be used for message passing and the overall orchestration of distributing the problem to be solved.

6 years ago I wrote a ditributed OpenMPI based password cracker: crackzor. Password cracking is a good distributed problem to solve because it’s a fairly simple problem: run through all the character permutations. The fact that it exhausts a known space also means benchmarking is easy. So to put the idea of a transient node javascript super computer in practice, I rewrote crackzor in JS instead of C, and for websockets instead of OpenMPI.

Every distributed problem is different and crackzor itself isn’t a magic way to distribute any problem to be solved. The magic of crackzor is its ability, given a space of character permutations, to divide it up in chunks which can be processed by the nodes. Given the problem, a start iteration and end iteration, a node can get to work without having to be provided the permutations themselves, thus removing the bandwidth bottleneck.

The first challenge: maximizing usage of the node’s CPU.

Javascript runs single threaded by default, so when the websocket sends code to be ran by a client, by default, the code running as fast as it can will only be able to fill one core of the CPU. A large majority of machines today have many more cores available. So we have to figure out how to use them or our super computer is going to loose a large portion of its processing power right off the bat.

Web workers to the rescue. With HTML5, it’s easy as pie to thread code. The one trick with the code we want to thread is that it can’t be gotten from a file as the web worker documentation suggests. That’s because our code doesn’t come from a static javascript file remember? It shows up out the the blue on the websocket, so it came from the network and is now in memory somewhere => not a file we can refer to.

The solution is to wrap it in a blob as such

var worker_code = 'alert( "this code is threaded on the nodes" );'

window.URL = window.URL || window.webkitURL;

var blob;
try {
    blob = new Blob([worker_code], {type: 'application/javascript'});
} catch (e) {
    window.BlobBuilder = window.BlobBuilder || window.WebKitBlobBuilder || window.MozBlobBuilder;
    blob = new BlobBuilder();
    blob.append(worker_code);
    blob = blob.getBlob();
}
workers.push( new Worker(URL.createObjectURL(blob)) ) ;

Here you’ll notice we have our first layer of encapsulation. The code relevant to the problem we are solving is in the variable worker_code, the rest of the javascript only threads it.

Having distributed amongst a node’s cores, we now look at

the second challenge: distributing between the nodes

This work is obviously up to the websocket server along with subsequent coordination. Without going into too much details, the websocket server keeps track of all the nodes as they come and go, it also keeps track of which ones are working or not, allocates new chunks of the problem to nodes as they become available.

A trick of the websocket server is that it is running at all times to handle node connections. Super computer problems however may change from one day to the next. To address that, I give it a function which reads a file and evals its code; the function is summoned by a process signal. As such:

function eval_code_from_file() {
    if( !file_exists("/tmp/code") ) {
        console.log( "error: file /tmp/code does not exist" ) ;
    } else {
        var code = read_file( "/tmp/code" ) ;
        code = code.toString() ;
        eval( code ) ;
    }
}

process.on('SIGUSR1', eval_code_from_file.bind() );

With this puppy in place, the next time I “kill -USR1 websocket_server_PID”, it will be imbued with new code that did not exist when it started. Does this sound familiar? Yup, javascript is super interesting in the ability it gives you to run arbitrary code at any time with full access to the established context.

Thus arrives the 2nd and 3rd layers of encapsulation, the code which will be distributed to the nodes is in a file which is to be evaled on the websocket server side and sent over the websocket to the clients.

The actual distribution to the nodes is simple, have them connect with a callback to eval code. Something like that:

Client:

var websocket_client=io.connect("http://websocket_server.domain.com") ; 
websocket_client.on( "eval_callback",function(data){data=atob(data),eval(data)}.bind() ) ;

Server:

client_socket.emit( "eval_callback", new Buffer("alert('this code will run on the client');").toString("base64") ) ;

Recapping where we are

So…

  1. all the transient nodes (web browser of website visitors) attach to a websocket server
  2. the websocket server receives SIGUSR1 which signals it to execute new code it gets from a file
  3. this new code gives the websocket server a packaged problem to be solved by the nodes
  4. this new code also instructs how the websocket server will distribute and coordinate the nodes
  5. once the packaged problem to be solved shows up on a node, it is evaled and it contains threading to maximize CPU usage.

all the pieces you need to make a super computer from your web traffic. I’m choosing not to publish the full code of my implementation for reasons of readability, security and complexity but I can go into more details if asked.

The same way that peer-to-peer protocols made any data available anywhere any time, could this do the same for computing power? Mind=blown, and your CPU along with it.

Show HN: Get Rails performance metrics into Chrome Dev Tools

0
0

README.md

Bring Ruby on Rails server-side performance metrics 📈 to Chrome's Developer Tools (and other browsers that support the Server Timing API) via the server_timing gem.

Metrics are collected from the scout_apm gem. A Scout account is not required.

server timing screenshot

Gem Installation

Add this line to your application's Gemfile:

And then execute:

$ bundle

Configuration

A minimal Scout config file is required. The server_timing gem reports metrics collected by the scout_apm gem (added as a dependency of server_timing).

If you don't have a Scout account, copy and paste the following minimal configuration into a RAILS_ROOT/config/scout_apm.yml file:

common: &defaultsmonitor: trueproduction:<<: *defaults

If you have a Scout account, no extra configuration is required. If you wish to see server timing metrics in development, ensure monitor: true is set for the development environment in the scout_apm.yml file.

See the scout_apm configuration reference for more information.

Browser Support

  • Chrome 65+
  • Firefox 59+
  • Opera 52+

Instrumentation

Auto-Instrumentation

By default, the total time consumed by each of the libraries scout_apm instruments is reported. This includes ActiveRecord, HTTP, Redis, and more. View the full list of supported libraries.

Custom Instrumentation

Collect performance data on additional method calls by adding custom instrumentation via scout_apm. See the docs for instructions.

Security

  • Non-Production Environments (ex: development, staging) - Server timing response headers are sent by default.
  • Production - The headers must be enabled.

Response headers can be enabled in production by calling ServerTiming::Auth.ok!:

# app/controllers/application_controller.rb

before_action doif current_user && current_user.admin?ServerTiming::Auth.ok!endend

To only enable response headers in development and for admins in production:

# app/controllers/application_controller.rb

before_action doif current_user && current_user.admin?ServerTiming::Auth.ok!elsifRails.env.development?ServerTiming::Auth.ok!elseServerTiming::Auth.deny!endend

Overhead

scout_apm is designed for use in production apps and is engineered for low overhead instrumentation.

Development

After checking out the repo, run bin/setup to install dependencies. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/scoutapp/ruby_server_timing.

License

The gem is available as open source under the terms of the MIT License.


How the Irish Teach Us to Die

0
0

In America Death is a whisper. Instinctively we feel we should dim the lights, lower our voices and draw the screens. We give the dead, dying and the grieving room. We say we do so because we don’t want to intrude. And that is true but not for these reasons.

We don’t want to intrude because we don’t want to look at the mirror of our own mortality. We have lost our way with death.

On the Irish island off the coast of County Mayo, where my family have lived in the same village for the last 200 years, death speaks with a louder voice.

Along with the weather reports of incoming Atlantic storms, the local country and western radio station runs a thrice daily “deaths” announcement enumerating the names and the funeral arrangements of the ten or so daily freshly departed. There is even a pay-by-the-minute phone line, 95 cents, just so you can check up on those corpses you might have missed.

Article continues after advertisement

There should be nothing strange about this. In the absence of war humans across the planet die at an annual rate of one per cent; 200,000 dead people a day, 73 million dead people a year. An even spread. It’s happening all around you even as you read this article; the block opposite, the neighboring street and your local hospital.

If the local radio in New York did the same as that Mayo radio station the announcer would have to read out the names of 230 dead strangers, three times day, just to keep up.

Of course, if you live in a city like New York, where 85,000 people die each year, you would never know of these things. Such a very public naming of the dead, an annunciation of our universal mortality, would be an act of revelation. And likely deemed an outrage against “public decency” that would almost certainly lead to advertising boycotts and protests.

More shocking still would be the discovery of another country where the dying, the living, the bereaved and the dead still openly share the world and remain bound together. . . in the Irish wake. Where death, in its very ordinariness, is no stranger.

My father Sonny Toolis was an ordinary man. He was never rich nor powerful nor important. He never held public office and his name never appeared in the newspapers. He was born poor in a village on an island, devoid of electricity, water mains and paved roads, in much the same way the poor have been born in such places for most of human history. He worked on building sites most of his life to pay for his seven children’s university education. The world never paid him much attention and Sonny also knew the world never would.

But Sonny really did have one advantage over most of us.

He knew how to die.

And he knew how to do that because his island mothers and fathers, and all the generations before, had shared their deaths at the Irish wake and showed him how to die too.

His dying, his wake, his willing sharing of his own death, would be his last parental lesson to his children and his community. A gift.

If you never been to an Irish wake, or only seen the movie version, you probably think a wake is just another Irish piss up, a few beers around the corpse and an open coffin. But you would be wrong.

The wake is amongst the oldest rites of humanity, first cited in the great 8th-century BCE Homeric war poem The Iliad and commonly practised across Europe up until around the last 200 years. The final verses of The Iliad, the display of the Trojan prince Hector’s corpse, the wailing women, the feasting and the funeral games, are devoted to his wake. And the same rituals would be easily recognisable to any Irish wake-goer today.

For our ancestors, a wake, with its weight of obligation between the living and the bodies of the dead, was a pathway to restore natural order to the world, heal up the mortal wound, and overcome, as a community, the death of any one individual. An act—in our thin, contemporary psychological jargon—of closure.

Through urbanization, industrialization and the medicalization of death the wake died away in most of the Western world at the hands of what we might call the Western Death Machine. But amongst the Celts this ancient form of death-sharing lives on.

When he was 70 my father was diagnosed with pancreatic cancer, which remains among the most fatal of cancers. Sonny never flinched. He did not want to die but when he knew he had no choice, he didn’t waste the time he had left. He wasn’t angry or embittered: he accepted his death, he got on with his dying the same way as he had got on living, day by day, pressing forward, husbanding his energy.

Sonny’s time had come but neither he nor his community would ignore his impending mortality. Unlike the inclination to absence and denial that can occur in the Anglo-Saxon world, Sonny’s house filled with visitors who came to see him because he was dying.

Dying is an exhausting, self-centring act. Sonny, always a physically imposing man, shed his powers like a snake shedding skin. His world shrank to two rooms and Sonny knew he wouldn’t see the end of summer.

As Sonny’s fatherhood was ending my own was beginning. Our last words together on his deathbed were very ordinary, bland. “I’ll let you go son,” he said as I left to return to the city.

But our parting was fitting. There was no more mystery to share. No revelation. Our identities as father and son had already been written out in the deeds of our life together; Sonny changing my diaper as a child, not losing his temper at my teenage tantrums, encouraging me in my education, the summers we shared on building sites when I worked alongside him while still a student. And in all the countless ways he showed me in his craft how to be a man and father myself.

Sonny died just before dawn on the longest day of the year, at home, in the village of our forefathers. No-one called for help, or the “authorities.” He was already home with us. His body was washed and prepared for his coffin by his daughter and sister-in-law. He was laid out in his own front sitting room in an open coffin as his grandchildren, three, five and nine, played at the coffin’s feet.

His community, his relatives, some strangers even, came in great numbers to pray at his side, feast, talk, gossip about sheep prices or the stock market, and openly mark his death in countless handshakes and “Sorry for your trouble” utterances.

We waked together through the night with Sonny’s corpse to guard the passage out for his departing soul and to man the Gate of Chaos against Hades’ invading horde lest the supernatural world enter the land of the living. The whole community, a perpetual quorum: dying in each other’s lives and living on in each other’s deaths at each wake since.

It was blessing of a kind, an act of grace.

We give ourselves, our mortal presence, in such death-sharings, or we give nothing at all; all the rest of our powers—wealth, position, status—are useless.

To be human is to bear the burden of our own mortality and to strive, in grace, to help others carry theirs—sometimes lightly, sometimes with great courage. In accepting death into our lives, our community, we relearn the first and oldest lessons of humanity: how to be brave in irreversible sorrow; how to reach out to the dying, the dead and the bereaved; how to go on living no matter how great the rupture or loss; how to face your own death.

And how, like Sonny, to teach your children to face their own deaths too.

__________________________________

Adapted from My Father’s Wake: How the Irish Teach Us to Live, Love, and Die by Kevin Toolis. Copyright ©2018. Available from Da Capo Press, an imprint of Perseus Books, LLC, a subsidiary of Hachette Book Group, Inc.

When Did Americans Stop Marrying Cousins? Ask the World’s Largest Family Tree

0
0

The researchers then used this data set to test several genetic and historical hypotheses, showing “you can harness the hard work of so many people around the globe just documenting their own family history, and learn something about humanity,” said Yaniv Erlich, the chief science officer of MyHeritage, the parent company of Geni.com, and senior author of the paper.

Photo
In a 6,000-person family tree, individuals spanning seven generations are represented in green and marriages in red.Credit Columbia University

“It’s very impressive as a data collection and harmonization effort, and of course they have only scratched the surface of what it might have to offer,” said Philip Cohen, a sociology professor at the University of Maryland, who was not involved in the research.

The study is the latest example of scientists using big, crowdsourced data collected by private companies to do research. Last year, one study spearheaded by Ancestry.com mapped North American migrations. There have also been efforts to track food poisoning via Yelp reviews and drug usage via Instagram.

The trend raises new questions for researchers to think about, such as how representative such data are of populations at large, and whether commercial entities like Geni.com have vested interests, said Emily Klancher Merchant, a science and technology studies professor at the University of California, Davis.

“When private companies control the data and fund the research, they’re the ones gatekeeping what kind of science gets done,” she said. A DNA testing company might be more interested, for instance, in financing studies that discover sellable genetic markers rather than open-ended, basic research.

What’s interesting about genealogies, however, is that there aren’t much better ways to get data currently. In the past, researchers had to laboriously cobble together church records or local birth and death certificates to construct large family trees.

With crowdsourced genealogy, “we have the ability to connect a much more vast network of individuals and locations around the world, in a faster, cheaper way,” said Dr. Erlich, who is also a computer science professor at Columbia University.

Geni.com has millions of users worldwide, and the website allows its members to merge trees, in an effort to create a single, massive family tree for the world.

Dr. Erlich and his collaborators took steps to validate the company’s trees, then reported several new findings from the data, such as a lower heritability of life span than others have reported, and a greater likelihood of mothers to migrate than fathers.

But with all of their findings, it’s important to consider that not everyone is represented equally.

“It’s like reading a Jane Austen novel — it gives you lots of great insights, but you have to be careful about generalizing to society as a whole,” Dr. Cohen said.

In addition to likely underreporting infant mortalities and people who never bore children, genealogy data skews toward families that have the privilege of accessing and maintaining a detailed history, Dr. Merchant said. Sites like Geni.com require a paid membership to access all features. And families that were forced to migrate, through the slave trade or to flee persecution, are probably not as well-represented in their databases.

While these studies have limitations, they also offer opportunities. Biomedical research that marries family trees with genetic and health information could lead to new discoveries about heritable disease risk. In social sciences, genealogies merged with census and tax records could yield findings on things like inequality.

“As soon as you start linking these things, your analytical power goes way up — as do privacy risks,” Dr. Cohen said.

With any of these efforts, he added, it’s important to insist “that the norms of science, as far as transparency and openness, still apply.”

Continue reading the main story

Large Scale NoSQL Database Migration Under Fire

0
0

The following post describes how we migrated a large NoSql database from one vendor to another in production without any downtime or data loss. The following methodology has worked for us and has been proven to be safe and simple. I am sure that there are several techniques for migrating NoSql databases, but here are some guidelines for doing so with minimal risk, while maintaining the ability to rollback at almost any point in the process. This post is based on a talk I gave at DevopsDays in Nov 2017.

In AppsFlyer, we have many NoSql databases. AppsFlyer’s data processing pipeline updates an internal state as it processes the events we get from mobile devices. This state is mission critical and is processed at high throughput.

One of those databases is our app-installs database. This database gets a write hit for roughly every new install that Appsflyer attributes and gets an additional read hit whenever Appsflyer gets any launch or in-app event.
Some numbers (true to the time of writing):

  • ~2000 write IOPS
  • ~180000 read IOPS.
  • ~2 Billion records stored with a replication a factor of 3.

And, yes, it is mission critical.

But the time had come to replace it. It brought us very far and for that it deserves credit. But lately, as traffic and resulting throughput demand has raised, it just has not been able to deliver the stability we need.
Here are a few examples of instability: Scale out went out of control- we were using 45 r3.4xl instances. Daily backups could not keep up and took longer than a day to complete. XDCR stopped working. Losing a node (and adding one back) came at a performance hit and system instability (and when you have so many nodes on AWS the probability that something will go wrong gets very high). We were not happy with the answers we got from the paid support. These were all issues with which we could no longer live.

Migrating data is always hard. It is already hard when dealing with relational databases, and it is of course harder to do so in production, when the database(s) are in use. But doing so in NoSql databases is even more complex. This is because amongst the dozens of NoSql technologies out there, there is nothing in common. There is no common way to serialize or represent data, no standard query language, no schema, and when it comes to out-of-the-box migration tools- you are all alone.

So after some research and trials we chose Aerospike. We had a few months until our enterprise plan with Couchbase was due to expire and we started planning. Since most of the vendors do not encourage data migration out of their database for obvious reasons we were left with two basic options: Write a migration tool ourselves (‘out of band migration’) or use the application to migrate the data for us (‘in-band migration’). In many cases, choosing only one of those methods is sufficient for reasons I will enumerate shortly, but for other cases, you will have to use a combination thereof, as we did.

First phase (migrating inserts and updates):

The basic idea here is to use your application code to duplicate both inserts and updates (‘upserts’) to your new database. In other words, every write or update that is done for the old database is also done on your new database. This effectively starts the migration of all of your new data to the new cluster going forward. In many cases, a delete operation is considered a write here, at least from the application point of view.

This has a few implications:

  1. You are changing your application code. This may impact the application performance, introduce new bugs, and complicate existing code. Take those risks into account.
  2. The data being written into the new database is completely redundant. If (when) you discover a bug, it is okay to just wipe out the entire database and start over.
  3. The data is still being served from the good, old database so the risk of data loss/ corruption is low.

You can start with a small cluster and grow up (scale out) as you go. This is a good opportunity to practice scaling out.

Testing the first phase:
This is the time to do some data sanity tests:

Compare the migrated data between the two databases and verify your expectations. Many databases even have different semantics for records TTL if you use it.

Get to know the new technology:

  1. Look at the logs
  2. Set up automatic backups and restore from them!
  3. Inspect the actual database RAM and disk usage and see that it meets your initial sizing expectations.
  4. Time to wire up your metrics system and configure the proper alerting.
  5. And yes, you guessed it: test your alerts. Remember, after all, this is soon to be your shiny new production database.

Second phase:

This phase is a bit trickier, but still very simple. After you have enough confidence in the new database it is time to start serving from it. The idea here is to serve data from the new database, and if you fail to find something there, fall back to the old database and serve from there.

An important note here is to configure a metric that should count the instances when a record was not found in the new database but was found in the old one. This means that this record was not migrated yet by our application. It also means that if this metric stops rising for a long enough time, the migration is over.

This phase has a few implications:

  1. If your new database crashes, the application code should quickly failover to the old cluster.
  2. Note that the old database is still updated with all data. This means that it is still safe to go back and start from scratch if (when) you find that something went wrong.
  3. This mechanism might have latency and other performance implications. Nothing is for free, you see.
  4. This is the first true combat test of your new database. For safety, if possible, the use of the new database should be deployed gradually. Try to serve any non-VIP data from it first, such as your beta users, or even generated test data.
    The important thing is that the birth pains of your new database will affect your less important data.
  5. If you are in a hurry, trigger an explicit migration for every such a miss-and-hit. This means that for every miss-and-hit event, copy the missing record to the new database from the application. This practice should speed up the migration process. Make sure you can support the additional I/O load both on the databases and on the application servers.

Testing time again.

Whatever you tested after the first migration phase, reiterate. The load should be much higher. The database should be larger also, in order to support new read load capacity, etc… If you can afford it, it is highly recommended to leave it working for a few days, or weeks. Traffic may vary, and some errors might appear only after some mileage.

Is that it?

For quite a lot of use cases this is enough. If your records are short lived and/or the application naturally hits every record frequently, you just need to wait it out. Sit back and watch the miss-hit metric diminish to zero. In order to follow the frequency of that metric firing, we used our metering stack, which includes Statsd, Graphite and Grafana. The stack itself does not matter, as long as you can track the frequency of this metric over time.

A record-by record explicit migration:

For more sensitive/complex use cases, you will need to make sure that the entire set of records was migrated to the new database before the sun sets the old one. What we had to do now was to enumerate the entire set of keys from the old database and make sure we migrate the missing ones.

Amazon will stop selling Nest smart home devices

0
0

Nest cofounder Matt Rogers introducing the company's new connected home devices last fall.
AP

  • Amazon decided not to sell any of the newer products from Google's smart home division Nest.
  • Amazon currently sells a limited number of Nest products, but those will disappear from the site after it sells the inventory it has left.Nest decided to no longer work with Amazon selling the limited number of products it was selling on the site.
  • Amazon's move heats up its war with Google over the future of the smart home. The two are also battling over video devices and services.

It was an awkward phone call, but one the Nest team had been expecting.

After weeks of silence, Amazon's retail team informed Nest employees on a conference call late last year that it would not list any of the newer Nest products recently announced by the company, according to a person familiar with the call. The products in question include the latest Nest thermostat and the Nest Secure home security system, among others.

On that call, says the person, Amazon told Nest that the decision came from the top — and that it had nothing to do with the quality of Nest products, which had great reviews on Amazon. Nest employees who were on the call ended the discussion under the impression that the decision had come from Amazon CEO Jeff Bezos, although Amazon's retail team didn't explicitly say that at any point, according to a person familiar with the call.

As a result of Amazon's decision, Nest decided to stop selling any of its products through Amazon, meaning the limited number of Nest devices listed on Amazon today are expected to disappear from the site once current inventory is sold out, according to a person familiar with the matter.

Amazon's decision not to sell Nest products has huge implications as it strives to carve out a new computing platform — and as it continues to clash with Google over the future of computing.

The Nest Learning Thermostat is the smart home maker's flagship product.
George Frey/Getty Images

After missing out on smartphones and finding limited success with its line of Fire tablets, Amazon is betting big on Alexa as a new computing platform. Alexa is both Amazon's AI assistant and its platform for smart home gadgets, including connected lights, door locks, and music speakers. The company has gotten more aggressive with competitors recently — especially Google, the owner of Nest, which is Amazon's biggest competitor in the smart home with its own Google Assistant platform. Amazon also announced in February that it would buy Ring, the maker of camera-equipped doorbells and other connected home security devices, in a deal said to be worth about $1 billion.

Nest was warned of Amazon's decision, even before that fateful call. Representatives from Google, its sister company at the time under their Alphabet parent company, told Nest that they had heard from Amazon that the ecommerce giant had decided not to sell newer Nest devices. Google reabsorbed Nest last month.

A person familiar with Nest's thinking said the company decided to remove its current set of older products from Amazon because it wanted to be able to offer its full portfolio of devices, or nothing at all.

It's possible you may be able to find Nest products on Amazon in the future through Amazon's Marketplace program, which lets third-party retailers sell items through Amazon. But it's unclear if Amazon plans to restrict Nest sales from its Marketplace partners too.

Nest CEO Marwan Fawaz
Nest

A Nest spokesperson declined to comment. An Amazon spokesperson also declined to comment.

Amazon doesn't appear to be blocking sales of smart home products from companies other than Nest. For example, Lighthouse, an AI-powered connected camera made by a startup of the same name, is available on Amazon. Products from August, a connected home company best known for its smart door locks, are also available to buy on Amazon, along with products from several other smart home device manufacturers.

Amazon's move against Nest comes as it works to beef up its smart home ambitions after a successful holiday season for the Alexa assistant and its Echo hardware. Last month's Ring acquisition puts Amazon in a much better position to integrate its products with Alexa, accelerating its ability to compete with Google's own smart home ambitions.

Nest is Google's smart home products division. It makes devices like connected cameras, thermostats, smoke detectors, and security systems. Google bought Nest in 2014 in a $3.2 billion deal. Nest later became its own company after Google reorganized into the Alphabet conglomerate, only to be reabsorbed back into Google in February.

Nest is now under the same hardware division as the rest of Google's hardware products, which will create a streamlined organization from which Google can better compete against Amazon.

Amazon CEO Jeff Bezos
Mark Lennihan/AP

Amazon built up an early lead in voice-controlled computing thanks to Alexa and its line of Echo devices. But Google is rapidly catching up. Google Assistant, a rival to Amazon Alexa and the Google Home speakers, which compete with the Amazon Echo, are rapidly gaining market traction.

Amazon had about two-thirds of the smart speaker market towards end of 2017, but that figure doesn't reflect the full holiday shopping season, when Google likely gained more market share.

The stakes are huge. Both Amazon and Google are building out a new voice-powered operating system that can control everything in your life — from your lights to your garage door to the music and video you stream. Amazon's acquisition of Ring will give it a nice boost on the hardware side as it continues to build out Alexa's AI. Ring was already one of Nest's biggest competitors. Now it has the nearly-limitless funding needed from Amazon to go after its Google-backed rival.

The rivalry between Amazon and Google extends beyond the smart home, though.

In addition to ending sales of Nest products, Amazon does not sell other Google hardware like the Google Home Speaker or Pixel smartphone. In a seemingly retaliatory move, Google has blocked its YouTube native app from running on Amazon's Fire TV and Echo Show. (Google, for its part, has said the block is because Amazon products violate YouTube's terms of service.) Amazon will start selling Google's Chromecast streaming devices soon, which may help ease tensions between the companies and convince YouTube to bring its service back to the Fire TV and Echo Show.

Beyond hardware and apps, Amazon is also beefing up its digital ads business, a direct threat to Google's core business, as CNBC reported last year. Amazon and Google also compete in providing cloud computing services to companies, through its respective Amazon Web Services and Google Cloud divisions.

The Nest Secure home security system
Nest

Amazon's beef with Google isn't unique in an industry dominated by a small handful of giants. Amazon also had a rocky relationship with Apple. It took about two years for whatever was going on between the two companies to get resolved after Apple opened up the Apple TV to all third-party developers. Amazon started selling the Apple TV again last year, and it later added its video-streaming app to the Apple TV App Store.

While Amazon's decision to keep Nest products off its site may seem nefarious to some, it likely isn't illegal under US antitrust law, as Chris Sagers, a professor of law at Cleveland State University told Business Insider in an interview. Because Amazon doesn't have a monopoly in the connected home, the move isn't anticompetitive.

"It's probably not illegal," Sagers said. "It's ugly... but American law says even monopolists have broad freedom to choose with whom they deal."

Now read Business Insider Prime's exclusive report on the turmoil at Magic Leap, the $6 billion startup creating the next big computing platform.

Flying Taxis May Be Years Away, but the Groundwork Is Accelerating

0
0

In November, Boeing acquired Aurora Flight Sciences, a company specializing in flight systems for pilotless aircraft, for an undisclosed sum. Before the acquisition, Aurora had been working with Uber to develop a flying taxi. And Joby Aviation, a start-up in Santa Cruz, Calif., building its own air taxi, said this month that it had raised $100 million in venture funding from a consortium of investors including the venture capital arms of Intel, Toyota Motor and JetBlue Airways.

“This is the natural progression of the vehicles we make,” said Ben Bridge, head of global business for Airbus Helicopters. “We want a seat at the table and a voice in the conversation that is happening.”

Flying cars even played a bit role in the recently settled legal fight over trade secrets between Uber and Waymo, the self-driving car service spun out of Google.

In court testimony this month, Travis Kalanick, Uber’s former chief executive, said he had heard that Larry Page — the chief executive of Waymo’s parent company, Alphabet, who has a side project building new types of aircraft — was upset because Uber was “doing their thing” with flying cars.

Whatever you imagine a flying car to be — stop. What these companies envision is something like a helicopter but much quieter and more affordable. Think of a hobbyist’s drone, but big enough to fit people. It would, in theory, be welcome in urban environments and affordable to more than well-heeled businesspeople. At least, that’s the dream.

Before there can be too much enthusiasm for these flying taxi services, it’s worth noting that self-driving cars have yet to turn into a notable business for anyone, despite about a decade of research at tech giants like Google and billions in investment from Silicon Valley and the auto industry.

Regulators are just starting to agree on rules for large-scale tests of self-driving cars on public roads. How would they deal with flying taxis? The details of the future service are far — very far — from being ironed out.

Photo
A rendering of the CityAirbus, which can carry up to four passengers and which Airbus plans to deploy in 2023.Credit Airbus

Still, there are some reasons for the new enthusiasm. Battery improvements and the wide use of drones have spawned technological breakthroughs. The taxis would take off and land vertically like a helicopter, so they’d take up less room. Because they would be battery-powered, they would be more environmentally friendly.

For now, Airbus executives hope to gain from Blade’s experience with an app that allows customers to reserve a seat on a helicopter. Airbus is expected to invest up to $15 million in Blade, which would represent about a 10 percent stake in the company, according to a person who is familiar with the transaction but not permitted to discuss the investment details publicly.

Both companies see helicopters as an intermediate step until a new type of aircraft and taxi service hits the market. Rob Wiesenthal, Blade’s chief executive, said a quieter and less expensive alternative to helicopters “opens up a whole new world.”

Airbus said it was preparing for a test flight by year-end for its CityAirbus aircraft, which carries up to four passengers and can reach a cruising speed of about 75 miles per hour. It plans to deploy the CityAirbus in 2023.

Uber has said it expects to begin testing of its urban air taxis in the Dallas-Fort Worth area, Los Angeles and Dubai in 2020. The company has landed exclusive deals for vertical takeoff and landing spots with real estate companies, including in the Dallas-Fort Worth area.

Uber declared its interest with its Uber Elevate initiative in 2016, forecasting a future when it could offer 15-minute Uber Air rides from San Francisco to San Jose for $20. The 50-mile ride with Uber Pool, its car-pooling option, usually takes about two hours in rush-hour traffic and costs $83.

It did not put a date on that future.

In a concept video for Uber Elevate, a woman opens the Uber app, which gives her a choice of taking a ride with Uber Pool, UberX or Uber Air. She enters a nearby building and rides the elevator to the top floor, where she boards an air taxi in Uber Skyport.

Uber is also establishing performance guidelines for its flying taxis, including asking that the new aircraft make only one-quarter of the noise of a small four-seat helicopter. Uber also has supplier deals in place with five manufacturers, including Textron’s Bell Helicopter.

“We believe the best way to get this idea off the ground is to partner with world-class companies and stakeholders that have a diverse set of specializations,” an Uber spokesman, Matt Wing, said in a statement.

Daniel Wiegand, a co-founder and the chief executive of the German air taxi company Lilium, said investors considered him totally crazy when he pitched the company in 2015.

“It has completely changed,” Mr. Wiegand said. Like many of his competitors, he dismisses the phrase “flying cars” because other companies are working on cars that fly as well as drive on the road. In September, Lilium announced that it had raised $90 million from investors, including the Chinese internet giant Tencent.

And then there is Google — or, to be more specific, Mr. Page, Google’s co-founder and Alphabet’s chief executive. He has invested part of his personal wealth in Kitty Hawk, a start-up that has demonstrated a personal aircraft that flies using propellers over water. A Kitty Hawk division called Zee Aero is also reportedly working on an air taxi concept aircraft.

Mr. Kalanick said he told Mr. Page that he believed a flying service “sounds pretty cool,” even though Uber wasn’t building one. But once the aircraft are available, he said, “I will make sure that our 50 million or so customers at the time can push a button and get a flying car.”

Continue reading the main story
Viewing all 25817 articles
Browse latest View live




Latest Images