Quantcast
Channel: Hacker News
Viewing all 25817 articles
Browse latest View live

Rowhammer-like attack on SSDs can provide root privileges to attacker

$
0
0

The way flash NAND memory of SSDs works makes it possible for an attacker with write access to get root privileges on a system, researchers from IBM demonstrated during the WOOT ’17 conference. The method works similar to that of the ‘Rowhammer’ attack.

Rowhammer is a vulnerability in DRAM memory that allows an attacker to manipulate memory without accessing it. By repeatedly reading a specific memory location somewhere else in memory a bit can ‘flip’. This means that a ‘1’ can flip to a ‘0’ or vice versa. By flipping bits it’s eventually possible to get read and write access to all physical memory after which it’s possible to get kernel rights.

The Rowhammer attack was already demonstrated in 2015. The researchers wanted to check whether a similar attack was also possible against SSD drives with MLC NAND flash memory.

“DRAM is not the only place that holds sensitive data that is essential to the correct working of security primitives implemented in software,” the researchers write in their report. Also through the filesystem used by the operating system it’s possible to gain access to sensitive data.

For their scenario, the researchers assume that the victim runs a filesystem on a SSD disk that consists of MLC NAND flash memory. To perform the attack, an attacker needs to have ‘unprivileged’ rights to the system. The attacker doesn’t need to have physical access to the system,  it could also be a server where users can login with a regular account. This account provides the attacker with limited write access.

Just like with the Rowhammer attack on DRAM memory, also flash chips of SSDs can be manipulated on a similar way and allows an attacker to elevate his rights on the system. To protect against the attack, a SSD can be encrypted. For the future the attackers hope to demonstrate a full system attack.

Several hardware manufacturers such as Apple released updates for their devices in response to the Rowhammer attack. The researchers from IBM don’t state whether they informed hardware manufacturers about their attack.


Puppeteer: Headless Chrome Node API

$
0
0

README.md

API | FAQ | Contributing

Puppeteer is a Node library which provides a high-level API to control headless Chrome over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome.

What can I do?

Most things that you can do manually in the browser can be done using Puppeteer! Here are a few examples to get you started:

  • Generate screenshots and PDFs of pages.
  • Crawl a SPA and generate pre-rendered content (i.e. "SSR").
  • Scrape content from websites.
  • Automate form submission, UI testing, keyboard input, etc.
  • Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.
  • Capture a timeline trace of your site to help diagnose performance issues.

Getting Started

Installation

Puppeteer requires Node version 7.10 or greater

To use Puppeteer in your project, run:

yarn add puppeteer
# or "npm i puppeteer"

Note: When you install Puppeteer, it downloads a recent version of Chromium (~71Mb Mac, ~90Mb Linux, ~110Mb Win) that is guaranteed to work with the API.

Usage

Puppeteer will be familiar to using other browser testing frameworks. You create an instance of Browser, open pages, and then manipulate them with Puppeteer's API.

Example - navigating to https://example.com and saving a screenshot as example.png:

constpuppeteer=require('puppeteer');

(async() => {constbrowser=awaitpuppeteer.launch();constpage=awaitbrowser.newPage();awaitpage.goto('https://example.com');awaitpage.screenshot({path:'example.png'});browser.close();
})();

Puppeteer sets an initial page size to 800px x 600px, which defines the screenshot size. The page size can be customized with Page.setViewport().

Example - create a PDF.

constpuppeteer=require('puppeteer');

(async() => {constbrowser=awaitpuppeteer.launch();constpage=awaitbrowser.newPage();awaitpage.goto('https://news.ycombinator.com', {waitUntil:'networkidle'});awaitpage.pdf({path:'hn.pdf', format:'A4'});browser.close();
})();

See Page.pdf() for more information about creating pdfs.

Default runtime settings

1. Uses Headless mode

Puppeteer launches Chromium in headless mode. To launch a full version of Chromium, set the 'headless' option when creating a browser:

constbrowser=awaitpuppeteer.launch({headless:false});

2. Runs a bundled version of Chromium

By default, Puppeteer downloads and uses a specific version of Chromium so its API is guaranteed to work out of the box. To use Puppeteer with a different version of Chrome, pass in the executable's path when creating a Browser instance:

constbrowser=awaitpuppeteer.launch({executablePath:'/path/to/Chrome'});

See Puppeteer.launch() for more information.

3. Creates a fresh user profile

Puppeteer creates its own Chromium user profile which it cleans up on every run.

API Documentation

Explore the API documentation and examples to learn more.

Contributing to Puppeteer

Check out contributing guide to get an overview of Puppeteer development.

Q: Which Chromium version does Puppeteer use?

Look for chromium_revision in package.json.

Puppeteer bundles Chromium to ensure that the latest features it uses are guaranteed to be available. As the DevTools protocol and browser improve over time, Puppeteer will be updated to depend on newer versions of Chromium.

Q: What is the difference between Puppeteer, Selenium / WebDriver, and PhantomJS?

Selenium / WebDriver is a well-established cross-browser API that is useful for testing cross-browser support.

Puppeteer works only with Chrome. However, many teams only run unit tests with a single browser (e.g. PhantomJS). In non-testing use cases, Puppeteer provides a powerful but simple API because it's only targeting one browser that enables you to rapidly develop automation scripts.

Puppeteer uses the latest versions of Chromium.

Q: Who maintains Puppeteer?

The Chrome DevTools team maintains the library, but we'd love your help and expertise on the project! See Contributing.

Q: Why is the Chrome team building Puppeteer?

The goals of the project are simple:

  • Provide a slim, canonical library that highlights the capabilities of the DevTools Protocol.
  • Provide a reference implementation for similar testing libraries. Eventually, these other frameworks could adopt Puppeteer as their foundational layer.
  • Grow the adoption of headless/automated browser testing.
  • Help dogfood new DevTools Protocol features...and catch bugs!
  • Learn more about the pain points of automated browser testing and help fill those gaps.

Q: How does Puppeteer compare with other headless Chrome projects?

The past few months have brought several new libraries for automating headless Chrome. As the team authoring the underlying DevTools Protocol, we're excited to witness and support this flourishing ecosystem.

We've reached out to a number of these projects to see if there are opportunities for collaboration, and we're happy to do what we can to help.

ProtonMail Now Supports Bitcoin Payments

$
0
0

Starting from version 3.11 of ProtonMail, it is now possible to pay for premium ProtonMail secure email account using Bitcoin. We have designed the payment system to work seamlessly so that Bitcoin payments are automatically converted to ProtonMail credits which can be used to pay for upgrades, pay invoices, or top up your account. This feature was long overdue, so we are glad to finally introduce this.

Long time followers of ProtonMail will know that we actually have a deep connection with Bitcoin, going all the way back to the very beginning of ProtonMail’s story. We have always been quite interested in cryptocurrencies and blockchain, as they empower the same principles that inspired us to create ProtonMail. These are the principles of freedom, privacy, and an Internet ecosystem where all have an equal opportunity to thrive, free from many of the artificial constructs that control commerce today.

Our appreciation of cryptocurrencies deepened after we had a first hand experience with the potential of Bitcoin. Back in 2014 when ProtonMail first launched, our first community fundraiser was infamously frozen by PayPal, leaving us without access to the funds necessary to run the service. However, thanks to Bitcoin and the support of the Bitcoin community, we were able to continue to receive funds via Bitcoin ensuring the success of the project.

Three years on, ProtonMail has come a long way, and we are still proud to have many users from the global cryptocurrency community. As we mentioned in our guide on how to buy Bitcoins, secure email can actually play an important role in keeping Bitcoins secure, either as an email address to use with online Bitcoin wallets or exchanges, or for the storage of sensitive data.

Over the past three years, we have had an unofficial way of accepting Bitcoins which involved manually sending BTC to our Bitcoin donation address, and manually confirming the transaction with our support team to receive account credits. However, the general increase in the number of Bitcoin transactions over the years has made that approach unfeasible and made development of an automated system necessary. We feel this is indicative of a broader trend, where the increased mainstream interest in Bitcoin will make it harder for merchants not to support Bitcoin, which will lead to more Bitcoin support, more transactions, and perhaps the continued appreciation of Bitcoin prices.

From a business perspective, we have witnessed the fragility and uncertainty of the traditional financial sector, with PayPal, credit card, and bank account freezes, fraud, or hacks becoming increasingly common. For any operating business, cryptocurrencies are an important form of diversification. Whereas companies in the past might have held multiple fiat currencies distributed cross multiple financial institutions as insurance, ProtonMail is now also holding a significant portion of our reserves in Bitcoin. Ultimately, we believe having a more significant portion of our revenue via Bitcoin or other cryptocurrencies actually reduces our business risk, by providing more redundancy in the event of the failure of more traditional payment methods. While this is fairly progressive thinking today, we believe that with time, more businesses will also adopt this view.

We’re excited to further deepen our connections with the cryptocurrency community with this step, and we look forward to doing our part to ensure that the brighter future promised by blockchain does become a reality. In the coming quarters, we plan to continue improving cryptocurrency support as part of our mission to build the most secure email service ever. Let us know which coins or tokens you would like us to support next on Twitter, Facebook, Reddit or Instagram.

Best Regards,
The ProtonMail Team

Andrew Ng is raising a $150M AI Fund

$
0
0

We knew that Andrew Ng had more than just a series of deep learning courses up his sleeve when he announced the first phase of his deeplearning.ai last week. It’s clear now that the turn of Ng’s three part act is a $150 million venture capital fund, first noted by PEHub, targeting AI investments.

Ng, who formerly founded Google’s Brain Team and served as chief scientist at Baidu has long evangelized the benefits AI could bring to the world. During an earlier conversation, Ng told me that his personal goal is to help bring about an AI-powered society. It would follow that education via his deep learning classes is one step of that and providing capital and other resources is another.

2017 has been a particularly active year for starting AI-focused venture capital funds. In the last few months we have seen Google roll out Gradient Ventures, Basis Set Ventures hall in $136 million, Element.AI raise $102 million, Microsoft Ventures start its own AI fund and Toyota corral $100 million for AI investment.

It’s unclear at this point how Ng’s AI Fund will differentiate from the pack. Many of these funds are putting time and resources into securing data sets, technical mentors and advanced simulation tools to support the unique needs of AI startups. Of course Ng’s name recognition and network should help ensure solid deal flow and enable Ng to poach and train talent for startups in need of scarce deep learning engineers.

I’ve sent a note to Andrew and we will update this post if and when we get more details.

Featured Image: Dawn Endico/Flickr UNDER A CC BY-SA 2.0 LICENSE

Facebook – You are the Product

$
0
0

You Are the Product

John Lanchester

  • The Attention Merchants: From the Daily Newspaper to Social Media, How Our Time and Attention Is Harvested and Sold by Tim Wu
    Atlantic, 416 pp, £20.00, January, ISBN 978 1 78239 482 2
  • Chaos Monkeys: Inside the Silicon Valley Money Machine by Antonio García Martínez
    Ebury, 528 pp, £8.99, June, ISBN 978 1 78503 455 8
  • Move Fast and Break Things: How Facebook, Google and Amazon have Cornered Culture and What It Means for All of Us by Jonathan Taplin
    Macmillan, 320 pp, £18.99, May, ISBN 978 1 5098 4769 3

At the end of June, Mark Zuckerberg announced that Facebook had hit a new level: two billion monthly active users. That number, the company’s preferred ‘metric’ when measuring its own size, means two billion different people used Facebook in the preceding month. It is hard to grasp just how extraordinary that is. Bear in mind that thefacebook – its original name – was launched exclusively for Harvard students in 2004. No human enterprise, no new technology or utility or service, has ever been adopted so widely so quickly. The speed of uptake far exceeds that of the internet itself, let alone ancient technologies such as television or cinema or radio.

Also amazing: as Facebook has grown, its users’ reliance on it has also grown. The increase in numbers is not, as one might expect, accompanied by a lower level of engagement. More does not mean worse – or worse, at least, from Facebook’s point of view. On the contrary. In the far distant days of October 2012, when Facebook hit one billion users, 55 per cent of them were using it every day. At two billion, 66 per cent are. Its user base is growing at 18 per cent a year – which you’d have thought impossible for a business already so enormous. Facebook’s biggest rival for logged-in users is YouTube, owned by its deadly rival Alphabet (the company formerly known as Google), in second place with 1.5 billion monthly users. Three of the next four biggest apps, or services, or whatever one wants to call them, are WhatsApp, Messenger and Instagram, with 1.2 billion, 1.2 billion, and 700 million users respectively (the Chinese app WeChat is the other one, with 889 million). Those three entities have something in common: they are all owned by Facebook. No wonder the company is the fifth most valuable in the world, with a market capitalisation of $445 billion.

Zuckerberg’s news about Facebook’s size came with an announcement which may or may not prove to be significant. He said that the company was changing its ‘mission statement’, its version of the canting pieties beloved of corporate America. Facebook’s mission used to be ‘making the world more open and connected’. A non-Facebooker reading that is likely to ask: why? Connection is presented as an end in itself, an inherently and automatically good thing. Is it, though? Flaubert was sceptical about trains because he thought (in Julian Barnes’s paraphrase) that ‘the railway would merely permit more people to move about, meet and be stupid.’ You don’t have to be as misanthropic as Flaubert to wonder if something similar isn’t true about connecting people on Facebook. For instance, Facebook is generally agreed to have played a big, perhaps even a crucial, role in the election of Donald Trump. The benefit to humanity is not clear. This thought, or something like it, seems to have occurred to Zuckerberg, because the new mission statement spells out a reason for all this connectedness. It says that the new mission is to ‘give people the power to build community and bring the world closer together’.

Hmm. Alphabet’s mission statement, ‘to organise the world’s information and make it universally accessible and useful’, came accompanied by the maxim ‘Don’t be evil,’ which has been the source of a lot of ridicule: Steve Jobs called it ‘bullshit’.[1] Which it is, but it isn’t only bullshit. Plenty of companies, indeed entire industries, base their business model on being evil. The insurance business, for instance, depends on the fact that insurers charge customers more than their insurance is worth; that’s fair enough, since if they didn’t do that they wouldn’t be viable as businesses. What isn’t fair is the panoply of cynical techniques that many insurers use to avoid, as far as possible, paying out when the insured-against event happens. Just ask anyone who has had a property suffer a major mishap. It’s worth saying ‘Don’t be evil,’ because lots of businesses are. This is especially an issue in the world of the internet. Internet companies are working in a field that is poorly understood (if understood at all) by customers and regulators. The stuff they’re doing, if they’re any good at all, is by definition new. In that overlapping area of novelty and ignorance and unregulation, it’s well worth reminding employees not to be evil, because if the company succeeds and grows, plenty of chances to be evil are going to come along.

Google and Facebook have both been walking this line from the beginning. Their styles of doing so are different. An internet entrepreneur I know has had dealings with both companies. ‘YouTube knows they have lots of dirty things going on and are keen to try and do some good to alleviate it,’ he told me. I asked what he meant by ‘dirty’. ‘Terrorist and extremist content, stolen content, copyright violations. That kind of thing. But Google in my experience knows that there are ambiguities, moral doubts, around some of what they do, and at least they try to think about it. Facebook just doesn’t care. When you’re in a room with them you can tell. They’re’ – he took a moment to find the right word – ‘scuzzy’.

That might sound harsh. There have, however, been ethical problems and ambiguities about Facebook since the moment of its creation, a fact we know because its creator was live-blogging at the time. The scene is as it was recounted in Aaron Sorkin’s movie about the birth of Facebook, The Social Network. While in his first year at Harvard, Zuckerberg suffered a romantic rebuff. Who wouldn’t respond to this by creating a website where undergraduates’ pictures are placed side by side so that users of the site can vote for the one they find more attractive? (The film makes it look as if it was only female undergraduates: in real life it was both.) The site was called Facemash. In the great man’s own words, at the time:

I’m a little intoxicated, I’m not gonna lie. So what if it’s not even 10 p.m. and it’s a Tuesday night? What? The Kirkland dormitory facebook is open on my desktop and some of these people have pretty horrendous facebook pics. I almost want to put some of these faces next to pictures of some farm animals and have people vote on which is the more attractive … Let the hacking begin.

As Tim Wu explains in his energetic and original new book The Attention Merchants, a ‘facebook’ in the sense Zuckerberg uses it here ‘traditionally referred to a physical booklet produced at American universities to promote socialisation in the way that “Hi, My Name Is” stickers do at events; the pages consisted of rows upon rows of head shots with the corresponding name’. Harvard was already working on an electronic version of its various dormitory facebooks. The leading social network, Friendster, already had three million users. The idea of putting these two things together was not entirely novel, but as Zuckerberg said at the time, ‘I think it’s kind of silly that it would take the University a couple of years to get around to it. I can do it better than they can, and I can do it in a week.’

Wu argues that capturing and reselling attention has been the basic model for a large number of modern businesses, from posters in late 19th-century Paris, through the invention of mass-market newspapers that made their money not through circulation but through ad sales, to the modern industries of advertising and ad-funded TV. Facebook is in a long line of such enterprises, though it might be the purest ever example of a company whose business is the capture and sale of attention. Very little new thinking was involved in its creation. As Wu observes, Facebook is ‘a business with an exceedingly low ratio of invention to success’. What Zuckerberg had instead of originality was the ability to get things done and to see the big issues clearly. The crucial thing with internet start-ups is the ability to execute plans and to adapt to changing circumstances. It’s Zuck’s skill at doing that – at hiring talented engineers, and at navigating the big-picture trends in his industry – that has taken his company to where it is today. Those two huge sister companies under Facebook’s giant wing, Instagram and WhatsApp, were bought for $1 billion and $19 billion respectively, at a point when they had no revenue. No banker or analyst or sage could have told Zuckerberg what those acquisitions were worth; nobody knew better than he did. He could see where things were going and help make them go there. That talent turned out to be worth several hundred billion dollars.

Jesse Eisenberg’s brilliant portrait of Zuckerberg in The Social Network is misleading, as Antonio García Martínez, a former Facebook manager, argues in Chaos Monkeys, his entertainingly caustic book about his time at the company. The movie Zuckerberg is a highly credible character, a computer genius located somewhere on the autistic spectrum with minimal to non-existent social skills. But that’s not what the man is really like. In real life, Zuckerberg was studying for a degree with a double concentration in computer science and – this is the part people tend to forget – psychology. People on the spectrum have a limited sense of how other people’s minds work; autists, it has been said, lack a ‘theory of mind’. Zuckerberg, not so much. He is very well aware of how people’s minds work and in particular of the social dynamics of popularity and status. The initial launch of Facebook was limited to people with a Harvard email address; the intention was to make access to the site seem exclusive and aspirational. (And also to control site traffic so that the servers never went down. Psychology and computer science, hand in hand.) Then it was extended to other elite campuses in the US. When it launched in the UK, it was limited to Oxbridge and the LSE. The idea was that people wanted to look at what other people like them were doing, to see their social networks, to compare, to boast and show off, to give full rein to every moment of longing and envy, to keep their noses pressed against the sweet-shop window of others’ lives.

This focus attracted the attention of Facebook’s first external investor, the now notorious Silicon Valley billionaire Peter Thiel. Again, The Social Network gets it right: Thiel’s $500,000 investment in 2004 was crucial to the success of the company. But there was a particular reason Facebook caught Thiel’s eye, rooted in a byway of intellectual history. In the course of his studies at Stanford – he majored in philosophy – Thiel became interested in the ideas of the US-based French philosopher René Girard, as advocated in his most influential book, Things Hidden since the Foundation of the World. Girard’s big idea was something he called ‘mimetic desire’. Human beings are born with a need for food and shelter. Once these fundamental necessities of life have been acquired, we look around us at what other people are doing, and wanting, and we copy them. In Thiel’s summary, the idea is ‘that imitation is at the root of all behaviour’.

Girard was a Christian, and his view of human nature is that it is fallen. We don’t know what we want or who we are; we don’t really have values and beliefs of our own; what we have instead is an instinct to copy and compare. We are homo mimeticus. ‘Man is the creature who does not know what to desire, and who turns to others in order to make up his mind. We desire what others desire because we imitate their desires.’ Look around, ye petty, and compare. The reason Thiel latched onto Facebook with such alacrity was that he saw in it for the first time a business that was Girardian to its core: built on people’s deep need to copy. ‘Facebook first spread by word of mouth, and it’s about word of mouth, so it’s doubly mimetic,’ Thiel said. ‘Social media proved to be more important than it looked, because it’s about our natures.’ We are keen to be seen as we want to be seen, and Facebook is the most popular tool humanity has ever had with which to do that.

*

The view of human nature implied by these ideas is pretty dark. If all people want to do is go and look at other people so that they can compare themselves to them and copy what they want – if that is the final, deepest truth about humanity and its motivations – then Facebook doesn’t really have to take too much trouble over humanity’s welfare, since all the bad things that happen to us are things we are doing to ourselves. For all the corporate uplift of its mission statement, Facebook is a company whose essential premise is misanthropic. It is perhaps for that reason that Facebook, more than any other company of its size, has a thread of malignity running through its story. The high-profile, tabloid version of this has come in the form of incidents such as the live-streaming of rapes, suicides, murders and cop-killings. But this is one of the areas where Facebook seems to me relatively blameless. People live-stream these terrible things over the site because it has the biggest audience; if Snapchat or Periscope were bigger, they’d be doing it there instead.

In many other areas, however, the site is far from blameless. The highest-profile recent criticisms of the company stem from its role in Trump’s election. There are two components to this, one of them implicit in the nature of the site, which has an inherent tendency to fragment and atomise its users into like-minded groups. The mission to ‘connect’ turns out to mean, in practice, connect with people who agree with you. We can’t prove just how dangerous these ‘filter bubbles’ are to our societies, but it seems clear that they are having a severe impact on our increasingly fragmented polity. Our conception of ‘we’ is becoming narrower.

This fragmentation created the conditions for the second strand of Facebook’s culpability in the Anglo-American political disasters of the last year. The portmanteau terms for these developments are ‘fake news’ and ‘post-truth’, and they were made possible by the retreat from a general agora of public debate into separate ideological bunkers. In the open air, fake news can be debated and exposed; on Facebook, if you aren’t a member of the community being served the lies, you’re quite likely never to know that they are in circulation. It’s crucial to this that Facebook has no financial interest in telling the truth. No company better exemplifies the internet-age dictum that if the product is free, you are the product. Facebook’s customers aren’t the people who are on the site: its customers are the advertisers who use its network and who relish its ability to direct ads to receptive audiences. Why would Facebook care if the news streaming over the site is fake? Its interest is in the targeting, not in the content. This is probably one reason for the change in the company’s mission statement. If your only interest is in connecting people, why would you care about falsehoods? They might even be better than the truth, since they are quicker to identify the like-minded. The newfound ambition to ‘build communities’ makes it seem as if the company is taking more of an interest in the consequence of the connections it fosters.

Fake news is not, as Facebook has acknowledged, the only way it was used to influence the outcome of the 2016 presidential election. On 6 January 2017 the director of national intelligence published a report saying that the Russians had waged an internet disinformation campaign to damage Hillary Clinton and help Trump. ‘Moscow’s influence campaign followed a Russian messaging strategy that blends covert intelligence operations – such as cyber-activity – with overt efforts by Russian government agencies, state-funded media, third-party intermediaries, and paid social media users or “trolls”,’ the report said. At the end of April, Facebook got around to admitting this (by then) fairly obvious truth, in an interesting paper published by its internal security division. ‘Fake news’, they argue, is an unhelpful, catch-all term because misinformation is in fact spread in a variety of ways:

Information (or Influence) Operations– Actions taken by governments or organised non-state actors to distort domestic or foreign political sentiment.

False News– News articles that purport to be factual, but which contain intentional misstatements of fact with the intention to arouse passions, attract viewership, or deceive.

False Amplifiers– Co-ordinated activity by inauthentic accounts with the intent of manipulating political discussion (e.g. by discouraging specific parties from participating in discussion, or amplifying sensationalistic voices over others).

Disinformation– Inaccurate or manipulated information/content that is spread intentionally. This can include false news, or it can involve more subtle methods, such as false flag operations, feeding inaccurate quotes or stories to innocent intermediaries, or knowingly amplifying biased or misleading information.

The company is promising to treat this problem or set of problems as seriously as it treats such other problems as malware, account hacking and spam. We’ll see. One man’s fake news is another’s truth-telling, and Facebook works hard at avoiding responsibility for the content on its site – except for sexual content, about which it is super-stringent. Nary a nipple on show. It’s a bizarre set of priorities, which only makes sense in an American context, where any whiff of explicit sexuality would immediately give the site a reputation for unwholesomeness. Photos of breastfeeding women are banned and rapidly get taken down. Lies and propaganda are fine.

The key to understanding this is to think about what advertisers want: they don’t want to appear next to pictures of breasts because it might damage their brands, but they don’t mind appearing alongside lies because the lies might be helping them find the consumers they’re trying to target. In Move Fast and Break Things, his polemic against the ‘digital-age robber barons’, Jonathan Taplin points to an analysis on Buzzfeed: ‘In the final three months of the US presidential campaign, the top-performing fake election news stories on Facebook generated more engagement than the top stories from major news outlets such as the New York Times, Washington Post, Huffington Post, NBC News and others.’ This doesn’t sound like a problem Facebook will be in any hurry to fix.

The fact is that fraudulent content, and stolen content, are rife on Facebook, and the company doesn’t really mind, because it isn’t in its interest to mind. Much of the video content on the site is stolen from the people who created it. An illuminating YouTube video from Kurzgesagt, a German outfit that makes high-quality short explanatory films, notes that in 2015, 725 of Facebook’s top one thousand most viewed videos were stolen. This is another area where Facebook’s interests contradict society’s. We may collectively have an interest in sustaining creative and imaginative work in many different forms and on many platforms. Facebook doesn’t. It has two priorities, as Martínez explains in Chaos Monkeys: growth and monetisation. It simply doesn’t care where the content comes from. It is only now starting to care about the perception that much of the content is fraudulent, because if that perception were to become general, it might affect the amount of trust and therefore the amount of time people give to the site.

Zuckerberg himself has spoken up on this issue, in a Facebook post addressing the question of ‘Facebook and the election’. After a certain amount of boilerplate bullshit (‘Our goal is to give every person a voice. We believe deeply in people’), he gets to the nub of it. ‘Of all the content on Facebook, more than 99 per cent of what people see is authentic. Only a very small amount is fake news and hoaxes.’ More than one Facebook user pointed out that in their own news feed, Zuckerberg’s post about authenticity ran next to fake news. In one case, the fake story pretended to be from the TV sports channel ESPN. When it was clicked on, it took users to an ad selling a diet supplement. As the writer Doc Searls pointed out, it’s a double fraud, ‘outright lies from a forged source’, which is quite something to have right slap next to the head of Facebook boasting about the absence of fraud. Evan Williams, co-founder of Twitter and founder of the long-read specialist Medium, found the same post by Zuckerberg next to a different fake ESPN story and another piece of fake news purporting to be from CNN, announcing that Congress had disqualified Trump from office. When clicked-through, that turned out to be from a company offering a 12-week programme to strengthen toes. (That’s right: strengthen toes.) Still, we now know that Zuck believes in people. That’s the main thing.

*

A neutral observer might wonder if Facebook’s attitude to content creators is sustainable. Facebook needs content, obviously, because that’s what the site consists of: content that other people have created. It’s just that it isn’t too keen on anyone apart from Facebook making any money from that content. Over time, that attitude is profoundly destructive to the creative and media industries. Access to an audience – that unprecedented two billion people – is a wonderful thing, but Facebook isn’t in any hurry to help you make money from it. If the content providers all eventually go broke, well, that might not be too much of a problem. There are, for now, lots of willing providers: anyone on Facebook is in a sense working for Facebook, adding value to the company. In 2014, the New York Times did the arithmetic and found that humanity was spending 39,757 collective years on the site, every single day. Jonathan Taplin points out that this is ‘almost fifteen million years of free labour per year’. That was back when it had a mere 1.23 billion users.

Taplin has worked in academia and in the film industry. The reason he feels so strongly about these questions is that he started out in the music business, as manager of The Band, and was on hand to watch the business being destroyed by the internet. What had been a $20 billion industry in 1999 was a $7 billion industry 15 years later. He saw musicians who had made a good living become destitute. That didn’t happen because people had stopped listening to their music – more people than ever were listening to it – but because music had become something people expected to be free. YouTube is the biggest source of music in the world, playing billions of tracks annually, but in 2015 musicians earned less from it and from its ad-supported rivals than they earned from sales of vinyl. Not CDs and recordings in general: vinyl.

Something similar has happened in the world of journalism. Facebook is in essence an advertising company which is indifferent to the content on its site except insofar as it helps to target and sell advertisements. A version of Gresham’s law is at work, in which fake news, which gets more clicks and is free to produce, drives out real news, which often tells people things they don’t want to hear, and is expensive to produce. In addition, Facebook uses an extensive set of tricks to increase its traffic and the revenue it makes from targeting ads, at the expense of the news-making institutions whose content it hosts. Its news feed directs traffic at you based not on your interests, but on how to make the maximum amount of advertising revenue from you. In September 2016, Alan Rusbridger, the former editor of the Guardian, told a Financial Times conference that Facebook had ‘sucked up $27 million’ of the newspaper’s projected ad revenue that year. ‘They are taking all the money because they have algorithms we don’t understand, which are a filter between what we do and how people receive it.’

This goes to the heart of the question of what Facebook is and what it does. For all the talk about connecting people, building community, and believing in people, Facebook is an advertising company. Martínez gives the clearest account both of how it ended up like that, and how Facebook advertising works. In the early years of Facebook, Zuckerberg was much more interested in the growth side of the company than in the monetisation. That changed when Facebook went in search of its big payday at the initial public offering, the shining day when shares in a business first go on sale to the general public. This is a huge turning-point for any start-up: in the case of many tech industry workers, the hope and expectation associated with ‘going public’ is what attracted them to their firm in the first place, and/or what has kept them glued to their workstations. It’s the point where the notional money of an early-days business turns into the real cash of a public company.

Martínez was there at the very moment when Zuck got everyone together to tell them they were going public, the moment when all Facebook employees knew that they were about to become rich:

I had chosen a seat behind a detached pair, who on further inspection turned out to be Chris Cox, head of FB product, and Naomi Gleit, a Harvard grad who joined as employee number 29, and was now reputed to be the current longest-serving employee other than Mark.

Naomi, between chats with Cox, was clicking away on her laptop, paying little attention to the Zuckian harangue. I peered over her shoulder at her screen. She was scrolling down an email with a number of links, and progressively clicking each one into existence as another tab on her browser. Clickathon finished, she began lingering on each with an appraiser’s eye. They were real estate listings, each for a different San Francisco property.

Martínez took note of one of the properties and looked it up later. Price: $2.4 million. He is fascinating, and fascinatingly bitter, on the subject of class and status differences in Silicon Valley, in particular the never publicly discussed issue of the huge gulf between early employees in a company, who have often been made unfathomably rich, and the wage slaves who join the firm later in its story. ‘The protocol is not to talk about it at all publicly.’ But, as Bonnie Brown, a masseuse at Google in the early days, wrote in her memoir, ‘a sharp contrast developed between Googlers working side by side. While one was looking at local movie times on their monitor, the other was booking a flight to Belize for the weekend. How was the conversation on Monday morning going to sound now?’

When the time came for the IPO, Facebook needed to turn from a company with amazing growth to one that was making amazing money. It was already making some, thanks to its sheer size – as Martínez observes, ‘a billion times any number is still a big fucking number’ – but not enough to guarantee a truly spectacular valuation on launch. It was at this stage that the question of how to monetise Facebook got Zuckerberg’s full attention. It’s interesting, and to his credit, that he hadn’t put too much focus on it before – perhaps because he isn’t particularly interested in money per se. But he does like to win.

The solution was to take the huge amount of information Facebook has about its ‘community’ and use it to let advertisers target ads with a specificity never known before, in any medium. Martínez: ‘It can be demographic in nature (e.g. 30-to-40-year-old females), geographic (people within five miles of Sarasota, Florida), or even based on Facebook profile data (do you have children; i.e. are you in the mommy segment?).’ Taplin makes the same point:

If I want to reach women between the ages of 25 and 30 in zip code 37206 who like country music and drink bourbon, Facebook can do that. Moreover, Facebook can often get friends of these women to post a ‘sponsored story’ on a targeted consumer’s news feed, so it doesn’t feel like an ad. As Zuckerberg said when he introduced Facebook Ads, ‘Nothing influences people more than a recommendation from a trusted friend. A trusted referral is the Holy Grail of advertising.’

That was the first part of the monetisation process for Facebook, when it turned its gigantic scale into a machine for making money. The company offered advertisers an unprecedentedly precise tool for targeting their ads at particular consumers. (Particular segments of voters too can be targeted with complete precision. One instance from 2016 was an anti-Clinton ad repeating a notorious speech she made in 1996 on the subject of ‘super-predators’. The ad was sent to African-American voters in areas where the Republicans were trying, successfully as it turned out, to suppress the Democrat vote. Nobody else saw the ads.)

The second big shift around monetisation came in 2012 when internet traffic began to switch away from desktop computers towards mobile devices. If you do most of your online reading on a desktop, you are in a minority. The switch was a potential disaster for all businesses which relied on internet advertising, because people don’t much like mobile ads, and were far less likely to click on them than on desktop ads. In other words, although general internet traffic was increasing rapidly, because the growth was coming from mobile, the traffic was becoming proportionately less valuable. If the trend were to continue, every internet business that depended on people clicking links – i.e. pretty much all of them, but especially the giants like Google and Facebook – would be worth much less money.

Facebook solved the problem by means of a technique called ‘onboarding’. As Martínez explains it, the best way to think about this is to consider our various kinds of name and address.

For example, if Bed, Bath and Beyond wants to get my attention with one of its wonderful 20 per cent off coupons, it calls out:

Antonio García Martínez
1 Clarence Place #13
San Francisco, CA 94107

If it wants to reach me on my mobile device, my name there is:

38400000-8cfo-11bd-b23e-10b96e40000d

That’s my quasi-immutable device ID, broadcast hundreds of times a day on mobile ad exchanges.

On my laptop, my name is this:

07J6yJPMB9juTowar.AWXGQnGPA1MCmThgb9wN4vLoUpg.BUUtWg.rg.FTN.0.AWUxZtUf

This is the content of the Facebook re-targeting cookie, which is used to target ads-are-you based on your mobile browsing.

Though it may not be obvious, each of these keys is associated with a wealth of our personal behaviour data: every website we’ve been to, many things we’ve bought in physical stores, and every app we’ve used and what we did there … The biggest thing going on in marketing right now, what is generating tens of billions of dollars in investment and endless scheming inside the bowels of Facebook, Google, Amazon and Apple, is how to tie these different sets of names together, and who controls the links. That’s it.

Facebook already had a huge amount of information about people and their social networks and their professed likes and dislikes.[2] After waking up to the importance of monetisation, they added to their own data a huge new store of data about offline, real-world behaviour, acquired through partnerships with big companies such as Experian, which have been monitoring consumer purchases for decades via their relationships with direct marketing firms, credit card companies, and retailers. There doesn’t seem to be a one-word description of these firms: ‘consumer credit agencies’ or something similar about sums it up. Their reach is much broader than that makes it sound, though.[3] Experian says its data is based on more than 850 million records and claims to have information on 49.7 million UK adults living in 25.2 million households in 1.73 million postcodes. These firms know all there is to know about your name and address, your income and level of education, your relationship status, plus everywhere you’ve ever paid for anything with a card. Facebook could now put your identity together with the unique device identifier on your phone.

That was crucial to Facebook’s new profitability. On mobiles, people tend to prefer the internet to apps, which corral the information they gather and don’t share it with other companies. A game app on your phone is unlikely to know anything about you except the level you’ve got to on that particular game. But because everyone in the world is on Facebook, the company knows everyone’s phone identifier. It was now able to set up an ad server delivering far better targeted mobile ads than anyone else could manage, and it did so in a more elegant and well-integrated form than anyone else had managed.

So Facebook knows your phone ID and can add it to your Facebook ID. It puts that together with the rest of your online activity: not just every site you’ve ever visited, but every click you’ve ever made – the Facebook button tracks every Facebook user, whether they click on it or not. Since the Facebook button is pretty much ubiquitous on the net, this means that Facebook sees you, everywhere. Now, thanks to its partnerships with the old-school credit firms, Facebook knew who everybody was, where they lived, and everything they’d ever bought with plastic in a real-world offline shop.[4] All this information is used for a purpose which is, in the final analysis, profoundly bathetic. It is to sell you things via online ads.

The ads work on two models. In one of them, advertisers ask Facebook to target consumers from a particular demographic – our thirty-something bourbon-drinking country music fan, or our African American in Philadelphia who was lukewarm about Hillary. But Facebook also delivers ads via a process of online auctions, which happen in real time whenever you click on a website. Because every website you’ve ever visited (more or less) has planted a cookie on your web browser, when you go to a new site, there is a real-time auction, in millionths of a second, to decide what your eyeballs are worth and what ads should be served to them, based on what your interests, and income level and whatnot, are known to be. This is the reason ads have that disconcerting tendency to follow you around, so that you look at a new telly or a pair of shoes or a holiday destination, and they’re still turning up on every site you visit weeks later. This was how, by chucking talent and resources at the problem, Facebook was able to turn mobile from a potential revenue disaster to a great hot steamy geyser of profit.

What this means is that even more than it is in the advertising business, Facebook is in the surveillance business. Facebook, in fact, is the biggest surveillance-based enterprise in the history of mankind. It knows far, far more about you than the most intrusive government has ever known about its citizens. It’s amazing that people haven’t really understood this about the company. I’ve spent time thinking about Facebook, and the thing I keep coming back to is that its users don’t realise what it is the company does. What Facebook does is watch you, and then use what it knows about you and your behaviour to sell ads. I’m not sure there has ever been a more complete disconnect between what a company says it does – ‘connect’, ‘build communities’ – and the commercial reality. Note that the company’s knowledge about its users isn’t used merely to target ads but to shape the flow of news to them. Since there is so much content posted on the site, the algorithms used to filter and direct that content are the thing that determines what you see: people think their news feed is largely to do with their friends and interests, and it sort of is, with the crucial proviso that it is their friends and interests as mediated by the commercial interests of Facebook. Your eyes are directed towards the place where they are most valuable for Facebook.

*

I’m left wondering what will happen when and if this $450 billion penny drops. Wu’s history of attention merchants shows that there is a suggestive pattern here: that a boom is more often than not followed by a backlash, that a period of explosive growth triggers a public and sometimes legislative reaction. Wu’s first example is the draconian anti-poster laws introduced in early 20th-century Paris (and still in force – one reason the city is by contemporary standards undisfigured by ads). As Wu says, ‘when the commodity in question is access to people’s minds, the perpetual quest for growth ensures that forms of backlash, both major and minor, are all but inevitable.’ Wu calls a minor form of this phenomenon the ‘disenchantment effect’.

Facebook seems vulnerable to these disenchantment effects. One place they are likely to begin is in the core area of its business model – ad-selling. The advertising it sells is ‘programmatic’, i.e. determined by computer algorithms that match the customer to the advertiser and deliver ads accordingly, via targeting and/or online auctions. The problem with this from the customer’s point of view – remember, the customer here is the advertiser, not the Facebook user – is that a lot of the clicks on these ads are fake. There is a mismatch of interests here. Facebook wants clicks, because that’s how it gets paid: when ads are clicked on. But what if the clicks aren’t real but are instead automated clicks from fake accounts run by computer bots? This is a well-known problem, which particularly affects Google, because it’s easy to set up a site, allow it to host programmatic ads, then set up a bot to click on those ads, and collect the money that comes rolling in. On Facebook the fraudulent clicks are more likely to be from competitors trying to drive each others’ costs up.

The industry publication Ad Week estimates the annual cost of click fraud at $7 billion, about a sixth of the entire market. One single fraud site, Methbot, whose existence was exposed at the end of last year, uses a network of hacked computers to generate between three and five million dollars’ worth of fraudulent clicks every day. Estimates of fraudulent traffic’s market share are variable, with some guesses coming in at around 50 per cent; some website owners say their own data indicates a fraudulent-click rate of 90 per cent. This is by no means entirely Facebook’s problem, but it isn’t hard to imagine how it could lead to a big revolt against ‘ad tech’, as this technology is generally known, on the part of the companies who are paying for it. I’ve heard academics in the field say that there is a form of corporate groupthink in the world of the big buyers of advertising, who are currently responsible for directing large parts of their budgets towards Facebook. That mindset could change. Also, many of Facebook’s metrics are tilted to catch the light at the angle which makes them look shiniest. A video is counted as ‘viewed’ on Facebook if it runs for three seconds, even if the user is scrolling past it in her news feed and even if the sound is off. Many Facebook videos with hundreds of thousands of ‘views’, if counted by the techniques that are used to count television audiences, would have no viewers at all.

A customers’ revolt could overlap with a backlash from regulators and governments. Google and Facebook have what amounts to a monopoly on digital advertising. That monopoly power is becoming more and more important as advertising spend migrates online. Between them, they have already destroyed large sections of the newspaper industry. Facebook has done a huge amount to lower the quality of public debate and to ensure that it is easier than ever before to tell what Hitler approvingly called ‘big lies’ and broadcast them to a big audience. The company has no business need to care about that, but it is the kind of issue that could attract the attention of regulators.

That isn’t the only external threat to the Google/Facebook duopoly. The US attitude to anti-trust law was shaped by Robert Bork, the judge whom Reagan nominated for the Supreme Court but the Senate failed to confirm. Bork’s most influential legal stance came in the area of competition law. He promulgated the doctrine that the only form of anti-competitive action which matters concerns the prices paid by consumers. His idea was that if the price is falling that means the market is working, and no questions of monopoly need be addressed. This philosophy still shapes regulatory attitudes in the US and it’s the reason Amazon, for instance, has been left alone by regulators despite the manifestly monopolistic position it holds in the world of online retail, books especially.

The big internet enterprises seem invulnerable on these narrow grounds. Or they do until you consider the question of individualised pricing. The huge data trail we all leave behind as we move around the internet is increasingly used to target us with prices which aren’t like the tags attached to goods in a shop. On the contrary, they are dynamic, moving with our perceived ability to pay.[5] Four researchers based in Spain studied the phenomenon by creating automated personas to behave as if, in one case, ‘budget conscious’ and in another ‘affluent’, and then checking to see if their different behaviour led to different prices. It did: a search for headphones returned a set of results which were on average four times more expensive for the affluent persona. An airline-ticket discount site charged higher fares to the affluent consumer. In general, the location of the searcher caused prices to vary by as much as 166 per cent. So in short, yes, personalised prices are a thing, and the ability to create them depends on tracking us across the internet. That seems to me a prima facie violation of the American post-Bork monopoly laws, focused as they are entirely on price. It’s sort of funny, and also sort of grotesque, that an unprecedentedly huge apparatus of consumer surveillance is fine, apparently, but an unprecedentedly huge apparatus of consumer surveillance which results in some people paying higher prices may well be illegal.

Perhaps the biggest potential threat to Facebook is that its users might go off it. Two billion monthly active users is a lot of people, and the ‘network effects’ – the scale of the connectivity – are, obviously, extraordinary. But there are other internet companies which connect people on the same scale – Snapchat has 166 million daily users, Twitter 328 million monthly users – and as we’ve seen in the disappearance of Myspace, the onetime leader in social media, when people change their minds about a service, they can go off it hard and fast.

For that reason, were it to be generally understood that Facebook’s business model is based on surveillance, the company would be in danger. The one time Facebook did poll its users about the surveillance model was in 2011, when it proposed a change to its terms and conditions – the change that underpins the current template for its use of data. The result of the poll was clear: 90 per cent of the vote was against the changes. Facebook went ahead and made them anyway, on the grounds that so few people had voted. No surprise there, neither in the users’ distaste for surveillance nor in the company’s indifference to that distaste. But this is something which could change.

The other thing that could happen at the level of individual users is that people stop using Facebook because it makes them unhappy. This isn’t the same issue as the scandal in 2014 when it turned out that social scientists at the company had deliberately manipulated some people’s news feeds to see what effect, if any, it had on their emotions. The resulting paper, published in the Proceedings of the National Academy of Sciences, was a study of ‘social contagion’, or the transfer of emotion among groups of people, as a result of a change in the nature of the stories seen by 689,003 users of Facebook. ‘When positive expressions were reduced, people produced fewer positive posts and more negative posts; when negative expressions were reduced, the opposite pattern occurred. These results indicate that emotions expressed by others on Facebook influence our own emotions, constituting experimental evidence for massive-scale contagion via social networks.’ The scientists seem not to have considered how this information would be received, and the story played quite big for a while.

Perhaps the fact that people already knew this story accidentally deflected attention from what should have been a bigger scandal, exposed earlier this year in a paper from the American Journal of Epidemiology. The paper was titled ‘Association of Facebook Use with Compromised Well-Being: A Longitudinal Study’. The researchers found quite simply that the more people use Facebook, the more unhappy they are. A 1 per cent increase in ‘likes’ and clicks and status updates was correlated with a 5 to 8 per cent decrease in mental health. In addition, they found that the positive effect of real-world interactions, which enhance well-being, was accurately paralleled by the ‘negative associations of Facebook use’. In effect people were swapping real relationships which made them feel good for time on Facebook which made them feel bad. That’s my gloss rather than that of the scientists, who take the trouble to make it clear that this is a correlation rather than a definite causal relationship, but they did go so far – unusually far – as to say that the data ‘suggests a possible trade-off between offline and online relationships’. This isn’t the first time something like this effect has been found. To sum up: there is a lot of research showing that Facebook makes people feel like shit. So maybe, one day, people will stop using it.[6]

*

What, though, if none of the above happens? What if advertisers don’t rebel, governments don’t act, users don’t quit, and the good ship Zuckerberg and all who sail in her continues blithely on? We should look again at that figure of two billion monthly active users. The total number of people who have any access to the internet – as broadly defined as possible, to include the slowest dial-up speeds and creakiest developing-world mobile service, as well as people who have access but don’t use it – is three and a half billion. Of those, about 750 million are in China and Iran, which block Facebook. Russians, about a hundred million of whom are on the net, tend not to use Facebook because they prefer their native copycat site VKontakte. So put the potential audience for the site at 2.6 billion. In developed countries where Facebook has been present for years, use of the site peaks at about 75 per cent of the population (that’s in the US). That would imply a total potential audience for Facebook of 1.95 billion. At two billion monthly active users, Facebook has already gone past that number, and is running out of connected humans. Martínez compares Zuckerberg to Alexander the Great, weeping because he has no more worlds to conquer. Perhaps this is one reason for the early signals Zuck has sent about running for president – the fifty-state pretending-to-give-a-shit tour, the thoughtful-listening pose he’s photographed in while sharing milkshakes in (Presidential Ambitions klaxon!) an Iowa diner.

Whatever comes next will take us back to those two pillars of the company, growth and monetisation. Growth can only come from connecting new areas of the planet. An early experiment came in the form of Free Basics, a program offering internet connectivity to remote villages in India, with the proviso that the range of sites on offer should be controlled by Facebook. ‘Who could possibly be against this?’ Zuckerberg wrote in the Times of India. The answer: lots and lots of angry Indians. The government ruled that Facebook shouldn’t be able to ‘shape users’ internet experience’ by restricting access to the broader internet. A Facebook board member tweeted that ‘anti-colonialism has been economically catastrophic for the Indian people for decades. Why stop now?’ As Taplin points out, that remark ‘unwittingly revealed a previously unspoken truth: Facebook and Google are the new colonial powers.’

So the growth side of the equation is not without its challenges, technological as well as political. Google (which has a similar running-out-of-humans problem) is working on ‘Project Loon’, ‘a network of balloons travelling on the edge of space, designed to extend internet connectivity to people in rural and remote areas worldwide’. Facebook is working on a project involving a solar-powered drone called the Aquila, which has the wingspan of a commercial airliner, weighs less than a car, and when cruising uses less energy than a microwave oven. The idea is that it will circle remote, currently unconnected areas of the planet, for flights that last as long as three months at a time. It connects users via laser and was developed in Bridgwater, Somerset. (Amazon’s drone programme is based in the UK too, near Cambridge. Our legal regime is pro-drone.) Even the most hardened Facebook sceptic has to be a little bit impressed by the ambition and energy. But the fact remains that the next two billion users are going to be hard to find.

That’s growth, which will mainly happen in the developing world. Here in the rich world, the focus is more on monetisation, and it’s in this area that I have to admit something which is probably already apparent. I am scared of Facebook. The company’s ambition, its ruthlessness, and its lack of a moral compass scare me. It goes back to that moment of its creation, Zuckerberg at his keyboard after a few drinks creating a website to compare people’s appearance, not for any real reason other than that he was able to do it. That’s the crucial thing about Facebook, the main thing which isn’t understood about its motivation: it does things because it can. Zuckerberg knows how to do something, and other people don’t, so he does it. Motivation of that type doesn’t work in the Hollywood version of life, so Aaron Sorkin had to give Zuck a motive to do with social aspiration and rejection. But that’s wrong, completely wrong. He isn’t motivated by that kind of garden-variety psychology. He does this because he can, and justifications about ‘connection’ and ‘community’ are ex post facto rationalisations. The drive is simpler and more basic. That’s why the impulse to growth has been so fundamental to the company, which is in many respects more like a virus than it is like a business. Grow and multiply and monetise. Why? There is no why. Because.

Automation and artificial intelligence are going to have a big impact in all kinds of worlds. These technologies are new and real and they are coming soon. Facebook is deeply interested in these trends. We don’t know where this is going, we don’t know what the social costs and consequences will be, we don’t know what will be the next area of life to be hollowed out, the next business model to be destroyed, the next company to go the way of Polaroid or the next business to go the way of journalism or the next set of tools and techniques to become available to the people who used Facebook to manipulate the elections of 2016. We just don’t know what’s next, but we know it’s likely to be consequential, and that a big part will be played by the world’s biggest social network. On the evidence of Facebook’s actions so far, it’s impossible to face this prospect without unease.

[1] When Google relaunched as Alphabet, ‘Don’t be evil’ was replaced as an official corporate code of conduct by ‘Do the right thing.’

[2] Note the ‘professed’. As Seth Stephens-Davidowitz points out in his new book Everybody Lies (Bloomsbury, £20), researchers have studied the difference between the language used on Google, where people tend to tell the truth because they are anonymously looking for answers, and the language used on Facebook, where people are projecting an image. On Facebook, the most common terms associated with the phrase ‘my husband is …’ are ‘the best’, ‘my best friend’, ‘amazing’, ‘the greatest’ and ‘so cute’. On Google, the top five are ‘amazing’, ‘a jerk’, ‘annoying’, ‘gay’ and ‘mean’. It would be interesting to know if there’s a husband out there who achieves the full Google set and is an amazing annoying mean gay jerk.

[3] One example of their work is Experian’s ‘Mosaic’ system of characterising consumer segments, which divides the population into 66 segments, from ‘Cafés and Catchments’ to ‘Penthouse Chic’, ‘Classic Grandparents’ and ‘Bus-Route Renters’.

[4] I should say that the information is hashed before it is exchanged, so that although the respective companies know everything about you and do share it, they do so in a pseudonymised form. Or a pseudo-pseudonymised form; there is an argument to be had about just how anonymous this form of anonymity actually is.

[5] The idea of one price for everyone is relatively recent. John Wanamaker gets the credit for having come up with the notion of fixed price tags in Philadelphia in 1861. The idea came from the Quakers, who thought that everyone should be treated equally.

[6] A study from 2015 in Computers in Human Behaviour, ‘Facebook Use, Envy and Depression among College Students: Is Facebooking Depressing?’ came to the answer no – except when the effects of envy were included, in which case the answer was yes. But since envious comparison is the entire Girardian basis of Facebook, that qualified ‘no’ looks an awful lot like a ‘yes’. A 2016 paper in Current Opinion in Psychiatry that studied ‘The Interplay between Facebook Use, Social Comparison, Envy and Depression’ found that Facebook use is linked to envy and depression, another discovery that would come as no surprise to Girard. A paper from 2013 in Plos One showed that ‘Facebook Use Predicts Declines in Subjective Well-Being in Young Adults’: in other words, Facebook makes young people sad. A 2016 paper in the journal Cyberpsychology, Behavior and Social Networking, entitled ‘The Facebook Experiment: Quitting Facebook Leads to Higher Levels of Well-Being’, found that Facebook makes people sad and that people were happier when they stopped using it.

The Life of a South Central Statistic

$
0
0

We, who are in prison, had to answer for our sins and our lives were taken from us. Our bodies became the property of the state of California. We are reduced to numbers and stripped of our identity. To the state of California I am not Michael Alexander Allen but I am K-10033. When they want to know anything about me they do not type my last name in the computer but it is my number that is inputted. My number is my name. . . . Dante was not in hell due to a fatal sin but somewhere in his life he strayed onto the path of error, away from his true self. I, K-10033, strayed away from my true self: Michael Alexander Allen.

What sets the course of a life? Three years before my beloved cousin’s murder—before the weeping, before the raging, before the heated self-recriminations and icy reckonings—I awoke with the most glorious sense of anticipation I’ve ever felt. It was June 29, 2006, the day that Michael was going to be freed. Outside my vacation condo in Hollywood, I climbed into the old white BMW I’d bought from my mother and headed to my aunt’s small stucco home, in South Central. On the corner, a fortified drug house stood like a sentry, but her pale cottage seemed serene, aglow in the morning sun. Poverty never looks quite as bad in the City of Angels as it does elsewhere.

Aunt Karen, my father’s youngest sister, then drove a crew of us to collect Michael from the California Rehabilitation Center-Norco, which lies on a dusty stretch of Riverside County. Michael, the youngest of her three kids, was born when I was eight years old. I had grown up with him. The baby of a sprawling family, he was also my baby, a child of magnetizing energy and good humor. We had lost him eleven years earlier, when he was arrested, at fifteen, for an attempted carjacking. Now we’d get him back. It felt like a resurrection.

At the parking lot for Tower 8, a white van drove up to deposit the prisoners being released. Michael stepped out, saw us, and smiled. His broad, toothy grin took up half his face, a bright flash of white against his dark skin. He had a little bob in his step, the same natural spring he’d had as a child. His late adolescence and early adulthood had been spent in captivity, yet he bounded toward us like a fawn.

The homecoming party was in the driveway of my aunt’s house, next to the postage stamp of a lawn. Uncles and friends, cousins and second cousins, and cousins who knows how many times removed pulled folding chairs up to folding tables, which were covered with paper tablecloths and laden with fried chicken and sweet tea. The merriment continued all afternoon, and seemed to attract some attention from the neighbors. More than once, a glamorous-looking woman drove past, slowly, in a low-slung two-door gold Mercedes sports car. Michael feasted and played Football Manager with the nephews and nieces who had been born while he was in prison.

After the party, we had little time to waste. That summer, I was telecommuting to my job as the dean of the humanities at the University of Chicago. Michael, for his part, was intent on making something of himself. He had spent some time as a firefighter when he was at Norco, and he was ready to rebuild his life. Making that happen, managing his reëntry in the months to come, was my job. Not mine alone, but mine consistently, day after day, as the cousin on duty, the one with resources, the one who had been to college and who had become a professional.

The plans we had were not the plans we had hoped to have. Michael should have been paroled to a fire camp or to a fire station in Riverside County, where we had family who were ready to take him in. He could have lived there and gone to school and kept on beating back wildfires. But the rule was that you had to be paroled to the county where your offense was committed—crime-ridden Los Angeles County, in his case. So we developed the best alternatives we could. We made task lists, and moved through them efficiently. We met the parole officer, opened a bank account, and went to the library, where Michael got a card and started learning how to use a computer. (Google hadn’t existed when he went to prison.) At the D.M.V., he took a test and got his driver’s license.

Then, under the scorching sun of the deadliest California heat wave in nearly sixty years, we returned each day to the cool library and scoured Web sites for jobs. We focussed on large chains, which would have room for advancement, and sent out a lot of applications. Most of the time, Michael never got a reply. Then he caught a break: Sears invited him to a job interview. One morning in late July, he donned a new pair of khaki trousers and a button-down shirt, and we headed to Hollywood, to Santa Monica and Western. It was the perfect opportunity—but also, to me, a fraught one. A man who had been imprisoned for more than a decade would have to make the case that he ought to be hired. We had practiced bits and pieces of his story, but never the whole thing. In fact, I never heard Michael recount his own tale from start to finish.

I wonder now whether this was because the full version would have led me to ask questions that Michael did not want to answer. He had so much to give—stories, reflection, engagement—that somehow none of us ever noticed just how much he was withholding. He could love everybody on the terms on which they needed to be loved, give everybody what they needed to receive; and so, in the end, none of us really knew him. I’ve come to realize that he didn’t quite know himself, either.

The trouble began in preadolescence. His mother got married to a man who had kept from her the fact that he had a criminal record, and who soon became abusive. Karen took her children to Mississippi and then to southern Georgia. There, a few months shy of twelve, Michael stole a jar of coins, amounting to something under ten dollars, from a white family across the street. He was starting to want things, impatiently, and he was also naïve, a California kid transplanted to the Deep South. Only out of naïveté could he have thought to steal from a white family in southern Georgia.

Rather than telling Karen and asking for the money back, the family pressed charges. It was Michael’s first encounter with the law, and he went to court with his mother. Karen had by then filed for divorce and bought plane tickets to California. The judge told her the charges would be dropped so long as they got on the plane and never came back.

In the fall of 1991, Michael and his family moved to Claremont, where my father, William, taught, and where my mother, Susan, worked as a college librarian. For my cousins, my parents’ house was a second home, screened with laurel bushes, framed by pink-blossomed crêpe myrtles, and shaded by a spreading loquat tree in front.

William and Karen—children of a Florida fisherman who became a charismatic Baptist preacher—were close, but their courses in life were not. My father, with the encouragement of a grade-school teacher, was academically ambitious, and he turned into a pipe-smoking, NPR-listening professor, a political scientist who chaired the U.S. Commission on Civil Rights. He spent much of his days amid heaps of paper in a book-filled study, orchestral harmonies from the radio perfumed by the tweedy, comforting smell of pipe tobacco. Karen’s story was different; she worked for a time as a certified nursing assistant, but bringing up three young kids while working full time was a struggle. Her ex-husband wasn’t the first abusive man she had been involved with, and plans for furthering her education were often derailed.

Now, with my brother and me away at college, my parents helped Karen find an apartment a few blocks away. Michael took piano lessons from a stern, diminutive woman who had been my own teacher and who taught us how to sit up straight, “like the Queen of England.” Michael earned money gardening for her, but resented the hectoring lessons about life that she delivered as he weeded.

He was becoming something of a rule breaker in Claremont. He and his new friend Adam were caught stealing chocolate-chip cookies from the school cafeteria, and sometimes had to be separated after making noise in class. Michael was also caught shoplifting at a nearby mall. Luckily, the store owner delivered Michael to my father, not to the police. But Michael’s pattern of petty theft worried his mother, and my father; the weeding job was meant to deal with his need for money.

Then, in early 1993, a fire swept through the family’s apartment complex, and they moved again, to the L.A. neighborhood of Inglewood. Although the area was scarred from the ravages of the previous year’s riots, the move meant that Karen could be closer to her new job, at an organization called Homeless Health Care Los Angeles. It also meant that Michael started a new school year in yet another district.

We know something about his experiences as a student, because the State of California surveyed its youth during the 1993-94 school year. Forty per cent of ninth graders reported being in a physical fight; nearly sixty per cent reported seeing someone at school with a weapon. Gangs filled in for family; almost one in five ninth graders reported belonging to one at some point. Michael, then just shy of fourteen, seems to have flirted with the Queen Street Bloods, who were active on the west side of Inglewood; later, he started hanging with a friend from the Crips, a rival gang.

Michael was testing out a new world. But in that summer of 1993 he would also return to his old one, riding a bus back to Claremont to hang out with Adam. During one of those visits, Adam’s parents were looking after the next-door neighbor’s house, and the two boys let themselves in and took a radio and some other items. The neighbor reported a burglary, and when Karen realized who was responsible she hauled Michael to the police station. The boys returned everything. They were given a two-year juvenile probation, which entailed a curfew but no court date.

The narrative so far is familiar. A kid from a troubled home, trapped in poverty, without a stable world of adults coördinating care for him, starts pilfering, mostly out of an impatience to have things. In Michael’s first fourteen years, his story includes not a single incidence of violence, aside from the usual wrestling matches with siblings. It could have had any number of possible endings. But events unfold along a single track. As we make decisions, and decisions are made for us, we shed the lives that might have been. In Michael’s fifteenth year, his life accelerated, like a cylinder in one of those pneumatic tubes, whisking off your deposit at a drive-through bank. To understand how that acceleration could happen, though, another story is needed.

Like Dante I am forced to descend lower into hell to achieve a full awakening. I am forced into depression, scarred by obscenities, war after war, but each war that I survive I am a step closer to a full awakening of self. My hell is no longer demonstrating what I am capable of doing in order to survive. It has become what I can tolerate and withstand in order to live.

Consider the visible surface of Los Angeles. Underpasses, bridges, alleyways, delivery trucks, service entrances, corner stores, mailboxes, water towers, exhaust vents, and the streets—in the nineties, at least, all were covered with graffiti. Few can read that graffiti. I couldn’t then, and have only now begun to learn how to decipher it. But it’s a language that represents a world. It records deaths and transactions, benefactions and trespasses, favors done and owed, vendettas pursued. Laws and punishments. If you can’t read that graffiti, you have no conception of the parallel universe, all around you, that is fundamentally at war with the legally recognized state. It’s a regime with its own rules and penalties—in effect, a parastate. Michael grew up there.

Behind that parastate’s economy and criminal-justice system lies the war on drugs. In the eighties, as the state sought to break the global drug-supply chain by rounding up low-level peddlers and deterring them with outsized penalties, the wholesalers established their own system of deterrence for gang members who served as retailers. If you didn’t do what you were supposed to do, you were shot. Maybe in the knee first. If you riled the gang system again, you or someone you loved might be killed. The drug business, dependent on a well-established witness-suppression program, operates a far more powerful system of deterrence, with far swifter punishment, than any lawful state could ever devise.

In these years, the Los Angeles County Sheriff’s Department created its first gang database. In 1988, after a much publicized drive-by shooting of a bystander, near U.C.L.A., the Los Angeles Police Department used the database to round up no fewer than fourteen hundred African-American youths and detain them in the parking lot of the L.A. Coliseum. More than eighteen thousand people were jailed in six months. Between 1982 and 1995, the African-American prison population in California grew from 12,470 to 42,296; the Latino prison population soared from 9,006 to 46,080. Los Angeles was a city ready to explode when the four police officers who had been caught on video beating Rodney King were acquitted.

When Michael stole the jar of coins in Georgia, and the judge dropped the charges, you might say that Michael met the “forgiving world.” The same happened when he shoplifted, and when he stole the radio in Claremont, in 1993. But, back in the City of Angels, Michael met the unforgiving world. Nearly half the black men in Los Angeles between the ages of twenty-one and twenty-four were officially identified as gang members, and this simple fact of classification, accurate or not, affected that community profoundly. The angels had turned their backs.

The summer before Michael’s junior year, in 1995, he began looking for a job. His cousin Marc—my younger brother—had worked in a grocery store as a bag boy throughout high school, and Michael wanted a similar gig. But, at fifteen, he needed a work permit, and nobody in his mother’s social network could help. He again began to roam the streets, and stayed out past his curfew. In math class, his grades plunged from straight A’s to an F. Karen had conferences with Michael and his teachers, who told him that he was smarter than this. He countered, “I don’t want to be smarter than this.” On those warm summer days, he spent as much time as he could out-of-doors. Sometimes he would stand in front of the house of a kid he’d come to know. Karen spotted him once, lean and muscled, standing shirtless in khaki trousers—gangbanging gear. Although he was only four blocks away from her apartment, it felt like a different neighborhood.

the relative best equipped to guide Michael’s reëntry.

Photograph by Matthew Tammaro for The New Yorker

Karen’s last day with her boy was Friday, September 15th. Michael didn’t have school. He went to work with his mother and hung out in her office. Then she took him to the Los Angeles Public Library, where she planned to meet him when she got off work, to take him shopping. But Michael was gone when she returned. The next time she saw him, he was in handcuffs.

Where were you when you were fifteen? When I close my eyes, I can still see a bedroom with a brass bed topped with a blue-and-white striped Laura Ashley comforter. There were matching valences on my windows, and I had a wooden rolltop desk, with a drawer that locked and held my secrets, including dirty letters that I couldn’t at the time translate from a German boy with whom I’d had a minor romance at summer music camp.

I grew up in a college town where everyone knew my parents. They had made a critical decision, early in the lives of their two children, not to move until we had graduated from high school. I was a faculty brat, an insecure and often lonely child; the only time I ever got grounded was when my mother caught me sneaking a ride to French class with a friend. I was younger than most of my classmates at Claremont High School, and, although my friends all had their driver’s licenses by the start of our junior year and I didn’t, I wasn’t allowed to ride in their cars. Eight years later, in L.A., my fifteen-year-old cousin, who also didn’t yet have a driver’s license, was arrested, for the first time, for an attempted carjacking.

It was September 17, 1995, a cool and foggy Sunday morning. Larry Smith, a lanky forty-four-year-old, was buffing the dashboard of his blue Cadillac Coupe de Ville in the alley behind his apartment, on Rosecrans Avenue. The street was lined with drab stucco apartment buildings, whose uncovered staircases led down to carports below. Michael appeared holding a chrome Lorcin .380, a cheap pistol prone to malfunction. An older friend, Devonn, a member of the Rollin 60s Crips, was apparently on lookout, but not visible to Smith as he worked in his car. (Both names have been changed.) Michael approached Smith, told him not to move, and demanded his watch. Smith handed it over.

Then Michael asked for his wallet. When he found that it was empty, he tossed it back into the car. Then, as the police report recounted, Michael “tapped Smith’s left knee with the gun and said he was going to take the car.” According to Smith, Michael kept the gun pointed at the ground. Smith lunged for the weapon. They wrestled. Michael punched him. Smith gained control of the gun and shot Michael through the neck.

As Michael lay bleeding on the ground, Smith hollered to his wife to call 911. When the police arrived, they collected evidence and looked for witnesses, although no one had anything to say. Meanwhile, paramedics took Michael to a hospital, where he was treated for a “through and through” bullet wound that had narrowly missed his spine.

A police officer accompanying Michael in the ambulance reported that, “during transport, Allen made a spontaneous statement that he was robbing a man when he got shot.” At the hospital, Michael was read his Miranda rights and additional juvenile admonishments in the presence of a second officer. According to the police report, he waived his rights and said again that he had tried to rob the man, using a gun that he claimed he had found about two and a half weeks earlier. He also confessed that he had robbed three people during the previous two days on the same block, and that he had robbed someone a week earlier, about ten blocks away. The police had no reports for two of the four robberies he confessed to; in the two that had been reported, Michael had taken twenty dollars from one victim and two dollars from another. In other words, on his way to the hospital, and upon admission, with no adults present other than the officers, a wounded fifteen-year-old talked a blue streak.

By the time Karen got to Michael’s bedside, he had wrapped up his confession. The only thing he didn’t mention was Devonn’s involvement. Did Devonn suggest the crime, or provide the gun? We have no way of knowing. I don’t believe that Michael was prepared, that morning, to be violent; he had a gun, but refrained from using it. Still, I was far away, a graduate student in England. Along the banks of the River Cam, I shared poems with friends and debated crime and punishment in ancient Athens. I had gravitated toward the subject upon being struck by how a sophisticated, democratic society had made next to no use of imprisonment. When the news of Michael’s arrest came, it was stupefying. My brain raced in endless loops. How could it be? How could it be? I now have a sense of an answer. But there were harder questions ahead.

I’m trapped in a hell with whom society decrees to be the worst of living and better off dead. Robbers, rapists, child molesters, carjackers, murderers, and dope fiends who would spend their mother’s monthly rent for a quick fix. And here I am, amongst them. As much as the mere thought disgusts me, I am one of them. Just another number, not deserving of a second chance.

Before his arrest, Michael did not have a criminal record. That day, he gained one with a vengeance. For the watch and the wallet, Michael was charged with robbery; for the car, attempted carjacking. Both charges were “enhanced” because of the gun. He was also charged with the two earlier robberies. Four felonies, two from one incident, and all in one week.

Eighteen months earlier, in March, 1994, California’s Three Strikes and You’re Out law, the nation’s first, had gone into effect. Once you were convicted of your third felony, it meant twenty-five years to life, or a plea deal. If Michael pursued a jury trial, convictions on at least three of his four charges would trigger the law. Worse, this was happening at the high point of L.A.’s panic about carjackings. In Los Angeles County alone, the number of carjackings had nearly doubled between 1991 and 1992, from 3,600 to 6,297. In 1993, the state legislature had unanimously passed a bill that made carjacking an offense for which sixteen-year-olds could be tried as adults. Two years later, the bar was lowered to fourteen. A Los Angeles Times article titled “Wave of Fear,” which ran the year before Michael’s arrest, quoted then Senator Joseph Biden saying, “Name me a person in L.A. who has a fender-bender and doesn’t fear an imminent carjacking. Yes, it’s still remote, but you’re in the statistical pool now. It’s like AIDS. Everyone’s in the pool now.”

California’s legislators had given up on the idea of rehabilitation in prison, even for juveniles. This is a point that critics of the penal system make all the time. Here is what they don’t say: legislators had also given up on retribution. Anger drives retribution. When the punishment fits the crime, retribution is achieved, and anger is sated; it softens. This is what makes it anger, not hatred, a distinction recognized by philosophers all the way back to antiquity. Retribution limits how much punishment you can impose.

The legislators who voted to try as adults sixteen-year-olds, and then fourteen-year-olds, were not interested in retribution. They had become deterrence theorists. They were designing sentences not for people but for a thing: the aggregate level of crime. They wanted to reduce that level, regardless of what constituted justice for any individual involved. The target of Michael’s sentence was not a bright fifteen-year-old boy with a mild proclivity for theft but the thousands of carjackings that occurred in Los Angeles. Deterrence dehumanizes. It directs at the individual the full hatred that society understandably has for an aggregate phenomenon. But no individual should bear that kind of responsibility.

On February 5, 1996, four and a half months after Michael’s last night at home, he sat in court, in an orange jumpsuit and handcuffs, as the judge told him to choose whether to stand trial and face a possible conviction of twenty-five years to life or to plead guilty and take a reduced sentence. The judge didn’t say how much the sentence would be reduced, but he did say, “Please take the plea.”

Michael could not choose. Now sixteen, he asked his mother to decide. Karen went outside the courtroom and prayed. “God told me,” she says, “that he would only get seven years, versus risking a trial of twenty-five years to life. I made the decision.” So Michael pleaded guilty. A few months later, he learned that his “earliest possible release date” was June 29, 2006. According to Karen, the only time Michael cried in court was when he got sentenced.

When you’re sixteen, the farthest back you can remember is about thirteen years, to the age of three. Michael’s sentence was almost equivalent, in psychological terms, to the whole of his life. It stretched past what was for him the limit of knowable time. The mind cannot fasten onto this sort of temporality; we are unable to give it concrete meaning in relation to our own lives. The imagination wanders into white space. For Michael, it was, he later wrote, “a mountain of time” to climb. It would be a steep one. The moment he turned seventeen, he was transferred to adult prison.

“How could it have happened?” is the question everyone asks. Where were the lawyers? What did your family do? I think back to the stolen radio. Michael came from a family who believed that if you did something wrong you admitted it, you fixed it, and you suffered the consequences. Michael was guilty of the attempted carjacking; he was going to have to suffer the consequences. Our family trusted in the fairness of the criminal-justice system. At each turn, we learned too late that this system was no longer what we thought it was, that its grip was mercilessly tightening, that our son would be but one among many millions soon lost in its vise.

When we read that the point of the Three Strikes law is to lock up repeat offenders, we do not think of the fifteen-year-old who has just been arrested for the first time. An underground nuclear test is conducted, and the land above craters only much later. This, I think, describes the effect of the Three Strikes law and the slow, constant escalation of penal severity. An explosion occurred underground. The people standing on the surface conducted their lives as usual. They figured out what was really going on only after the earth had collapsed beneath them.

The years between the ages of fifteen and twenty-six are punctuated by familiar milestones: high school, driver’s license, college, first love, first job, first serious relationship, perhaps marriage, possibly a child. For those who pass adolescence in prison, some of these rites disappear; the ones that occur take on a distorted shape. And extra milestones get added. First long-term separation from family. First racial melee. First time in solitary, formally known as “administrative segregation.” First time sodomized.

Between his arrest and his sentencing, Michael was mainly in Central, the juvenile prison, where only parents and legal guardians could visit. When Michael and I reconnected properly, in the late nineties, he was making his way through Chino—a notoriously tough prison—before landing in Norco. Its full name was the California Rehabilitation Center-Norco, but little rehabilitation was on offer. There was the obligatory library, but no classes past the G.E.D. level. In the nineties, college and university classes were scrapped because of budget cuts, and the state and federal governments ceased providing prisoners access to Pell Grants for correspondence courses. Higher education, once seen as an antidote to recidivism, had come to be seen as a privilege that inmates hadn’t earned.

After I started teaching at the University of Chicago, in 1998, Michael and I began talking regularly on the phone. Once he was at Norco, I began to visit him, too, every other week in the summer and during the Christmas holidays. Michael would call at least once a week, sometimes more, except when the prison was on lockdown owing to outbreaks of violence. Then weeks might pass without a word. I was a good phone partner, because I could afford the astronomical collect-call charges. Every call began with a reminder, a robotic voice saying, “This is the California Department of Corrections. Will you accept the charges?” And then, every fifteen seconds, as if we could forget, there was another interruption: “This call has originated from a California state prison.”

Michael, who had already completed his G.E.D., desperately wanted to go to college, and I understood his desire to learn. I believed in education; I believed in Michael. So I researched how Michael might be able to get a college degree. On November 8, 2001, Michael sent me his application to Indiana University’s Program in General Studies, and I mailed it with a check nine days later. He would aim for a bachelor’s degree. The day he was admitted was as exhilarating as the day I received my fat envelope from Princeton, thirteen years earlier.

There was a catch, however. No hardcover books were allowed into the prison. Michael could enroll only in classes for which the textbooks had soft covers. I made a round of phone calls. The remaining choices for introductory classes were Intro to Ethics and Intro to Writing and Study of Literature. Michael chose the second, Lit 141. I paid the fees and ordered the books.

New Year’s came and so did the Bible, the Odyssey, the Inferno, “The Canterbury Tales,” and “Persian Letters.” But there was no shortage of distractions, and Michael had trouble completing the assignments. At one point, suspected of participating in a “racial melee,” he was transferred to Chino and placed in solitary, until an investigation absolved him. A year later, he repeated the class. This time he churned out one essay after another, with readings that were full of insight and personal connection to ancient texts. He was finding his voice. “I don’t take kindly to seeing myself in Hell but Dante’s writing makes it impossible to just read without visualization,” he wrote in one essay. “It is the life I live in Prison which to me is Hell. . . . I think of Dante’s use of ice as nothing but a mere deception. Ice within itself is enticing to the burning soul. Ice can get so cold that it burns flesh. And it’s parallel to any sin committed on earth.”

Along the way, Michael fell in love. I remember his words on the phone: “I’ve met someone, Danielle. She’s beautiful.” And I remember my sense of confusion. Met someone? How? Where? I was thinking of the female guards whom I’d got to know in the course of my visits. But in a fumbling way we came to understand each other. Michael had fallen in love with a fellow-inmate who had implants or hormone-induced breasts, and who dressed and lived as Bree. (I’ve changed the name.) She was, he said, unquestionably the most beautiful woman in the prison. He hadn’t told his mother, and he made me promise not to say anything. He knew Karen would be upset and he feared she would judge him, as he trusted I would not.

Like freedom, desire was dizzying to Michael. A month later, he mailed me a piece of writing unlike anything he had ever sent me. “The world has change and brothas far from the same,” he rapped, and continued:

Am I losing my mind

No; I think I found it

Realizing greatness in one’s self is very astounding

and truth be told, I recognize a King

cause when I look in the mirror all I see is me

And us, so please trust, we can’t be touch

standing together forever is a necessary must.

Soon afterward, he sent me Bree’s annual prison shot. She was posed as a woman, lying on the floor like a sports pinup, made up and in colorful clothing. Why did he love her? He loved her because she was the most beautiful woman he had ever seen. He loved her because, of all the men in prison, she had chosen him—and that was a gift of surpassing value. But it was also a gift that came to blind him. When he was finally released from prison, I failed to grasp that he was not yet free.

Upstairs, in the Sears personnel department, everything was beige and brightly lit. I settled into a metal chair and waited while Michael had his interview, in an office down the hall. I did a lot of waiting that summer, but I never questioned why I was there. My brother and I had long ago formed a tight circle with Aunt Karen’s three kids—Nicholas, Roslyn, and Michael, each born about two years apart—and, as the oldest, I was always the one in charge. As I waited, I typically spent my time thinking about my task lists, about what had to be done next. Forty-five minutes into this particular wait, the door opened and I learned that the managers had offered Michael a job as an inventory clerk.

It felt as if time had begun. I could imagine a future, even a happy ending. There was still school and housing to be arranged, but we were steadily assembling the pieces of a possible life, as if doing a jigsaw puzzle. The goal was for Michael to work full time and to enroll in one of California’s famed community colleges. No one in his immediate family had a degree, but I was in my element—pretty much my deepest expertise was in going to school.

Los Angeles Valley College, in Valley Glen, was the obvious place, a decent school with good general-education courses and—our goal—a fire-technology program. The subway’s Red Line had stops at Santa Monica and Vermont, about a mile from the Sears, and in North Hollywood, not too far from campus. We battled our way through the thicket of federal financial-aid forms, visited the tutoring center, and hungrily collected flyers posting apartments for rent.

We needed a place cheap enough to manage on Michael’s wages. Together, we searched the listings, drove by addresses, and made calls and appointments. We landed on a promising place on Ethel Avenue, in Valley Glen, a few blocks north of the college. The advertisement was for a studio apartment in a converted garage behind a modest home. Once again, Michael practiced telling his story, and we scheduled a visit.

The home was impeccable, a white bungalow circled by a white iron fence. Alongside the fence stood some small shrubs, neatly tended, and rosebushes spraying white flowers. I went up to the house by myself. Two women met me at the door, a mother, perhaps in her sixties, and her daughter. Dressed in linen trousers and a black T-shirt, I introduced myself. I was a professor, I told them, and I was helping my cousin, who had recently been released from prison. He had just enrolled at Los Angeles Valley College and been hired at Sears. I would be paying his deposit and guaranteeing his rent. He’d been sentenced as a young person and this was his second chance. Were they willing to meet him and hear him out?

They agreed, and I sat outside while Michael spoke to his prospective landlords. He could charm anyone with that bouncing gait and electric grin. Finally, the three emerged, in good spirits, and the women took us around to the back to see the studio. It was clean and peaceful, and equipped with a hot plate and an electric heater. I could imagine being comfortable there. And it was walking distance from the school.

Michael said he wanted it; we all shook hands in the gaze of the late-afternoon sun. I was moved by the trust and the generosity of these two women, and I still am. Driving back to South Central, my mood was all melody. I imagined Michael felt the same. Little more than a month out and here he was, with a driver’s license, a bank account, a library card, and a job. He was enrolled in college, with a clean, safe, comfortable place to live. This was a starter set for a life, enabling him to defy the pattern of parolees.

I dropped him off in South Central and headed back to Hollywood, expecting to sleep soundly for the first time in a while. But that night Michael called. He wasn’t sure he should take the apartment. I felt a stone drop to the bottom of a well.

Why not? I asked.

He couldn’t explain, he said. He just didn’t feel quite right about it.

I told him to sleep on it, and when we talked in the morning he told me he wanted the apartment after all. Relieved, I headed off to collect a cashier’s check for the security deposit, and Michael headed off under yet another cloudless sky to his job at Sears. At midday, he called me again: Had I taken the check over yet? He said that he had changed his mind again.

“Michael, what on earth are you talking about?”

He told me that he wasn’t sure what it would be like if his associates came by.

The word surprised me, but I didn’t ask him what he meant by “associates.” The purpose of the word, somehow, was to insist on his privacy, and it brought me up short. I paused, didn’t ask questions. I told him to think about it some more. Disagreement was rare for us.

He called me a few hours later. He said he would take the apartment and asked me to pick him up after work. Then, just before I did so, he called again. “I’ve made up my mind,” he said. “I don’t want the apartment.”

My memory of the conversation is hazy, but it’s likely we exchanged some sharp words. His plan, it emerged, was to live with his mother and to ride the bus the nine miles from there to Sears and the ten miles from there to Los Angeles Valley College, and then the twenty-two miles home—through the worst of Los Angeles traffic. It was madness, but there was nothing I could do. It was well into August. School would start soon. I would have incoming students to welcome, new faculty to orient, budgets to plan. I bought him more khakis and button-down shirts, spent as much time with him as I could. A few weeks later, I headed back to Chicago.

The root of sin is lust and the desire to satisfy that lust. . . . Lust only creates wanting and wanting creates greed and greed burns Flesh. It is lust that causes us to believe we have to have something at all cost. This is my suffering, this is my hell. 24 hours all night. There is no day. My soul in its entirety is in darkness.

The jigsaw puzzle soon fell apart, and college was the first piece to go. The commute was just too much; I doubt Michael made it through even two weeks of classes. The job, meanwhile, lasted until November, when I got a nearly hysterical call. Michael said he couldn’t do it. He was drowning. He wasn’t going to make it. When I left L.A., I had promised him that if he ever needed me I would be there. After the call, I went straight to the airport, and arrived in L.A. just in time to take him to dinner.

Michael was teary and despondent. After work, he said, some of his Latino co-workers had called him a nigger. He fought them in the parking lot, and walked away from the job. Never told his bosses or co-workers that he was quitting—just didn’t return. So now he was back to square one. Worse than that, really, since he’d proved himself unreliable to an employer. He was mostly spending his time at home, playing video games with his nephews. He no longer saw a future for himself.

I mainly tried to listen; I didn’t have much to offer. I could promise to get him into an apartment, if he could get another job. But I was no longer in a position to stay and help him find one. I had too many obligations in Chicago. November was tenure-review time, with mounds of papers to read and unending cycles of meetings that the dean, in particular, was not supposed to miss. My professional reputation was at stake. Michael would have to make the next push for himself.

When I visited L.A. just before the winter break, it seemed as if Michael had made that push. He had found an apartment, he told me, and was ready to put down a deposit. Could I come and see it? The place was on the fourth floor of a vintage Craftsman-style building overlooking the 101 freeway. It was big and spacious, with gleaming wood floors. As I wound through the rooms, Michael began telling me about how he and Bree wanted to move in.

I had no idea he was still seeing Bree, let alone making plans to move in together. My face must have conveyed surprise, though I tried not to react too strongly. (Learning how to suppress visible emotion is an occupational demand of being a dean.) I told him that I wanted to know what the job situation was. Had he lined up a new gig? What did Bree do—did she have a job? Our voices echoed in the empty apartment. Michael leaned against a windowsill, the sky and the freeway shining behind him.

There was something shamefaced in him as he answered. No, he didn’t have a job. Bree was into hair styling, but, no, she didn’t have one, either. What, exactly, were they thinking? Michael didn’t have much of an answer. Plainly, the plan involved taking advantage of me to some degree.

In that moment, I encountered a different Michael from the one I knew. I saw something calculating, something I’d never seen before. I didn’t ask to talk to Bree, who I’d come to realize was the woman in the gold Mercedes crawling past our homecoming party. All I was able to say was that I couldn’t possibly pay the deposit—plus some number of months’ rent, plus co-sign a lease—when neither of them had a job.

Michael’s face tensed. He said he understood.

This was the day I understood that the idea that I could stand my baby cousin up on his own two feet was a fantasy; it had always had too much of me in it. From this point on, Michael ceased confiding in me. Our phone conversations never burrowed below the surface. I no longer knew how to help.

Michael spent more and more time with Bree, whose possessiveness was violent. According to Karen, Bree cut Michael three times between December and May, and each time Michael tried to pass the injuries off as the result of someone attempting to rob him. He had also begun to suspect Bree of cheating. Late one night, he sneaked under her window, in the hope—he told me later—that catching her in the act would give him an easy out from the relationship. That night, he got into a fight with a lover of Bree’s, and the police were called. Michael went straight to prison for a parole violation, and remained there for around a year.

It was a catastrophic defeat. Despite the fact that we wrote each other letters, I somehow obliterated from my memory all traces of Michael’s second stint in prison. When he got out again, just months before the 2008 stock-market crash, he returned to what we hoped would be the comfort of his mother’s house. Just a short time later, though, he began living with Bree.

In the months before Michael’s parole violation, Karen and Bree had waged a battle rooted in a strong mutual dislike. Now Bree sought a formal treaty. She called Karen to say that Michael would be living with her, and that she didn’t want any conflict. This was hard for Karen. She knew that her son’s relationship was violent. As Karen understood it, Bree had been in prison for attempting to kill a boyfriend, and the only time she had seen Michael get physical with anyone was when he fought Bree on her pin-neat front lawn. Bree had been going down the street, breaking car windows and throwing things at Karen’s house. Michael had gone outside to warn her away. The two came to blows. Through a window, Karen saw Michael knock Bree out. That night, Karen added to her prayers the hope that the Lord would liberate Michael from his misery.

By December, Michael’s world had fully contracted. While living at Bree’s house, he became known on the street as Big Mike. That winter, he revealed to his sister a gun, hidden in a towel, in Bree’s Mercedes. By the spring, he was running drugs, including at least one trip to Texas. Later, the detectives investigating his murder found PCP in his room.

In June, 2009, I got married, in New Jersey, where I had recently accepted an appointment at a distinguished research institute. Michael came to the wedding—his first airplane flight since his release. He was handsome in a beige jacket and crimson shirt, with matching crimson alligator-skin shoes. But there was so much I couldn’t see: I couldn’t make out the demons chasing Michael as he greeted the other guests at the door to the chapel.

Five weeks after that champagne-filled wedding day, my father called me from Maryland with the news: Michael had been discovered in a car in South Los Angeles, dead from multiple gunshots. I was in England, and I remember my father’s voice, the careful, clipped speech of a retired professor, crackling as if through the first transatlantic cables. Heading to the airport, I knew that the police were looking for a woman, and that Bree had disappeared. Two weeks later, she was charged with his murder.

She had, evidently, shot Michael in her kitchen. There had been one witness, a middle-school-age boy. He hadn’t seen anything, but he had heard voices and gunshots. With the help of relatives, Bree cleaned Michael up nicely. She then bundled him in a blanket, put him in his little hatchback, and drove him to the street corner where he was found. Three accessories—all members of Bree’s family—were also charged. Eventually, Bree pleaded no contest to voluntary manslaughter, and was sentenced to twenty-two years in prison. Having by now undergone gender-reassignment surgery, she was sent to a women’s prison.

Michael and Bree had first met and become lovers when they were both inmates at Norco, which she had entered at the age of twenty-five. Bree was a little more than two years older than Michael. She was just his height and just his weight, a transgender woman still early in the process of transitioning. As far as the public record reveals, she’d been convicted for assault with a firearm.

I thought back to Michael’s homecoming in 2006, to Bree cruising by in her chariot, coming for to carry Michael home. We all had thought the relationship ended when Bree left prison a year ahead of Michael, and we believed that Michael’s home was with us. What Michael himself thought or wanted that homecoming day, I will never know. He hadn’t invited Bree to the picnic. Yet she came and would stay.

When Michael contemplated renting that tidy little studio apartment on Ethel Avenue, with its white fence and pearly roses, it was voluptuous Bree in her tight clothes and gold Mercedes whom he was visualizing having to introduce to those kindly landladies. How would it have gone if he had taken this “associate” home with him? When he spent those twenty-four hours dithering over whether to rent the apartment, I see now that his real choice was whether to repudiate the first and only love of his life. He chose Bree, and it would prove to be his life’s defining decision.

There are those who await to fulfill their destiny. I see in them a sincere and apologetic heart for their ill misdeeds. They are the one who will change the world positively or positively change someone’s world. Hell cannot hold the latter of the two opposites but in time will only spit them back out into society to do what is right. The hell that I live in cannot hold Dante. Hell can test and try one’s self but it cannot hold Dante and it will not hold me. In the Inferno, the dead are trapped forever. Surely, the biggest and most important difference in the Inferno and my hell called prison, is that I have a way out.

Bethlehem Temple, Karen’s parish, mounted a funeral service like those from my childhood, when I visited my grandfather the Baptist preacher. There were soul-busting songs and unpainted, teetotalling women; women in hats, with fans, on the verge of fainting. Karen had to be held, and the preacher lifted the roof off. We wept enough to make our own riverside. Oh, we’ll wait till Jesus comes / Down by the riverside.

The service was followed by a brief lunch back at Karen’s house, and then it was onward to a second service, at the church that Michael belonged to. The street had turned out for this service, bringing its jive step. The place was filled with people we didn’t recognize. The detectives were here, too, working. They hadn’t yet solved the murder of the man they knew as Big Mike, and were watching to see who showed up. The pastor had nothing to say about Michael; instead, he spent the eulogy giving himself credit for the worldly success of this or that parishioner, before descending into an anti-Semitic rant about moneylenders and lawyers.

Where was Michael in all of these remarks? He wasn’t there. Not in those words, or, in fact, in his casket. We’d had a viewing a few days earlier. I’d been taken aback, seeing him, his still face so sombre in repose, with a slightly grayish tinge. In the satin-lined casket, he was dressed in the very suit he’d worn to my wedding, a month earlier. I was struck by his solidity. I had never noticed how much he had bulked up. In the casket, there was no smile. The light was gone, and with it, I suppose, the lightness. Later, much later, writing this, I’ve had to face the fact that on that day I was looking at Big Mike, not at little Michael. The hardest part of my effort to understand what happened to my cousin has been learning when and how Big Mike replaced Michael.

After the service, we went back to Aunt Karen’s house to celebrate what we called Michael’s homegoing, his passage to the promised land. Next to that postage stamp of a lawn, we gathered around folding chairs pulled up to folding tables, laden with fried chicken and sweet tea, to commemorate the baby of the family. We had lost him at fifteen to jail; we regained him eleven years later. At twenty-nine, he was lost to us again, gone for good. My cousin’s idea of hell was to be reduced to a number; now he became a statistic, joined to the nearly two hundred thousand black Americans who have died violently in the years since his arrest on Rosecrans Avenue.

In my heart’s locket, five gangly brown-skinned kids, cousins, will be forever at play beneath a pair of crêpe-myrtle trees bathed in June sunshine. Michael and I loved to climb trees. An arm here, a leg there, juts out from the trees’ floral sundress, a delicate skein of pink and purple blooms. When we found unbloomed buds on the dichondra lawn, we would gently press at their nubs until the skins slit and fragile, crinkled blossoms emerged whole. Meanwhile, inside the house, through the living-room picture window, the adults, beloved, pass their time in glancing, distracted talk. ♦

Assume positive intent

$
0
0

Chris could have become my worst nightmare. He was a key stakeholder and decision maker on the largest active account that my company was dependent upon for its financial stability. That wasn’t the problem. The issue was that we were midway through the initial project deliverables and it was becoming apparent to team members on both sides that the budget specified on the contract was insufficient to deliver a solution that would actually meet the needs of their business. This was a huge problem. I was doing my best to navigate the situation and keep calm on the outside, but on the inside, I was stressed to the max knowing that it was possible that we could lose the account and suffer a significant financial loss.

As time went on, tensions continued to rise. Additional meetings were scheduled to discuss budgets and contracts. People started to become irritable, dig in their heels, and act in ways that were clear signs of CYA (cover your ass) in progress.

Chris could have crushed me, and yet he didn’t. In fact, he did the exact opposite and taught me an incredibly valuable lesson. Amidst the bickering on one phone call, he asked his colleagues to stop this behavior and to assume positive intent instead. He went on to describe how this is a philosophy he’s adopted in his personal and professional life as a means of being more efficient, effective, and solution focused. After all, approaching any situation with the opposite mindset results in wasted time and energy. Assuming negative intent means you’re spending lots of time second guessing everyone’s motivations, being combative instead of collaborative, and slowing everything down by having to update contracts meticulously instead of going off of a handshake.

What About When…?

No, I wouldn’t recommend applying this advice universally and haphazardly. A few caveats:

1) It’s 100% reasonable to have a high degree of skepticism within a low-trust environment. For example, I would never assume positive intent and allow my daughter to be alone with a registered sex offender just because the person claimed they had changed. I would also never trust an alcoholic with a house full of liquor. Once a person has violated trust against a particular metric, it's OK to take a different position in order not to put yourself in harm's way.

2) This advice is not without risk. Some people will leverage this against you. There are con artists and sociopaths out there. Marth Stout claims 4% of the population falls into this category. In short, if you start from a place of positive intent, you are going to get screwed over by at least one out of 1 out of every 25 people you encounter because they literally have no moral compass. However, in my experience, the rewards outweigh the risks. To invert the statistic by Martha, 24 out of 25 people can feel legitimately bad if they know they’ve caused harm to someone else. Essentially, the vast majority of the population has a conscience that acts as the moral compass to steer them from intentionally causing hurting others. It doesn’t always work, but it’s better than nothing. And given that 96% of people fall into this category, I feel there is a greater risk in approaching those relationships from a place of distrust.

The ROI of Trust

There is an entire book called The Speed of Trust that provides qualitative and quantitative evidence that high trust situations result in massive gains in efficiency and effectiveness in relationships. We know this intuitively. If we are skeptical of an expensive new product or service, we might spend hours of time researching to make sure we’re making the right decision. However, if we get a recommendation from someone we trust, that may be all we need to decide in an instant.

So if trust is so valuable, how can we get there more quickly? Well, there are two different ways. We can start from a place of skepticism until someone has proven themselves worthy to be trusted. Or we can start from a place of positive intent right from the beginning and keep it there until they violate that trust. Both are feasible, but only by extending trust first will most relationships ever get to the place where you can experience the gains outlined in Stephen M. R. Covey’s book.

Just remember there are caveats. It’s all a matter of if you’re willing to accept the risks!

Photo by Charles Deluvio on Unsplash

Video is Coming to Reddit

$
0
0

Reddit is home to more than 100,000 of the internet’s most passionate and engaged communities, and we want to give all of them the best tools to express themselves and engage in deeper conversations. That’s why we’re excited to announce the extended rollout of Reddit video beta, which makes it easier than ever for redditors to capture, upload, and share videos and gifs with all their favorite communities.

This launch is a giant step forward in our efforts to bring rich content to Reddit, following last year’s release of native image hosting and our introduction of native video ads earlier this year.

So what is Reddit video hosting? Here’s everything you need to know:

  • You can record video within the official Reddit apps or select a pre-recorded video from your gallery and upload it to Reddit.
  • On desktop, you can select a video from your files to upload.
  • Allowed video file types are MP4 and MOV, with a maximum length of 15 minutes.
  • You can convert uploaded videos into MP4 gifs with our native gif converter.
  • On the mobile apps, you can trim uploaded videos and gifs to just the section you want to feature.
  • Best of all, you can watch Reddit video while you read comments and engage in conversation. (Check it out by clicking this post and scrolling down.)

Why Native Video?

If you’ve spent any time on r/HighQualityGifs, r/mealtimevideos, or just about any other Reddit community, you know that videos and gifs represent a major proportion of the content shared on our site. But prior to this launch, content creators had to go through a time-consuming, circuitous process to post videos, using third-party hosting platforms, copying URLs, and sharing them as link posts. This inhibited many users, especially those who capture videos on their phones and want to share them quickly with their favorite subreddits.

With native video, we’ve streamlined this process dramatically, allowing both content creators and commenters to focus on the conversation taking place on Reddit. No third-party uploads, new tabs, or back arrows necessary.

Beta Tests and User Feedback

Of course, given Reddit’s scale, implementing video hosting that could support our vast and varied userbase was a massive undertaking. We also wanted to ensure that video would be available to redditors across all our platforms, so we worked to launch hosting on iOS, Android, and desktop at the same time. And to ensure we’re offering our users a best-in-class experience, it was important to roll this out slowly, soliciting feedback from individual redditors, video and gif communities, mod teams, and content creators along the way.

As we began beta testing, we found that the most engaging types of video weren’t coming from popular users trying to establish their individual brands, as you might see on other platforms.Instead, our video adopters so far have been creating and sharing videos to engage within their specific Reddit communities, as an organic extension of conversations already happening on each subreddit. These users don’t need to worry about building a huge subscriber base or posting to an abyss of content because they’re posting directly to the communities that share their passion.

We’ve seen redditors recording videos of their head to ask r/Hair for recommendations before their next haircut:

One user shared a slow-motion shot of his golf swing to ask the pros at r/golf for tips:

Redditors who attended Tesla’s Model 3 launch posted videos of the car’s new features to r/teslamotors:

And of course the pet lovers have already taken to Reddit video to share their adorable doggos with communities like r/aww:

Throughout the beta process, we’ve found that native video doesn’t merely improve the content creating process; it offers a unique new format for expression and interruption-free engagement once the post is live. Because our video platform keeps a small preview window open at the top of the screen as you browse the comments below, you can transition seamlessly between viewing, lurking, and commenting on Reddit videos.

It’s already proving its value to our communities, content creators, publishing partners, and brands, who are eager to start using native video to engage in conversation with Reddit communities in new ways.

Your First Reddit Video

We will continue to evolve Reddit video as we collect additional feedback from our users, so you can expect more updates in the coming months. In the meantime, if you’d like to try out your first video or gif post, here’s a step-by-step guide to uploading:

Desktop:

  • Click “Submit Link” on a Reddit video-enabled subreddit
  • Click “Choose File”
  • Select an MP4 or MOV video file to upload
  • Choose a thumbnail for your video
  • Add a title
  • Click “Submit”

Mobile Apps:

  • Click “Post something interesting” at the top of your “Home” feed
    • Or, on a Reddit video-enabled subreddit, click “Post to r/___”
  • Select “Image/Video”
  • Click “Camera” to record a new video or “Library” to select one you’ve already recorded
  • Add a title
  • Click “Post” to submit

(Startled cats, sneezing seals, and doggo eye-bleach not included.)


Want to give us feedback? Let us know your thoughts on r/beta and stay tuned for more updates.


Serverless Event Gateway

$
0
0

#Background

The innovation of serverless compute (FaaS) was huge. It dramatically decreased operational complexity and allowed developers to perform compute more easily than ever.

Then entered the Serverless Framework, which offered an application experience of functions and events around serverless computing. This is now widely known as serverless architecture.

Despite being relatively new, serverless architectures have proven themselves well. Serverless teams consistently exhibit shortened time to market, increased developer productivity and reduced operational overhead.

But there has been a missing piece. Developers have been locked into a single cloud provider, unable to perform service communication between various services. They have been left without a good way to perform service discovery across different teams and applications.

That is exactly why we made Event Gateway.

#Introducing: The Serverless Event Gateway

The Event Gateway is an open-source communication fabric for serverless architectures. It combines both API gateway and pub/sub functionality into a single experience.

Inside the Event Gateway, all data is considered to be an event. This lets developers react to data flows of all their applications in a centralized way, with serverless compute.

This is powerful; when developers can manage those data flows from a single place, they can take events from one provider and trigger functions on another provider. Serverless architectures become truly cross-cloud.

#Features

The Serverless Event Gateway is the missing piece of serverless architectures.

#Cross-cloud

Businesses do not want to be limited by where they can access their data. With Event Gateway, any of your events can have multiple subscribers from any other cloud service. Lambda can talk to Azure can talk to OpenWhisk.

This makes businesses completely flexible. Building an events-first experience that exists cross-cloud and on-premise protects you from lock-in, while also keeping you open for whatever else the future may bring.

#Open Source

The Event Gateway is open-source and platform agnostic. Use it to create the cohesive nervous system of your digital business.

Run it on all the major cloud providers, on-premise or in a hybrid architecture. Unify events from all over your system. Even teams who are working on separate applications can easily share resources that shave time and overhead.

#Tightly integrates with Serverless Framework

The Event Gateway ties right into the Serverless Framework and is available for developers to use locally today.

#Get started

Use the Event Gateway to start taking full advantage of the serverless cloud. Serverless architectures just got their missing backbone.

The Event Gateway is currently in beta, and is available to use locally via the Serverless Framework. To check out the code, see the repo here and walk through the example app.

The Matasano Crypto Challenges (2013)

$
0
0

The Matasano Crypto Challenges

I recently took some time to work through the Matasano crypto challenges, a set of 48 practical programming exercises that Thomas Ptacek and his team at Matasano Security have developed as a kind of teaching tool (and baited hook).

Much of what I know (or think I know) about security has come from reading tptacek's comments on Hacker News, so I was intrigued when I first saw him mention the security challenges a few months ago. At the same time, I worried that I'd be way out of my depth attempting them.

As a programmer, my core strengths have always been knowing how to apologize to users, and composing funny tweets. While I can hook up a web template to a database and make the squigglies come out right, I cannot efficiently sort something for you on a whiteboard, or tell you where to get a monad. From my vantage point, crypto looms as high as Mount Olympus.

To my delight, though, I was able to get through the entire sequence. It took diligence, coffee, and a lot of graph paper, but the problems were tractable. And having completed them, I've become convinced that anyone whose job it is to run a production website should try them, particularly if you have no experience with application security.

Since the challenges aren't really documented anywhere, I wanted to describe what they're like in the hopes of persuading busy people to take the plunge.

You get the challenges in batches of eight by emailing cryptopals at Matasano, and solve them at your own pace, in the programming language of your choice. Once you finish a set, you send in the solutions and Sean unlocks the next eight. (Curiously, after the third set, Gmail started rejecting my tarball as malware.)

Most of the challenges take the form of practical attacks against common vulnerabilities, many of which will be sadly familiar to you from your own web apps. To keep things fun and fair for everyone, they ask you not to post the questions or answers online. (I cleared this post with Thomas to make sure it was spoiler-free.)

The challenges start with some basic string manipulation tasks, but after that they are grouped by theme. In most cases, you first implement something, then break it in several enlightening ways. The constructions you use will be familiar to any web programmer, but this may be the first time you have ever taken off the lid and looked at the moving parts inside.

Here are the cryptographic topics covered:

Going into the challenges, I worried that my math wouldn't be up to the task. My impression of Serious Crypto was that it required all kinds of group theory, abstract algebra, elliptic curves, vector spaces, and other scary stuff. But while this may be true, the math content for the practical challenges was much gentler:

While the math concepts weren't hard, getting a real feel for them took work (and this was the point of the exercise).

If you're an experienced programmer, the Matasano challenges are also a terrific excuse to try a new programming language. It's always much more fun to solve real problems than it is to write a Manager object that inherits from Employee.

Here are the language features I found myself using most:

  • string manipulation (ranges, substrings)
  • bitwise operators
  • lookup hashes
  • conversion between string and number formats
  • big integer operations
  • packing and unpacking binary data
  • pattern matching
  • url manipulation
  • client/server interaction over a socket

Altogether it took me about three weeks to do the full cycle, working pretty intensively. Skilled programmers will find the going much faster, especially if you're comfortable with bit twiddling. Very few of the problems were downright hard, though some required several hours of work. I spent most of my time stepping through algorithms in pursuit of bugs, and in the process really got a feel for the moving parts in various cryptographic constructions.

I would compare the experience to having only ever read cookbooks and watched cooking shows, and then being asked to fry an egg. You know exactly what to do... in principle.

Some of the challenges have a payoff, in that you decrypt a short bit of secret text. This is incredibly fun. Seeing a cracked message come up on the screen after an evening of bug chasing reminded me of how it felt to be a kid in front of my Apple ][, finally getting it to beep or draw a circle or print DONGS all over the screen. Some of the later challenges even display the answer 'Hollywood style', where you get to see it decrypt one letter at a time in a cascade of print statements.

While the rules don't stipulate it, I think it's a good idea not to look at anyone's code if you try the challenges. The goal here is to convert message-board levels of understanding into actual knowledge, and the only way that works is if you bang your head on the task without seeing how anyone else has done it. Sean was really helpful in helping me navigate difficult spots, and the challenges are not set up to intentionally trick you. But you will need the kind of graph paper with the small squares.

What surprised me most:

  1. How practical these attacks were. A lot of stuff that I knew was weak in principle (like re-using a nonce or using a timestamp as a 'random' seed) turns out to be crackable within seconds by an art major writing crappy Python.

  2. There is no difference, from the attacker's point of view, between gross and tiny errors. Both of them are equally exploitable. In at least three challenges, the mere fact of getting distinguishable error messages was enough to recover the entire message.

  3. This lesson is very hard to internalize. In the real world, if you build a bookshelf and forget to tighten one of the screws all the way, it does not burn down your house

  4. Timing attacks are much more effective than I imagined.

  5. Someone who can muck with your ciphertext is halfway to reading it, possibly with your secret key for dessert.

  6. Some mistakes are incredibly non-obvious. I had no idea you had to super-carefully pad RSA, for example.

  7. Even on a laptop, in 10 minutes you can do a terrifying amount of computation. It really is 2013.

I mentioned earlier that I thought every web programmer should try their hand at these. It is very illuminating to look at your own web app from the vantage point of an attacker actually writing code. At the very least, you will never be confused about cipher block modes again, or have to worry that someone will ask you to explain how a public key works in an interview. And there is a whole slew of dumb mistakes you will now avoid (replacing them with smarter mistakes that will become the subject matter of challenges 48-96).

The best part, from a web app developer's perspective, is that you never once write a SQL statement or HTML tag.

Here are some specific lessons from the challenges that I will apply to my own work:

  1. Keep meaningful data out of tokens (like cookies) that I hand out to clients. Use random values keyed against a database, memory store, or wherever.

  2. If I have to put data in tokens, include an integrity check, and pay a real crypto person to vet it.

  3. I must never seed a PRNG with a timestamp. I used to do this with microsecond precision thinking I was being clever. Then I went ahead and wrote a script that guessed the seed value in just a few seconds, and now I will never do that again.

  4. Use constant-time string comparisons when testing incoming data against some target value for authentication purposes. This is easy enough to do in most languages to make it cheap insurance.

  5. Anything related to authentication should only fail in one way. I must not provide distinguishable errors to the user.

  6. If possible, find a way to log the fact that someone is making a lot of weird queries against my site. For extra points, try not to make the logger itself hackable.

  7. No third-party javascript. I hated it already, now I hate it more.

  8. Cut off one of my fingers each time I re-use a nonce.

Having read this post, you can go to Hacker News and comment in Talmudic detail about what is right or wrong in the conclusions I drew. But a much better idea is to just email Sean and have a crack at the challenges yourself. You will have a good time!

One final observation. Crypto is like catnip for programmers. It is hard to keep us away from it, because it's challenging and fun to play with. And programmers respond very badly to the insinuation that they're not clever enough to do something. We see the F-16 just sitting there, keys in the ignition, no one watching, lights blinking, ladder extended. And some infosec nerd is telling us we're can't climb in there, even though we just want to taxi around a little and we've totally read the manual.

Doing these challenges is a great way to 'shake your sillies out', as Raffi might say, without hurting yourself or your users. You get to put on the flight suit, climb into the simulator, and crash that plane in every conceivable way.

I would like to sincerely thank Thomas and Sean and everyone at Matasano who worked on these challenges, and implore people in other technical fields to consider offering something similar. It's the most fun I've had programming in years!

—maciej on April 18, 2013

The Attention Economy and the Net (1997)

$
0
0
The Attention Economy: The Natural Economy of the NetFirst Monday
The Attention Economy: The Natural Economy of the Net

If the Web and the Net can be viewed as spaces in which we will increasingly live our lives, the economic laws we will live under have to be natural to this new space. These laws turn out to be quite different from what the old economics teaches, or what rubrics such as "the information age" suggest. What counts most is what is most scarce now, namely attention. The attention economy brings with it its own kind of wealth, its own class divisions - stars vs. fans - and its own forms of property, all of which make it incompatible with the industrial-money-market based economy it bids fair to replace. Success will come to those who best accommodate to this new reality.

Greetings
Change Happens
A Feudal Hope
The New Natural Economy
A Driving Force
Chatting, But Not Necessarily About Anything
Illusory Attention
The Effect of the Audience
A Miniature Working Model
A Material Economy Falls Victim to Its Own Success
It's Not for Productivity
A Point Worth Repeating, Though Not Too Often
Organizations Diminish as Transparency Grows
Material Things Reinterpreted
Wealth and Property Take New Forms, Too
Money and Attention
Business as Performance
Further Expectations
Advice for the Transition
A Closing Scenario
The End
Notes

Explanatory note: This article began as a draft of a conference[ * ] presentation, and has been left pretty much in that form. Another version was actually presented.

This is a conference on the "Economics of Digital Information." My guess is that most of the speakers, and most of the listeners interpret that title to mean that while "digital information" requires special consideration enough to justify a special conference, the basic meaning of the word "economics" can be taken for granted. What we are to be concerned with is how prices, costs, productivity, and so forth apply to digital information.

My vantage point is quite different. What we mean by economics cannot be taken for granted if what we are talking about is the economics which applies, say, to the Internet, or more generally to cyberspace, or more generally still, to life in the foreseeable future. We are moving into a period wholly different from the past era of factory-based mass production of material items when talk of money, prices, returns on investment, laws of supply and demand, and so on all made excellent sense. We now have to think in wholly new economic terms, for we are entering an entirely new kind of economy. The old concepts will just not have value in that new context.

Of course, there is nothing so new about the insight that the Internet is part of a revolutionary change in the way we do things and also in why we do them. Many names for the new era have been invoked: the information age, the Third Wave, the move towards cyberspace, all of which point, vaguely at least to the fact that new patterns of activity and of interrelationships among people are now emerging. The trouble with that insight is that it is so vague that you can easily agree with it without feeling the necessity of changing your economic thinking in the least. My effort over the past several years - it's embarrassing to admit how many - has been to overcome that vagueness, to come up with specifics about what this revolution actually implies. My conclusions are that we are headed into what I call the attention economy.

Before offering any details about the new economy itself I want to deal with a feeling you no doubt have. "Economics is economics; it really can't change." Even if you are not saying that in so many words, I feel fairly confident it is somewhere in your mind at this point. To try to convince you at least to have some doubts about that certainty, let me invoke two different analogies. (Since it is obviously beyond my capabilities to explain the full workings of an entire new economy in the brief time available here, getting you to take the thought of it seriously would not be a useless accomplishment.) The first analogy comes from science. Most scientists would agree that early in its existence, the planet Earth held no life. There were various kinds of minerals, volcanoes, sea water, chemicals in solution - lots going on, but all of it understandable in terms of the laws of physics, chemistry, geology. Then, fairly suddenly, some chemical molecules began to commingle in a new way, capable of growing and reproducing. Life had emerged, and, in its tremendous variety, grew and flourished according to completely new laws, the laws of molecular biology, of physiology, of ecology and so on.

To try to understand life solely on the basis of the old laws of physics and chemistry, would be an enormous, crippling mistake; you couldn't talk about the most obvious things, like sex or aging or digestion or species or parasites, since those are all biological concepts that have no place in physics or chemistry. The parallel I want to draw is that the new kinds of connection that the Net and cyberspace make possible also demand a whole new way of thinking if you are to understand what is going on between people, the kinds of organized effort that are now possible, the motivations that most matter, and a host of other facets of life.

This analogy is imperfect in one way though. I don't mean to imply that the new concepts of economics we need come on top of or in addition to the old concepts. Rather, economics is about the overall patterns of effort and motivation that shape our lives, and it is these patterns and motivations that are changing. That implies a wholly new set of economic laws that replace the ones we all have learned.

My second analogy should make this point more clearly. It also involves looking back to an earlier time, but, instead of billions of years ago we now must think back a mere five centuries. The expansion into cyberspace now underway parallels the expansion of European civilization into North and South America that followed Columbus's discoveries, exactly 498 years before Tim Berners-Lee discovered, or rather invented, the Web. Europe back then in the 15th century was still ruled pretty much on feudal lines, and the feudal lords took it for granted that the new world would be a space for more of a feudal economy, with dukes and counts and barons and earls ruling over serfs throughout the newly discovered continents. They did in fact begin to set up that system, but it was not what turned out to flourish in the new space. Instead, the capitalist, market-based industrial economy, then just starting out, found the new soil much more congenial. Eventually it grew so strong in North America that, when it re-crossed the ocean, it finally completed its move to dominance in Western Europe and then elsewhere in the world [ 1 ].

Contemporary economic ideas stem from that selfsame market-based industrialism, which was thoroughly different from the feudal, subsistence-farming-based economy that preceded it. We tend to think, as we are taught, that economic laws are timeless. That is plain wrong. Those laws hold true in particular periods and in a particular kind of space. The characteristic landscape of feudalism, dotted with small fields, walled villages, and castles, differs markedly from the industrial landscape of cities, smokestack factories and railroads, canals, or superhighways. The "landscape" of cyberspace exists only in our minds, perhaps, but even so it is where we are increasingly coming to live, and it looks nothing like either of those others. If cyberspace grows to encompass interactions between the billions of people now on the planet, those kinds of interaction will be utterly different from what prevailed for the last few centuries, or ever before [ 2 ].

If you want to thrive in this new world, it behooves you not to mistake it for a place where the dukes and earls of today will naturally continue to prosper, but rather to learn to think in terms of the economy natural to it [ 3 ].

So, at last, what is this new economy about? Well if the Net exemplifies it, then you might guess it has less to do with material things than with the kinds of entity that can flow through the Net. We are told over and over just what that is: information. Information, however, would be an impossible basis for an economy, for one simple reason: economies are governed by what is scarce, and information, especially on the Net, is not only abundant, but overflowing. We are drowning in the stuff, and yet more and more comes at us daily. That is why terms like "information glut" have become commonplace, after all. Furthermore, if you have any particular piece of information on the Net, you can share it easily with anyone else who might want it. It is not in any way scarce, and therefore it is not an information economy towards which we are moving. What would be the incentive in organizing our lives around spewing out more information if there is already far too much?

Well, my title gives it away, of course. There is something else that moves through the Net, flowing in the opposite direction from information, namely attention. So seeking attention could be the very incentive we are looking for. Parenthetically, I have now rejected both parts of the conference title; no economics in the conventional sense, and not digital information either. You might conclude I am speaking at the wrong conference. I would rather say it has the wrong title. Except the title did serve its purpose. It did get your attention, and that was something, in fact a lot.

Attention, at least the kind we care about, is an intrinsically scarce resource [ 4 ]. Consider yours, right now. You are reading this paper, or more likely, since it is intended to be delivered at a conference, listening to me speaking it. You have a certain stock of attention at your disposal, and right now, a large proportion of the stock available to you is going to me, or to my words. Note that if I am standing in front of you it is difficult to distinguish between paying attention to me and paying attention to my words or thoughts; you can hardly do one without doing the other. If you are just reading this, assuming it gets printed in a book, the fact that your attention is going to me and not just to what I write may be slightly less obvious. So it is convenient to think of being in the audience at this conference in order to consider what attention economics is all about.

First of all, if this talk is not a total bust, at this moment I am getting attention from a considerable audience. There is a net flow of attention towards me. If this is a reasonably polite group, there may be no great competition for your attention at the moment, but nonetheless, if there were, you would have to choose, or someone else, say the chair, would. The assembled audience cannot really pay attention to very many people speaking at once, usually not to more than one, in fact. Which is another way to say that the scarcity of attention is real and limiting.

Now this might not matter if attention were not desirable and valuable in itself, but it is. In fact, it is a very nice feeling to have respectful attention from everybody within earshot, no matter how many people that may include. We have a word to describe a very attentive audience, and that word is "enthralled." A thrall is basically a slave. If, for instance, I should take it in my head to mention panda bears, you who are paying attention are forced to think "panda bears," a thought you had no inkling would come up when you decided to listen to this talk. Now let me ask, how many of you, on hearing the word "panda" saw a glimpse of a panda in your imagination? Raise your hands, please. Thank you. ... A ha.

What just happened? I had your attention and I was able to convert it into a physical action on some of your parts, raising your hands. It comes with the territory. That is part of the power that goes with having attention, a point I will have reason to return to. Right now, it should be evident that having your attention means that I have the power to bend your minds and your bodies to my will, within limits that in turn have to do with how good I am at enthralling you. This can be a remarkable power. When you have superb control over your own body, so that you can perform great athletic feats, it feels great; likewise, it feels good when your mind feels focused and powerful; how much more wonderful then to be able to have the minds and bodies of others at your disposal! On the rather rare occasions when I have felt I was holding an audience "in the palm of my hand, hanging on my every word," I have very much enjoyed the feeling, and of course others who have felt the same have reported their feelings in the same terms. The elation is independent of what you happen to be talking about, even if it is to decry something you think is horrible.

This is not a particularly huge audience, but it is possible to enthrall any number of people if you can reach them and if you are good enough at it. So having attention is very, very desirable, in some ways infinitely so, since the larger the audience, the better. And, yet, attention is also difficult to achieve owing to its intrinsic scarcity. That combination makes it the potential driving force of a very intense economy.

Of course, not everybody necessarily wants a great deal of attention, just as in a money economy not everybody wants a great deal of money or many of the material goods that money can buy. But, just as in a money economy practically everyone must have some money to survive, so attention in some quantities is pretty much a prerequisite for survival, and attention is actually far more basic. This has always been the case for tiny babies. About the only thing they can get for themselves, or can give, is attention, which they begin to do within a half hour of birth, by smiling at those who smile at them. Without attention an infant could never satisfy its material needs, for food, warmth, fresh diapers, being burped, and so on. At a slightly later stage infants and toddlers need attention if they are to develop any sense of themselves as persons, and neither of those needs ever completely goes away. So even if you do not especially make a point of reaching for attention, even if you are very shy and reclusive, you still probably cannot do without some minimum, which however reluctantly, you may have to fight for. And no matter how humble you now may be, at some time in your own childhood you certainly sought attention, or you wouldn't be here.

As we move towards an attention economy in a fuller sense, the ethos of the old economy which makes it often bad taste or a poor strategy to consciously seek attention seems to be giving way to an attitude that makes having a lot of attention rather admirable and seeking it not at all to be frowned upon. Think of the sorts of things people are now willing to admit about themselves just to get on the likes of Oprah or the Sally Jesse Raphael show. Even the President of the United States is willing to discuss his underwear on nationwide television.

But I am running a bit ahead of myself. Before saying more about the workings of the attention economy and its ramifications, I have to offer you a bit more of an idea about how to view different situations in terms of the exchange of attention. Earlier I suggested that when information flows one way through the Net, attention has to be flowing the other. Now I want to say that it would be even better to think in terms of attention of some kind flowing both ways.

Consider an ordinary conversation. You could describe it as the exchange of information, but except in a highly technical sense that is rarely a very accurate description of what takes place. A conversation is primarily an exchange of attention. When you say "how are you?" for instance, you don't really want to know, as a rule, but if whomever you're talking with chooses to say how he or she is, it is more to get attention from you than to convey information. Even if this person genuinely thought you did want to know about her/his health, in answering, s/he would be attempting to pay attention to you. And even if you, in turn genuinely did want to know, the usual reason would be to pay attention to her/him.

Information, in the sense of something not previously known to one of the parties or another is secondary, if present at all. If I want your attention for any reason, I might begin by asking you for information, such as who you are and what you do, not necessarily because that is of great interest to me, but because it is a good way to get your attention. Children ask countless questions with this motive often patently obvious, and adults are not necessarily any different. Even if I am desperately searching for some fact that you happen to know, to get it from you I first have to get your attention. So what really matters in every conversation is the exchange of attention -- an exchange that normally must be kept more or less equal if one party or the other isn't likely to lose interest.

Now, let us come back to the example of this conference, in fact the very interchange going on between me and you at this moment. If you are still paying attention, it is at least in part because what I am saying interests you; that is, to some extent I am addressing some need or desire that you now have. Thus it appears, in a certain sense that I am paying each of you attention individually, even though I can't really be doing that. Of course, in this setting it helps that I have some idea of why you are here, but I obviously am not in a position to focus on your individual needs. If just the two of us were having a conversation, rather than my standing up here and reading this paper to this whole audience, you would be quite rightly incensed if instead of pausing to answer your questions or seeing whether you were still interested I just talked on and on in this fashion. As another sign of the asymmetry between us, if I leave the room after this talk, I would be extremely unlikely to be able to recognize a particular one of you three months from now, though you might well be able to recognize me.

What I am trying to get at here is that while you would normally want a conversation to involve a more or less equal exchange of attention, in the special circumstances that you are listening to a speaker, your feelings about what is a fair exchange are altered. What I would suggest is going on is less that I am providing you with information that you deem in advance will be of value, than that I am offering you individually the illusion of my full attention. I don't claim to be very good at this, but what I have done to some extent is to set up some expectations in you about what I will get to by the time the talk is finished, and any sense of progress towards that goal then feels as if I am filling your need, even though it is a need I have subtly created. (Any speaker must somehow do this, of course, to hold attention.)

If rhetoric is the art of persuasive speech, then anyone who speaks or writes or seeks attention in any way has to become something of a success in the special rhetoric of persuading listeners, readers, and so on, that he or she is meeting their individual needs, when in fact some of these needs have been artfully set up in advance [ 5 ]. You want to know what I am driving at, for instance, because I have already provided clues galore that I am driving at something that should matter to you.

My success, if any, in meeting these expectations I have myself set up in you will appear to be attention - call it illusory attention - that flows from me to you. That helps create an apparent equality of attention, and it can in fact go beyond that to create a feeling of obligation on your part or the part of other readers or listeners. The audience members can each feel they have not paid as much attention to a speaker as the speaker has paid personally to them, even though, in a very real sense the reverse is closer to the truth. The speaker may still not know them from Adam though they have the speaker's visage, voice, and thoughts permanently etched in memory.

Much more is going on here. One thing is the question of why you started listening in the first place. Well one reason is that I was introduced by the chair, who had your attention already, she was paying attention to the committee that set up this conference, in particular to Brian Kahin. He in turn paid attention to Esther Dyson, who gets paid a lot of attention. And indeed you possibly came here because you saw Esther's name on the organizing committee, and you already had gotten used to paying her attention. A key truth is that if you have the attention of an audience, you can then pass that on to someone else. For instance, if I happened to spot a friend of mine in the audience, or just chose someone at random, I could turn over all of your attention to that person.

Now, the fact that attention can be passed on from someone who has it to someone else, and on and on, is of course a vital feature if there is to be anything resembling an economy. We will return to this general point. But right now, I want to combine the idea that I could pass the whole audience's attention on to you with the thought I introduced before that you can feel in a certain sense that I am paying attention to you specifically - what I referred to as illusory attention. Since I observably do have at least a good fraction of the whole audience's attention, if I were to pay attention specifically to you in reality, by singling you out, I would of course be paying not only my own attention but that of everyone else here, and yet, it would seem to be arriving at you through me.

And now, just a few more quick points about this conference. First, the whole conference works pretty much as an attention economy. While you are here, your main concern is how you pay attention and where you pay it, perhaps whether you get enough in return to have a chance at being one of the conference stars, perhaps only through the brilliance of the questions you ask. Even between sessions, the exchange of attention is what mostly tends to occupy people at a conference. Of course, there are material considerations, such as having enough to eat, a comfortable chair, etc. But they tend to be secondary issues, taken for granted, and not occupying much attention. We are living a temporary attention economy in miniature right at this moment.

It bears repeating: We are living a temporary attention economy in miniature right at this moment.quote It should be evident by now that everyone has always lived with some degree of an attention economy, but through most of human history it hasn't been primary. Material needs and the production of material goods or the provision of purely material and basically impersonal services such as railways held sway. Even fifty years ago, the percentage of the American population that could take basic material needs for granted and didn't work directly in factories or on farms was much smaller than it is today.

If you look at how you live your life when you are not attending this conference, you will probably see that quite a bit of what you personally do is better characterized as involving attention transactions than monetary transactions. You most likely make many more decisions every day about where and towards whom your attention should now go than about where your or anyone else's money should go. It is an issue every time you get a phone call, receive a memo, see someone you know waving at you, decide whether to go to a movie, or surf the Web, to list just a few examples. You are probably quite concerned too with getting attention in one way or another, or perhaps helping someone else get it. In this you are typical of a growing proportion of our society, and indeed of almost every sizable society on this globe now.

The simple fact, which I have no time to discuss at any length, is that compared with our capacity to produce material things, our net capacity to consume those things can no longer keep pace. Thus fewer and fewer of us, on a percentage basis are involved in producing standard items than ever before, and this is true despite the fact that per capita consumption of material goods keep rising. It just cannot rise fast enough to keep pace with possible production. There just is not enough work of the older kinds to keep us as busy as we once were. So, for example, actual manufacturing employment as a fraction of the total population continues its slow decline. Even in so-called developing nations, the Green Revolution in agriculture has led to the same sort of decline in the number employed producing material things, including food crops.

Yet strangely, we are all busier than ever. In fact, in the light of what I have been saying so far, that is not so odd. It is precisely because material needs at the creature comfort level are fairly well satisfied for all those in a position to demand them that the need for attention, or what is closely related to attention, meaning or meaningfulness of life, takes on increasing importance. In other words, the energies set free by the successes of what I refer to as the money-industrial economy go more and more in the direction of obtaining attention. And that leads to growing competition for what is increasingly scarce, which is of course attention. It sets up an unending scramble, a scramble that also increases the demands on each of us to pay what scarce attention we can.

And because we all need some attention, as competition for it rises, the effort begins to take on still more importance. When real attention of the right sort is unavailable, one has to make do to make do with the illusory kind, which comes through an increasing variety of media: paperback books, sound recordings, movies, radio, magazines, TV, video, and most recently computer software, CD-ROMs and the Web.

But the longing to get real attention and lots of it is only intensified by that experience. If the average kid today at age twenty has seen over 30,000 hours of TV, and, if, as is often suggested, TV offers young viewers role models for acceptable behavior, then the one thing everyone visible on the tube has in common to model is going after attention and getting it. This is also what is universally modeled by rock stars, successful athletes, politicians, and to a lesser degree even by school teachers and college professors.

So it is no coincidence that some of the most popular uses of computers, fax machines, networks, phone systems, etc., have more to do with getting attention than with directly aiding what they are supposedly about, increasing productivity of an organization or society as a whole [ 6 ]. quote For an important truth is getting attention is of primary value to individuals rather than organizations, and attention also flows from individuals. This conference is sponsored by several organizations, most notably Harvard University, and quite possibly additional organizations have sent more than one attendee apiece. However, within the confines of the conference, attention flows primarily irrespective of organizational affiliation.

If you are after attention, you use whatever organization you are part of as a stage upon which to perform for as wide an audience as you can manage. The Web and the Internet fit well in this model. The physical walls and barriers that might once have defined a university, a government bureau or an industrial corporation, making outside and inside sharply distinct, are pretty much no barriers at all on the Web or the Internet, or even on a phone system equipped not with a central switchboard allowing an operator to direct every incoming call but, as most are today, with direct inward dialing. You often don't even know what organization goes with the number you are dialing, the e-mail message you are responding to or the particular Web site you have been linked to.

In a full attention economy practically all organizations will be basically temporary, either communities in which attention is shared around pretty equally, or, more often, entourages of fans who form around one or a few stars to help them achieve the performances they are attempting. Think of the groups that come together to make a movie or to create a new piece of software, etc. More often than not, a few stars dominate the process; in the case of a movie, it is not only the main actors, but the directors, writer, producer, and possibly the cinematographer, the chief editor, and a few others. If the movie is to be made, everyone else involved focuses their attention on these stars; afterwards, the stars usually go their separate ways, bringing together different entourages for their next performance.

This might be good point to add that since it is hard to get new attention by repeating exactly what you or someone else has done before, this new economy is based on endless originality, or at least attempts at originality. By contrast, the old industrial economy worked on the basis of making interchangeable objects in huge numbers. One could spend a lifetime of work in a factory, for instance, repeating the same motions over and over, polishing the same small area on car after car, for instance. And it was such repetition that allowed standard prices for things and standard wages for definite jobs to make sense. The entire money system is based on the simultaneous inter-changeability of units of money, on the one hand, and of standardized goods on the other. One dollar is as good as another; one quart of non-fat milk is as good as another; both statements must be true, or non-fat milk will have no price.

With the endless originality and diversity of the attention economy, that kind of exchange is no longer possible. Even though one can loosely compare amounts of attention paid to different performances, attention does not come in precise, indistinguishable units, and neither does the illusory attention for which it is exchanged.

Again, I digress. Let me return to the thread I have been trying to follow: the breakdown of organizational barriers. The Web and other media aid this development by allowing you to look behind the scenes as easily as at them. Gossip, interviews, biographies of individuals involved in specific efforts, photos, videos of rehearsals, documentaries of pre-performance steps, all are visible or can be visible on the Web, taking equal status with the final performances themselves. Documentaries about the production of movies are common by now; a movie about a movie is just as accessible as the first movie.

This transparency will even more be the case in the very near future, and, as a result, organizations will diminish in importance at rapid pace, relative to the importance of the individuals who are temporarily in them. Even as stable and long-lasting an institution as Harvard will be less its familiar buildings and more the people in the buildings, and the networks of attention among them. And whether these people are physically at Harvard or somewhere else will matter less and less, until the institution loses all coherence, all distinctness from other universities or from any one of hundreds of other organizations which have audiences in common.

In a full-fledged attention economy the goal is simply to get either enough attention or as much as possible. Recall now what I pointed out earlier: if you have a person's full attention, you can get them to perform physical acts, ranging from moving their eyes to follow you, to raising their hands, to applauding, to bringing you a glass of water, to handing you a sandwich, or, as is not uncommon in the case of rock groupies or sports fans, having sex with you (to cite a notorious example). Just as a parent paying attention to a child fills its material wants and desires, so a fan, that is anyone paying attention can feel an obligation or a desire to do the same for whomever they are paying attention to.

In an attention economy as confined as a conference of this sort, the material goods such as a snack or a sandwich come from outside the system. If the whole world is an attention economy, then making material goods, growing food from scratch in a garden or on a farm, or obtaining resources in any other fashion, and ultimately turning these over to you can be a direct act of attention paying. Thus, if you have enough attention, you can get anything you want. If you don't have enough your options will be distinctly more limited, but supplying you with some range of items, produced in a fairly automated fashion, can also be a successful form of paying you illusory attention, in return for some real attention that you pay to whomever is apparently doing this for you.

One lesson to draw is that material goods and the acts of producing them are only secondary in an attention economy. quote What is primary is attention in the form of hanging on your every word or gesture. Paying attention in that sense is not over when its over. If what I say to you today makes any impression at all, for instance, you will remember me as well as some of the message for some time, possibly even for the rest of you life. Even if you find what I say outrageous or stupid, it will be easier for you to tune into me the next time I come across your field of vision, however that might happen. That is, getting attention is not a momentary thing; you build on the stock you have every time you get any, and the larger your audience at one time, the larger your potential audience in the future. Thus obtaining attention is obtaining a kind of enduring wealth, a form of wealth that puts you in a preferred position to get anything this new economy offers.

Wealth that can endure and sometimes be added to is what we mean by property. Thus, in the new economy attention itself is property. Where is it? Primarily it is located in the minds of those who have paid you attention in the past, whether years ago or seconds ago. You may have forgotten all about some children's author whose books you had read to you as a child, but if you come across the book again, your memory will very likely be reawakened. Likewise you will remember actors you saw on television, sports figures who captured your attention in the past, professors, teachers, politicians, business leaders, etc. Thus, attention wealth can apparently decline, only to revive later. It is rarely entirely lost.

Seeing this kind of wealth as property suggests a strategy for maintaining and enlarging what you have that is far different from what is usually considered to be the case when dealing with ideas or information. Suppose you get attention through some text you send out over the Internet. Would you want your audience to copy this and pass it on to others who might pay attention in turn? Of course you would. It would be insane to want to stop or restrain such copying, since that would deprive you of much attention you could otherwise get. This is an area, clearly, where the new economy and the old are at sharp odds. Thus the fight over intellectual property and rights to make copies is actually a struggle between the outlooks of the new economy and the old, a reason why they cannot both coexist forever, and thus a feature of the period of transition from old to new.

So let's now take up the topic of this of transition, which has been underway for some time and will loom still larger in the next few years. I have described the attention economy itself without saying anything about the role of money in it, which was easy because in a pure attention economy money has no essential function, no real role to play. In the period of transition from old economy to new, however, the connection between money and attention is significant and needs examining. If you have a lot of attention, you are a star of one sort or another, and we all know that these days stars generally have little trouble obtaining money in large amounts. Just think of the amounts that go to movie stars, sports stars, or even leading politicians or generals who retire to the lecture circuit or propose to oversee the ghostwriting of their memoirs. And if they have some pet project, such as a movie they want to make or a cause they want supported they can often influence their publics or bankers to cough up many millions more.

Within the framework I have suggested, there is little mystery as to why this should be. If fans are willing to do anything up to some limit for stars, such as wait in long lines to see them perform, avidly make sure to be there when they come to town, applaud them and sing their praises however they can, often paying more attention to stars than to members of their own families and so on, then it should come as no surprise that fans are also willing to pay out money at the stars' behest. It is just one more way to follow a star's wishes.

In other words, money now flows along with attention, or, to put this in more general terms, when there is a transition between economies, the old kind of wealth easily flows to the holders of the new. Thus, when the market-based, proto-industrial economy first began to replace the feudal system of Western Europe, in which the prime form of wealth was aristocratic lineage and inheritance of land, both the noble titles and the lands that went with them soon ended up disproportionately in the hands of those who were good at obtaining what was then the new kind of wealth, namely money.

With considerable ease, the rising merchant and industrialist class could buy old titles, induce governments to grant them brand new ones, or marry into the old impoverished gentry. The parallel today, again, is that possessors of today's rising kind of wealth, which is attention, and whom we label stars of every sort, have an easy time getting money.

But now let me point out that the other way round doesn't work nearly as easily. Contrary to what you are sometimes urged to believe, money cannot reliably buy attention. Suppose it did work that way. Then you could have been paid to sit here and listen closely even if I were to read you something as boring as the phone book or an unabridged dictionary. Presumably it wouldn't even matter if I kept repeating the same few syllables over and over. If money could reliably buy attention, all I would have to do is pay you the required amount and you would keep listening carefully through all that, not falling asleep en masse, nor allowing your minds to wander. In truth, even if you had been paid a huge sum, this would be most difficult, and if you did it, it would be a testament more to your own deep sense of principle than to a general condition in which another roomful of similar people could be expected to do equally well.

Someone who wants your attention just can't rely on paying you money to get it, but has to do more, has to be interesting, that is must offer you illusory attention, in just about the same amounts as they would if you had instead been paying money to listen to them -- which by the way is closer to the case here. Money flows to attention, and much less well does attention flow to money.

There are exceptions, if only in a peculiar way. Business is still a lively spectator sport for instance, and just as we care about who is the best or the record breaker in football or basketball or any other such game, so we are interested in who heads lists like the Forbes 400 of those with the most money. According to some, like Ted Turner, who are on that list, in fact the main motive for trying to earn still more boils down to wanting to be recognized as number one. Presumably, Bill Gates might want to hold the record for more annual first-place finishes than anyone else. Even in this regard however, when the amount of monetary wealth you have draws attention to you, the price of such fame keeps going up. Even more literally does that happen in the well-known cases of the ultra-rich seeking political office. The price they must pay per vote keeps rising, and no matter how good the advertising geniuses they hire, they have to be good at attracting attention on their own.

Bill Gates is also a good example of how even monetary fortunes of his magnitude are in larger and larger measure just covers for stardom. A century ago, Gates' analog would have been John D. Rockefeller, leader and chief owner of the Standard Oil Trust. His wealth consisted chiefly of oil fields, oil wells, tanker cars, refineries, and so on -- material things that would have been worth just as much if someone else bought him out. Rockefeller could have sold his interests and still kept about the same net worth, which is what monetary net worth is supposed to mean. But the share value of a company such as Microsoft is already far more a result of attention-getting and the star process. Its future sales, for instance, largely depend on software that is yet to be completely designed.

If Gates were to decide to sell out and buy control of the XYZ Corp. instead of staying at the helm of Microsoft, as soon as he let this be known, his Microsoft stock would fall precipitously and XYZ's would rise. His own net worth would plummet, at least temporarily, but such is the attention wealth he has, that as soon as he began to issue pronouncements from his new stage, XYZ's stock would probably rise further, and Gates' former monetary wealth might magically reappear. Despite the fact that the arena in which he made his mark happens to be business, it is already true that Gates' actual wealth, and that of many like him, is less in money or shares of stock than in attention.

I hope that by now you have some sense that there is far more to discuss here or to think about than I can conveniently explain or you can take in at one sitting. So let me now just summarize a few developments that seem reasonable to expect over the next decade or so:
  • A continuing rapid rise in the number of people attached to the Web and trying to get attention through it.
  • A continuing growth in the capacity of those on the Web to send out multimedia or virtual reality signals, and thus to capture attention through all these means. Say you are primarily a writer of mere words, i.e. text; still, on the Web you will be able to supplement your writings with your picture, with video images, with recordings of your voice, with interviews or pieces of autobiography. The advantage of doing that is that by offering potential readers a more vivid and rounded sense of who you are, you can both increase their sense of who it is who is offering them illusory attention, and have them have a clearer and more definite feeling than otherwise of what it is like to pay attention to you, rather than to some other writer of similar sounding words. Both these effects can help you hold their attention better. This of course helps explain why authors' pictures are so commonly stuck on book jackets, and increasingly on the front cover rather than the back.
  • All this and more will make the Web a better and better means of transmitting and circulating attention, a circulation that is essential for a full-fledged economy to emerge. To show that most strikingly, consider an author in the distant past, say the ancient Greek philosopher Aristotle. Over the past than two thousand years and more, his writings have gotten the direct attention of probably millions of readers. Still, except for contributing to his "immortality," the vast majority of that attention did him little personal good, since it came when he, along with all known descendants, had been long dead. Very few of today's attention getters can expect to remain in the public eye for thousands of years, but they do have a far better shot at reaping the benefits of attention from millions of people through the Web while they are still alive. Thus they can live, and live well, in the new economy.
  • Individual attention getters of all sorts will find it ever easier to get attention directly through the Web, without any corporate packaging necessary. They will also find diminishing advantage in trying to make use of money, since attention in a wider and wider a variety of forms, filling more and more of their needs will be able to flow to them either directly through the Web, or as a kind of adjunct to it.
  • Companies of all kinds will have less definite and fixed structures, since they will be structured not by physical walls and buildings, but through the Net itself, and more and more of their proceedings will be done in the full glare of Web attention, as temporary and rapidly re-forming projects. This means that companies will be unable to provide even what loyalty they do now to their employees, or say, in the case of publishers, to authors who have signed with them. Just as baseball stars move around from team to team or TV stars from network to network, so employee loyalties to companies will decrease as well. What will matter more for everyone is the stars one has particular loyalty to, or the Net communities of which one is a part and through which one gets attention.
  • Attention transactions, which already are far more numerous than monetary transactions will come to dominate even further. So even if you have lots of money, you will find it less and less convenient or worthwhile to bother to use it. As a result, our deeply ingrained desire for monetary recompense will begin to fade as well.
All this means that the changeover to total domination by the new economy, while by no means complete or about to be, is moving very rapidly and is already quite far along, and probably unstoppable. Any individual today who fails to take that into account in her or his personal plans may be in for a rude awakening. Efforts that fit in with the overall flow of things are far more likely to work to your benefit than those that ignore them or are consciously opposed.

Say for example you work for a book publisher today. If you have any sense, you understand your employer as temporary. You will either strive to achieve stardom through what you do in your current job directly - say by being a great editor, a great marketer of books, a very visible cover designer or something of the sort - or (and this is not an exclusive but an inclusive or) you will want to be as visible and indispensable a part of what I call the entourages of bigger stars, so that through them you can get indirect attention. Your interest in your company's success as such is like a Major League baseball player's interest in his current team's success, something that can help him shine, and valuable to the extent that it does, but less valuable if it keeps him from displaying what he does best.

Simply amassing money (say by investing a large chunk of your salary in stocks) is not necessarily the best strategy if you believe you can do that without bothering to capture and in some way maintain some attention of your own. Even if the stock market never goes down, money, like the aristocratic titles of the past, may turn out to be less and less meaningful in the future.

A publisher also has to decide how to deal with the Internet. At present, for instance, it is impractical to distribute books directly over the Net, though it is easy to foresee that need not be the case for long. We still do understand material things as objects that generally are to be bought and sold in exchange for money, but we also understand that more people are likely to pay attention to a book if they find out about it than if they don't. So in the case of a book, the Internet should now be viewed as a useful and free publicity mechanism. Let passages be freely copied and circulated on the Net, because most of the time, the more of copying that takes place, the more customers there will be for the physical printed version. If you have a Web site, don't charge for it, because that will only reduce the attention it gets. If you can't figure out how to afford it without charging, you may be doing something wrong.

In due time, publishing companies as such will hardly be necessary, for actual physical books will be seen as cumbersome and quaint. quote Still, many of the kinds of tasks once performed by publishing company employees such as acquisition and line editors, designers, publicists, and so, will still be done, but on much more ad hoc and free-lance, eventually even unpaid basis. All of this will take place over the Web. No one will earn monetary profits from it. And this disappearance of the involvement of capital will be equally the case for attention-getting objects of just about any sort.

Money will not necessarily fade in value, in other words inflation will not set in, in the old sense; neither will recession nor deflation. Instead, money will just lose importance, just as noble titles have over the past few centuries. The stock market might not even fall; stockholders may simply lose interest, ceasing to sell and buy in equal ratio.

Am I speaking about the far future? I think not. Already, if you are reading this, you are probably involved in far more organized person-to-person or audience-type situations where what is being exchanged is attention, real and illusory, than you are in direct monetary transactions or the direct production of material goods. The fraction of time spent in pursuits more closely tied to the new economy is, even now, well above fifty per cent and rising. The new practices are already almost fully functioning for some, and more and more in place for others.

At the end of the feudal period, the pomp and display of the nobility reached a level never before attained; the most gorgeous armor, the most magnificent tournaments of knights, the most elaborate ceremonies between rival nobles, the most brilliant marriages, the greatest interest in noble lineage. But by then it had lost all real function or importance. So today, when the stock market goes up and up, when money wealth itself seems a source of fame more than ever, when being number one on Forbes 400 list seems the height of perfection, when every basketball superstar wants a contract that is at least a million more than the last record one, we seem to be more dazzled by money than ever, just as we seem to be more intrigued by material goods than ever. But these interests are superficial and faddish. They are signs of decadence not of a glorious future for the money economy. Even in themselves they speak to the growing desire for attention, the need for it as well. Money is now little more numbers, one number among many, and as a source of lasting attention it can fade in an instant. The attention economy is already here, and more completely so every day.End of article

Michael H. Goldhaber is completing a book on the attention economy. Formerly a theoretical physicist, a Fellow of the Institute for Policy Studies in Washington, D. C. and editor of Post-Industrial Issues, he is currently head of his own think tank, The Center for Technology and Democracy, and is a visiting scholar at UC Berkeley's institute for the Study of Social Change. is previous book was Reinventing Technology.

His Web site is http://www.well.com/user/mgoldh/ E-mail: mgoldh@well.com
©Michael H. Goldhaber, 1997

The conference was on "Economics of Digital Information," hosted by the Kennedy School of Government , Harvard University, Cambridge, Mass., January 23-26, 1997.

1. To be more exact, in Western Europe as whole, feudalism as an economic system reached its high point around the eleventh or twelfth century (i.e. between 1000 and 1200). After that the market economy began its slow rise. But the outward forms and ways of thinking long remained feudal, certainly in the Iberian peninsula whence the first explorers came. In the Americas, where feudal systems hadn't previously existed, they were unable to compete with the new economic ways that most of the settlers brought with them. As is most obvious in the case of the Puritan colonists in New England, many of these settlers quite consciously had come to escape the old forms of rule. The "Puritan Ethic" they brought with them was much more suitable to a capitalistic, market economy than to feudalism. The great text that argues the last point (though ignoring earlier economic history) is Max Weber, 1958. The Protestant Ethic and the Spirit of Capitalism. New York: Scribner.

2. Just as settlers in the Americas fashioned the geography they found to fit their purposes and values, so cyberspace is being shaped largely by those who want a space for their own new purposes. As I suggest elsewhere (Michael Goldhaber, 1986. Reinventing Technology. New York: Routledge & Kegan Paul) technology (such as that which goes into cyberspace) is shaped by the values of those who create it and it then helps promote those values, in the main, as it allows certain actions and not others. In the case of the kinds of technology (such as software) that make up cyberspace, the users play a very large role in deciding in what directions the technology as a whole will advance, and their underlying purposes and values are more in the direction of the new economy I will outline than the old.

3. Despite its seeming generality, the following definition, (Paul Samuelson, 1973. Economics. New York: McGraw-Hill, p. 3) as read by millions of students of basic economics, shows why this new thinking must be very basic:

Economics is the study of how men and society end up choosing, with or without the use of money, to employ scarce productive resources that could have alternate uses, to produce various commodities and distribute them for consumption, now or in the future among various people and groups in society. It analyzes the costs and benefits of improving patterns of resource allocation.

As will become evident, "employing scarce productive resources," "produc[ing] various commodities and distributing them for consumption" and "improving patterns of resource allocation" are simply not relevant for what I will argue is unfolding. Nor is this a particularly perspicacious way of examining older economies, .e.g. feudalism.

4. On attention's scarcity and its economic importance, see also Michael H. Goldhaber, 1989. "Equality and Education in America Now," In: Education and the American Dream, H. Holtz, I. Marcus, J. Dougherty, J. Michaels, and R. Peduzzi (eds.), Granby, Mass.: Bergin & Garvey, Chapter 6, pp. 70-76; Michael H. Goldhaber, 1992. "The Attention Society," Release 1.0, ( 26 March), No. 3, E. Dyson (ed.), New York, EDventure Holdings, pp. 1-20; Michael H. Goldhaber, 1992. "Attention: The System of Post Industrialism?" Z papers, Vol. 1, No. 2 (April-June); and, Michael H. Goldhaber, 1996-97, Web site: http://www.well.com/user/mgoldh/

I still remember the thunderclap of insight that attention, not information is the key to the new system, a thought that struck me in 1984. While the details I present about the new economy stem from my own explorations, the fact that the following people, among others, have independently arrived at similar conclusions about the economic centrality of attention scarcity adds weight to the argument. See, for example, Richard Lanham, 1994. "The Economics of Attention," Proceedings of 124th Annual Meeting, Association of Research Librarians, Austin, Texas, http://sunsite.berkeley.edu/ARL/Proceedings/124/ps2econ.html or W. Thorngate, 1988. "On Paying Attention." In: Recent Trends in Theoretical Psychology, W. Baker, L. Mos, H. VanRappard, and H. Stam (eds.), New York: Springer-Verlag, (pp. 247-264), or W. Thorngate, 1990. "The Economy of Attention and the Development of Psychology," Canadian Psychology/Psychologie Canadienne, Vol. 31, pp. 262-271.

5. The rhetorician Kenneth Burke (in his 1931 book Counter-Statement, New York: Harcourt, Brace, p. 157) describes literary form in a very similar manner: "Form in literature is an arousing and fulfilling of desires."

6. Controversy continues to swirl around this point. It is argued at length by Thomas K. Landauer ( in his 1995 book The Trouble with Computers, Cambridge, Mass.: MIT Press) among others. Erik Brynjolfsson and Lorin Hitt (1995, "Information Technology as a Factor of Production: The Role of Differences Among Firms," Econ. Innov. New Techn., Vol. 3, pp. 183-199) present data revealing an overall positive correlation between total amounts of spending on information technology and total output for Fortune 500 companies. However, they do not show an increase in labor productivity per se, as is commonly presumed to be the case. What is indisputable is in the two decades since the introduction of the personal computer and related technologies, national measured productivity growth was lower than in the two decades following World War II, when such technology was either non-existent or much more limited. That is totally the opposite from what intuitive estimates of the value of these technologies would suggest and what has repeatedly been predicted.


While it would be impossible to thank everyone who has contributed to this lengthy project, I would like especially to thank Anatole Anton, Sandra Braman, Erik Brynjolfsson, Esther Dyson, Rishab Ghosh, William Gladstone, Nat Goldhaber, Peter Oppenheimer, Bruce Sterling, Edward Valauskas, and Terry Winograd for comments and/or encouragement that aided in the writing of this article. In addition I would like to thank Ilene Philipson. No one on this list should be held responsible for anything said here, however.


Copyright © 1997, First Monday

The Attention Economy and the Net by Michael H. Goldhaber
First Monday, Volume 2, Number 4 - 7 April 1997
http://firstmonday.org/ojs/index.php/fm/article/view/519/440


A Great Cities Initiative of the University of Illinois at Chicago University Library.

© First Monday, 1995-2017. ISSN 1396-0466.

Peer-to-Peer Communication Across Network Address Translators (2005)

$
0
0
Peer-to-Peer Communication Across Network Address Translators

Bryan Ford
Massachusetts Institute of Technology
baford (at) mit.edu

Pyda Srisuresh
Caymas Systems, Inc.
srisuresh (at) yahoo.com

Dan Kegel
dank (at) kegel.com

J'fais des trous, des petits trous$\dots$
toujours des petits trous

     - S. Gainsbourg

Abstract:

Network Address Translation (NAT) causes well-known difficulties for peer-to-peer (P2P) communication, since the peers involved may not be reachable at any globally valid IP address. Several NAT traversal techniques are known, but their documentation is slim, and data about their robustness or relative merits is slimmer. This paper documents and analyzes one of the simplest but most robust and practical NAT traversal techniques, commonly known as “hole punching.” Hole punching is moderately well-understood for UDP communication, but we show how it can be reliably used to set up peer-to-peer TCP streams as well. After gathering data on the reliability of this technique on a wide variety of deployed NATs, we find that about 82% of the NATs tested support hole punching for UDP, and about 64% support hole punching for TCP streams. As NAT vendors become increasingly conscious of the needs of important P2P applications such as Voice over IP and online gaming protocols, support for hole punching is likely to increase in the future.

The combined pressures of tremendous growth and massive security challenges have forced the Internet to evolve in ways that make life difficult for many applications. The Internet's original uniform address architecture, in which every node has a globally unique IP address and can communicate directly with every other node, has been replaced with a new de facto Internet address architecture, consisting of a global address realm and many private address realms interconnected by Network Address Translators (NAT). In this new address architecture, illustrated in Figure 1, only nodes in the “main,” global address realm can be easily contacted from anywhere in the network, because only they have unique, globally routable IP addresses. Nodes on private networks can connect to other nodes on the same private network, and they can usually open TCP or UDP connections to “well-known” nodes in the global address realm. NATs on the path allocate temporary public endpoints for outgoing connections, and translate the addresses and port numbers in packets comprising those sessions, while generally blocking all incoming traffic unless otherwise specifically configured.

The Internet's new de facto address architecture is suitable for client/server communication in the typical case when the client is on a private network and the server is in the global address realm. The architecture makes it difficult for two nodes on different private networks to contact each other directly, however, which is often important to the “peer-to-peer” communication protocols used in applications such as teleconferencing and online gaming. We clearly need a way to make such protocols function smoothly in the presence of NAT.

One of the most effective methods of establishing peer-to-peer communication between hosts on different private networks is known as “hole punching.” This technique is widely used already in UDP-based applications, but essentially the same technique also works for TCP. Contrary to what its name may suggest, hole punching does not compromise the security of a private network. Instead, hole punching enables applications to function within the the default security policy of most NATs, effectively signaling to NATs on the path that peer-to-peer communication sessions are “solicited” and thus should be accepted. This paper documents hole punching for both UDP and TCP, and details the crucial aspects of both application and NAT behavior that make hole punching work.

Unfortunately, no traversal technique works with all existing NATs, because NAT behavior is not standardized. This paper presents some experimental results evaluating hole punching support in current NATs. Our data is derived from results submitted by users throughout the Internet by running our “NAT Check” tool over a wide variety of NATs by different vendors. While the data points were gathered from a “self-selecting” user community and may not be representative of the true distribution of NAT implementations deployed on the Internet, the results are nevertheless generally encouraging.

While evaluating basic hole punching, we also point out variations that can make hole punching work on a wider variety of existing NATs at the cost of greater complexity. Our primary focus, however, is on developing the simplest hole punching technique that works cleanly and robustly in the presence of “well-behaved” NATs in any reasonable network topology. We deliberately avoid excessively clever tricks that may increase compatibility with some existing “broken” NATs in the short term, but which only work some of the time and may cause additional unpredictability and network brittleness in the long term.

Although the larger address space of IPv6 [3] may eventually reduce the need for NAT, in the short term IPv6 is increasing the demand for NAT, because NAT itself provides the easiest way to achieve interoperability between IPv4 and IPv6 address domains [24]. Further, the anonymity and inaccessibility of hosts on private networks has widely perceived security and privacy benefits. Firewalls are unlikely to go away even when there are enough IP addresses: IPv6 firewalls will still commonly block unsolicited incoming traffic by default, making hole punching useful even to IPv6 applications.

The rest of this paper is organized as follows. Section 2 introduces basic terminology and NAT traversal concepts. Section 3 details hole punching for UDP, and Section 4 introduces hole punching for TCP. Section 5 summarizes important properties a NAT must have in order to enable hole punching. Section 6 presents our experimental results on hole punching support in popular NATs, Section 7 discusses related work, and Section 8 concludes.

This section introduces basic NAT terminology used throughout the paper, and then outlines general NAT traversal techniques that apply equally to TCP and UDP.

This paper adopts the NAT terminology and taxonomy defined in RFC 2663 [21], as well as additional terms defined more recently in RFC 3489 [19].

Of particular importance is the notion of session. A session endpoint for TCP or UDP is an (IP address, port number) pair, and a particular session is uniquely identified by its two session endpoints. From the perspective of one of the hosts involved, a session is effectively identified by the 4-tuple (local IP, local port, remote IP, remote port). The direction of a session is normally the flow direction of the packet that initiates the session: the initial SYN packet for TCP, or the first user datagram for UDP.

Of the various flavors of NAT, the most common type is traditional or outbound NAT, which provides an asymmetric bridge between a private network and a public network. Outbound NAT by default allows only outbound sessions to traverse the NAT: incoming packets are dropped unless the NAT identifies them as being part of an existing session initiated from within the private network. Outbound NAT conflicts with peer-to-peer protocols because when both peers desiring to communicate are “behind” (on the private network side of) two different NATs, whichever peer tries to initiate a session, the other peer's NAT rejects it. NAT traversal entails making P2P sessions look like “outbound” sessions to both NATs.

Outbound NAT has two sub-varieties:Basic NAT, which only translates IP addresses, and Network Address/Port Translation (NAPT), which translates entire session endpoints. NAPT, the more general variety, has also become the most common because it enables the hosts on a private network to share the use of a single public IP address. Throughout this paper we assume NAPT, though the principles and techniques we discuss apply equally well (if sometimes trivially) to Basic NAT.

The most reliable--but least efficient--method of P2P communication across NAT is simply to make the communication look to the network like standard client/server communication, through relaying. Suppose two client hosts $A$ and $B$ have each initiated TCP or UDP connections to a well-known server $S$, at $S$'s global IP address 18.181.0.31 and port number 1234. As shown in Figure 2, the clients reside on separate private networks, and their respective NATs prevent either client from directly initiating a connection to the other. Instead of attempting a direct connection, the two clients can simply use the server $S$ to relay messages between them. For example, to send a message to client $B$, client $A$ simply sends the message to server $S$ along its already-established client/server connection, and server $S$ forwards the message on to client $B$ using its existing client/server connection with $B$.

Relaying always works as long as both clients can connect to the server. Its disadvantages are that it consumes the server's processing power and network bandwidth, and communication latency between the peering clients is likely increased even if the server is well-connected. Nevertheless, since there is no more efficient technique that works reliably on all existing NATs, relaying is a useful fall-back strategy if maximum robustness is desired. The TURN protocol [18] defines a method of implementing relaying in a relatively secure fashion.

Some P2P applications use a straightforward but limited technique, known as connection reversal, to enable communication when both hosts have connections to a well-known rendezvous server $S$ and only one of the peers is behind a NAT, as shown in Figure 3. If $A$ wants to initiate a connection to $B$, then a direct connection attempt works automatically, because $B$ is not behind a NAT and $A$'s NAT interprets the connection as an outgoing session. If $B$ wants to initiate a connection to $A$, however, any direct connection attempt to $A$ is blocked by $A$'s NAT.$B$ can instead relay a connection request to $A$ through a well-known server $S$, asking $A$ to attempt a “reverse” connection back to $B$. Despite the obvious limitations of this technique, the central idea of using a well-known rendezvous server as an intermediary to help set up direct peer-to-peer connections is fundamental to the more general hole punching techniques described next.

UDP hole punching enables two clients to set up a direct peer-to-peer UDP session with the help of a well-known rendezvous server, even if the clients are both behind NATs. This technique was mentioned in section 5.1 of RFC 3027 [10], documented more thoroughly elsewhere on the Web [13], and used in recent experimental Internet protocols [17,11]. Various proprietary protocols, such as those for on-line gaming, also use UDP hole punching.

Hole punching assumes that the two clients,$A$ and $B$, already have active UDP sessions with a rendezvous server $S$. When a client registers with $S$, the server records two endpoints for that client: the (IP address, UDP port) pair that the client believes itself to be using to talk with $S$, and the (IP address, UDP port) pair that the server observes the client to be using to talk with it. We refer to the first pair as the client's private endpoint and the second as the client's public endpoint. The server might obtain the client's private endpoint from the client itself in a field in the body of the client's registration message, and obtain the client's public endpoint from the source IP address and source UDP port fields in the IP and UDP headers of that registration message. If the client is not behind a NAT, then its private and public endpoints should be identical.

A few poorly behaved NATs are known to scan the body of UDP datagrams for 4-byte fields that look like IP addresses, and translate them as they would the IP address fields in the IP header. To be robust against such behavior, applications may wish to obfuscate IP addresses in messages bodies slightly, for example by transmitting the one's complement of the IP address instead of the IP address itself. Of course, if the application is encrypting its messages, then this behavior is not likely to be a problem.

Suppose client $A$ wants to establish a UDP session directly with client $B$. Hole punching proceeds as follows:

  1. $A$ initially does not know how to reach $B$, so $A$ asks $S$ for help establishing a UDP session with $B$.
  2. $S$ replies to $A$ with a message containing $B$'s public and private endpoints. At the same time,$S$ uses its UDP session with $B$ to send $B$ a connection request message containing $A$'s public and private endpoints. Once these messages are received,$A$ and $B$ know each other's public and private endpoints.
  3. When $A$ receives $B$'s public and private endpoints from $S$,$A$ starts sending UDP packets toboth of these endpoints, and subsequently “locks in” whichever endpoint first elicits a valid response from $B$. Similarly, when $B$ receives $A$'s public and private endpoints in the forwarded connection request,$B$ starts sending UDP packets to $A$ at each of $A$'s known endpoints, locking in the first endpoint that works. The order and timing of these messages are not critical as long as they are asynchronous.

We now consider how UDP hole punching handles each of three specific network scenarios. In the first situation, representing the “easy” case, the two clients actually reside behind the same NAT, on one private network. In the second, most common case, the clients reside behind different NATs. In the third scenario, the clients each reside behind two levels of NAT: a common “first-level” NAT deployed by an ISP for example, and distinct “second-level” NATs such as consumer NAT routers for home networks.

It is in general difficult or impossible for the application itself to determine the exact physical layout of the network, and thus which of these scenarios (or the many other possible ones) actually applies at a given time. Protocols such as STUN [19] can provide some information about the NATs present on a communication path, but this information may not always be complete or reliable, especially when multiple levels of NAT are involved. Nevertheless, hole punching works automatically in all of these scenarioswithout the application having to know the specific network organization, as long as the NATs involved behave in a reasonable fashion. (“Reasonable” behavior for NATs will be described later in Section 5.)

First consider the simple scenario in which the two clients (probably unknowingly) happen to reside behind the same NAT, and are therefore located in the same private IP address realm, as shown in Figure 4. Client $A$ has established a UDP session with server $S$, to which the common NAT has assigned its own public port number 62000. Client $B$ has similarly established a session with $S$, to which the NAT has assigned public port number 62005.

Suppose that client $A$ uses the hole punching technique outlined above to establish a UDP session with $B$, using server $S$ as an introducer. Client $A$ sends $S$ a message requesting a connection to $B$.$S$ responds to $A$ with $B$'s public and private endpoints, and also forwards $A$'s public and private endpoints to $B$. Both clients then attempt to send UDP datagrams to each other directly at each of these endpoints. The messages directed to the public endpoints may or may not reach their destination, depending on whether or not the NAT supports hairpin translation as described below in Section 3.5. The messages directed at the private endpointsdo reach their destinations, however, and since this direct route through the private network is likely to be faster than an indirect route through the NAT anyway, the clients are most likely to select the private endpoints for subsequent regular communication.

By assuming that NATs support hairpin translation, the application might dispense with the complexity of trying private as well as public endpoints, at the cost of making local communication behind a common NAT unnecessarily pass through the NAT. As our results in Section 6 show, however, hairpin translation is still much less common among existing NATs than are other “P2P-friendly” NAT behaviors. For now, therefore, applications may benefit substantially by using both public and private endpoints.

Suppose clients $A$ and $B$ have private IP addresses behind different NATs, as shown in Figure 5.$A$ and $B$ have each initiated UDP communication sessions from their local port 4321 to port 1234 on server $S$. In handling these outbound sessions, NAT $A$ has assigned port 62000 at its own public IP address, 155.99.25.11, for the use of $A$'s session with $S$, and NAT $B$ has assigned port 31000 at its IP address, 138.76.29.7, to $B$'s session with $S$.

In $A$'s registration message to $S$,$A$ reports its private endpoint to $S$ as 10.0.0.1:4321, where 10.0.0.1 is $A$'s IP address on its own private network.$S$ records $A$'s reported private endpoint, along with $A$'s public endpoint as observed by $S$ itself.$A$'s public endpoint in this case is 155.99.25.11:62000, the temporary endpoint assigned to the session by the NAT. Similarly, when client $B$ registers,$S$ records $B$'s private endpoint as 10.1.1.3:4321 and $B$'s public endpoint as 138.76.29.7:31000.

Now client $A$ follows the hole punching procedure described above to establish a UDP communication session directly with $B$. First, $A$ sends a request message to $S$ asking for help connecting with $B$. In response, $S$ sends $B$'s public and private endpoints to $A$, and sends $A$'s public and private endpoints to $B$.$A$ and $B$ each start trying to send UDP datagrams directly to each of these endpoints.

Since $A$ and $B$ are on different private networks and their respective private IP addresses are not globally routable, the messages sent to these endpoints will reach either the wrong host or no host at all. Because many NATs also act as DHCP servers, handing out IP addresses in a fairly deterministic way from a private address pool usually determined by the NAT vendor by default, it is quite likely in practice that $A$'s messages directed at $B$'s private endpoint will reach some (incorrect) host on $A$'s private network that happens to have the same private IP address as $B$ does. Applications must therefore authenticate all messages in some way to filter out such stray traffic robustly. The messages might include application-specific names or cryptographic tokens, for example, or at least a random nonce pre-arranged through $S$.

Now consider $A$'s first message sent to $B$'s public endpoint, as shown in Figure 5. As this outbound message passes through $A$'s NAT, this NAT notices that this is the first UDP packet in a new outgoing session. The new session's source endpoint (10.0.0.1:4321) is the same as that of the existing session between $A$ and $S$, but its destination endpoint is different. If NAT $A$ is well-behaved, it preserves the identity of $A$'s private endpoint, consistently translatingall outbound sessions from private source endpoint 10.0.0.1:4321 to the corresponding public source endpoint 155.99.25.11:62000.$A$'s first outgoing message to $B$'s public endpoint thus, in effect, “punches a hole” in $A$'s NAT for a new UDP session identified by the endpoints (10.0.0.1:4321, 138.76.29.7:31000) on $A$'s private network, and by the endpoints (155.99.25.11:62000, 138.76.29.7:31000) on the main Internet.

If $A$'s message to $B$'s public endpoint reaches $B$'s NAT before $B$'s first message to $A$ has crossed $B$'s own NAT, then $B$'s NAT may interpret $A$'s inbound message as unsolicited incoming traffic and drop it.$B$'s first message to $A$'s public address, however, similarly opens a hole in $B$'s NAT, for a new UDP session identified by the endpoints (10.1.1.3:4321, 155.99.25.11:62000) on $B$'s private network, and by the endpoints (138.76.29.7:31000, 155.99.25.11:62000) on the Internet. Once the first messages from $A$ and $B$ have crossed their respective NATs, holes are open in each direction and UDP communication can proceed normally. Once the clients have verified that the public endpoints work, they can stop sending messages to the alternative private endpoints.

3.5 Peers Behind Multiple Levels of NAT

In some topologies involving multiple NAT devices, two clients cannot establish an “optimal” P2P route between them without specific knowledge of the topology. Consider a final scenario, depicted in Figure 6. Suppose NAT $C$ is a large industrial NAT deployed by an internet service provider (ISP) to multiplex many customers onto a few public IP addresses, and NATs $A$ and $B$ are small consumer NAT routers deployed independently by two of the ISP's customers to multiplex their private home networks onto their respective ISP-provided IP addresses. Only server $S$ and NAT $C$ have globally routable IP addresses; the “public” IP addresses used by NAT $A$ and NAT $B$ are actually private to the ISP's address realm, while client $A$'s and$B$'s addresses in turn are private to the addressing realms of NAT $A$ and NAT $B$, respectively. Each client initiates an outgoing connection to server $S$ as before, causing NATs $A$ and $B$ each to create a single public/private translation, and causing NAT $C$ to establish a public/private translation for each session.

Now suppose $A$ and $B$ attempt to establish a direct peer-to-peer UDP connection via hole punching. The optimal routing strategy would be for client $A$ to send messages to client $B$'s “semi-public” endpoint at NAT $B$, 10.0.1.2:55000 in the ISP's addressing realm, and for client $B$ to send messages to $A$'s “semi-public” endpoint at NAT $B$, namely 10.0.1.1:45000. Unfortunately, $A$ and $B$ have no way to learn these addresses, because server $S$ only sees the truly global public endpoints of the clients, 155.99.25.11:62000 and 155.99.25.11:62005 respectively. Even if $A$ and $B$ had some way to learn these addresses, there is still no guarantee that they would be usable, because the address assignments in the ISP's private address realm might conflict with unrelated address assignments in the clients' private realms. (NAT $A$'s IP address in NAT $C$'s realm might just as easily have been 10.1.1.3, for example, the same as client $B$'s private address in NAT $B$'s realm.)

The clients therefore have no choice but to use their global public addresses as seen by $S$ for their P2P communication, and rely on NAT $C$ providing hairpin or loopback translation. When $A$ sends a UDP datagram to $B$'s global endpoint, 155.99.25.11:62005, NAT $A$ first translates the datagram's source endpoint from 10.0.0.1:4321 to 10.0.1.1:45000. The datagram now reaches NAT $C$, which recognizes that the datagram's destination address is one of NAT $C$'s own translated public endpoints. If NAT $C$ is well-behaved, it then translates both the source and destination addresses in the datagram and “loops” the datagram back onto the private network, now with a source endpoint of 155.99.25.11:62000 and a destination endpoint of 10.0.1.2:55000. NAT $B$ finally translates the datagram's destination address as the datagram enters $B$'s private network, and the datagram reaches $B$. The path back to $A$ works similarly. Many NATs do not yet support hairpin translation, but it is becoming more common as NAT vendors become aware of this issue.

Since the UDP transport protocol provides NATs with no reliable, application-independent way to determine the lifetime of a session crossing the NAT, most NATs simply associate an idle timer with UDP translations, closing the hole if no traffic has used it for some time period. There is unfortunately no standard value for this timer: some NATs have timeouts as short as 20 seconds. If the application needs to keep an idle UDP session active after establishing the session via hole punching, the application must send periodic keep-alive packets to ensure that the relevant translation state in the NATs does not disappear.

Unfortunately, many NATs associate UDP idle timers with individual UDP sessions defined by a particular pair of endpoints, so sending keep-alives on one session will not keep other sessions active even if all the sessions originate from the same private endpoint. Instead of sending keep-alives on many different P2P sessions, applications can avoid excessive keep-alive traffic by detecting when a UDP session no longer works, and re-running the original hole punching procedure again “on demand.”

Establishing peer-to-peer TCP connections between hosts behind NATs is slightly more complex than for UDP, but TCP hole punching is remarkably similar at the protocol level. Since it is not as well-understood, it is currently supported by fewer existing NATs. When the NATs involved do support it, however, TCP hole punching is just as fast and reliable as UDP hole punching. Peer-to-peer TCP communication across well-behaved NATs may in fact be more robust than UDP communication, because unlike UDP, the TCP protocol's state machine gives NATs on the path a standard way to determine the precise lifetime of a particular TCP session.

The main practical challenge to applications wishing to implement TCP hole punching is not a protocol issue but an application programming interface (API) issue. Because the standard Berkeley sockets API was designed around the client/server paradigm, the API allows a TCP stream socket to be used to initiate an outgoing connection via connect(), or to listen for incoming connections via listen() and accept(),but not both. Further, TCP sockets usually have a one-to-one correspondence to TCP port numbers on the local host: after the application binds one socket to a particular local TCP port, attempts to bind a second socket to the same TCP port fail.

For TCP hole punching to work, however, we need to use a single local TCP port to listen for incoming TCP connections and to initiate multiple outgoing TCP connections concurrently. Fortunately, all major operating systems support a special TCP socket option, commonly named SO_REUSEADDR, which allows the application to bind multiple sockets to the same local endpoint as long as this option is set on all of the sockets involved. BSD systems have introduced a SO_REUSEPORT option that controls port reuse separately from address reuse; on such systems both of these options must be set.

4.2 Opening Peer-to-Peer TCP Streams

Suppose that client $A$ wishes to set up a TCP connection with client $B$. We assume as usual that both $A$ and $B$ already have active TCP connections with a well-known rendezvous server $S$. The server records each registered client's public and private endpoints, just as for UDP. At the protocol level, TCP hole punching works almost exactly as for UDP:

  1. Client $A$ uses its active TCP session with $S$ to ask $S$ for help connecting to $B$.
  2. $S$ replies to $A$ with $B$'s public and private TCP endpoints, and at the same time sends $A$'s public and private endpoints to $B$.
  3. From the same local TCP ports that $A$ and $B$ used to register with $S$,$A$ and $B$ each asynchronously make outgoing connection attempts to the other's public and private endpoints as reported by $S$, while simultaneously listening for incoming connections on their respective local TCP ports.
  4. $A$ and $B$ wait for outgoing connection attempts to succeed, and/or for incoming connections to appear. If one of the outgoing connection attempts fails due to a network error such as “connection reset” or “host unreachable,” the host simply re-tries that connection attempt after a short delay (e.g., one second), up to an application-defind maximum timeout period.
  5. When a TCP connection is made, the hosts authenticate each other to verify that they connected to the intended host. If authentication fails, the clients close that connection and continue waiting for others to succeed. The clients use the first successfully authenticated TCP stream resulting from this process.

Unlike with UDP, where each client only needs one socket to communicate with both $S$ and any number of peers simultaneously, with TCP each client application must manage several sockets bound to a single local TCP port on that client node, as shown in Figure 7. Each client needs a stream socket representing its connection to $S$, a listen socket on which to accept incoming connections from peers, and at least two additional stream sockets with which to initiate outgoing connections to the other peer's public and private TCP endpoints.

Consider the common-case scenario in which the clients $A$ and $B$ are behind different NATs, as shown in Figure 5, and assume that the port numbers shown in the figure are now for TCP rather than UDP ports. The outgoing connection attempts $A$ and $B$ make to each other's private endpoints either fail or connect to the wrong host. As with UDP, it is important that TCP applications authenticate their peer-to-peer sessions, due of the likelihood of mistakenly connecting to a random host on the local network that happens to have the same private IP address as the desired host on a remote private network.

The clients' outgoing connection attempts to each other's public endpoints, however, cause the respective NATs to open up new “holes” enabling direct TCP communication between $A$ and $B$. If the NATs are well-behaved, then a new peer-to-peer TCP stream automatically forms between them. If $A$'s first SYN packet to $B$ reaches $B$'s NAT before $B$'s first SYN packet to $A$ reaches $B$'s NAT, for example, then $B$'s NAT may interpret $A$'s SYN as an unsolicited incoming connection attempt and drop it.$B$'s first SYN packet to $A$ should subsequently get through, however, because $A$'s NAT sees this SYN as being part of the outbound session to $B$ that $A$'s first SYN had already initiated.

What the client applications observe to happen with their sockets during TCP hole punching depends on the timing and the TCP implementations involved. Suppose that $A$'s first outbound SYN packet to $B$'s public endpoint is dropped by NAT $B$, but $B$'s first subsequent SYN packet to $A$'s public endpoint gets through to $A$ before $A$'s TCP retransmits its SYN. Depending on the operating system involved, one of two things may happen:

  • $A$'s TCP implementation notices that the session endpoints for the incoming SYN match those of an outbound session $A$ was attempting to initiate.$A$'s TCP stack therefore associates this new session with the socket that the local application on $A$ was using to connect() to $B$'s public endpoint. The application's asynchronous connect() call succeeds, and nothing happens with the application's listen socket.

    Since the received SYN packet did not include an ACK for $A$'s previous outbound SYN,$A$'s TCP replies to $B$'s public endpoint with a SYN-ACK packet, the SYN part being merely a replay of $A$'s original outbound SYN, using the same sequence number. Once $B$'s TCP receives $A$'s SYN-ACK, it responds with its own ACK for $A$'s SYN, and the TCP session enters the connected state on both ends.

  • Alternatively,$A$'s TCP implementation might instead notice that $A$ has an active listen socket on that port waiting for incoming connection attempts. Since $B$'s SYN looks like an incoming connection attempt,$A$'s TCP creates a new stream socket with which to associate the new TCP session, and hands this new socket to the application via the application's next accept() call on its listen socket.$A$'s TCP then responds to $B$ with a SYN-ACK as above, and TCP connection setup proceeds as usual for client/server-style connections.

    Since $A$'s prior outbound connect() attempt to $B$ used a combination of source and destination endpoints that is now in use by another socket, namely the one just returned to the application via accept(),$A$'s asynchronous connect() attempt must fail at some point, typically with an “address in use” error. The application nevertheless has the working peer-to-peer stream socket it needs to communicate with $B$, so it ignores this failure.

The first behavior above appears to be usual for BSD-based operating systems, whereas the second behavior appears more common under Linux and Windows.

4.4 Simultaneous TCP Open

Suppose that the timing of the various connection attempts during the hole punching process works out so that the initial outgoing SYN packets from both clients traverse their respective local NATs, opening new outbound TCP sessions in each NAT, before reaching the remote NAT. In this “lucky” case, the NATs do not reject either of the initial SYN packets, and the SYNs cross on the wire between the two NATs. In this case, the clients observe an event known as a simultaneous TCP open: each peer's TCP receives a “raw” SYN while waiting for a SYN-ACK. Each peer's TCP responds with a SYN-ACK, whose SYN part essentially “replays” the peer's previous outgoing SYN, and whose ACK part acknowledges the SYN received from the other peer.

What the respective applications observe in this case again depends on the behavior of the TCP implementations involved, as described in the previous section. If both clients implement the second behavior above, it may be thatall of the asynchronous connect() calls made by the application ultimately fail, but the application running on each client nevertheless receives a new, working peer-to-peer TCP stream socket via accept()--as if this TCP stream had magically “created itself” on the wire and was merely passively accepted at the endpoints! As long as the application does not care whether it ultimately receives its peer-to-peer TCP sockets via connect() or accept(), the process results in a working stream on any TCP implementation that properly implements the standard TCP state machine specified in RFC 793 [23].

Each of the alternative network organization scenarios discussed in Section 3 for UDP works in exactly the same way for TCP. For example, TCP hole punching works in multi-level NAT scenarios such as the one in Figure 6 as long as the NATs involved are well-behaved.

4.5 Sequential Hole Punching

In a variant of the above TCP hole punching procedure implemented by the NatTrav library [4], the clients attempt connections to each other sequentially rather than in parallel. For example: (1) $A$ informs $B$ via $S$ of its desire to communicate,without simultaneously listening on its local port; (2) $B$ makes a connect() attempt to $A$, which opens a hole in $B$'s NAT but then fails due to a timeout or RST from $A$'s NAT or a RST from $A$ itself; (3) $B$ closes its connection to $S$ and does a listen() on its local port; (4) $S$ in turn closes its connection with $A$, signaling $A$ to attempt a connect() directly to $B$.

This sequential procedure may be particularly useful on Windows hosts prior to XP Service Pack 2, which did not correctly implement simultaneous TCP open, or on sockets APIs that do not support the SO_REUSEADDR functionality. The sequential procedure is more timing-dependent, however, and may be slower in the common case and less robust in unusual situations. In step (2), for example,$B$ must allow its “doomed-to-fail” connect() attempt enough time to ensure that at least one SYN packet traverses all NATs on its side of the network. Too little delay risks a lost SYN derailing the process, whereas too much delay increases the total time required for hole punching. The sequential hole punching procedure also effectively “consumes” both clients' connections to the server $S$, requiring the clients to open fresh connections to $S$ for each new P2P connection to be forged. The parallel hole punching procedure, in contrast, typically completes as soon as both clients make their outgoing connect() attempts, and allows each client to retain and re-use a single connection to $S$ indefinitely.

This section describes the key behavioral properties NATs must have in order for the hole punching techniques described above to work properly. Not all current NAT implementations satisfy these properties, but many do, and NATs are gradually becoming more “P2P-friendly” as NAT vendors recognize the demand for peer-to-peer protocols such as voice over IP and on-line gaming.

This section is not meant to be a complete or definitive specification for how NATs “should” behave; we provide it merely for information about the most commonly observed behaviors that enable or break P2P hole punching. The IETF has started a new working group, BEHAVE, to define official “best current practices” for NAT behavior. The BEHAVE group's initial drafts include the considerations outlined in this section and others; NAT vendors should of course follow the IETF working group directly as official behavioral standards are formulated.

5.1 Consistent Endpoint Translation

The hole punching techniques described here only work automatically if the NAT consistently maps a given TCP or UDP source endpoint on the private network to a single corresponding public endpoint controlled by the NAT. A NAT that behaves in this way is referred to as a cone NAT in RFC 3489 [19] and elsewhere, because the NAT “focuses” all sessions originating from a single private endpoint through the same public endpoint on the NAT.

Consider again the scenario in Figure 5, for example. When client $A$ initially contacted the well-known server $S$, NAT $A$ chose to use port 62000 at its own public IP address, 155.99.25.11, as a temporary public endpoint to representing $A$'s private endpoint 10.0.0.1:4321. When $A$ later attempts to establish a peer-to-peer session with $B$ by sending a message from the same local private endpoint to $B$'s public endpoint,$A$ depends on NAT $A$ preserving the identity of this private endpoint, and re-using the existing public endpoint of 155.99.25.11:62000, because that is the public endpoint for $A$ to which $B$ will be sending its corresponding messages.

A NAT that is only designed to support client/server protocols will not necessarily preserve the identities of private endpoints in this way. Such a NAT is a symmetric NAT in RFC 3489 terminology. For example, after the NAT assigns the public endpoint 155.99.25.11:62000 to client $A$'s session with server $S$, the NAT might assign a different public endpoint, such as 155.99.25.11:62001, to the P2P session that $A$ tries to initiate with $B$. In this case, the hole punching process fails to provide connectivity, because the subsequent incoming messages from $B$ reach NAT $A$ at the wrong port number.

Many symmetric NATs allocate port numbers for successive sessions in a fairly predictable way. Exploiting this fact, variants of hole punching algorithms [9,1] can be made to work “much of the time” even over symmetric NATs by first probing the NAT's behavior using a protocol such as STUN [19], and using the resulting information to “predict” the public port number the NAT will assign to a new session. Such prediction techniques amount to chasing a moving target, however, and many things can go wrong along the way. The predicted port number might already be in use causing the NAT to jump to another port number, for example, or another client behind the same NAT might initiate an unrelated session at the wrong time so as to allocate the predicted port number. While port number prediction can be a useful trick for achieving maximum compatibility with badly-behaved existing NATs, it does not represent a robust long-term solution. Since symmetric NAT provides no greater security than a cone NAT with per-session traffic filtering, symmetric NAT is becoming less common as NAT vendors adapt their algorithms to support P2P protocols.

5.2 Handling Unsolicited TCP Connections

When a NAT receives a SYN packet on its public side for what appears to be an unsolicited incoming connection attempt, it is important that the NAT just silently drop the SYN packet. Some NATs instead actively reject such incoming connections by sending back a TCP RST packet or even an ICMP error report, which interferes with the TCP hole punching process. Such behavior is not necessarily fatal, as long as the applications re-try outgoing connection attempts as specified in step 4 of the process described in Section 4.2, but the resulting transient errors can make hole punching take longer.

A few existing NATs are known to scan “blindly” through packet payloads for 4-byte values that look like IP addresses, and translate them as they would the IP address in the packet header, without knowing anything about the application protocol in use. This bad behavior fortunately appears to be uncommon, and applications can easily protect themselves against it by obfuscating IP addresses they send in messages, for example by sending the bitwise complement of the desired IP address.

5.4 Hairpin Translation

Some multi-level NAT situations require hairpin translation support in order for either TCP or UDP hole punching to work, as described in Section 3.5. The scenario shown in Figure 6, for example, depends on NAT $C$ providing hairpin translation. Support for hairpin translation is unfortunately rare in current NATs, but fortunately so are the network scenarios that require it. Multi-level NAT is becoming more common as IPv4 address space depletion continues, however, so support for hairpin translation is important in future NAT implementations.

To evaluate the robustness of the TCP and UDP hole punching techniques described in this paper on a variety of existing NATs, we implemented and distributed a test program called NAT Check [16], and solicited data from Internet users about their NATs.

NAT Check's primary purpose is to test NATs for the two behavioral properties most crucial to reliable UDP and TCP hole punching: namely, consistent identity-preserving endpoint translation (Section 5.1), and silently dropping unsolicited incoming TCP SYNs instead of rejecting them with RSTs or ICMP errors (Section 5.2). In addition, NAT Check separately tests whether the NAT supports hairpin translation (Section 5.4), and whether the NAT filters unsolicited incoming traffic at all. This last property does not affect hole punching, but provides a useful indication the NAT's firewall policy.

NAT Check makes no attempt to test every relevant facet of NAT behavior individually: a wide variety of subtle behavioral differences are known, some of which are difficult to test reliably [12]. Instead, NAT Check merely attempts to answer the question, “how commonly can the proposed hole punching techniques be expected to work on deployed NATs, under typical network conditions?”

NAT Check consists of a client program to be run on a machine behind the NAT to be tested, and three well-known servers at different global IP addresses. The client cooperates with the three servers to check the NAT behavior relevant to both TCP and UDP hole punching. The client program is small and relatively portable, currently running on Windows, Linux, BSD, and Mac OS X. The machines hosting the well-known servers all run FreeBSD.

6.1.1 UDP Test

To test the NAT's behavior for UDP, the client opens a socket and binds it to a local UDP port, then successively sends “ping”-like requests to servers 1 and 2, as shown in Figure 8. These servers each respond to the client's pings with a reply that includes the client's public UDP endpoint: the client's own IP address and UDP port number as observed by the server. If the two servers report the same public endpoint for the client, NAT Check assumes that the NAT properly preserves the identity of the client's private endpoint, satisfying the primary precondition for reliable UDP hole punching.

When server 2 receives a UDP request from the client, besides replying directly to the client it also forwards the request to server 3, which in turn replies to the client from its own IP address. If the NAT's firewall properly filters “unsolicited” incoming traffic on a per-session basis, then the client never sees these replies from server 3, even though they are directed at the same public port as the replies from servers 1 and 2.

To test the NAT for hairpin translation support, the client simply opens a second UDP socket at a different local port and uses it to send messages to the public endpoint representing the client's first UDP socket, as reported by server 2. If these messages reach the client's first private endpoint, then the NAT supports hairpin translation.

6.1.2 TCP Test

The TCP test follows a similar pattern as for UDP. The client uses a single local TCP port to initiate outbound sessions to servers 1 and 2, and checks whether the public endpoints reported by servers 1 and 2 are the same, the first precondition for reliable TCP hole punching.

The NAT's response to unsolicited incoming connection attempts also impacts the speed and reliability of TCP hole punching, however, so NAT Check also tests this behavior. When server 2 receives the client's request, instead of immediately replying to the client, it forwards a request to server 3 and waits for server 3 to respond with a “go-ahead” signal. When server 3 receives this forwarded request, it attempts to initiate an inbound connection to the client's public TCP endpoint. Server 3 waits up to five seconds for this connection to succeed or fail, and if the connection attempt is still “in progress” after five seconds, server 3 responds to server 2 with the “go-ahead” signal and continues waiting for up to 20 seconds. Once the client finally receives server 2's reply (which server 2 delayed waiting for server 3's “go-ahead” signal), the client attempts an outbound connection to server 3, effectively causing a simultaneous TCP open with server 3.

What happens during this test depends on the NAT's behavior as follows. If the NAT properly just drops server 3's “unsolicited” incoming SYN packets, then nothing happens on the client's listen socket during the five second period before server 2 replies to the client. When the client finally initiates its own connection to server 3, opening a hole through the NAT, the attempt succeeds immediately. If on the other hand the NAT does not drop server 3's unsolicited incoming SYNs but allows them through (which is fine for hole punching but not ideal for security), then the client receives an incoming TCP connection on its listen socket before receiving server 2's reply. Finally, if the NAT actively rejects server 3's unsolicited incoming SYNs by sending back TCP RST packets, then server 3 gives up and the client's subsequent attempt to connect to server 3 fails.

To test hairpin translation for TCP, the client simply uses a secondary local TCP port to attempt a connection to the public endpoint corresponding to its primary TCP port, in the same way as for UDP.

The NAT Check data we gathered consists of 380 reported data points covering a variety of NAT router hardware from 68 vendors, as well as the NAT functionality built into different versions of eight popular operating systems. Only 335 of the total data points include results for UDP hairpin translation, and only 286 data points include results for TCP, because we implemented these features in later versions of NAT Check after we had already started gathering results. The data is summarized by NAT vendor in Table 1; the table only individually lists vendors for which at least five data points were available. The variations in the test results for a given vendor can be accounted for by a variety of factors, such as different NAT devices or product lines sold by the same vendor, different software or firmware versions of the same NAT implementation, different configurations, and probably occasional NAT Check testing or reporting errors.

Table 1: User Reports of NAT Support for UDP and TCP Hole Punching
  UDPTCP
  Hole Hole 
  PunchingHairpinPunchingHairpin
NAT Hardware        
 Linksys45/46(98%)5/42(12%)33/38(87%)3/38(8%)
 Netgear31/37(84%)3/35(9%)19/30(63%)0/30(0%)
 D-Link16/21(76%)11/21(52%)9/19(47%)2/19(11%)
 Draytek2/17(12%)3/12(25%)2/7(29%)0/7(0%)
 Belkin14/14(100%)1/14(7%)11/11(100%)0/11(0%)
 Cisco12/12(100%)3/9(33%)6/7(86%)2/7(29%)
 SMC12/12(100%)3/10(30%)8/9(89%)2/9(22%)
 ZyXEL7/9(78%)1/8(13%)0/7(0%)0/7(0%)
 3Com7/7(100%)1/7(14%)5/6(83%)0/6(0%)
OS-based NAT        
 Windows31/33(94%)11/32(34%)16/31(52%)28/31(90%)
 Linux26/32(81%)3/25(12%)16/24(67%)2/24(8%)
 FreeBSD7/9(78%)3/6(50%)2/3(67%)1/1(100%)
All Vendors310/380(82%)80/335(24%)184/286(64%)37/286(13%)

Out of the 380 reported data points for UDP, in 310 cases (82%) the NAT consistently translated the client's private endpoint, indicating basic compatibility with UDP hole punching. Support for hairpin translation is much less common, however: of the 335 data points that include UDP hairpin translation results, only 80 (24%) show hairpin translation support.

Out of the 286 data points for TCP, 184 (64%) show compatibility with TCP hole punching: the NAT consistently translates the client's private TCP endpoint, and does not send back RST packets in response to unsolicited incoming connection attempts. Hairpin translation support is again much less common: only 37 (13%) of the reports showed hairpin support for TCP.

Since these reports were generated by a “self-selecting” community of volunteers, they do not constitute a random sample and thus do not necessarily represent the true distribution of the NATs in common use. The results are nevertheless encouraging: it appears that the majority of commonly-deployed NATs already support UDP and TCP hole punching at least in single-level NAT scenarios.

There are a few limitations in NAT Check's current testing protocol that may cause misleading results in some cases. First, we only learned recently that a few NAT implementations blindly translate IP addresses they find in unknown application payloads, and the NAT Check protocol currently does not protect itself from this behavior by obfuscating the IP addresses it transmits.

Second, NAT Check's current hairpin translation checking may yield unnecessarily pessimistic results because it does not use the full, two-way hole punching procedure for this test. NAT Check currently assumes that a NAT supporting hairpin translation does not filter “incoming” hairpin connections arriving from the private network in the way it would filter incoming connections arriving at the public side of the NAT, because such filtering is unnecessary for security. We later realized, however, that a NAT might simplistically treat any traffic directed at the NAT's public ports as “untrusted” regardless of its origin. We do not yet know which behavior is more common.

Finally, NAT implementations exist that consistently translate the client's private endpoint as long as only one client behind the NAT is using a particular private port number, but switch to symmetric NAT or even worse behaviors if two or more clients with different IP addresses on the private network try to communicate through the NAT from the same private port number. NAT Check could only detect this behavior by requiring the user to run it on two or more client hosts behind the NAT at the same time. Doing so would make NAT Check much more difficult to use, however, and impossible for users who only have one usable machine behind the NAT. Nevertheless, we plan to implement this testing functionality as an option in a future version of NAT Check.

Despite testing difficulties such as those above, our results are generally corroborated by those of a large ISP, who recently found that of the top three consumer NAT router vendors, representing 86% of the NATs observed on their network, all three vendors currently produce NATs compatible with UDP hole punching [25]. Additional independent results recently obtained using the UDP-oriented STUN protocol [12], and STUNT, a TCP-enabled extension [8,9], also appear consistent with our results. These latter studies provide more information on each NAT by testing a wider variety of behaviors individually, instead of just testing for basic hole punching compatibility as NAT Check does. Since these more extensive tests require multiple cooperating clients behind the NAT and thus are more difficult to run, however, these results are so far available on a more limited variety of NATs.

UDP hole punching was first explored and publicly documented by Dan Kegel [13], and is by now well-known in peer-to-peer application communities. Important aspects of UDP hole punching have also been indirectly documented in the specifications of several experimental protocols, such as STUN [19], ICE [17], and Teredo [11]. We know of no existing published work that thoroughly analyzes hole punching, however, or that points out the hairpin translation issue for multi-level NAT (Section 3.5).

We also know of no prior work that develops TCP hole punching in the symmetric fashion described here. Even the existence of the crucialSO_REUSEADDR/SO_REUSEPORT options in the Berkeley sockets API appears to be little-known among P2P application developers. NatTrav [4] implements a similar but asymmetric TCP hole punching procedure outlined earlier in Section 4.5. NUTSS [9] and NATBLASTER [1] implement more complex TCP hole punching tricks that can work around some of the bad NAT behaviors mentioned in Section 5, but they require the rendezvous server to spoof source IP addresses, and they also require the client applications to have access to “raw” sockets, usually available only at root or administrator privilege levels.

Protocols such as SOCKS [14], UPnP [26], and MIDCOM [22] allow applications to traverse a NAT through explicit cooperation with the NAT. These protocols are not widely or consistently supported by NAT vendors or applications, however, and do not appear to address the increasingly important multi-level NAT scenarios. Explicit control of a NAT further requires the application to locate the NAT and perhaps authenticate itself, which typically involves explicit user configuration. When hole punching works, in contrast, it works with no user intervention.

Recent proposals such as HIP [15] and FARA [2] extend the Internet's basic architecture by decoupling a host's identity from its location [20]. IPNL [7], UIP [5,6], and DOA [27] propose schemes for routing across NATs in such an architecture. While such extensions are probably needed in the long term, hole punching enables applications to work over the existing network infrastructure immediately with no protocol stack upgrades, and leaves the notion of “host identity” for applications to define.

Hole punching is a general-purpose technique for establishing peer-to-peer connections in the presence of NAT. As long as the NATs involved meet certain behavioral requirements, hole punching works consistently and robustly for both TCP and UDP communication, and can be implemented by ordinary applications with no special privileges or specific network topology information. Hole punching fully preserves the transparency that is one of the most important hallmarks and attractions of NAT, and works even with multiple levels of NAT--though certain corner case situations require hairpin translation, a NAT feature not yet widely implemented.

The authors wish to thank Dave Andersen for his crucial support in gathering the results presented in Section 6. We also wish to thank Henrik Nordstrom, Christian Huitema, Justin Uberti, Mema Roussopoulos, and the anonymous USENIX reviewers for valuable feedback on early drafts of this paper. Finally, we wish to thank the many volunteers who took the time to run NAT Check on their systems and submit the results.

1
Andrew Biggadike, Daniel Ferullo, Geoffrey Wilson, and Adrian Perrig.
NATBLASTER: Establishing TCP connections between hosts behind NATs.
In ACM SIGCOMM Asia Workshop, Beijing, China, April 2005.
2
David Clark, Robert Braden, Aaron Falk, and Venkata Pingali.
FARA: Reorganizing the addressing architecture.
In ACM SIGCOMM FDNA Workshop, August 2003.
3
S. Deering and R. Hinden.
Internet protocol, version 6 (IPv6) specification, December 1998.
RFC 2460.
4
Jeffrey L. Eppinger.
TCP connections for P2P apps: A software approach to solving the NAT problem.
Technical Report CMU-ISRI-05-104, Carnegie Mellon University, January 2005.
5
Bryan Ford.
Scalable Internet routing on topology-independent node identities.
Technical Report MIT-LCS-TR-926, MIT Laboratory for Computer Science, October 2003.
6
Bryan Ford.
Unmanaged internet protocol: Taming the edge network management crisis.
In Second Workshop on Hot Topics in Networks, Cambridge, MA, November 2003.
7
Paul Francis and Ramakrishna Gummadi.
IPNL: A NAT-extended Internet architecture.
In ACM SIGCOMM, August 2002.
8
Saikat Guha and Paul Francis.
Simple traversal of UDP through NATs and TCP too (STUNT).
http://nutss.gforge.cis.cornell.edu/.
9
Saikat Guha, Yutaka Takeday, and Paul Francis.
NUTSS: A SIP-based approach to UDP and TCP network connectivity.
In SIGCOMM 2004 Workshops, August 2004.
10
M. Holdrege and P. Srisuresh.
Protocol complications with the IP network address translator, January 2001.
RFC 3027.
11
C. Huitema.
Teredo: Tunneling IPv6 over UDP through NATs, March 2004.
Internet-Draft (Work in Progress).
12
C. Jennings.
NAT classification results using STUN, October 2004.
Internet-Draft (Work in Progress).
13
Dan Kegel.
NAT and peer-to-peer networking, July 1999.
http://www.alumni.caltech.edu/~dank/peer-nat.html.
14
M. Leech et al.
SOCKS protocol, March 1996.
RFC 1928.
15
R. Moskowitz and P. Nikander.
Host identity protocol architecture, April 2003.
Internet-Draft (Work in Progress).
16
NAT check.
http://midcom-p2p.sourceforge.net/.
17
J. Rosenberg.
Interactive connectivity establishment (ICE), October 2003.
Internet-Draft (Work in Progress).
18
J. Rosenberg, C. Huitema, and R. Mahy.
Traversal using relay NAT (TURN), October 2003.
Internet-Draft (Work in Progress).
19
J. Rosenberg, J. Weinberger, C. Huitema, and R. Mahy.
STUN - simple traversal of user datagram protocol (UDP) through network address translators (NATs), March 2003.
RFC 3489.
20
J. Saltzer.
On the naming and binding of network destinations.
In P. Ravasio et al., editor, Local Computer Networks, pages 311-317. North-Holland, Amsterdam, 1982.
RFC 1498.
21
P. Srisuresh and M. Holdrege.
IP network address translator (NAT) terminology and considerations, August 1999.
RFC 2663.
22
P. Srisuresh, J. Kuthan, J. Rosenberg, A. Molitor, and A. Rayhan.
Middlebox communication architecture and framework, August 2002.
RFC 3303.
23
Transmission control protocol, September 1981.
RFC 793.
24
G. Tsirtsis and P. Srisuresh.
Network address translation - protocol translation (NAT-PT), February 2000.
RFC 2766.
25
Justin Uberti.
E-mail on IETF MIDCOM mailing list, February 2004.
Message-ID: <402CEB11.1060906@aol.com>.
26
UPnP Forum.
Internet gateway device (IGD) standardized device control protocol, November 2001.
http://www.upnp.org/.
27
Michael Walfish, Jeremy Stribling, Maxwell Krohn, Hari Balakrishnan, Robert Morris, and Scott Shenker.
Middleboxes no longer considered harmful.
In USENIX Symposium on Operating Systems Design and Implementation, San Francisco, CA, December 2004.

Bryan Ford 2005-02-17

Launch HN: VergeSense (YC S17) – AI-Powered Sensors for Building Management

$
0
0
Hello HN! This is Dan and Kelby (tripleplay369), the founders of VergeSense (http://www.vergesense.com). We're building an AI-powered facility management platform that helps companies use their buildings more efficiently. The cost of real estate is typically the #2 cost center for any company (after people), but most companies don't have a good way of measuring how their building is being used. Our product solves this by identifying wasted areas and recommending more productive uses for that space (e.g. turning unused offices into conference rooms or employee lounge areas).

The core of our offering is a discrete sensor that leverages multiple inputs (primarily an imaging sensor + PIR-based motion sensing), which feed into a neural network model that executes inference directly on the device. This allows us to do powerful processing on inexpensive hardware.

Our machine-learning stack is built around Tensorflow, which we use in two ways: 1) for inference (we embed Tensorflow directly on a Raspberry Pi), and 2) training new models in the cloud. New models can be pushed remotely to the devices over-the-air to make the sensors “smarter”.

While our sensors are currently trained to count people, our vision is to evolve into a 100% passive "super-sensor" that can be configured to detect thousands of different types of events. Examples that we've explored include things like detecting falls (e.g. during an emergency), counting assets (equipment, furniture, cars), and monitoring equipment usage (for preventative maintenance).

We're happy to chat and would love to hear your thoughts. Some things we've worked on that might be interesting to discuss: rapid-prototyping for hardware (Raspberry Pis +ESP8266), machine-learning, computer-vision, building automation, BLE, B2B sales, keeping sane while drawing bounding boxes, or anything else that comes to mind!

We look forward to your feedback!

Dan + Kelby

Hacker claims to have decrypted Apple's Secure Enclave

$
0
0

A hacker going by the handle xerub has just released what he claims to be a full decryption key for Apple's Secure Enclave Processor (SEP) firmware.

This could be a major blow for iOS security because of the importance of the SEP: It handles Touch ID transactions and is completely isolated from the rest of its host device. Your iPhone, iPad, or iPod has no idea what's going on in the SEP, and that means no one else does either—at least until today.

Now that its firmware code is exposed it's open season on SEP vulnerabilities.

What the Secure Enclave Processor is

Photographer: Denys Prykhodov, Getty Images

Ever since Touch ID came out with the iPhone 5S, there has been a tiny coprocessor embedded in the main S-series, and now A-series, processor chip. That tiny coprocessor runs completely on its own—it has a separate OS, updates separately, and nothing it does is known to the rest of the device.

One of the key points of the SEP is its generation of the device's Unique ID (UID). That UID is further secured by tangling it up with an ephemeral key that changes every time the device is rebooted.

SEE: Special report: Cybersecurity in an IoT and mobile world (free PDF) (TechRepublic)

Protecting the UID is why the SEP exists, and why all Touch ID actions, password verification, and other security processes happen inside it. .

Why the SEP's decryption is a big deal

The SEP's firmware code is now open to the world, thanks to xerub's efforts. The key is published here, this GitHub repository contains what you need to decrypt it, and this one has the tools to process it.

"The fact that [the SEP] was hidden behind a key worries me," said xerub. "Is Apple not confident enough to push SEP decrypted as they did with kernels past iOS 10?" He added that while SEP is amazing tech the fact that it's a "black box" adds very little, if anything to security. "Obscurity helps security—I'm not denying that," he said, but added that relying on it for security isn't a good idea.

Expert hackers, he added, won't be stopped by black boxes. Just slowed down.

"I think public scrutiny will add to the security of SEP in the long run," xerub said, noting that was also his intention with releasing the key. It's another act in the arms race between tech companies and hackers, who poke and prod software in a way that ultimately can make users safer.

"Apple's job is to make [SEP] as secure as possible," xerub said. "It's a continuous process ... there's no actual point at which you can say 'right now it's 100% secure.'"

Decrypting the SEP's firmware is huge for both security analysts and hackers. It could be possible, though xerub says it's very hard, to watch the SEP do its work and reverse engineer its process, gain access to passwords and fingerprint data, and go even further toward rendering any security relying on the SEP completely ineffective.

"Decrypting the firmware itself does not equate to decrypting user data," xerub said. There's a lot of additional work that would need to go into exploiting decrypted firmware—in short it's probably not going to have a massive impact.

SEE: Every iOS user should update to 10.3.3 now to avoid this Wi-Fi hack (TechRepublic)

There's no telling when any potential effects of the SEP's decryption could start being felt, or in what way. Ideally, Apple will release a fix as soon as possible, but failing that be on the lookout for Touch ID hacks, password harvesting scams, or other attacks that could take advantage of the decryption.

We reached out to Apple and xerub for comments. This article will be updated with any response.

Update 8/17: Added information from a conversation with xerub.

Top three takeaways for TechRepublic readers:

  1. A hacker has released what they claim to be a decryption key for the Apple Secure Enclave Processor (SEP) firmware. The SEP handles password and Touch ID encryption, and decrypting it could have serious security consequences.
  2. It's still too early to know what the full fallout from the SEP's decryption will be, but it could open the door for password harvesting, spoofing, and other security-compromising attacks.
  3. Decryption of firmware doesn't equate to decryption of personal data. While SEP's firmware may have been opened up your personal data isn't necessarily at risk.

Also see:

China’s embrace of embryo selection raises thorny questions

$
0
0

Victor Fraile/Corbis/Getty

China’s move to a two-child policy has been a boon for fertility clinics.

Getting time with Qiao Jie is not easy. At 7:30 a.m., the line coming out of the fertility centre that she runs blocks the doorway and extends some 80 metres down the street. Inside, about 50 physicians on her team are discussing recent findings, but Qiao, a fertility specialist and president of Peking University Third Hospital in Beijing, is still in an early-morning consult.

When she finally emerges, she jumps to the topic at hand: spreading awareness of preimplantation genetic diagnosis (PGD), a procedure that helps couples undergoing in vitro fertilization (IVF) to avoid passing on genetic mutations that could cause disease or disability in their children. Qiao typically refuses interview requests, but she’s concerned that people aren’t getting the message about PGD fast enough. “Now, more and more diseases can be stopped — if not immediately, in the generation after next,” she says. 

Early experiments are beginning to show how genome-editing technologies such as CRISPR might one day fix disease-causing mutations before embryos are implanted. But refining the techniques and getting regulatory approval will take years. PGD has already helped thousands of couples. And whereas the expansion of PGD around the world has generally been slow, in China, it is starting to explode.

The conditions there are ripe: genetic diseases carry heavy stigma, people with disabilities get very little support and religious and ethical push-back against PGD is almost non-existent. China has also lifted some restrictions on family size and seen a subsequent rise in fertility treatments among older couples. Genetic screening during pregnancy for chromosomal abnormalities linked to maternal age has taken off throughout the country, and many see this as a precursor to wider adoption of PGD.

Although Chinese fertility doctors were late to the game in adopting the procedure, they have been pursuing a more aggressive, comprehensive and systematic path towards its use there than anywhere else. The country’s central government, known for its long-term thinking, has over the past decade stepped up efforts to bring high-quality health care to the people, and its current 5-year plan has made reproductive medicine, including PGD, a priority, an effort that Qiao is leading. Researchers are hunting down various mutations in the Chinese population that might be screened for in PGD. And well-equipped and powerful clinical-research groups, including Qiao’s, are stepping up efforts to improve the technology, increase awareness and bring down costs. 

LISTEN

Why is preimplantation genetic diagnosis is taking off in China?

You may need a more recent browser or to install the latest version of the Adobe Flash Plugin.

Comprehensive figures are difficult to come by, but estimates from leading PGD providers show that China’s use of the technique already outpaces that in the United States, and it is growing up to five times faster. Qiao’s clinic alone now performs more procedures with PGD each year than all of the United Kingdom. 

“Looking over the development in China over the past 10 years, they might start to think it’s possible to get rid of these diseases,” says Kangpu Xu, a Chinese-born reproductive biologist at Weill Cornell Medical College in New York City.

Such systematic efforts raise thorny questions for bioethicists. Some worry that pushes to eliminate disabilities devalue the lives of those who already have them. The cost and accessibility of the procedure raises concerns about genetic traits further widening the divide between rich and poor people. Then there are concerns about the push to select for non-disease-related traits, such as intelligence or athletic ability. The ever-present spectre of eugenics lurks in the shadows. But in China, although these concerns are considered, most thoughts are focused on the benefits of the procedures. “There are ethical problems, but if you bring an end to the disease, I think it’s good for society,” says Qiao. 

Heyday for PGD

Physicians in the United Kingdom pioneered PGD in humans about 30 years ago, initially to help genetic carriers of a disorder that affects mainly boys. Thanks to the procedure, the parents were able to select for girls. Generally, the process involves removing one or a few cells from an embryo created during IVF and then using various techniques to test the structure and number of chromosomes and even the sequence of individual genes. Physicians typically discard embryos that don’t pass the tests.

Uncertain about the procedure’s safety, and wary of its potential for abuse (selecting for males in China is illegal, for example), the Chinese government restricted the practice to hospitals with a licence. By the end of 2004, only four centres in the entire country had such a licence. By 2016, the number had risen to 40. 

The clinics are huge and growing. Qiao’s centre carried out 18,000 IVF procedures in 2016. The biggest clinic, the Reproductive and Genetic Hospital CITIC-Xiangya in Changsha, recorded 41,000 IVF procedures in the same year. That’s roughly one-quarter of the annual number for the entire United States. One reason for the dramatic rise is China’s policy change last year that now allows families to have two children. This has led to a huge number of older women seeking fertility treatment. Another factor is the changing culture in China. Ten years ago, people who couldn’t conceive would take traditional Chinese medicine, or they might adopt a child. “Now they know assisted reproductive technologies can help,” says Qiao. 

And the centres with licences to do PGD have created a buzz in their race to claim firsts with the technology. In 2015, CITIC-Xiangya boasted China’s first “cancer-free baby”. The boy’s parents had terminated a prior pregnancy after genetic testing showed the presence of retinoblastoma, a cancer that forms in the eyes during early development and often leads to blindness. In their next try, the couple used PGD to ensure that the gene variant that causes retinoblastoma wasn’t present. Other groups have helped couples to avoid passing on a slew of conditions: short-rib-polydactyly syndrome, Brittle-bone disease, Huntington’s disease, polycystic kidney disease and deafness, among others. Qiao, working with biochemist Sunney Xie at Harvard University in Cambridge, Massachusetts, has also introduced a method that can do both chromosomal analyses and next-generation genetic analyses on a single cell. China might have got a slow start, but it is now overtaking Western nations in its use of PGD.

Andy Wong/AP/REX/Shutterstock

Chinese clinics are pioneering new methods for embryo testing.

Qiao’s clinic screened embryos for individual disease-causing genes about 100 times last year. It screened for abnormal chromosome counts, such as that associated with Down’s syndrome, in another 670 cases. For comparison, 578 such procedures were done in the entire United Kingdom in 2014, the latest year for which numbers are available. And China’s uptake is growing fast. At CITIC-Xiangya, the number of preimplantation testing procedures rose by 277% over just 2 years, from 876 in 2014 to 2,429 in 2016, and 700 of these were for single-gene disorders.  

What’s more, many fertility centres in China have the capacity for high-quality research. Qiao is interested in safety and is studying whether extracting the cells for PGD causes subtle damage to the embryo. She is in the middle of compiling data from all IVF clinics in China for a 10-year study on such effects. 

Qiao is also working with Xie and Sijia Lu, the chief technology officer of Shanghai-based Yikon Genomics, to develop a technique to do all the necessary sequencing without removing cells, by sampling free-floating DNA in the media the embryos are cultured in. Such an advance could make PGD safer and easier to do.

Joe Leigh Simpson, a medical geneticist at Florida International University in Miami, and former president of the Preimplantation Genetic Diagnosis International Society, is impressed by the quality and size of the Chinese fertility clinics. They “are superb and have gigantic units. They came out of nowhere in just 2 or 3 years,” he says. 

Chinese researchers are also looking for more disease-associated gene variants, specifically to expand the impact of PGD. The most concentrated efforts are being orchestrated by He Lin, a geneticist at Shanghai Jiao Tong University. He has set out an ambitious project: to pin down all the mutations in all the genes that cause diseases and put them into a single database. “We just do them one by one until we get the whole set,” he says, referring to the roughly 6,000 known genetic diseases. As disease–gene links are verified, they could be added to the list of things that PGD can screen for. 

The first target, He says, is deafness. Wang Qiuju, a hearing-loss specialist at the Chinese PLA General Hospital in Beijing and head of the project, says that she plans to get up to 200,000 samples from 150 hospitals throughout China to identify associated mutations. 

The large numbers are needed because there are a handful of genes involved in hearing loss, and each of them have dozens, even hundreds, of mutations. “When we have big databases, we can see the contribution of each gene more clearly. Then it’s easy to do PGD,” says Wang. 

Culture clash

Such efforts, for hearing loss in particular, can seem jarring because many people in the West do not consider it a problem to be avoided. In the United States, some deaf couples have used PGD to select for congenital deafness, in an effort to preserve Deaf culture. Such sentiments wouldn’t make sense to many parents in China, says Wang, because there is little support for them: “If they have a deaf child, they feel the need to have a normal child to help them take care of the deaf child.”

People in China seem more likely to feel an obligation to bear the healthiest child possible than to protect an embryo. The Chinese appetite for using genetic technology to ensure healthy births can be seen in the rapid rise of pregnancy testing for Down’s syndrome and other chromosomal abnormalities. Since Shenzhen-based BGI introduced a test for Down’s syndrome in 2013, it has sold more than 2 million kits; half of those sales were in the past year.

Although such testing has become routine in the United Kingdom and United States, many in the West won’t terminate a pregnancy just because of Down’s syndrome. 

Jiani Chen, a genetic counsellor at the University of Oklahoma Health Sciences Center in Oklahoma City, says that this isn’t the case in China. “In China, if you want to abort a baby with Down’s syndrome, no one will scold you.” Since moving from her native Taiwan to Oklahoma, Chen herself says that she is no longer sure what she would do.

“There are ethical problems, but if you bring an end to the disease, I think it’s good for society.”

In the West, PGD still raises fears about the creation of an elite genetic class, and critics talk of a slippery slope towards eugenics, a word that elicits thoughts of Nazi Germany and racial cleansing. In China, however, PGD lacks such baggage. The Chinese word for eugenics, yousheng, is used explicitly as a positive in almost all conversations about PGD. Yousheng is about giving birth to children of better quality. Not smoking during pregnancy is also part of yousheng

This is not to say that the Chinese haven’t thought about abuses of the technology. The Chinese government was worried, as were many Western governments, that PGD would be used to select physical characteristics, such as height or intelligence. The clinics licensed to do PGD can use it only to avoid serious disease or assist infertility treatments. And sex selection through PGD is off the table. Yikon’s Lu says that some families ask to weed out the mutation that renders many Asians unable to process alcohol, something that could affect the ability to take part in the often alcohol-fuelled Chinese business lunches. “They want their son to be able to drink,” says Lu. “We say no.” Shanghai Jiao Tong University’s He has made training genetic counsellors — people versed in the risks, benefits and ethical issues related to PGD — a priority. Currently, they are almost non-existent in China.

The UK Human Fertilisation & Embryology Authority also tightly regulates PGD — limiting its use to 400 conditions. But in the United States, clinics have fairly free rein. Sex selection, for example, is acknowledged as controversial by the American Society for Reproductive Medicine, but its ethics committee largely leaves it to individual clinics to decide what is permissible. 

To many fertility specialists, what’s most striking about China’s adoption of PGD is the speed and organization of its uptake. China already seems to provide more procedures than the United States, and with growth estimated at 60–70% per year, is on target to catch up in per capita terms in the next few years.

This could be a boon for the country, given the economic arguments for PGD. For instance, one study has compared the average costs of the PGD procedure needed to avoid cystic fibrosis — US$57,500 — with the medical costs incurred in a lifetime by an average patient, which amounted to $2.3 million (I. Tur-Kaspa et al. Reprod. Biomed. Online 21, 186–195; 2010). The authors calculated net savings on health care of all patients born in a year over the average patient’s lifespan of 37 years to be $33.3 billion. That is just for one of hundreds of diseases that can be avoided with PGD. 

But PGD has not been an easy sell in the West. The Catholic Church, for example, opposes embryo manipulation, including the removal of cells for testing, as well as the destruction of embryos. “The idea that scientists are playing god is always a theme,” says Natasha Bonhomme, chief strategy officer at Genetic Alliance, a lobbying group in Washington DC that focuses on genetic diseases. 

There are also social and economic concerns. Some parents of affected children argue that reducing the number of children with those diseases would reduce government funding for research into treatments. Others object to the idea that they are being discouraged from conceiving children the usual way.

The debate has made physicians and scientists wary. “The scientific community is not interested in getting too forward out in front of public opinion,” says Simpson, even though he thinks that the evidence is on the side of employing more PGD. “With every reproductive-biology advance,” he says, “we get the same questions: ‘won’t there be a slippery slope that leads to abuse?’ But it never happens.” 

The upshot is that there has never really been advocacy organized around PGD in the United States, says Bonhomme. And without government support, it remains for many a prohibitively expensive procedure. Insurance coverage is “pitiful”, says Svetlana Rechitsky, director of the genetic-testing firm Reproductive Genetic Innovations in Northbrook, Illinois. Sitting at her desk, sorting through letters from insurers — mostly refusals to offer coverage for PGD — she says, “It’s getting worse and worse.” 

Already the procedure is much cheaper in China — about one-third of what it costs in the United States. Cheaper tests will make it more palatable for national insurance coverage, something Qiao has already started pushing for. “Before I retire, I want to get the government involved. I have 12 years,” she says.


Weebly (YC W07) Is Hiring Senior Infrastructure Engineers

$
0
0
Weebly is looking for Senior Infrastructure Engineers to join our Infrastructure Engineering team. We’re building orchestration components that will run Weebly. The ideal candidate will have a strong software and distributed systems development background. We are looking for candidates who have a passion for tackling complex problems in a performant and scalable way.

Qualifications include: Proficiency in multiple languages, but an expert in at least one object oriented and one procedural language. Experience with HTTP/RPC/REST API development Experience with Relational databases, such as MySQL or Postgres Experience with caching mechanisms and unstructured data stores Experience with Message Queues Strong knowledge of Unix/Linux systems

Ideally, you'll: Have deployed and scaled high availability distributed services Be comfortable helping lead the direction of our core infrastructure architecture Have previous experience with Borg, Omega, Mesos or Kubernetes Golang experience

Your responsibilities will include: Work with geographically distributed backend systems Design and build core backend software components and services Instrumentation for monitoring the health and availability of services Performance management including benchmarking and monitoring of vital metrics Analyze and improve scalability and efficiency of system resources

Perks: Competitive salary Stock options Employer-paid health care Employer match of retirement contribution up to the maximum allowed by law Company credit card Multiple large monitors, Aeron chair Electronically operated sit to stand desks Free food, drinks & snacks at the office (including breakfast, lunch, and dinner prepared by our professional chef) In-office gym, weekly yoga classes and massages Fun, get-things-done work environment -- work whenever you want, take days off whenever you want, as long as you're getting things done

Company:

Weebly is a complete platform that allows anyone to start and grow an online business with curated website templates, powerful ecommerce and integrated marketing. More than 40 million entrepreneurs around the world use Weebly to grow their customer base, fuel sales and market their idea. Designed for any entrepreneur who wants to reach a global audience, Weebly gives everyone the freedom to create a high quality online presence that works brilliantly across any device. Over 325 million people a month visit a site or store made on Weebly.

The office culture at Weebly encourages new ideas and teamwork above all else. Our team is collaborative and sociable.

http://careers.weebly.com/#111159

Peanut allergy cured in majority of children in immunotherapy trial

$
0
0

Australian researchers have made a breakthrough in the treatment of peanut allergy in children.

A small clinical trial conducted at the Murdoch Children’s Research Institute has led to two-thirds of children treated with an experimental immunotherapy treatment being cured of their allergy. Importantly, this desensitisation to peanuts persisted for up to four years after treatment.

“These children had been eating peanut freely in their diet without having to follow any particular program of peanut intake in the years after treatment was completed,” said the lead researcher, Prof Mimi Tang.

Peanut allergy is the most common cause of anaphylaxis, a life-threatening allergic reaction, and one of the most common causes of death from food allergy.

To combat this Tang, an immunologist and allergist, pioneered a new form of treatment that combines a probiotic with peanut oral immunotherapy, known as PPOIT. Instead of avoiding the allergen, the treatment is designed to reprogram the immune system’s response to peanuts and eventually develop a tolerance.

It’s thought that combining the probiotic with the immunotherapy gives the immune system the “nudge” it needs to do this, according to Tang.

Forty-eight children were enrolled in the PPOIT trial and were randomly given either a combination of the probiotic Lactobacillus rhamnosus with peanut protein in increasing amounts, or a placebo, once daily for 18 months.

At the end of the original trial in 2013, 82% of children who received the immunotherapy treatment were deemed tolerant to peanuts compared with just 4% in the placebo group.

Four years later, the majority of the children who gained initial tolerance were still eating peanuts as part of their normal diet and 70% passed a further challenge test to confirm long-term tolerance.

Tang said the results were exciting and had been life-changing for participants. “The way I see it is that we had children who came into the study allergic to peanuts, having to avoid peanuts in their diet, being very vigilant around that, carrying a lot of anxiety with that and, at the end of treatment and even four years later, many of these children who had benefited from our probiotic peanut therapy could now live like a child who didn’t have peanut allergy.”

The results are published in the Lancet Child & Adolescent Health.

If confirmed by larger clinical studies, the broader hope is that this treatment can have an impact on the high rates of food allergy among children.

“This is a major step forward in identifying an effective treatment to address the food allergy problem in western societies,” Tang said.

Dilution of whisky – the molecular perspective

$
0
0
  • 1.

    Gill, V. A whisky tour-is this the most chemically complicated drink in the world? Victoria Gill attempts to unlock some of the mysteries of scotch malt whisky. Chem. World UK5, 40–44 (2008).

  • 2.

    Kew, W., Goodall, I., Clarke, D. & Uhrín, D. Chemical diversity and complexity of scotch whisky as revealed by high-resolution mass spectrometry. J. Am. Soc. Mass Spectrom.28, 200–213 (2017).

  • 3.

    Jounela-Eriksson, P. & Lehtonen, M. In The Quality of Foods and Beverages V1: Chemistry and Technology (Elsevier Science, 1980).

  • 4.

    Lehtonen, M. Phenols in whisky. Chromatographia16, 201–203 (1982).

  • 5.

    Nie, Y. & Kleine-Benne, E. Determining phenolic compounds in whisky using direct large volume injection and stir bar sorptive extraction. Gerstel Glob. Anal. Solut. AppNote 02/2012.

  • 6.

    Guo, J.-H. et al. Molecular structure of alcohol-water mixtures. Phys. Rev. Lett.91, 157401 (2003).

  • 7.

    Dixit, S., Crain, J., Poon, W. C. K., Finney, J. L. & Soper, A. K. Molecular segregation observed in a concentrated alcohol–water solution. Nature416, 829–832 (2002).

  • 8.

    Lam, R. K., Smith, J. W. & Saykally, R. J. Communication: hydrogen bonding interactions in water-alcohol mixtures from x-ray absorption spectroscopy. J. Chem. Phys.144, 191103 (2016).

  • 9.

    Takamuku, T., Saisho, K., Nozawa, S. & Yamaguchi, T. X-ray diffraction studies on methanol–water, ethanol–water, and 2-propanol–water mixtures at low temperatures. J. Mol. Liq.119, 133–146 (2005).

  • 10.

    Asenbaum, A. et al. Structural changes in ethanol–water mixtures: Ultrasonics, Brillouin scattering and molecular dynamics studies. Vib. Spectros.60, 102–106 (2012).

  • 11.

    Guo, J.-H. et al. The molecular structure of alcohol-water mixtures determined by soft-x-ray absorption and emission spectroscopy. J. Electron Spec. Relat. Phenom.137–140, 425–428 (2004).

  • 12.

    Nagasaka, M., Mochizuki, K., Leloup, V. & Kosugi, N. Local structures of methanol-water binary solutions studied by soft X-ray absorption spectroscopy. J. Phys. Chem. B118, 4388–4396 (2014).

  • 13.

    Bako, I., Megyes, T., Balint, S., Grosz, T. & Chihaia, V. Water-methanol mixtures: topology of hydrogen bonded network. Phys. Chem. Chem. Phys.10, 5004–5011 (2008).

  • 14.

    Wensink, E. J. W., Hoffmann, A. C., van Maaren, P. J. & van der Spoel, D. Dynamic properties of water/alcohol mixtures studied by computer simulation. J. Chem. Phys.119, 7308–7317 (2003).

  • 15.

    Rankin, B. M., Ben-Amotz, D., van der Post, S. T. & Bakker, H. J. Contacts between alcohols in water are random rather than hydrophobic. J. Phys. Chem. Lett.6, 688–692 (2015).

  • 16.

    Tarek, M., Tobias, D. J. & Klein, M. L. Molecular dynamics investigation of the surface/bulk equilibrium in an ethanol-water solution. J. Chem. Soc., Faraday Trans.92, 559–563 (1996).

  • 17.

    Stewart, E., Shields, R. L. & Taylor, R. S. Molecular dynamics simulations of the liquid/vapor interface of aqueous ethanol solutions as a function of concentration. J. Phys. Chem. B107, 2333–2343 (2003).

  • 18.

    Wilson, M. A. & Pohorille, A. Adsorption and solvation of ethanol at the water liquid−vapor interface:  a molecular dynamics study. J. Phys. Chem. B101, 3130–3135 (1997).

  • 19.

    Hub, J. S., Caleman, C. & van der Spoel, D. Organic molecules on the surface of water droplets - an energetic perspective. Phys. Chem. Chem. Phys.14, 9537–9545 (2012).

  • 20.

    Ben-Amotz, D. Interfacial solvation thermodynamics. J. Phys.-Condens. Matter28, 414013 (2016).

  • 21.

    Habartová, A., Obisesan, A., Minofar, B. & Roeselová, M. Partial hydration of n-alkyl halides at the water–vapor interface: a molecular simulation study with atmospheric implications. Theor. Chem. Acc.133, 1455 (2014).

  • 22.

    Hanwell, M. D. et al. Avogadro: an advanced semantic chemical editor, visualization, and analysis platform. J. Cheminform.4, 17 (2012).

  • 23.

    Halgren, T. A. Merck molecular force field. I. Basis, form, scope, parameterization, and performance of MMFF94. J. Comput. Chem.17, 490–519 (1996).

  • 24.

    Frisch, M. J. et al. Gaussian09. (Gaussian Inc., 2009).

  • 25.

    Ponder, J. W. & Case, D. A. Force fields for protein simulations. Adv. Protein Chem.66, 27–85 (2003).

  • 26.

    Wang, J., Cieplak, P. & Kollman, P. A. How well does a restrained electrostatic potential (RESP) model perform in calculating conformational energies of organic and biological molecules? J. Comput. Chem.21, 1049–1074 (2000).

  • 27.

    Martínez, L., Andrade, R., Birgin, E. G. & Martínez, J. M. PACKMOL: a package for building initial configurations for molecular dynamics simulations. J. Comput. Chem.30, 2157–2164 (2009).

  • 28.

    Case, D. A. et al. The Amber biomolecular simulation programs. J. Comput. Chem.26, 1668–1688 (2005).

  • 29.

    Case, D. A. et al. AMBER 10. (University of California, San Francisco, USA).

  • 30.

    Wang, J., Wolf, R. M., Caldwell, J. W., Kollman, P. A. & Case, D. A. Development and testing of a general amber force field. J. Comput. Chem.25, 1157–1174 (2004).

  • 31.

    Berendsen, H. J. C., Grigera, J. R. & Straatsma, T. P. The missing term in effective pair potentials. J. Phys. Chem.91, 6269–6271 (1987).

  • 32.

    Mark, P. & Nilsson, L. Structure and dynamics of the TIP3P, SPC, and SPC/E water models at 298 K. J. Phys. Chem. A105, 9954–9960 (2001).

  • 33.

    Taylor, R. S., Dang, L. X. & Garrett, B. C. Molecular dynamics simulations of the liquid/vapor interface of SPC/E water. J. Phys. Chem.100, 11720–11725 (1996).

  • 34.

    Andoh, Y. & Yasuoka, K. Two-dimensional supercritical behavior of an ethanol monolayer:  a molecular dynamics study. Langmuir21, 10885–10894 (2005).

  • 35.

    Chickos, J. S. & Acree, W. E. Enthalpies of vaporization of organic and organometallic compounds, 1880–2002. J. Phys. Chem. Ref. Data32, 519–878 (2003).

  • 36.

    Budavari, S., O’Neil, M., Smith, A., Heckelman, P. & Obenchain, J. The Merck Index, Print Version, Twelfth Edition. (CRC Press, 1996).

  • 37.

    Grolier, J. & Wilhelm, E. Excess volumes and excess heat-capacities of water + ethanol at 298.15-K. Fluid Phase Equilib.6, 283–287 (1981).

  • 38.

    Gereben, O. & Pusztai, L. On the accurate calculation of the dielectric constant from molecular dynamics simulations: The case of SPC/E and SWM4-DP water. Chem. Phys. Lett.507, 80–83 (2011).

  • 39.

    Neumann, M. Dipole moment fluctuation formulas in computer simulations of polar Systems. Mol. Phys.50, 841–858 (1983).

  • 40.

    Petong, P., Pottel, R. & Kaatze, U. Water−ethanol mixtures at different compositions and temperatures. A dieletric relaxation study. J. Phys. Chem. A104, 7420–7428 (2000).

  • 41.

    Ryckaert, J.-P., Ciccotti, G. & Berendsen, H. J. C. Numerical integration of the cartesian equations of motion of a system with constraints: molecular dynamics of n-alkanes. J. Comput. Chem.23, 327–341 (1977).

  • 42.

    Giorgino, T. Computing 1-D atomic densities in macromolecular simulations: the Density Profile Tool for VMD. Comp. Phys. Commun.185, 317–322 (2014).

  • 43.

    Humphrey, W., Dalke, A. & Schulten, K. VMD: visual molecular dynamics. J. Mol. Graph. 14, 33–8, 27–8 (1996).

  • 44.

    Gil, P. S. & Lacks, D. J. Effect of surfactant shape on solvophobicity and surface activity in alcohol-water systems. J. Chem. Phys.145, 204705 (2016).

  • 45.

    Carteau, D., Bassani, D. & Pianet, I. The ‘Ouzo effect’: following the spontaneous emulsification of trans-anethole in water by NMR. Comptes Rendus Chim.11, 493–498 (2008).

  • WebRender newsletter #1

    $
    0
    0

    The Quantum Flow and Photon projects have exciting newsletters. The Quantum graphics project (integrating WebRender in Firefox) hasn’t provided a newsletter so far and people have asked for it, so let’s give it a try!

    This newsletter will not capture everything that is happening in the project, only some highlights, and some of the terminology might be a bit hard to understand at first for someone not familiar with the internals of Gecko and WebRender. I will try to find the time to write a bit about WebRender’s internals and it will hopefully provide more keys to understanding what’s going on here.

    The terms layer-full/layers-free used below refer to the way WebRender is integrated in Gecko. Our first plan was to talk to WebRender using the layers infrastructure in the short term, because it is the simplest approach. This is the “layers-full” integration. Unfortunately the cost of building many layers to transform into WebRender display items is high and we found out that we may not be able to ship WebRender using this strategy. The “layers-free” integration plan is to translate Gecko’s display items into WebRender display items directly without building layers. It is more work but we are getting some encouraging results so far.

    Some notable (recent) changes in WebRender

    • Glyph Cache optimizations – Glenn profiled and optimized the glyph cache and made it a lot faster.
    • Texture cache rewrite (issue #1572) – The new cache use pixel buffer objects to transfer images to the GPU (previously used glTexSubImage2D), and does not suffer from fragmentation issues the way the previous one did, and has a better eviction policy.
    • Other text related optimization in the display list serialization.
    • Sub-pixel positioning on Linux.

    Some notable (recent) changes in Gecko

    • Clipping in layers free mode (Bug 1386483) – This reuses clips instead of having new ones for every display item. This will reduce the display list processing that happens on the Gecko side as well as the WebRender side. This was one of the big things missing from getting functional parity with current layers-full WebRender.
    • Rounded rectangle clipping in layers free mode (Bug 1370682) – This is a noticeable difference from what we do in layer-full mode. In layer-full mode we currently use mask layers for rounded clipping. Doing this directly with WebRender gives a noticeable performance improvement.

    How to get the most exciting WebRender experience today:

    Go to about:config and change the following prefs:

    • turn off layers.async-pan-zoom.enabled
    • turn on gfx.webrender.enabled
    • turn on gfx.webrendest.enabled
    • turn on gfx.webrender.layers-free
    • add and turn on gfx.webrender.blob-images
    • if you are on Linux, turn on layers.acceleration.force-enabled

    This will give you a peek at the future but beware there are lots of rough edges. Don’t expect the performance of WebRender in Gecko to be representative yet (Probably better to try Servo for that).

    All of the integration work is now happening in mozilla-central and bugzilla, WebRender development happens on the servo/webrender github repository.

    A big, successful trial of probiotics

    $
    0
    0

    For all the hype that surrounds them, probiotics—products that contain supposedly beneficial bacteria—have rarely proven their worth in large, rigorous studies. There are good reasons for this disappointing performance. The strains in most commercially produced probiotics were chosen for historical reasons, because they were easy to grow and manufacture, and not because they are well-adapted to the human body. When they enter our gut, they fail to colonize. As I wrote in my recent book, they’re like a breeze that blows between two open windows.

    But even though probiotic products might be underwhelming, the probiotic concept is sound. Bacteria can beneficially tune our immune systems and protect us from disease. It’s just a matter of finding the right strains, and helping them to establish themselves. Many scientists are now trying to do just that, and one such team, led by Pinaki Panigrahi at the University of Nebraska Medical Center, has just scored a big win.

    Since 2008, Panigrahi’s team has been running a large clinical trial in rural India, where they gave a probiotic of their own devising to thousands of randomly selected newborn babies. Their product contained a strain of Lactobacillus plantarum, chosen for its ability to attach to gut cells. The team also added a sugar, chosen to nourish the microbe and give it a foothold when it enters a baby’s gut. Together, this combination is called a synbiotic. And it was strikingly effective.

    The team found that babies who took this concoction had a significantly lower risk of developing sepsis—a life-threatening condition where infections trigger body-wide inflammation, restricted blood flow, and organ failure. Sepsis is one of the biggest killers of newborn babies, ending around 600,000 lives every year when they’ve barely begun. Some proportion of these cases begin in the gut, and probiotics might be able to prevent them by ousting harmful microbes, or by stopping benign ones from crossing into the bloodstream and causing infections.

    Sure enough, in Panigrahi’s trial, just 5.4 percent of the infants who took the synbiotic developed sepsis in their first two months of life, compared to 9 percent of those who received a placebo. That’s a reduction of 40 percent. Such estimates always come with a margin of error, but the team calculate that the reduction in risk should still be somewhere between 25 and 50 percent.

    The effect was twice as large as what the team expected, especially since the infants took daily doses of the synbiotic for just one week. And given the clear evidence of benefits, independent experts who were monitoring the study decided to stop the trial early: It would have been unethical to continue depriving half the newborns of the treatment. Panigrahi originally planned to enroll 8,000 babies into the study. He stopped at 4,557.

    Which is still a huge number! Probiotics trials have been criticized in the past for being small and statistically underpowered. Those that looked at sepsis, for example, usually involved just 100 to 200 babies, making it hard to know whether any beneficial effects were the result of random chance. The biggest trial to date included 1,315 infants; Panigrahi’s study is over three times bigger. “[It] exemplifies how intervention research should be done,” writes Daniel Tancredi from the University of California, Davis, in a commentary that accompanies the paper.

    “In most studies, people take the probiotics that are available on the shelf without asking why that probiotic should work in the disease they’re interested in. And they think they’ll stumble onto something good,” says Panigrahi. “It’s counter-intuitive, but we did the same thing.”

    At first, his team tested Lactobacillus GG and Lactobacillus sporogenes—the most commonly used probiotics in India—in small pilot studies. Both strains are claimed to colonize the gut. “We did the trial and the colonization was almost zero,” says Panigrahi. To find more suitable strains, the team collected stool from healthy volunteers and screened the microbes within for those that could stick to human cells, and could prevent disease-causing bacteria from doing so. They ended up with a strain called Lactobacillus plantarum ATCC strain 202195, which not only colonized infant guts successfully, but stayed there for up to four months. That’s when they launched the big trial.

    Aside from preventing sepsis, it also reduced the risk of infections by both the major groups of bacteria: the Gram-positives, by 82 percent; and the Gram-negatives, which are harder to treat with antibiotics, by 75 percent. It even reduced the risk of pneumonia and other infections of the airways by 34 percent. That was “completely unexpected,” says Panigrahi, and it’s the result he’s especially excited about. It suggests that the synbiotic isn’t just acting within the gut, but also giving the infants’ immune systems a body-wide boost.

    Probiotics are not without risk. There have been rare cases where the bacteria in these products have caused sepsis in newborn or preterm infants. But Panigrahi saw no signs of that in his study: His synbiotic didn’t seem to cause any harmful side effects.

    Beyond protecting infants, Panigrahi says that this approach would also reduce the use of antibiotics, and slow the spread of drug-resistant infections. And perhaps best of all, it can be done cheaply. You’d need to treat 27 infants to prevent one case of sepsis, and each week-long course costs just one U.S. dollar.

    “It’s a very important study,” says Marie-Claire Arrieta from the University of Calgary. “It not only shows an effective and low-cost way to prevent a horrible infant disease that kills millions worldwide, but provides important clues on how to improve strategies to change the infant-gut microbiome.”

    Two earlier trials tested off-the-shelf probiotics on 1,099 and 1,315 premature infants respectively. Neither found any benefits for sepsis. Nor did an Indian trial involving 668 babies born with a low birth weight. In retrospect, such failures were to be expected. Sepsis is a varied and complicated condition. The microbiome is also incredibly varied in early life, and changes in ways we barely understand. “It’s not surprising that a one-size-fits-all approach hasn’t worked thus far,” says Arrieta. Success probably depends on choosing the right strain, administering it at the right time, and feeding it appropriately.

    Then again, Panigrahi’s trial only included healthy newborns of normal weight, whose mothers had begun to breastfeed them. They already had the best odds of fighting off infections, so it’s unclear if his synbiotic would work equally well with weaker or smaller babies, who are more prone to sepsis. It’s also unclear exactly why the synbiotic worked, or what effect it might have on the infants’ microbiomes in the long run.

    “We may need to test this in different settings and we’re working with the government to do so,” says Panigrahi. “But this should be the standard of care. The money involved is very small. The synbiotic can be manufactured anywhere without fancy technology. And it can do so much good.”


    Related Video

    Viewing all 25817 articles
    Browse latest View live


    <script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>