A team of Microsoft researchers said Wednesday that they believe they have created the first machine translation system that can translate sentences of news articles from Chinese to English with the same quality and accuracy as a person.

Researchers in the company’s Asia and U.S. labs said that their system achieved human parity on a commonly used test set of news stories, called newstest2017, which was developed by a group of industry and academic partners and released at a research conference called WMT17 last fall. To ensure the results were both accurate and on par with what people would have done, the team hired external bilingual human evaluators, who compared Microsoft’s results to two independently produced human reference translations.

Xuedong Huang, a technical fellow in charge of Microsoft’s speech, natural language and machine translation efforts, called it a major milestone in one of the most challenging natural language processing tasks.

“Hitting human parity in a machine translation task is a dream that all of us have had,” Huang said. “We just didn’t realize we’d be able to hit it so soon.”

Huang, who also led the group that recently achieved human parity in a conversational speech recognition task, said the translation milestone was especially gratifying because of the possibilities it has for helping people understand each other better.

“The pursuit of removing language barriers to help people communicate better is fantastic,” he said. “It’s very, very rewarding.”

Machine translation is a problem researchers have worked on for decades – and, experts say, for much of that time many believed human parity could never be achieved. Still, the researchers cautioned that the milestone does not mean that machine translation is a solved problem.

Ming Zhou, assistant managing director of Microsoft Research Asia and head of a natural language processing group that worked on the project, said that the team was thrilled to achieve the human parity milestone on the dataset. But he cautioned that there are still many challenges ahead, such as testing the system on real-time news stories.

Arul Menezes, partner research manager of Microsoft’s machine translation team, said the team set out to prove that its systems could perform about as well as a person when it used a language pair – Chinese and English – for which there is a lot of data, on a test set that includes the more commonplace vocabulary of general interest news stories.

“Given the best-case situation as far as data and availability of resources goes, we wanted to find out if we could actually match the performance of a professional human translator,” said Menezes, who helped lead the project.

Menezes said the research team can apply the technical breakthroughs they made for this achievement to Microsoft’s commercially available translation products in multiple languages. That will pave the way for more accurate and natural-sounding translations across other languages and for texts with more complex or niche vocabulary.

Dual learning, deliberation, joint training and agreement regularization

Although academic and industry researchers have worked on translation for years, they’ve recently achieved substantial breakthroughs by using a method of training AI systems called deep neural networks. That has allowed them to create more fluent, natural-sounding translations that take into account an even broader context than the previous approach, known as statistical machine translation.

To reach the human parity milestone on this dataset, three research teams in Microsoft’s Beijing and Redmond, Washington, research labs worked together to add a number of other training methods that would make the system more fluent and accurate. In many cases, these new methods mimic how people improve their own work iteratively, by going over it again and again until they get it right.

“Much of our research is really inspired by how we humans do things,” said Tie-Yan Liu, a principal research manager with Microsoft Research Asia in Beijing, who leads a machine learning team that worked on this project.

One method they used is dual learning. Think of this as a way of fact-checking the system’s work: Every time they sent a sentence through the system to be translated from Chinese to English, the research team also translated it back from English to Chinese. That’s similar to what people might do to make sure that their automated translations were accurate, and it allowed the system to refine and learn from its own mistakes. Dual learning, which was developed by the Microsoft research team, also can be used to improve results in other AI tasks.

Another method, called deliberation networks, is similar to how people edit and revise their own writing by going through it again and again. The researchers taught the system to repeat the process of translating the same sentence over and over, gradually refining and improving the response.

The researchers also developed two new techniques to improve the accuracy of their translations, Zhou said.

One technique, called joint training, was used to iteratively boost the English-to-Chinese and Chinese-to-English translation systems. With this method, the English-to-Chinese translation system translates new English sentences into Chinese in order to obtain new sentence pairs. Those are then used to augment the training dataset that is going in the opposite direction, from Chinese to English. The same procedure is then applied in the other direction. As they converge, the performance of both systems improves.

Another technique is called agreement regularization. With this method, the translation can be generated by having the system read from left to right or from right to left. If these two translation techniques generate the same translation, the result is considered more trustworthy than if they don’t get the same results. The method is used to encourage the systems to generate a consensus translation.

Zhou said he expects these methods and techniques to be useful for improving machine translation in other languages and situations as well. He said they also could be used to make other AI breakthroughs beyond translation.

“This is an area where machine translation research can apply to the whole field of AI research,” he said.

No ‘right’ answer

The test set the team used to reach the human parity milestone includes about 2,000 sentences from a sample of online newspapers that have been professionally translated.

Microsoft ran multiple evaluation rounds on the test set, randomly selecting hundreds of translations for evaluation each time. To verify that Microsoft’s machine translation was as good as a person’s translation, the company went beyond the specifications of the test set and hired a group of outside bilingual language consultants to compare Microsoft’s results against manually produced human translations.

The method of verifying the results highlights the complexity of teaching systems to translate accurately. With other tasks, such as speech recognition, it’s pretty straightforward to tell if a system is performing as well as a person, because the ideal result will be the exact same for a person and a machine. Researchers call that a pattern recognition task.

With translation, there’s more nuance. Even two fluent human translators might translate the exact same sentence slightly differently, and neither would be wrong. That’s because there’s more than one “right” way to say the same thing.

“Machine translation is much more complex than a pure pattern recognition task,” Zhou said.  “People can use different words to express the exact same thing, but you cannot necessarily say which one is better.”

The researchers say that complexity is what makes machine translation such a challenging problem, but also such a rewarding one.

Liu said no one knows whether machine translation systems will ever get good enough to translate any text in any language pair with the accuracy and lyricism of a human translator. But, he said, these recent breakthroughs allow the teams to move on to the next big steps toward that goal and other big AI achievements, such as reaching human parity in speech-to-speech translation.

“What we can predict is that definitely we will do better and better,” Liu said.

Related:

Allison Linn is a senior writer at Microsoft. Follow her on Twitter.

$ git push ...remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
remote: error: GH013: Your push could infringe someone's copyright.
remote: If you believe this is a false positive (e.g., it's yours, open
remote: source, not copyrightable, subject to exceptions) contact us:
remote: https://github.com/contact
remote: We're sorry for interrupting your work, but automated copyright
remote: filters are mandated by the EU's Article 13.
To github.com/vollmera/atom.git ! [remote rejected] patch-1 -> patch-1 (push declined due to article 13 filters)

The EU is considering a copyright proposal that would require code-sharing platforms to monitor all content that users upload for potential copyright infringement (see the EU Commission’s proposed Article 13 of the Copyright Directive). The proposal is aimed at music and videos on streaming platforms, based on a theory of a “value gap” between the profits those platforms make from uploaded works and what copyright holders of some uploaded works receive. However, the way it’s written captures many other types of content, including code.

We’d like to make sure developers in the EU who understand that automated filtering of code would make software less reliable and more expensive—and can explain this to EU policymakers—participate in the conversation.

Why you should care about upload filters

Upload filters (“censorship machines”) are one of the most controversial elements of the copyright proposal, raising a number of concerns, including:

• Privacy: Upload filters are a form of surveillance, effectively a “general monitoring obligation” prohibited by EU law
• Free speech: Requiring platforms to monitor content contradicts intermediary liability protections in EU law and creates incentives to remove content
• Ineffectiveness: Content detection tools are flawed (generate false positives, don’t fit all kinds of content) and overly burdensome, especially for small and medium-sized businesses that might not be able to afford them or the resulting litigation

Upload filters are especially concerning for software developers given that:

• Software developers create copyrightable works—their code—and those who choose an open source license want to allow that code to be shared
• False positives (and negatives) are especially likely for software code because code often has many contributors and layers, often with different licensing for different components
• Requiring code-hosting platforms to scan and automatically remove content could drastically impact software developers when their dependencies are removed due to false positives

The EU Parliament continues to introduce new proposals for Article 13 but these issues remain. MEP Julia Reda explains further in a recent proposal from Parliament.

EU policymakers want and need to hear from developers

As part of our ongoing collaboration with others affected, GitHub will help represent developers at an upcoming breakfast in Parliament on Tuesday, March 20, intended to show the human impact of this copyright proposal.

EU policymakers have told us it would be very useful to hear directly from more developers. In particular, developers at European companies can make a significant impact.

How to reach EU policymakers

1. Write to EU policymakers (MEPs, Council Members, or Commissioners) and ask them to exclude “software repositories” from Article 13. Please explain how important the ability to freely share code is for software developers and how important open source software is to the software industry and the EU economy

2. Explain this in person to EU policymakers

GitHub can help connect you with policymakers, provide additional background, or chat if you might be interested in representing software developers in defending your ability to share code and not have your builds break. Get in touch!

Sci-Hub is currently available at:

https://sci-hub.tw

Tip: bookmark the following link — it's always up to date! Sci-Hub.

Amazon’s Japanese headquarters in Tokyo have been raided by the country’s fair trade watchdog on suspicion of violation of antitrust regulations.

The Japanese Fair Trade Commission (JFTC) said on Thursday that Amazon Japan was being investigated after allegations that the company improperly asked suppliers to shoulder part of the costs of discounting their products on the retail site. Amazon Japan said Thursday that it was “fully cooperating” with JFTC, but declined to comment on the details of the allegations.

Local media the Asahi Shimbun daily reported that the firm may have demanded suppliers pay a “collaboration fee”, measured as a percentage of the selling price of the product. The Kyodo news agency, citing unnamed sources, said the fees were to cover discounting.

Public broadcaster NHK reported that Amazon told suppliers it would stop working with them if they did not pay the fees.

The JFTC declined to comment.

Amazon Japan has faced regulatory scrutiny before. In a recent antitrust probe, which involved raids on Amazon’s Tokyo offices in 2016, the JFTC found the firm had required suppliers sell items on Amazon Japan at the same or lower price as any listings they may have on on other platforms.

Japan’s antitrust law prohibits a firm from abusing a superior bargaining position to illicitly make a business partner accept unprofitable trade conditions.

The JFTC ended its probe in June after Amazon Japan agreed to drop the practice.

Amazon has been operating in Japan since 1998, with its Amazon.co.jp store front opening in 2000, making it one of the first of the US retail firm’s international expansions. Amazon Japan follows the model established by the firm’s US site, including its Marketplace where third parties can list and sell goods through Amazon’s storefront.

The firm has offices in the Japanese capital and in Osaka, as well as 13 fulfilment centres and five Prime Now warehouses for its same-day delivery services. The company last year reportedly agreed to pay up to 40% more to domestic delivery service firm Yamato, which said its workforce was struggling to keep up with rising demand for package delivery.

The assault was the most alarming in a string of cyberattacks on petrochemical plants in Saudi Arabia. In January 2017, computers went dark at the National Industrialization Company, Tasnee for short, which is one of the few privately owned Saudi petrochemical companies. Computers also crashed 15 miles away at Sadara Chemical Company, a joint venture between the oil and chemical giants Saudi Aramco and Dow Chemical.

Within minutes of the attack at Tasnee, the hard drives inside the company’s computers were destroyed and their data wiped clean, replaced with an image of Alan Kurdi, the small Syrian child who drowned off the coast of Turkey during his family’s attempt to flee that country’s civil war.

The intent of the January attacks, Tasnee officials and researchers at the security company Symantec believe, was to inflict lasting damage on the petrochemical companies and send a political message. Recovery took months.

Energy experts said the August attack could have been an attempt to complicate Crown Prince Mohammed bin Salman’s plans to encourage foreign and domestic private investment to diversify the Saudi economy and produce jobs for the country’s growing youth population.

“Not only is it an attack on the private sector, which is being touted to help promote growth in the Saudi economy, but it is also focused on the petrochemical sector, which is a core part of the Saudi economy,” said Amy Myers Jaffe, an expert on Middle East energy at the Council on Foreign Relations.

Saudi Arabia has cut oil exports in recent years to support global oil prices, a strategy central to its efforts to make a potential public offering of shares of government-controlled Saudi Aramco more attractive to international investors. The kingdom has tried to compensate for its lost revenue by expanding its petrochemical and refining industry.

Some technical details of the attack in August have been previously reported, but this is the first time the earlier attacks on Tasnee and other Saudi petrochemical companies have been reported.

Security analysts at Mandiant, a division of the security firm FireEye, are still investigating what happened in August, with the help of several companies in the United States that investigate cyberattacks on industrial control systems.

A team at Schneider Electric, which made the industrial systems that were targeted, called Triconex safety controllers, is also looking into the attack, the people who spoke to The Times said. So are the National Security Agency, the F.B.I., the Department of Homeland Security and the Pentagon’s Defense Advanced Research Projects Agency, which has been supporting research into forensic tools designed to assist hacking investigations.

All of the investigators believe the attack was most likely intended to cause an explosion that would have killed people. In the last few years, explosions at petrochemical plants in China and Mexico — though not triggered by hackers — have killed several employees, injured hundreds and forced evacuations of surrounding communities.

What worries investigators and intelligence analysts the most is that the attackers compromised Schneider’s Triconex controllers, which keep equipment operating safely by performing tasks like regulating voltage, pressure and temperatures. Those controllers are used in about 18,000 plants around the world, including nuclear and water treatment facilities, oil and gas refineries, and chemical plants.

“If attackers developed a technique against Schneider equipment in Saudi Arabia, they could very well deploy the same technique here in the United States,” said James A. Lewis, a cybersecurity expert at the Center for Strategic and International Studies, a Washington think tank.

The Triconex system was believed to be a “lock and key operation.” In other words, the safety controllers could be tweaked or dismantled only with physical contact.

So how did the hackers get in? Investigators found an odd digital file in a computer at an engineering workstation that looked like a legitimate part of the Schneider controllers but was designed to sabotage the system. Investigators will not say how it got there, but they do not believe it was an inside job. This was the first time these systems were sabotaged remotely.

The only thing that prevented significant damage was a bug in the attackers’ computer code that inadvertently shut down the plant’s production systems.

Investigators believe that the hackers have probably fixed their mistake by now, and that it is only a matter of time before they deploy the same technique against another industrial control system. A different group could also use those tools for its own attack.

The August attack was also a significant step up from earlier attacks in Saudi Arabia. Starting on Nov. 17, 2016, computer screens at a number of Saudi government computers went dark and their hard drives were erased, according to researchers at Symantec, which investigated the attacks.

Two weeks later, the same attackers hit other Saudi targets with the same computer virus. On Jan. 23, 2017, they struck again, at Tasnee and other petrochemical firms, deploying a computer virus known as Shamoon, after a word embedded in its code.

The Shamoon virus first surfaced five years earlier at Saudi Aramco, wiping out tens of thousands of computers and replacing the data with a partial image of a burning American flag. Leon E. Panetta, the United States defense secretary at the time, said the attack could be a harbinger.

“An aggressor nation or extremist group could use these kinds of cyber tools to gain control of critical switches,” he said.

Government officials and cybersecurity experts in Saudi Arabia and the United States attributed the 2012 Shamoon attack to Iranian hackers.

“Another attacker could have adopted that code” for the January 2017 attacks, said Vikram Thakur, a senior researcher at Symantec, “but our analysis showed the likelihood it was the same perpetrator was pretty high.”

The attack in August was not a Shamoon attack. It was much more dangerous.

Investigators believe a nation-state was responsible because there was no obvious profit motive, even though the attack would have required significant financial resources. And the computer code had not been seen in any earlier assaults. Every hacking tool had been custom built.

The attackers not only had to figure out how to get into that system, they had to understand its design well enough to know the layout of the facility — what pipes went where and which valves to turn in order to trigger an explosion.

Investigators believe someone would have had to buy the same version of the Triconex safety system to figure out how it worked. The components, investigators said, could be purchased for $40,000 on eBay.

The attack has also shown the challenge of attributing with unquestionable evidence an attack to one country.

Cybersecurity experts said Iran, China, Russia the United States and Israel had the technical sophistication to launch such attacks. But most of those countries had no motivation to do so. China and Russia are increasingly making energy deals with Saudi Arabia, and Israel and the United States have moved to cooperate with the kingdom against Iran.

That leaves Iran, which experts said had a growing cyberspace military program, although the Iranian government has denied any involvement in cyberattacks.

Tensions between Iran and Saudi Arabia have steadily escalated in recent years, and the conflict has drifted into cyberspace.

United States officials and security analysts blamed Iranian hackers for a spate of attacks on American banks in 2012 and more recent espionage attacks on the airline industry. Iranian hackers were blamed for the 2012 Aramco attack and are also the leading suspects in the more recent Shamoon attacks.

The August attack was far more sophisticated than any previous attack originating from Iran, Mr. Thakur of Symantec said, but there is a chance Iran could have improved its cyberwarfare abilities or worked with another country, like Russia or North Korea.

Tasnee said in an email that it had hired experts from Symantec and IBM to study the attack against it. The company said it had also “completely overhauled our security standards” and started using new tools to prevent cyberattacks.

“Being a global business,” the company said, “we believe that cybersecurity is a concern wherever you are in the world.”

Introduction

The housing market of coastal California is a good example of a market where growing demand and stagnant supply are leading to high and rising prices. From San Diego to Sonoma County, a strong economy, rich culture, and natural beauty attract Americans and immigrants from all age groups. At the same time, tight restrictions on building new housing at either urban centers or the exurban fringe limit the availability of new housing. When tight supply meets burgeoning demand, prices rise.

In a bid to allow for more new housing and thus slow or even reverse the ongoing growth in prices, State Senator Scott Wiener has introduced a bill, SB 827, in the California State Senate that would preempt some local restrictions on housing construction near transit stations and frequent bus routes. The bill, in its present form, is an appropriate use of the state’s preemptive power and is likely to slow or reverse the growth in housing costs. State law already includes a requirement for cities to permit new housing to meet targets for new supply, but the law has no teeth. SB 827, in turn, would more effectively limit localities’ ability to restrict housing supply.

In this article, we discuss the California housing market, the principled case for preempting municipal ordinances in California’s institutional environment, the specifics of SB 827, and the likely effects and limits of passing SB 827 into law.

California’s Housing Market

California contains 10 of the 11 most-expensive metropolitan areas in the country. The San Jose area has an eye-popping median single-family home value of $1,275,800. The median single-family home in the San Diego area (the 11th highest nationally) is worth half that, which is still about 50 percent more than the median value in the expensive New York, Washington, or Denver metro areas.

For renters, the picture is no better. Median apartment rent is above $3,000 per month in the San Jose and San Francisco areas. Eight more California metro areas round out the nation’s 12 costliest rental markets, along with New York and Boston. Statewide, monthly apartment rent rose $87 in 2017, compared to $25 nationwide.

Strict and time-consuming land use regulations and ordinances are a major source of inflexibility in California’s housing supply. The Wharton Residential Land Use Regulatory Index measures several dimensions of land use regulation based on a 2005 survey of local governments. Figures 1 and 2 compare California’s land use approval processes with the rest of the United States.

Many studies have shown that housing regulations are an important factor in lower construction rates, with a growing consensus that delays in permitting and construction are particularly insidious. Although California is far from the only high-regulation state, its inflexibility is particularly detrimental because the state is home to so many high-demand regions. Many people want to live in California because it is home to some of the country’s most productive labor markets in addition to providing geographic, climatic, and urban amenities.

As a consequence of land use restrictions, California metropolitan areas’ housing supply and populations have grown less than high-demand cities elsewhere. Although the housing growth champions of the 2010s are spacious, low-regulation cities like Austin (+19 percent), many areas that share California’s topographical barriers to growth have nonetheless added more housing in the last seven years than California metros, as figure 3 illustrates.

California’s small coastal cities are even more exclusive: the metropolitan areas that make up Ventura, Monterey, Santa Cruz, Sonoma, and Napa counties each added less than 3 percent to their housing stocks during the same time period.

Despite the demand for housing and infamous road congestion, many California transit stations are surrounded by low-density retail centers, suburban homes, and parking lots. The potential gains from the enactment of SB 827 in terms of both housing supply and increasing transit use are largest around low-density suburban stations, such as the area around Bay Fair Station on the Bay Area Rapid Transit (BART) system (see figure 4 on page 4).

In urban Los Angeles, single-family homes dominate the landscape in high-income and low-income neighborhoods alike. The closest buildings to the Expo Line’s Westwood/Rancho Park station on the light-rail line that runs between downtown Los Angeles and Santa Monica are suburban-style homes (see figure 5 on page 4). Local zoning prevents their owners from taking advantage of the high prices and availability of rapid transit to downtown LA.

Midrise development in the 45- to 85-foot range, which SB 827 would allow, would certainly transform areas like these. A Google Street View image (see figure 6 on page 4) from newly developed Mission Bay shows a main street with mid-rise housing.

California cities offer amenities and economic opportunities that are attracting new residents from across the country and around the world. To accommodate a growing population, the state needs a new regulatory approach, one that allows new growth and offers clearly delineated rules to minimize bureaucratic delays. 

California’s Role in Preempting Local Land Use Rules

SB 827 would implement state preemption of local land use regulation near public transit. In other words, the state would be setting limits on the extent to which municipalities can restrict development. Preemption of local land use regulations is a legal recourse of state governments because municipalities are “creatures of their states.” The State of California should have a role to play in restraining local land use regulations for three key reasons: protecting individual property rights, facilitating economic growth, and supporting efficient use of state transportation investments.

Individual Property Rights

Economist Michael Farren has argued that, in general, rulemaking should rest at the most local level of government possible, but that higher level governments should use their authority to ensure that lower levels of government do not violate individual rights.

This principled case for the preemption embodied by SB 827 rests on the state’s duty to protect property owners’ rights to determine the best use of their land. The bill would allow more development options than current local rules do, in turn increasing access to high-demand areas and allowing more people to take advantage of existing transit investments.

Economic Growth

Aside from protecting individual rights, easing land use regulation in California would improve conditions for economic growth. The density of people, firms, and industries within cities results in agglomeration benefits; living in cities makes those residents more productive by giving them an opportunity to learn from one another and creating an environment that supports innovation.

Kyle Herkenhoff, Lee Ohanian, and Edward Prescott used a macroeconomic model to conclude that “Deregulating only California to its 1980 level and leaving the land-use regulation level of all other states unchanged, raises [national] output, investment, [productivity], and consumption by about 1.5%, and increases California’s population by about 6.0 million workers.” While SB 827 does not go as far as this simulation in deregulation, it is nevertheless a policy step toward promoting economic growth.

When local jurisdictions permit new housing construction, they get only a fraction of the resulting benefits (such as economic growth and lower regional housing prices) while absorbing most of the costs. Consequently, local governments constrain new developments at the expense of regional welfare gains. State action provides a coordinating mechanism for citizens to share the costs as well as the benefits of growth. 

Efficient Transportation Investments

The role that California plays in funding transportation provides an additional nexus for state preemption of local land use rules. Last year, the state allocated about $400 million to transit projects. When local governments don’t permit housing development near these transit investments, buses and expensive rail projects benefit a smaller number of riders—and require larger subsidies—than if denser development were allowed near transit corridors. 

Los Angeles’ most recent Exposition Line extension, for example, connects neighborhoods that are largely zoned for single family development from Santa Monica to Culver City. The line covers less than a quarter of its operating expenses with ticket sales, in part because under current zoning, dense housing development is prohibited, so few people are able to take advantage of it. With liberal zoning near transit, transit fares could cover a larger portion of its expenses, freeing taxpayers from having to subsidize it so heavily.

Housing growth along the Caltrain corridor from San Jose to San Francisco has been more than twice the Bay Area average since 2010, allowing Caltrain to almost double its ridership, lower its per-passenger operating subsidy from $3.80 to $1.00, and cut its taxpayer subsidy in half. Even Californians who never use transit still benefit from efficient private development around transit stations.

SB 827’s Requirements

SB 827 would set limits on local housing regulations through “transit-rich housing bonuses.” These would apply to housing projects that sit within half a mile of a major transit stop or within a quarter mile of a high-quality transit corridor.

The bill would preempt the following local regulations on new housing for projects that meet the bonus requirements:

• Limits on residential density,
• Parking requirements, and
• Design standards that restrict residential density beyond building code requirements.

The bill would also preempt local height limits based on the following criteria:

• Eighty-five feet would be the lowest allowable minimum height limit for projects within one-quarter mile of a high-quality transit corridor or within one block of a major transit stop, except for parcels facing streets that are less than 45 feet wide. In these cases, the lowest allowable height limit would be 55 feet.
• For projects that are within a half mile of a major transit stop, the bill would preempt height limits below 55 feet, except when the parcel is facing a street that is less than 45 feet wide, in which case the lowest allowable height limit would be 45 feet.

Today, many of California’s desirable, transit-accessible neighborhoods are zoned for only low-density residential uses. This bill would make it legal for property owners to sell these homes to developers who would replace them with mid-rise residential buildings that would allow more people to access these locations.

All of the preemptions in SB 827 would reduce current barriers to housing supply that result in high and rising house prices in many California markets. Height limits and density restrictions limit supply directly by preventing developers from taking advantage of high-value land with apartment buildings that allow many households to live in prime locations. Minimum lot size and minimum unit size requirements are the quintessential “snob zoning” rules that communities have used to outlaw inexpensive housing in their jurisdictions, preventing lower-income households from moving in.

Rules that require open space rather than allowing developers to build to parcels’ lot lines interact with height limits to restrict the potential supply of housing within a local jurisdiction. Similarly, while parking requirements may not seem to limit housing supply directly, they mandate that space is dedicated to car storage rather than alternative uses. When developers respond to parking requirements by providing surface lots, less land is available for development. In high-cost locations, developers may choose to provide very expensive above-ground or underground garages. In Los Angeles, one estimate finds that parking requirements contribute over $100,000 to the cost of each new apartment unit. Under SB 827, some apartments would be built with less parking, resulting in lower-cost options for those who are willing to trade fewer parking spots for cheaper rent.

SB 827 would establish limits on local regulations, but it would not set any minimum sizes on new buildings or establish rules requiring new developments to meet minimum density standards. It would simply transfer some development decisions from local governments to developers.

Institutional Interactions

Preempting local ordinances that limit citizens’ property rights and worsen economic outcomes is certainly within the state’s purview, but it should be done carefully. As written, SB 827 could lead to unintended responses from local policymakers who are looking for ways to circumvent the bill’s preemption. 

Potential for Municipal Workarounds

Local governments will likely respond to a reduction in their power to implement exclusionary zoning and attempt to find creative ways to limit the effects of SB 827 without running directly afoul of the law. To prevent the construction of new market-rate affordable housing, municipalities could pursue historical designations or shut down bus routes.

While SB 827 prevents zoning code design standards that drive up housing costs, it leaves open the door for local regulators to put costly requirements in their building codes—a platinum toilet mandate, for example. Or a municipality could mandate that 100 percent of new apartments near transit be leased or sold at below-market rates, ensuring that only nonprofit housing is built. By increasing the cost of development, requirements that developers provide below-market-rate housing decreases the potential for SB 827 to increase housing supply. If localities scramble to update zoning codes, there is a risk of more mid-project rule changes that currently bedevil construction projects.

Analysis from the San Francisco Planning Department points out that, as written, SB 827 leaves local review processes in place. Policymakers who wish to severely restrict new development in their jurisdiction could simply allow the queue of proposed projects to accumulate without accelerating the rate at which they approve new development.

SB 827 could lead to rapid municipal rulemaking by local policymakers who are seeking to avoid its intended outcome. In turn, this could create significant policy confusion and an unnecessary rush by developers to secure permits before new local rules take effect. SB 827 should include temporary relief from local responses in the defined transit-rich zones. Any project that applies for a permit within the next five years should be eligible to use January 1, 2018, zoning and building codes as well as the SB 827 preemptions. This exemption would remove the incentive for localities to respond too rapidly in either trying to sabotage or accommodate SB 827 and provide builders with precious clarity as they proceed.

Environmental Protection

The California Environmental Quality Act (CEQA) offers neighbors the chance to tie up any unpopular development project in court. Under CEQA’s “private right of action,” anyone can file a lawsuit to halt a project, public or private, that has allegedly failed to consider some adverse outcome, such as congestion or air quality. Jennifer Hernandez, David Friedman, and Stephanie DeHerrera note that “any party can file a CEQA lawsuit, even if it has no environmental purpose. For example, a competitor can file a CEQA lawsuit to delay or derail a competing project.” Despite the name, CEQA is used in urban California to stop the very infrastructure, infill, and apartment construction that would make California a more environmentally friendly place. 

Since transit-oriented infill development is far more environmentally friendly than the alternatives, exempting construction in transit-rich areas from CEQA challenges would be an effective measure in the efforts to control housing costs. These projects would still have to abide by environmental rules, but enforcement would be left in the hands of the state, as it is elsewhere in the United States. Without CEQA, residents would still have recourse against true harms under existing nuisance laws.

Conclusion

Ahead of SB 827’s first conference hearing, Senator Wiener has introduced amendments that would exempt some parcels from the bill’s preemptions and add relocation benefits for tenants affected by demolition. If the bill moves forward in the political process, it will likely continue to evolve. Changes that reduce the amount of land where local zoning rules are preempted will also reduce potential new housing supply that puts downward pressure on prices. However, as long as the bill still upzones substantial amounts of land near transit stations, it can be expected to increase housing supply growth and improve affordability.

States have an important role to play in protecting individual rights from local restrictions. When states preempt local land use regulations, they also facilitate competition between local jurisdictions, promote economic growth, and, as in the case of SB 827, reduce transit subsidies. 

SB 827 is an attempt by state policymakers to rein in local land-use regulations. Because California residents suffer the pains of high and rising housing prices induced by regulatory constraints on supply, state preemption has the potential to create opportunities for increased access to housing in several markets.

Localities could escape preemption by putting new design standards in their building codes, excluding themselves from public transit routes, implementing historic preservation or affordability requirements, or encouraging lawsuits under the guise of environmentalism. Failing to anticipate these workarounds will limit the chances that the bill achieves its intended goal of increasing access to housing in transit-rich locations.

It has been pretty obvious for a few years now that Theranos Inc. was a huge fraud. Theranos is a blood-testing startup that developed devices, which it called "TSPUs" and "miniLabs," that were supposed to be able to do a wide range of laboratory tests on a finger-prick blood sample. It seems like Theranos founder Elizabeth Holmes really wanted to build devices that would actually do these things, and thought she could, and tried to. But it didn't work, and Theranos ran out of time: It talked Walgreens into offering Theranos tests at its stores, but "it became clear to Holmes that the miniLab would not be ready" in time for the Walgreens rollout. So she went with Plan B: "Theranos never used its miniLab for patient testing in its clinical laboratory," but did a dozen tests on the earlier-generation Theranos TSPU, 50 to 60 more tests on blood-test-analysis devices that it bought from other companies and modified to take finger-prick samples, and "the remaining 100-plus tests it offered" on regular unmodified devices bought from other companies or sent out to third-party laboratories. Meanwhile Theranos and Holmes were going around giving interviews about how revolutionary their technology was, without ever mentioning that it didn't work and they didn't use it. This got them a lot of favorable press and a $9 billion valuation, which went on for a while until the Wall Street Journal's John Carreyrou reported in 2015 that the product didn't work and that Theranos was lying about using it, after which Theranos fairly quickly collapsed.

But the fact that Theranos was a gigantic fraud doesn't quite mean that it committed fraud. It isn't exactly fraud to go around lying to journalists. People do it all the time! If you decide that you want to be a celebrity, and that the easiest path to fame is by convincing people that you've found a magical new blood test, you can lie about that to your heart's content, and if you fool people then that's their problem, not yours. Undeserved celebrity is a central fact of American life; if it was illegal to lie your way to fame then our politics, for one thing, would be very different.

It becomes fraud in the legal sense if you use those lies to get money. Theranos, in parallel with being a massive fraud, was also raising a lot of money. I used to refer to it pretty regularly as the Blood Unicorn, Elasmotherium haimatos, because it was a Silicon Valley unicorn with a peak valuation of $9 billion that managed to raise $700 million from investors. If you are going around lying publicly about your technology while also raising hundreds of millions of dollars from investors, that certainly suggests that you were defrauding those investors. But it's not a certainty. Theranos wasn't a public company; it raised all that money in negotiated private fundraising rounds where investors received disclosure documents and had the opportunity to conduct due diligence. Perhaps while it was going around talking up its fake product to the press, it was simultaneously giving investors thorough disclosure documents that made clear exactly where its technology stood and exactly what were the risks to its business. Perhaps the investors knew that the technology wasn't ready yet, but invested in the company anyway because they believed that it would be ready one day, and they were kept sufficiently informed of the actual progress for that to be a reasonable belief. This would be a bit of a strange way to roll -- telling self-flattering lies to the press while giving your investors the unvarnished truth -- but it is not impossible, and it would give Theranos a defense against fraud charges.

Unicorns

But no, no, that's not what happened at all. Instead the Securities and Exchange Commission today brought fraud charges against Holmes, Theranos and its former president, Sunny Balwani, and its complaint alleges pretty strongly that the investors were just as bamboozled as everybody else. In fact, Theranos made direct use of its positive press to raise money: It "sent investors a binder of background materials," which included "articles and profiles about Theranos, including the 2013 and 2014 articles from The Wall Street Journal, Wired, and Fortune that were written after Holmes provided them with interviews" and that included her misleading claims about the state of Theranos's technology. She also repeated those claims to investors directly: "For instance, Holmes and Balwani told one investor that Theranos' proprietary analyzer could process over 1,000 Current Procedural Terminology ('CPT') codes and that Theranos had developed a technological solution for an additional 300 CPT codes," even though "Theranos' analyzers never performed comprehensive testing or processed 1,000 CPT codes in its clinical lab," and in fact never processed more than 12 tests on its TSPU. And Theranos would even do a little pantomime blood-draw demonstration directly on the investors:

This initial meeting was often followed by a purported demonstration of Theranos' proprietary analyzers, the TSPU, and the miniLab. In several instances, potential investors would be taken by Holmes and Balwani to a different room to view Theranos' desktop computer-like analyzers. A phlebotomist would arrive to draw their blood through fingerstick, using a nanotainer, a Theranos-developed collection device. Then the sample was either inserted into the TSPU or taken away for processing. Based on what they saw, potential investors believed that Theranos had tested their blood on either an earlier-generation TSPU or the miniLab. As Holmes knew, or was reckless in not knowing, however, Theranos often actually tested their blood on third-party analyzers, because Theranos could not conduct all of the tests it offered prospective investors on its proprietary analyzers.

And so the SEC decided that this all did indeed amount to securities fraud. Theranos and Holmes settled with the SEC without admitting or denying the allegations; Balwani will apparently fight the accusations. Holmes agreed to pay a $500,000 penalty to the SEC, "return" 18.9 million Theranos shares to the company and relinquish her super-voting control, and be barred from serving as a public-company director or officer for 10 years.  That is a pretty small fine for such a big fraud: Martin Shkreli had to forfeit $7.4 million for what a judge found to be a $10.4 million fraud, while Holmes will pay just $500,000 for a $700 million fraud. But Holmes, unlike Shkreli, does not seem to have a lavish collection of Picassos and Wu-Tang Clan albums to liquidate to pay a fine. Unlike most people who run nine-digit frauds, she never took much money out: The SEC notes that she "was paid a salary of approximately $200,000 to $390,000 per year between 2013 and 2015" and "has never sold any of her Theranos stock." Forbes once estimated Holmes's net worth at $4.5 billion, but essentially all of that was in stock that is now probably worthless. In a very real sense she was the biggest victim of her own fraud.

Two other points. First, when the Theranos story first broke in 2015, you would occasionally see people saying things like "if this were a public company it would clearly be securities fraud." That always struck me as a curious analysis: Securities fraud just means committing fraud in connection with the sale of securities; whether those securities are public or private doesn't have anything to do with it. The SEC's case today doesn't just confirm that Theranos was a fraud; it also confirms that the SEC will pursue securities fraud in private markets:

"Investors are entitled to nothing less than complete truth and candor from companies and their executives," Steven Peikin, the co-director of the SEC's enforcement division, said in a statement. "There is no exemption from the anti-fraud provisions of the federal securities laws simply because a company is non-public, development-stage, or the subject of exuberant media attention."

Second: We talk sometimes around here about how U.S. law seems to classify a lot of random kinds of misbehavior as securities fraud. Intentionally slowing down iPhones or mispricing chickens or denying climate change or lobbying against fiduciary regulation or overprescribing opioids or municipal bribery can all count as securities fraud: If a securities issuer does a bad thing and doesn't tell its shareholders about it, then that's enough to make out a case of securities fraud, and it is often easier to punish the company for that securities fraud than for the underlying bad thing. There is something morally strange about this: "Securities fraud" suggests that the company's shareholders are the victims, while often what actually happened is that the company victimized someone else in order to make more money for its shareholders.  

That's not what's going on here: Theranos really did deceive its investors, and they really were victims of its fraud. But they weren't the only victims. The problem with launching a blood-test machine that doesn't work isn't just that you swindle the investors who funded the machine's development. You are also out there performing a lot of fake blood tests. The Wall Street Journal has reported on the "trail of agonized patients" who got blood-test results from Theranos that turned out to be wrong, and Theranos ultimately "voided" two years of results from its machines because they were not sufficiently accurate. Building a fake blood-testing company that raises hundreds of millions of dollars from investors is bad, certainly, but it's not really any worse than any of the other securities fraud that we so often delight in around here. Building a fake blood-testing company that performs fake blood tests on thousands of people is much worse, even if it doesn't count as securities fraud.

This column does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.

To contact the author of this story:
Matt Levine at mlevine51@bloomberg.net

To contact the editor responsible for this story:
James Greiff at jgreiff@bloomberg.net

Blue Apron Holdings Inc. will try to give its struggling business a boost by selling meal kits in stores, acknowledging that its subscription-only model isn’t enough in the intensifying fight to fill people’s dinner plates.

One of the first meal-kit companies, Blue Apron has been losing customers amid increased competition and distribution problems. The company reported 750,000 subscribers last month, down from a peak of over 1 million last year. Its shares, down 46% so far this year, are trading around a fifth of their debut...

In addition to operating the music streaming service iHeartRadio, iHeartMedia runs over 850 stations, stages music festivals and produces events and concerts. It's unclear if its "balance sheet restructuring" would put some of its less popular properties off the air, but the company believes that the cash it has on hand along with the money its ongoing projects will generate will be enough to support its business during the bankruptcy proceedings.

iHeartMedia chief Bob Pittman also sounds optimistic, since this means the company "only" has another $10 billion in debt to worry about. He said in a statement:

"The agreement we announced today is a significant accomplishment, as it allows us to definitively address the more than $20 billion in debt that has burdened our capital structure. Achieving a capital structure that finally matches our impressive operating business will further enhance iHeartMedia's position as America's #1 audio company."

(Reuters) - Amazon.com Inc’s top television shows drew more than 5 million people worldwide to its Prime shopping club by early 2017, according to company documents, revealing for the first time how the retailer’s bet on original video is paying off.

The documents also show that Amazon’s U.S. audience for all video programming on Prime, including films and TV shows it licenses from other companies, was about 26 million customers. Amazon has never released figures for its total audience.

The internal documents compare metrics that have never been reported for 19 shows exclusive to Amazon: their cost, their viewership and the number of people they helped lure to Prime. Known as Prime Originals, the shows account for as much as a quarter of what analysts estimate to be total Prime sign-ups from late 2014 to early 2017, the period covered by the documents.

Core to Amazon’s strategy is the use of video to convert viewers into shoppers. Fans access Amazon’s lineup by joining Prime, a club that includes two-day package delivery and other perks, for an annual fee.

The company declined to comment on the documents seen by Reuters. But Chief Executive Jeff Bezos has been upfront about the company’s use of entertainment to drive merchandise sales. The world’s biggest online retailer launched Amazon Studios in 2010 to develop original programs that have since grabbed awards and Hollywood buzz.

“When we win a Golden Globe, it helps us sell more shoes,” Bezos said at a 2016 technology conference near Los Angeles. He said film and TV customers renew their subscriptions “at higher rates, and they convert from free trials at higher rates” than members who do not stream videos on Prime.

Video has grown to be one of Amazon’s biggest expenditures at $5 billion per year for original and licensed content, two people familiar with the matter said. The company has never disclosed how many subscribers it won as a result, making it hard for investors to evaluate its programming decisions.

The internal documents show what Amazon considers to be the financial logic of its strategy, and why the company is now making more commercial projects in addition to high-brow shows aimed at winning awards, the people said.

For example, the first season of the popular drama “The Man in the High Castle,” an alternate history depicting Germany as the victor of World War Two, had 8 million U.S. viewers as of early 2017, according to the documents. The program cost $72 million (£51.5 million) in production and marketing and attracted 1.15 million new subscribers worldwide based on Amazon’s accounting, the documents showed.

Amazon calculated that the show drew new Prime members at an average cost of $63 per subscriber.

That is far less than the $99 that subscribers pay in the United States for Prime; the company charges similar fees abroad. Prime members also buy more goods from Amazon than non-members, Bezos has said, further boosting profit.

AMAZON’S SECRET MATH

Precisely how Amazon determines a customer’s motivation for joining its Prime club is not clear from the documents viewed by Reuters.

But a person familiar with its strategy said the company credits a specific show for luring someone to start or extend a Prime subscription if that program is the first one a customer streams after signing up. That metric, referenced throughout the documents, is known as a “first stream.”

The company then calculates how expensive the viewer was to acquire by dividing the show’s costs by the number of first streams it had. The lower that figure, the better.

(For a graphic on how Amazon evaluates its TV shows, see: tmsnrt.rs/2FkFGHgtmsnrt.rs/2FkFGHg))

The internal documents do not show how long subscribers stayed with Prime, nor do they indicate how much shopping they do on Amazon. The company reviews other metrics for its programs as well. Consequently, the documents do not provide enough information to determine the overall profitability of Amazon’s Hollywood endeavour.

Still, the numbers indicate that broad-interest shows can lure Prime members cheaply by Amazon’s calculations. One big winner was the motoring series “The Grand Tour,” which stars the former presenters of BBC’s “Top Gear.” The show had more than 1.5 million first streams from Prime members worldwide, at a cost of $49 per subscriber in its first season.

The documents seen by Reuters reflect Prime subscribers in the United States, United Kingdom, Germany, Austria and Japan, where Amazon’s programs were available before Prime Video rolled out globally in December 2016.

Analysts estimate that 75 million or more customers have Prime subscriptions worldwide, including about half of all households in the United States.

BIGGER BETS

About 26 million U.S. Prime members watched television and movies on Amazon as of early 2017. Reuters calculated this number from the documents, which showed how many viewers a TV series had as a percentage of total Prime Video customers.

Rival Netflix Inc had twice that many U.S. subscribers in the first quarter of last year. It does not disclose how many were active viewers.

For years, Amazon Studios aimed to win credibility in Hollywood with sophisticated shows beloved by critics. Its marquee series “Transparent,” about a transgender father and his family, won eight Primetime Emmy Awards and created the buzz Amazon wanted to attract top producers and actors.

Yet “Transparent” lagged Amazon’s top shows in viewership. Its first season drew a U.S. audience half as large as that of “The Man in the High Castle,” and it fell to 1.3 million viewers for its third season, according to the documents.

Similarly, “Good Girls Revolt,” a critically-acclaimed show about gender inequality in a New York newsroom, had total U.S. viewership of 1.6 million but cost $81 million, with only 52,000 first streams worldwide by Prime members.

The program’s cost per new customer was about $1560, according to the documents. Amazon cancelled it after one season.

Amazon is now working on more commercial dramas and spin-offs with appeal outside the United States, where Prime membership has far more room to grow, people familiar with the matter said.

The effort to broaden Amazon’s lineup, long in the works, will be in the hands of Jennifer Salke, NBC Entertainment’s president whom Amazon hired last month as its studio chief. Amazon’s Bezos has wanted a drama to rival HBO’s global hit “Game of Thrones,” according to the people.

In November, Amazon announced it will make a prequel to the fantasy hit “The Lord of the Rings.” The company had offered $250 million for the rights alone; production and marketing could raise costs to $500 million or more for two seasons, one of the people said.

At half a billion dollars, the prequel would cost triple what Amazon paid for “The Man in the High Castle” seasons one and two, the documents show. That means it would need to draw three times the number of Prime members as “The Man in the High Castle” for an equal payoff.

After nearly two weeks of continuous curling coverage, a viewer of the Winter Olympics could be forgiven for concluding that curling, while it may technically qualify as a sport, does not seem terribly Olympian. Archery, too, is fringe and unexciting, but at least the Greek goddess Artemis did it. Hockey—Ares would have loved it. Hermes probably kept a luge uniform in his closet. But curling exists because (and with all due respect to the many fine athletes involved) . . . why?

Scientists have long wondered the same thing, and lately the urge to explain it has taken on a competitive edge. By now, you know the basic rules: two teams take turns sliding, or “throwing,” a fortyish-pound puck of granite down a lane of ice toward the center of a horizontal target. When the stone is set in motion, it is also made to rotate slightly, and this rotation causes it to curve, or curl, to one side or another. If you’re the thrower, you can aim your stone to block your opponents’ stones, knock them out of the way, or even slide around them; own the curl, you own the game. To help the stone reach its intended target, your teammates can, using special brooms, furiously sweep the ice directly in front of it, warming the ice, reducing the friction, and making the stone travel slightly farther. It’s shuffleboard meets Swiffer ad.

“It’s way harder than you think,” Mark Shegelski, a physicist at the University of Northern British Columbia, and a recreational curler, told me recently. “It’s like golf: it’s easy to watch a guy hit a golf ball, and you think, ‘This isn’t very athletic.’ And then you get out there yourself and find that it’s incredibly difficult.” Also incredibly difficult: understanding why curling stones curl the way they do, a problem that Shegelski has been chipping away at for two decades.

Unlike skating ice, which is made to be as smooth as possible— “burnt,” in industry parlance—curling ice is pebbled. Between games, it is sprayed with droplets of water, which freeze to form microscopic bumps. As all curlers know, pebbling is essential to the sport; without it, a curling stone wouldn’t curl. This, however, is where the certainty ends. In most other respects, Shegelski told me, curling defies traditional logic.

The bottom of a curling stone resembles the bottom of a beer bottle. It’s concave, not flat, so as it slides only a narrow ring of stone—the running band—actually interacts with the ice. Take a beer bottle or an upturned glass and send it spinning down a table: if it rotates to the right, clockwise, it will curl to the left; if it rotates to the left, it will curl to the right. That’s because the bottle, as it moves forward, also tips forward slightly, adding weight to the leading edge. More weight means more friction. As the leading edge turns to the right, it meets with greater resistance than the back edge, turning to the left, does. So the clockwise-spinning bottle follows the path of least resistance, curling to the left.

Weirdly, a curling stone on ice does exactly the opposite: if it rotates to the right, it curls right, and vice versa. Shegelski said that, at the bar after a game (“Drinking beer after curling is absolutely required; it’s a must”), he sometimes blows the minds of fellow-curlers by sending an upturned glass spinning across the table. “To their horror, the drinking glass curls the wrong way,” he said. “All the curlers would be, like, ‘Whoa, how’d you do that?!’ ”

Early attempts to explain a curling stone’s behavior essentially worked the beer-bottle mechanism in reverse. If a clockwise-spinning bottle curls left because there’s more friction in the front than in the back, a stone spinning the same way must curl right because there’s more friction in the back than in the front. But why? Several theories emerged under the umbrella of “asymmetrical friction,” including, in the late nineteen-nineties, one by Shegelski. He proposed that, like the beer bottle, the curling stone tips forward slightly as it slides; the added pressure warms the ice, creating a thin film of water that acts as a lubricant, which reduces the friction in the front and, by comparison, increases it in the back.

This became known as the thin-liquid-film model, and it reigned for a few years, largely for want of challengers. But there’s more to the mystery of the curl than just “Why?”; there’s also “How much?” A curling stone can curl by as much as a metre and a half to either side. It’s clear that the curl is caused by rotation, since a stone that’s thrown without rotation doesn’t curl. But, in a game, the typical stone rotates only a couple of times during its long slide, and asymmetrical friction doesn’t generate enough force to produce that much curl. Even stranger, the curl stays pretty much the same whether the stone rotates twice or twenty times. “These models will not work, because the effect will never be strong enough to explain what we see,” Harald Nyberg, a materials scientist and friction expert at Uppsala University, in Sweden, told me.

In June, 2013, Nyberg and his colleagues made that argument in an equation-laden paper in the journal Tribology Letters (“tribology” being the fancy word for “friction science”). Not long before, in the journal Wear, they hadproposed a model of their own, which became known as scratch-guiding theory. Using images from an electron microscope, the researchers showed that, as a curling stone slides along, it leaves fine scratches on the ice in the direction of rotation. The scratches are laid down by the front edge of the running band, but when the back edge encounters them it has a tendency to follow them, causing the stone to curl in the direction of rotation. In follow-up experiments, Nyberg’s group found that, by scratching the ice themselves in various ways, they could alter the trajectory of sliding stones, even ones that weren’t rotating. In one setup, they created a lane in which the ice was scratched in one direction and then, farther on, in the other direction. Then they slid a stone down the lane and watched as it curled, first one way and then the other.

“All of this added up to a mechanism that we felt was reasonable,” Nyberg said. He feels further vindicated by a controversy that broke out a couple of years ago: curlers were using new brooms that scratched the ice rather than merely warming it, enabling them to control the curl like never before. The “Frankenbrooms” have since been banned by the World Curling Federation. “I don’t think they’d read our papers, though,” Nyberg said.

One person who did read Nyberg’s papers was Shegelski, and in January, 2016, he wrote a comment piece in Tribology Letters that called the dismissal of asymmetrical-friction models “inappropriate.” Nyberg and his colleagues soon replied, reiterating their earlier points. When I spoke with Shegelski, he noted that, while scratch-guiding theory is intriguing, the theorists haven’t attempted to demonstrate that the mechanism can generate a metre-wide curl, much less explain why the curl is the same regardless of the rotation rate. “The fact is, in a theory, you need to have quantitative results, and there aren’t any,” he said.

Recently, Shegelski teamed up with Edward Lozowski, a physicist and atmospheric scientist at the University of Alberta, to reconsider the curling conundrum. Some years earlier, Lozowski had published papers on the science of bobsledding and speed skating. “Ed is just a wizard at the physics of ice,” Shegelski told me. Together, the two men developed an improved explanation, which they unveiled in the latest issue of Cold Regions Science and Technology. They call it the pivot-slide model. Lozowski, whom I spoke to over Skype, explained it to me by holding up a hair comb and running his finger slowly across the teeth. Notice, he said, that his fingertip adheres to each tooth long enough to bend it, until the elastic force becomes large enough that the tooth breaks free and snaps back into place—that’s stick-slip friction. The same force will cause a circular saw, when it binds, to jump up and try to pivot around the obstacle. That’s what’s happening with a curling stone, he said.

“Every time the running band encounters a pebble, it catches on it,” Lozowski explained. “And, because ice is elastic, the pebble is deflected, then snaps back, and the rock moves on, pivoting during the deflection.” The pebbles and the pivots are small but they are many, and Lozowski and Shegelski calculated that the net result is enough to curl the stone by a metre. They emphasized that there is still a lot of research to be done—making models of the curling ice, quantifying the number of pebbles per unit area. “It could be all wrong,” Lozowski said. “But whether it’s right or not, at least it’s testable.” Nyberg, for one, is unpersuaded. “To be honest, I don’t really understand what they’re getting at,” he said.

“We aren’t by any means saying we’ve figured it all out,” Shegelski said. “What curling rocks do is so complicated that there’s got to be more than one thing going on.” He said that he wouldn’t be at all surprised if the stone’s path was shaped by a series of mechanisms—one at the beginning of the slide, another in the middle, maybe another toward the end. “It will take more than two people working on this to solve it,” he said. “We all need to pitch in.”

An amazing virtual reality OLED display under development by Google and LG has some downright impressive specifications:

• 4.3 inch 18 megapixel AMOLED display
• 5500 x 3000 resolution
• 1443 dots per inch
• 120 Hz refresh rate

Previous OLEDs were only 600 dpi, and this is twice the resolution of the HTC VIVE Pro.

Google made a large nearly-billion-dollar investment in LG to secure a supply of OLEDs for their Pixel phones, and it appears that this co-operation also extended to VR OLED displays. They will be showing this VR display at Display Week 2018 expo in May.

Being an OLED, it is anticipated that it will likely have a low-persistence mode similar to HTC Vive and Oculus Rift, to eliminate OLED motion blur.

We look forward to seeing VR headsets containing this display!

“DESIGNED by Apple in California. Assembled in China”. For the past decade the words embossed on the back of iPhones have served as shorthand for the technological bargain between the world’s two biggest economies: America supplies the brains and China the brawn.

Not any more. China’s world-class tech giants, Alibaba and Tencent, have market values of around $500bn, rivalling Facebook’s. China has the largest online-payments market. Its equipment is being exported across the world. It has the fastest supercomputer. It is building the world’s most lavish quantum-computing research centre. Its forthcoming satellite-navigation system will compete with America’s GPS by 2020.

America is rattled. An investigation is under way that is expected to conclude that China’s theft of intellectual property has cost American companies around $1trn; stinging tariffs may follow. Earlier this year Congress introduced a bill to stop the government doing business with two Chinese telecoms firms, Huawei and ZTE. Eric Schmidt, the former chairman of Alphabet, Google’s parent, has warned that China will overtake America in artificial intelligence (AI) by 2025.

This week President Donald Trump abruptly blocked a $142bn hostile takeover of Qualcomm, an American chipmaker, by Broadcom, a Singapore-domiciled rival, citing national-security fears over Chinese leadership in 5G, a new wireless technology. As so often, Mr Trump has identified a genuine challenge, but is bungling the response. China’s technological rise requires a strategic answer, not a knee-jerk one.

The motherboard of all wars

To understand what America’s strategy should be, first define the problem. It is entirely natural for a continent-sized, rapidly growing economy with a culture of scientific inquiry to enjoy a technological renaissance. Already, China has one of the biggest clusters of AI scientists. It has over 800m internet users, more than any other country, which means more data on which to hone its new AI. The technological advances this brings will benefit countless people, Americans among them. For the United States to seek to keep China down merely to preserve its place in the pecking order by, say, further balkanising the internet, is a recipe for a poorer, discordant—and possibly warlike—world.

Yet it is one thing for a country to dominate televisions and toys, another the core information technologies. They are the basis for the manufacture, networking and destructive power of advanced weapons systems. More generally, they are often subject to extreme network effects, in which one winner establishes an unassailable position in each market. This means that a country may be squeezed out of vital technologies by foreign rivals pumped up by state support. In the case of China, those rivals answer to an oppressive authoritarian regime that increasingly holds itself up as an alternative to liberal democracy—particularly in its part of Asia. China insists that it wants a win-win world. America has no choice but to see Chinese technology as a means to an unwelcome end.

The question is how to respond. The most important part of the answer is to remember the reasons for America’s success in the 1950s and 1960s. Government programmes, intended to surpass the Soviet Union in space and weapons systems, galvanised investment in education, research and engineering across a broad range of technologies. This ultimately gave rise to Silicon Valley, where it was infused by a spirit of free inquiry, vigorous competition and a healthy capitalist incentive to make money. It was supercharged by an immigration system that welcomed promising minds from every corner of the planet. Sixty years after the Sputnik moment, America needs the same combination of public investment and private enterprise in pursuit of a national project.

Why use a scalpel when a hammer will do?

The other part of the answer is to update national-security safeguards for the realities of China’s potential digital threats. The remit of the Committee on Foreign Investment in the US (CFIUS), a multi-agency body charged with screening deals that affect national security, should be expanded so that minority investments in AI, say, can be scrutinised as well as outright acquisitions. Worries about a supplier of critical components do not have to result in outright bans. Britain found a creative way to mitigate some of its China-related security concerns, by using an evaluation centre with the power to dig right down into every detail of the hardware and software of the systems that Huawei supplies for the telephone network.

Set against these standards, Mr Trump falls short on every count. The Broadcom decision suggests that valid suspicion of Chinese technology is blurring into out-and-out protectionism. Broadcom is not even Chinese; the justification for blocking the deal was that it was likely to invest less in R&D than Qualcomm, letting China seize a lead in setting standards.

Mr Trump has reportedly already rejected one plan for tariffs on China to compensate for forced technology transfer but only because the amounts were too small. Were America to impose duties on Chinese consumer electronics, for example, it would harm its own prosperity without doing anything for national security. An aggressively anti-China tack has the obvious risk of a trade tit-for-tat that would leave the world’s two largest economies both worse off and also more insecure.

Mr Trump’s approach is defined only by what he can do to stifle China, not by what he can do to improve America’s prospects. His record on that score is abysmal. America’s federal-government spending on R&D was 0.6% of GDP in 2015, a third of what it was in 1964. Yet the president’s budget proposal for 2019 includes a 42.3% cut in non-defence discretionary spending by 2028, which is where funding for scientific research sits. He has made it harder for skilled immigrants to get visas to enter America. He and some of his party treat scientific evidence with contempt—specifically the science which warns of the looming threat of climate change. America is right to worry about Chinese tech. But for America to turn its back on the things that made it great is no answer.

Money in the type system where it belongs

Money is a measurable abstraction over wealth. Like the number 7, money has little to no intrinsic value. It is civilization who agrees on a currency's utility and on whether the wealth conveyed by 7 units of a particular monetary currency are enough or not. That is, the value of money is extrinsic. Interestingly, the value of most currencies is measured in terms of the value of other currencies, most of which have no value other than the trust recursively vested in them by civilization. Money, in other words, mostly measures trust. This is less obvious for precious metals or cryptocurrencies because they have an intrinsic value of their own as well, whether civilization embraces them as currency or not. But still, that alone is not enough, civilization will drive their value up or down depending on how much they trust them.

Notwithstanding the value civilization gives to a particular currency, the amount of said currency one owns can't spontaneously increase nor decrease in number. As programmers, we do play a crucial role in ensuring amounts of currency are never made up nor lost. In this article we will explore how we can leverage types, functional programming, and in particular the safe-money Haskell library to ensure that our software deals with monetary values and world currencies as carefully as civilization requires. Mostly, we will be exploring the type system and learning how to reason through types.

Representing exact monetary values in computers is a subtle matter, and even more so is operating with them. Let's investigate some of the alternatives.

Most world currencies allow using fractional numbers to represent some amount of a currency unit. For example, 32.15 USD or 3.64 EUR are valid amounts. In programming languages, the tool we have most readily available for representing fractional numbers are floating point numbers. Unfortunately, they are the wrong tool. Consider the following example where we try to add 0.10 USD and 0.60 USD together. Naturally, we expect 0.70 USD as a result, but instead we get something else:

> 0.10 + 0.60 :: Float
0.70000005

There are some problems with this result. It is not exact, sure, but more importantly: We are making up money! We can't do that. Where are those extra 0.00000005 coming from? Floating points are intended to be an approximation of numeric values belonging to an infinite set in a finite and predictable amount of computer memory. But, of course, not every numeric value in such infinite set will be exactly representable within the same finite amount of memory, so some sacrifices will be necessary at times, and rounding errors will happen. Our ideal 0.7 was one such sacrifice, a value not exactly representable as a floating point number. We got an approximation instead.

Floating point numbers are acceptable for many applications, but not for money. Otherwise, who pays for these rounding errors? Remember, money can't be made up, money can't vanish. A better representation for money is integer numbers. For example, instead of talking about 0.60 units of USD, we could talk about 60 units of “a hundredth of an USD”. That is, we would talk about whole amounts of USD cents instead of fractional amounts of USD dollars. We simply change our unit to be smaller than the dollar. Computers are excellent at working with integer numbers, so, in our previous example 10 plus 60 would have been precisely 70. And indeed, integers are a good representation for monetary values for fiat and crypto currencies to a good extent.

However, integers are still not enough for all our needs. We can identify at least two problems with this approach. The first problem is representing currencies that don't have a “canonical smallest unit” like USD had in the cent. Think of precious metals, for example. What is the smallest unit of gold? Is it a kilo? Is it a gram? Is it a tenth of a gram? Is it a troy ounce? There's no correct answer. For all practical intents and purposes, you can repeatedly split your gold into smaller pieces and still call each of the smmaller pieces “gold”. So, since we can't find a canonical smallest unit for this currency, we can't count our gold as a whole amount of them. Otherwise, we'd be unable to count pieces smaller than any arbitrarily chosen smallest unit. And yes, presumably we could treat the atomic mass of the gold atom as our smallest unit, but that would be rather small and unpractical.

A more subtle problem with integer numbers is representing intermediate values. Say, what is the half of 5 USD cents? It is 2.5 cents, but of course can't be represented as a whole amount of cents. An “alternative” result, 2, is an approximation in which we lose 1 cent. 3 is another approximation in which we make up 1 cent. But we already know that losing or making up money is not acceptable. Yet, values such as “half 5 cents” are important even though it's impossible to represent them using coins or other tangible currency support. In particular, these values are very important when they are the intermediate results of some calculation. Suppose you want to write a function that calculates “ten times half of its input”. That is, f x = 10 * (x / 2). If our input is 5 cents, ideally, we would get 25 cents as a result, which is a perfectly representable whole amount of cents. However, if the intermediate result for x / 2 would have been rounded down or up, we would have gotten either 20 or 30 cents as result, respectively. Generally, this is unacceptable when dealing with money, so integer numbers are not sufficient representation for monetary values for manipulation purposes. What is the answer then?

The correct way of representing monetary values for manipulation is as rational numbers. Rational numbers always give precise results, they don't introduce rounding errors and they don't require us to acknowledge a “smallest representable unit”. For example, going back to our original example: Adding 0.10 USD cents plus 0.60 USD cents is, conceptually, the same as adding one tenth of a dollar plus six tenths of it:

> (1 % 10) + (6 % 10) :: Rational
7 % 10

Note: In Haskell, we use the syntax numerator % denominator to construct a rational number.

As expected, we got a precise value. Moreover, since these are just rational numbers, if we wanted to represent 0.60 cents as the equivalent three fifths of a dollar, that would have been acceptable as well.

> (1 % 10) + (3 % 5) :: Rational
7 % 10

This gives us a lot of flexibility when working with those intermediate values mentioned before. In particular, we can use values such as 25 % 1000, 1 % 40 or (1 % 2) * (5 % 100) to represent our idea of “the half of 5 cents”.

Eventually, when performing a real-world transaction for this rational amount, we will of course need to somehow approximate it to a whole number that is actually representable in our smallest currency unit of choice. This is fine, however, because by then we have already performed all the calculations we needed in a lossless manner, so our approximation will be as close to the ideal value as possible, and at that point we can make a concious decision of whether to round up or down our value if necessary. Additionally, we will see later how we can improve this by requiring that any remainder which doesn't fit in the approximated value be acknowledged. That way, money is never lost.

Something to also keep in mind is that rational numbers are not free. In principle, they could potentially occupy large amounts of memory because internally they are represented as two unbounded integer values. In practice, those integers will usually be small enough that from a memory usage point of view the difference won't be that significant. Performance will likely be worse than when working with floating point numbers or bounded integers. You'll have to benchmark and decide for yourself whether that's more important than precision.

To sum up, we now know what the correct representations for monetary values are. Integer numbers indicating an amount of some chosen “smallest currency unit” suffice when we want to represent amounts that can be evenly divided by said currency unit. On the other hand, if we perform any arithmetic operations on this amount, then rational numbers are a better representation because they will prevent any loss of money in case one of these operations results in an amount not evenly divisible by said smallest currency unit. You might recognize these two representations by other names: The integer representation gives us a discrete view of monetary values in a particular currency unit, whereas the rational representation gives us a dense view of currency amounts, whatever our smallest unit of choice may be. In safe-money, we have two datatypes for representing monetary values, they are called Discrete and Dense.

Let's look at the Dense type first:

newtype Dense (currency :: Symbol)

Even if we don't know what Symbol is, we should still be able to grasp the idea that a Dense monetary value is somehow related to a particular currency, because the word currency shows up as a type parameter. If you haven't seen Symbol in Haskell before, don't worry too much about it: Symbol is like String, but it exists at the type level and not at the term level. For us, this currency type parameter will be things such as "USD", "BTC" or "XAU", and the reason why conveying the name of the currency as a type parameter is so important is because it prevents us from accidentally mixing amounts of different currencies. Later on we will learn more about the Symbols that can be used here.

Let's try and represent 2 USD, for example. Following the recommendation from the safe-money Haddock documentation, we see that fromInteger should do the trick. However, notice that we explicitly need to specify the "USD" type, as otherwise the compiler can't infer it because fromInteger can be used to create Dense values for any currency, not just USD.

> fromInteger 2 :: Dense "USD"
Dense USD 2%1    -- USD 2.00

Notice that in Haskell you don't have to say fromInteger explicitly when you use a literal number. That is, saying 2 is the same as saying fromInteger 2. So, the previous example could have been written as follows.

> 2 :: Dense "USD"
Dense "USD" 2%1    -- USD 2.00

If we want to represent an amount not evenly divisible by the main USD unit (the dollar) such as 3.41 dollars, then fromInteger is not enough. We need to use the fromRational smart constructor instead, as recommended by the Haddock documentation.

> 'fromRational' (341 % 100) :: Dense "USD"
Dense "USD" 341%100     -- USD 3.41

Note: In some cases, when trying to build a Dense value out of a Rational number obtained from untrusted sources, dense would be a better alternative than fromRational. Refer to the Haddocks to understand why, but not now.

And as promised, we can safely operate with Dense monetary values without any loss of precision, even when working with values not evenly divisible by the currency's smallest unit. Dense supports the usual plethora of numeric-like typeclasses: Eq, Ord, Num, Fractional, Real.

> let x = 4 :: Dense "USD"   -- USD 4.00> x + 60
Dense "USD" 64%1             -- USD 64.00> x / 3
Dense "USD" 4%3              -- USD 1.3333333...> (x / 3) * 3 == x
True                         -- No loss of intermediate amounts!

That's it, really. There's not much more to Dense values in terms of operating with them. You can do anything you can do with a Rational number.

Let's go back to currencies for a bit. Here is an example of using the currency type parameter to Dense in order to prevent accidentally mixing different currencies:

> (1 :: Dense "BTC") + (2 :: Dense "JPY")<interactive>:6:35: error:• Couldn't match type ‘"JPY"’ with ‘"BTC"’
      Expected type: Dense "BTC"
        Actual type: Dense "JPY"
      ...

Great, that is exactly what we wanted. However, sometimes, instead of triggering a compiler error, we may want to exchange these different Dense values into a common currency so that they can be used together. For this, we have the aptly named exchange function.

exchange :: ExchangeRate src dst -> Dense src -> Dense dst

To seasoned Haskell developers, the behaviour of this function will be obvious from its type and the names given to the type parameters. This function is an excellent example of why Haskell, and type systems in general, are such powerful tools, so let's explore it a bit. Let's start with the asumption that we want to convert "JPY" into "BTC". Here's a transcript of what might be thought process of a seasoned Haskell developer:

1. The function is called exchange, and seeing how this safe-money library is about money, then quite likely “exchange” here means “exchange one currency for another”. I might be wrong, but sounds like a safe bet.

2. Yes! This must be the correct function, because I see something called ExchangeRate in the type signature, and I know exchange rates convey the idea of how much one currency is worth in another currency, something quite likely necessary when exchanging one currency into another.

3. Ah, it also takes a Dense monetary value as argument, given another Dense as result. But... these Dense values are indexed by different src and dst currencies, so clearly a monetary value in currency src is being converted to another monetary value in currency dst.

4. It looks like src is probably a short for “source” and dst is probably a short for “destination”. So, we convert from some source currency to some destination currency. In our case, src shall be "JPY" and dst will be "BTC".

5. Hmm... The ExchangeRate parameter is also indexed by both src and dst, which presumably means that the exchange rate only works for a particular combination of currencies. It makes sense. I wouldn't want to use the exchange rate for a different currency with my monetary amount.

6. What is a ExchangeRate anyway? Well, the function is pure, so it's certainly not the current market value or something like that. I'm glad this function is pure, by the way, it means I should be able to do something like x + exchange foo y and it will work just fine. This will be easy!

7. Now... how do I build an ExchangeRate "USD" "JPY"? ...

Luckily, this Haskell programmer knew a thing or two about currencies, so there was some prior knowledge about the problem domain that influenced the reasoning progress. However, what if we didn't have that advantage? What if we are trying to solve a problem in a domain we know nothing about? Let's explore this approach a bit as well, because this is a place where Haskell truly shines.

If we recall, we were trying to add two Dense values together. However, these Dense values were indexed by different type parameters, which prevented the addition from compiling. That's the problem we ultimately need to solve, and this is how another seasoned Haskell programmer that knows nothing about currencies might approach it:

1. I have a Dense "JPY" but I need a Dense "BTC". Is there any function Dense a -> Dense b or similar exported by safe-money?

2. It seems the only function that has a similar type is exchange :: ExchangeRate src dst -> Dense src -> Dense dst, so this function looks like a good candidate.

3. I have src ~ "JPY" and dst ~ "BTC", so, how do I obtain an ExchangeRate "JPY" "BTC", whatever that might be?

The subtle difference between this approach and the one above is that here we just don't care about domain specific concepts such as currencies or exchange rates, yet we are able to successfully make effective use of them because they have been encoded in the type system. In fact, let's take this reasoning approach one step further by repeating the steps above obfuscating the names of the things we are dealing with:

1. I have a Foo a but I need a Foo b. Is there any function Foo a -> Foo b or similar exported by this library?

2. It seems the only function that has a similar type is qux :: Bar a b -> Foo a -> Foo b, so this function looks like a good candidate.

3. How do I obtain an Bar a b, whatever that might be?

There's so much being said in just a type. Everything in the type of exchange is of essence. Everything is safe and clear. Programming is suddenly not about reading documentation anymore, nor about navigating executable code, but about learning how to quickly recognize the shapes of the pieces of a puzzle, knowing that as long as two pieces fit together, it will be alright. Haskell is beautiful, it makes programming fun.

We took a small detour, but hopefully we learned or were reminded of a nice thing or two about Haskell. Now it's time to go back to our safe-money library. In particular, as witnessed by our thought transcripts before, we need to learn about ExchangeRate. Luckily, however, there's not much to learn about it. An ExchangeRate src dst, as explained by its documentation, is just a rational number by which we multiply monetary values in a currency src in order to obtain an equivalent monetary value in a different currency src. We construct an ExchangeRate using exchangeRate, which takes the mentioned rational number as an argument.

> let Just jpybtc = exchangeRate (3 % 1000000) :: Maybe (ExchangeRate "JPY" "BTC")> (1 :: Dense "BTC") + exchange jpybtc (2 :: Dense "JPY")
Dense "BTC" 500003%500000

Finally, as expected, if we wanted to use this exchange rate for currencies other than JPY and BTC, the type checker would complain.

> (1 :: Dense "BTC") + exchange jpybtc (2 :: Dense "EUR")<interactive>:17:39: error:• Couldn't match type ‘"EUR"’ with ‘"JPY"’
      Expected type: Dense "JPY"
        Actual type: Dense "EUR"
      ...

We now know how to operate with different currencies in safe-money.

Interestingly, and just as a side note, ExchangeRates can be composed as well. What does this mean? Picture yourself in Japan with gold in your pockets, a plane ticket to Spain, and nobody willing to trade your gold directly for the Euros you'll need in Spain. Do you cancel your trip? No, you sell your gold for Japanese Yen to the local goldsmith, and then exchange those Yen for Euros at one of the fiat currency exchange shops at the airport. Sure, you'll likely lose some money in the process due to the many hoops and the terrible exchange rates, but you will finally get those much needed Euros to buy some tapas and vinos in Spain. What you did in Japan was composing the exchange rate from gold to Yen, with the exchange rate from Yen to Euro. And as ugly as the rate you got might have been, what you did was beautiful. In safe-money, ExchangeRate forms a category where composition using (.) allows you to convert from one currency to another by means of some intermediate currency. In our case, we went from gold (XAU) to Euros (EUR) by means of Japanese Yen (JPY).

> :t goldsmith
goldsmith :: ExchangeRate "XAU" "JPY"> :t fiatshop
fiatshop :: ExchangeRate "JPY" "EUR"> let you = fiatshop . goldsmith> :t you
you :: ExchangeRate "XAU" "EUR"

As mentioned before, we can observe monetary values either through an idealized representation as a members of a dense set, or through a more realistic representation as multiples of some small unit agreed to be aceptable by the parties involved in the monetary transaction.

Let's take the Euro (EUR), for example. This currency has a widely accepted small unit called the Cent, where 100 Cents equal one Euro. So, if we were to represent amounts of this currency intended to be acceptable for transactions, we could represent them as values in a discrete set of multiples of one Cent. That is, we would convey the idea of “6 Euros” using the representation “600 Cents”. In practice, finding ths “small unit” can be tricky. For example, in The Netherlands, even though coins for 1 and 2 Cents are valid and accepted, they are not minted anymore, so for most cash transactions the amount is rounded up or down to the nearest 5 cents multiple, effectively excluding amounts that are not evenly divisible by 5 cents from the discrete set. Let's keep it simple today though. Let's asume 1 cent is an acceptable small unit for EUR.

As we saw before, we use Dense currency to represent monetary amounts of a given currency as rational numbers belonging to a dense set. Perhaps unsurprisingly, we use Discrete currency unit to represent amounts of a given currency in the discrete set of multiples of a chosen “smallest unit”. That is, Discrete is just an integer number. The unit parameter conveys our idea of a “smallest unit”, and like currency, it is also a Symbol. So, for example, if we want our discrete set to be a representation of monetary values that are multiples of one EUR Cent (that is, a discrete set of monetary amounts that can be effectively rendered as some combination of EUR bank notes and coins), then Discrete "EUR" "cent" would be our type. This type is an instance of Num, so we can rely on Haskell's polymorphic literal integers to construct them, and also perform some arithmetic operations on them.

> 1 :: Discrete "EUR" "cent"
Discrete "EUR" 100%1 1           -- 0.01 EUR> 200 :: Discrete "EUR" "cent"
Discrete "EUR" 100%1 200         -- 2.00 EUR> 5394 :: Discrete "EUR" "cent"
Discrete "EUR" 100%1 5394        -- 53.94 EUR> 100 + 70 :: Discrete "EUR" "cent"
Discrete "EUR" 100%1 170         -- 1.70 EUR

If we wanted to use a different “small unit”, we could. For example, you can disregard Cents altogether, and instead make the Euro itself the smallest representable unit.

> 2 :: Discrete "EUR" "euro"
Discrete "EUR" 1%1 2             -- 2.00 EUR

This time, however, with Discrete "EUR" "euro" we can't represen't values such as 2.74 EUR, because insofar as this discrete representation is concerned, there are no values between 2.00 EUR and 3.00 EUR, as witnessed by succ from the Enum class, which gives us the successive value to 2.00 EUR.

> succ (2 :: Discrete "EUR" "euro")
Discrete "EUR" 1%1 3            -- 3.00 EUR

And, of course, we can't accidentally mix Euros and Cents:

>  (2 :: Discrete "EUR" "euro") + (5 :: Discrete "EUR" "cent")<interactive>:32:34: error:• Couldn't match type ‘100’ with ‘1’
      Expected type: Discrete "EUR" "euro"
        Actual type: Discrete "EUR" "cent"
      ...

In order to add Euros and Cents together, much like we did for exchange rates, we need to convert one or both of these values to a common “small unit” that can represent them. In our case it is easy, because we know that we can represent 2 Euros as 200 cents, and we already have Cents as the second operand's small unit, so we just need to find a function to make that conversion.

eurosToCents :: Discrete "EUR" "euro" -> Discrete "EUR" "cent"

Perhaps surprisingly, a quick glance at the safe-money documentation doesn't mention a function resembling this reasonable lossless function. The reason for this becomes more apparent if you reverse the arrow.

centsToEuros :: Discrete "EUR" "cent" -> Discrete "EUR" "euro"

You see, centsToEuros is a lossy function. That is, if you were to convert an integer number of cents to an integer number of euros, you would inevitably lose some precision because 0.72 EUR can't be represented as a whole number of Euros. This is the kind bug that a good type-system, like Haskell's, can outright eradicate.

While it is possible to whitelist source and target “small unit” pairs that are can be converted in a lossless manner, for example by checking for common factors, safe-money prefers to take a general approach that works fine whatever small units you might be dealing with.

It's time to grow up and learn the truth. So far we've been talking about “small units” when referring to things like the Euro or the Cent. However, as the attentive reader might have figured out by now, safe-money doesn't really care about names like Euro or Cent, all it cares about is the ratio between this “small unit” and the value of the currency unit. In safe-money we call this ratio “scale”, and it is simply a number that tells the number of “small units” necessary to add up exactly to the currency's main unit. For example, the Cent has a scale of 100 in the EUR currency, because 100 Cents add up to one EUR. The Euro, on the other hand, has a scale of 1, because the Euro itself is EUR's main currency already.

It is not always this straightforward, though. Something interesting about Discrete is its support for currencies where the number of “small units” in the larger unit is not a multiple of 10. For example, consider a precious metal such as gold. According to the ISO-4217 standard for currency codes that we have been following, the currency symbol for gold is XAU, and one unit of XAU amounts to one troy-ounce of gold.

> 1 :: Discrete "XAU" "troy-ounce"
Discrete "XAU" 1%1 1

But what if we want to represent a smaller amount as a Discrete value? We need to find a smaller “small unit”. The troy-ounce, as most non-metric units of measurement, is terribly confusing and doesn't use a base-10 system for establishing relationships between bigger or smaller related units of mass. It's common to see gold measured in other units such as the gram. However, the relationship between the gram and the troy ounce is not a pretty number: There are 31.103477 grams in a troy-ounce. This means that a Discrete "XAU" "gram" value would be conveying an integer number that, when divided by 31.103477, will amount to a possibly fractional number of troy ounces of gold.

We have learned, however, that representing non-integer numbers involved in monetary amounts is tricky, and that we need to rely on rational numbers instead of floating point approximations. For this reason, scales in safe-money are rational numbers as well. So when we said before that the scale of Cents in EUR is 100, we actually meant 100 % 1, and when we say now 31.103477, we actually mean 31103477 % 1000000. If you have been paying attention to the REPL output in our examples, you probably noticed this fraction showing up everywhere.

> 2 :: Discrete "EUR" "euro"
Discrete "EUR" 1%1 2                -- The scale is: 1 % 1> 130 :: Discrete "EUR" "cent"
Discrete "EUR" 100%1 130            -- The scale is: 100 % 1> 4 :: Discrete "XAU" "gram"
Discrete "XAU" 31103477%1000000 4   -- The scale is: 31103477 % 1000000

Where does the scale come from, though? If you look closer at Discrete, you'll see that it is actually a type synonym for another type called Discrete'.

data Discrete' (currency :: Symbol) (scale :: (Nat, Nat))

The currency type parameter is exactly the same as in Discrete, it allows us to tell currencies apart from each other. And the scale type parameter is, of course, the scale, which as we just mentioned is a rational number which we choose to represent at the type level as a tuple consisting of its numerator and its denominator. Nat, in case you are not familiar with it, is to Natural what Symbol is to String. That is, Nat is a type-level representation of natural numbers. Having the scale at the type level prevents monetary values with different scales to be accidentally used together, as we have seen before.

The Discrete type synonym is defined as follows:

type Discrete (currency :: Symbol) (unit :: Symbol)
  = Discrete' currency (Scale currency unit)

Most of the times we use Discrete instead of Discrete' because it's easier and less error-prone to remember words like “cent” or “gram” than dealing with rational numbers explicitly. But at the end of the day, these two approaches are fully interchangeable, and it's up to the user of safe-money to decide what approach works best for them. Discrete relies on a helper type family called Scale to convert these memorable names to rational numbers.

A type family is basically a function. However, instead of taking terms as arguments and returning terms as a results, it takes types as arguments and returns types as results. The Scale type family works pretty much like this term-level function:

f1 :: String -> String -> Rational
f1 "EUR" "euro"       = 1 % 1
f1 "EUR" "cent"       = 100 % 1
f1 "XAU" "troy-ounce" = 1 % 1
f1 "XAU" "gram"       = 31103477 % 1000000
f1 ...

However, not quite. The problem with f1 above is that the scales for all currencies and units must be defined in the same function, and that's not something we want, since it would mean that the only supported currencies and units are the ones explicitly handled by safe-money in this function. Instead, we want to allow users of this library to add their own currencies and units. We achieve that by making Scale an open type family, meaning we are not forced to make a closed-world assumption of the currencies and units we'll be able to handle. Instead, users will be able to add more as needed, possibly across different modules and packages. In this sense, open type families are more closely related to type-class methods than to normal functions. Scale is defined like this:

type family Scale (currency :: Symbol) (unit :: Symbol) :: (Nat, Nat)

That is, Scale is a type-level function that takes a currency identifier as first argument and a unit identifier as second argument, and returns the unit's scale for that currency as a rational number which we've chosen to represent as a tuple of nominator and denominator. Instances of this type family, akin to methods in instances of a type-class, are defined like so:

type instance Scale "EUR" "euro"       = '(1, 1)
type instance Scale "EUR" "cent"       = '(100, 1)
type instance Scale "XAU" "troy-ounce" = '(1, 1)
type instance Scale "XAU" "gram"       = '(31103477, 1000000)
...

The ' preceding the tuple is there to convince GHC that the kind of this tuple is (Nat, Nat). Otherwise, things like (100, 1) look just like a two element tuple of kind *. But... don't pay much attention to this tick. If you ever get it wrong the compiler will let you know.

So, with this new knowledge, we can easily see that something like Discrete "EUR" "cent" reduces to Discrete "EUR" '(100, 1). The unit name is not important when typechecking a Discrete value, only its scale is. Again, unit is there only as a convenience, since something like “gram” is easier to remember, recognize and get right than (31103477, 1000000). The choice of word “gram” is an arbitrary convention. We could have used “g” or “Gram” or “whatever” instead.

Once we understand scales, understanding how the conversion from a Discrete value to a Dense value happens should be easy. The discrete amount is simply divided by the discrete scale. In safe-money, there is a function for making this conversion readily available:

fromDiscrete :: GoodScale scale => Discrete' currency scale -> Dense currency

Let's try it in the REPL:

> fromDiscrete (1 :: Discrete "EUR" "cent")
Dense "EUR" 1%100> fromDiscrete (1 :: Discrete "EUR" "euro")
Dense "EUR" 1%1> fromDiscrete (1 :: Discrete "XAU" "gram")
Dense "XAU" 1000000%31103477> fromDiscrete (34 :: Discrete "XAU" "gram")
Dense "XAU" 34000000%31103477

The GoodScale constraint in fromDiscrete is there to ensure that we don't accidentally use a non-positive number as either the numerator or numerator of our scale. This is very interesting. We are using the type system again to prevent a negative scale and a denominator value of 0. A negative scale could replace losses with gains and vice-versa, and a denominator value of 0 would result in a runtime error. Here, we are prevent both situations from ever occurring, at compile time.

type GoodScale (scale :: (Nat, Nat))
  = ( CmpNat 0 (Fst scale) ~ 'LT
    , CmpNat 0 (Snd scale) ~ 'LT,
    , KnownNat (Fst scale)
    , KnownNat (Snd scale)
    ) :: Constraint

In our case, GoodScale is implemented as a type family resulting in a Constraint that simply checks whether the number 0 is smaller than first (Fst) and second (Snd) elements of the given tuple. To accomplish this, it checks whether the result of comparing 0 to each element is 'LT. CmpNat, whose kind is Nat -> Nat -> Ordering, is basically the type-level version of compare :: Integer -> Integer -> Ordering. That is, CmpNat x y ~ 'LT is the type level version of compare x y == LT.

We also have KnownNat constraints in GoodScale. These witness that there exists a term-level representation as Integer of the elements in the tuple. This witness will be required when converting our type-level representation of rational numbers to a term-level Rational. And indeed, if we look inside the implementation of fromDiscrete, we will see that it calls a helper function called scale which uses KnownNat to build said Rational number.

scale :: GoodScale scale => proxy scale -> Rational

The proxy there can be anything of kind (Nat, Nat) -> Type, such as Proxy from Data.Proxy, or Discrete currency, as in the implementation of fromDiscrete. Let's try in the REPL.

> scale (Proxy :: Proxy (Scale "EUR" "cent"))
100 % 1> scale (Proxy :: Proxy (Scale "XAU" "gram"))
31103477 % 1000000> scale (1 :: Discrete "XAU" "gram")
31103477 % 1000000> scale (999 :: Discrete "XAU" "gram")
31103477 % 1000000> scale (999 :: Discrete' "XAU" '(31103477, 1000000))
31103477 % 1000000> scale (Proxy :: Proxy '(31103477, 1000000))
31103477 % 1000000

We now know how to convert a Dense monetary value to a Discrete one. Going in the other direction, however, is a bit more complicated. Imagine a situation where we want to convert 4.54 EUR to a whole number of Euro units. We can't readily do so because there would be a reminder of 0.54 EUR that we would never be able to represent as a whole number of Euro units. Simply rounding up or down the amount of round up or down is unacceptable, because we would be either losing or making up money, and that's something we just can't do. What we need to do instead is to let the users decide how they would like to approximate the value to the closest unit multiple as a Discrete value, and force them to acknowledge the existence of a reminder as a Dense value. safe-money exports the four usual rounding functions round, floor, ceiling and truncate, and they all share this behaviour.

round, floor, ceiling, truncate
  :: GoodScale scale
  => Dense currency
  -> (Discrete' currency scale, Dense currency)

Given a Dense monetary value, these functions return a tuple where the first value is the largest possible approximation of said value to a Discrete amount in a the requested scale, rounded up or down depending on the choice of round, floor, ceiling and truncate. The second element in the returned tuple is the difference between the Discrete value and the original Dense value. That is, this value is the exact remainder of the approximation, which may of course be zero in case there's no reminder, or a negative number in case the Discrete value was rounded up.

> -- Sample dense value: EUR -1.24> let x = fromRational (-124 % 100) :: Dense "EUR"> -- Approximating (floor) to cent: EUR -1.24, no reminder> Money.floor x :: (Discrete "EUR" "cent", Dense "EUR")
(Discrete "EUR" (100 % 1) -124,Dense "EUR" 0%1)> -- Approximating (floor) to euro: EUR -2.00, reminder EUR 0.76> Money.floor x :: (Discrete "EUR" "euro", Dense "EUR")
(Discrete "EUR" (1 % 1) -2,Dense "EUR" (19 % 25))

As a quick reminder: floor rounds down, ceiling rounds up, round rounds to the nearest integer, and truncate rounds towards 0 (that is, truncate behaves like floor for positive numbers and like ceiling for negative numbers).

The lesson here is that by always returning the reminder of the approximation, we prevent any accidental monetary losses or gains. It's up to the caller of these rounding functions to decide how to deal with the reminder. For example, the caller could choose to save the remainder until a later time when it can be combined with other reminders, potentially adding up to a multiple of our Discrete scale. Of course, the caller can also chose to simply ignore the remainder, in which case the operation would be lossy. However, if that happens, we can still rest at ease knowing that any losses are due to a concious choice made by the caller, and not due to a faulty design in our library.

The safe-money library provides, optionally, serialization support for its datatypes using the aeson, binary, cereal, store, serialise and xmlbf libraries. Serializing indexed types such as Dense and Discrete is very straightforward. Let's try serializing as JSON using the aeson library, for example.

> Aeson.encode (32 :: Dense "EUR")
"[\"EUR\",32,1]"> Aeson.encode (4 :: Discrete "XAU" "gram")
"[\"XAU\",31103477,1000000,4]"

Dense values are serialized as a 3-element tuple containing the essentials: The currency name, the amount numerator, and the amount denominator. Discrete values are serialized as a 4-element tuple containing the currency name, the scale numerator, the scale denominator, and the integer amount. Deserializing to the same type works as expected.

> Aeson.decode (Aeson.encode (32 :: Dense "EUR")) :: Maybe (Dense "EUR")
Just (Dense "EUR" (32 % 1))> Aeson.decode (Aeson.encode (4 :: Discrete "XAU" "gram")) :: Maybe (Discrete "XAU" "gram")
Just (Discrete "XAU" (31103477 % 1000000) 4)

And trying to deserialize to the wrong currency or unit, of course, fails.

> Aeson.decode (Aeson.encode (32 :: Dense "EUR")) :: Maybe (Dense "JPY")
Nothing> Aeson.decode (Aeson.encode (4 :: Discrete "XAU" "gram")) :: Maybe (Discrete "XAU" "kilogram")
Nothing

So far, everything seems quite straightforward. To serialize or encode is always easy: We start with a value with lot of type information and end up with a value where all or most of that type information is gone. It's easy to see this if we look closer at type of the Aeson.encode function specialized to Discrete'.

discreteToJson :: Discrete' currency scale -> ByteString

Notice that discreteFromJson doesn't exist in safe-copy. It is simply Data.Aeson.decode with a different name.

This function takes a value with a lot of type information, presumably makes use of it internally, and then discards it. Discarding things is easy. However, what if we need to go in the other direction? That is, what if we need to deserialize or decode a value that carries no type information into some type-indexed value? Let's try to imagine what such decoding function could look like.

discreteFromJson :: ByteString -> Maybe (Discrete' currency scale)
discreteFromJson = \bs ->
 -- 1. Check that `bs` is a serialization of a 4-element tuple mentioned above.
 -- 2. Check that the currency encoded in `bs` is the same as `currency`.
 -- 3. Check that the scale encocded in `bs` is the same as `scale`.
 -- 4. Return a `Discrete'` value using the integer amount encoded in `bs`.

That is, the implementation of this function relies on knowning the specific types of currency and scale even before attempting to construct a Discrete' value indexed by them. This function can't work “for all currencys and scales”, because even if it manages to find the string "USD" or "EUR" in bs, there's no way it can put that string, which was obtained during runtime as a term-level value, back in the type-level as an index to Discrete'. And the same is of course true for scale. Essentially, currency and scale are also inputs to this function. Funnily enough, they show up to the right of the last -> in this function signature, which can be a bit confusing at first. Maybe seeing this type written in a different style can help.

discreteFromJson
  :: Proxy currency
  -> Proxy scale
  -> ByteString
  -> Maybe (Discrete' currency scale)

This new type signature is essentially the same as the previous one. Those two new Proxy arguments are redundant, but perhaps they help us appreciate how currency and scale are essentially types provided as inputs to this function.

So, what do we do if get a ByteString supposedly representing someDiscrete' value, yet we are not sure in what currency and scale that might be? If our set of acceptable currencies and scales is small enough, we can simply use the Alternative support in Maybe in order to try decoding as different types in order. Sometimes, however, we may actually want to be able to deal with any Discrete' monetary value that comes our way without explicitly observing the related currency or scale. Perhaps all we want to do is show the Discrete' value, and for that we don't really need to known what the currency and scale type indexes are, all we need to know is that there is a Show instance for them. However, as things currently stand, in order to deserialize into a Discrete, we are always forced to choose the currency and scale indexes anyway. Have we reached an impasse? Well, not exactly.

In Haskell, it's the caller who must pick the types when there's some polymorphic type involved in a call. For example, consider the type of the dense function, used to construct a Dense value.

dense :: Rational -> Maybe (Dense currency)

In dense, currency is a polymorphic type that must be chosen by the caller. See what happens in the REPL if we try to use dense without specifying the currency type.

> dense (4 % 7)<interactive>:16:1: error:• No instance for (GHC.TypeLits.KnownSymbol currency0)
        arising from a use of ‘print’
    • In a stmt of an interactive GHCi command: print it

The error is saying that since currency is not known, the KnownSymbol instance necessary to render this Dense value using print could not be found for it, which is a convoluted way of saying that currency must be specified. A similar kind of error would happen if we tried to, say, deserialize some raw JSON string without specifying the target type. If we ask for a specific currency, then everything works.

> dense (4 % 7) :: Maybe (Dense "EUR")
Just (Dense "EUR" (4 % 7))

In Haskell there is a way to “workaround” this behavior so that it is not the caller, but the callee, who can chose the currency, and we can embrace this in order to allow Dense values for any currency to be deserialized from a raw JSON input, without having to enumerate the possible currencies anywhere in the type system. The solution is RankNTypes, and it requires a different arrangement of our deserializing function. So, instead of having a function denseFromJson :: ByteString -> Maybe (Dense currency), we would have something like this, which is only possible when the RankNTypes extension is enabled.

withDenseJSON
  :: ByteString    -- ^ Raw JSON
  -> (forall currency. Dense currency -> r)
  -> Maybe r

Let's carefuly try and understand what withDenseJSON means. This function returns a Maybe because it's possible that parsing doesn't succeed, and Maybe is a cheap way for us to communicate that. Leaving this small detail out of the way, let's proceed to taking a serious look at the second argument to this function.

Unfortunately, the forall keyword is used for many different purposes in Haskell, so at times its meaning can be confusing. This forall here is quantifying currency so that it exists only within this limitted scope delimited by the surrounding parentheses. This means that if we were to refer to currency outside these parentheses, it would be a different type. We can see an extreme example of this:

f :: a -> (forall a. a -> b) -> (a, b)
f oa g = (oa, g false)

This function takes an a, some function for obtaining a b, and returns that same a along with a b. This “outer a”, however, is different from the a inside the parentheses. We can see a proof of that if we partially apply f in a way that would force the outer a to be fixed to a concrete type.

> :t f ()
f () :: (forall a. a -> b) -> ((), b)

The outer a is now gone, it has been replaced with a specific type (). The inner a remains, however, since was is different from the outer a. What is happening is that the scopes over which these two as are quantified are different. Or, more precisely, the outer a is available within the scope of the second argument to f, but not the other way around. That is, the inner a can't share the same scope as the outer a. Or in more precise terms, the inner a has a higher rank than the outer a, thus the extension name RankNTypes. In our definition of f we deliberately used the same name a for two different type variables. While this works, much like term-level variable name shadowing, it can be confusing, and GHC will warn us about it. We deliberately chose to use name shadowing to make a point, but in practice you should probably use different variable names for the sake of your users.

<interactive>:89:31: warning: [-Wname-shadowing]
    This binding for ‘a’ shadows the existing binding
      bound at <interactive>:89:13

We used False, of type Bool, somewhere in our definition of f. However, nowhere in f's type we can see Bool. What is happening? Well, this is the crux of RankNTypes. Do you remember how we said earlier that when faced with a polymorphic type, the outermost caller that first encounters that polymorphism is the one who must chose what the concrete type should be? Well, this is exactly what is happening here. The important detail, however, is that our higher-ranked a is first introduced within the implementation of the callee, so our callee becomes the caller that needs to choose a concrete type for a. And, perhaps surprisingly, it can chose any type it wants. Here we chose Bool because one of its constructors was readily available, but we could have used any other type.

With this new understanding, let's go back to withDenseJSON and assume that we have some JSON blob that represents the idea of some Dense monetary value.

> withDenseJSON myJsonBlob toRational
Just (3 % 1)

Great! We could convert the Dense value to a Rational even though we don't know what currency is inside myJsonBlob. But... seeing that the second argument to withDenseJSON is just a forall currency. Dense currency -> r, and that the whole withDenseJSON expression is expected to return that r, we can just pass id so that the whole expression returns said Dense currency, right? No, we can't.

> withDenseJSON myJsonBlob id<interactive>:1:21: error:• Couldn't match type ‘r’ with ‘Dense currency’
        because type variable ‘currency’ would escape its scope
      This (rigid, skolem) type variable is bound by
        a type expected by the context:
          Dense currency -> r
        at <interactive>:1:1-22
      Expected type: Dense currency -> r
        Actual type: r -> r• In the second argument of ‘withDenseJSON’, namely ‘id’
      In the expression: withDenseJSON "foo" id

The type-checker quite clearly tells us that r can't be Dense currency because currency would escape its scope. And we should have been expecting this. As we discussed before, the purpose of functions using higher-rank types is to be able to introduce new types in an inner scope, smaller than the scope of the function's caller. That is, currency is not defined in the outer scope where r exists. This shouldn't be surprising, since it is the same as how lexical scoping rules work for term-level values.

Note: Techically, there is a way for currency to escape this inner scope, but it requires knowing beforehand which currency you are expecting, as well as runtime type-checking, which will ofcourse fail if we don't get the currency we are expecting. We won't be exploring this.

Let's try something else. Let's try using show on our deserialized Dense value, whatever its currency might be.

> :t withDenseJSON myJsonBlob show<interactive>:1:21: error:• No instance for (KnownSymbol currency)
        arising from a use of ‘show’
      Possible fix:
        add (KnownSymbol currency) to the context of
          a type expected by the context:
            Dense currency -> String• In the second argument of ‘withDenseJSON’, namely ‘show’
      In the expression: withDenseJSON "foo" show

The error is actually quite clear. We are applying show in the second argument of withDenseJSON, and there is no visible KnownSymbol instance for currency there. Where is that KnownSymbol requirement coming from, though? Well, we know that we are trying to show some Dense currency value, so the type of our show function would be Dense currency -> String. However, if we go to the Haddocks and search for this Show instance, we will see that there is an additional superclass to it.

instance KnownSymbol currency => Show (Dense currency) where
   ...

We don't really need to go to the Haddocks for this, but sometimes it is just easier. We could have learned about this superclass by explicitly typing show in the REPL:

> :t show :: Dense currency -> String<interactive>:1:1: error:• No instance for (KnownSymbol currency)
        arising from a use of ‘show’
      Possible fix:
        add (KnownSymbol currency) to the context of
          an expression type signature:
            Dense currency -> String• In the expression: show :: Dense currency -> String

The error is pretty much the same one we got before, which suggests we are on the right track to solving this problem. Let's explicitly add that constraint to the type signature.

> :t show :: KnownSymbol currency => Dense currency -> String

show :: KnownSymbol currency => Dense currency -> String
  :: KnownSymbol currency => Dense currency -> String

So, the issue is that much like in our very first encounter with RankNTypes, the function we are passing as a second argument to withDenseJSON doesn't include this KnownSymbol constraint, so everything depending on it fails.

Unfortunately, when you are free to pick any type like currency in withDenseJSON, or like the higher ranked a in our previous example f, the function becomes rather useless. The problem is that the function that we pass as a second argument to f knows absolutely nothing about the higher-rank a, meaning it can't do anything with it. We want something better. In our example, we want to be able to at least show the pased in higher-rank value. That's easy enough. Mostly, we just need to add the constraint to be satisfied by a inside the parentheses that are typing the passed in function.

h :: (forall a. Show a => a -> b) -> b
h g = g False> h show :: String
"False"> h (length . show) :: Int
5> h (head . show) :: Char
'F'

This new h function uses RankNTypes to allow the callee to take over the usual role of the caller of picking up a. This a, contrary the higher-rank a in to our previous example f, is not just any a. Instead, it must have Show instance. Applying our g function to False works because Bool is an instance of Show. However, if we tried to apply g to something like id :: x -> x, it would fail because there is no Show instance for x -> x.

h' :: (forall a. Show a => a -> b) -> b
h' g = g id<interactive>:14:7: error:• No instance for (Show (x -> x)) arising from a use of ‘g’
        (maybe you haven't applied a function to enough arguments?)
    • In the expression: g id
      In an equation for ‘h'’: h' g = g id

What this implies is that a witness that a satisfies Show is necessary at h's definition site, not at its call site. This shift of responsibility is very powerful. Suddenly we are able to show things even if we can't produce the necessary Show witness at the call site, by relying on a higher-rank function to provide it for us. And with this new understanding, let's go back to our monetary parsing problems. We know that in order for the show function on a Dense currency to work, we need a KnownSymbol currency constraint satisfied as well. Let's change the type signature of withDenseJSON just like we did in h, so that this witness is available inside the higher rank function.

withDenseJSON
  :: ByteString    -- ^ Raw JSON
  -> (forall currency. KnownSymbol currency => Dense currency -> r)
  -> Maybe r

The KnownSymbol constraint is now satisfied, so Show should be automatically satisfied as well.

> withDenseJSON myJsonBlob show
Dense "EUR" (4 % 7)

So far we've talked about JSON. But what about other serialization formats? What about parsing some XML format, or perhaps some binary format? Do we have to implement withDenseXML, withDenseBinary, etc.? Hopefully not, since not only their implementations will be quite redundant, but also we can't really leverage many of the already existing tools like FromJSON, binary's Binary, or even Read, since we are always forced to use this RankNTypes approach. So... let's try and understand a bit more what we mean when we talk about deserialization to see if we can find a common approach to all of these formats.

Supose we have these four raw representations of the value fromRational (4 % 5) :: Dense "EUR" as ByteStrings.

> s1 = "[\"EUR\",4,5]"       -- ^ The JSON representation we saw before.> s2 = "4/5 EUR"             -- ^ A supposedly user-friendly representation.> s3 = "\x03\x01EUR\x04\x05" -- ^ A binary representation.> s4 = "EUR\xff\x04\x05"     -- ^ Another binary representation.

These four ByteString representations are term-level values, and in them we encode the essentials that convey the idea of fromRational (4 % 5) :: Dense "EUR". That is, they mention the name of the currency "EUR", the numerator 4 of the rational number representing the monetary amount, and its denominator 5. In other words, they are isomorphic to a (Rational, String) tuple where the first element is the monetary amount as a rational number, and the second element is the currency identifier. This is the same data conveyed by a Dense value, if we consider that its rational number representing the monetary amount at the term level is always accompanied by a currency identifier at the type level. However, therein lies our problem: The currency identifier in Dense only exists at the type level, making it impossible for us to implement a function going from that raw ByteString to some arbitrary Dense currency, as this would imply that currency is a type dependent on a some data present in said ByteString. We already encountered a similar situation in discreteFromJson and we know it doesn't work without RankNTypes. However, we are deliberately trying to avoid having a RankNTypes-based solution for each of our four ByteString representations. We need a different approach.

Let's take a step back and recall what we are trying to do: We have many possible ByteString representations for a same Dense value of an unknown currency—possibly an infinite number of them—and we want to be able to write parsers for these raw representations without having to do the RankNTypes dance every time, which we know we need to do every time we go from a “less typed” representation to a “more typed” representation. That is, every time we go from a representation where the currency identifier is not in the types, to one where it is. Can you see where this is going?

What we need to do is reduce the number of times we go from a “less typed” to a “more typed” representation to just one, so that we only need to worry about RankNTypes once. How do we accomplish this? Well, we simply acknowledge that the tuple we briefly mentioned before is a perfectly acceptable term-level representation of all the data conveyed by both the Dense datatype, and all of the raw representations we care about, and as such we can treat it as some canonical “less typed” intermediate representation between them. The idea is that all of our raw ByteString representations will be parsed into this intermediate representation, which should be very straightforward because we won't need to concern ourselves with coming up with currency information at the type level. Once we have this intermediate value, we can simply rely on a unique function to do the RankNTypes work. In safe-money, we have this intermediate type and the relevant RankNTypes function out-of-the-box. They are called SomeDense and withSomeDense. Of course, SomeDense is not just a tuple, as this makes it easier to give specific instances for it, such as FromJSON. And more generally, because is always a good idea to introduce new types for any abstraction that can be conceptually identified as unique, so that we don't accidentally mix them with different yet structurally compatible abstractions, while improving type inference and error messages at the same time.

Of course, we can only deserialize raw representations that have been correctly serialized before, so SomeDense provides instances for this such as ToJSON. Serialization, however, is different than deserialization in the sense that when we serialize we are going from a “more typed” value to a “less typed” value. Exactly the opposite than in deserialization. In our case, it means that if we were to start with a value like fromRational (4 % 5) :: Dense "EUR", then we would always have that currency identifier available at the type level as input, and there would be no reason for us to worry about things like RankNTypes. This implies that whereas it is not possible for us to provide a FromJSON instance for Dense that works for all currencies, it is perfectly possible to write a ToJSON instance that works for all of them. In turn, this suggests that our intermediate representation SomeDense is not really necessary for serialization. And indeed, it is not. However, for completeness, safe-money always provides serialization and deserialization support for both Dense and SomeDense values that are compatible among them. That is, a serialized Dense value can be correctly deserialized both as both SomeDense and Dense, although in the latter case you would need to know the currency identifier beforehand. The same is true for the serialization of SomeDense, which can also be deserialized as both SomeDense or Dense.

Similarly, much like for Dense monetary values we have SomeDense, for Discrete values we have SomeDiscrete, with the small difference that SomeDiscrete hides two type level arguments currency and scale, not just one. And we also have SomeExchangeRate which hides the two type parameters to ExchangeRate.

What is it about this currency identifier that complicates our Haskell so much? It has to do with the position where we want to use Dense in function calls, a topic often overlooked yet important in order for us to develop a stronger understanding and intuition of how things compose together. Let's take a diversion to explore this topic. Consider the following hypotetical function.

f1 :: a -> b

What can we say about a and b without knowing the implementation of f1? Well, for one, it seems that this function can't possibly be implemented because it is simply not possible to create a b out of thin air since we don't really know what b is supposed to be, so how would we produce one? Would something change if we were to change b to be some concrete type?

f2 :: a -> Int

Certainly. Now we can at least imagine f2 being implemented. Here are two possible implementations.

-- | One possible implementation
f2 :: a -> Int
f2 _ = 5

-- | Another possible implementation
f2 :: a -> Int
f2 _ = 895

Although perfectly valid, these are rather silly functions. What we are seeing in these functions is the position of our original b argument come into play. b is said to be a function argument in positive position, which for practical purposes is mostly a fancy way of saying that a value of type b is being produced within the function. If we are producing a value of type b, however, we as producers must now what b is in order to produce it. This is why why we end up being forced to pick some arbitrary integer number. The positive position of b, and the fact that we don't already have some value of type b at hand, force us to make a decision and pick one, which necessarily makes the type of b less polymorphic.

There are also function arguments in negative position. These are the arguments consumed by the function. That is, they are taken as input somehow. In our example, a is one such argument in negative position. We chose not to do anything with a in our example, but this is besides the point. We might as well had done something with it.

It is of course possible to have a same type ocuppy both a positive position and a negative position at the same time. The simplest example of this is the identity function, whose type a -> a says the first a is in negative position while the second a is in positive position. The fact that the way id“produces” its positive a is by just reusing the negative a that was given as input is not problematic. All that matters is that id can obtain some a to provide as output somehow, and then our positive position requirements will be satisfied.

Identifying the position of function arguments is not as straightforward as it may seem at first, however. Consider the following example.

f3 :: (a -> b) -> a -> b

In this example, we see that a is received as an input in the a -> b function, and also as the second argument to f3, which suggests that a appears in negative position twice. Also, we see that b is an output of both the a -> b function, as well as an output of the whole f3 expression, which suggests b appears in positive position twice. However, that's not entirely correct.

When we talk about argument positions in a particular function, we need to look at said function's arguments as a whole, and not individually. It's not sufficient to analyse the a -> b and a arguments individually, we must also analyse all of (a -> b) -> a -> b at the same time. From this point of view, we can see that the obvious inputs to f3 are its arguments a -> b and a. That is, these two arguments are being consumed by f3, which means these are arguments in negative position. Similarly, the obvious output of f3 is the b that shows up as its return type. This particular value of b is being produced by f3, which implies that this b is in positive position. Indeed, these are all truths. And moreover, our occurrences of a and b here as negative and positive arguments, respectively, agree with our previous assessment of the positions of a and b in a -> b individually. So, are we done? Well, no, this is not the full story. The best way to understand why is to look at the usage of a -> b in an implementation of f3—which, by the way, due to parametricity, happens to be the only possible implementation of f2.

f3 :: (a -> b) -> a -> b
f3 g x = g x

Isn't it true, when looking at this code, that we are calling g within the implementation of f3? And isn't it true that in order to call a function a caller must be able to produce all of the function inputs. Yes, it is all true. So, even while it's true that when assessing a -> b individually, a is in negative position, and that when assessing the type of f3, a is in negative position. It is also true that from f3's implementation point of view, an a must somehow be produced in order to be able to call g. Hence, the a in a -> b is in positive position. This is fine, a same type can be in both positive and negative positions at the same time. Similarly, while b appears in a positive position, it also appears negative position. Why? Well, isn't it true that the implementation of f3 is able to consume the b produced by g x? Indeed. This implies that b is actually in negative position here. The fact that as soon as we obtain this b we simply reuse it as f3's final value is incidental.

Enough with the contrived abstract examples. Why do we care about all of this? Well, observe the similarities between f3 and withSomeDense.

f3            :: (a                               -> b) -> a         -> b
withSomeDense :: (forall currency. Dense currency -> b) -> SomeDense -> b

If you keep in mind that Dense currency and SomeDense are simply two different ways if representing the same thing, then the similarities are even more obvious. The dense monetary value is our a. However, as we learned before when we introduced RankNTypes, we need to quantify the currency variable so that it exists only within the scope of the given function, and finally we can explain why in simple words: Dense currency appears in positive position, which forces the implementation of withSomeDense to provide a value of this type after having chosen a currency. However, we can't, since we won't learn about our currency until runtime when we inspect what's inside the given SomeDense. So, with RankNTypes, we can defer that decision until later while still satisfying the positive position requirements of this argument, provided there is a function able to handle any Dense currency it may be given.

And with this, we will stop our tour full of distractions. Hopefully we learned a thing or two about Haskell and using the type system as a reasoning tool. And if not, perhaps at least we grew curious about this fascinating tool that allows us to stop trusting and start proving.

Regarding safe-money, you can start using it today. It is available in Hackage and GitHub, with plenty of documentation and tests. Production ready, if you fancy that term.

By Renzo Carbonara. First published in December 2017.
This work is licensed under a Creative Commons Attribution 4.0 International License.

README.org

PINNED:  Index


               +-Python              Game development-+-Game design
               +-Programming books           |
   Programming-+-Emacs                       |
         |                                   |
         +-----------------+-----------------+
                           |
                           ▽
                    Game programming ←→ Computer games

Game Maker  Unity

--- Resources ---------------------------------

• https://en.wikipedia.org/wiki/Game_programming
• Passing Through Ghosts in Pac-Man
• In-House Engine Development: Technical Tips

--- Text --------------------------------------

Game programming is the art of programming computer games...

(use-package org-brain :ensuret:init
  (setq org-brain-path "directory/path/where-i-want-org-brain");; For Evil users
  (eval-after-load'evil
    (evil-set-initial-state 'org-brain-visualize-mode'emacs)):config
  (setq org-id-track-globally t)
  (setq org-id-locations-file "~/.emacs.d/.org-id-locations")
  (push '("b""Brain" plain (function org-brain-goto-end)"* %i%?":empty-lines1)
        org-capture-templates)
  (setq org-brain-visualize-default-choices 'all)
  (setq org-brain-title-max-length 12))

* org-brain.el
  :PROPERTIES:
  :ID:       7cf9563e-24e7-4823-8742-64b7fcc5cafb
  :END:

  #+BEGIN_description
  Package for concept mapping using org-mode.
  #+END_description

  ...

(defunorg-brain-cliplink-resource ()"Add a URL from the clipboard as an org-brain resource.Suggest the URL title as a description for resource."
  (interactive)
  (let ((url (org-cliplink-clipboard-content)))
    (org-brain-add-resource
     url
     (org-cliplink-retrieve-title-synchronously url)t)))

(define-key org-brain-visualize-mode-map (kbd"L") #'org-brain-cliplink-resource)

(define-key org-brain-visualize-mode-map (kbd"C-l") #'link-hint-open-link)

(defunaa2u-buffer ()
  (aa2u (point-min) (point-max)))

(add-hook'org-brain-after-visualize-hook#'aa2u-buffer)

(defunorg-brain-insert-resource-icon (link)"Insert an icon, based on content of org-mode LINK."
  (insert (format"%s"
                  (cond ((string-prefix-p"http"link)
                         (cond ((string-match"wikipedia\\.org"link)
                                (all-the-icons-faicon "wikipedia-w"))
                               ((string-match"github\\.com"link)
                                (all-the-icons-octicon "mark-github"))
                               ((string-match"vimeo\\.com"link)
                                (all-the-icons-faicon "vimeo"))
                               ((string-match"youtube\\.com"link)
                                (all-the-icons-faicon "youtube"))
                               (t
                                (all-the-icons-faicon "globe"))))
                        ((string-prefix-p"brain:"link)
                         (all-the-icons-fileicon "brain"))
                        (t
                         (all-the-icons-icon-for-file link))))))

(add-hook'org-brain-after-resource-button-functions#'org-brain-insert-resource-icon)

(defunorg-brain-deft ()"Use `deft' for files in `org-brain-path'."
  (interactive)
  (let ((deft-directory org-brain-path)
        (deft-recursive t)
        (deft-extensions '("org")))
    (deft)))

(defunhelm-org-rifle-brain ()"Rifle files in `org-brain-path'."
  (interactive)
  (helm-org-rifle-directories (list org-brain-path)))

Since the public revelation of the Meltdown and Spectre CPU vulnerabilities early this year, Intel has spent virtually the entire time in a reactionary mode, starting from the moment the vulnerabilities were revealed ahead of schedule. Since then the company has been making progress, albeit not without some significant steps backwards such as faulty microcode updates. However in recent weeks the company finally seems to be turning a corner on their most pressing issues, and this morning is releasing a more forward-looking update to their security issues.

Jumping straight to what AnandTech readers will consider the biggest news, Intel is finally talking a bit about future hardware. Intel is announcing that they have developed hardware fixes for both the Meltdown and Spectre v2 vulnerabilities, which in turn will be implemented into future processors. Both the next version of Intel’s Xeon server/HEDT platform – Cascade Lake – as well as new 8^th gen Core processors set to ship in the second half of this year will include the mitigations.

For those not up to date with their Intel codenames, Cascade Lake is the 14nm refresh of Intel’s current Skylake-E/X family. Little official information is available about Cascade Lake, but importantly for datacenter vendors, this lays out a clear timetable for when they can expect to have access to Meltdown and Spectre-hardened silicon for use in new virtual machine servers. Given that virtual machine hosts were among those at the greatest risk here – and more impacted by the performance regressions of the software Meltdown mitigations – this is understandably most crucial market for Intel to address.

Meanwhile for updating Intel’s consumer chips, this is a bit more nebulous. While Intel hasn’t shared the complete text of their announcement with us ahead of press time, their specific wording is that the changes will be included in 8^th gen Core processors “expected to ship in the second half of 2018.” Intel hasn’t said what processor family these are (e.g. Cannon Lake?), or for that matter whether these are even going to be traditional consumer chips or just the Core HEDT releases of Cascade Lake. So there is a lot of uncertainty here over just what this will entail. In the interim we have reached out to Intel about how consumers will be able to identify post-mitigation chips, and while we’re still waiting on a more complete response, Intel has told us that they want to be transparent about the matter.

As for the hardware changes themselves, it’s important to note that Intel’s changes only mitigate Meltdown (what Intel calls “variant 3”) and Spectre variant 2. In both cases the company has mitigated the vulnerabilities through a new partitioning system that improves both process and privilege-level separation, going with a “protective walls” analogy.

Unfortunately these hardware changes won’t mitigate Spectre variant 1. And admittedly, I haven’t been expecting Intel (or anyone else) to figure that one out in 2018. The best mitigations for Spectre v2 will remain developer-focused software techniques such as retpoline.

The catch is that the more worrying risk with Spectre has always been the v1 variant, as the attack works against rather fundamental principles of speculative out-of-order execution. Which has been why the initial research on the vulnerability class noted that researchers weren’t sure they completely understood the full depth of the issue at the time. And indeed, it seems like the industry as a whole is still trying to fully understand the matter. The one silver lining here is that Spectre v1 isn’t a privilege escalation attack, so it is believed that it can only be used against same-level processes. Which can still be used for plenty of naughtiness with user data in other user-level applications, but can’t reach into more secure processes.

Moving on, for Intel’s current processors the company has updated their guidance for releasing the mitigation microcode updates. As of last week, the company has released production microcode updates for all of their products released in the last 5 years. In fact on the Core architecture side it goes even farther than that; Intel has now released microcode updates for all 2^nd gen Core (Sandy Bridge) and newer processors, including their Xeon and HEDT variants. There are some outstanding questions here on how these updates will be delivered, as it seems unlikely that manufacturers will release BIOS updates for motherboards going back quite that far, but judging from how Intel and Microsoft have cooperated thus far, I’d expect to see these microcode updates also released to Windows Update in some fashion.

Finally, Intel will also be going even further back with their microcode updates. Their latest schedule calls for processors as old as the Core 2 lineup to get updates, including the 1^st gen Core processors (Nehalem/Gulftown/Westmere/Lynnfield/Clarksfield/Bloomfield/Arrandale/Clarkdale), and the 45nm Core 2 processors (Penryn/Yorkfield/Wolfdale/Hapertown), which would cover most Intel processors going back to late 2007 or so. It’s worth noting that the 65nm Core 2 processors (Conroe, etc) are not on this list, but then the latter Core 2 processors weren’t on the list either at one point.