Shopping isn't always as easy as it should be. When was the last time you needed to pick up something from the store but didn’t have the time to make the trip? Or you went to the store only to realize they didn’t have the brand you wanted? Wouldn’t it be nice if you could get what you want, however you want, from the stores where you already shop? We launched Google Express and shopping on the Google Assistant to do just that: make it faster and easier for you to shop your stores like Costco, Target and now Walmart.

Welcome Walmart

We’re entering an exciting partnership with Walmart to bring you hundreds of thousands of products at Walmart’s Every Day Low Prices—everything from laundry detergent to Legos—that you can buy through voice with your Assistant on Google Home or on the Google Express website or app.

If you’re an existing Walmart customer, you can choose to link your Walmart account to Google and receive personalized shopping results based on your online and in-store Walmart purchases.  For example, if you order Tide PODS or Gatorade, your Google Assistant will let you know which size and type you previously ordered from Walmart, making it easy for you to buy the right product again.

In 1977, four recent MIT graduates who’d met at MIT’s Laboratory for Computer Science used the lab’s PDP-10 mainframe to develop a computer game that captivated the world. Called Zork, which was a nonsense word then popular on campus, their creation would become one of the most influential computer games in the medium’s half-century-long history.

The text-based adventure challenged players to navigate a byzantine underground world full of caves and rivers as they battled gnomes, a troll, and a Cyclops to collect such treasures as a jewel-encrusted egg and a silver chalice.

During its 1980s heyday, commercial versions of Zork released for personal computers sold more than 800,000 copies. Today, unofficial versions of the game can be played online, on smartphones, and on Amazon Echo devices, and Zork is inspiring young technologists well beyond the gaming field.

It’s an impressive legacy for a project described by its developers as a hobby, a lark, and a “good hack.” Here’s the story of Zork’s creation, as recounted by its four inventors—and a look at its ongoing impact.

Tim Anderson, Marc Blank, Bruce Daniels, and Dave Lebling—who between them earned seven MIT degrees in electrical engineering and computer science, political science, and biology—bonded over their interest in computer games, then in their infancy, as they worked or consulted for the Laboratory for Computer Science’s Dynamic Modeling Group. By day, all of them but Blank (who was in medical school) developed software for the U.S. Department of Defense’s Advanced Research Projects Agency (DARPA), which funded projects at MIT. On nights and weekends, they used their coding skills—and mainframe access—to work on Zork.

In early 1977, a text-only game called Colossal Cave Adventure—originally written by MIT grad Will Crowther—was tweaked and distributed over the ARPANET by a Stanford graduate student. “The four of us spent a lot of time trying to solve Adventure,” says Lebling. “And when we finally did, we said, ‘That was pretty good, but we could do a better job.’”

By June, they’d devised many of Zork’s core features and building blocks, including a word parser that took words the players typed and translated them into commands the game could process and respond to, propelling the story forward. The parser, which the group continued to fine-tune, allowed Zork to understand far more words than previous games, including adjectives, conjunctions, prepositions, and complex verbs. That meant Zork could support intricate puzzles, such as one that let players obtain a key by sliding paper under a door, pushing the key out of the lock so it would drop onto the paper, and retrieving the paper. The parser also let players input sentences like “Take all but rug” to scoop up multiple treasures, rather than making them type “Take [object]” over and over.

Vibrant, witty writing set Zork apart. It had no graphics, but lines like “Phosphorescent mosses, fed by a trickle of water from some unseen source above, make [the crystal grotto] glow and sparkle with every color of the rainbow” helped players envision the “Great Underground Empire” they were exploring as they brandished such weapons as glowing “Elvish swords.” “We played with language just like we played with computers,” says Daniels. Wordplay also cropped up in irreverent character names such as “Lord Dimwit Flathead the Excessive” and “The Wizard of Frobozz.”

Within weeks of its creation, Zork’s clever writing and inventive puzzles attracted players from across the U.S. and England. “The MIT machines were a nerd magnet for kids who had access to the ARPANET,” says Anderson. “They would see someone running something called Zork, rummage around in the MIT file system, find and play the game, and tell their friends.” The MIT mainframe operating system (called ITS) let Zork’s creators remotely watch users type in real time, which revealed common mistakes. “If we found a lot of people using a word the game didn’t support, we would add it as a synonym,” says Daniels.

The four kept refining and expanding Zork until February 1979. A few months later, three of them, plus seven other Dynamic Modeling Group members, founded the software company Infocom. Its first product: a modified version of Zork, split into three parts, released over three years, to fit PCs’ limited memory size and processing power.

Nearly 40 years later, those PC games, which ran on everything from the Apple II to the Commodore 64 in their 1980s heyday, are available online—and still inspire technologists. Ben Brown, founder and CEO of Howdy.ai, says Zork helped him design AI-powered chatbots. “Zork is a narrative, but embedded within it are clues about how the user can interact with and affect the story,” he says. “It’s a good model for how chatbots should teach users how to respond to and use commands without being heavy-handed and repetitive.” For example, the line “You are in a dark and quite creepy crawlway with passages leaving to the north, east, south, and southwest” hints to players that they must choose a direction to move, but it doesn’t make those instructions as explicit as actually telling them, “Type ‘north,’ ‘east,’ ‘south,’ or ‘southwest.’” Brown’s chatbot, Howdy, operates similarly, using bold and highlighted fonts to draw attention to keywords, like “check in,” and “schedule,” that people can use to communicate with the bot.

Jessica Brillhart, a filmmaker who creates virtual-reality videos, also cites Zork as an influence: “It provides a great way to script immersive experiences and shows how to craft a full universe for people to explore.”

README.md

The United States research fleet is made up of dozens of vessels, large and small, that take scientists out to sea to collect data on the ocean and climate and marine life, and most of them look like regular old boats or barges. But not the Research Platform FLIP, short for Floating Instrument Platform. It looks like the bow of a regular boat, chopped off and tacked onto end of a submarine. FLIP lives up to its name; it’s designed to do what no other boat does without sinking—flip a full 90 degrees by submerging 300 feet of its hull and leaving just 55 feet of bow above the water. The result is a quiet, stable platform for precise scientific measurements at sea.

This unusual purpose affects every part of the boat’s design, including the quirky and cramped living quarters, which must work for life in two orientations. Despite this, or perhaps because of it, FLIP is a favorite among the researchers who spend weeks aboard. “It was built in the decade that people were trying to reach the Moon and so thinking big was on everyone’s mind,” says Robert Pinkel. Pinkel is a professor emeritus at Scripps Institution of Oceanography at the University of California, San Diego, and has been using FLIP for research nearly as long as it has been in service. The platform, which is technically known as a “spar buoy,” was designed and constructed in 1962 to conduct research for the U.S. Navy on how ocean conditions affect acoustics.

The ship’s creators likely didn’t expect FLIP to be so popular. The original designs were built on the assumption that scientists would board the platform each day to do research, then return to a “mothership” at night to sleep. “Very quickly, it became clear that nobody wanted to go back at night because FLIP was not only more comfortable, but it provided a good platform for working 24 hours a day,” says Pinkel. “When things were working, you didn’t want to leave your experiment to go back to a mothership.” So bunks for 16 people were soon installed—wherever there was room—and now scientists spend weeks at a time on the platform.

FLIP isn’t exactly a ship; it doesn’t have engines of its own, so it must be towed out to sea, laying flat like a boat, before it can sit up. Researchers ride along, making do with awkward furniture arrangements until the big flip. “All of the rooms, basically, are on edge, waiting to come to life when they’re in the vertical,” says Pinkel. “The amount of useful space that you have once you’ve flipped suddenly triples.”

The flip takes about half an hour. When the platform reaches its planned research site, the long, thin spar is filled with water, and that end begins to sink, thrusting the bow high into the air. “You go from being 5 feet above the ocean to being 30 feet or 40 feet up in the course of the very last two minutes while it’s taking its Titanic dive,” says Pinkel. “A big concern is if everything isn’t stowed correctly, things will fall off shelves, and so to minimize any chance of danger, we all stand outside on the working decks and hang on out there.”

Once the ship is flipped, and the seasickness and disorientation subside, it’s time for the scientists to get to work. “FLIP would be enormously fun if you didn’t have a job to do,” says Pinkel. Instruments and computers are turned on and begin collecting data from sensors placed all over. Today the platform is used for a wide array of projects. Pinkel, for example, has focused on slow-moving waves that propagate in the interior of the ocean, below the choppy surface. Because FLIP doesn’t have engines below the water, it’s a quiet place, and its flipped configuration is stable, even in 30-foot seas. Pinkel’s colleague, John Hildebrand, has used FLIP to monitor the vocalizations of whale populations, while others listen to some of the smallest animals in the ocean—plankton. The platform can shut off its generators and run on electrical power from batteries for short periods of time so scientists can hear the faint Doppler effect of the microorganisms moving toward or away from the ship.

As research budgets shrink, and fewer people go into oceanographic research, it’s hard to say what is on the horizon for FLIP. “The future of FLIP is totally determined by the future of people in science,” says Pinkel. “FLIP has been a kind of quiet resource for the country for 50 years and it’s not so much the ship, but that the next generation of people who are willing to use it—that’s what hangs in the balance.”

The study team should get back to you in a few business days.

You will also receive an email with next steps. Check your junk/spam folder if needed.

If you do not hear from the study team, please call 888-689-8273 and tell them you’re interested in study number NCT02950467.

The rate of reports of severe allergic reactions to foods like peanuts has increased by nearly five times over the past decade, according to a new analysis of private insurance claims.

The analysis looked at private insurance claims with a diagnosis of an anaphylactic food reaction from 2007 to 2016. Anaphylaxis is a systemic allergic reaction in which the immune system affects multiple parts of the body at the same time, often...

Vim is my default editor. There’s no particular reason for this, except that I ended up learning it when I moved over to Linux many years ago. I ended up liking it because I could edit my small source files on my quad-core machine without needing to wait forever for the file to open.

Sure Vim isn’t a bad editor, it’s highly extensible, it’s easy to shell out to the, err well shell, its everywhere so when you ssh into some obscure server you can just type vim (or vi) and you’re good to go.

But this isn’t a pitch about Vim being a great editor, that’s a matter of subjective taste. I’ve primarily stuck with it because it’s an extensible editor that doesn’t hog all the resources and kill my machines. Using Atom or Code I experience frequent freezes for several minutes when just typing a single character.

How much memory would you expect an editor needs to open the following C file?

#include<stdio.h>

int main() {
printf("Hello, world!\n");
}

Memory Usage

The answer is… crazy.

Code requires a whopping 349 megabytes in order to open a 60 byte file. Atom comes in at 256 megabytes. Where Vim “only” needs 5 megabytes, which is still kind of high, but representative of an average configuration.

I’ve also included Nano to have another text mode editor to compare Vim with, which came out at less than a megabyte.

What about larger files? Opening a 6 megabyte XML file in Vim consumes around 12 megabytes. Nano is pretty much neck-and-neck with Vim. Code needs 392 megabytes, and Atom needs a whopping 845 megabytes.

Startup Time

What about the amount of time necessary to open that same XML file, then move your cursor to the end of it? This tells a similar story. Atom and Code take nearly 20 seconds. Vim takes around 4 seconds. Sublime is surprisingly fast here taking only a mere second.

Doing a search and replace for 100,000 instances of a word in that same XML file yields somewhat surprising results. Nano and Atom failed, taking nearly 10 minutes on average to complete. Atom crashed quite a few times trying to get a result. Code took around 80 seconds. Sublime finished in 6 seconds. And Vim only took 4 seconds.

Conclusion

Learn Vim. It’s worth checking out http://vimcasts.org, which is basically Vim golfing, tips, and tricks made by Drew Neil, who also wrote this awesome book

If not Vim, then maybe Emacs. Or, well, just anything that is not a web browser masquerading as a text editor.

It’s just flat out ridiculous to have an editor consume all the processing power and memory available on a “modern” expensive laptop, when it doesn’t need to do so at all.

The test files used in these benchmarks were taken from this repository, results were averaged between that dataset and my own.

How do employers motivate workers to do the unglamorous tasks that keep companies running?

Mozilla Corp., maker of the Firefox web browser, faced that question last year when it weighed how to fix the bugs in its bug-fixing process. Programmers considered the task, which solves issues that slow or impede the browser, tedious compared with the flashier work of designing new features.

To kindle interest and get more employees...

Image copyrightTom Wall

A headless torso found in waters off Denmark has been identified as missing Swedish journalist Kim Wall, Danish police say.

DNA from the torso matched that from Ms Wall's hairbrush and toothbrush.

Chief investigator Jens Moller Jensen said that the torso had been weighted down with metal in an apparent attempt to stop it floating.

Ms Wall was last seen alive on 10 August as she departed on a submarine trip with inventor Peter Madsen.

The submarine sank hours after the search for Ms Wall began, after her partner reported that she had not returned from the trip. Mr Madsen, who designed and built the submarine, was charged with negligent manslaughter.

He initially said he had dropped her off safely near Copenhagen, but has since said she died in an accident and that he had "buried" her at sea.

Danish police believe the 40-tonne submarine was deliberately sunk by Mr Madsen.

Traces of blood have been found inside the submarine, and they also match Ms Wall.

The remains were found on a beach south of Copenhagen on Monday.

Mr Jensen would not comment on the cause of death but said forensic investigations were still being carried out and police were looking for the rest of her body.

As well as the metal attached to the torso, Mr Jensen said the remains were mutilated in what appeared to be an attempt to ensure that decomposition gases passed out of the body, to make it less likely to float.

Kim Wall's mother Ingrid wrote of the family's "boundless sorrow" at the news that her daughter's remains had been found.

"During the horrendous days since Kim disappeared, we have received countless examples of how loved and appreciated she was, as a person and as a friend, as well as a professional journalist," Ingrid Wall said in a family statement released on Facebook.

"From all corners of the world we have received testimony to how she was able to be a person who made a difference."

Ms Wall, 30, was reported missing by her boyfriend in the early hours of 11 August, after she failed to return from the trip on Peter Madsen's homemade submarine, the Nautilus.

A freelance journalist who had written for the Guardian, New York Times and South China Morning Post, she was researching a feature about the inventor and the Nautilus, which he built in 2008 with crowdfunding.

Emergency services scoured the area of sea to the east of Copenhagen and the submarine was eventually spotted from a lighthouse south of the Oresund bridge between Denmark and Sweden. Within 30 minutes the vessel had sunk and Mr Madsen was rescued.

For 10 days, the search for the journalist continued. A torso was found by a passing cyclist on a beach near Koge Bay on Monday.

Police said the next day that the arms, legs and head had been deliberately cut off. They finally confirmed it was Kim Wall in a tweet early on Wednesday.

Mr Madsen's lawyer, Betina Hald Engmark, said the news that the torso was Ms Wall did not change her client's position, that the journalist had died in an accident. Both she and her client were pleased that the remains had been found, she told Danish media.

Peter Madsen pleaded not guilty in a closed-door judicial hearing earlier this month.

Copenhagen police are re-examining old cases, as is standard in such investigations, Mr Jensen said, including the death of a 22-year-old Japanese woman, whose torso was found in Copenhagen harbour in 1986. That case has never been solved.

Kick Start a JavaScript GraphQL 3 Layers API with Apollo-Server, Dataloader and Knex (comments)

A history of branch prediction from 1500000 BC to 1995 (comments)

Companies from Y Combinator’s Summer 2017 Demo Day (Day 2) (comments)

Why I haven't jumped ship from Common Lisp to Racket just yet (comments)

Apple Scales Back Its Ambitions for a Self-Driving Car (comments)

HackerNews grid (comments)

Kim Wall: Headless body identified as missing journalist (comments)

The Dark Side of Resilience (comments)

Mathematicians Tame Rogue Waves, Illuminating Future of LED Lighting (comments)

The Enduring Legacy of Zork (comments)

Dangers of Sitting (comments)

Sun Flyer, an Electric Trainer Aircraft (comments)

The Beauty of Bresenham's Algorithm (comments)

A Striking Rise in Serious Allergy Cases (comments)

A glut has used-car depreciation accelerating (comments)

A ship that flips 90 degrees for precise scientific measurements (comments)

Psilocybin-assisted group therapy for demoralization in long-term AIDS survivors (comments)

Why George Guidall Is the Undisputed King of Audiobooks (comments)

The Management Fix That Made Firefox Faster (comments)

Show HN: The best time to visit any city (comments)

The Icewind Dale Problem (comments)

Show HN: RNN that generates Game of Thrones text (comments)

Atom uses nearly 1GB of RAM to open a 6MB file (comments)

The Book He Wasn't Supposed to Write (comments)

Winner-take-all effects in autonomous cars (comments)

Chrome Enterprise (comments)

Warriors Who Once Feared Elephants Now Protect Them (comments)

Swift Local Refactoring (comments)

Inside a fast CSS engine (comments)

Last updated at: Wed Aug 23 2017 03:20:23 GMT-0700 (PDT)

Prerequisites (~12min)

• Have Yarn installed (~5min)
• Have Docker and Docker-compose installed (~5min)
• Have Postman installed (~2min)

Thanks

Thanks to Tycho Tatitscheff and Yann Leflour for helping me with their great BAM API repo

Context

During this standard, we will create a Heroes graphQL API. We will have a Hero model with superheroes real and hero names. We will add one example of association.

Our API will be lightly protected and use batching to minimise DB round-trips.

Steps (~61min)

Note: You should commit between each step.

Initialise a new project (~6min)

• Create and go to a new directory for the project: mkdir graphql_formation && cd graphql_formation
• Init a git repository: git init

• Create two services with Docker-compose, one postgres database and one node server:

  • For this step, notice that our final folder architecture looks like this:

    📂 graphql_formation
    ├ 📂 api
    │ └ 🗋 Dockerfile
    ├ 📂 db
    │ └ 🗋 Dockerfile
    ├ 🗋 config.env
    └ 🗋 docker-copose.yml

  • Make sure your local 3000 port is available as we will use this port to reach our API

  • In a new api/Dockerfile file, write all the commands to assemble the API image:

  FROM node:8.1.0-alpineWORKDIR/usr/src/apiEXPOSE3000
  CMD ["yarn", "run", "serve"]

  • In a new db/Dockerfile file, write all the commands to assemble the db image:

  FROM postgres:9.6.3

  • In a new docker-compose.yml file, declare the two services:

  version:'3'services:  api:    build:      context: ./api    image: heroes-api    env_file: config.env    volumes:      - ./api:/usr/src/api    ports:      -3000:3000    links:      - db:db  db:    build:      context: ./db    env_file: config.env    image: heroes-db    ports:      -5431:5432

  • In a new config.env file, declare your environnement variable for these Docker containers:

  POSTGRES_USER=heroesuser
  POSTGRES_PASSWORD=heroespassword
  POSTGRES_DB=heroesdb
  PGDATA=/data
  DB_HOST=db

• Build these services with the command: docker-compose build

CHECK 1: Your terminal should prompt successively these lines confirming Docker images have been built:

Successfully tagged heroes-db:latest

Successfully tagged heroes-api:latest

Install nodemon and run our project (~5min)

• Add this to the project .gitignore: echo "node_modules" > .gitignore
• In the api folder, interactively create a api/package.json file: cd api && yarn init
• Add nodemon, babel-cli, babel-plugin-transform-class-properties, babel-preset-flow and babel-preset-es2015 to our dev dependencies: yarn add nodemon babel-cli babel-plugin-transform-class-properties babel-preset-es2015 babel-preset-flow -D
• In a new api/.babelrc file, write the babel configuration:

  {"presets":["es2015","flow"
  ],"plugins":["transform-class-properties"
  ]
  }

• In our api/package.json, write the command to launch the server:

  "scripts": {"serve": "nodemon index.js --exec babel-node"
  }

• Create a new empty file api/index.js
• Go back to the root of the project: cd ..
• Run the project: docker-compose up

CHECK 1: You terminal should prompt the logs of the two containers together with two different colors

CHECK 2: From another terminal, you can access the API and see the following folder structure: docker-compose exec api /bin/sh then inside the container: ls -lath;

drwxrwxr-x    3 node     node        4.0K Aug 17 12:37 .
-rw-rw-r--    1 node     node           0 Aug 17 12:37 index.js
drwxrwxr-x  222 node     node       12.0K Aug 17 12:37 node_modules
-rw-rw-r--    1 node     node         426 Aug 17 12:37 package.json
-rw-rw-r--    1 node     node       66.2K Aug 17 12:37 yarn.lock
-rw-rw-r--    1 node     node          86 Aug 17 12:32 Dockerfile
drwxr-xr-x    3 root     root        4.0K Aug  3 11:50 ..

Exit with: CTRL-D

CHECK 3: You can access the db and prompt the PostgreSQL version: docker-compose exec db psql -U heroesuser -d heroesdb then inside the container: select version();

PostgreSQL 9.6.3 on x86_64-pc-linux-gnu, compiled by gcc (Debian 4.9.2-10) 4.9.2, 64-bit

Exit with: CTRL-D

Create a koa server (~3min)

• Install koa and koa-router in our API: cd api && yarn add koa koa-router
• In the index.js file, create our server:

import Koa from'koa';import koaRouter from'koa-router';const app = new Koa();const router = new koaRouter();

router.get('/', ctx => {
  ctx.body = 'Hello World!';
});

app.use(router.routes());
app.use(router.allowedMethods());
app.listen(3000);console.log('Server is up and running');

CHECK 1: In your terminal which run docker-compose, you should see Server is up and running

CHECK 2: Hitting localhost:3000 should return Hello World!: curl localhost:3000

Create a presentation layer with graphQL (~6min)

This layer will let our API know how to present data: what data one user can query? How should front end query this data (fields, root queries, sub queries...)?

• Install graphQL, graphQL Server Koa, graphQL tools and Koa body-parser: yarn add graphql graphql-server-koa graphql-tools koa-bodyparser
• In a new folder api/presentation add a new schema.js file describing a simple graphQL schema:

import { makeExecutableSchema } from'graphql-tools';const typeDefs = [`
  type Hero {
    id: Int!
    firstName: String
    lastName: String
  }

  type Query {
    heroes: [Hero]
  }

  schema {
    query: Query
  }
`];const resolvers = {
  Query: {
    heroes: () => ([
      {
        id: 1,
        firstName: 'Clark',
        lastName: 'Kent',
      },
      {
        id: 2,
        firstName: 'Bruce',
        lastName: 'Wayne',
      }
    ]),
  },
}const schema = makeExecutableSchema({ typeDefs, resolvers });exportdefault schema;

• In the api/index.js file, add our api endpoint:

import koaBody from'koa-bodyparser';import { graphqlKoa } from'graphql-server-koa';import schema from'./presentation/schema';

...

router.post(
  '/api',
  graphqlKoa(async ctx => {return {
      schema: schema,
      context: {},
      debug: true,
    };
  })
);

...


app.use(koaBody());

CHECK 1: In Postman, making a POST request to localhost:3000/api which content-type is JSON(application/json) with the following raw body:

{"query": "{heroes { firstName lastName }}"
}

...should return our two heroes, Clark and Bruce:

• Install Koa graphiQL: yarn add koa-graphiql
• In the index.js file, let our API knows it should use Koa-graphiql:

import graphiql from'koa-graphiql';

...


router.get('/graphiql', graphiql(async (ctx) => ({
  url: '/api',
})));

CHECK 2: Hitting localhost:3000/graphiql should return graphiql interface and show the Docs

CHECK 3: Using graphiql interface with the following query:

{
  heroes {
    firstName
    lastName
  }
}

...should return our two heroes, Clark and Bruce:

Create a business layer (~5min)

This layer will contain all business logic: access controll, scoping / whitelisting, batching and caching and computed properties. More explanations can be found here, in the bam-api repo. In this MO, we will only cover access control logic and batching / caching.

• In a new api/business folder add a new hero.js file describing our class for this business object:

const mockedHeroes = [
  {
    id: 1,
    firstName: 'Clark',
    lastName: 'Kent',
  },
  {
    id: 2,
    firstName: 'Bruce',
    lastName: 'Wayne',
  }
];classHero{
  id: number;
  firstName: string;
  lastName: string;constructor(data) {this.id = data.id;this.firstName = data.firstName;this.lastName = data.lastName;
  }staticasync load(ctx, args) {const data = mockedHeroes[args.id];if (!data) returnnull;returnnew Hero(data);
  }staticasync loadAll(ctx, args) {const data = mockedHeroes;return data.map(row => new Hero(row));
  }

}

exportdefault Hero;

• In our previous presentation/schema.js file, modify our mocked resolvers to use our business layer:

+import Hero from '../business/hero';

  type Query {
    heroes: [Hero]
+    hero(id: Int!): Hero
  }

const resolvers = {
  Query: {
-    heroes: () => ([-      {-        firstName: 'Clark',-        lastName: 'Kent',-      },-      {-        firstName: 'Bruce',-        lastName: 'Wayne',-      }-    ]),+    heroes: async (_, args, ctx) => Hero.loadAll(ctx, args),+    hero: async (_, args, ctx) => Hero.load(ctx, args),
  }
}

CHECK 1: Using graphiql interface with the following query:

{
  heroes {
    id
    firstName
    lastName
  }
}

...should return our two heroes, Clark and Bruce.

CHECK 2: Using graphiql interface with the following query:

{
  hero(id:0) {
    id
    firstName
    lastName
  }
}

...should return Clark Kent with its id: 1.

CHECK 3: Using graphiql interface with the following query:

{
  hero(id:1) {
    id
    firstName
    lastName
  }
}

...should return Bruce Wayne with its id: 2.

Seed our database (~8min)

• Install knex and pg at the root of the project: cd .. && yarn add knex pg
• At the root of our project, add a knexfile.js file:

module.exports = {
  development: {
    client: 'pg',
    connection: {
      host: 'localhost',
      port: 5431,
      user: 'heroesuser',
      password: 'heroespassword',
      database: 'heroesdb',
    },
    migrations: {
      directory: './api/db/migrations',
    },
    seeds: {
      directory: './api/db/seeds',
    },
  },
};

• Create a migration file: yarn knex migrate:make add_heroes_table and complete the new created file with this:

exports.up = function(knex, Promise) {return knex.schema.createTableIfNotExists('Heroes', function(table) {
    table.increments('id');
    table.string('firstName');
    table.string('lastName');
    table.string('heroName');
  });
};

exports.down = function(knex, Promise) {return knex.schema.dropTableIfExists('Heroes');
};

• Create a seed file: yarn knex seed:make heroes and complete the new created file with this:

exports.seed = function(knex, Promise) {return knex('Heroes').del()
    .then(function () {return knex('Heroes').insert([
        {id: 1, firstName: 'Clark', lastName: 'Kent', heroName: 'Superman'},
        {id: 2, firstName: 'Bruce', lastName: 'Wayne', heroName: 'Batman'},
        {id: 3, firstName: 'Peter', lastName: 'Parker', heroName: 'Spiderman'},
        {id: 4, firstName: 'Susan', lastName: 'Storm-Richards', heroName: 'Invisible Woman'},
      ]);
    });
};

• Run the migration and the seed: yarn knex migrate:latest && yarn knex seed:run

CHECK 1: You can access the db and prompt content of the Heroes table: docker-compose exec db psql -U heroesuser -d heroesdb then inside the container: select * from "Heroes";;

 id | firstName |    lastName    |    heroName     
----+-----------+----------------+-----------------
  1 | Clark     | Kent           | Superman        
  2 | Bruce     | Wayne          | Batman          
  3 | Peter     | Parker         | Spiderman       
  4 | Susan     | Storm-Richards | Invisible Woman 
(4 rows)

Exit with: CTRL-D

Create a db layer with knex (~6min)

This layer let our API query the data using knex query builder.

• Install knex and pg in our API: cd api && yarn add knex pg
• In the api/db folder add a new index.js file:

import knex from'knex';exportdefault knex({
  client: 'pg',
  connection: {
    host: process.env.DB_HOST,
    user: process.env.POSTGRES_USER,
    password: process.env.POSTGRES_PASSWORD,
    database: process.env.POSTGRES_DB,
  },
  debug: true,
});

• In a new api/db/queryBuilders subfolder, create a new hero.js file and add these few methods to query our data:

import db from'..';classHero{staticasync getById(id: number) {return db
    .first()
    .table('Heroes')
    .where('id', id);
  }staticasync getByIds(ids: Array<number>) {return db
    .select()
    .table('Heroes')
    .whereIn('id', ids);
  }staticasync getAll() {return db
    .select()
    .table('Heroes');
  }
}exportdefault Hero;

• Modify the api/db/queryBuilders/hero.js file in our business layer this way:

-const heroes = [-  {-    id: 0,-    firstName: 'Clark',-    lastName: 'Kent',-  },-  {-    id: 1,-    firstName: 'Bruce',-    lastName: 'Wayne',-  }-];+import HeroDB from '../db/queryBuilders/hero';

 class Hero {

   static async load(ctx, args) {
-    const data = heroes[args.id];+    const data = await HeroDB.getById(args.id);
     if (!data) return null;

     return new Hero(data);
   }

   static async loadAll({ authToken, dataLoaders }) {
-    const data = heroes;+    const data = await HeroDB.getAll();

     return data.map(row => new Hero(row));
   }

CHECK 1: Using graphiql interface with the following query:

{
  hero(id:1) {
    id
    firstName
    lastName
  }
}

...should return Clark Kent with its id: 1.

CHECK 2: Using graphiql interface with the following query:

{
  heroes {
    id
    firstName
    lastName
  }
}

...should return all 4 heroes of our database.

Add association to our API (~6min)

Association are made both in our db and in our API, in our presentation layer.

• Create a new migration: cd .. && yarn knex migrate:make add_heroes_enemies
• Complete the newly created migration file with this:

exports.up = function(knex, Promise) {return knex.schema.table('Heroes', function(table) {
    table.integer('enemyId').references('id').inTable('Heroes');
  });
};

exports.down = function(knex, Promise) {return knex.schema.table('Heroes', function(table) {
    table.dropColumn('heroName');
  });
};

• Modify our api/db/seeds/heroes.js seeds:

exports.seed = function(knex, Promise) {return knex('Heroes').del()
    .then(function () {return knex('Heroes').insert([
        {id: 1, firstName: 'Clark', lastName: 'Kent', heroName: 'Superman', enemyId: 2},
        {id: 2, firstName: 'Bruce', lastName: 'Wayne', heroName: 'Batman', enemyId: 1},
        {id: 3, firstName: 'Peter', lastName: 'Parker', heroName: 'Spiderman'},
        {id: 4, firstName: 'Susan', lastName: 'Storm-Richards', heroName: 'Invisible Woman'},
      ]);
    });
};

• Run these migrations: yarn knex migrate:latest && yarn knex seed:run
• In our business layer, modify api/business/hero.js this way:

class Hero {
  id: number;
  firstName: string;
  lastName: string;+  heroName: string;+  enemyId: number;

  constructor(data) {
    this.id = data.id;
    this.firstName = data.firstName;
    this.lastName = data.lastName;
+    this.heroName = data.heroName;+    this.enemyId = data.enemyId;
  }

• In our API, in our presentation layer, modify our api/presentation/schema.js:

const typeDefs = [`
  type Hero {
    id: Int
    firstName: String
    lastName: String+    heroName: String+    enemy: Hero
  }
...
`];

const resolvers = {
  Query: {
...
  },
+  Hero: {+    enemy: async (hero, args, ctx) => Hero.load(ctx, {id: hero.enemyId}),+  },
}

CHECK 1: Using graphiql interface with the following query:

{
  hero(id:1) {
    id
    firstName
    lastName
    heroName
    enemy {
      heroName
    }
  }
}

...should return Clark Kent with its heroName and its enemy: Batman.

Push your API to the next level: use caching with Dataloader (~6min)

Trying to query heroes and their enemies'heroName will show up a N+1 problem. Indeed, our API make 5 round-trips to our database! Try yourself:

{"query": "{heroes { id firstName lastName heroName enemy { heroName } }}"
}

We can reduce these calls adding caching to our business layer

• Install Dataloader: cd api && yarn add dataloader
• Add a getLoaders method to our api/business/hero.js file in our business layer:

import DataLoader from'dataloader';classHero{static getLoaders() {const getById = new DataLoader(ids => HeroDB.getByIds(ids));const primeLoaders = (heroes) => {
      heroes.forEach(hero =>
        getById.clear(hero.id).prime(hero.id, hero))
      ;
    };return { getById, primeLoaders };
  }
}

• In our api/index.js file, add a new dataloader to our context for each query on /api route:

+import Hero from './business/hero';

router.post(
  '/api',
  graphqlKoa(async ctx => {
    return {
      schema: schema,+      context: {+        dataLoaders: {+          hero: Hero.getLoaders(),+        }+      },
      debug: true,
    };
  })
);

• Back in our api/business/hero.js business layer file, modify load and loadAll methods to use our dataloader:

  static async load(ctx, args) {+    const data = await ctx.dataLoaders.hero.getById.load(args.id);
    if (!data) return null;

    return new Hero(data);
  }

  static async loadAll(ctx, args) {
    const data = await HeroDB.getAll();
+    ctx.dataLoaders.hero.primeLoaders(data);

    return data.map(row => new Hero(row));
  }

• Protect loader.load() function call if no argument is supplied:

  static async load(ctx, args) {+    if (!args.id) return null;
    const data = await ctx.dataLoaders.hero.getById.load(args.id);
    if (!data) return null;

    return new Hero(data);
  }

CHECK 1: Using graphiql interface with the following query:

{
  heroes {
    id
    firstName
    lastName
    heroName
    enemy {
      heroName
    }
  }
}

...should return all heroes and their enemies and your terminal should prompt only one request to the DB.

CHECK 2: Using graphiql interface with the following query:

{
  h1: hero(id:1) {
    id
    firstName
    lastName
    heroName
    enemy {
      heroName
    }
  }
  h2: hero(id:2) {
    id
    firstName
    lastName
    heroName
    enemy {
      heroName
    }
  }
}

...should return Clark Kent and Bruce Wayne; and only one SELECT call should have beeen made to our DB.

Add access control to our API (~5min)

This is a very simple example, for a more advanced solution, prefer using Koa Jwt.

• In a new api/utils.js file, add these two methods to parse Authorization header and verify token:

exportconst parseAuthorizationHeader = (req) => {const header = req.headers.authorization;if (typeof header === 'undefined' || header === 'null') {returnnull;
  }const [, scheme, token] = (/(\w+) ([\w.-]+)/g).exec(header);return token;
};exportconst verifyToken = token => newPromise((resolve, reject) => {if (token !== 'authorized') {const error = newError('UNAUTHORIZED');
    error.code = 401;
    reject(error);
  }
  resolve();
});

• In our api/index.js file, parse authorization header and pass it to our context:

+import { parseAuthorizationHeader } from './utils';

router.post(
  '/api',
  graphqlKoa(async ctx => {
    return {
      schema: schema,
      context: {+        authToken: parseAuthorizationHeader(ctx.req),
        dataLoaders: {
          hero: Hero.getLoaders(),
        }
      },
      debug: true,
    };
  })
);

• In our business layer, modify api/business/hero.js:

+import { verifyToken } from '../utils';


  static async load(ctx, args) {
+    await verifyToken(ctx.authToken);

  static async loadAll(ctx, args) {
+    await verifyToken(ctx.authToken);

CHECK 1: In Postman, making a POST request to localhost:3000/api which content-type is JSON(application/json) with the following raw body:

{"query": "{hero(id:1) { id firstName lastName heroName }}"
}

...should return UNAUTHORIZED.

CHECK 2: In Postman, making a POST request to localhost:3000/api which content-type is JSON(application/json) with the following raw body:

{"query": "{heroes { id firstName lastName heroName }}"
}

...should return UNAUTHORIZED.

CHECK 3: In Postman, making a POST request to localhost:3000/api which content-type is JSON(application/json) and Authorization Header is Bearer authorized with the following raw body:

{"query": "{hero(id:1) { firstName lastName }}"
}

...should return Clark Kent.

Troubleshooting: Accessing data by id in the correct order (~5min)

You should notice that in Postman making a POST request to localhost:3000/api which content-type is JSON(application/json) and Authorization Header is Bearer authorized with the following raw body:

{"query": "{h1: hero(id:1) { id firstName lastName heroName enemy { heroName } } h2: hero(id:2) { id firstName lastName heroName enemy { heroName } }}"
}

...returns the same than the following request (ids switched):

{"query": "{h1: hero(id:2) { id firstName lastName heroName enemy { heroName } } h2: hero(id:1) { id firstName lastName heroName enemy { heroName } }}"
}

This is due to our DB query: select * from "Heroes" where "id" in (1, 2) return the same result than: select * from "Heroes" where "id" in (2, 1).

• In utils.js, add the following method:

exportconst orderByArgIdsOrder = ids => ("array_position(string_to_array(?, ',')::integer[], id)", ids.join(','));

• In our db layer, modify api/db/queryBuilders/hero.js like this:

+import { orderByArgIdsOrder } from '../../utils';

class Hero {

  static async getByIds(ids: Array<number>): Promise<Array<CostDBType>> {
    return db
    .select()
    .table('Heroes')+    .whereIn('id', ids)+    .orderByRaw(orderByArgIdsOrder(ids));
  }

CHECK 1: In Postman, making a POST request to localhost:3000/api which content-type is JSON(application/json) and Authorization Header is Bearer authorized with the following raw body:

{
    "query": "{h1: hero(id:2) { heroName } h2: hero(id:1) { heroName }}""
}

...should return Batman (as h1) then Superman (as h2).

Next steps

• Add Koa Jwt
• Add graphiQL with authorization header (get inspired by BAM API)

Version: 5.2.1

Danny Yoo <dyoo@hashcollision.org>

Source code can be found at:https://github.com/dyoo/whalesong. The latest version of this document lives in http://hashcollision.org/whalesong.

Current commit head is 0bcdd61f10af9739e4b757d879b2a1880907ca09.

1 Introduction

Whalesong is a compiler from Racket to JavaScript; it takes Racket programs and translates them so that they can run stand-alone on a user’s web browser. It should allow Racket programs to run with (hopefully!) little modification, and provide access through the foreign-function interface to native JavaScript APIs. The included runtime library supports the numeric tower, an image library, and a framework to program the web in functional event-driven style.

The GitHub source repository to Whalesong can be found athttps://github.com/dyoo/whalesong. If you have any questions or comments, please feel free to use theRacket-users mailing list.

Prerequisites: at least Racket 5.1.1. If you wish to use the JavaScript compression option, you will need Java 1.6 SDK.

1.1 Examples

Here are a collection of programs that use the web-world library described later in this document:

2 Getting started

2.1 Installing Whalesong

Before you begin, if you are using DrRacket,

This is to avoid an installation-time issue that prevents Whalesong from fully compiling.

If you want to use Whalesong, run the following to create the "whalesong" launcher:

This may take a few minutes, as Racket is compiling Whalesong, its dependencies, and its documentation. When it finally finishes, you should see a "whalesong" launcher in the current directory.

You should also see a "whalesong-gui" launcher that includes a minimal graphical user interface.

At this point, you should be able to run the "whalesong" executable from the command line.

and if this does appear, then Whalesong should be installed successfully.

To repeat: whenever Whalesong’s source code is updated from Github, please re-run the raco setup step. Otherwise, Racket will try to recompile Whalesong on every single use, which can be very expensive.

2.2 Making .html files with Whalesong

Let’s try making a simple, standalone executable. At the moment, the program must be written in the base language of (planetdyoo/whalesong). This restriction unfortunately prevents arbitraryracket/base programs from compiling at the moment; the developers (namely, dyoo) will be working to remove this restriction as quickly as possible.

Write a "hello.rkt" with the following content

This program is a regular Racket program, and can be executed normally,

However, it can also be packaged with "whalesong".

Running whalesong build on a Racket program will produce a".html" and ".js" file. If you open the".html" in your favorite web browser, you should see a triumphant message show on screen.

We can do something slightly more interesting. Let’s write a Whalesong program that accesses the JavaScript DOM. Call this file "dom-play.rkt".

2.3 Using Whalesong functions from JavaScript

Whalesong also allows functions defined from Racket to be used from JavaScript. As an example, we can take the boring factorial function and define it in a module called "fact.rkt":

The files can also be downloaded here:

with generated JavaScript binaries here:

"fact.rkt"

Instead of creating a standalone .html, we can use whalesong to get us the module’s code. From the command-line:

This file does require some runtime support not included in"fact.js"; let’s generate the runtime.js and save it as well. At the command-line:

Replacing the 10000 with "one-billion-dollars" should reliably produce a proper error message.

3 Using whalesong

Whalesong provides a command-line utility called whalesong for translating Racket to JavaScript. It can be run in several modes:

Using whalesong to generate HTML+js documents is relatively straightforward with the build command. To use it, pass the name of the file to it:

A ".html" and ".js" will be written to the current directory, as will any external resources that the program uses.

The whalesong commands support these command line options:

• Use Google Closure’s JavaScript compiler to significantly compress the JavaScript. Using this currently requires a Java 1.6 JDK.

• Write verbose debugging information to standard error.

• Write files to a separate directory, rather than the current directory.

• Write each dependent module as a separate file, rather than in one large ".js". This may be necessary if your browser environment prohibits large ".js" files. The files will be numbered starting from 1.

For more advanced users, whalesong can be used to generate JavaScript in non-standalone mode. This gives the web developer more fine-grained control over how to control and deploy the outputted program.

3.1 build

Given the name of a program, this builds".html" and ".js" files into the current working directory.

The ".html" and ".js" should be self-contained, with an exception: if the file uses any external resources by usingdefine-resource, those resources are written into the current working directory, if they do not already exist there.

3.2 get-javascript

Given the name of a program, writes the JavaScript to standard output, as well as its dependent modules. The outputted file is meant to be used as a SCRIPT source.

By default, the given program will be treated as a main module. All main modules will be executed when the JavaScript functionplt.runtime.invokeMains() is called.

3.3 get-runtime

Prints out the core runtime library that the files generated by get-javascript depend on.

4 Including external resources

Programs may need to use external file resources that aren’t themselves Racket programs, but instead some other kind of data. Graphical programs will often use ".png"s, and web-related programs ".html"s, for example. Whalesong provides the(planetdyoo/whalesong:1:=18/resource) library to refer and use these external resources. When Whalesong compiles a program into a package, these resources will be bundled alongside the JavaScript-compiled output.

For example,

As a convenience, you can also write

If the resource given has an extension one of the following:

then it can be treated as an image for which image? will be true.

If the resource has the extension ".html", then it will be run through an HTML purifying process to make sure the HTML is well-formed.

For example,

5 The web-world API

The web-world library allows you to write functional event-drivenWorld programs for the web; the user defines functional callbacks to handle events, and receive and consume a world argument.

One difference introduced by the web is the web page itself: because the page itself is a source of state, it too will be passed to callbacks. This library presents a functional version of the DOM in the form of a view.

Let’s demonstrate this by creating a basic ticker that counts on the screen every second.

The first thing we can do is mock up a web page with a user interface, like this.

We can even look at this in a standard web browser.

Once we’re happy with the statics of our program, we can inject dynamic behavior. Write a file called "tick-tock.rkt" with the following content.

Several things are happening here.

5.1 big-bang and its options

By default, this will send a clock tick 28 times a second, but if given delay, it will use that instead.

During the extent of a big-bang, a form widget will appear in thedocument.body to allow us to manually send location-changing events.

5.2 Views

A view is a functional representation of the browser DOM tree. A view is always focused on an element, and the functions in this subsection show how to traverse and manipulate the view.

Return true if the view can be focused using the given id.

Focuses the view on an element, given the id. The view will be searched starting from the toplevelmost node.

See if the view can be moved to the previous sibling.

Move the focus to the previous sibling.

See if the view can be moved to the next sibling.

Move the focus to the next sibling.

See if the view can be moved to the parent.

Move the focus to the parent.

See if the view can be moved to the first child.

Move the view to the first child.

See if the view can be moved forward.

Move the view forward, assuming a pre-order traversal.

See if the view can be moved backward.

Move the view backward, assuming a pre-order traversal.

Get the textual content at the focus.

Update the textual content at the focus.

Attach a world-updating event to the focus.

Attach a world-updating event to the focus. When the world-updater is called, the view will be focused on the element that triggered the event.

Common event types include "click", "mouseenter","change". Note that the name of the event should not include an "on" prefix.

A view may have many elements to bind, and it’s a common pattern to focus and view. As a convenience the API provides some syntactic support to bind multiple handlers at once:

Common event types include "click", "mouseenter", or"change". Note that the name of each event should not include an "on" prefix.

As an example:

If the collection of ids, types, and handlers can’t be represented as a static list, then view-bind-many* is an alternate helper function that may be helpful to bind a bulk number of handlers to a view.

Common event types include "click", "mouseenter", or"change". Note that the name of each event should not include an "on" prefix.

As an example:

Show the element at the focus.

Hide the element at the focus.

Get the attribute name at the focus.

Returns true if the element at the focus has an attribute name.

Update the attribute name with the value value at the focus.

Remove the attribute name at the focus.

Get the css value name at the focus.

Update the css value n with the value v at the focus.

Get the unique identifier of the node at the focus.

Get the form value of the node at the focus.

Update the form value of the node at the focus.

Dom nodes can be created by using xexp->dom, which converts axexp to a node, and attached to the view by usingview-append-child, view-insert-left, andview-insert-right.

Add the dom node d as the last child of the focused node. Focus moves to the inserted node.

Add the dom node d as the previous sibling of the focused node. Focus moves to the inserted node.

Add the dom node d as the next sibling of the focused node. Focus moves to the inserted node.

Remove the dom node at the focus from the view v. Focus tries to move to the right, if there’s a next sibling. If that fails, focus then moves to the left, if there’s a previous sibling. If that fails too, then focus moves to the parent.

5.3 Events

An event is a structure that holds name-value pairs. Whenever an event occurs in web-world, it may include some auxiliary information about the event. As a concrete example, location events from on-location-change and on-mock-location-change can send latitude and longitude values, as long as the world callback can accept the event as an argument.

kvpairs : (listof (list symbol (or/c string number)))

Get an value from the event, given its name.

Get an list of the event’s keys.

5.4 Dynamic DOM generation with xexps

We often need to dynamically inject new dom nodes into an existing view. As an example where the UI is entirely in code:

Normally, we’ll want to do as much of the statics as possible with ".html" resources, but when nothing else will do, we can generate DOM nodes programmatically.

We can create new DOMs from an xexp, which is a s-expression representation for a DOM node. Here are examples of expressions that evaluate to xexps:

"hello world"

'(p "hello, this" "is an item")

(local [(define name "josh")]

`(p "hello" (i ,name)))

'(div (@ (id "my-div-0"))

(span "This is a span in a div"))

`(div (@ ,(fresh-id))

(span "This is another span in a div whose id is dynamically generated"))

Return true if x is a xexp.

Return a dom from the xexp.

Return a string that can be used as a DOM node id.

5.5 Tips and tricks: Hiding standard output or directing it to an element

For a web-world program, output is normally done by usingto-draw. However, side effecting functions, such asprintf or display, are still available, and will append to document.body.

We may want to disable such printing or redirect it to a particular element on the page. For such purposes, use a combination ofcurrent-output-port and open-output-element to redirect the output of these side effect functions to somewhere else.

For example:

All subsequent I/O side effects after the call tocurrent-output-port will be written out to thestdout-div, which can be easily styled with display: none to hide it from normal browser display.

Opens an output port that will be directed to write to the DOM element whose id is id. Note: writing to this port shouldn’t fail, even if the id does not currently exist on the page.

6 The JavaScript Foreign Function Interface

[[This needs to describe what hooks we’ve got from the JavaScript side of things.

In particular, we need to talk about the plt namespace constructed by the runtime, and the major, external bindings, likeplt.runtime.invokeMains.

The contracts here are not quite right either. I want to use JQuery as the type in several of the bindings here, but don’t quite know how to teach Scribble about them yet.]]

Caveats: the lifted function does no auto-coersion of values.

Caveats: the lifted function does no auto-coersion of values.

Returns the width of the viewport.

Returns the height of the viewport.

6.1 Adding new event handlers to world programs with the FFI

The callback-driven asynchronous APIs in JavaScript can act as sources of World events. The following function allows web-world programs to bind to these sources.

The setup and shutdown functions are usually constructed with js-function->procedure in order to bind to native JavaScript APIs.

The setup function is called with an JavaScript function value that, when called, emits a new event into the world’s event loop. The return value of the setup function will be saved, and when the shutdown procedure calls, that value is passed to it, with the intent that shutting down a service will likely require information that’s produced at setup-time.

7 Simple world programming

Whalesong provides a library to support writing functional I/O programs (A Functional I/O System). Here’s an example of such a world program:

[FIXME: embed a world program here.]

8 Acknowledgements

Whalesong uses code and utilities from the following external projects:

The following folks have helped tremendously in the implementation of Whalesong by implementing libraries, giving guidence, reporting bugs, and suggesting improvements.

• Asumu Takikawa

• Cristina Teodoropol

• Doug Orleans

• Emmanuel Schanzer

• Eric Hanchrow

• Ethan Cecchetti

• Greg Hendershott

• Gregor Kiczales

• Jay McCarthy

• Jens Axel Søgaard

• Keith Decker

• Matthew Flatt

• Richard Cleis

• Robby Findler

• Sam Tobin-Hochstadt

• Scott Newman

• Shriram Krishnamurthi

• Zhe Zhang

Bruce Hoffman, the Acting Director of the Federal Trade Commission’s Bureau of Competition, issued this statement on the Commission’s decision not to further pursue an investigation of Amazon.com, Inc.’s acquisition of Whole Foods Market Inc.:

“The FTC conducted an investigation of this proposed acquisition to determine whether it substantially lessened competition under Section 7 of the Clayton Act, or constituted an unfair method of competition under Section 5 of the FTC Act. Based on our investigation we have decided not to pursue this matter further. Of course, the FTC always has the ability to investigate anticompetitive conduct should such action be warranted.”

The Federal Trade Commission works to promote competition, and protect and educate consumers. You can learn more about how competition benefits consumers or file an antitrust complaint. Like the FTC on Facebook, follow us on Twitter, read our blogs and subscribe to press releases for the latest FTC news and resources

ThoughtWorks, my employer, had some big news to share today. Our founder and owner, Roy Singham, has decided to sell ThoughtWorks to Apax Funds - a private equity firm based in London. Apax wishes the current management team to continue running and growing ThoughtWorks, using the same model that's driven our growth and success for the last twenty-odd years.

Bicyclists ride to commute, to exercise, for sport, for leisure. But no matter the reason, most cyclists will ride on the road at some point, and when they do, they’ll think about the safety and comfort of their route.

Should you take a longer route to ride on a road with a dedicated cycling lane? Do you take a left turn to avoid those difficult hills? Different cyclists will make different choices, as the ideal route for one bicyclist may not be so ideal for another.

Valhalla, the open-source routing engine that powers Mapzen Turn-by-Turn, has always had customization parameters. While these include use_roads and use_hills, setting these to zero didn’t quite define what most would consider a “low stress” bike route. At least not until now!

This summer, I and the Mapzen routing team enhanced Valhalla’s bicycle costing system. Not only is overall routing improved, but use_roads and use_hills are now much more versatile: when both of these parameters are set to 0, Valhalla generates a nice, “low-stress” biking route. As these parameters get closer to 1, it linearly increases to a more “professional” biking route.

Along with this new update, our default bike settings have also changed — use_roads and use_hills previously defaulted to 0.5, but now they default to 0.25. The default bike (and thus speed) has also been changed from a road bike to a hybrid bike, which is more suited for city riding.

Technical Details

The meat of this update is the added support for low-stress biking. Our original goal was to introduce a new “low-stress bike” costing class derived from the regular bike costing class. However, after doing this, we realized that the algorithms of the new LowStressBicycleCost class were actually cleaner and easier to manage than our original bike costing algorithms. This inspried us to rewrite the old bike code to look like the new low-stress bike code. This way, the use_roads and use_hills parameters acted as variables that scale the costs of edges (i.e. road segments) with particular attributes in a way that provides low-stress biking at one end of the spectrum, and professional road biking on the other.

When originally making the LowStressBicycleCost class, we made use of research by Michael B. Lowry at the University of Idaho, Peter Furth at Northeastern, and Tracy Hadden-Loh with the Rails-to-Trails Conservancy. Their paper titled “Low-Stress Neighborhood Bikeability Assessment to Prioritize Bicycle Infrastructure” provides a way to score a road based on how stressful it is to bike on. Here are the variables that are included in the score:

    stress_factor = accomodation_factor * road_stress

    edge_weight = 1.0 + stress_factor + slope_factor

• accomodation_factor is a value between 0 and 1, where 0 means the road is very well accommodated for biking and 1 is not accomodated for biking.
• road_stress represents how stressful it is to bike on the road due to things like road speed and lane count. This number can be greater than 1 — the higher it is, the more stressful the road segement.
• Multiplying these two values gets the total_stress value which is added with the slope_factor and 1 to get edge_weight.
• We multiply edge_weight at the end of our EdgeCost function to the estimated time it takes to travel the edge. This way, edges that have unfavorable attributes for our needs are less likely to be taken.

Due to the predictability this algorithm had over our previous one, we folded this code back into the original BicycleCost class and removed the LowStressBicycleCost class all together, leaving the control with use_roads and use_hills. As a result, our routing is better and our API has stayed simple and intuitive.

Examples

Here are a few examples of the “new and improved” versus the “old” bike routing (both with use_roads and use_hills set to 0).

Here the old bike route was skipping what is known as “The Wiggle” in San Francisco. The Wiggle is comprised of shared lanes in a zig-zag shape and is the flattest ways to move in that direction. With the low-stress settings, our new and improved bike routing now follows the entire Wiggle.

On this San Francisco route, the old system suggested taking Guerrero Street all the way to the right turn onto Cesar Chavez Street, but the new and improved system takes a shortcut through Tiffany Avenue, then onto the protected Valencia cycleway. While Guerrero has painted bike lanes, it is a very busy street with significant car traffic, while Tiffany Avenue is a numbered bike route, on a residential road classified as a “living street”. The new bike routing gives living streets and numbered routes much higher preference when calculating lower-stress routes.

In Arlington, Virginia, the old bike system here suggestes an odd zig-zag route to try and avoid busy streets like South Glebe Road. The new and improved biking system goes straight north, using a bike path and some residential roads to riders’ advantage. This is not only quicker but is also safer to ride.

Turn Costing

Turns are tricky business. In Valhalla, turns are penalized to a certain degree to keep routes from becoming ridiculous. However, with low-stress biking, there were certain areas (like the Wiggle and Tiffany Avenue) where we wanted more turns. In those situations, it’s worth taking a few more turns if it means you can ride along a nice dedicated cycle lanes or a calm side street.

So inside the transition costing, we had to penalize the turns onto low-stress roads a little less to allow more turns onto those roads. Still, we didn’t want to encourage too many unnecessary turns. After some tuning, we think we’ve struck a reasonable balance for turn costing.

(There aren’t similar issues with use_hills, since grade and elevation are calculated along edges, rather than at turning/transition nodes.)

Learn More

You can access the new and improved bike routes, as well as the “low-stress” options now in our Turn-by-Turn API and mapzen.js!

Or directly make an API request like this:

https://valhalla.mapzen.com/route?json= {"costing":"bicycle",,"costing_options":{"bicycle":{"use_roads":0,"use_hills":0}},"directions_options":{"language":"en-US","units":"miles"},"locations":[{"lat":37.799201,"lon":-122.399584},{"lat":37.7431177,"lon":-122.424318}]}&api_key=YOUR-API-KEY

In mapzen.js, just tell the built-in router to use bicycle costing, and set use_roads and use_hills to 0 if you want the lowest-stress route.

            router = L.Mapzen.routing.control({
                waypoints: [
                    L.latLng(sf_start),
                    L.latLng(sf_stop)
                ],
                router: L.Mapzen.routing.router('YOUR-API-KEY', {
                    costing: "bicyle",
                    costing_options: {
                        "bicycle": {
                            use_roads: 0,
                            use_hills: 0
                        }
                    },
                    directions_options: {
                        language: 'en-US',
                        units: 'miles'
                    }
                }),
            }).addTo(map);

Want to display your stress-free bike routes on an attractive map? Or want to browse bike lanes and paths, before you plan a particular trip? See Mapzen’s updated bike map.

In a sign of the soaring demand for zeroday attacks that target software that's becoming increasingly secure, a market-leading broker is offering serious cash for weaponized exploits that work against Signal, WhatsApp, and other mobile apps that offer confidential messaging or privacy.

Zerodium, the Washington, DC-based broker that launched in 2015, said on Wednesday that it would pay $500,000 for fully functional attacks that work against Signal, WhatsApp, iMessage, Viber, WeChat, and Telegram. The broker said it would start paying the same rate for exploits against default mobile e-mail apps. Those are among the highest prices Zerodium offers. Only remote jailbreaks for Apple's iOS devices fetch a higher fee, with $1.5 million offered for those that require no user interaction and $1 million for those that do. The jailbreak fees were announced in September 2016 and September 2015, respectively.

"Overall prices are trending up—and quite significantly in many cases, and there's an increased focus on mobile," Adam Caudill, a senior application security consultant at AppSec Consulting, told Ars. "The new $500k targets for messaging and default e-mail apps show what a priority attacking individuals via their devices has become (which makes sense, given the recent increase in state-sponsored malware targeting mobile devices via SMS and the like)."

One of the best-known state-sponsored attacks targeting a mobile device was exposed last year when a highly advanced iPhone attack targeted a political dissident in the United Arab Emirates. It combined three separate vulnerabilities that had remained unpatched in iOS to surreptitiously jailbreak an iPhone and install malware that could steal confidential messages from a large number of apps, including Gmail, Facebook, and WhatsApp.

The attack platform—which cost about $8 million for 300 licenses—was developed by NSO Group, an Israeli-based division of US-headquartered company Francisco Partners Management. Apple patched the vulnerabilities shortly after the attack came to light.

Zerodium has long been criticized by some as a broker that, like NSO Group, exposes dissidents, journalists, attorneys, and others to dangerous hacks by governments with no or little oversight. Zerodium officials reject that claim and say access to the exploits the company sells is tightly restricted and available to only a select number of carefully vetted organizations, mostly located in Europe and North America. Zerodium, however, has never disclosed its list of past or current customers, so the assurances are impossible to verify.

The new categories and rising prices in Wednesday's updated list underscore the growing demand for mobile exploits, as more and more targets of interest rely on phones to send text messages and e-mail. These exploits increasingly use specific high-value apps, such as Signal, that experts say are harder to hack. The hikes are also the result of a steady stream of security protections that, over the past few years, have made phones and the apps that run on them much harder to compromise.

The Zerodium update has other noteworthy mobile additions. For the first time, the broker said it's actively seeking attacks that exploit basebands, which are the parts of a phone that manage radio communications with a cell carrier. Over the past few years, hackers have increasingly targeted basebands because they often run proprietary firmware that's usually not as carefully secured as other parts of the phone. Despite being distinct from most phones' CPUs, baseband hacks can sometimes have ways of compromising the entire device. Zerodium said it will pay $150,000 for advanced baseband exploits.

Other mobile additions included $150,000 for media files or documents that can execute malicious code on a phone and $100,000 for files that bypass security sandbox protections or code-signing requirements or that exploit Wi-Fi functions. That $100,000 fee also applied to hacks that target Signalling System No. 7. The telephony signaling language that more than 800 telecommunications companies around the world use to ensure their networks interoperate is under increased scrutiny as a way to target mobile phone users. Thieves reportedly abused SS7 in May to drain bank accounts protected by two-factor authentication.

Other new exploit categories for servers and desktop computers include:

• Windows 10 that require no user interaction: $300,000
• Apache Web Server: $150,000
• Microsoft Outlook: $100,000
• Mozilla Thunderbird: $80,000
• VMware escapes: $80,000
• USB code execution: $30,000

Zerodium is also increasing the prices it will pay for a range of other exploits, including:

• Chrome, to $150,000 from $80,000
• PHP Web programming language, to $100,000 from $50,000
• OpenSSL crypto library used to implement TLS, to $100,000 from $50,000
• Microsoft Exchange Server, to $100,000 from $40,000
• The TOR version of Firefox for Linux, to $100,000 from $30,000
• The TOR version of Firefox for Windows, to $80,000 from $30,000

The prices can represent a significant sum in many cases, particularly when compared to many bug bounty programs, which pay only a fraction. Then again, the attacks Zerodium buys generally require more work because they must be fully functional, as opposed to less complex proof-of-concept exploits accepted by many bounty programs. The other big drawback to submitting to Zerodium: exploit developers don't know where their creations wind up or how, or against whom, they're used.

The prices also provide a portal into the exploit market. Besides increased appetite for mobile attacks, Wednesday's update shows growing demand for server-based attacks and those that work against users of the Tor privacy service. If end users didn't already have good reason to keep their devices updated and limit attack surfaces, they sure have it now.

As students have been celebrating their exam results, pundits from across the political spectrum have been commiserating the state of British universities. Andrew Adonis, an education minister during the Blair years, has excoriated universities for offering costly courses while jacking up the pay of their senior leaders. Nick Timothy, Theresa May’s ex-advisor, thinks UK universities are an unsustainable “Ponzi scheme”. The universities minister, Jo Johnson, has written about the need to put further pressure on seats of higher learning so students get good value for money.

Behind the political point-scoring are more serious issues. The university sector has been growing for decades, but now that growth is going into reverse. The number of undergraduates applying to universities has fallen by 4% this year. Although close to 50% of the population goes through higher education, only about 20% of jobs require an undergraduate degree. One US study found that 46% of students showed no improvement in their cognitive skills during their time at university. In some courses, like business administration, students’ capacity to think got worse for the first few years. And after they graduated, many struggled to find full-time work while being loaded down with debt. Nearly a quarter of graduates were living with their parents or relatives.

On top of all this, UK universities have some significant financial difficulties. The university pension scheme is £17.5bn in the red. Senior managers have been on a building spree that has been almost entirely funded by new borrowing on the bond market. Many institutions are locked into costly private finance initiatives.

Underlying all this bad news is an often overlooked fact. Universities have been growing for a decade, but most of the resources fuelling that growth have gone into expanding university administration, not faculty. One US study found that between 1975 and 2008, the number of faculty had grown about 10% while the number of administrators had grown 221%. In the UK, two thirds of universities now have more administrators than they do faculty staff. One higher education policy expert has predicted the birth of the “all-administrative university”.

The massive expansion of administration has also fuelled an equally stark expansion of empty activities. These include costly rebranding exercises, compliance with audits and ranking initiatives, struggling with poorly designed IT systems, engaging with strategic initiatives and failed attempts at “visionary leadership”. All the while, faculty are under pressure to show they are producing world-class research, outstanding teaching and are having an impact on wider society. No wonder some faculty complain that they are “drowning in shit”.

The expansion of empty administration has some up sides. By showing universities are willing to keep up with the latest management fads and fashions, they gain credibility in the eyes of business and government. Empty administration makes some members of the university feel good about themselves. But empty administration also comes at a significant cost. It is expensive, it is disheartening, and often it diverts universities from their core tasks. Instead of educating students, doing research and contributing to broader society, universities end up developing policies, ticking boxes and trying to climb up rankings.

If universities are interested in addressing the problems they face, what is not needed is spectacular reform. Such reforms are often costly, disturbing distractions with a short shelf life. What is needed is something more modest, but more far-reaching: cutting back empty administration.

The first step in cutting back empty administration is eliminating the demand. An important aspect of this is to remove creeping government attempts to micro-manage the sector. Putting an end to the research excellence framework would save the sector £250m. It would also eliminate pressure on faculty to publish obtuse articles which are read by few people. Killing off the new teaching excellence framework will immediately save the sector £20m, plus countless hours of staff time spent complying with the exercise.

The next step is to root out the supply of empty administration in universities. A modest first step would be the elimination of “bullshit jobs” in universities. These are jobs which the people doing them think should not exist. Creeping forms of corporate escapism in universities would also be wound back. This includes everything from fanciful strategy development exercises, managerial vanity projects like opening campuses in exotic locations and overly elaborate leadership retreats. Staff need to be given space to question and even veto any new administrative initiatives. When any new initiative is proposed, faculty need to ask: “Is there any evidence this works? What is the logic behind it? And is it meaningful to staff and students?” Answering these three simple questions is likely to cut back empty administration substantially.

Finally, universities need to stop rewarding the creation of empty administration. They can do this by not rewarding empty talk with attention. Simply switching off as soon as someone begins to use empty business jargon will mean people who present new ideas think them through thoroughly. They need to make stupidity costly. One way to do this is by requiring people to fully carry out their own fanciful ideas. When people have to implement an idea, they are likely to think twice before proposing it. Finally, universities need to make it costly for individuals to increase organisational load. For instance, before introducing a new procedure they would need to eliminate an old one.

Cutting back on empty administration comes with risks. For instance, powerful groups like politicians may see universities as “out of touch”. Accessing resources used to support empty administration can become more difficult. But cutting empty administration is likely to be worth it. Universities will no longer be burdened with the cost. Staff will feel like they are no longer being crushed by meaningless and wasteful processes. Students will no longer be faced with thickets of maddening processes. The institutions themselves are less likely to be diverted from their core tasks of educating students, carrying out research and contributing to the broader society. Finally, the public is more likely to trust higher education institutions which carry out their purpose of educating students, conducting research and contributing to wider society.

• André Spicer’s new book, Business Bullshit, is out in September

Principal Component Analysis, or PCA, is a well-known and widely used technique applicable to a wide variety of applications such as dimensionality reduction, data compression, feature extraction, and visualization. The basic idea is to project a dataset from many correlated coordinates onto fewer uncorrelated coordinates called principal components while still retaining most of the variability present in the data.

Singular Value Decomposition, or SVD, is a computational method often employed to calculate principal components for a dataset. Using SVD to perform PCA is efficient and numerically robust. Moreover, the intimate relationship between them can guide our intuition about what PCA actually does and help us gain additional insights into this technique.

In this post, I will explicitly describe the mathematical relationship between SVD and PCA and highlight some benefits of doing so. If you have used these techniques in the past but aren’t sure how they work internally this article is for you. By the end you should have an understanding of the motivation for PCA and SVD, and hopefully a better intuition about how to effectively employ them.

Before we discuss how to actually perform PCA it will be useful to standardize the way in which data is encoded in matrix form. Since PCA and SVD are linear techniques, this will allow us to manipulate the data using linear transformations more easily.

It’s simplest to start with an example, so suppose that you are given the width $w$, height $h$, and length $l$ measurements of $n = 1000$ rectangular boxes. We can encode the measurements of box $i$ as a $(w_i, h_i, l_i)$ tuple called a sample. Each sample is a vector in $d = 3$ dimensions, since there are 3 numbers describing it. Because vectors are typically written horizontally, we transpose the vectors to write them vertically:

$$ x_i = \left( \begin{array}{c} w_i \\ h_i \\ l_i \end{array} \right) \quad\text{and}\quad x_i^T = \left( w_i, h_i, l_i \right)\,. $$

To pack the data into a single object we simply stack the samples as rows of a data matrix,

$$ X = \left( \begin{array}{ccc} w_1 & h_1 & l_1 \\ \hline w_2 & h_2 & l_2 \\ \hline&\vdots& \\ \hline w_{1000} & h_{1000} & l_{1000} \end{array} \right)\,. $$

The columns of this matrix correspond to a single coordinate, i.e., all the measurements of a particular type.

In the general case, we are working with a $d$-dimensional dataset comprised of $n$ samples. Instead of using letters like $h$, $w$, and $l$ to denote the different coordinates, we simply enumerate the entries of each data-vector so that $x_i^T = (x_{i1}, \ldots, x_{id})$. Before placing these vectors into a data matrix, we will actually subtract the mean of the data

$$ \mu = \frac{1}{n} \sum_{i=1}^n x_i = (\frac{1}{n} \sum_{i=1}^n x_{i1}, \ldots, \frac{1}{n} \sum_{i=1}^n x_{id} )^T $$

from each sample for later convenience. We then use these zero-centered vectors as rows of the matrix

$$ X = \left( \begin{array}{ccccc}&& x_1^T - \mu^T && \\ \hline&& x_2^T - \mu^T && \\ \hline&& \vdots && \\ \hline&& x_n^T - \mu^T && \end{array} \right)\,. $$

See Figure 1 for a visual illustration of the case $n = 1000$, $d = 2$. Placing data into a matrix is particularly convenient because it allows us to write the sample covariance around the mean using matrices as

$$ S = \frac{1}{n-1} \sum_{i=1}^n (x_i-\mu)(x_i-\mu)^T = \frac{1}{n-1} X^T X\,. $$

Dividing by $n - 1$ is a typical way to correct for the bias introduced by using the sample mean instead of the true population mean. The covariance matrix will take center stage as we work through understanding principal components and singular value decomposition.

One of the first papers to introduce PCA as its known today was published in 1933 by Hotelling. The author’s motivation was to transform a set of possibly correlated variables into “some more fundamental set of independent variables … which determine the values [the original variables] will take.” Coming from psychology, his first choice for naming them was “factors,” but given that this term already had a meaning in mathematics he called the reduced set of variables “components” and the technique to find them “the method of principal components.” These components are chosen sequentially in a way that lets their “contributions to the variances of [the original variables] have as great a total as possible.”

Mathematically, the goal of Principal Component Analysis, or PCA, is to find a collection of $k \leq d$ unit vectors $v_i \in \mathbb R^d$ (for $i \in {1, \ldots, k }$) called Principal Components, or PCs, such that

1. the variance of the dataset projected onto the direction determined by $v_i$ is maximized and
2. $v_i$ is chosen to be orthogonal to $v_1, \ldots, v_{i-1}$.

Now, the projection of a vector $x \in \mathbb R^d$ onto the line determined by any $v_i$ is simply given as the dot product $v_i^T x$. This means that the variance of the dataset projected onto the first Principal Component $v_1$ can be written as

$$ \frac{1}{n-1}\sum_{i=1}^{n}(v_1^Tx_i-v_1^T\mu)^2 = v_1^TSv_1. $$

To actually find $v_1$ we have to maximize this quantity, subject to the additional constraint that $\| v_1 \|= 1$. Solving this optimization problem using the method of Lagrange multipliers turns out to imply that

$$ Sv_1 = \lambda_1v_1\,, $$

which just means that $v_1$ is an eigenvector of the covariance matrix $S$. In fact, since $\| v_1 \| = v_1^T v_1 = 1$ we also conclude that the corresponding eigenvalue is exactly equal to the variance of the dataset along $v_1$, i.e.,

$$ v_1^T S v_1 = \lambda_1\,. $$

You can continue this process by projecting the data onto a new direction $v_2$ while enforcing the additional constraint that $v_1 \perp v_2$, then onto $v_3$ while enforcing $v_3 \perp v_1, v_2$, and so on.

The end result is that the first $k$ principal components of $X$ correspond exactly to the eigenvectors of the covariance matrix $S$ ordered by their eigenvalues. Moreover, the eigenvalues are exactly equal to the variance of the dataset along the corresponding eigenvectors.

You may have noticed that this result suggests that there exists a full set of orthonormal eigenvectors for $S$ over $\mathbb R$. Indeed, since $S$ is a real symmetric matrix, meaning that $S = S^T$, the Real Spectral Theorem implies exactly that. This is a non-trivial result which we will make use of later in the article, so let’s expand on it a little bit. Consider the case of $k = d < n$. Taking $k = d$ principal components may seem like a strange choice if our purpose is to understand $X$ through a lower dimensional subspace, but doing so allows us to construct a $d \times d$ matrix $V$ whose columns are the eigenvectors of $S$ and which therefore diagnoalizes $S$, i.e.,

$$ S = V \Lambda V^T = \sum_{i = 1}^r \lambda_i v_i v_i^T \,, $$

where $\Lambda = \text{diag}(\lambda_1, \ldots, \lambda_d )$ and $r = \text{rank}(X)$. In other words, the principal components are the columns of a rotation matrix and form the axes of a new basis which can be thought of as “aligning” with the dataset $X$. Of course, this image works best when $X$ is blobby or looks approximately normal.

Singular Value Decomposition is a matrix factorization method utilized in many numerical applications of linear algebra such as PCA. This technique enhances our understanding of what principal components are and provides a robust computational framework that lets us compute them accurately for more datasets.

Let’s start with a review of SVD for an arbitrary $n \times d$ matrix $A$. SVD is motivated by the fact that when viewed as a linear transformation, $A$ maps the unit sphere $\mathbb S^d \subset \mathbb R^d$ to a (hyper)ellipse in $\mathbb R^n$. Let’s consider an example with $n = d = 2$ in order to more easily visualize this fact.

From this figure we can extract a few useful definitions which hold for arbitrary dimensions with a few extra caveats:

• The lengths $\sigma_i$ of the semi-axes of the ellipse $A\mathbb S^d \in \mathbb R^n$ are the singular values of $A$.
• The unit vectors ${ u_i }$ along the semi-axes of the ellipse are called the “left” singular vectors of $A$.
• The unit vectors $v_i$ such that $Av_i = \sigma_i u_i$ are called the “right” singular vectors of $A$.

Proving the existence and uniqueness of the singular values and singular vectors is a bit beyond the scope of this article, but the proof is relatively straightforward and can be found in any book on numerical linear algebra (see below for a reference). The two-dimensional image in Figure 2 hopefully provides enough guidance for the result to be intuitive, however.

What may not be immediately apparent from the picture is that depending on $n, d$ and $r = \text{rank}(A)$, some of the left singular vectors may “collapse” to zero. This happens when the matrix $A$ does not have full rank for instance, since then its range must be a subspace of $\mathbb R^n$ with dimension $r < d$. In general, there are exactly $r = \text{rank}(A)$ singular values and the same number of left singular vectors.

By stacking the vectors $v_i$ and $u_i$ into columns of matrices $\widehat V$ and $\widehat U​$, respectively, we can write the relations $Av_i = \sigma_i u_i$ as

$$ A\widehat V = \widehat U \widehat \Sigma\,, $$

where $\widehat \Sigma = \text{diag}(\sigma_1, \ldots, \sigma_r)$. By padding $\widehat \Sigma$ with zeros and adding arbitrary orthonormal columns to $\widehat U$ and $\widehat V$ we obtain a more convenient factorization

$$ A V = U \Sigma\,. $$

Since $V$ is unitary – that is, it has unit length, orthonormal columns – it follows that $V^{-1} = V^T$, so multiplying by $V^T$ gives us the singular value decomposition of $A$:

$$ A = U \Sigma V^T\,. $$

Things got a bit busy here, so here’s a visual summary of this result.

Note that this result is in the end consistent with the ellipse picture above since the effect of the three transformations is to switch to a new basis using $V^T$, stretch axes to get a (hyper)ellipse using $\Sigma$, and rotate and reflect using $U$ (which doesn’t affect the shape of the ellipse).

Since any matrix has a singular value decomposition, let’s take $A = X$ and write

$$ X = U \Sigma V^T\,. $$

We have so far thought of $A$ as a linear transformation, but there’s nothing preventing us from using SVD on a data matrix. In fact, note that from the decomposition we have

$$ X^TX = (U\Sigma V^T)^T(U \Sigma V^T) = V \Sigma^T U^T U \Sigma V^T = V (\Sigma^T \Sigma) V^T\,, $$

which means that $X^T X$ and $\Sigma^T \Sigma$ are similar. Similar matrices have the same eigenvalues, so the eigenvalues $\lambda_i$ of the covariance matrix $S = \frac{1}{n-1} X^TX$ are related to the singular values $\sigma_i$ of the matrix $X$ via

$$ \sigma_i^2 = (n-1) \lambda_i\,, $$

for $i \in \{1, \ldots, r\}$, where as usual $r = \text{rank}(A)$.

To fully relate SVD and PCA we also have to describe the correspondence between principal components and singular vectors. For the right singular vectors we take

$$ \widehat V^T = \left( \begin{array}{ccccc}&& v_1^T && \\ \hline&& \vdots && \\ \hline&& v_r^T && \\ \end{array} \right) $$

where $v_i$ are the principal components of $X$. For the left singular vectors we take

$$ u_i = \frac{1}{\sqrt{(n-1)\lambda_i}} Xv_i\,. $$

Before proving that these choices are correct, let’s verify that they at least make intuitive sense. From Figure 2 we can see that the right singular vectors $v_i$ are an orthonormal set of $d$-dimensional vectors that span the row space of the data. Since $X$ is zero centered we can think of them as capturing the spread of the data around the mean in a sense reminiscent of PCA. We also see that the column space of $X$ is generated by the left singular vectors $u_i$. The column space of a data matrix is a summary of the different coordinates of the data, so it makes sense that we’ve chosen $u_i$ to be a (scaled) projection of $X$ into the direction of the $i$-th principal component. These observations are encouraging but imprecise, so let’s actually prove that our choices are correct algebraically. The result will follow from this general fact.

Fact. Let $A = U \Sigma V^T$ be the singular decomposition of the real matrix $A$. Then,

$$ A = \sum_{i=1}^r \sigma_i u_i v_j^T\,. $$

Proof. Let’s prove this statement in a small example to simplify the notation. The general case will follow analogously. So, let $v_i = (v_{1i}, v_{2i})^T$ for $i = 1, 2$ and $u_i = (u_{1i}, u_{2i}, u_{3i})$ for $i = 1, 2, 3$. That is,

$$ \begin{align*} U \Sigma V^T&= \left( \begin{array}{c|c}& \\ u_{1} & u_{2} \\& \end{array} \right) \left( \begin{array}{cc} \sigma_1 & 0 \\ 0 & \sigma_2 \end{array} \right) \left( \begin{array}{ccc}& v_1^T & \\ \hline& v_2^T & \end{array} \right) \\& = \left( \begin{array}{cc} u_{11} & u_{12} \\ u_{21} & u_{22} \\ u_{31} & u_{32} \end{array} \right) \left( \begin{array}{cc} \sigma_1 & 0 \\ 0 & \sigma_2 \end{array} \right) \left( \begin{array}{cc} v_{11} & v_{21} \\ v_{12} & v_{22} \end{array} \right) \end{align*} $$

Next, split up the matrix $\Sigma$ so that

$$ U\Sigma V^T = U \left( \begin{array}{cc} \sigma_1 & 0 \\ 0 & 0 \end{array} \right) V^T + U \left( \begin{array}{cc} 0 & 0 \\ 0 & \sigma_2 \end{array} \right) V^T $$

and tackle the terms individually. One way to rewrite the first term is

$$ \begin{align*} U \left( \begin{array}{cc} \sigma_1 & 0 \\ 0 & 0 \end{array} \right) V^T&= \left( \begin{array}{cc} u_{11} \sigma_1 & \color{blue}{0} \\ u_{21} \sigma_1 & \color{blue}{0} \\ u_{31} \sigma_1 & \color{blue}{0} \end{array} \right) \left( \begin{array}{cc} v_{11} & v_{21} \\ \color{blue}{v_{12}} & \color{blue}{v_{22}} \end{array} \right) \\&= \sigma_1 u_1 v_1^T\,, \end{align*} $$

where the last step follows because the entries highlighted in blue (the second column of the first matrix and the second row of $V^T$) do not affect the result of the matrix multiplication. A similar calculation shows that the second term is $\sigma_2 u_2 v_2^T$, which proves the claim for the given example. Note that in general some of the singular values could be 0, which makes the sum go only up to $r = \text{rank}(A)$. $\square$

Let’s apply this fact to $X$. We have that

$$ \begin{align*} U \Sigma V^T&= \sum_{i=1}^r \sigma_i u_i v_i \\&= \sum_{i=1}^r \sqrt{(n-1)\lambda_i} \frac{1}{\sqrt{(n-1)\lambda_i}} X v_i v_i^T \\&= X \sum_{i=1}^r v_i v_i^T \\&= X\,, \end{align*} $$

where the last step follows from $I = V^T V = \sum_{i=1}^r v_i v_i^T$. We have thus established the connection between SVD and PCA.

Let’s explore a few consequences of this correspondence. From the definition of $U$ we already know that it projects and slightly scales the data onto the principal components. It is more satisfying, however, to derive this result as a general consequence of being able to write $X = U \Sigma V^T$, so let’s turn out attention to that. Multiplying the SVD of $X$ by $V$ we obtain

$$ XV = U\Sigma V^T V = U\Sigma\,. $$

From the left hand side we see that the $i$-th column of this matrix is the projection

$$ z_i = Xv_i = ((x_1 - \mu)^T v_i, \ldots, (x_n - \mu)^T v_i)^T $$

of the data onto the $i$-th PC. From the right hand side we see that this corresponds to $z_i = u_i \sigma_i$, which is what we were trying to recover.

We can extract another useful morsel by thinking about the scaling of $U$. Recall that each singular value corresponds to the length of a semi-axis of an ellipse that describes the column space of $X$. By dividing out the singular value (and multiplying by $\sqrt{n-1}$ to deal with the fact that we started with the covariance) we obtain a transformed dataset which is in some sense spherical:

$$ Y = \sqrt{n-1}\, U\Sigma^{-1}\,. $$

This process produces a unit variance dataset in a way that can be difficult to achieve by just centering and scaling.

As a final remark, let’s discuss the numerical advantages of using SVD. A basic approach to actually calculating PCA on a computer would be to perform the eigenvalue decomposition of $X^TX$ directly. It turns out that doing so would introduce some potentially serious numerical issues that could be avoided by using SVD.

The problem lies in the fact that on a computer real numbers are represented with finite precision. Since each number is stored in a finite amount of memory there is necessarily a gap between consecutive representable numbers. For double precision floating point numbers, for instance, the relative error between a real number and its closest floating point approximation is on the order of $\varepsilon \approx 10^{-16}$. Algorithms that take this limitation into account are called backward stable. The basic idea is that such an algorithm produces the correct output for an input value that’s within $\varepsilon$ of the value you actually asked about.

If $\tilde \sigma_i$ is the output of a backward stable algorithm calculating singular values for $X$ and $\sigma_i$ is the true singular value, it can be shown that

$$ | \tilde \sigma_i - \sigma_i | = O(\varepsilon \| X \|)\,, $$

where $\| X \|$ is a reasonable measure of the size of $X$. On the other hand, a backward stable algorithm that calculates the eigenvalues $l_i$ of $X^TX$ can be shown to satisfy

$$ | \tilde l_i - l_i | = O(\varepsilon \| X^TX \|) = O(\varepsilon \| X \|^2)\,, $$

which when given in terms of the singular values of $X$ becomes

$$ | \tilde \sigma_i - \sigma_i | = O(\varepsilon \frac{\| X \|^2}{\sigma_i})\,. $$

The end result here is that if you are interested in relatively small singular values, e.g. $\sigma_i \ll \| X \|$, working directly with $X$ will produce much more accurate results than working with $X^TX$. Luckily, SVD lets us do exactly that.

There is obviously a lot more to say about SVD and PCA. If you’d like to know more about ways to extend these ideas and adapt them to different scenarios, a few of the references that I have found useful while writing this article are Numerical Linear Algebra by Trefethen and Bau (1997, SIAM) and Principal Component Analysis by Jolliffe (2002, Springer).

If you need a custom data sourcing, transformation and analysis solution, get in touch with us to see how our experts can help you make the most out of your data.