When you're playing your favorite game on Dolphin with a powerful computer, things should run fairly well. The game is running full speed, there are no graphical glitches, and you can use your favorite controller if you want. Yet, every time you go to a new area, or load a new effect, there's a very slight but noticeable "stutter." You turn off the framelimiter to check and your computer can run the game at well over full speed. What's going on?

The slowdown when loading new areas, effects, models, and more is commonly referred to as "Shader Compilation Stuttering," by users and developers alike. This problem has been a part of Dolphin since the very beginning, but has only recently become more of a focus.

When games barely ran at all, a little stutter here and there wasn't a big deal. Though emulation has improved to near perfection in many titles, the stuttering has remained the same over the years. Since the release of Dolphin 4.0, users have actually complained about shader compilation stuttering at an increasing rate even. While some of this may be partially due to increased GPU requirements from integer math, the bigger cause was actually that the stuttering stuck out more with there now being fewer serious issues otherwise.

There was some frustration and even antipathy from the developers toward shader compilation stuttering. It was something that was deemed unfixable and was garnering a lot of ill will and frustration within the community. Ironically, we hated the stuttering as much as anyone else, but the sheer insanity of the task was enough to keep most developers away. Despite this, some still privately held onto a glimmer of hope. It started out as a theory that had a chance of working. A theory that would take hundreds, if not thousands, of person-hours just to see if it was possible.

That hope is what fueled an arduous journey against seemingly impossible odds. A journey that would take multiple GPU engineers across two years. All in an effort to emulate the full range of the GameCube/Wii's proto-programmable pipeline without falling victim to this pesky stuttering.

This is the dawn of the Ubershader era.

The Problem¶

Modern GPUs are incredibly flexible, but this flexibility comes at a cost - they are insanely complicated. To unlock this power, developers use shaders - programs that the GPU runs just like a CPU runs an application - to program the GPU to perform effects and complex rendering techniques. Devs write code in a shader language from an API (such as OpenGL) and a shader compiler in the video driver translates that code into binaries that your PC's GPU can run. This compiling takes processing power and time to complete, so modern PC games usually get around this by compiling shaders during periods in which framerate doesn't matter, such as loadtimes. Due to the number of different PC GPUs out there, it's impossible for PC games to pre-compile their shaders for a specific GPU, and the only way to get shaders to run on specific PC hardware is for the video drivers to compile at some point in the game.

Flipper, the GameCube GPU, is the largest chip on the motherboard.
Image Credit: Anandtech

Consoles are very different. When you know the precise hardware you are going to run the game on, and you know that the hardware will never change, you can pre-compile GPU programs and just include them on the disc, giving your game faster load times and more consistent performance. This is especially important on older consoles, which may not have enough memory for or possibly even the capability to store shaders in memory. Flipper, the GameCube GPU, is the latter.

While it has some fixed-function parts, Flipper features a programmable TEV (Texture EnVironment) unit that can be configured to perform a huge variety of effects and rendering techniques - much the same way that pixel shaders do. In fact, the TEV unit has very similar capabilities to the DirectX 8 pixel shaders of the Xbox! It was so flexible and powerful that Flipper was reused as the Wii GPU (redubbed Hollywood) with few modifications. Unfortunately for us though, the TEV unit is designed for the game to configure and run TEV configurations immediately when an effect is needed. There is no preloading of the TEV configurations whatsoever, since the TEV unit doesn't have the memory for that.

That instantaneous loading is the source of all our problems. Dolphin has to translate each Flipper/Hollywood configuration that a game uses into a specialized shader that current GPUs can run, and shaders have to be compiled, which takes time. But the TEV unit doesn't have the ability to store configurations, so GC/Wii games must configure it to render an effect the instant it is needed, without any delay or notice. To deal with this disparity, Dolphin's only option is to delay the CPU thread while the GPU thread and the video driver perform the compilation - essentially pausing the emulated GC/Wii. Usually the compilation will take place in under a frame and users will be none the wiser, but when it takes longer than a frame, the game will visibly stop until the compilation is complete. This is shader compilation stuttering. Typically a stutter only lasts a couple of frames, but on really demanding scenes with multiple compiling shaders, stutters of over a second are possible.

As the first emulator to emulate a system with a highly programmable GPU at full speed, Dolphin has had to go it alone at tackling this problem. We implemented shader caching so if any configuration occurred a second time it would not stutter, but it would take hours of playing a game to build a reliable cache for it, and a GPU change, GPU driver update, or even going to a new Dolphin version would invalidate the cache and start the stuttering all over again. For years, it seemed like there was nothing more we could do about shader compilation stuttering, and many wondered if it would ever be solved...

Solving an Impossible Problem¶

Of all of Dolphin's remaining issues, shader compilation stuttering is the most complained about. Whether it be on the issue trackers, forums, social media, or IRC, this problem comes up all the time. Over the years, the reaction has shifted. At first, this stuttering was ignored as a non-issue. What did it matter if there was a slight stutter here and there if games barely ran at all in the first place? Things shifted in January of 2015, when this stuttering was formally accepted as a bug on Dolphin's issue tracker, and awareness spread.

Over the past few years, we've had users ask many questions about shader stuttering, demand action, declare the emulator useless, and some even cuss developers out over the lack of attention to shader compilation stuttering. The truth is that we hated the stuttering as much as anyone else, and we had thought about the problem for many years. Tons of solutions had been pondered, some even attempted. It just didn't seem possible to fix without serious side-effects.

The Potential Solutions¶

Generate All the Shaders Beforehand!¶

As a reference, there are a mere 7.5 × 10^¹⁵ grains of sand on Earth.

Dolphin is pretty fast at generating the shaders it needs, but compiling them is a problem. But, if we could somehow generate and compile shaders for every single configuration, that would solve the problem, right? Unfortunately, this is simply not possible.

There are roughly 5.64 × 10^⁵¹¹ potential configurations of TEV unit alone, and we'd have to make a unique shader for each and every configuration. Vertex shaders are also used to emulate the semi-programmable Hardware Transform and Lighting unit, and this raises the number of combinations even higher.

Even if we were able to compile them, these shaders would only be usable on the version of Dolphin they were generated on. Upgrading to a new build would require a new set of shaders. Other necessary occasions like upgrading your graphics card or upgrading your graphics drivers would also necessitate a recompile. And all of this relies on the driver having a low-level cache, which not all drivers do.

Predict What Shaders the Game Needs!¶

If we could just generate and compile shaders during loading screens and whatnot, there wouldn't be any stuttering when it mattered. Trying to predict what the game wants to do simply isn't feasible to a degree that would solve this problem. The performance and implementation implications around having Dolphin try to "see ahead" either by fastforwarding and predicting inputs cost way too much for the situations that they could possibly help.

Blind prediction doesn't work either - a game can choose to run whatever configurations it wants without any warning, and past configurations don't tell us anything about future configurations. The only way to know what shaders a game would need would be go through a game and find out every configuration it could possibly want.

...Which leads us to the next proposed solution.

Sharing Shaders¶

Dolphin uses a "Unique ID" object, or "UID" to represent a configuration of the emulated GPU, and these UIDs are then turned into shader code and handed to the video driver for compilation. Because UIDs are before compilation and have not been tailored to any specific PC GPU, they are compatible with any computer and could theoretically be shared. Users refer to this as "sharing shaders" and in theory if users shared UID files, they could compile shaders ahead of time and not encounter stuttering. Currently, the Vulkan video backend already has this feature as was necessitated to avoid shader caching issues on certain drivers.

So why hasn't extending this solution been pursued?

• Dolphin is still improving. If a graphics fix is merged, all of those UIDs may have to be thrown out.
• Not all games will be serviced. While popular games may get near complete UID collections, people playing hidden gems probably won't get any help.
  • In testing, there is very little UID overlap between games. The Legend of Zelda: The Wind Waker and The Legend of Zelda: Twilight Princess do share a small portion (15%) of configurations, but they are both running on the same base engine. Most games will have far less in common with each other, so sharing popular games will definitely not benefit lesser known games.
• Users may miss various UIDs. There are a near limitless number of configurations. Even 100%ing a game isn't a guarantee that you've hit every configuration.

Developers pondered this idea for a while, but building the infrastructure for sharing UIDs and finding a good way to distribute them proved to create more disagreements than solutions. While this could possibly be used to improve an already working solution, it is not a working solution on its own.

Asynchronous Shader Compilation¶

Popularized by a fork, asynchronous shader compilation is a creative solution to the shader compilation dilemma. Tino looked at the problem more like how some some modern games handle the same issue of having to dynamically compile new shaders - when you spawn into a new area, sometimes new objects will just "pop" in as they are loaded. He wondered if he could achieve something similar in an emulator and began rewriting how shaders were handled in his fork.

The asynchronous shader compiling concept changes how Dolphin behaves when there isn't a cached shader for an encountered Flipper/Hollywood configuration. Instead of pausing the game and waiting for a shader to compile, it simply skips rendering the object. This means that there is no pause or stutter, but some objects may be missing from view until the shader is ready.

This works well for some games. Depending on how the game's engine culls objects when drawing the world, objects that fall outside the field of view of the camera, or only cover a few pixels on-screen may still be rendered. In this case, skipping rendering of these objects is hardly noticable. However, depending on the game, it can result in the "pop in" described earlier.

One of the things users wondered was why Dolphin didn't at least implement Tino's asynchronous shaders as an option to fight shader compilation stuttering. In the end, it just came down to the fact that the people who could have implemented it along with other core developers were against it as a solution. They saw it as nothing more than a hack that would cause a lot of false positives on the issue tracker and cause bigger issues down the road. Those worries were proven somewhat valid when you realize that some games need objects to be rendered on the frame they expect it to be. The EFB Copies used to make Mii heads come up a bit empty if the objects aren't rendering due to async shader compilation!

Despite its flaws, users of Tino's fork swear by asynchronous shader compilation. For everything wrong with asynchronous shaders, they do solve the problem of shader compilation stuttering at all costs. The stark downsides were too steep for it to be merged into Dolphin master, but, this solution definitely brought the spotlight on how shader generation compilation was a big problem. Tino's work on asynchronous shader compilation really let us know how much users cared about this problem, and further motivated the team to come up with a more complete solution.

The Solution¶

Write an Interpreter for the GameCube/Wii Rendering Pipeline within Shaders and Run it on the Host Graphics Card¶

Sometimes, one of the best ways to solve an impossible problem is to change your perspective. No matter what we tried, there was no way to compile specialized shaders as fast as games could change configurations.

But what if we don't have to rely on specialized shaders? The crazy idea was born to emulate the rendering pipeline itself with an interpreter that runs directly on the GPU as a set of monsterous flexible shaders. If we compile these massive shaders on game start, whenever a game configures Flipper/Hollywood to render something, these "uber shaders" would configure themselves and render it without needing any new shaders. Theoretically, this would solve shader compilation stuttering by avoiding compilation altogether.

This idea is all kinds of crazy, but it was also the first idea that had the potential to actually solve this impossible problem. The difficulty with this solution instead came from the absurd amount of work and expertise required to even get to the point of trying it. To put it into perspective, even among all the developers that work on Dolphin, only two or three people at most have the necessary knowledge on not only the GameCube/Wii hardware, but also modern GPUs, APIs, and the drivers to write, debug, and optimize the shaders. Not to mention running an interpreter as huge shaders is not exactly easy on the GPU, and many were afraid that all that work might not even run full speed on current video cards.

Hundreds, if not thousands of hours of mindnumbing, repetitive, yet difficult work were needed with no guarantee of any payoff.

It was only first attempted in 2015, when phire became so frustrated with the shader compilation stuttering on his brand new computer that he actually made a proposal and designed the framework for an ubershader. While he was well aware of the difficulties, he seemed intent on proving that Ubershaders were the solution to this age old problem. phire went in all alone in an attempt to teach Dolphin how to render all over again.

After grinding at the feature for more than a month, he managed to get the pixel Ubershaders to the point where some games started to look like their fast-shader counterparts. The surprising part wasn't that it worked, but that the prototype Ubershaders actually ran full speed. phire himself recollected that his initial reaction consisted of, Holy shit, it's actually running at full speed and further admitted that GPUs shouldn't really be able to run these at playable speeds, but they do. Despite all the odds stacked against them, the prototypes only proved the Ubershaders could be our solution to shader compilation stuttering. And thus, the grind began to improve the accuracy of Ubershaders, fix the many bugs, and implement the missing features.

The effort to even get Ubershaders this far left phire completely exhausted with the project. On top of that, phire had to put in a ton of work cleaning up other projects for the Dolphin 5.0 release. The delays proved costly, as he lost his fire to continue working on ubershaders thanks to burnout and increasing worries about driver and API limitations regarding the solution. Despite being around 90% complete, the last 90% still remained to be done, including some key features.

• Finishing the vertex Ubershaders
• Infrastructure/linking pixel and vertex Ubershaders
• Solving OpenGL and (after a rebase) Vulkan performance issues
• Cleanups, bug fixes, and making rendering identical to the specialized shaders
• GUI Options
• Optional - Hybrid Mode for integrated/weaker GPUs

To see them on the cusp of working was painful. But, there weren't any developers capable of working on it with the will to take on such a massive project. Even those who would have considered working on it weren't ready to take on the cleanups, bug fixes, and infrastructure work. For well over a year, Ubershaders sat and bitrotted on the backburner within an evergrowing list of features that were never finished, and hope began to fade once again...

Ubershaders 2.0¶

Shader compilation stuttering is one of the most complained about bugs in Dolphin, so after Ubershaders development stopped, people didn't forget about it. The pull request, though long abandoned, still saw comments, got linked around the forums and even posted on the bug tracker in various forms.

Ubershaders was the first real hope to eliminate shader compilation stuttering, and it was still brought up on a monthly basis. If anything, the progress on it only inflamed the community's desire for a solution. After much pleading, begging, and much, much blackmail honest coercion, Stenzek reluctantly took over the mantle of Ubershaders.

Even before Stenzek began working on Ubershaders, the team made some decisions toward maintainability of the graphics backends. One of those decisions that was met with a mixed, if not negative, reaction was the removal of the D3D12 backend. Unlike D3D9, we didn't go through a deprecation process; we removed it once it was obvious no one was going to maintain it.

This was a fortuitous decision however, as the removal of that backend aided with the rebase and revival of Ubershaders when Stenzek was ready to give it his best shot. As he was the architect of Dolphin's Vulkan backend, he was already more than willing to go through the extra work to setup ubershaders to work on Vulkan.

When the pixel and vertex Ubershaders were finally hooked up together and ready for a run, testers immediately took them to some of the worst case scenario titles. Considering that none of the previous solutions really worked for a game like Metroid Prime 3, it was first on the docket.

The initial Ubershaders test was a massive success, with stuttering completely eliminated in D3D, and only a few strange stutters early on in runs within OpenGL and Vulkan. Continued work on Ubershaders has made things better in each backend, with a few exceptions that we'll note later. But, just running games on Ubershaders wasn't the end-game; pure Ubershaders are a massive performance drain on the host graphics card. While each game's requirements will vary, your graphics card will greatly affect how high of a resolution you can run. At 1x Internal Resolution (480p) most dedicated GPUs should be able to get the job done, with higher end cards still able to push 1080p or higher even exclusively using Ubershaders. Unfortunately, many of our users don't have the necessary hardware to run Ubershaders at the resolution they'd prefer, which would put them in the unfortunate position of choosing between resolution and smoothness.

A very large portion of Dolphin's users are running onboard graphics. In our testing, onboard solutions at best could get roughly 50% speed with Ubershaders at 1x IR in a typical 3D game! Developers felt like ignoring a huge group of Dolphin's users would be a mistake and make Ubershaders a limited victory at best. Thus work continued on an even more robust solution that would cure these performance ailments once and for all.

Hybrid Mode Ubershaders¶

Hybrid Mode Ubershaders is a marriage of Ubershaders and Asynchronous Shader Generation into a beautiful solution that takes the best parts of both with none of their flaws. Because Hybrid Mode greatly reduces the performance cost of Ubershaders, we expect it to be the most commonly used Ubershader mode.

Under Hybrid Mode, whenever a new pipeline configuration appears, Dolphin will use the already compiled Ubershaders to immediately render the effect without stuttering while still compiling the specialized shader in the background. Once the specialized shader is done, Dolphin will then hand the objects rendering through the Ubershader over to these newly generated specialized shaders.

Assuming that drivers and APIs behave the way we want, this is the perfect solution. Because Ubershaders are only running on a fraction of the objects on a scene and only for frames at a time, the performance hit is almost entirely negated and stuttering is completely eliminated. Unfortunately, drivers and APIs aren't perfect, limiting the effectiveness of Hybrid in some setups. That brings us to...

Ubershaders API and Driver Hall of Shame¶

GPU driver teams have a tough job squeezing as much power as possible out of their products while also providing a stable experience for their users. We mean no disrespect to anyone who works on these drivers, but one of the biggest obstacles to this project has been a ridiculous number of driver and API quirks that forced workarounds and other changes in functionality.

By bringing these to light, we hope to get some attention on them. Maybe someone outside of the project can come up with a workaround for us or at least monitor it in case a future driver/API update fixes the issue listed.

Shader Variant Generation¶

Drivers can do things in ways we don't expect and cannot control. When we have to generate a new pipeline for a different blend or depth states, some drivers aren't smart enough to share the shaders between pipelines. This will cause a minor stutter the first time a new blending mode is used. Most of the time the variants a game will use are generated within the first few minutes of play, but it can still be frustrating when you're seeking perfection.

Thankfully, some drivers are smart enough to share shaders between pipelines, such as the Mesa driver, and there appears to be no additional stuttering. All of the other available drivers appear to suffer from some form of stuttering during variant generation. While we can't do anything about this currently, we're hopeful that as Vulkan drivers mature, they'll take on Mesa's more favorable behavior.

NVIDIA Shader Locking on OpenGL and Vulkan¶

Some users have reported that on OpenGL and Vulkan (particularly in Hybrid Mode) there is some very slight stuttering when shaders are compiled. While we're not sure exactly what's wrong, because this does not happen in D3D, we're fairly certain that it's a quirk in NVIDIA's driver rather than a fault in how Dolphin handles things. Based on our testing, this appears to be separate from variant generation.

NVIDIA's Compiled Shaders on OpenGL and Vulkan are Much Slower than D3D¶

This one is particularly frustrating as there is no great way for us to debug this. We're feeding the same shaders to the host GPU on OpenGL, Vulkan and D3D, yet, D3D ends up with shaders that are much faster than the other two backends. This means that on a GTX 760, you may only get 1x internal resolution in a particular game on OpenGL or Vulkan, but on D3D, be able to comfortably get double or even triple before seeing slowdown.

Since NVIDIA does not allow us to disassemble shaders despite every other desktop GPU vendor having open shader disassembly, we have no way to debug this or figure out why the compiled code is so much more efficient on D3D. Think about how ridiculous this is: we want to make Dolphin run better on NVIDIA and they don't provide the tools to let us even attempt it. It's a baffling decision that we hope is rectified in the future. Without the shader disassembly tools provided by other vendors, fixing various bugs would have been much more difficult.

The sad thing is, the tools we need do exist - - if you're a big enough game studio.

AMD's Vulkan Driver still Lacks Shader Cache Support¶

During the writing of this article, our wishes were answered! AMD's Vulkan driver now supports a shader cache! This greatly improves what was a dire situation with Ubershaders, as it meant we'd have to recompile the Ubershaders every single run. It also improves variant stuttering as mentioned above.

macOS Graphics Drivers are Still Terrible¶

As with any exciting feature, macOS users were probably waiting for the inevitable "but on macOS..." and here it is. The outdated, inefficient OpenGL 4.1 drivers on macOS simply aren't up to the task of handling Ubershaders to any useful degree. Hybrid Mode will reduce stuttering, but, Exclusive is too slow to be useful. As another downside macOS still doesn't support a shader cache under any drivers.

Recommended Ubershader Settings¶

With all of the driver issues above, it isn't a surprise that some graphics cards work better on some backends and settings. We've outlined general recommended settings based on various video cards. Depending on your preference or particular graphics card, you may wish to deviate from the recommendations. Do note that changing certain settings while a game is running, such as per-pixel lighting and anti-aliasing level, will require different ubershaders to be compiled and may cause a sizable pause while this is done. Also remember that Ubershaders require more GPU power, so those same settings will also require a beefier graphics card.

In Conclusion¶

It feels strange to be talking about the ubershader project in past tense now. It is completed, it is merged, and you can use it right now in the latest development builds. While there may be some growing pains here and there, we finally have our solution to shader compilation stuttering. Ubershaders are going to get better over the years as graphics cards get stronger and Exclusive Mode can get more widespread use. Hybrid Mode should also get better as Vulkan drivers mature and other driver quirks are hopefully addressed. And, of course, we're going to continue working on our side to make sure emulation continues to improve.

While shader compilation stuttering is effectively solved, Dolphin still requires the host computer to be fast enough to emulate the game. Additionally there are some flaws in the JIT that can cause stuttering. Currently, Dolphin's JIT with branching support really struggles on games that use a JIT (such as N64 VC games), causing a stutter that feels like shader compilation stuttering but actually is not. While we were hoping to solve that issue, we'll likely be including an option to disable branching support if this cannot be rectified, so users have the power to turn it off for problematic titles.

Because of this giant article, we're not going to be doing a Progress Report this month. We will have July's many amazing changes combo'd into the August Progress Report. It's going to be a big one, so look forward to it! Until then, enjoy.

When I’ve started writing my stories on Medium it felt really​ good. Medium was the perfect “medium” to share your thoughts with all your followers. However in couple of years it changed a lot and I didn’t liked those changes. What were these?

1. Setting up a domain was difficult. You had to sent them an email and wait for it for couple of days. Even with that, you couldn’t create domains for your own profile, you had to create a “Publication“, which is a profile that can be used by multiple authors. The whole thing was just confusing and I think it was deliberately done to prevent personal users using their own domains. To make it even worse they started to charge a one time fee of $75 (which I think is totally ok, but read more why it doesn’t make sense when combined with other issues).
2. Content on Medium is great to read. The content just flows. I really feel how they poured hours of thinking into their designs. But Medium also has the power to change it suddenly. It slowly started to adding more and more social media buttons on their website. The design that once clean and simple, started to feel heavy. And because they “have to” grow they started to experiment  changing their famous design to increase the engagement (and therefore user growth). One of the end products of this way is that they  started using the infamous “dickbars”. I won’t go into details here much as John Gruber already ranted pretty well about it, but I just want to add that I truly hate it.
3. Lack of readership communication. Writing comments to Medium posts feels awkward because each comment is treated as a blog post. Even if you just write couple of words, that particular comment will now be seen in your profile page forever. Medium discourages the readers to comment on any of the blog posts. The whole system is created in way that discouraged active sharing of knowledge. Say you decided to write anyway. In this case the frustrating behavior is that nobody can see the response of your comment if I  didn’t follow you back. You have to make an additional click just to see them. I don’t like this at all.
4. Private sharing and reviewing never worked well. I’ve had several peer reviews of my posts and every single time the highlighted text would just disappear. I thought it was a great idea to get feedback before publishing it but It never worked well. Still they have great little touches (for example it shows a message that thanks to the reviewers of your post).
5. Lack of a front page for your own profile. If you tried to jump to your own profile page, you’ll find out that you have to follow multiple URL’s get to your profile page. Just clicking on your avatar doesn’t work. The bad part though is that you don’t have the ability to change your profile page in a meaningful way. Medium decides how to show your own content. And as I said before it’ll show your comments made on other blog posts as they were posted as a story itself (which is not!). Again the whole system is designed to make individuals powerless and just to promote constant reading.

I know that Ev tries to establish and create a vision around Medium. However, It seems like I’m not a part of his vision. I know that many others like it and are totally ok with.

I was ok to pay a monthly fee to get a better features that promotes individuals and fixes some of the problems above. But that never happened. And seems like it’ll never happen in foreseeable future.

So here I am. I’ve moved all my Medium posts to WordPress.com (and selected the plan to use my own domain). There are several reasons why I’ve opted for this (which might be coming odd for some of you), but the number one reason was that I didn’t wanted to manage anything about my blog. I know things like hugo, jekyll, etc.. is super easy to setup, work great and are highly customizable, but they have their own tradeoffs as well (asset management, commenting, etc…). WordPress just works. It’s simple and and they have years of experience running a profitable blogging business. Plus I like the company (hint: Automattic) behind WordPress 🙂

This is how my new home page looks like. I usually add the screenshots of my blogs to see how it evolves in the years of usage. This time I’ve opted for something more clean and minimalist (more on this later here!). I’ve started to travel more minimal and over the past year made many choices to be better at this. The design reflects some of it as well.

A Look into NASA’s Coding Philosophy

Having done work for Kennedy does not imply I’m NASA’s mouthpiece. This write-up was made at a personal capacity, and came from a live talk I gave on Twitch.

Who, out of all of us, speaks the truth?

The World of Modern Software

There’s a recurring theme in the programming community that’s tied to finding “better ways” to write “modern software.” Whether or not the term “modern” is actually any useful — computer programming hasn’t been around for very long — I’m definitely left with the impression folks always have something “new” or “better” to say on the subject. And so if we pay attention to the conversations surrounding software development today, we’ll quickly realize how important it is to separate the wheat from the chaff: what’s useful and what isn’t.

NASA has to Cut through the Bullshit

Given all of this modern software talk, I have seen beginner programmers asking what I think is the most important question of their careers: “Which things should we pay attention to?”, and honestly, it’s generally quite cruel to encourage them to follow that up with:

• Should OOP be the future of programming?
• Let’s join the Rust Evangelism Strikeforce LUL
• Is Go like totally better than<language_with_other_design_goals>?

At least, that’s not how NASA set out to answer it when it had asked that question too. The story of the agency is well-known, which is that the space program may suffer irreversible consequences if any of its software is incorrect, among them death. The shock of that reality has allowed them to develop a certain attitude towards programming. It could be instructive to take a look at what they value.

With some years of work there, I wish to provide a first-hand account on the philosophy that’s allowed the space agency to produce some of the world’s most reliable software, and I’ll frame their attitude towards programming with a set of four assumptions I think they make for programmers in the workforce (and which I have experienced directly.)

NASA’s Four Assumptions

You will have Access to a Mentor

“Who’s your mentor?” is a common ice-breaker with any pair of sweaty NASA programmers avoiding eye contact, and it’s thanks to the agency’s commitment to making sure each programmer can have a mentor. It could be at the level of business, with the OSBP’s Mentor-Protégé program, the internal NEXT project, or with the Pathways Agency Cross-Center Connection (PAXC). Interestingly, you don’t need to go out of your way to get access to one, because we’re all assigned personal mentors from the very beginning. The idea that a programmer can’t have someone capable and with senior experience to overlook their individual progress is close to horrifying for the space program.

We Trust Each Other’s Potential

There’s a reasonable degree of trust when entering the workforce due to the federal background investigations you’re subjected to. It is also assumed, however, that everyone has something valuable to offer, and we trust the potential of the individual. Show that you’re good and the title becomes almost meaningless. It’s not uncommon for an intern to contribute to high-profile NASA software projects.

You’ll Say “I Don’t Understand”

This statement is thrown around left-and-right. You will have knowledgeable senior developers asking interns “I don’t understand this part of the codebase. Is that a language feature I don’t know about yet?” or management asking engineers “I’m not sure what the implications of your work today are. Could we go over it a few times until I have a good grasp of it?” These questions are at the core of their daily work routine.

We Value Insights into how Computer Systems Operate

NASA hungrily searches for insights into how computers work, and they find value in discovering fundamental system limitations to prevent backing into bad software development decisions.

Consider Gene Amdahl. He was an early computer pioneer with an interesting inquiry: what’s the effectiveness of improving one component of a system? Say that fifty percent of your program’s running time was improved by a factor of four (wow!) and now we want to know how fast the application is. In other words, something took time T1 to run to completion, and now a part of that something (which takes some fraction f from T1) is k times faster. I’ll spare you any math, although it’s very simple — define the new time T2 using T1, f, k, and express the relative performance increase as a ratio of T1 and T2. You’ll end up with the following equation:

Let Amdahl’s Law sink in... Going with the earlier example, to know how fast your application is now simply apply the formula:

This gives 1.6. Though you took half of your entire program, and optimized it to be four times faster, the overall system is only 1.6x faster. Let’s shock ourselves further. Say you take some other component and the time it takes to run it is now zero (k tends to infinity). You get a special case of Amdahl’s Law:

Assume that component comprises 60% of your program’s running time, and it now takes effectively no time to run it. The theoretical maximum speedup is 2.5… Amdahl’s Law is telling us that to significantly improve the speed of a system, we’ll have to improve a very large fraction of it. Simple truth, but easily forgotten. The law is used often in parallel computing to predict speedups using multicore processors, and Julian Browne reformulates it in that context: “Essentially the law states that whilst a process can be decomposed into steps which may then be run in parallel, the time taken for the whole process will be significantly limited by the steps that remain serialised.” (source)

With these assumptions in mind, the rest of the article gives a more personal view on what lessons could be gleaned from them.

Avoid Unreliable Sources of Knowledge

Some of my friends agree the programming profession hasn’t matured yet because it’s at a pre-scientific stage of its existence. I claim many of us are subconsciously subscribing to traditional sources of knowledge that have proven to be unreliable, and as far as I can tell NASA has done well to guard themselves against them:

Faith

I don’t mean faith in the sense of hope. Faith is also used when we want to believe in something with no good reason to believe in it:

Lead: “I like our app’s new features for our customers, and you said you downloaded a library to write them?”

Junior: “Yeah. It’s worked fine so far.”

Lead: “Is it open source? Otherwise who is the vendor and how good are they on support? Have you proven it to be robust and tested it across our parameter space?”

Junior: “No but I’ve been using it for a while and it hasn’t caused any issues.”

Lead: *PLONK*

This happens not just for random downloads, but when your code wasn’t working but then it magically did and you decide to call it a day. It may be difficult to notice when someone is relying on faith to decide how good their software is, but if it drags on long enough it will lead to slow and buggy software.

Revelation

I mean revelation when a figure of authority reveals something as incontrovertibly true, or some fact has been passed down from earlier times, and you believe it to be true because it has been believed to be true. If you tie it with the need for people to identify with some random subset of the programming community, it’s the root cause of heated arguments about which text editor reigns supreme, and how much your use of one language makes you a real programmer.

Charisma

The conference is packed with an atmosphere of euphoria; excitement for what’s to come as the next framework is announced. A passionate author comes to the stage with a singing voice. The talk and powerpoint slides discuss the newest principles that promise bullet-proof programming practices — of the kind that’ll make you the 10x programmer (for real this time!). With a natural ability to work the crowd, and the official setting in which the author talks, it can be hard to realize we’ve come to believe the author’s claims. And most importantly buying their book.

Glow

I realize it’s a weird word to use, but when an idea is compelling and exciting enough it just seems to give off a certain “glow.” It is shiny and it must therefore be true.

We could go on, but moving away from this kind of list is what it means to be scientific about our technological progress.

Expand Your Computer Literacy

A conversation about the nature of the universe is generally more useful when those talking are scientifically literate. Similarly, a discussion on programming benefits from everyone reaching a certain threshold in computer literacy. Indeed, there are enduring ideas giving life to computer systems, and programmers ought to value them. If we call ourselves professionals, we have a real responsibility to learn the underlying concepts behind computers. It protects us from unreliable sources of knowledge, and it shows us the things we should pay attention to. NASA realized this, and has made sure the organization’s culture allows for everyone’s literacy to skyrocket.

Recommendations

Did any of this resonate? Are you a programmer looking to improve your computer literacy? If so I have a couple of personal suggestions.

Computer Systems: A Programmer’s Perspective

Computer System’s: A Programmer’s Perspective was very formative in my career, and if it matters, it is my favorite programming book. The topics covered here are the kind of foundations I think everyone should have. At the very least, a thorough study of Chapters 1–3, the memory hierarchy, and System I/O are eye-opening. If you are working on anything web-related, of course the chapter on Network Programming is relevant.

Finding Like-Minded People

There’s a small community called Handmade Network, with an incredible team (spoiler: I’m part of the team) as we try to build a place where we can step away from the modern software conversation, and instead talk about the things I have laid out here. We are as vulnerable as other communities to fall into dogma (and we have in the past), but we try to be cognizant of it.

The community also has interviews with well-known programmers discussing the state of affairs, whether or not they agree with our conclusions. Previous ones are available here.

Related Press

About the Author

2017 marked three years of work at Kennedy Space Center, writing and maintaining launch control system software. I was awarded Kennedy’s Intern of the Year by 2015 which lead me to NASA’s Pathways program, where I got tasked to research ways to simplify the debugging of (wait for it) flying space robots. The fruit of that work landed me on Kennedy’s 2016 Top 10 list of Innovators — though just barely!

Most of my work is unfortunately SBU so it may not be disclosed online. I technically could, but I’ve been told federal prison does not have chocolate wine. So no dice. You can find me over at Twitter, Twitch, or e-mail.

Warring factions within factions, conflicting back-channeling, intense media scrutiny, questionable foreign influences and a capricious leader whose jarring moves leave everyone in a state of perpetual uncertainly.

The Trump administration, right?

Well, yes, but also Uber, as it nears its much anticipated decision on who will be its next CEO.

And, according to sources, that top leader is not going to be a woman, as the board of the car-hailing company struggles to move forward.

To add to the drama: Some directors worry that its former CEO Travis Kalanick — who was ousted — is trying to game the outcome in his favor, after he told several people that he was “Steve Jobs-ing it.” It is a reference to the late leader of Apple, who was fired from the company, only to later return in triumph.

That’s why, while I am always loath to dump a kitchen sink’s worth of reporting in one story, it’s hard not to since this particular pile of dirty dishes is so stuck together at Uber that it’s almost impossible to pry them apart.

“If there was no hair on this dog, this would be a no brainer for anyone to take this job,” said one person close to the search of Uber’s next leader. “But this is the hairiest company anyone has ever seen.”

Ew. But pretty much true.

So, in what is probably a vain attempt to clarify the situation, here’s the state of what is a very confused play.

Boys club

With the exit of Hewlett Packard Enterprise CEO Meg Whitman from consideration, several sources with knowledge of the situation said the group of four final candidates being considered are all men and all CEOs. (Also: Only one of those is a person of color.)

Among those choices is outgoing GE CEO Jeff Immelt. But Immelt is not, said several sources, the top choice of several members of the search committee and also within Uber’s top ranks. Some are worried that — while he is obviously a very experienced manager — he lacks the entrepreneurial drive to take the company to the next level.

In other words, he’s too much a corporate suit and not enough a geek pirate. (Anywhere else but in tech, this is not seen as a bad thing, but here we are.)

Immelt aside, sources said that the inclusion of more women in key decision-making roles at Uber will come via the addition of more independent board members and top execs rather than as its main leader.

That, of course, will be a major disappointment to some, especially given major problems at Uber involving sexism and sexual harassment. While gender of a leader should not matter to dispatching such appalling behaviors, the symbolism over the appointment of a female CEO at Silicon Valley’s most toxic-bro startup is unquestionable.

But there are also not that many top women CEOs to pick from. Execs like Facebook’s Sheryl Sandberg and General Motors’ Mary Barra were not interested, said sources, and others pursued, like EasyJet CEO Carolyn McCall, did not pan out.

It had been hoped that the decision on who will be the next leader of Uber would be completed within the next week, but trouble getting the board to coalesce and trust each other has complicated and continues to complicate the effort.

In addition, sources said that not all board members had met all the top candidates as of this weekend, so that no vote on any one of them had been planned or is planned immediately. (Uber’s board now meets regularly every two weeks, so that could change.)

That lack of cohesion impacted the courting of Whitman, which never resulted in any vote (and which had never been scheduled at all, despite one report that it was). This was largely because she had not met with all directors before she suddenly pulled her name out of consideration late last week.

Sudden indeed. Whitman was expected this past weekend to meet in person with more directors, including Wan Ling Martello of Nestlé and Arianna Huffington, which never happened.

Whitman out!

Whitman’s departure from the Uber fray was considerably more complicated than has been previously reported. The well-known exec, who is also one of the most prominent women leaders in tech, was indeed interested in the job. That included having several substantive discussions with directors, where she made a series of suggestions on how to fix Uber.

Still, sources close to Whitman said she was never formally offered the job by Uber’s board, even though she certainly was intrigued by the possibility.

And why not? Such a role would have been the capstone of a long and largely successful career, most especially in growing eBay to its heights from a small startup. Turning around Uber and taking it public would have been the ultimate addition to her legacy.

But board disagreements, worries about whether all the myriad of problems at Uber had surfaced and concerns about the continued involvement of Kalanick, as well as the public disclosure of Whitman as a possible candidate, ended that possibility.

“Uber’s CEO will not be Meg Whitman,” she tweeted on Thursday, days after media reports on Tuesday by Recode and Bloomberg that disclosed Uber’s interest in her. That then put pressure on Whitman, a situation made worse by another report in Axios that indicated that she was not a unanimous choice of the board.

Such mishegas irked Whitman, who expected a more — ahem— professional process. “What a mess,” said one person. “Why should she get dirtied by their playground antics? It’s like a sandbox over there.”

Well, to be fair, more like a really nerdy version of “Game of Thrones,” with 100 percent less dismemberment but 53 percent more intrigue.

Still, as Recode previously reported, Whitman was an early investor in Uber and has helped out with coaching from time to time, especially with Kalanick, so she did have some familiarity with the unusually juvenile management style there.

But sources said she did not like the process, and gave the board a 48-hour deadline to decide if they seriously wanted her to be CEO. While some on the board wanted to move forward immediately, others — Huffington, for example — did not since she had not met them all in person.

“It was artificial urgency, even if Whitman might be the right choice,” said one source.

No matter! Whitman out!

We have to talk about Travis

That might not have mattered, as Whitman was also worried more specifically about further disclosures to come from the company related to its lawsuit with Alphabet’s Waymo, especially how involved Kalanick was in the allegations leveled against Uber.

“Who knows what could fall out of the closet over there?” said one person, reflecting the concerns of many candidates who have spoken to Uber, including Whitman. “There could be even more skeletons.”

Kalanick’s continued role figures large in the mind of every person I spoke to who has been contacted about the CEO job.

What’s the biggest problem at Uber? I asked.

“Travis,” said one

“Oh, Travis,” said another.

“Man, he’s brilliant and so important, but who wants to deal with Travis?” said yet another.

If that sounds like a real sad country song, titled “The Travis Blues,” you are not far off the mark.

This was certainly not the plan when Kalanick was dispatched to Tahiti to sail around in a glamourous yacht — owned by media mogul Barry Diller, with fellow guests like CNN’s Anderson Cooper — to cool off after he was forced out at Uber.

He initially had agreed to take just a temporary leave, a move that became permanent after a group of key investors — including Benchmark — demanded his full departure a week later.

But once he returned from the South Seas, his ardor for meddling in the company did not end. According to numerous insiders, the pugnacious entrepreneur has continued to try to involve himself in daily operating decisions, so much so that top execs have been mulling how to get help from the board to rein him in.

“It’s not stopped,” said one top exec about Kalanick up in the grill of the operating group that is running Uber. “None of us know what to do since it is Travis.”

In addition, to cut off Kalanick’s access, the Uber board has reinforced a policy that all directors get the same limited access to information about Uber’s ongoing operations. “We have had to put guardrails on him,” said one person involved. “Even if he keeps trying to break through them.”

Indeed, Kalanick was considered by many directors and investors to be obstructive to the process of finding a COO before his departure as CEO.

And there have been more signals that he has been unhappy about this status. Since he left, Kalanick has told numerous people, including at least one job candidate, that he was “Steve Jobs-ing it,” an apparent reference to the purge and later return of the legendary Apple founder at the company.

(As I have said before, I knew Steve Jobs, Steve Jobs was a person I wrote about and — you got it — Travis Kalanick is no Steve Jobs. But I digress!)

But the key tenet in this comparison is that the tarnished tech hero leaves the scene to wander in the desert for years before the triumphant return. Instead, what has been described to me by many is an entrepreneur who cannot let go and, in fact, has been trying to plot ways to increase his grip.

Why? “If you think about it, Uber has been his life and it’s only more so with his terrible personal situation,” said one person, who is fond of Kalanick, referring to the recent tragic death of his mother and the serious injuries sustained by his father in a boating accident. “Giving up Uber is not easy.”

As the board turns

And being with each other in a cohesive manner is not something Uber’s current board seems capable of either, given all the odd back-channeling and frequent miscommunications that seem to crop up.

While there was a recent dinner at San Francisco’s Garabaldi’s restaurant that included Huffington, Ling and two other Uber board members, TPG’s David Trujillo and Benchmark’s Matt Cohler, to try to create some level of comfort, there is a lot to repair.

Here’s a clue to how much: The continued media leakage from the CEO process. While this is not completely uncommon — the Microsoft CEO search got a lot of ink — what is unusual is how varying the accounts of the same meetings or circumstances are at Uber. In fact, they are often diametrically opposed to each other.

This is something that many who have been in touch with Uber have experienced. “Consensus is not something you are feeling is happening there,” said one person, which makes every possible move seem suspect.

(Have you seen the really complexly plotted spy thriller “Atomic Blonde,” where everyone seems to be lying and then lying about lying? Uber is more confusing.)

That’s why a recent attempt to discuss a funding offer involving both secondary sales and a new investment from Japanese investor SoftBank has been so riven. While I will not get into the particulars in this hairball of a story here — unless you really want to hear about transfer restrictions, tender offers and more right now — suffice it to say that it has turned into a drama about whose side SoftBank leader Masa Son will land on.

Let me be clear, this is before any investment, which really should not take place without the cooperation of the new CEO, even if there is some urgency in making sure SoftBank’s $100 billion fund does not favor only Uber’s rivals. Since it just made a big investment in Southeast Asia’s Grab, many at Uber remain fretful.

Also, a related cause of tension was a Bloomberg report that Benchmark was mulling selling a chunk of its 10 percent stake in Uber to SoftBank. While the venture firm might sell some of its stake if any transaction takes place, sources said Benchmark was wary that such a story was meant to weaken it.

That might not be so far-fetched given the deep tension between Benchmark’s Bill Gurley — who left the Uber board in favor of his partner Cohler — and Kalanick and the level of mistrust that has developed over time.

“Every single act feels like it might have another meaning,” said one person familiar with the situation. “Even if it does not.”

What is all boils down to is the deep concerns around control of Uber’s fate: Who has it and who is angling for it.

That distrust has left the board at times in a kind of odd face-off that appears to be more perception than reality, once you really plumb the depth of the concerns.

Consider the possible return of Kalanick as CEO, which most will finally admit is overblown unless he decides to go full throttle and end up in a legal fight with the company he founded. In reality, without the support of board members Ryan Graves and Garrett Camp, Kalanick has none of the kind of leverage that has been reported.

“Travis would have to blow it up completely to get his job back,” said one major investor familiar with the cap table of Uber. “And maybe he is crazy enough to do that, but he’d better bring a lot more ammo.”

What I can say for sure is that the entire company is leaking like a sieve and that the Trump White House has a tighter press ship. Which brings us back to the beginning: If Uber wants to move forward, it had better lose that meme and get back to building the kind of company its employees so desperately want to.

Said one of those top execs to me tonight: “When is this going to stop so we can do our jobs?”

It’s a very good question for the Uber board. Whether they can do their only real duty as directors — hiring a CEO — and make that happen soon is unclear.

One silver lining: Anthony Scaramucci is currently busy with another gig.

README.md

npm install lambda-webpack-zip
yarn add lambda-webpack-zip

const {pack} =require("lambda-webpack-zip");pack({// webpack configs
  entry:"/source/code/path",// Specify the output file containing our bundled code
  output: {
    filename:"index.js"
  }
})
.then(zipFilePath=> {// upload zipFilePath
});

Special note: we’re teaching a fully updated part 1, in person, for seven weeks from Oct 30, 2017, at the USF Data Institute. See the course page for details and application form.

When we launched course.fast.ai we said that we wanted to provide a good education in deep learning. Part 1 of the course has now been viewed by tens of thousands of students, introducing them to nearly all of today’s best practices in deep learning, and providing many hours of hands-on practical coding exercises. We have collected some stories from graduates of part 1 on our testimonials page.

Today, we are launching Part 2: Cutting Edge Deep Learning for Coders. These 15 hours of lessons take you from part 1’s best practices, all the way to cutting edge research. You’ll learn how to:

• Read and implement the latest research papers (even if you don’t have a math background)
• Build a state of the art neural translation system
• Create generative models for art, super resolution, segmentation, and more (including generative adversarial networks)
• Apply deep learning to structured data and time series (such as for logistics, marketing, predictive maintenance, and fraud detection)
• …and much more.

We believe that we have created a unique path to deep learning expertise, and many of our students have shown what’s possible, such as:

• Sara Hooker, who only started coding 2 years ago, and is now part of the elite Google Brain Residency
• Tim Anglade, who used Tensorflow to create the Not Hot Dog app for HBO’s Silicon Valley, leading Google’s CEO to tweet “our work here is done”
• Gleb Esman, who created a new fraud product for Splunk using the tools he learnt in the course, and was featured on Splunk’s blog
• Jacques Mattheij, who built a robotic system to sort two tons of lego
• Karthik Kannan, founder of letsenvision.com, who told us “Today I’ve picked up steam enough to confidently work on my own CV startup and the seed for it was sowed by fast.ai with Pt1. and Pt.2”
• Matthew Kleinsmith and Brendon Fortuner, who in 24 hours built a system to add filters to the background and foreground of videos, giving them victory in the 2017 Deep Learning Hackathon.

The prerequisites are that you’ve either completed part 1 of the course, or that you are already a confident deep learning practictioner who is comfortable implementing and using:

• CNNs (including resnets)
• RNNs (including LSTM and GRU)
• SGD/Adam/etc
• Batch normalization
• Data augmentation
• Keras and numpy

The course covers a lot of territory - here’s a brief summary of what you’ll learn in each lesson:

Lesson 8: Artistic Style

We begin with a discussion of a big change compared to part 1: from Theano to Tensorflow. You’ll learn about some of the exciting new developments in Tensorflow that have led us to the decision to make this change. We’ll also talk about a major project we highly recommend: build your own deep learning box!

We’ll also talk about how to approach one of the biggest challenges in this part of the course: reading acadmic papers. Don’t worry, it’s not as terrifying as it first sounds—especially once you know some of our little tricks.

Then, we start our deep dive into creative and generative applications, with artistic style transfer. You’ll be able to create beautiful and interesting images even if your artistics skills are as limited as Jeremy’s… :)

Lesson 9: Generative Models

We’ll learn about the extraordinarily powerful and widely useful technique of generative models. These are models that don’t just spit out a classification, but create a whole new image, sound, etc. They can be used, for example, with images, to:

• Improve photos (colorization, noise removal, increase resolution, etc)
• Create art
• Find and segment (localize) objects
• and much more…

We’ll try using this approach for super resolution (i.e. increasing the resolution of an image), and then you’ll get to try building your own system for rapidly adding the style of any artist to your photos. Have a look at the image on the right - the top very low resolution image has been input to the algorithm (see for instance the very pixelated fingers), and the bottom image has been created automatically from that!

Lesson 10: Multi-modal & GANs

A surprising result in deep learning is that models created from totally different types of data, such as text and images, can learn to share a consistent feature space. This means that we can create multi-modal models; that is, models which can combine multiple types of data. We will show how to combine text and images in a single model using a technique called DeVISE, and will use it to create a variety of search algorithms:

• Text to image (which will also handle multi-word text descriptions)
• Image to text (including handling types of image we didn’t train with)
• And even image to image!

Doing this will require training a model using the whole imagenet competition dataset, which is a bigger dataset than we’ve used before. So we’re going to look at some techniques that make this faster and easier than you might expect.

We’re going to close our studies into generative models by looking at generative adversarial networks (GANs), a tool which has been rapidly gaining in popularity in recent months, and which may have the potential to create entirely new application areas for deep learning. We will be using them to create entirely new images from scratch.

Lesson 11: Memory Networks

We’ve covered a lot of different architectures, training algorithms, and all kinds of other CNN tricks during this course—so you might be wondering: what should I be using, when? The good news is that other folks have wondered that too, and have provided some great analyses of the pros and cons of various techniques in practice. We’ll be taking a look at a few highlights of these papers today.

Then we’re going to learn to GPU accelerate algorithms by taking advantage of Pytorch, which provides an interface that’s so similar to numpy that often you can move your algorithm onto the GPU in just an hour or two. In particular, we’re going to try to create the first (that we know of) GPU implementation of mean-shift clustering, a really useful algorithm that deserves to be more widely known.

To close out the lesson we will implement the heavily publicized “Memory Networks” algorithm, and will answer the question: does it live up to the hype?

Lesson 12: Attentional Models

It turns out that Memory Networks provide much of the key foundations we need to understand something which have become one of the most important advances in the last year or two: Attentional Models. These models allow us to build systems that focus on the most important part of the input for the current task, and are critical, for instance, in creating translation systems (which we’ll cover in the next lesson).

Lesson 13: Neural Translation

One application of deep learning that has progressed perhaps more than any other in the last couple of years is Neural Machine Translation. In late 2016 it was implemented by Google in what the New York Times called The Great A.I. Awakening. There’s a lot of tricks needed to reach Google’s level of translation capability, so we’ll be doing a deep dive in this lesson to learn nearly all the tricks used by state of the art systems.

Next up, we’ll learn about Densenets, which in July 2017 were awarded the CVPR Best Paper award, and have been shown to provide state of the art results in computer vision, particularly with small datasets. They are very similar to resnets, but with one key difference: the branches in each section are combined through concatenation, rather than addition. This apparently minor change makes a big difference in how they learn. We’ll also be using this technique in the next lesson to create a state of the art system for image segmentation.

Lesson 14: Time Series & Segmentation

Deep learning has generally been associated with unstructured data such as images, language, and audio. However it turns out that the structured data found in the columns of a database table or spreadsheet, where the columns can each represent different types of information in different ways (e.g. sales in dollars, area as zip code, product id, etc), can also be used very effectively by a neural network. This is equally true if the data can be represented as a time series (i.e. the rows represent different times or time periods).

In particular, what we learnt in part 1 about embeddings can be used not just for collaborative filtering and word encodings, but also for arbitrary categorical variables representing products, places, channels, and so forth. This has been highlighted by the results of two Kaggle competitions that were won by teams using this approach. We will study both of these datasets and competition winning strategies in this lesson.

Finally, we’ll look at how the Densenet architecture we studied in the last lesson can be used for image segmentation - that is, exactly specifying the location of every object in an image. This is another type of generative model, as we learnt in lesson 9, so many of the basic ideas from there will be equally applicable here.

To discuss this post at Hacker News, click here

A small company called hiQ is locked in a high-stakes battle over Web scraping with LinkedIn. It's a fight that could determine whether an anti-hacking law can be used to curtail the use of scraping tools across the Web.

HiQ scrapes data about thousands of employees from public LinkedIn profiles, then packages the data for sale to employers worried about their employees quitting. LinkedIn, which was acquired by Microsoft last year, sent hiQ a cease-and-desist letter warning that this scraping violated the Computer Fraud and Abuse Act, the controversial 1986 law that makes computer hacking a crime. HiQ sued, asking courts to rule that its activities did not, in fact, violate the CFAA.

James Grimmelmann, a professor at Cornell Law School, told Ars that the stakes here go well beyond the fate of one little-known company.

"Lots of businesses are built on connecting data from a lot of sources," Grimmelmann said. He argued that scraping is a key way that companies bootstrap themselves into "having the scale to do something interesting with that data." If scraping without consent becomes illegal, startups like hiQ will have a harder time getting off the ground.

But the law may be on the side of LinkedIn—especially in Northern California, where the case is being heard. In a 2016 ruling, the 9th Circuit Court of Appeals, which has jurisdiction over California, found that a startup called Power Ventures had violated the CFAA when it continued accessing Facebook's servers despite a cease-and-desist letter from Facebook.

Some details of that case were different—Power Ventures was sending out private messages with the permission and cooperation of Facebook users, while hiQ is scraping data on public webpages. But experts told Ars that the Power Ventures precedent is likely to be bad news for hiQ because it suggests that continuing to access a site after being asked to stop is enough to trigger the anti-hacking law.

LinkedIn's position disturbs Orin Kerr, a legal scholar at George Washington University. "You can't publish to the world and then say 'no, you can't look at it,'" Kerr told Ars.

The CFAA makes it a crime to "access a computer without authorization or exceed authorized access." Courts have been struggling to figure out what this means ever since Congress passed it more than 30 years ago.

One plausible reading of the law—the one LinkedIn is advocating—is that once a website operator asks you to stop accessing its site, you commit a crime if you don't comply.

That's the interpretation suggested by the 2016 Power Ventures decision, which is a binding precedent in California. Power.com was a social network that functioned as a social network aggregator. Through the Power.com website, users could log into other social networks like Facebook, allowing them to access information from multiple social networks simultaneously.

To expand its user base, Power asked users to provide their Facebook credentials and then—with their permission—sent Power.com invitations to their Facebook friends. Facebook, naturally, didn't appreciate this marketing tactic. They sent Power a cease-and-desist letter and also blocked the IP addresses Power was using to communicate with Facebook's servers.

Facebook sued, claiming that its cease-and-desist letter made Power's access unauthorized under the terms of the CFAA. Power disagreed and argued that having permission from Facebook users was good enough—it didn't need separate approval from Facebook itself.

But the 9th Circuit Court of Appeals sided with Facebook last year.

"Power users arguably gave Power permission to use Facebook's computers to disseminate messages," the court wrote. "But Facebook expressly rescinded that permission when Facebook issued its written cease-and-desist letter." After this point, the court held, "Power knew it no longer had authorization to access Facebook's computers, but continued to do so anyway."

That result bothers Kerr.

For example, he said, imagine if CNN sent out letters to reporters at rival news organizations demanding that their reporters not access cnn.com. Under an expansive reading of the law, Kerr told Ars, it would then "become a federal crime to visit a public website."

Kerr argues sites wanting to limit access to their site should be required to use a technical mechanism like a password to signal that the website is not, in fact, available to the public.

"It's hugely problematic to let the subjective wishes of the website owner and not their objective action" determine what's legal, Kerr told Ars.

The Power Ventures case isn't over. Power Ventures asked the Supreme Court to consider the case in May, and the high court hasn't decided whether to do so yet. And for now, the Power Ventures precedent only applies within the 9th Circuit, which covers California and other Western states. Unfortunately for hiQ, the LinkedIn dispute is being heard by California federal courts.

Ultimately, Grimmelmann believes, the text of the CFAA doesn't clearly settle this question. Both Kerr's view that running a public website implicitly gives the public authorization to access it and LinkedIn's view that companies can rescind authorization on a case-by-case basis are plausible interpretations of the law.

But both scholars argue there are good reasons to favor the more permissive reading of the law. The LinkedIn interpretation of the law gives big website operators like LinkedIn plenty of power over how their sites are used. They argue the courts should preserve the rights of small companies, watchdog groups, and others to gather information from the Web using scraping tools.

The EFF has pointed out just how ridiculous it is to argue that violating a terms of service is a potential felony, noting how that even makes children who read online news sites potential felons for violating terms of service. This is, in part, due to another bad law that we've spoken about, the Children's Online Privacy Protection Act, or COPPA. The issue here is that online sites have stricter rules if they're seen as targeting children under the age of 13. To avoid this potential liability, many websites simply inserted a clause into their terms of service saying that you can only read the site if you're over 13 (some sites say 18 and others say between 13 and 18 need a parent's approval). While this is somewhat lazy lawyering on the part of those sites (to ban outright), those are their terms of service. And violating such terms violates the CFAA under the DOJ's interpretation.

The EFF notes that such age exclusion provisions are pretty common, and sites like the NY Times and NBC News bar children under 13 entirely.

This means that inquisitive 12-year-olds who visit NBCNews.com to learn about current events would be, by default, misrepresenting their ages. Again, this could be criminal under the DOJ's interpretation of the CFAA.

We’d like to say that we’re being facetious, but, unfortunately, the Justice Department has already demonstrated its willingness to pursue CFAA to absurd extremes. Luckily, the Ninth Circuit rejected the government’s arguments, concluding that, under such an ruling, millions of unsuspecting citizens would suddenly find themselves on the wrong side of the law. As Judge Alex Kozinski so aptly wrote: "Under the government’s proposed interpretation of the CFAA...describing yourself as 'tall, dark and handsome,' when you’re actually short and homely, will earn you a handsome orange jumpsuit."

And it’s no excuse to say that the vast majority of these cases will never be prosecuted. As the Ninth Circuit explained, “Ubiquitous, seldom-prosecuted crimes invite arbitrary and discriminatory enforcement.” Instead of pursuing only suspects of actual crimes, it opens the door for prosecutors to go after people because the government doesn’t like them.

Unfortunately, there’s no sign the Justice Department has given up on this interpretation outside the Ninth and Fourth Circuits, which is why the Professor Tim Wu in the New Yorker recently called the CFAA “the most outrageous criminal law you’ve never heard of.”

The EFF followed it up by pointing out that, until just recently, if you were a 17-year-old girl (or younger!) reading the magazine Seventeen online, you were almost certainly breaking the law under the DOJ's interpretation of the CFAA, since its terms restricted visitors to those 18 and older.

Rather than "trusting" the DOJ not to abuse this kind of thing, wouldn't we all be better off fixing it?

GoCardless Ltd. 338-346 Goswell Road, London, EC1V 7LQ, United Kingdom

GoCardless (company registration number 07495895) is authorised by the Financial Conduct Authority under the Payment Services Regulations 2009, registration number 597190, for the provision of payment services.

Editor’s note: On July 23, the writer Zhang Wumao published an essay called “In Beijing, 20 Million People Pretend to Live” to his public WeChat account. As of the following morning, it had accumulated more than 5 million views and nearly 20,000 comments. 

Of course, the article was removed that very afternoon.

But by then, the essay had attracted thousands of responses. As our correspondent Megan Pan wrote for Radii:

Though the hubbub online has died down, the essay, a meditation on varying facets of life in Beijing, has since spawned over a hundred thousand countering essays in response. Titles include plays on the original essay’s title, such as “In Beijing, 20 Million People and “In Beijing, 20 Million People are Bravely Living,” and even direct digs at the author, such as “Mr. Zhang, You Aren’t Even a Beijing Kid So Why Are You Acting Like a Know-it-all.” The original essay has been lambasted as “making a fuss over nothing.”

But “In Beijing, 20 Million People Pretend to Live” resists easy summarization – it’s framed as a series of Zhang’s loosely related reflections on living in Beijing, heavily supported by anecdotes. He touches on a variety of topics that hit close to home: the everyday absurdities of urban sprawl, the never-ending struggle to buy a house, and alienation from home. As a nonlocal from Shaanxi who has been living in Beijing for the past eleven years, he also attempts to negotiate the tensions and differences between locals and nonlocals.

What follows is Megan Pan’s translation of that now-censored essay.

1

北京没有人情味.

Beijing has no human warmth.

经常被外地朋友批评：北京人钱多装逼不热情。都到了同一个城市，干嘛不一起聚聚？几十年的交情，还不把我送到机场？事实上，北京人很难像外地人一样热情——来去接送，全程陪同，北京人真的很难做得到。

I am often criticized by nonlocal friends: Beijingers have a lot of money and act unfriendly. We’ve all made it to the same city, why don’t we get together? A few decades of friendship, and you won’t even send me to the airport? In reality, it is hard for Beijingers to be as friendly as outsiders – picking up and dropping off, accompanying all the way, these things truly are hard for Beijingers to do.

北京人很忙，忙到晚上11点，还在三环路上堵着；北京社交时间成本真的太高，高到从石景山去通州吃饭，还不如去天津来得快；北京真的太大，大到根本就不像一个城市。

Beijingers are very busy, busy all the way until 11 o’clock at night, and even then they are still stuck in traffic on the Third Ring Road; the time cost of socializing in Beijing is too high, so high that it is faster to go to Tianjin than it is to go from Shijingshan to Tongzhou to eat; Beijing is really too big, so big that it isn’t like a city at all.

北京到底有多大？它相当于2.5个上海，8.4个深圳，15个香港，21个纽约，27个首尔。2006年，张先生来北京，地铁只有1号，2号，13号线，现在的北京地铁到底有多少条线，不用百度还真记不住。10年前我坐着公交去找工作，拒绝去四环外的公司面试。现在，京东、腾讯、百度这些大公司都在五环外。

How big is Beijing? It is equivalent to 2.5 Shanghais, 8.4 Shenzhens, 15 Hong Kongs, 21 New Yorks, and 27 Seouls. In 2006, when Mr. Zhang [referring to himself] came to Beijing, the subway only had Lines 1, 2, and 13; if I didn’t use Baidu, I really wouldn’t be able to remember just how many lines the Beijing subway has now. Ten years ago, I took the bus to look for work and refused to go beyond the Fourth Ring for job interviews. Now, big companies like JD, Tencent, and Baidu are all outside of the Fifth Ring.

外地朋友来了北京，以为我们就很近了，实际上咱们不在同一个城市，咱们可能是在若干个城市，它们是中国海淀，中国国贸，中国通州，中国石景山……如果以时间为尺度，通州人和石景山人谈恋爱就算是异地恋，从北五环来趟亦庄就可以说是出差。

When nonlocal friends came to Beijing, they thought we were closer, but we weren’t actually in the same city, we may have been in a number of cities: they are Haidian, China; Guomao, China; Tongzhou, China; Shijingshan, China… If we use time as a measure, then someone from Tongzhou dating someone from Shijingshan would count as long-distance, and going from North Fifth Ring to Yizhuang can be called a business trip.

十年间，北京一直在控房控车控人口，但这块大饼却越摊越大，以至于西安的同学给我打电话，也说自己在北京，我问他在北京哪里？他说：我在北京十三环。

For the last ten years, Beijing has been controlling housing, controlling cars, and controlling population, but this large flatbread continues to sprawl and grow larger, to the point where my Xi’an classmate called me and said he was also in Beijing, and when I asked where in Beijing he was, he said: I’m in Beijing’s Thirteenth Ring.

北京是个肿瘤，没有人能控制它的发展速度；北京是一条河流，没人能划清它的边界。北京是一个信徒，只有雄安能将它超度。

Beijing is a tumor, whose speed of development no one can control; Beijing is a river, whose boundaries no one can draw. Beijing is a disciple, and only Xiongan can release it from purgatory. [Editor's note: Xiongan is a recently established state-level development hub in nearby Hebei province.]

北京的人情淡薄不只是针对外地朋友，对同处一城的北京朋友同样适用。每次有外地同学来京，聚会时外地同学会说，你们在北京的应该经常聚吧？我说，你们一年来几次北京，我们差不多就聚几次。

Beijing’s coldness is directed not only at nonlocal friends, it is also similarly applied to Beijing friends who live in the same city. Every time a nonlocal classmate comes to Beijing, when we all get together the classmate will say, you guys in Beijing often meet up, right? And I will say, however many times you guys come to Beijing is about how many times we meet up.

在北京，交换过名片就算认识；一年能打几个电话就算至交；如果还有人愿意从城东跑到城西，和你吃一顿不谈事的饭，就可以说是生死之交了；至于那些天天见面，天天聚在一起吃午饭的，只能是同事。

In Beijing, exchanging business cards counts as recognition; calling a couple times a year counts as best friends; if someone is willing to go from the east to the west side to have a meal with you without talking business, then you could be called friends for life; as for the people you see every day, eat lunch with every day, they are only coworkers.

2

北京其实是外地人的北京。

Beijing is actually the outsider’s Beijing.

如果要让中国人评选一生中必去的城市，我相信大多数人会选择北京。因为这里是首都，这里有天安门，有故宫，有长城，有几百家大大小小的剧院。话剧歌剧传统戏，相声小品二人转，不管你是阳春白雪，还是下里巴人，都可以在北京找到属于自己的精神食粮。但这些东西其实和北京的人没多大关系。

If you let Chinese choose their must-go cities in this lifetime, I believe that most people would pick Beijing. Because here is the capital, here is Tiananmen, the Forbidden City, the Great Wall, the hundreds of theaters, big and small. Drama, opera, traditional drama, crosstalk, two-person skits, whether you like highbrow or popular art, you can always find what your spirit needs in Beijing. But these things actually do not have much to do with Beijingers.

走进北京各大剧院，十个人里面有六个人是口音各异的外地人，还有三个是刚来北京，没新鲜够的文艺青年，最后剩下一个是坐在角落里刷手机，熬时间的北京地陪。

Walking into Beijing’s various big theaters, I see that among ten people, six are outsiders with differing accents, three are young literary types that have just arrived in Beijing and haven’t gotten enough of the novelty, and the last remaining one is the local guide sitting in the corner, playing with his phone to kill time.

来京11年，我去过11次长城，12次故宫，9次颐和园，20次鸟巢。我对这个城市牛逼的建筑，悠久的历史完全无感。登上长城，只会想起孟姜女，很难再升腾起世界奇迹的民族豪情；走进故宫，看到的只是一个接一个的空房子，还没我老家的猪圈生动有趣。

In the 11 years since arriving in Beijing, I have gone to the Great Wall 11 times, the Forbidden City 12 times, the Summer Palace nine times, and the Bird’s Nest 20 times. I feel complete indifference for this city’s awesome structures and long history. Climbing the Great Wall, I only think of Lady Meng Jiang, finding it difficult to stir up that lofty pride for the wonders of the world once more; walking into the Forbidden Palace, I see only one empty building after another, which is even less lively and interesting than my hometown’s pigpen.

[Lady Meng Jiang, according to folklore, wept bitterly at the Great Wall for her dead husband, who helped build it.]

很多人一提北京，首先想到的是故宫后海798，是有历史有文化有高楼大厦。这些东西好不好？好！自豪不自豪？自豪！但这些东西不能当饭吃。北京人感受更深的是拥堵雾霾高房价，是出门不能动弹，在家不能呼吸。

When bringing up Beijing, so many people think first of the Forbidden City, Houhai, and 798 [Art Zone], of how Beijing has history and culture and high-rises. Are these things good? They are good! Am I proud? I am proud! But these things cannot be what we live off. What Beijingers experience more deeply is the congestion, the smog, the high housing prices; it is how, when leaving the house, you cannot move, and when at home, you cannot breathe.

3

北京是终归是北京人的北京。

Beijing, in the end, is the Beijingers’ Beijing.

如果说北京还有那么一点烟火味的话，那么这烟火味属于那些祖孙三代都居住在这个城市的老北京人。这烟火味是从老北京人的鸟笼子里钻出来的，是从晚饭后那气定神闲的芭蕉扇里扇出来的，是从出租车司机那傲慢的腔调里扯出来的……

If Beijing is said to have that hint of the smell of smoke, then that smell of smoke belongs to the old Beijingers who have been living in this city for generations. This smell of smoke curls out of old Beijingers’ birdcages, fans out from the leisurely palm-leaf fan after dinner, is pulled out from the taxi driver’s haughty tone of voice…

老北京人正在努力为这个城市保留一丝生活气息，让这个城市看起来，像是个人类居住的地方。

Old Beijingers are currently trying to preserve a bit of breath of life for this city, in order to make this city look like a place where humans live.

老北京人的这点生活气息是从基因里传下来的，也是从屁股下面五套房子里升腾起来的。当西城的金融白领沉浸在年终奖的亢奋中时，南城的北京土豪会气定神闲地说，我有五套房；当海淀的码农们敲完一串代码，看着奶茶的照片，幻想自己成为下一个刘强东的时候，南城的北京土豪会气定神闲地说，我有五套房；当朝阳的传媒精英签完一个大单，站在CBD落地窗前展望人生时，依旧会听到南城土豪气定神闲地说，我有五套房。

This breath of life that old Beijingers have is passed down through genes, and also rises up from the five houses underneath their asses. When Xicheng’s [Beijing district to the west] financial white-collars are absorbed in the excitement of their year-end bonuses, Nancheng’s [district to the south] Beijing tuhao [Chinese term for people of wealth/nouveau riche] will leisurely say, I have five houses; when Haidian’s manong [coders] finish typing out a string of code, looking at pictures of milk tea* and fantasizing about when they will become a Richard Liu [founder of JD.com], Nancheng’s Beijing tuhao will leisurely say, I have five houses; when Chaoyang [District]’s media elite finish signing a large order, standing in front of the CBD’s [Central Business District] floor-to-ceiling windows forecasting life, they will still hear Nancheng’s tuhao leisurely saying, I have five houses.

* ["Milk tea" is a reference to JD.com founder Richard Liu's wife, Zhang Zetian, whose nickname is "milk tea sister."]

没有五套房，你凭什么气定神闲？凭什么感受生活气息？凭什么像北京大爷一样逗鸟下棋，听戏喝茶？

If you don’t have five houses, on what basis can you act leisurely? On what basis can you feel that breath of life? On what basis can you be like an old Beijing uncle, playing with birds, playing chess, listening to operas, and drinking tea?

在北京，没有祖产的移民一代，注定一辈子要困在房子里。十几年奋斗买一套鸟笼子大小的首套房；再花十几年奋斗换一套大一点的二套房，如果发展得快，恭喜你，可以考虑学区房了。

In Beijing, this generation of migrants without inherited property are destined to be trapped within the housing system their whole lives. They struggle for decades to buy a house the size of a birdcage, then struggle a few more decades to swap it out for a slightly bigger second house, and if you make strides, congratulations, you can now consider school district housing.

好像有了学区房，孩子就可以上清华上北大，但是清华北大毕业的孩子依旧买不起房。那时候，孩子要么跟我们一起挤在破旧的老房子里，要么从头开始，奋斗一套房。

It is as though if you have school district housing, your kids will be able to go to Tsinghua and Peking University, but kids that graduate from Tsinghua and Peking still can’t afford to buy a house. Then, they will come live with us in that shabby old house, or start all over again, struggling to buy a house.

4

2015年，电影《老炮儿》热映，朋友圈里有好多人吐槽电影里六爷的北京味。我深有感触。

In 2015, Mr. Six was popular in theaters, and on my Wechat Moments were many people complaining about Mr. Six’s Beijing flavor. I felt very much the same way.

来北京十多年，我拒绝去五棵松看首钢，拒绝去工体看国安，因为没有发自肺腑的热爱，也学不会京腔国骂。但在北京久了，你会和老北京人达成某种和解。对他们有了更立体的了解，就没法再把他们简单地标签化。

Having been in Beijing for 10 or so years, I refuse to go to Wukesong to see the Beijing Ducks [basketball team] and to go to the Workers’ Stadium to see Beijing Guoan [soccer team], as I have no love for it from the bottom of my heart and I can’t learn Beijing-style cursing. But if you stay in Beijing for a while, you will reach a kind of understanding with the old Beijingers. Once you have a richer understanding of them, there is no way to stereotype them.

事实上，不是所有的北京人都排外，我身边就有很多友好的北京土著；也不是所有的北京年轻人都不求上进，坐享其成，大部分的北京年轻人和我们一样勤奋。

In reality, not all Beijingers oppose outsiders, many of my friends are Beijing natives themselves, and not all young Beijingers are idle and only enjoy what they already have. Most young Beijingers are just as assiduous as we are.

你可以不喜欢《老炮儿》，不喜欢北京人的自大京骂吹牛逼，但你得尊重他们，就像尊重东北人戴金项链，尊重山东人吃大葱一样，这就是人家的文化和习性，不能入乡随俗，至少也得敬而远之。

You can dislike Mr. Six, dislike Beijingers’ swaggering style of cursing and bragging, but you must respect them, just like you respect Dongbei [northeastern] people wearing gold chains and Shandong people eating scallions. These are people’s culture and habits, and if you can’t do as the Romans do, you must at least respect them from a distance.

有一次打车去林萃路，怕师傅不认识路，我打开导航准备帮师傅找路。师傅说不用导航了，那地方我知道，30年前那里是个面粉厂，十年前面粉厂拆了，建成了保障房。我说，你咋这么清楚？师傅满脸忧愁地说，那是我老家。

The first time I took a taxi to Lincui Road, I was worried the taxi driver wouldn’t know the way, so I opened up my navigation app to help guide him. The driver said he didn’t need it, I know that place, 30 years ago it was a flour factory, 10 years ago the flour factory was torn down and turned into affordable housing. I said, how do you know so much? With a face full of sorrow, the driver said, That was my old home.

我从师傅的话里能听出一丝乡愁和怨恨，北京对于新移民是站不住的远方；对老北京人却是回不去的故乡。

I could hear in his words a hint of nostalgia and resentment; to new migrants, Beijing is the distant place where they cannot stay, to old Beijingers, it is the home to where they cannot return.

我们这些外来人一边吐槽北京，一边怀念故乡。事实上，我们的故乡还回得去。它依旧存在，只是日益落败，我们已经无法适应而已。但对于老北京人而言，他们的故乡才是真的回不去了，他们的故乡正在以前所未有的速度发生物理上的改变，我们还能找到爷爷当年的房子，但多数北京人，只能通过地球经纬度来寻找自己的故乡。

We outsiders complain about Beijing while missing our homes. In reality, we can still go back to our homes. They still exist, it is only that they fall increasingly behind day by day and we cannot adjust anymore. But for old Beijingers, they truly cannot go back to their home, their home is now undergoing a physical change at an unprecedented speed. We can still find grandpa’s house from back then, but many Beijingers can only search for their own home by the earth’s coordinates.

有人说，是我们外地人建设了北京，没有外地人北京人连早餐都吃不上；是因为大量的外来人口抬高了北京的房价，造就了北京的繁华。但是你想过没有？老北京人也许并不需要这繁华，也不需要我们来抬高房价，他们和我们一样，只需要一个说青水秀，车少人稀的故乡。

Some people say, it is we outsiders who built up Beijing, if there were no outsiders, Beijingers wouldn’t even be able to have breakfast; it is because the migrant population has raised Beijing’s housing prices, creating Beijing’s prosperity. But have you ever thought about it? Perhaps old Beijingers don’t need this prosperity and don’t need us to raise housing prices. They are just like us, only needing a home with idyllic scenery, with few cars and less people.

5

今年，北京核心城区开始治理开墙打洞，越来越多的小商店、小饭店、小旅馆被迫关门，越来越多低端行业的从业者被迫离开，这种脱衣服减肥的管理方式让北京在高大上的道路上一路狂奔，但它离生活便利的宜居之都却越来越远，离包容开放的城市精神越来越远。

This year, Beijing’s core city area has begun to clean up “holes in the wall.” More and more small shops, restaurants, and hotels are being forced to close, more and more people working in low-end sectors are being forced to leave. This kind of shed-clothing-to-lose-weight style of management has allowed Beijing to hurtle down the road to sophistication, but it draws further away from the livable city, further away from the open and inclusive spirit of the city.

那些追梦成功的人正在逃离，他们去了澳洲，新西兰，加拿大，美国西海岸。那些追梦无望的人也在逃离，他们退回到河北，东北和故乡。

Those who have successfully achieved their dreams are currently fleeing to Australia, New Zealand, Canada, and the west coast of America. Those who have chased their dreams in vain are also fleeing, they are returning to Hebei, Dongbei, and their hometowns.

还剩下2000多万人留在这个城市，假装在生活。事实上，这座城市根本就没有生活。这里只有少数人的梦想和多数人的工作。

And in this city remain 20 million people, pretending to live. In reality, there is no life in this city. Here, there are only the dreams of few and the work of many.

~

Megan Pan is a writer and undergraduate at Northwestern University majoring in Philosophy and double-minoring in Poetry and Chinese.

Connected home security device maker Ring has a growing product line – and it’s now three products bigger. Joining the Floodlight Cam the company debuted at CES this year, Ring is now also offering three Spotlight Cam variants, each of which offers a different power source option to work with a variety of potential installation scenarios and locations.

The Spotlight Cam – Wired works as you would expect from its name, by plugging physically into a power outlet via an included 20-foot cable. The Spotlight Cam – Battery includes two 6,000 mAH removable, rechargeable Ring batteries (the same used on the new Ring Doorbell 2) for long-lasting power, with one automatically tapping in when the first is drained, so you can charge the first for continuous use. The Spotlight Cam – Solar ships with a bundled Ring solar panel to power the unit’s two 6,000 mAH batteries.

All the cameras feature 1080p HD cameras, 160-degree motion detection and 140-degree fields of view, with two-way audio and live streaming capabilities to your smartphone via the Ring app, along with two powerful LED light strips to illuminate what they spot when motion is detected. Each goes on sale today, July 31, with the wired version at $199 shipping in 7 – 10 days. The battery ($199) and solar ($299) versions are up for pre-order, with shipping beginning later this “fall,” per the company.

Ring’s product roadmap ramp has been impressive, with recent releases including the Floodlight camera and the second generation of their initial Wi-Fi doorbell with a swappable battery design all released already this year. It’s rare to see a hardware startup achieve this kind of product variety this quickly, in fact, so it’ll be interesting to see if Ring can keep up the momentum.

ML, and in particular Deep Learning (DL), has achieved transformative success in speech recognition, image recognition, and natural language processing, and recommendation/search engines recently. These technologies have very promising applications in self-driving cars, digital health systems, CRM, advertising, internet of things, etc. Of course, the money leads/drives the technological progress at an accelerated rate, and we have seen many ML platforms built recently.

Due to the huge dataset and model sizes involved in training, the ML platforms are often distributed ML platforms and employ 10s and 100s of workers in parallel to train the models. It is estimated that an overwhelming majority of the tasks in datacenters will be machine learning tasks in the near future.

My background is in distributed systems, so we decided to study these ML platforms from a distributed systems perspective and analyze the communication and control bottlenecks for these platforms. We also looked at fault-tolerance and ease-of-programming in these platforms.

We categorize the distributed ML platforms under 3 basic design approaches:
1. basic dataflow, 2. parameter-server model, and 3. advanced dataflow.

We talk about each approach in brief, using Apache Spark as an example of the basic dataflow approach, PMLS (Petuum) as an example of the parameter-server model, and TensorFlow and MXNet as examples of the advanced dataflow model. We provide a couple evaluation results comparing their performance. See the paper for more evaluation results. Unfortunately, we were unable to evaluate at scale as a small team from academia.

At the end of this post, I present concluding remarks and recommendation for future work for distributed ML platforms. Skip to the end, if you already have some experience with these distributed ML platforms.

Spark

On a DAG, an edge E from vertex A to vertex B implies that RDD B is a result of performing operation E on RDD A. There are two kinds of operations: transformations and actions. A transformation (e.g., map, filter, join) performs an operation on a RDD and produces a new RDD.

The Spark user models the computation as a DAG which transforms & runs actions on RDDs. The DAG is compiled into stages. Each stage is executed as a series of tasks that run in parallel (one task for each partition). Narrow dependencies are good for efficient execution, whereas wide dependencies introduce bottlenecks since they disrupt pipelining and require communication intensive shuffle operations.

Distributed execution in Spark is performed by partitioning this DAG stages on machines. The figure shows the master-worker architecture clearly. The driver
contains two scheduler components, the DAG scheduler and the task scheduler, and tasks and coordinates the workers.

Spark was designed for general data processing, and not specifically for machine learning. However, using the MLlib for Spark, it is possible to do ML on Spark. In the basic setup, Spark stores the model parameters in the driver node, and the workers communicate with the driver to update the parameters after each iteration. For large scale deployments, the model parameters may not fit into the driver and would be maintained as an RDD. This introduces a lot of overhead because a new RDD will need to be created in each iteration to hold the updated model parameters. Updating the model involves shuffling data across machines/disks, this limits the scalability of Spark. This is where the basic dataflow model (the DAG) in Spark falls short. Spark does not support iterations needed in ML well.

PMLS

The PS (shown in the green boxes in the figure) is maintained as distributed in-memory key-value store. It is replicated & sharded: Each node serves as primary for a shard of the model (parameter space), and secondary/replica for other shards. Thus the PS scales well with respect to the number of nodes.

The PS nodes store & update model parameters, and respond to the requests from workers. The workers request up-to-date model parameters from their local PS copy and carry out computation over the partition of dataset assigned to them.

PMLS also adopts the Stale Synchronous Parallelism (SSP) model, which relaxes the Bulk Synchronous Parellelism (BSP) model where workers synchronize at the end of each iteration. SSP cuts some slack to the workers for synchronization, ensures the fastest worker cannot be *s* iteration ahead of the slowest worker. The relaxed consistency model is still OK for ML training due to noise tolerance of the process. I had covered this in an April 2016 blog post.

TensorFlow

So TensorFlow is designed to enable that goal. TensorFlow adopts the dataflow paradigm, but the advanced version where the computation graph does not need to be a DAG but can include cycles and support mutable state. I think Naiad design might have some influence on TensorFlow design.

TensorFlow denotes computation with a directed graph of nodes and edges. The nodes represent computations, with mutable state. And the edges represent multidimensional data arrays (tensors) communicated between nodes. TensorFlow requires the user to statically declare this symbolic computation graph, and uses rewrite & partitioning of the graph to machines for distributed execution. (MXNet, and particularly DyNet, uses dynamic declaration of the graph, which improves on ease & flexibility of programming.)

The distributed ML training in TensorFlow  uses parameter-server approach as the figure shows. When you use the PS abstraction in TensorFlow, you use a parameter-server and data parallelism. TensorFlow says you can do more complicated stuff, but that requires writing custom code and marching into uncharted territory.

Some evaluation results

This figure shows the speed of platforms for logistic regression. Spark performs good here behind PMLS and MXNet.

This figure shows the speed of platforms for DNNs. Spark sees greater performance loss going to two layers NN compared to single layer logistic regression. This is due to more iterative computation needed. We kept the parameters at the driver in Spark because they could fit, things would have been much worse if we kept the parameters in an RDD and updated after every iteration.

This figure shows the CPU utilization of the platforms. Spark application seems to have significantly high CPU utilization, which comes mainly as serialization overhead. This problem has been pointed out before by earlier work. 

Concluding remarks and future directions

As far as bottlenecks is concerned, network still remains as a bottleneck for distributed ML applications. Instead of work on more advanced general purpose dataflow platforms, it is more useful to provide better data/model staging; treat data/model as first class citizen.

However, there can be some surprises and subtleties. In Spark, the CPU overhead was becoming the bottleneck before the network limitations. The programming language used in Spark, i.e., Scala/JVMs, affected its performance significantly. Therefore there is especially a need for better tools for monitoring and/or performance-prediction of distributed ML platforms. Some tools addressing the problem for Spark data processing applications have been proposed recently, such as Ernest and CherryPick.

There are many open questions for distributed systems support for ML runtime, such as resource scheduling and runtime performance improvement. With runtime monitoring/profiling of the application, the next generation distributed ML platforms should provide informed runtime elastic provisioning/scheduling of the computation, memory, network resources for the tasks running atop.

Finally there are open questions for programming & software engineering support. What are suitable [distributed] programming abstractions for ML applications? Also more research needed for verification and validation (testing DNNs with particularly problematic input) of distributed ML applications.

I’m spending more of my time online on Discord, a social network of sorts for small communities of gaming fans. This service combines elements of Reddit, chat rooms, and Skype into a single app, and it has facilitated most of my multiplayer sessions over the last couple of months. And while Discord’s clearly one of the best tools for communication in games, it has become something more than that for me. It’s turning into the place I check before Twitter to talk about games and to see what my friends are up to.

Discord is the latest evolution of a long-running type of third-party app that wants to act as the default service that people go to for voice chat and similar features. We’ve seen programs that focused on voice chat or game-specific communities like Raptr, Player.me, or Razer Comms. People also use services like Skype, Slack, and Hangouts to communicate in games. Discord, however, combines all of those ideas into one app that is free, easy to use, and won’t affect game performance.

One of the big reasons that Discord is embedding itself in my life so deeply is PlayerUnknown’s Battlegrounds. This last-player-standing shooter is great with groups, and Discord’s voice-chat quality makes it easy to hear your squad and even include their audio in a livestream.

Having an active game where a lot of people are always looking for a group to play with has led to me joining a huge number of servers, which is what Discord calls the individual user-created communities. But I think I’m going back to Discord between gaming sessions because of the way the company set up those servers to work.

A Discord server feels like an isolated pocket on the internet that you have some ownership of. It’s like creating one of Reddit’s subreddit forums, except it’s more exclusive. The Discord app doesn’t offer you a bunch of fun communities to join, and it won’t put your server in a master list for other people to jump in. Instead, it’s private and controlled by whoever started the server or whoever has admin rights.

You also don’t need to make the server about a game or a topic. I think that, the exclusivity, and even calling them “severs” (instead of “clans” or something else) gives Discord users a sense of ownership over their tiny communities.

Even in the Discords I don’t personally run, I feel a sense of membership. That combined with playing games with the people in these communities has me returning to share news and interesting links with people based on whatever we talked about during our last match. I’m doing that instead of just posting the links into the void on Twitter or occasionally browsing Reddit.

I’m still doing those other things, but it’s just that Discord has found its way into my personal online social flow. And that’s not something I thought would happen with an app like this. I’ve even subscribed to Discord Nitro for $5 per month. This gets you just a handful of random bonus features like an animated icon, and I’m not sure who exactly it’s for. I probably won’t renew my membership next month, but I’ll still have access to all the features that make Discord crucial to my day-to-day gaming life.

README.md

git clone https://github.com/made2591/go-perceptron-go
cd go-perceptron-go
go get https://github.com/sirupsen/logrus
go run main.go

README.md

console.log('hello, world!') //> hello, world!

// Assigning a function to foo constantconstfoo=function () {}

!!'false'==!!'true'// -> true!!'false'===!!'true'// -> true

true=='true'// -> truefalse=='false'// -> false// 'false' is not empty string, so it's truthy value!!'false'// -> true!!'true'// -> true

"foo"++"bar"// -> 'fooNaN'

(![]+[])[+[]]+(![]+[])[+!+[]]+([![]]+[][[]])[+!+[]+[+[]]]+(![]+[])[!+[]+!+[]]// -> 'fail'

(![]+[]) // -> 'false'![]      // -> false

(![]+[].toString()) // 'false'

!![]       // -> true
[] ==true// -> false

!!null// -> falsenull==false// -> false

0==false// -> true''==false// -> true

Number.MIN_VALUE>0// -> true

// Declare a class which extends nullclassFooextendsnull {}// -> [Function: Foo]newFooinstanceofnull//> TypeError: function is not a function//>     at … … …

[1, 2, 3] + [4, 5, 6]  // -> '1,2,34,5,6'

[1, 2, 3] + [4, 5, 6]// joining
[1, 2, 3].join() + [4, 5, 6].join()// concatenation'1,2,3'+'4,5,6'// ->'1,2,34,5,6'

Number()          // -> 0Number(undefined) // -> NaN

parseInt('f*ck');     // -> NaNparseInt('f*ck', 16); // -> 15

//parseInt('Infinity', 10) // -> NaN// ...parseInt('Infinity', 18) // -> NaN...parseInt('Infinity', 19) // -> 18// ...parseInt('Infinity', 23) // -> 18...parseInt('Infinity', 24) // -> 151176378// ...parseInt('Infinity', 29) // -> 385849803parseInt('Infinity', 30) // -> 13693557269// ...parseInt('Infinity', 34) // -> 28872273981parseInt('Infinity', 35) // -> 1201203301724parseInt('Infinity', 36) // -> 1461559270678...parseInt('Infinity', 37) // -> NaN

parseInt(null, 24) // -> 23

parseInt('06'); // 6parseInt('08'); // 0

true+true// -> 2
(true+true) * (true+true) -true// -> 3

// valid comment<!-- valid comment too

typeofNaN// -> 'number'NaNinstanceofNumber// -> false

typeof []   // -> 'object'typeofnull// -> 'object'// howevernullinstanceofObject// false

Object.prototype.toString.call([])// -> '[object Array]'Object.prototype.toString.call(newDate)// -> '[object Date]'Object.prototype.toString.call(null)// -> '[object Null]'

999999999999999// -> 9999999999999999999999999999999// -> 10000000000000000

0.1+0.2// -> 0.30000000000000004

Number.prototype.isOne=function () {returnNumber(this) ===1
}1.0.isOne() // -> true1..isOne()  // -> true2.0.isOne() // -> false
(7).isOne() // -> false

1<2<3// -> true3>2>1// -> false

1<2<3// 1 < 2 -> truetrue<3// true -> 11<3// -> true3>2>1// 3 > 2 -> truetrue>1// true -> 11>1// -> false

3-1// -> 23+1// -> 4'3'-1// -> 2'3'+1// -> '31'''+''// -> ''
[] + [] // -> ''
{} + [] // -> 0
[] + {} // -> '[object Object]'
{} + {} // -> '[object Object][object Object]''222'--'111'// -> 333

[4] * [4]       // -> 16
[] * []         // -> 0
[4, 4] * [4, 4] // NaN

Number  + Number  -> addition
Boolean + Number  -> addition
Boolean + Boolean -> addition
Number  + String  -> concatenation
String  + Boolean -> concatenation
String  + String  -> concatenation

// Patch a toString methodRegExp.prototype.toString=function() {returnthis.source
}/7/-/5/// -> 2

'str'// -> 'str'typeof'str'// -> 'string''str'instanceofString// -> false

typeofString('str')   // -> 'string'String('str')          // -> 'str'String('str') =='str'// -> true

newString('str') =='str'// -> truetypeofnewString('str')   // -> 'object'

newString('str') // -> [String: 'str']

functionf(...args) {return args
}

f(1, 2, 3) // -> [ 1, 2, 3 ]

f`true is ${true}, false is ${false}, array is ${[1,2,3]}`// -> [ [ 'true is ', ', false is ', ', array is ', '' ],// ->   true,// ->   false,// ->   [ 1, 2, 3 ] ]

functiontemplate(strings, ...keys) {// do something with strings and keys…
}

console.log.call.call.call.call.call.apply(a=> a, [1, 2])

constc='constructor'
c[c][c]('console.log("WTF?")')() //> WTF?

// Declare a new constant which is a string 'constructor'constc='constructor'// c is a string
c // -> 'constructor'// Getting a constructor of string
c[c] // -> [Function: String]// Getting a constructor of constructor
c[c][c] // -> [Function: Function]// Call the Function constructor and pass// the body of new function as an argument
c[c][c]('console.log("WTF?")') // -> [Function: anonymous]// And then call this anonymous function// The result is console-logging a string 'WTF'
c[c][c]('console.log("WTF?")')() //> WTF

{ [{}]: {} } // -> { '[object Object]': {} }

({[{}]:{[{}]:{}}})[{}][{}] // -> {}// structure:// {//   '[object Object]': {//     '[object Object]': {}//   }// }

(1).__proto__.__proto__.__proto__// -> null

(1).__proto__// -> [Number: 0]
(1).__proto__.__proto__// -> {}
(1).__proto__.__proto__.__proto__// -> null

let x, { x: y =1 } = { x }; y;

let x, { x: y =1 } = { x }; y;//↑       ↑           ↑    ↑//  1       3           2    4

[...[...'...']].length// -> 3

[...'...']             // -> [ '.', '.', '.' ]
[...[...'...']]        // -> [ '.', '.', '.' ]
[...[...'...']].length// -> 3

[...'...']                 // -> [ '.', '.', '.' ]
[...[...'...']]            // -> [ '.', '.', '.' ]
[...[...[...'...']]]       // -> [ '.', '.', '.' ]
[...[...[...[...'...']]]]  // -> [ '.', '.', '.' ]// and so on …

foo: {console.log('first');break foo;console.log('second');
}//> first// -> undefined

a: b: c: d: e: f: g:1, 2, 3, 4, 5; // -> 5

(() => {try {return2;
  } finally {return3;
  }
})()

new (classFextends (String, Array) { }) // -> F []

(function*f() { yield f })().next()// -> { value: [GeneratorFunction: f], done: false }

(function*f() { yield f })().next().value().next()// -> { value: [GeneratorFunction: f], done: false }// and again
(function*f() { yield f })().next().value().next().value().next()// -> { value: [GeneratorFunction: f], done: false }// and again
(function*f() { yield f })().next().value().next().value().next().value().next()// -> { value: [GeneratorFunction: f], done: false }// and so on//…

(typeof (new (class { class () {} }))) // -> 'object'

constfoo= {class:function() {}
};

functionnonCoercible(val) {if (val ==null) {throwTypeError('nonCoercible should not be called with null or undefined')
  }constres=Object(val)

  res[Symbol.toPrimitive] = () => {throwTypeError('Trying to coerce non-coercible object')
  }return res
}

// objectsconstfoo=nonCoercible({foo:'foo'})

foo *10// -> TypeError: Trying to coerce non-coercible object
foo +'evil'// -> TypeError: Trying to coerce non-coercible object// stringsconstbar=nonCoercible('bar')

bar +'1'// -> TypeError: Trying to coerce non-coercible objectbar.toString() +1// -> bar1
bar ==='bar'// -> falsebar.toString() ==='bar'// -> true
bar =='bar'// -> TypeError: Trying to coerce non-coercible object// numbersconstbaz=nonCoercible(1)

baz ==1// -> TypeError: Trying to coerce non-coercible object
baz ===1// -> falsebaz.valueOf() ===1// -> true

At Roompact, we make a SaaS product used by university residence life departments across the United States. Our software provides an array of features that range from form-based tools, to digital roommate agreements, to email and text message broadcasting, to a central news feed that acts as both a communications tool and data aggregator for residence hall staff.

Roompact was founded in 2013, and since its inception, a combination of plain JavaScript, jQuery, and an assortment of jQuery-esque libraries had been what the entire front end of the application was built with. In October of 2016, I realized that we were long past due for an upgrade. The straw that broke the camel’s back occurred when a feature we had worked on over the summer and released in August had already started feeling like a legacy application. Built as a single-page application (SPA) making very heavy use of Ajax calls to a JSON-based RESTful API as its back end, its 5,000 lines of front end code were already becoming very difficult to work with and modify, hardly a month after release. “When did this turn into jQuery spaghetti?” I thought.

Trying Elm

The search for a replacement front end framework had me considering React and Vue.js as the top candidates for several weeks. But I had a feeling that I should examine Elm more closely. I had read about Elm in the past and it had been showcased at a Haskell meetup I attended in Chicago. I was not by any means an advanced Haskeller when making the consideration to use Elm, but I knew that several years of writing small Haskell programs and reading about the language would have left me with knowledge transferable to Elm. It was also my belief in the benefits of statically typed functional programming that made the opportunity to at least try Elm too tempting to pass up.

I decided to follow the general strategy outlined in Evan Czaplicki’s How to Use Elm at Work post. One day in November of 2016 I set out to make a small internal tool that solved the problem of having no UI to configure a certain one of our features. The project took me about three full days to complete. The experience involved a lot of fighting against the compiler and resisting the feeling of being trapped by having to write in a functional style. By the end, I had written a bit over 500 lines of Elm. I realize that despite running into compiler errors often, and having to completely eschew techniques that were commonplace in imperative code, the constraints imposed by Elm were actually rather helpful. The compiler offered protection against silly mistakes, and functional code was easy to read and naturally highly composable. These were the types of benefits that I had read about when learning the fundamentals of Haskell.

Teaching Elm

A few weeks after finishing my little project, I introduced one of my engineers to Elm. He would be writing another internal tool that was quite a bit larger than mine. More importantly, he had no previous functional programming experience whatsoever. In order to be able to adopt Elm at our company, it was absolutely imperative that I would be able to get him productive in what was to him a completely foreign language rather quickly. In order to accomplish this, my approach was to:

1. Pair program with him on a significant portion of this initial project, but make him write most of the code
2. Emphasize the importance of reading type signatures, both in our own code and in any documentation we were referencing
3. Treat compiler errors as helpful feedback, rather than as a signal indicating failure
4. Practice approaching each problem with a functional mindset (e.g. “How can we apply List.map or List.filter rather than a for loop and array mutation?”)

This project took him a few weeks to complete. The end result was highly successful, and both him and I learned quite a bit about Elm as we worked on it together. Most importantly, by the end, my engineer was comfortable enough with Elm as to work independently for extended periods of time. The lesson had been a success, and it also proved that about a week of intense training, someone who has never written functional code can build a solid understanding of the basics of Elm. Within month, they should be able to work independently on code that will eventually make it into production.

Work has gotten really interesting again.

Another important factor that emerged during this process was a human one: writing Elm code was both fun and interesting. My favorite quote from my new-to-Elm engineer during this process was the one above. Hearing this was not a top initial priority when looking for a new tool, but it was a very reassuring thing to hear. As the Chief Technology Officer of a company whose main focus is to build software, my responsibilities do not end at ensuring my team is productive. I view it as an obligation to ensure that each team member feels the importance of their work, engages in work that they have a personal affinity towards, and continually develops professionally. Simply writing code in Elm was immediately hitting two out of three of these goals.

Using Elm

With these two trials of Elm being very successful, I had all the evidence I needed: we were going to move forward with Elm. Our first user-facing application of Elm would come in short order. Without going into extensive detail, we spent the entire first half of 2017 making the largest and most complex feature that Roompact has ever seen: a highly customizable form-builder system with integrations to the rest of the data in our software.

With almost every single piece of data on each page in this feature being dynamic (questions, input types, order values, tags, answers, form template and submission edit histories, etc.), the need for managing all of this data effectively was paramount. Moreover, this data would have to be shared across multiple views seamlessly: an edit to a form template would have to be reflected in the corresponding form submission creation page immediately; a new form submission would have to be visible in the multiple tabular views in addition to its own individual page view.

Not only was the scope and complexity of this feature to be extremely broad, but it would also serve as a replacement for two existing features that were no longer up-to-par, and not worth updating. This would easily be the most high-stakes project we had ever undertaken.

With all of this in mind, I was quite convinced that Elm would be the best tool for the job. And so, we used Elm for the entire front end.

Released in early June, it is now over 22,000 lines of Elm code in the form of a single Elm application. Feedback from our summer users has been nothing short of glowing. I am certain that it was the decision to use Elm that make it possible to build such an intricate front end to such a high degree complexity without making any compromises in performance or reliability.

Reflecting on Elm

In the rest of this post I am going to outline what we have learned as we have used Elm; both its strengths and its weaknesses. This is not meant to be an Elm tutorial, but it is meant to inform someone with little-to-no knowledge of the language of its distinguishing features. Every point discussed below is aimed at addressing the experiences of using Elm in a production setting. That is, software that will likely be written by several people, that must be bug-free, performant, address a certain set of functionality requirements, and will see significant use by end users.

Elm has an incredibly powerful type system

Relative to other front end tools, Elm’s type system is its most distinct and powerful feature. Elm is statically typed, meaning all code is verified during the compilation process (more on that later). More importantly, Elm allows for the creation of Algebraic Data Types, which are referred to as Union Types in Elm. This allows the programmer to model much of the business logic of the application in type system, to be verified statically by the compiler, rather than in code that will be evaluated at runtime.

One simple example is that of a three-way state. Suppose I have a tag input field. When I arrive to the page with this input, certain tags may already be present. I can then edit the tag list, either by adding or removing tags. But here’s the catch: if I press “Cancel” to return to a previous view in the application, the tag list much revert to its original state. If I press “Save” the changes must be applied.

There are a number of ways to do this in JavaScript, one solution would be:

1. in addition to the master tag array for that field, create temporary arrays for added and for removed tags
2. for each tag change, apply it to the master array and keep a record of the change in the corresponding temporary array
   • if a tag is removed, remove it from the master array and add it to the removed array
   • if a tag is added, add it to the master array and to the added array
3. apply or revert the changes depending on the final user action
   • if the “Save” button is pressed, use the tags stored in both temporary arrays to permanently save the changes (e.g. http request to the back end)
   • if the “Cancel” button is pressed, use the temporary arrays to identify which tags to add back into or remove from the master array
4. clear the temporary arrays

In JavaScript, our solution relies on using several data structures to help with the bookkeeping of keeping track of which tags were added or removed. This might work, but I have to make many considerations in order to avoid possible errors. What if something is not re-initialized correctly after the user visits the page for a second time? Are we resetting them every time that we need to? How do we keep the tag list that the user sees in the DOM in sync with the state?

I chose this approach because there is no great way to model the current state of each tag in the tag itself. Attempting to do so might involve adding a status field containing a string that indicates one of the three possible states. Or worse yet, I could try to model this by juggling several boolean fields (e.g. added_status, removed_status).

Even the single-field approach likely to result in chains of if-statements that perform string comparisons in several places in the program. I will also have to ensure that my status field is always initialized with any tag object in order to protect against runtime errors. I could attempt to solve this latter problem by creating prototype functions that extend a Tag constructor object, but there is nothing forcing anyone to use these prototypal functions to create Tag objects to begin with. In short, adding such a field to my tag object is an encumbrance, as I have to remember to handle this extra field throughout my program.

By contrast, in Elm, modeling the possible states of each tag is incredibly easy. We can write this as follows:

-- The type representing the possible states of the tagtypeTagState=Current|Added|Removed-- The type representing a single tagtype alias Tag=
    { tagId :Int
    , tagName :String
    , tagStatus :TagState
    }

In the above example, every value of type Tag will contain a tagStatus field, which will contain a value of type TagState, which in turn has to be one of the three states I want to represent. The important thing to note here is that now every tag value must have a tagStatus field, and it must always contain one of the three defined TagState values. If this field is not initialized (e.g. at the JSON decoder for a tag) or its values are not handled exhaustively (e.g. in my view code), the program will not compile. I will show an example of the latter scenario below.

Elm has a great compiler

Elm’s compiler is what does the heavy lifting of enforcing the constraints of the type system, and I would consider it to be a huge asset when writing Elm code. Other statically typed languages have compilers, but Elm’s is in a league of its own.

Let’s consider the snippet of code in the section above. With the above types, every Tag in my application to always have a TagState. Let’s see how we would use this to our advantage to address the problem of keeping our DOM in sync with our data:

-- Function that takes a tag value and returns an html-- value that will be rendered by the Elm program
displayTag :Tag->HtmlMsg
displayTag tag =-- perform a match against the tagStatus field of the tag parametercase tag.tagStatus ofCurrent->-- Display a standard tag
            div [ class"tag" ] [ text tag.tagName ]Added->-- Display a tag, but include the 'tag-added' class-- so that we can style these tags differently
            div [ class"tag tag-added" ] [ text tag.tagName ]Removed->-- Display an empty div*
            div [] []

* Note that it would be possible to restructure this code in a way that does not display an empty <div> in the case of a removed tag, but the above code results in the clearest example.

But what if we had forgotten to handle one of our cases? Suppose we did not include the Removed case.

==================================== ERRORS ====================================

-- MISSING PATTERNS --------------------------------------------------- TagDisplay.elm

This `case` does not have branches for all possibilities.

72|>    case tag.tagStatus of
73|>        Current ->
74|>            div [ class "tag" ] [ text tag.tagName ]
75|>
76|>        Added ->
77|>            div [ class "tag tag-added" ] [ text tag.tagName ]

You need to account for the following values:

    Removed

Add a branch to cover this pattern!

If you are seeing this error for the first time, check out these hints:
<https://github.com/elm-lang/elm-compiler/blob/0.18.0/hints/missing-patterns.md>
The recommendations about wildcard patterns and `Debug.crash` are important!

Detected errors in 1 module.

The compiler tells us what the cause of the error is, exactly where it is, and what we need to do to fix it. The large majority of compiler errors in Elm are written in this manner. This is wildly helpful when dealing with something like potentially dozens of user-defined types, each with numerous possible values. This is particularly helpful when one developer might be editing another developer’s code: it is not necessary to look at the definition of the TagState type in order to safely edit code that relies on it; if I miss something the compiler will let me know.

There is one other key benefit that comes out of this combination of static typing a powerful compiler: Elm absolutely never encounters runtime exceptions. In a talk recorded in April of 2017, Richard Feldman from NoRedInk describes how the 100,000 lines of Elm code they have built up since 2015 have never thrown a runtime exception.

Writing in a functional style has significant productivity benefits

Everyone who knows about Elm knows that it is a functional language. However, I think that relatively few developers have written enough functional code to build an appreciation for just how pleasant reading and writing functional code is, and the productivity gains that come as a result. Here are some of the hallmark features of Elm code:

Pure Functions - Virtually all functions in Elm are considered ‘pure’. This means that given a set of parameters, a function will always produce the same result. Such functions also referred to as being referentially transparent, meaning they can be replaced with their corresponding return values without altering the behavior of the program. Because of the constraints that enforce this property, pure functions lack the ability to produce side effects (making HTTP requests, changing HTML on the page, printing output somewhere, etc.).

These traits combine to result in a significantly lower the cognitive load required to read Elm code. For example, if you see a function whose type signature is Int -> Int, meaning that it takes one Int parameter and returns an Int value, you can safely assume that it will at most be doing some sort of numerical manipulation without any other side effects. If you are searching for code that validates email addresses, you know that you can look elsewhere (perhaps for a function that takes a String and returns a Bool).

Immutable Values - All values in Elm are immutable; they cannot be changed after they are set. This may seem limiting to someone coming from writing JavaScript, but in a functional paradigm, changing values is not necessary. The standard approach is to return a new value rather that overwrite an old one. Immutability eliminates a whole array of possible issues in a program, ranging from race conditions caused by concurrent code, to uncontrolled global state modification.

Higher-Order Functions - A higher order function is simply a function that takes another function as a parameter, or returns a function as its result. This style of programming is very common in Elm and leads to code that is well suited function composition, and in turn, reusability. Take a look at an example of the map function below (which takes a function and applies that function to every element in a list):

square :Int->Int
square x = x * x

normalList = [1, 2, 3, 4, 5]

squaredList = List.map square normalList

In the above code, the square function is applied to every element in normalList. When this code is evaluated, squaredList will contain [1, 4, 9, 16, 25].

Pattern Matching - This is without a doubt one of the most useful features in Elm. Pattern matching allows you to to write code that will only get evaluated when the “shape” of the value being examined matches the defined pattern. Consider the following example:

typePermissionLevel=AdministratorPermissionLevel|StandardPermissionLeveltypeUserGroup=AdministratorUserGroup|StandardUserGroup-- Function checks whether a user can edit a post

checkIfUserCanEditPost :PermissionLevel->UserGroup->Bool
checkIfUserCanEditPost requiredPermissionLevel currentUserGroup =case (requiredPermissionLevel, currentUserGroup) of-- A standard user can edit a standard post
        (StandardPermissionLevel, StandardUserGroup) ->True-- An administrator can edit any post
        (_, AdministratorUserGroup) ->True-- Deny any other possible combination of values
        _ ->False

In the above code, we combine the requiredPermissionLevel and currentUserGroup parameters into a tuple in the case statement and evaluate them together as we try to match one of the cases. The _ value will match any value in the case statement. We use it to avoid having to define (StandardPermissionLevel, AdministratorUserGroup) and (AdministratorPermissionLevel, AdministratorUserGroup) as two separate cases. Instead, we tell compiler to produce code that evaluates to true anytime the user is an administrator. We also use the _ value in the final case statement as a catch-all, to deny all other possible combinations of required permission level and user group.

Elm’s Model-Update-View architecture is very well-suited for building web applications

The core architectural pattern in every Elm web application is what is referred to as The Elm Architecture, consisting of three main parts:

The Model is the state of the application. The Model consists of a single data structure that contains every piece of data used in the application. It will usually grow incrementally as an application grows in features and complexity, but how it is structured is up to the developer. In most applications the Model will take the form of a record type (similar to an object in JavaScript) which may have any number of top level fields that may be any type of data structure (including themselves being record types).

The Update is the portion of the application that handles both changes to the Model as well as any I/O that the Elm application has to perform (http requests, calling external JavaScript functions, etc.). No other portion of the program can change the state of the Model or perform I/O. The Update is called anytime a Msg value is produced in the application. Such a Msg value will usually represent the action and may have additional data bound to it (e.g. UserSearchInput "john").

The View is the portion of the program that renders HTML and handles user inputs. The View always takes the Model as an argument, so any conditional logic that uses data to dictate how the HTML on the page is changed during runtime must be based off of data in the Model. The View is automatically called by the Elm runtime anytime any value in the Model changes. User inputs in the View will produce Msg values, which will result in the Elm runtime invoking the Update.

So in general, the execution of an Elm program is as follows:

1. The Model enters a particular state
2. The View is rendered based off of the state of the Model
3. The user interacts with the application, a Msg value is produced
4. The Update is called, receiving the Msg value as a parameter, which results in a change to the Model (return to step 1)

This structure and the separation of concerns between the different portions of the application make it easy to both build and later refactor even extremely large applications. This structure also eliminates nearly all issues with data going out of sync with the DOM, or different DOM elements being out of sync from one-another, as the View will always re-render the DOM based off of the contents of the Model. To someone who is not used to using this type of architecture, seeing it in action for the first time may feel like magic. It is not uncommon to think “wow all I did was change the value in the model and all of the HTML relies on that value updated automatically”.

Elm has a very powerful debugger

One of the features released in the latest version of Elm (0.18) is known throughout the Elm community as the Time Traveling Debugger. When opened, the debugger displays the current state of the program as well as the history of Msg values produced as the user has interacted with the program. When one of the older Msg values in the list is clicked, the entire Elm application will revert to that point in history in the execution of the program. This which will include the state at that point in time as well as the entire contents of the DOM. Clicking through the Msg list effectively allows one to replay the entire history of the current session.

What’s more is that this entire history can be exported as a JSON file. So a user can be asked to reproduce the steps that led to a bug, export the history, and send that history file to a developer that will fix the problem. The debugger is easily enabled via a --debug flag appended during compilation time, and requires no external tools or plugins.

Elm has a readable syntax

This may be a point of contention, but it is my opinion that Elm’s syntax (largely taken from Haskell), is very readable:

• It tends to be quite terse, with individual lines of Elm code being very short.
• It does away with unnecessary curly brackets and semicolons.
• Function parameters are delineated via spaces.
• Functions and variables start with lower case letters and are always written in camelCase.
• Types and module names always start with uppercase letters and are written in PascalCase.
• Elm code is indented only with spaces (killing the tabs vs. spaces debate).
• Indentation largely does not matter in most places to begin with (though most of the community uses a utility to auto-format their code, discussed below).

Elm has a great set of standardized development tools

The Elm community has developed a number of tools over time that remove many of the pain points of development.

• The standard elm binary comes with four primary utilities:
  • elm make - the compiler (which is also what enables the debugger via the --debug flag)
  • elm package - the package build tool; can be used to quickly install new packages or download all dependencies in an existing project
  • elm repl - a simple read-eval-print loop; a good way to test code and ideas in the command line
  • elm reactor - a simple web server and websocket-based live reload used for rudimentary development
• elm-format - a utility that auto-formats code to the community standards; most editors will have plugins to have to auto-run each time a file is saved
• elm-upgrade - a utility that helps automatically upgrade much of your code when a new version of Elm is released (more on this later)

The Elm community is very friendly and helpful

In my experiences, the following three communities are the best place for Elm help and discussion:

• /r/elm - The Elm Reddit community. A place for news and general discussion.
• Elm Slack - The Elm Slack team. A great place to go for help on a certain topic. There are always users active here, and in my experiences, skill levels range from total beginner to expert. I have been surprised to hear answers from people who obviously have very advanced knowledge of things like the Elm compiler; it is reassuring to know that language experts have a willingness to help Elm novices.
• Elm Discuss - The Elm Google Group. Like the other two, this community is very active, with discussions ranging from beginners asking for help, to discussions of very advanced topics, project proposals, etc.

Elm applications have excellent performance and additional optimization is easy

Although Elm is a very high level language, the JavaScript that the compiler produces is extremely fast.

In August of 2016, Elm 0.17 (last version of Elm at the time of this writing), even non-optimized Elm code was able to outperform React, Angular 2, and Ember.

In a different set of benchmarks posted in May of 2017, Elm’s performance was on par with React and Angular 4, which were all among the fastest frameworks in the benchmark (having a slowdown of 1.30 - 1.40 relative to the optimal vanilla JavaScript implementation of the benchmark). The fastest framework intended for production use, Inferno, scored 1.07.

In practice, runtime performance should rarely be a concern with Elm for normal web applications. In the event that additional optimizations are required, elm has two libraries which can increase the performance of the application.

• Html.Lazy - contains functions that can be used to cache the results of view functions, reducing the need to re-render certain elements.
• Html.Keyed - contains functions that can be used to optimize situations when elements are getting added, removed, or re-ordered (such as in a list)

Functions in both of the above modules are drop-in replacements for functions from the standard Html module, meaning no substantial rearchitecting of code is required to apply such functions.

Elm applications can be rather large when compiled

The amount of code produced by the Elm compiler can be somewhat lengthy. Our 22,000 line application compiles to a file with over 53,000 lines of JavaScript that is 1.6MB in size.

Fortunately, using the Google Closure Compiler with simple optimizations enabled, this file can be reduced to be a mere 450KB in size. A minor downside of using the GCC is that it is written in Java, so the Java Runtime is required to run it.

There is a version written in Node.js, but it is considered an experimental release, and in my experiences, this compiler is far slower on large JavaScript files such as the one in our case. I have a powerful Core i7 6700K @ 4.00Ghz on my Linux machine, and whereas the Java version takes 2-3 seconds to complete, the Node.js version takes 43 seconds. On substantially weaker hardware, like an EC2 instance used for a development or build environment, the JavaScript version would take several minutes to run.

The Elm core libraries do not yet have certain web API bindings, but interoperability with JavaScript code is safe and easy

Not everything is possible in native Elm code. For example, there is no official library for using the localStorage API, as the Elm API is still being developed. However, the good news is that Elm has very well thought out JavaScript interoperability. There are two primary methods for doing so:

• Ports can be used to talk to any JavaScript code outside of the Elm application during runtime. This technique can also be used to send data into Elm with functions that the Elm application object exposes.
• Flags can be used to set values in Elm during the initialization of the application. This may be used to set values passed to the browser from the web server that Elm must have access to, such as user configuration settings or authentication tokens. This technique would likely be used when Elm is being used in a full stack MVC framework (such as Ruby on Rails or Laravel).

A distinguishing feature of this interoperability model is that it ensures type safety and the integrity of the program. Even though an Elm application makes the use of flags and ports, it will still use compile-time type checking and never encounter runtime exceptions.

The Elm language and core libraries are prone to change as new versions of the compiler are released

This point may sound scarier than it is, but its implications should certainly be considered before using Elm for production use. Elm is currently on version 0.18, with a 0.19 release coming likely in the next few months. 0.17 was released in December 2016, right as we were finishing our second small Elm project. As a result, we experienced the process of upgrading Elm in our first two small applications. To give a summary of the experience:

• The process was well documented and relatively simple. There were several syntax changes, but most of this was handled by the elm-upgrade tool I mentioned above.
• There were a few changes to the way the HTTP library worked, which we had to spend some time on. This portion of the process likely took a lot longer than it would today, as we were so new to Elm.
• We had to wait to upgrade until all of our dependency libraries were upgraded as well.

I will add that this last point may have huge potential ramifications in a production application. I have been somewhat reserved in using external libraries in our application, particularly avoiding ones with large dependency trees or a lot of code. I can foresee a scenario where some dependency used by your application gets abandoned, and your team may have to support it. Certainly this is a risk with any library in any language, but the Elm compiler will refuse to even attempt to compile your application unless every dependency in your project supports the current version.

If such an abandonment scenario takes place with single module library that has 150 lines of code, maintaining that library will likely be quite straightforward. It may not even require any changes, short of a version bump in the package file. But I would stay far away from a library like elm-mdl, a material design implementation in Elm, which contains 10,000 lines of Elm code. If your application becomes highly dependent on such a library, and the maintainer stops supporting it, you will have to make the decision between forking the library and maintaining it on your own, or never upgrading to the newest version of Elm.

In Elm, doing what might seem hard can actually be quite easy, and the inverse is also true

All of Elm’s unique features often come together to produce a language that often flips the definitions of ‘easy task’ and ‘difficult task’ on their heads.

• Want to add a new possible value to one of your union types, which will necessitate changes to the code in dozens of places in your application? Easy, just start by changing your type definition and the compiler errors will help you find everywhere that needs updating.

• Want to decode some JSON? Hard, especially if the JSON is heavily nested and it must be decoded to custom types defined in your application. Doing this will require an understanding of how JSON decoding works in Elm and will usually result in quite a bit of code (our application contains over 900 lines of JSON decoders alone).

• Want to create multiple different views in your application that each have complex data interdependencies? Easy, The Elm Architecture’s Model-View relationship make this type of thing almost trivial to do. It will also be almost impossible for the data to get out of sync across different views.

• Want to measure the height of an element on the page at the moment a user clicks on it? Hard, in order to do this we had to make heavy use of event bubbling and writing JSON decoders for the native event.target object that is produced by an onclick event.

Generally speaking, however, the trade off is worth it. The easy tasks that become difficult usually do so because you gain some sort of benefit (such as type safety) as you do them in Elm. The difficult tasks that become easy usually result in massive productivity and reliability gains, particularly as an Elm application reaches large sizes. Our 22,000 line Elm application is easier to refactor than our 5,000 line jQuery-based application by a wide margin. The Elm application will age well, only becoming more reliable and performant as we make improvements and add features over time. The jQuery-based one will not, and will be slated for replacement when its limitations become too prominent.

Conclusion

Using Elm in production has been a been a very successful endeavor at Roompact. Our latest project, with a front end written solely in Elm, has exceeded all expectations, both those of our users as well as our own. We have managed to take a set of functionality that would have been exceptionally difficult to build using our old methods, and using the strengths of the Elm language and architecture, successfully developed the largest feature in our entire software product to date. All of this done with a very high degree of maintainability and reliability. This post has been a record of our experiences with Elm up to this point.

The decision to use Elm for the first time was difficult due to the risks associated with the unknowns that would come with a departure from normalcy. The decision to continue using Elm will not be.