This started out as a Hacker News comment about a blog post about select being fundamentally broken After that the same author wrote why epoll is broken as well. I don’t know what problem author is trying to solve but let’s pick this one:

Let’s say you run an HTTP server, serving a large number of short lived connections. You want to accept() as many connections per second as possible. Doing accept() in only one process will surely be CPU-bound. How to fix it?

Now I’m no network programming expert but I have written enough of networking code in C (and C++, Python, Java) over the last 17+ years and I have thought about the C10K problem and read papers on architecture of servers, proxies, load balancers and using socket API efficiently. People working on libev and HAProxy definitely have a better understanding and might find flaws in my arguments.

Using select the correct way

Literature ussually says that fd_set is limited to 1024 descriptors but some mention that you can increase that by defining FD_SETSIZE to a larger number. Last time I checked it did not work on Linux although it did on SunOS and AIX. However fd_set is just an array of bits and you can allocate memory for as many bits as you need. And turns out that all select implementations I worked with accepted larger arrays of bits than the standard fd_set allocated. I still had to put #define FD_SETSIZE 65536 in my code for SunOS to use select_large_fdset function instead of select.

fd_set *wfds_out = calloc(FDS_BYTES(65536), 1);

To find out which sockets have events some loop through data structures and check each socket they contain. But a much better way is to iterate through fd_set as array of long and check 32-64 bits at a time. If any of bits is set only then check them individually in a nested loop. Some implementations even check read, write and exception sets at the same time by ORing bits - Linux kernel does that in fs/select.c.

unsigned long *wfds = (unsigned long *)wfds_out;
for (unsigned i = 0; i < FDS_LONGS(maxfd); i++, *wfds++) {
  unsigned long bits = *wfds;
  if (bits != 0) {

    unsigned long bit = 1;
    for (unsigned j = 0; j < FDS_BITPERLONG; j++, bit <<= 1) {

      if (bits & bit) {
        int fd = i * FDS_BITPERLONG + j;

Once a notification is received for listener socket the best approach is to call accept multiple times up to certain limit (10? 50?) or until EAGAIN or EWOULDBLOCK error is returned indicating there are no connections pending. This way more useful work is done per iteration amortizing the cost of “expensive” select call. I found this reading “Scalability of Linux Event-Dispatch Mechanisms” by Abhishek Chandra and David Mosberger.

while (1) {
  int client = accept(server, (struct sockaddr *)&sin4, &sin4_len);
  if (client == -1 && (errno == EAGAIN || errno == EWOULDBLOCK)) {
    break;
  }

Another useful idea is to trade latency for throughput with interval control. It works by limiting how many select calls are performed per second and enforce it by sleeping before the next call to select. During that time more events will occur on sockets and the next select call will return more events to process. Again, more useful work will be done per iteration. The source for this is paper “Efficient Network I/O Polling with Fine-Grained Interval Control” by Eiji Kawai, Youki Kadobayashi, Suguru Yamaguchi.

while (1) {
  tnow = usec();
  if (tnow - tprev < 1000) {
    struct timespec spec;
    spec.tv_nsec = 1000 - (tnow - tprev);
    nanosleep(&spec, NULL);
  } else {
    tprev = tnow;
  }

  int nready = select(server + 1, &rfds_out, NULL, NULL, NULL);

All the small things

select modifies descriptor sets passed in. A common mistake is to create descriptor set from zero before calling select by looping through data structures and adding all sockets they contain. I’ve seen that to take huge amount of time due to locking and synchronization between threads. The correct way is to maintain a descriptor set all the time and create a copy of it and pass the copy to select.

select returns number of file descriptors in resulting sets. You should count events you have processed and stop once you reach number returned by select.

Most programs will need to store some state associated with the socket. Map seems to be a good choice at first but you should use an array indexed by socket (it’s an int). The POSIX API says that kernel always has to use the lowest unused file descriptor when creating a new one. So the array will be as big as many connections your program handles and kernel will take care of finding a free slot in your array.

And if you use the hard limit returned by getrlimit you can always pre-allocate both the array for state of connections and descriptor sets for select. Which means no re-allocations during runtime and no synchronization between threads. If the state of your connections is really big, allocate array of pointers.

Results

So I wrote sample code using most of the ideas I described above. I wrote it from scratch not using any of my previous work so it “works on my computer” and there are bugs. I used a a small Linux virtual machine to run it with just 1 CPU and 512 Mb of RAM. This is how I started the server:

./accept-server 1025 > accept-server.log

And client in a different console:

./accept-client 100 127.0.0.1 1025 > /dev/null 

And it gave me following results:

awk '{count[$1]++} END {for (word in count) print word, count[word]}' accept-server.log | sort -n
1496608453 40154
1496608454 46292
1496608455 44843
1496608456 47250
1496608457 48005
1496608458 47204
1496608459 45685
1496608460 45992
1496608461 41870
1496608462 44020
1496608463 45392
1496608464 44592
...

First column is the timestamp (seconds since epoch) and second is number of connections accepted.

accept-server was consuming 24% of CPU time and accept-client 71% of CPU time. I think results are pretty good: 40,000 accept calls per second. Passing new sockets to worker threads would take some time but it’s still possible to handle tens of thousands of accept calls with a single thread.

select vs others

poll transfers 8-16 bytes per descriptor to the kernel compared up to 3 bits for select. poll is more efficient for very sparse descriptor set and I would use it if I had a listener thread (poll for few sockets) and separate thread for connections (select for thousands of sockets).

epoll uses system calls (“slow”) to add or remove descriptor to the set. select does the same thing in userspace. If you have to add and remove many descriptors per iteration which is typical for short-lived connections then select might be more efficient. In fact it was for synthetic tests I did years ago.

Closing words

I usually ignore topics about the fastest networking code since for problems I work on application logic and database access code is the bottleneck. I have seen badly written networking code that caused problems but not code where the accept call is the bottleneck. The other thing is once you reach certain number events per second you should distribute work among several processes and machines anyway. That may differ if you work for Google or large ISP or try to develop a load balancer or a proxy.

Martin Thompson often talks about mechanical sympathy - that you have to understand how the hardware works and take that into consideration when you design software. Well, you should have sympathy for your Operating System and it’s APIs and find the best way to use it.

Use libev, Boost.Asio or other libraries. I do. Just because I know how to use select efficiently does not mean I want to write low-level code.

Introduction

My name is Eric Wing and we’re going to talk about Swift on Android.

Why Swift on Android?

I want to assume everybody can imagine useful cases for having a cross-platform native language like Swift available on Android. I’m using the real Apple Swift. If you’ve heard of the RIM object Silver Swift, this is not that. I’ve given talks about Android before and I’m brutally honest about Android. While the experienced Android developers always thank me at the the end for telling it as it is, people who are not familiar with Android development are in disbelief about how bad things can be and start to question my credentials.

My Background: Worn lots of hats

I’m not just an Android developer but a Cocoa developer, and I’m trying to bridge the words for the audience. I worked on a global satellite communication system called Globalstar (e.g. launching satellites into space with rockets!).

I worked in cross-platform development during the end of the Unix wars and the peak of the Microsoft monopoly. Scientific Visualization was a specialization I had. At the time, certain engineering niches needed higher liability and Windows was too unstable, but all the Unix vendors are dying. Then a solution came with Mac OS X, which promised as real Unix combined with a good GUI. Eventually, my work shifted from porting to Mac to embracing Cocoa to create first class UIS deeply integrated with OpenGL visualization systems.

I got involved in open source projects. It usually related to improving Apple platform support. SDL is the ultimate cross-platform foundation layer used heavily by the video game industry. It provides access to the graphics system, audio, input events, and anything you need to create a game or multimedia app. If you’ve played a cross-platform triple-A game in the past 15 years, it is better than a coin flip that uses SDL. Valve Steam has adopted SDL as a foundational component, it’s important to the video game industry. CMake is a cross-platform meta build system. You list the files that you need to build in a text file, and CMake will create native projects for it like Visual Studio, X Code and Make Files.

I wrote the world’s first full featured bridge between the Lua language and Cocoa. I’ve used many low-level parts of the eject and see runtime that most people don’t even know exists. I understand language bridging.

I coauthored the book Beginning iPhone Games Development, published by Apress.

I worked on some commercial game engines. I was the chief architect of the Corona SDK, which allowed people to write native cross-platform games in Lua. I later co-founded Lanica to build a game engine for accelerator people could write native cross-platform games in JavaScript. Since Android development has many problems, I inadvertently became an Android expert out of necessity.

As far as I know, I was the first to create a proper Swift Android app - February 27, 2016.

Ouroboros: Eternal cycle of life & death. What’s old is new again

Ouroboros is an ancient symbol of a serpent or a dragon eating itself. It represents an infinite cycle or something constantly recreating itself. With this talk, I hope to give you much more than a checklist of things to do. I want you to see the fundamental concepts that allow everything to work. Nothing here is new. We are reapplying old concepts in slightly different ways, when we talk about the Swift Compiler in native code, think about C compilers in native code. When we talk about Android, think about Unix and Linux. Seeing these analogies may help you understand how everything works.

The most important fundamental that keeps reappearing in this talk is C. C makes everything possible. C is like the building block for everything else because it has special properties. C is the most portable language. Every platform has a C compiler, even the web has a C compiler now. The C ABI is stable and everything is built on top of it. Almost all languages have a way to talk to C. Swift in particular, has one of the best. There is software written in C, which all the other languages can use because they all know how to talk to C.

Language vs. Libraries

When we talk about using Swift on Android or any other language for that matter, it is helpful to separate the difference between the language and the libraries because when I talk about using Swift on Android, I do not mean using UIkit on Android. UIkit is a library and one that is unlikely to ever be reported to Android, I’m talking about the language.

There is this fuzzy area with the Swift standard library base, which I boxed in yellow. You can see there are multiple libraries. Swift Core is the only one I call essential. It contains definitions for basic types we take for granted like int, and I believe it contains the Swift runtime. The others are more optional. I want to emphasize that we are getting the base Swift language working on Android and you don’t need the rest to make fully capable, shippable apps with Swift.

Here’s an example. Imagine these apps are written in Swift and all you must use Swift Core, but after that, each uses a different set of libraries. At the top is our native iOS app. It uses all the Swift standard libraries plus Uikit and Core Audio. Next, consider a native Android app written in Swift. This app uses the Swift C standard library, but there is no Uikit or Core Audio on Android, for native Android, we would use the Android SDK and OpenSL ES.

Finally, consider a native cross-platform game that goes to a whole bunch of platforms beyond Apple and Android written in Swift. SDL provides us all the functionality we need. SDL even provides audio, but for symmetry, I decided to include OpenAL for 3D Audio.

App Development vs. Server Development

I care about making shippable user facing apps that would be deployed on an app store, not command line server apps. As Mac and iOS developers, you know what I’m talking about: we must do extra things that other don’t. All our resources and dependencies must be deployed with the app. We can’t require users to install things. End users can’t be expect to compile apps from Source.

I bring this up because these are the official instructions for using Swift on Android. There are four steps here. The last two steps are completely wrong (see video) because they’re trying to build a Command line up and install it and run it on Android. You iOS developers should be able to imagine how pointless and useless this is. It masks a whole class of bugs because the environment and conditions this runs in is not like the normal Android environment, we’re going to do things the right way. But first, I must introduce Android development.

Android NDK (Native Development Kit)

All real Android apps must be written using the Android SDK, which is in Java. Android originally was Java only, but had to cave to game developers, they created the NDK, which lets developers create dynamic libraries in C and C++, but the NDK is the bare minimum they needed to do. You still must create a normal Android SDK app and start in Java. You must use Java’s loan library to load the native dynamic libraries and then use Java JNI to cross between languages. You might be asking how Swift fits in.

While the NDK was intended for C and C++, it is for all things related to native code and Swift generates native code. There is nothing new here; old concepts reapplied in a slightly different way.

Swift becomes our stand-in for C and C++. Unfortunately, the Android NDK is awful. Legendary game developer John Carmack of Doom and Quake fame called it “half-baked” and says “It really does suck.” People like to remind me that Carmack has rarely complained about developer environments, even ones that are notoriously famous for being hard, such as the Sega Saturn. This is damning, but this quote went viral because every NDK developer knows this pain. The NDK is a second-class citizen on Android with poor integration. The word on the street used to be that Google only has two full-time engineers working on the NDK.

Why am I getting on Google’s case? Android is eight years old. They’ve ignored our pleas, but reports go into the void. Public ridicule is the only tool we have left. This is one of the riches, most powerful companies in the world with complete market share dominance dominance in mobile. It is shameful, it is bad. I want to give you a taste of what it’s like you’re prepared.

Bionic (Android’s C standard library)

Android does not use glibc. They wrote their own standard C library for Android called Bionic. It doesn’t care about POSIX compliance. It doesn’t even care about ANSI compliance and eight years in Android, it is still terrible.

Lua is an embeddable scripting language. It is renowned for how portable and clean its code base is. Lua has been built on everything. It doesn’t resort to pound dip dash everywhere for different platforms. It is a singular code base written in pure standards compliant ANSI C. To demonstrate how awesome the code base is, somebody build modern day Lua in Boroland Turbo C 1.0 for MS-DOS from 1990 without modifying the source. It built and ran successfully.

If we try this with a modern Android NDK we get build failure. This is a post from a Bionic engineer. I’m not picking on this engineer. This poor soul is trying to fix this mess, but this gives different insights into why Android is awful.

Let me read an excerpt: “As you may be aware, Android’s C library, Bionic, is a hybrid of code from different sources, home grown, FreeBSD, NetBSD, and OpenBSd. NetBSD files aren’t from a single NetBSD release. hell, they’re not necessarily from any release. I found the bug that had been fixed in Upstream in like, 1996, but we had some random old version of that file.”

Terrible performance bug for strlcpy

In a prior project, I wrote a program to run a JavaScript performance to suite called Test262, which goes through 11,000-plus files. I use strlcpy in a hotspot. For a hot loop that should have taken 14 milliseconds, it took 9,000 milliseconds.

On the SDK side, I was trying to get my list of 11,000 files inside a directory in the APK. Little did I know, I walked right into a well-known Android performance bug. Pessimistically, this should take no more than a few seconds. Three hours later, I gave up and killed the process.

Speaking of files, files that ship with your app are inside the .ap. Think of it as a zip file. You can’t use the standard C library file functions like fopen and fread on anything in the .apk. Google makes you use a different set of functions. On top of that, they require a parameter that comes from a “God” object from the Java Android Activity or Context class. This means existing cross-platform libraries won’t work without special modification for Android. How many projects want to do this? And it’s ridiculous that this problem even exists.

Dynamic Library System wonky too

Android has wonky behavioral rules for dynamic libraries. I wish I had time to cover this in detail because things are affected by this. It is a common source of breakage and often leads to needing to specially modify cross-platform code.

Another example I’m going to call out is the soname. Soname is a meta data field and dynamic libraries for the Elf Executable and Linker format, which is used on Linux systems and much all Unix systems except Apple. There is a strong convention that put versioning information in the soname on Linux distributions combined with a series of symlinks. This will break on Android. You need to go one step deeper: C++.

Swift & C++ (and how it affects you)

Once upon a time, it was well known that you do not write libraries in C++, but I see a lack of library developers giving talks nowadays, and people seem to be forgetting these important lessons. I have to mention this because Swift uses C++. Swift itself is implemented in C++ and the standard libraries depend on the standard C++ library. Thus, the C++ fitpals are passed onto you. With the adnroid NDK, these problems are much more apparent.

Android NDK and C++

The NDK provides five difference C++ standard libraries you have to choose from. Choice is not a good thing. All are incompatible with each other. The C++ standard library does not guarantee a stable ABI, ever NDK upgrade potentially breaks things. If you use dynamic linking for your shared library of choice, you must bundle it with your app because Android does not ship a copy with the OS. This is in contrast to Apple, which ships you one to spare you this mess.

In the real world, people build library binaries and share them. But people don’t upgrade their NDKs all at the same time, and versions get mixed. People use multiple libraries and each library could be built with a different NDK version. You’d expect that the final application must include a copy of all these different C++ standard library versions that all your different libraries depend on. But Android doesn’t name versions differently, and files will override each other. Some of your libraries will start calling into the wrong version and bad things happen.

In contrast, Microsoft Visual Studio has the common sense to put version numbers in the file name to avoid this problem. we should statically link. This is the warning that Android has in their documentation about static linking. It’s loose. Thanks for nothing, Google. In practice, I personally found static linking to be the better of the two. Unfortunately, modifying this with build system is hard. I’ve successfully modified it in Swift 2 and submitted a patch, but the patch was never incorporated and somebody upstream re-based, the patch is broken now. It’s back to dynamic linking for now.

Let’s build Swift for Android: Dependency hell

We are application developers. The Android NDK provides almost no libraries we are responsible for building every library we need and shipping it with our app.

I showed these instructions earlier and said this mostly works. But you need to know about this gotcha. This is an Android variation of “DLL hell”. The main dependency for Swift Core that we need is libICU, the international components for unicode library, which is written in C++.

ICU is a popular enough library that Android manufacturers or Android itself may use internally. If used, when we try to load our ICU library with load library, the call silently does nothing because the operating system thinks it’s already loaded. If we use the exact same version of ICU with the same build options, things will work, but ICU is notorious for breaking compatibility every version. Throw in Android fragmentation, and it is guaranteed that somebody will be running with a different version.

Now that the versions don’t match, when your code uses ICU, bad things start to happen. If you’re lucky, you’ll get a crash. One way round this is statically link with ICU. But, if you must dynamically link, you should rename the libraries and the symbol names they do not conflict with the version in the operating system. ICU has switches to help mangle names differently because they are at least aware of the problems caused by breaking compatibility all of the time. You need to disable soname versioning because libIC’s build system is overly aggressive about setting it. good news.

Let’s use Swift on Android!

If we made it this far, we have a perfectly usable Swift for Android. Unfortunately, I need to skip foundation for time. It’s not part of the official build yet, but this slide has a few notes on its dependencies.

Earlier I said the official instructions were wrong. We’re going to do it the right way. Remember, all Android apps must use the Android SDK, which is in Java. We need to start in Java and work our way to Swift. We need to create a proper Android Java app and follow the standard techniques to cross into the NDK. We’re going to start in Java, cross into C, and then cross into Swift.

Android/Java

We start in Java. On Android, every Android app must have an activity to start in. An activity is like a window or a view controller. I made this class my starting activity. I’m using the on create method as my starting point. You can think of it as a knit and Cocoa.

public class MyLaunchActivity extends Activity {
    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        // Note: Order matters!
        System.loadLibrary("c++_shared"); // (if we didn't compile statically)

        // Notice that I renamed the ICU libraries & symbols
        System.loadLibrary(“icudataswift");
        System.loadLibrary("icuucswift");
        System.loadLibrary("icui18nswift");

        System.loadLibrary("swiftSwiftOnoneSupport"); // used by swiftCore
        System.loadLibrary("swiftCore");

        System.loadLibrary("swiftGlibc"); // Should be named Bionic

        // System.loadLibrary("dispatch");
        // System.loadLibrary(“Foundation"); // (may have additional dependencies)
        // This is our Swift code compiled into a dynamic library
        System.loadLibrary("MySwiftMainProgram");

Here we must load all dynamic libraries for Swift. Order matters, we load all the dependencies first, then Swift itself, and then our program, which we wrote in Swift.

package com.blurrrsdk.app;

public class MyLaunchActivity extends Activity {

    // Tells Java/JNI we promise to implement this function in native code
    // Activity is unused in this example, but useful as ‘God’ object
    public native void MyMainEntry(Activity activity);

    @Override
    protected void onStart() {
        super.onStart();
          // Call into native code
          MyMainEntry(this);
    }
}

Remember, we cannot build normal X cable files for Android, everything we write must be built into a dynamic library it can be called from Java. This is the same class as before, but I’ve hidden the on create method we can see this. On start is roughly the equivalent to awake from NIB or applications that finished launching. In here, we want to call a native function that will get us to C. The mechanism that makes this work is Java JNI. JNI is well-documented, it comes down to following a bunch of rules and boilerplate. In this example, I’ve declared that I promise to implement the function called MyMainEntry in C. The native keyword tells Java that this will be a native function.

The Java compiler is now trusting me to fulfill my promise of implementing it. I want to point out the package name declaration here because you will see how it is used in a second.

Now in C: call into swift

This is the implementation for that native function. Notice that the package name is part of the function name. This is one of those rules of JNI.

#include <jni.h>

// Promise C we've implemented this function.
extern void MyMain(void);

JNIEXPORT void JNICALL
Java_com_blurrrsdk_MyLaunchActivity_MyMainEntry(
    JNIEnv* jni_env, jobject thiz, jobject activity)
{

    // Call into Swift
    MyMain();
}

Now that we are in C, we need to call into Swift. I’m going to do a trick, but it’s a trick based on old fundamentals. Let’s call a function that looks like a C function but is a Swift function behind the scenes. We’ll call it MyMain().

Finally, we’re in Swift and here’s my Swift function. The trick is, the app’s _cdecl parameter. This tells the Swift compiler to make the function you see call in conventions it can be called like any other C function. Notice we declare the function as public. Make sure the symbol is accessible to be called across the dynamic library boundary. We transformed and reduced the problem so it looks like C. Again, there is nothing new here. We took our Swift ideas and transformed it to look like how we would solve it with C in the past. Now that we’re in Swift, we can start having fun and write things in Swift.

@_cdecl("MyMain")
public func MyMain()
{

    // Now the fun can begin!
    // <Your Swift code starts here ...>

    let four = 2 + 2
    // TODO/FIXME: print goes to /dev/null on Android
    print("2 + 2 = \(four)”)


    // Remember to not block the event loop
}

We must never block the event loop. Android is event-driven, like iOS. You will need to figure out how you want to use Swift in your app. You’ll likely create more callbacks like this to call Swift for different events. Alternatively, you could spin off the background thread. However, I strongly discourage this unless you know exactly what you’re in for. If you ever need to call system APIs from Swift, you’re in for a world of pain from doing it from a background thread. This is like trying to write a full-feature Cocoa app from a background thread: you are asking for trouble.

Caveats

There’s no objective C runtime on Android. Also, be extremely careful about the initialization of static and global variables. In Android, these variables may not get re-initialized on subsequent launches. In this example, I have a global variable called g_isInit. It is initialized to false. You can see when we first run this program, the top block in the with else gets run because our variable is false. We then set that variable to true. In Android, let’s assume you exit the program with the back button, which is an Android difference compared to iOS.

Let’s relaunch the program. Depending on whether Android purged all the NDK memory between runs or not, our goal variable may or may not get reinitialized. If Android let this memory alone, when you relaunch the program, the dynamic library you loaded is still in memory. Google made the decision to not reload and reinitialize things. In this case, even though this is a brand new start, our global variable is still set the true from the last run. If we take the bottom block of the with else, we will take the bottom block, which is probably not what we are expecting and wrong.

This problem scares me the most because it is subtle and I don’t know how far reaching the implications of this are yet. If you need this pattern, try to design it in a way you can explicitly reinitialize your variables manually at start.

Build Systems (Ugh…)

As somebody who has done cross-platform work, this tweet resonates with me “when you’re trying to ship on 3 platforms, you’re dealing with portability issues. But on 13 platforms, it’s all about build issues”, by Fabian Giesen.

Even at three different enough platforms, the build system differences are crippling. Take X Code, Android Studio, and Visual Studio as three of them. They’re completely alien to one another.

Build systems are the worst. While languages are standardized and cross-platform at least try to minimize the differences, build systems are completely different and the platform vendors have no interest in minimizing the cross-platform pain.

Build System Options

There are three basic techniques to build your projects for Android:

1. Swift Package Manager. Personally, I haven’t found this useful. The main problem is that things we need to do with Android Studio are a far cry from what Swift Package Manager does.
2. You can string together the cause to the Swift compiler to build everything yourself. Look at what XCode does and reproduce the commands. Obviously, this is a pain and it’s hard to scale, especially as you need to maintain additional platforms.
3. CMake.

CMake is well-used by cross-platform projects, even LLVM, Cland, and Swift are using CMake internally. Surprisingly, Android recent announced their support in CMake as part of Android Studio. CMake already has support for each platform’s native build process. For example, it can generate Esco projects. But since Swift is a new language, CMake doesn’t know anything about it. I started implementing support for it. My repo’s on Get Hub if you’d like to help. It’s a work in progress, but there is enough to make shippable applications.

cmake_minimum_required(VERSION 3.4)
project(MySwiftProject C Swift)
if(ANDROID)
    # Must build library on Android
    add_library( SwiftApp CCode.c SwiftCode.swift)
else()
    add_executable(SwiftApp CCode.c SwiftCode.swift main.swift)
endif()

set_property(TARGET SwiftApp PROPERTY SWIFT_BRIDGING_HEADER "MyBridgingHeader.h")

This is a simple CMake script showing how to make a library or executable for Swift code. CMake script isn’t going to win any beauty awards, but the CMake project generation capabilities are unparalleled.

Let’s talk about Libraries for writing Android apps

There are three major categories:

1. Native Android-only development in Swift
2. Cross-platform development (non-native GUI)
3. Cross-platform native GUI

Native Android in Swift

Can we write native Android-only apps in Swift? Yes, but it isn’t pleasant. We must go through JNI. Furthermore, not everything is possible through JNI. For example, you cannot subclass through JNI, you’ll still need the right Java for those cases, unless you resort by code hacks or code generation.

But again, there’s nothing new. JNI has been around for a long time. We know how to deal with it, and the world has come up with all sorts of solutions. But they all come down to the same basic ideas. You can write higher level libraries to encapsulate and hide all the JNI so when you use the libraries, you don’t have to touch JNI. Or, you can develop co-generation tools. While I’m not a huge fan of co-generation because it often introduces new problems (e.g. complex build systems, long build times, and application bloat), you can see high-profile companies have resorted to this on Android.

Cross-platform (C) libraries

Moving beyond pure native Android, let’s look at cross-platform libraries. Again, there’s nothing new. This has been solved many times over. Developers have been doing cross-platform for decades, especially by the video game industry. There’s a C library for everything. And, if there’s a C library, we can use it with Swift.

Here’s a list of the some of the usual suspects: SDL, OpenGL, OpenAL, FreType does fonts, libpng and libjpeg for your image needs, cURL for networking. Chipmunk even provides a physics engine in PureC.

FlappyBlurrr Swift

There’s already more than enough to make full apps in Swift on Android and any other platform you can think of. This is a FlappyBlurrr clone I wrote in Swift using SDL and company.

I’m a FlappyBlurrr perfectionist and most clones are terrible, I focused to get the timing details right (even though I don’t have time to prove it to you). And because our libraries already work cross-platform to everywhere, this program can work everywhere. This even runs hat a smooth 60 frames per second on a Raspberry Pi.

What about (non-native) GUI?

Let’s start easy with non-native GUIs, ones that display the exact same thing no matter what platform you’re on. My favorite is Nuklear. It is pure C, it can be used easily with Swift, and it has zero dependencies, it is easy to drop into a project. It is designed to be easy to adapt to any drawing toolkit. OpenGL is the most common, but it can be adapted to anything.

Here’s a screenshot, and a nice view of it running on Android (see video). This is a particle designer I wrote for the desktop. One aspect I wanted to not is that these non-native GUIs scale linearly with your screen size. I intentionally made the widget small I could fit more, my design is not ideal on a phone. But, it does work and it is surprisingly usable on an iPad or Android tablet. I’m showing this to you to get you to think about your designs if you need to support a wide range of screens, like TVs to phones.

What about cross-platform native GUI?

But I know most of you are native developers. We know how much the native experience can matter to users, and how much better it can be. Is there a cross-platform solution to native GUI?

I know everyone here wishes they could take their Cocoa apps and recompile them everywhere else, but this is a hard problem. If you look at history, there is a long list of serious attempts. Every one of these has failed to achieve the dream. If we look beyond Cocoa, we see that there are very few actual native toolkits out there, and the ones that we can use with Swift are even fewer. But there is one I like, called IUP.

IUP (Portable User Interface), A Hidden Gem

IUP originated as a research project from the same university in Brazil that created the Lua language, but it has since become a production-quality library used in certain circles like the oil industry and scientific visualization.

It uses true native widgets. It is small and focused only on GUI. Most other libraries have become a massive kitchen sink, reinventing everything and has tons of bloat. For us, this is a perfect fit because foundation + IUP has no overlap, like foundation + UIkit.

It is written in C, but it is written with language bindings in mind, we can easily use IUP in Swift. It also comes with a textual layout description language called LED, separating code from data, and can be used for per-platform customized layouts.

IUP’s research roots were focused on how to deal with the wide variations between different platforms not only in the widget differences, but on how not all platforms use object-oriented languages. It is very well thought out. Much of the API solution centers around a key value attribute system. Think NS user defaults. This allows API access to native features that may not exist on other all-platforms without constantly breaking the API design, nor preventing you from accessing platform-specific features if you want them.

Squint hard and see…

While some people may not be able to look past the stringy API, this is protocol-oriented design. At least this best can be done within C’s weak type system. I think there are amazing possibilities for a Swift wrapper for IUP. There’s already been talk in the Swift community about what if we could have a GUI API designed around protocol-oriented design.

Here’s a hidden gem that needs polish. And the great news is that the Swift community has already developed many techniques to build nice wrappers. For example, last year’s Try a Swift talk, Swift Eye for the Stringly Typed API by Andy Hope.

IUP drawbacks

But every solution has its drawbacks. For IUP, it has native windows, GTK2, GTK3, and Motif, but no Cocoa. But we can fix this.

This is a screenshot from a Rio program I wrote (see video). The screenshots are from Windows and Linux. This is a work in progress, but here it is, running on Mac.

What about mobile? We can fix that, too. IUP is well-designed. I believe it can do mobile, too. You can look up a YouTube presentation I made on this topic. As a simple example, most other libraries made the mistake of modeling their APIs off of existing desktop APIs and everything is shoehorned into that, and we get things like a Window API. In contrast, IUP created a more ambiguous dialogue API. On the desktop, this may map to an actual Window API. On mobile, we get to decide what makes sense. For iOS, we map dialogue to use UI navigation controller and view controllers, and on Android, we map it to activities.

Demo Program: Create Dialog & Button (recursive)

var g_buttonCounter = 0;

func BlurrrMain() -> Int32 {
    IupOpen(nil, nil)
    IupSetFunction("ENTRY_POINT", IupEntryPoint);
    IupMainLoop()
    return 0;
}
func IupEntryPoint(_ ih:OpaquePointer?) -> Int32 {
    g_buttonCounter = 0
    return ShowNewDialogCallback(nil)
}

func ShowNewDialogCallback(_ ih:OpaquePointer?) -> Int32 {
let button = IupButton(nil, nil)
IupSetStrAttribute(button, "TITLE", "Iup Button \(g_buttonCounter)")
IupSetCallback(button, "ACTION", ShowNewDialogCallback)
let dialog = IupDialog(button)
IupSetAttribute(dialog, "SIZE", "QUARTERxQUARTER")
IupSetStrAttribute(dialog, "TITLE", "Iup Dialog \(g_buttonCounter)")
IupShow(dialog)
g_buttonCounter += 1
return IUP_DEFAULT
}

You don’t need to understand this code, but I want you to know that this is a complete IUP program written in Swift. For clarity, I did not write any high-level Swift wrappers, you are seeing everything. This program creates a dialogue containing a button. When you press the button, it creates a new dialogue with a button and forth.

See the video to see the program running on Ubuntu Linux, Raspberry Pi, Mac, iOS. We are mapping dialogue to use UI Navigation Controller and UI View Controllers under the hood. And finally, Android. Here we map dialogue to activity. And… Surprise! Swift on Windows.

Re-cap of what we just saw

You saw native GUI cross-platform desktop and mobile with a single code base in Swift. If you’d like to help out, my research are in GitHub.

IUP development links

Blurrr SDK (My current endeavor)

There is no magic in cross-platform, but there is tedious work involved. I’d like to make cross-platform native development more accessible to more people. I created Blurrr SDK. I want you to leverage my experience in dealing with all the annoying platforms specific to the differences you can focus on your core program. I use Blurrr to help me create all the examples in this presentation.

Blurrr handles the build system challenges using CMake under the hood. Blurrr provides a bunch of deployment-ready pre-built libraries so you can build your app and ship it. I tried to provide a download and go experience for Blurrr itself.

This is the Blurrr workflow (see video). Start it up, open your project, then you get the appropriate native project for the platform you’re on. Then you work on your code, build and run. Currently, I include libraries, like SDL. Blurrr is currently best for games and multimedia apps.

I am very serious about IUP. It is in development but quite capable. I’m still nervous, but I want people to use this, I’m opening up the beta today you can all start playing with it. I’m still trying to get off the ground, I could use your support. I will be doing a game workshop at the Hackathon.

Carlos M. Icaza, June 5, 1966 - May 17, 2016

I want to switch gears and say a few words about my friend, mentor, and former co-founder, Carlos Icaza. He passed away unexpectedly this summer. While he was known by many different groups for different accomplishments, the Swift community knows him as @codinginswift, with 18,000 followers. He held Swift meetups. This one (see video) was in Silicon Valley; he even managed to somehow get the elusive Chris Lattner to show up that night.

So…

I wanted to try to write something in his memory that would have a special significance for events we shared together. In some sense, this is a bucket list of things we wanted to achieve together, but time ran out. For example, he was an expert at splines. We wanted to do a spline project together, but it could never get a high enough priority.

I wanted to leave you with something that inspires you with how much is already possible with cross-platform Swift today.

Q & A

Q: How that is Apple in these respects, in the same fragmentation and not following standards? Eric: It’s good. Our rules from generally, and all the companies I’ve worked with, Android is about four to 10 times harder.

Q: How about the memory management difference between Android platform and Swift? Eric: If you’re on Swift, you’re using the NDK side of the memory, not the SDK side unless you’re calling in to native widgets. The funny thing about Android memory management is that the Java side is completely managed and Google has done stupid things and they have imposed artificial memory elements for your Java apps. A long time ago, the memory limits were artificially, insanely low and nobody could write real apps with it, the NDK people figured out if you do stuff on the NDK side, you can use all the memory that’s on the device and those limits don’t apply to you. And because Android is bloated and general, they ship with more RAM than their iOS device. you actually, generally, get more memory with Android than you do with iOS.

What with the arrival of the category-defining Commodore VIC-20 and the dramatic growth of the British PC market, 1981 has provided us with no shortage of new machines and other technical developments to talk about. Yet I’ve saved the biggest event of all for last: the introduction of the IBM PC, the debut of an architecture that is still with us over 30 years later. As such a pivotal event in the history of computing, there’s been plenty written about it already, and no small amount of folklore of dubious veracity has also clustered around it. Still, it’s not something we can ignore here, for the introduction of the IBM PC in late 1981 marks the end of the first era of PCs as consumer products as surely as the arrival of the trinity of 1977 spelled the end of the Altair era of home-built systems. So, I’ll tell the tale here again. Along the way, I’ll try to knock down some pervasive myths.

One could claim that the IBM PC was not really IBM’s first PC at all. In September of 1975 the company introduced the IBM 5100, their first “portable” computer. (“Portable” meant that it weighed just 55 pounds and you could buy a special travel case to lug it around in.)

The 5100 was not technically a microcomputer; it used a processor IBM had developed in-house called the PALM which was spread over an entire circuit board rather than being housed in a single microchip. From the end user’s standpoint, however, that made little difference; certainly it would seem to qualify as a personal computer if not a microcomputer. It was a self-contained, Turing complete, programmable machine no larger than a suitcase, with a tape drive for loading and saving programs, a keyboard, and a 5-inch screen all built right in along with 16 K or more of RAM. What made the 5100 feel different from the first wave of PCs were its price and its promoted purpose. The former started at around $10,000 and could quickly climb to the $20,000 range. As for the latter: IBM pushed the machine as a serious tool for field engineers and the like in remote locations where they couldn’t access IBM’s big machines, not as anything for fun, education, hacking, or even office work. The last of these at least changed with two later iterations of the concept, the 5110 and 5120, which were advertised as systems suitable for the office, with accounting, database, and even word processing applications available. Still, the prices remained very high, and actually outfitting one for this sort of office work would entail connecting it to a free-standing disk array that was larger than the machine itself, making the system look and feel more like a minicomputer and less like a PC. It’s nevertheless telling that, although it was almost never referred to by this name, the IBM PC when it finally arrived had the official designation of (with apologies to Van Halen) the IBM 5150, a continuation of the 5100 line of portable computers rather than an entirely new thing — this even though it shared none of the architecture of its older siblings.

In February of 1978 IBM began working on its first microcomputer — and it still wasn’t the IBM PC. It was a machine called the System/23 Datamaster.

Designed once again for an office environment, the Datamaster was built around an Intel 8085 microprocessor. It was large and heavy (95 pounds), and still cost in the $10,000 range, which combined with its very business-oriented, buttoned-down personality continued to make it feel qualitatively different from machines like the Apple II. Yet it was technically a microcomputer. IBM was a huge company with a legendarily labyrinthine bureaucracy, meaning that projects could sometimes take an inordinately long time to complete. Despite the Datamaster project predating the PC project by two years, the former didn’t actually come out until July of 1981, just in time to have its thunder stolen by the announcement of the IBM PC the following month. Still, if the question of IBM’s first microcomputer ever comes up in a trivia game, there’s your answer.

The machine that would become known as the real IBM PC begins, of all places, at Atari. Apparently feeling their oats in the wake of the Atari VCS’s sudden Space Invaders-driven explosion in popularity and the release of the their own first PCs, the Atari 400 and 800, they made a proposal to IBM’s chairman Frank Cary in July of 1980: if IBM wished to have a PC of their own, Atari would deign to build it for them. Far from being the hidebound mainframer that’s he often portrayed as, Cary was actually something of a champion of small systems — even if “small systems” in the context of IBM often meant something quite different from what it meant to the outside world. Cary turned the proposal over to IBM’s Director of Entry Systems, Bill Lowe, based out of Boca Raton, Florida. Lowe in turn took it to IBM’s management committee, who pronounced it “the dumbest thing we’ve ever heard of.” (Indeed, IBM and Atari make about the oddest couple imaginable.) But at the same time, everyone knew that Lowe was acting at the personal behest of the chairman, not something to be dismissed lightly if they cared at all about their careers. So they told Lowe to assemble a team to put together a detailed proposal for how IBM could build a PC themselves — and to please come back with it in just one month.

Lowe assembled a team of twelve or thirteen (sources vary) to draft the proposal. In defiance of all IBM tradition, he deliberately kept the team small, the management structure informal, hoping to capture some of the hacker magic that had spawned PCs in the first place. His day-to-day project manager, Don Estridge, said, “If you’re competing against people who started in a garage, you have to start in a garage.” One might have expected IBM, the Goliath of the computer industry, to bludgeon their way into the PC market. Indeed, and even as they congratulated themselves for having built this new market using daring, creativity, and flexibility stolid IBM could not hope to match, many PC players lived in a sort of unvoiced dread of exactly this development. IBM, however, effectively decided to be a good citizen, to look at what was already out there and talk to those who had built the PC market to find out what was needed, where a theoretical IBM PC might fit. In that spirit, Jack Sams, head of software development, recommended that they talk to Microsoft. Sams was unusually aware of the PC world for an IBMer; he had actually strongly pressed for IBM to buy the BASIC for the Datamaster from Microsoft, but had been overruled in favor of an in-house effort. “It just took longer and cost us more,” he later said. Sams called Bill Gates on July 21, 1980, asking if he (Sams) could drop by their Seattle office the next day for a friendly chat about PCs. “Don’t get too excited, and don’t think anything big is about to happen,” he said.

Gates and Steve Ballmer, his right-hand man and the only one in this company of hackers with a business education, nevertheless both realized that this could be very big indeed. When Sams arrived with two corporate types in tow to function largely as “witnesses,” Gates came out personally to meet them. (Sams initially assumed that Gates, who still had the face, physique, and voice of a twelve-year-old, was the office boy.) Sams immediately whipped out the non-disclosure agreement that was standard operating procedure for IBM. Gates: “IBM didn’t make it easy. You had to sign all these funny agreements that sort of said IBM could do whatever they wanted, whenever they wanted, and use your secrets however they felt. So it took a little bit of faith.” Nevertheless, he signed it immediately. Sams wanted to get a general sense of the PC market from Gates, a man who was as intimately familiar with it as anyone. In this respect, Gates was merely one of a number of prominent figures he spoke with. However, he also had an ulterior motive: to see just what kind of shop Gates was running, to try to get a sense of whether Microsoft might be a resource his team could use. He was very impressed.

After consulting with Gates and others, Lowe presented a proposal for the machine that IBM should build on August 8. Many popular histories, such as the old PBS Triumph of the Nerds, give the impression that the IBM PC was just sort of slapped together in a mad rush. Actually, a lot of thought went into the design. There were two very interesting aspects.

At that time, almost all PCs used one of two CPUs: the MOS 6502 or the Zilog Z80. Each was the product of a relatively small, upstart company, and each “borrowed” its basic instruction set and much of its design from another, more expensive CPU produced by a larger company — the Motorola 6800 and the Intel 8080 respectively. (To add to the ethical questions, both were largely designed by engineers who had also been involved with the creation of their “inspirations.”) Of more immediate import, both were 8-bit chips capable of addressing only 64 K of memory. This was already becoming a problem. The Apple II, for example, was limited, due to the need to also address 16 K of ROM, to 48 K of RAM at this time. We’ve already seen the hoops that forced Apple and the UCSD team to run through to get UCSD Pascal running on the machine. Even where these CPUs’ limitation weren’t yet a problem, it was clear they were going to be soon. The team therefore decided to go with a next-generation CPU that would make such constraints a thing of the past. IBM had a long history of working with Intel, and so it chose the Intel 8088, a hybrid 8-bit / 16-bit design that could be clocked at up to 5 MHz (far faster than the 6502 or Z80) and, best of all, could address a full 1 MB of memory. The IBM PC would have room to grow that its predecessors lacked.

The other interesting aspect was this much-vaunted idea of an “open architecture.” In Accidental Empires and even more so in Triumph of the Nerds Robert X. Cringely makes it out to be a choice born of necessity, just another symptom of the machine as a whole’s slapdash origins: “An IBM product in a year! Ridiculous! To save time, instead of building a computer from scratch, they would buy components off the shelf and assemble them — what in IBM speak was called ‘open architecture.'” Well, for starters “open architecture” is hardly “IBM speak”; it’s a term used to describe the IBM PC almost everywhere — and probably least of all within IBM. (In his meticulous, technically detailed Byte magazine article “The Creation of the IBM PC,” for example, team-member David J. Bradley doesn’t use it once.) But what do people mean when they talk about “open architecture?” Unfortunately for flip technology journalists, the “openness” or “closedness” of an architecture is not an either/or proposition, but rather, like so much else in life, a continuum. The Apple II, for example, was also a relatively open system in having all those slots Steve Wozniak had battled so hard for (just about the only battle the poor fellow ever won over Steve Jobs), slots which let people take the machine to places its creators had never anticipated and which bear a big part of the responsibility for its remarkable longevity. Like IBM, Apple also published detailed schematics for the Apple II to enable people to take the machine places they never anticipated. The CP/M machines that were very common in business were even more open, being based on a common, well-documented design specification, the S-100 bus, and having plenty of slots themselves. This let them share both hardware and software.

Rather than talking of an open architecture, we might do better to talk of a modular architecture. The IBM would be a sort of computer erector set, a set of interchangeable components that the purchaser could snap together in whatever combination suited her needs and her pocketbook. Right from launch she could choose between a color video card that could do some graphics and play games, or a monochrome card that could display 80 columns of text. She could choose anywhere from 16 K to 256 K of onboard memory; choose one or two floppy drives, or just a cassette drive; etc. Eventually, as third-party companies got into the game and IBM expanded its product line, she would be all but drowned in choices. Most of the individual components were indeed sourced from other companies, and this greatly sped development. Yet using proven, well-understood components has other advantages too, advantages from which would derive the IBM PC’s reputation for stolid reliability.

While sourcing so much equipment from outside vendors was a major departure for IBM, in other ways the IBM PC was a continuation of the company’s normal design philosophy. There was no single, one-size-fits-all IBM mainframe. When you called to say you were interested in buying one of these monsters, IBM sent a rep or two out to your business to discuss your needs, your finances, and your available space with you. Then together you designed the system that would best suit, deciding how much disk storage, how much memory, how many and what kind of tape drives, what printers and terminals and punched-card readers, etc. In this light, the IBM PC was just a continuation of business as usual in miniature. Most other PCs of course offered some of this flexibility. It is nevertheless significant that IBM decided to go all-in for modularity, expandability, or, if we must, openness. Like the CPU choice, it gave the machine room to grow, as hard drives, better video cards, eventually sound cards became available. It’s the key reason that the architecture designed all those years ago remains with us today — in much modified form, of course.

The committee gave Lowe the go-ahead to build the computer. IBM, recognizing itself that its bureaucracy was an impediment to anyone really, you know, getting anything done, had recently come up with a concept it called the Independent Business Unit. The idea was that an IBU would work as a semi-independent entity, freed from the normal bureaucracy, with IBM acting essentially as the venture capitalists. Fortune magazine called the IBU, “How to start your own company without leaving IBM.” Chairman Cary, in a quote that has often been garbled and misattributed, called the IBU IBM’s answer to the question, “How do you make an elephant [IBM] tap dance?” Lowe’s IBU would be code-named Project Chess, and the machine they would create would be code-named the Acorn. (Apparently no one was aware of the British computer company of the same name.) They were given essentially free rein, with one stipulation: the Acorn must be ready to go in just one year.

On average, foregoing college in favor of entering the workforce + investing tuition costs nets higher returns

College is obviously expensive, but is it still a wise investment?

We've all heard how expensive college is getting, along with plenty of criticism surrounding its value in a changing job market. Of course, there are many benefits beyond the monetary ones that should be considered when exploring college options, but for the purpose of this post I'm going to limit the scope and purely assess the financial benefit of attending college.

The main financial benefit of attending college is the earnings differential received by a college graduate over a high school graduate; Payscale provides 20-year return on investment (ROI) figures for exactly that. The website compares the gain in median pay from graduating over a high school graduate across ~1,250 4 and 5-year educational institutions in the United States.

Below I'll compare the ROI of college to the return generated from simply joining the workforce after high school, but investing college tuition costs* into the stock market (using S&P 500 as a proxy).

Option A - Skip college, enter workforce, and invest your saved tuition costs

Lets say we skip college and dump our saved money from avoided tuition costs into the stock market. The S&P 500 (our stock market proxy) has risen from ~435 at the beginning of 1993 to ~2,260 at the beginning of 2017. This represents a 520% return, which equates to an annualized return of ~7.1% for 24 years. Note that we get 24 years for our investment return, rather than the 20 years college gets, since we can make our investment upon high school graduation and immediately realize returns.

Option B - Attend college, pay tuition, generate higher levels of income if you graduate

Here, we'll attend college as usual, and realize the average ROI across U.S. colleges. Using our Payscale ROI data mentioned above, and normalizing for graduation rates (ROI*graduation rate = Expected ROI), the average annualized ROI with financial aid is ~4.5% (median 4.1%).

The winner?

As shown, on average, forgoing college costs, investing that money in equities, and beginning work would generate significantly higher returns than going to college. Below you can see the distribution of ROI across U.S. colleges, along with our 'Option A' comparison point.

A couple of additional stats:

• Only ~10% of the ~1250 colleges listed on Payscale generate an average ROI higher than the 7.1% generated by foregoing college and investing that money. In other words, a student planning on enrolling in college would generate higher returns entering the labor force and investing tuition into the stock market 90% of the time.
• If we remove financial aid, only ~2.5% of colleges generate a better ROI than our 'skip college invest in stock market' option!

*Footnotes/disclaimers:

• Of course, lenders won't give out large loans for an individual to skip college and dump tuition into the stock market, this is just a hypothetical scenario.
• One could incorporate undergraduate population size of each school to volume-weight the ROIs provided by Payscale, for an improved average across the pool. (I did not feel like scraping undergrad population size for all schools listed).
• Returns vary significantly by school and major. If you're trying to decide whether or not to attend college from a financial point of view, I'd recommend being thoughtful about your school choice, major, potential financial aid, and ability to graduate enrolled with a degree from said school in said major. As mentioned above, there are also many additional (non-financial) factors to consider.
• I went to college, and would recommend it.

In 2014, a heartwarming letter sent to year 6 pupils at Barrowford primary school in Lancashire went viral. Handed out with their Key Stage 2 exam results, it reassured them: “These tests do not always assess all of what it is that make each of you special and unique… They do not know that your friends count on you to be there for them or that your laughter can brighten the dreariest day. They do not know that you write poetry or songs, play sports, wonder about the future, or that sometimes you take care of your little brother or sister.”

At Barrowford, people learned, teachers were discouraged from issuing punishments, defining a child as “naughty” and raising their voices. The school’s guiding philosophy, said headteacher Rachel Tomlinson, was that kids were to be treated with “unconditional positive regard”.

A little more than a year later, Barrowford found itself in the news again. Ofsted had given the school one of its lowest possible ratings, finding the quality of teaching and exam results inadequate. The school, their report said, “emphasised developing pupils’ emotional and social wellbeing more than the attainment of high standards”. Somehow, it seemed, the nurturing of self-esteem had not translated into higher achievement.

The flawed yet infectious notion that, in order to thrive, people need to be treated with unconditional positivity first gained traction in the late 80s. Since then, the self-esteem movement has helped transform the way we raise our children – prioritising their feelings of self-worth, telling them they are special and amazing, and cocooning them from everyday consequences.

One manifestation of this has been grade inflation. In 2012, the chief executive of British exams regulator Ofqual admitted the value of GCSEs and A-levels had been eroded by years of “persistent grade inflation”. In the US, between the late 60s and 2004, the proportion of first year university students claiming an A average in high school rose from 18% to 48%, despite the fact that SAT scores had actually fallen. None of this, says Keith Campbell, professor of psychology at the University of Georgia and expert on narcissism, serves our youngsters well. “Burning yourself on a stove is really useful in telling you where you stand,” he says, “but we live in a world of trophies for everyone. Fourteenth place ribbon. I am not making this stuff up. My daughter got one.”

Campbell, with his colleague Jean Twenge at San Diego State University, has argued that this kind of parenting and teaching has contributed to a measurable rise in narcissism: witness the selfie-snapping millennials. Although their findings are disputed, Twenge points to other research done in the US and beyond – “twenty-two studies or samples [that] show a generational increase in positive self-views, including narcissism, and only two [that] do not”.

How did we get here? To answer that, you have to go back to 1986 and the work of an eccentric and powerful California politician, John “Vasco” Vasconcellos. That year, the Democrat Vasconcellos managed to persuade a deeply sceptical Republican state governor to fund a three-year task force to explore the value of self-esteem. Vasco was convinced that low self-esteem was the source of a huge array of social issues, including unemployment, educational failure, child abuse, domestic violence, homelessness and gang warfare. He became convinced that raising the population’s self-esteem would act as a “social vaccine”, saving the state billions.

But Vasco’s plan backfired spectacularly, with the fallout lasting to this day. I spent a year trying to find out why – and discovered that there was, at the heart of his project, a lie.

***

John Vasconcellos grew up an obedient Catholic, an altar boy, the smartest kid in his class, whose mother swore that he never misbehaved. But, being such a devout Catholic, he knew that no matter how good he was, he could only ever be a sinner. At primary school, he ran for class president. “I lost by one vote. Mine,” he later said. He didn’t vote for himself because “I’d been drilled never to use the word ‘I’, never to think or speak well of myself.”

After a spell as a lawyer, Vasco entered politics. In 1966, aged 33, he was elected to the California state assembly. But there was a problem: his professional success was at odds with how he thought of himself; he felt he didn’t deserve it. At 6ft 3in and over 200lb, he would stalk the Capitol building in Sacramento, glowering and anxious in his smart black suit, perfect white shirt and arrow-straight tie, his hair cropped with military precision. “I found my identity and my life coming utterly apart,” he later said. “I had to go and seek help.”

That help came from an unusual Catholic priest: Father Leo Rock was a psychologist who had trained under the pioneer of humanistic psychology, Carl Rogers, a man who believed that the Catholics had it absolutely wrong. At their core, he thought, humans weren’t bad; they were good. And in order to thrive, people needed to be treated with “unconditional positive regard” (Rogers coined the phrase). Vasco began studying under Rogers himself, a man he later described as “almost my second father”. Through intense group therapy workshops at the Esalen Institute in Big Sur, Vasco became a devotee of the human potential movement, based partly on the Rogerian idea that all you need to do to live well is discover your authentic inner self.

Around the state capitol, Vasco’s colleagues began to notice the buttoned-up Catholic was unbuttoning. He grew his hair and wore half-open Hawaiian shirts on the floor of the senate, a gold chain nestled in his chest hair. One reporter described him as looking like “a cross between a rock star and a drug smuggler”. He became a human potential evangelist, preaching the innate goodness of humans and handing long book lists to colleagues. His self-hating Catholic self had washed away, and in its place was a great, glowing letter “I”.

Vasco knew he was in a unique position. As a politician, he could take everything he’d learned about human potential and turn it into policy that would have a real effect on thousands, perhaps millions, of lives. He decided to campaign for a state-financed task force to promote self-esteem: this would give the movement official affirmation and allow politicians to fashion legislation around it. Best of all, they could recruit the world’s finest researchers to prove, scientifically, that it worked.

In the mid-80s, the notion that feeling good about yourself was the answer to all your problems sounded to many like a silly Californian fad. But it was also a period when Thatcher and Reagan were busily redesigning western society around their project of neoliberalism. By breaking the unions, slashing protections for workers and deregulating banking and business, they wanted to turn as much of human life as possible into a competition of self versus self. To get along and get ahead in this new competitive age, you had to be ambitious, ruthless, relentless. You had to believe in yourself. What Vasco was offering was a simple hack that would make you a more winning contestant.

Vasco’s first attempt at having his task force mandated into law came to a halt in 1984, when he suffered a heart attack. His belief in positive thinking was such that, in an attempt to cure himself, he wrote to his constituents asking them to picture themselves with tiny brushes swimming through his arteries, scrubbing at the cholesterol, while singing, to the tune of Row, Row, Row Your Boat: “Now let’s swim ourselves/ up and down my streams/Touch and rub and warm and melt/the plaque that blocks my streams.” It didn’t work. As the senate voted on his proposal, Vasco was recovering from seven-way coronary bypass surgery.

After a second attempt was vetoed by the state governor, Vasco decided to enhance the name of his project, upgrading it to the Task Force to Promote Self-Esteem and Personal and Social Responsibility. He reduced the proposed budget from $750,000 a year to $735,000 over three, to be spent on academic research and the gathering of evidence in the form of public testimony. On 23 September 1986, Assembly Bill 3659 was signed into law.

The response from the California media was immediate and savage. One editorial, in the San Francisco Chronicle, called Vasco’s task force “naive and absurd”. Nothing made Vasco more angry than his ideas not being taken seriously, but he was about to become the joke of America.

***

Until Monday 9 February 1987, Vasco’s task force had been largely state news. But on that morning, the cartoonist Garry Trudeau, who had been tickled by the politician’s crusade, began an extraordinary two-week run of his popular Doonesbury strip devoted to it. By the end of that day, reporters were crowding Vasco on the floor of the assembly chamber. Rival politicians gave dismissive briefings – “You could buy the Bible for $2.50 and do better” – while the Wall Street Journal’s story bore the headline Maybe Folks Would Feel Better If They Got To Split The $735,000.

Vasco was livid. The media, he complained, were “terrible, cynical, sceptical and cheap”. Their problem? “Low self-esteem.”

Meanwhile, something remarkable seemed to be happening. The response from the people of California had been great. Between its announcement and the task force’s first public meeting in March 1987, the office received more than 2,000 calls and letters, and almost 400 applications to volunteer. More than 300 people came forward to speak in support of self-esteem at public hearings across the state. And even if the media’s tone wasn’t always respectful, Vasco himself was now a national figure. He appeared everywhere from Newsweek to the CBS Morning Show to the BBC. This, he sensed, could be a major opportunity.

But first he needed to find a way to wrench the media conversation upwards. And things, on that front, were going from unfortunate to ridiculous. It began with the announcement of the task force’s 25 members. On the upside, it was a diverse group, including women, men, people of colour, gay people, straight people, Republicans, Democrats, a former police officer and Vietnam veteran who’d been awarded two Purple Hearts. On the downside, it also included a white man in a turban who predicted the work of the task force would be so powerful, it would cause the sun to rise in the west. A delighted Los Angeles Herald told how, in front of the press, one member of the task force had asked others to close their eyes and imagine a “self-esteem maintenance kit” of magic hats, wands and amulets.

Vasco’s team began hearing testimony from people up and down California. They heard from an LA deputy sheriff who toured schools, attempting to reduce drug use by telling pupils, “You are special. You are a wonderful individual.” They heard from masked members of the Crips, who blamed their violent criminality on low self-esteem. One school principal recommended having elementary pupils increase their self-importance by doing evaluations on their teachers. A woman called Helice Bridges explained how she’d dedicated her life to distributing hundreds of thousands of blue ribbons that read Who I Am Makes A Difference.

With the national media given so much to snigger over, it was beginning to look as if Vasco’s mission was a bust. But there had been some good news: the University of California had agreed to recruit seven professors to research the links between low self-esteem and societal ills. They would report back in two years’ time. For Vasco, their findings would be personal. If the professors decided he was wrong, it was all over.

***

At 7.30pm on 8 September 1988, Vasco met the scientists at El Rancho Inn in Millbrae, just outside San Francisco, to hear the results. Everything hinged on Dr Neil Smelser, an emeritus professor of sociology who had coordinated the work, leading a team who reviewed all the existing research on self-esteem. And the news was good: four months later, in January, the task force issued a newsletter: “In the words of Smelser, ‘The correlational findings are very positive and compelling.’”

The headlines quickly piled up: Self-Esteem Panel Finally Being Taken Seriously; Commission On Self-Esteem Finally Getting Some Respect. The state governor sent the professors’ research to his fellow governors, saying, “I’m convinced that these studies lay the foundation for a new day in American problem solving.”

Vasco’s task force was almost done: all they had to do now was build upon this positive tone with the publication of their final report, Toward A State Of Esteem, in January 1990. That report turned out to be a victory beyond the reasonable hopes of anyone who had witnessed its humiliating origins. The governor of Arkansas, Bill Clinton, who’d privately mocked Vasco and his project, now publicly endorsed it, as did figures including Barbara Bush and Colin Powell. Time magazine ran with the headline, “The sneers are turning to cheers.”

The man they were calling the Johnny Appleseed of Self-Esteem appeared on the Today Show and Nightline, on the BBC and Australia’s ABC. The report went into reprint in its debut week and went on to sell an extraordinary 60,000 copies. Vasco’s publicists approached Oprah Winfrey, who ran a prime-time special examining why she believed self-esteem was going to be one of the “catch-all phrases for the 1990s”. Interviewed were Maya Angelou, Drew Barrymore and John Vasconcellos.

Four months after the launch of Toward A State Of Esteem, the papers were reporting that self-esteem was “sweeping through California’s public schools”, with 86% of the state’s elementary school districts and 83% of high school districts implementing self-esteem programmes. In Sacramento, students began meeting twice a week to decide how to discipline other students; in Simi Valley, kids were taught, “It doesn’t matter what you do, but who you are.” Political leaders from Arkansas to Hawaii to Mississippi began considering their own task forces.

As the months became years, the self-love movement spread. Defendants in drug trials were rewarded with special key chains for appearing in court, while those who completed treatment were given applause and doughnuts. Children were awarded sports trophies just for turning up; a Massachusetts school district ordered children in gym classes to skip without actual ropes lest they suffer the self-esteem catastrophe of tripping. Meanwhile, police in Michigan seeking a serial rapist instructed the public to look out for a thirtysomething male with medium build and “low self-esteem”.

The credibility of Vasco’s task force turned largely on a single fact: that, in 1988, the esteemed professors of the University of California had analysed the data and confirmed his hunch. The only problem was, they hadn’t. When I tracked down one renegade task force member, he described what happened as “a fucking lie”. And Vasco was behind it.

***

In an attempt to discover how America, and then the world, got conned so spectacularly, I travelled to Del Mar, California, to meet the task force member who’d predicted their work would cause the sun to rise in the west. David Shannahoff-Khalsa welcomed me into his bungalow, looking little changed from the old photographs I’d seen: face narrow, eyes sharp, turban blue. A kundalini yoga practitioner who believed meditation to be an “ancient technology of the mind”, Shannahoff-Khalsa had been so disillusioned by the final report, he’d refused to sign it.

As we sat and nibbled cheese, he picked up a thick book with a shiny red cover: The Social Importance Of Self-Esteem. This was the collected work of the University of California professors. He flicked through its pages, settling eventually on Smelser’s summary of the findings. “The news most consistently reported,” he read out loud, “is that the association between self-esteem and its expected consequences are mixed, insignificant or absent.”

This was a radically different conclusion from that fed to the public. Shannahoff-Khalsa told me he was present when Vasco first saw preliminary drafts of the professors’ work. “I remember him going through them – and he looks up and says, ‘You know, if the legislature finds out what’s in these reports, they could cut the funding to the task force.’ And then all of that stuff started to get brushed under the table.”

How did they do that?

“They tried to hide it. They published a [positive] report before this one,” he said, tapping the red book, which deliberately “ignored and covered up” the science.

It was hard to believe that Vasco’s task force had been so rash as simply to invent the quote, the one that stated the findings were “positive and compelling”. What had really happened at that meeting in September 1988? I found the answer on an old audio cassette in the California state archives.

The sound was hissy and faint. What I heard, though, was clear enough. It was a recording of Smelser’s presentation to Vasco’s task force at that meeting in El Rancho Inn, and it was nowhere near as upbeat as the task force had claimed. I listened as he announced the professors’ work to be complete but worryingly mixed. He talked through a few areas, such as academic achievement, and said: “These correlational findings are really pretty positive, pretty compelling.” This, then, was the quote the task force used. They’d sexed it up a little for the public. But they had completely omitted what he said next: “In other areas, the correlations don’t seem to be so great, and we’re not quite sure why. And we’re not sure, when we have correlations, what the causes might be.”

Smelser then gave the task force a warning. The data was not going to give them something they could “hand on a platter to the legislature and say, ‘This is what you’ve got to do and you’re going to expect the following kind of results.’ That is another sin,” he said. “It’s the sin of overselling. And nobody can want to do that.”

I wondered whether Smelser was angry about the quote that got used. So I called him. He told me the university got involved in the first place only because Vasco was in charge of its budget. “The pressure [from Vasco] was indirect. He didn’t say, ‘I’m going to cut your budget if you don’t do it.’ But, ‘Wouldn’t it be a good idea if the university could devote some of its resources to this problem?’” It turned out that Smelser wasn’t at all surprised about their dubious treatment of the data. “The task force would welcome all kinds of good news and either ignore or deny bad news,” he said. “I found this was a quasi-religious movement, and that’s the sort of thing that happens in those dynamics.”

Vasco passed away, aged 82, in 2014, but I traced his right-hand man, task force chairman and veteran politician Andrew Mecca. When we finally spoke, he confirmed that it was the prestige of the University of California that had turned things around for Vasco. “That earned us some credibility stripes,” he said. Like Smelser, he felt that the university became involved only out of fear of Vasco. “John chaired their lifeblood. Their budget!” he chuckled.

How did he rate the academics’ research? “As you read the book,” he said, “it’s a bunch of scholarly gobbledegook.”

What was Mecca’s response when the data didn’t say what he wanted?

“I didn’t care,” he said. “I thought it was beyond science. It was a leap of faith. And I think only a blind idiot wouldn’t believe that self-esteem isn’t central to one’s character and health and vitality.”

Was Vasconcellos angry when he read the professors’ reports?

“The thing is, John was an incredible politician. He was pragmatic enough that he felt he had what he needed, and that was a scholarly report that pretty much said, ‘Self-esteem’s important.’ At least, that’s the spin we got in the media.”

Mecca told me that, prior to the final report’s publication, he and Vasco visited editors and television producers up and down the country, in a deliberate attempt to construct the story before it could be subverted. An extraordinary $30,000 was spent on their PR campaign: at its height, five publicists were working full time. “We decided to make sure we got out there to tell our story and not let them interpret it from the stuff that was being written by Smelser. We cultivated the message. And that positiveness prevailed.”

So nobody listened to what Smelser and Shannahoff-Khalsa were saying?

“I’m not sure anybody cared,” Mecca said. “Who remembers Neil Smelser or Shannahoff-Khalsa? Nobody! They were tiny ripples in a big tsunami of positive change.”

***

More than 20 years on, the effects of Vasco’s mission linger. Whether the tsunami of change he brought about was wholly positive remains doubtful. I spoke to educational psychologist Dr Laura Warren, who taught in British schools in the 90s, and remembers her school’s edict that staff use mauve pens to mark errors, in place of the negative red. “It was a policy of ‘reward everything that they do’,” she told me. “That turned out to be a terribly bad idea.”

The Ofsted inspectors discovered as much when they visited Barrowford primary school in 2015. But after their critical report became public, the headteacher, Rachel Tomlinson, defended herself in her local newspaper. “When we introduced the policy, it was after an awful lot of research and deliberation,” she said. “And I think it has been a success.”

• Adapted from Selfie: How We Became So Self-Obsessed And What It’s Doing To Us by Will Storr, published by Picador on 15 June at £18.99. To order a copy for £16.14, go to bookshop.theguardian.com or call 0330 333 6846

tl;dr: A hot take on a recent ‘simply stats’ post. You can still use deep learning in (some) small data settings, if you train your model carefully.

Over at Simply Stats Jeff Leek posted an article entitled “Don’t use deep learning your data isn’t that big” that I’ll admit, rustled my jimmies a little bit. To be clear, I don’t think deep learning is a universal panacea and I mostly agree with his central thesis (more on that later), but I think there are several things going on at once, and I’d like to explore a few of those further in this post.

Jeff takes a look at the performance of two approaches to classify handwritten 0s vs. 1s from the well known MNIST data set. He compares the performance of a 5-layer neural net with hyperbolic tangent activations to the Leekasso, which just uses the 10 pixels with the smallest marginal p-values. He shows, perhaps surprisingly, that the Leekasso outperforms the neural net when you only have a dozen or so samples.

Here is the figure of merit:

So that’s it right? Don’t use deep learning if you have < 100 samples because the model will overfit and you will get bad out of sample performance. Well, not so fast. I think there are several things going on here that are worth unpacking. Deep learning models are complex and tricky to train, and I had a hunch that lack of model convergence/difficulties training probably explained the poor performance, not overfitting.

Deep Learning vs. the Leekasso Redux

The first thing to do is to build a deep learning model someone would actually use on this data, namely modern versions of multilayer perceptrons (MLPs) and convolutional neural networks (CNNs). If the thesis of the original post is correct, these models should overfit badly when we only have a few samples.

We built a simple MLP with relu activations and a VGG-like convolutional model to see how they perform relative to the Leekasso. All of the code is available here. Big thanks to my awesome summer intern Michael Chen who did the heavy lifting and implemented most of this using python and Keras.

The MLP model is pretty standard and looks like this:

defget_mlp(n_classes):model=Sequential()model.add(Dense(128,activation='relu',input_shape=(784,)))model.add(Dropout(0.5))model.add(Dense(128,activation='relu'))model.add(Dropout(0.5))model.add(Dense(n_classes,activation='softmax'))model.compile(optimizer='Adam',loss='categorical_crossentropy',metrics=['accuracy'])returnmodel

and the CNN will also look familiar to anyone who has worked with these models before:

defget_cnn(n_classes):model=Sequential()model.add(Convolution2D(32,3,3,activation='relu',input_shape=(28,28,1)))model.add(Convolution2D(32,3,3,activation='relu'))model.add(MaxPooling2D())model.add(Convolution2D(64,3,3,activation='relu'))model.add(Convolution2D(64,3,3,activation='relu'))model.add(MaxPooling2D())model.add(Flatten())model.add(Dense(128,activation='relu'))model.add(Dropout(0.5))model.add(Dense(n_classes,activation='softmax'))model.compile(optimizer='Adam',loss='categorical_crossentropy',metrics=['accuracy'])returnmodel

For a point of reference, the MLP has about parameters while the CNN has nearly ! According to the hypothesis in the original post, we are going to be really screwed when we have this many parameters and only a handful of samples.

We tried to mirror the original analysis as closely as possible - we did 5-fold cross validation but used the standard MNIST test set for evaluation (about validation samples for 0s and 1s). We split the test set into 2 pieces. The first half was used to assess convergence of the training procedure while the second half was used to measure out of sample predictive accuracy. We didn’t really even tune these models, but just went with sensible defaults for most of the parameters.

We recreated python versions of the Leekasso and MLP used in the original post to the best of our ability, and the code is available here. Here are the out of sample accuracies for each model. The bottom plot is just a zoom-in of the best performing models to make it easier to read:

Wow, this looks pretty different than the original analysis! The MLP used in the original analysis still looks pretty bad for small sample sizes, but our neural nets get essentially perfect accuracy for all sample sizes. This all leads to the question…

What’s going on here?

Deep learning models are notoriously finicky to train, and knowing how to ‘babysit’ them is an important skill. A lot of parameters are problem specific (especially the parameters related to SGD) and poor choices will result in misleadingly bad performance. You should always keep this in mind when working on a deep learning model:

Model details are very important and you should be wary of blackbox calls to anything the looks like deeplearning()

Here are my best guesses at what is going on in the original post:

• Activation functions are very important and tanh networks are hard to train. That’s why the field has largely moved to the ‘relu’ family of functions.
• Make sure that stochastic gradient descent has converged. In the original comparison, the model was trained for only 20 epochs, which is probably not enough. With only samples, epochs results in only total gradient updates. One full pass over the full MNIST data of is equivalent to gradient updates, and it’s common to do hundreds or thousands of passes, roughly ~ gradient updates. If you are only going to perform gradient updates, you will probably need to use a very large learning rate or your model will likely not converge. The default learning rate for h2o.deeplearning() is 0.005, which is likely way too small if you are only doing a few updates. The models we used were trained for 200 epochs and we witnessed significant fluctuation in the out of sample accuracy during the first 50 epochs. If I had to guess, I would say lack of model convergence explains most of the difference observed in the original post.
• Always check default values for parameters. Keras is nice because the default parameters are an attempt to reflect current best practices, but you still need to make sure the parameters you’ve selected are good for your problem.
• Different frameworks can give you very different results. I attempted to go back to the original R code to see if I could get the results to line up. However, I was never able to get the h2o.deeplearning() function to produce good results. If I had to guess, I would say it’s related to the optimization procedure it uses. It looks like it’s using elastic averaging SGD to push the computation onto multiple nodes to speed up training. I don’t know if this breaks down when you only have a few samples, but that would be my best guess. I don’t have a ton of experience using h2o, so maybe someone else can figure this out.

Thankfully, the good folks at RStudio just released an R interface to Keras, so I was able to recreate my python code in pure R. The MLP we used before looks like this implemented in R:

model<-keras_model_sequential()model%>%layer_dense(units=128,input_shape=c(784),activation='relu')%>%layer_dropout(0.5)%>%layer_dense(units=128,activation='relu')%>%layer_dropout(0.5)%>%layer_dense(units=1,activation='sigmoid')%>%compile(optimizer=optimizer_adam(lr=0.001),loss='binary_crossentropy',metrics=c('accuracy'))model%>%fit(as.matrix(training0[,-1]),as.matrix(training0[,1]),verbose=0,epochs=200,batch_size=1)score<-model%>%evaluate(as.matrix(testing[,-1]),as.matrix(testing[,1]),batch_size=128,verbose=0)deep[b,i]<-score[[2]]

I dropped this into Jeff’s R code and regenerated the original plot. I changed the Leekasso a bit too. The original code used lm() (i.e. linear regression) which I found strange, so I switched it to glm() (e.g. logistic regression). The new plot is shown below:

Deep learning redemption! A similar phenomenon probably explains the difference between the python and R versions of the Leekasso. The python version of logistic regression uses liblinear as its solver, which I would would guess is a little more robust than the default solvers in R. This probably matters since the variables selected by the Leekasso are highly collinear.

• This problem is too easy to say anything meaningful. I reran the Leekasso but used only the top predictor and the results are nearly identical to the full Leekasso. In fact, I’m sure I could come up with a data-free classifier that has high accuracy. Just take the center pixel and if it’s black predict , else predict . As David Robinson pointed out:

David also showed (in an aspirational piece of dplyr wizardry) that most pairs of numbers can be classified by a single pixel. So, it’s unlikely that this problem will give us any insight into a ‘real’ small data scenario, and conclusions should be taken with an appropriate grain of salt.

Misconceptions On Why Deep Learning Works

Finally, I wanted to revisit a point Jeff made in the original post, specifically this statement:

The issue is that only a very few places actually have the data to do deep learning […] But I’ve always thought that the major advantage of using deep learning over simpler models is that if you have a massive amount of data you can fit a massive number of parameters.

This passage, especially the last part, is not the whole story in my opinion. Many people seem to think of deep learning as a huge black box with a ton of parameters that can learn any function, provided you have enough data (where enough is some where between a million and Graham’s number of samples). It is of course true that neural networks are extremely flexible, and this flexibility is part of the reason for their success. But this can’t be the only reason they work, right?

After all, there is a 70+ year history of super flexible models in machine learning in statistics, and I don’t think neural nets are a priori any more flexible than other algorithms of the same complexity.

Here’s a quick run down of some reasons why I think they’ve been successful:

• Everything is an exercise in the bias/variance tradeoff. Just to be clear, the actual argument I think Jeff is making is about model complexity and the bias/variance trade off. If you don’t have a lot of data it’s probably better to go with a simple model (high bias/low variance) than a really complex one (low bias/high variance). I think that this is objectively good advice in most cases, however…
• Neural nets have a large library of techniques to combat overfitting. Neural nets are going to have a lot of parameters, and to Jeff’s point, this will result in really high variance if we don’t have enough data to stably estimate values for those parameters. The field is very aware of this problem and has developed a lot of techniques aimed at variance reduction. Things like dropout combined with stochastic gradient descent result in a process that looks an awful lot like bagging, but over network parameters instead of input variables. Variance reduction techniques like dropout are baked into the training procedure in a manner that is difficult to replicate for other models. This let’s you train really big models (like our MLP with parameters) even if you don’t have a ton of data.
• Deep learning allows you to easily incorporate problem specific constraints directly into the model to reduce variance. This is the most important point I wanted to make and I think gets overlooked too often. Due to their modularity, neural nets let you incorporate really strong constraints (or priors if you prefer) that can drastically reduce the model’s variance. The best example of this is in a convolutional neural network. In a CNN, we actually encode properties about images into the model itself. For instance, when we specify a filter size of 3x3, we are directly telling the network that small clusters of locally-connected pixels will contain useful information. Additionally, we can encode things like the translation and rotation invariance of images directly into the model. All of this serves to bias the model towards properties of images to drastically reduce variance and improve predictive performance.
• You don’t need Google-scale data to use deep learning. Using all of the above means that even your average person with only a 100-1000 samples can see some benefit from deep learning. With all of these techniques you can mitigate the variance issue, while still benefitting from the flexibility. You can even build on others work through things like transfer learning.

To sum up, I think the above reasons are good explanations for why deep learning works in practice, more so than the lots of parameters and lots of data hypothesis. Finally, the point of this post wasn’t to say Jeff’s message was wrong, but to offer a different perspective on his main thesis. I hope others will find it useful.

[Editor’s note: At nearly 7,000 words, you probably don’t want to try reading this on an iDevice.  Bookmark it and come back later.]

Imagine something a wee bit outside your comfort zone.  Nothing scandalous: just something you don’t do often, don’t particularly enjoy, and slightly more challenging than “totally trivial.”  Maybe reciting poetry while simultaneously standing on one foot.

If I told you I would pay you a hundred thousand dollars if you did five minutes of poetry recital while standing on one foot, would you do it?  It’s an absurd image, but play it straight.  There is no hidden gotcha here.  You won’t be videotaped.  Your friends will never see you make a fool of yourself.  The revolution will not be YouTubed.  The offer is exactly as simple as you think it is: poetry, foot, $100,000.

Would you read poetry for me?

Of course you would. You’d be screamingly stupid not to.  In fact, not only would you read poetry, you’d probably take a poetry class to make sure you did it right, or go to the gym to verify “Yep, sure enough, I can stand on one foot.  Phew.  Pass me the Shakespeare.”  If you couldn’t stand on one foot, you’d fix that, because you know that is much easier than other things you routinely accomplish and you suddenly have a hundred thousand wonderful reasons to learn it, too.

What if you were talking about this at dinner with your friends, and one of them said “Oh, no, I’d never do that.  I just don’t do poetry.  I’m an engineer.  And besides, my father told me that people who stand on one foot look silly.  And what do I need $100,000 for anyhow?”  You would not clap them on the back and say “Damn straight!  Man, poets, always trying to tempt virtuous engineers into their weird poetry-spouting flamingo-standing ways.”  You’d say “Dude, it’s five minutes.  Heck, I’ll help you practice.”

This is pretty much how I feel every time I talk to my engineering friends about salary negotiation.  We overwhelmingly suck at it.  We have turned sucking at it into a perverse badge of virtue.  We make no affirmative efforts to un-suck ourselves and, to the extent we read about it at all, we read bad advice and repeat it, pretending that this makes us wise.

Dude, it’s five minutes.  Let’s un-suck your negotiation.

(New to the blog?  Hiya.  I generally write as an engineer for engineers.  Non-engineers can benefit from many of the same techniques, though the hiring market isn’t nearly as in your favor at the moment as it is for engineers in most major US metro areas.)

Why Negotiation Matters

Your salary negotiation — which routinely takes less than 5 minutes to conclude — has an outsized influence on what your compensation is.  Compensation can include money or things which are more-or-less fungible replacements for money, but it can also include interesting things which you value from “more time with your family” to “opportunities to do tasks which you find fulfilling” to “perks which make a meaningful difference in your day-to-day quality of life.”  That makes your negotiation five very important minutes.  You generally can’t do a totally bang up job on any five minutes of work this year and have your boss give you an extra $5,000.  You can trivially pick up $5,000 in salary negotiations just by sucking less.

Since salaries are shockingly durable over time, particularly if one is not good at negotiating, you can expect a $5,000 increase in salary to compound with your standard annual read-the-HR-chart-percent raise, cause a similar increase in your 401k contribution (which also compounds), and establish a higher peg for any further jobs you take (if you’re unsavvy and allow these other jobs access to your prior salary history, at any rate).  Accordingly, over the next ten years, the value of $5,000 a year extra salary is close to $100k gross, and the value of $15,000 a year extra (very achievable if you’re e.g. a young engineer who doesn’t realize that the hiring market is on fire right now, even outside of tech epicenters like Silicon Valley) is over $100k even net of taxes.

Shifting Your Mindset To Embrace Negotiation

We’ll discuss tactical advice in a moment, but let’s talk about the psychology of negotiation first.  I think that middle class Americans are socialized from a very young age to view negotiation as something that is vaguely disreputable and engaged in only by poor people.  Think of the associations you have with the word “haggling”: do you think of a successful young professional talking about thousands of dollars in a brightly lit office?  No, you probably think of an old woman arguing over a trivial sum of money in a dirty flea market.

If I were a little more paranoid and a little more Marxist, I’d honestly think that you’re so misinformed about reality that that is almost prima facie evidence of a conspiracy to keep you in the dark about this, to the advantage of people who a) you won’t negotiate with and b) who will feel absolutely no compunctions about negotiating with you.  Principally, this will be your employers.  People say that your house is the biggest purchase you’ll ever make, but it won’t be the most consequential negotiation.  If you’re sane only about 25% or so of your gross income is subject to the results of real estate negotiations.  Close to 100% is subject to the results of salary negotiations.  Thus, your salary negotiations are probably going to be the most important financial decisions you will ever make.  We socialize middle class Americans to go into them unprepared, demotivated, and fearful of success.

The reality is that rich, successful people negotiate.  (This is one important way in which they get — and stay — rich.)  It is an all-day-every-day thing in much of the business world, which is where most rich people get their money.

Your Counterparty Does Not Share Your Mental Model of Negotiation

Salary negotiations are very asymmetrical.   Companies know this and routinely exploit it.  Job seekers don’t, perhaps because they think doing so would be unfair and the word “exploit” makes them acutely uncomfortable.  So we often default by pretending that the employer is evaluating the negotiation like we would.  This is not true, and acting like it is true will harm both your interests and the interests of your future employer.

For example, many people’s mental model of employment is that an employee with a $60,000 a year salary costs about $60,000 a year to hire.  If they negotiate $65,000 instead, that’s $5,000 extra which has to come from… somewhere.  If the negotiation breaks down, then that is $60,000 saved.  This mental model is broken.

First, get into the habit of seeing employees like employers see them: in terms of fully-loaded costs.  To hire someone you need to pay for their salary, true, but you also have taxes, a benefits package, employer contributions to retirement, healthcare, that free soda your HR department loves mentioning in the job ads, and what have you.  (Trivia: for a US employer of professionals, the largest component after salary is usually healthcare, followed by payroll taxes.)  The fully-loaded costs of employees are much higher than their salary: exactly how much higher depends on your locality’s laws, your benefits package, and a bunch of other HR administrivia, but a reasonable guesstimate is between 150% and 200% of their salary.

The fully loaded cost of an engineer receiving market salaries these days in California or New York is close to $20,000 a month.  It is “only” $10,000 a month if they’re receiving a heavily below-market salary, such as if they’re working for a startup.  If you have a kid brother who majored in Flemish Dance and got a modest full-time job at a non-profit, his fully-loaded cost is still probably $4,000 a month or more.

This is a roundabout way of telling you that companies are not sensitive to small differences in employee wages because employees are so darned expensive anyhow.  You see $5,000 and think “Holy cow, even after taxes that’s a whole new vacation.  Five thousand dollars.  Five thousand dollars.  It would be so very, very greedy of me to ask for five thousand whole dollars.”  The HR department sees $5,000 and thinks “Meh, even after we kick in the extra taxes, that is only about 3% of their fully-loaded cost for this year anyhow, or seven hundredths of one percent of that team’s hiring budget.  I wonder if the cafeteria has carrot cake today?”

Virtually any amount of money available to you personally is mouse droppings to your prospective employer.  They will not feel offended if you ask for it.  (I received a comment that this is untrue for startups by someone today.  For a funded startup which has enough engineers to warrant a foosball table, the company payroll is well north of $100,000 a month.  Making a new hire is a big commitment, but they still have a lot of flexibility on the  details because the details do not shave months off of their runway.)

We’ve been talking about your employer as an abstraction, but in the instant case you’re talking to an actual person.  Let’s call him Bob.  It is Bob’s job to get you signed with the company as cheaply as possible, but Bob is not super motivated to do so, because Bob is not spending Bob’s money to hire you.  Bob is spending Bob’s budget.  Bob generally does not get large performance incentives for shaving money off of his hiring budget: you get a new Macbook if you convince Bob to give you $5k extra, but Bob gets (if he is anomalously lucky) a dinner at TGIFridays if he convinces you to take $5k less.  In fact, there are many organizations (and Bobs) for whom power, status, and money come from asking for more budget every year.  If you turn out to be on the expensive side, that is great for Bob, because a) he manages a high-powered peon so he must be a high-powered manager and b) this will help Bob get more budget next quarter.  So if you’re worried about what Bob will think of your moral character, or you want to compensate Bob because you feel you owe him for this job opportunity, do Bob a solid and negotiate in a spirited fashion with him.

You don’t owe Bob for giving you this job opportunity, by the way.  Internalize this: everyone in this discussion is a businessman.  (Some might call themselves “regular employees,” which just means they’re businessmen with self-confidence issues and poor business skills.)  If the deal makes economic sense, it will happen.  If it doesn’t, firm handshakes will be exchanged, non-specific promises will be uttered, and everyone will forget about this discussion in a matter of hours.  You will not be blackballed for negotiating.  Bob couldn’t care less and, even if he did care, he has better things to do with his time than worry about a candidate he didn’t hire.  Bob is working through a list of a dozen people right now, and his manager Dave is being such a hard case about that project’s schedule, and he’s not sure he can make his daughter’s piano recital, and the cafeteria’s carrot cake was a little dry.  Bob is far, far less invested in this negotiation than you are.

Your Negotiation Started Before You Applied To This Job

Your negotiation doesn’t happen in a vacuum.  Generic career advice is a little outside the scope of this post (though I’ve previously written a bit with engineers in mind that folks from many walks of life tell me was useful), but to make a long story short, many people think job searches go something like this:

1. See ad for job on Monster.com
2. Send in a resume.
3. Get an interview.
4. Get asked for salary requirements.
5. Get offered your salary requirement plus 5%.
6. Try to negotiate that offer, if you can bring yourself to.

This is an effective strategy for job searching if you enjoy alternating bouts of being unemployed, being poorly compensated, and then treated like a disposable peon.  (I served three years as a disposable peon in a Japanese megacorp and might be projecting a tad bit here.  Regardless, my loss is your gain.)

You will have much, much better results if your job search looks something more like:

1. (Optional but recommended) Establish a reputation in your field as someone who delivers measurable results vis-a-vis improving revenue or reducing costs.
2. Have a hiring manager talk with you, specifically, about an opening that they want you, specifically, to fill.
3. Talk informally (and then possibly formally) and come to the conclusion that this would be a great thing if both sides could come to a mutually fulfilling offer.
4. Let them take a stab at what that mutually fulfilling offer would look like.
5. Suggest ways that they could improve it such that the path is cleared for you doing that voodoo that you do so well to improve their revenue and/or reduce their costs.
6. (Optional) Give the guy hiring you a resume to send to HR, for their records.  Nobody will read it, because resumes are an institution created to mean that no one has to read resumes.  Since no one will read it, we put it in the process where it literally doesn’t matter whether it happens or not, because if you had your job offer contingent on a document that everyone knows no one reads, that would be pretty effing stupid now wouldn’t it.

You might think that desirable jobs at well-managed companies (Google, Microsoft, hot startup FooWithTheWhat.ly, etc) have layers and layers of bureaucratic scar tissue (a great image from 37Signals) to ensure that their hiring will conform to established processes and that offers will not be given to candidates sourced by using informal networks and interpersonal connections.  If you believe this, you have a dangerously incomplete mental model of how the world operates.  I have a specific recommendation for you to make that model more complete: start talking to people who actually work for those companies and who have hiring authority.  Virtually no company has a hiring process which is accurately explained by blog posts about the company.  No company anywhere has a hiring process which is accurately explained by their own documents about how the hiring process works.

I won’t give names, but all of the following are companies you’ve heard of:

• Ironclad non-compete with an IP assignment provision of major multinational… struck from the contract with four sentences of discussion.
• Major popular tech employer offered desirable employee $X as a salary because “it was the max the HR department allows for that position.”  He got hired that week at $2X.  All parties — hiring organization, HR, and employee — think they pulled one over on the other participants.
• Funny story goes here. I now can’t tell you the funny story, because literally two hours before publication someone emailed me for advice about a situation that he believes is incredibly unjust at his company, and it is exactly the funny story to the letter.  Now if I tell you the funny story he might think “Dang, I write Patrick in confidence and it ends up on the blog.”  So, no funny story today.  Suffice it to say that in my old age I treat Dilbert less as farce and more as documentary.
• “We can’t hire engineers fast enough through our standard processes so, meh, I guess we’ll circumvent them by just tossing $1 million per employee at whomever they currently work for.  Who cares, it isn’t my million.”

When Does A Salary Negotiation Happen?

Only negotiate salary after you have agreement in principle from someone with hiring authority that, if a mutually acceptable compensation can be agreed upon, you will be hired.

This is really, really important because it has direct implications for your negotiating strategy.  First, the company is going to spend a lot of time and effort on getting you to the point of agreement-in-principle.  Pretend you’ve gone through six rounds of interviews.  (You probably won’t if you get hired on informal networks, because all barriers vanish when important people want a deal to get done, but let me give some advice to someone a little less well-situated.)  Do some quick mental math on what that actually cost the company, with reference to “one man-month of an engineer’s time costs $20k” like we discussed earlier.  You’ll quickly reach the conclusion that the company has spent thousands of dollars just talking to you, and that doesn’t even count the thousands they spent deciding to talk to you instead of whoever isn’t in the room right now.  Walking away from the negotiation means that they lose all that investment.  (Yeah, sunk cost fallacy and all, but since people predictably act in this fashion you should, well, predict that they will act in this fashion.)  They really want to reach an agreement with you.

The second implication is that the inner serf worrying “If I even attempt to negotiate this, the deal will fall through” is worrying for nothing.  They’ve got thousands invested in this discussion by this point.  They want you.  The absolute worst outcome of negotiating an offer in good faith is that you will get exactly the contents of that offer.  Let me say that again for emphasis: negotiating never makes (worthwhile) offers worse.  This means you need what political scientists call a commitment strategy: you always, as a matter of policy, negotiate all offers.  (In this wide world I’m sure you can find a company who still makes exploding offers, where you get one yay-or-nay and then the offer is gone.  You have a simple recourse to them: refuse them and deal with people who are willing to be professionals.  You’re not a peasant.  Don’t act like one.)

This also means you do not start negotiating until you already have a Yes-If.  (Yes-If we agree on terms.)  Do not start negotiating from No-But.  (No-But we might hire you anyway if you’re really, really effing cheap.)  You don’t want to work for a No-But for the same reasons that smart employers hate hiring candidates who are a No-But (No-But maybe if not on my team, etc).  If they’re leaning to not hiring you, you will compromise excessively on negotiation to get them to hire you.  Compromising excessively is not the point of the exercise.  It is a seller’s market for talent right now: sell to someone who is happy to buy.

This means that any discussion of compensation prior to hearing Yes-If is premature.  If you’re still at the job interview and you’re talking price you are doing something wrong.  (Read the room: it is entirely possible that you came for a job interview, finished it, and proceeded directly to a salary negotiation.  That’s probably suboptimal, but it is OK.  Just don’t give the employer the option of having the schedule be job interview, salary negotiation, and back to job interview if they discover that you have a spine.)  The ideal resolution to the job interview is for both sides to be optimistic about the arrangement, and then you close with a warm handshake and “I look forward to receiving your offer by, oh, would tomorrow be enough time for you to run the numbers?”

You then have a high likelihood of doing your salary negotiation over email, which is likely to your advantage versus doing it in real time.  Email gives you arbitrary time to prepare your responses.  Especially for engineers, you are likely less disadvantaged by email than you are by having an experienced negotiator talking to you.

The First Rule Is What Everyone Tells You It Is: Never Give A Number First

Every handbook on negotiation and every blog post will tell you not to give a number first.  This advice is almost always right.  It is so right, you have to construct crazy hypotheticals to find edge cases where it would not be right.

For example, if your previous salary was set during the dot-com bubble and you are negotiating after the bubble popped, you might mention it to anchor your price higher such that the step down will be less severe than it would be if you engaged in free negotiations unencumbered by the bubbilicious history.  Does this sound vaguely disreputable to you?  Good.  This vaguely disreputable abuse of history is what every employer asking for salary history, salary range, or desired salary is doing.  They are all using your previous anomalously low salary — a salary which did not reflect your true market worth, because you were young or inexperienced or unskilled at negotiation or working at a different firm or in another line of work entirely — to justify paying you an anomalously low salary in the future.

Never give a number.

“But Patrick,” you cry.  “I don’t want to be difficult.”  You’re not being difficult.  You’re not doing anything immoral.  You’re not being unprofessional.  They’re businessmen, sometimes they don’t have all the information they would love to have prior to making a decision.  They’ll deal.

They already deal with every employee that they’ve ever had who was not a doormat at negotiations, which includes essentially all of the employees they really value.  Ramit Sethi (more on him later) introduced me to a concept that he calls Competence Triggers: basically, if you have to judge someone’s skill based on a series of brief interactions, you’re going to pattern match their behavior against other people who you like.  When people with hiring authority think of winners, they think of people like them who live and breathe this business thing.  They negotiate things as a matter of course: that is a major portion of the value they bring to the company.  Volunteering a number when asked says the same thing to people with hiring authority that flunking FizzBuzz says to an engineer: this person may be a wonderful snowflake in other regards, but on the thing I care about, they’re catastrophically incompetent.  It will also cause them to retroactively question competencies they’d previously credited you with.

I have literally heard that feedback, in so many words, from folks with whom I’ve had successful business dealings.  (A funny in hindsight story: I cost myself five figures with a single email.  The particulars are boring, but suffice it to say I fairly recently made a wet-behind-the-ears-engineer error in quoting a client.  He noticed.  So did my bank statement.  My bank statement kept quiet, but the client opined that it made him think less of me until we actually got to work together.)

So anyhow, you may well hear reasons why you should give a number.

Objection: “I really need a number to move the process forward.”

What you should think: “You’re lying to me to attempt to get me to compromise my negotiating position.”

What you should say: “I’m more concerned at the moment with talking to you about discovering whether we’re a mutual fit.  If we’re a great fit, then I can be flexible on the numbers with you and you can be flexible on the numbers with me.  If we’re not a great fit, then the numbers are ultimately irrelevant, because your company only hires A players and I only work at roles I would be an A player at.”

(Don’t talk like that normally?  Fine then, talk like yourself, but say substantially the same things.  Engineers overestimate how different we really are from business people: we say “10x engineer,” they say “A player,” but at the end of the day we believe that there are vast differences in productivity between workers.  OK, gut check: is this something we actually believe to be true or just something we wish for?  If it is actually your best guess about the state of reality, that has immediate news-you-can-use implications about how you should conduct your life.)

Objection: “This form needs a number.”

What you should think: “You’re lying to me to attempt to get me to compromise my negotiating position.”

What you should say: “Give me git access and I’ll fix it in a jiffy!  both people laugh No, seriously, speaking, I’m more concerned at the moment with discovering whether we’re a mutual fit…  Oh, it’s physically impossible?  Put in $1 then to get the ball rolling, and we’ll circle back to this later.”

Objection: “We want to figure out whether you’re an appropriate candidate for the position.”

What you should think: “You’re lying to me to attempt to get me to compromise my negotiating position.”

What you should say: “It’s so important to me that this is a good mutual fit for us.  Let’s talk about why I’m a great fit for this position: I know you’re concerned about $FILL_IN_THE_BLANK.  In addition to my previous successes doing it, I have some great ideas for what I’d do about that if I was working at your company.  Would you like to drill into those or is there another job area you’re more concerned about to start with?”

Objection: “I’m sorry, great try at a dodge there, but I just can’t go forward without a number.”

What you should think: “You’re lying to me to attempt to get me to compromise my negotiating position.”

What you should say (if you’re an engineer): “Well, you know, I would hate to have to walk away from the negotiation over this.  Working with your company looked it would have been such a wonderful opportunity.  I hear the hiring market is super-tight right now, would you like me to introduce you to other candidates?  Maybe we can shave a couple of months off of you filling this position.”

What you should say (if you’re not an engineer): “Damn, I guess I should have studied engineering.”

What you should say (if you’re a little put out by that comment): “Well, you know, salary is only one component of the total compensation package.  In terms of total compensation, we’re probably looking at something like $FILL_IN_NUMBER_HERE.”  (Suggested calculation: take the package value from your last company and add 5~10%.  If you don’t know how to calculate the value of your compensation package, learn that, but as a rough guesstimate salary + 30 ~ 50% for full-time employees in professional roles and the multiplier tends to scale up as your base salary scales up.)

P.S. I double majored in making things and making things up.  The joking comes from a place of love.  OK, love and schadenfreude, in solution with each other.

Listen To What People Tell You.  Repeat It Back To Them.

Properly run negotiations are not jockeying contests, they’re persuasive exercises.  (We’ll give the company a pass on the “what’s your number?” question because it is an established social ritual that they get one free pass at screwing you.  You still don’t have to cooperate with it, though.)  You know what people find persuasive?  Their own words.  People love their own words.  When you talk to them, you should use their own words.  Seriously, watch the eyes light up.

Did the solicitation for the job say “We are seeking someone with strong skills at scaling traffic in a fast-moving environment”?  Pick out the key words.  Scaling traffic.  Fast-moving environment.  “Scaling traffic” doesn’t sound like how I’d phrase it if I were writing or speaking for myself, but if you’ve just described your need to me as scaling traffic, by golly I will tell you how great I am at scaling traffic.  Reinterpret or rephrase the (true!) bits of your own story such that it fits the narrative framework which they have conveniently told you that they are going to respond to.  Did you previously work at a small business which was unencumbered by lots of process?  Sounds like a fast-moving environment, right?  Call it exactly that, then.

Micro-tip: Take notes during job interviews and salary negotiations.  It’s easy: go to the convenience store before the job interview, buy a writing instrument and a $1 notebook, jot down occasional notes when appropriate.

Can I do that?!  Of course you can.  Do you know anyone who you’ve ever thought “Man, I thought they were competent, but then it turned out they had a notebook so I had to write them off?”  No.  Taking notes says “I’m attentive and detail-oriented and I care about what you say.”  (Make sure you can take notes without playing with your pen or otherwise appearing to fidget.)  In terms of specific things that should get your pen moving, among others, I would focus on specific words they use and concerns they have so that you can come back to them later in the conversation.  Numbers are another good thing to hit the notebook, because numbers should only ever trend in a direction of “Better to you,” so you don’t want to do something stupid like saying “So how many days of vacation was that again?” and let a 24 suddenly become a 20.  (You might think “I’m going to write down the offer so I have proof of it for later.”  Get offers written, that goes hopefully without saying, but get it written by them and/or follow-up the discussion with an email recapping the important points and asking if you understood them correctly.  Your notes will not convince their HR apparatus to honor the agreement in event of a retroactive miscommunication, but an email from their decisionmaker likely will.)

People say the damnedest things.  For example, someone might spontaneously volunteer during a job interview that they’ve been interviewing for the position for six months.  (None of my clients would ever say that, of course, but then again one would hope none of their consultants would chop five figures off their own invoice with an email.)  If they say the position has been open for six months, take a note of that.  During the salary negotiation, if they have a pricing objection, one of your first responses should be “I appreciate that this is a little more money than you might have been thinking about, but this is an opportunity to get this position filled without delaying your business by another six months.  What is the value of that six months of execution to you?”  (Conversely, don’t say stupid things during job interviews such as “I need this job because…”  You never need a job.  Being needy means that the party who is not needy has automatic leverage over you: your BATNA to the negotiation is very poor.  Instead of being needy, aim for “I’m enthusiastic about the opportunity with working with you, assuming we can come to mutually satisfactory terms.”)

Micro-tip: Notice how often I say “We” and variations on “mutual win.”  Those work pretty well.  The only thing better than “We” is “You” (and variants), because people care a heck of a lot more about their problems than about your problems.  (This advice is stolen shamelessly from Dale Carnegie.)  This means that a) you should talk about their problems, concerns, and wishes and b) you should guard against your own natural tendency to bring up irrelevant things like your own problems, which typically will not help you sell the decisionmaker on adopting the mutual win you’re proposing.  Similarly, I generally try to phrase things positively rather than score debating points.  (“You just said X, but that was contradicted by your earlier statement Y, which means…” wins debating points but does not win friends and influence people.  You might try something like “Good good, but taking into account your earlier concerns about Y…”)

Research, Research, Research

Many people will tell you that you should familiarize yourself with the approximate salary range for the position in your region.  This advice is easy to act on (go to a salary aggregation site, guess what “the position” is, pray that this gives you a better number than rand(40000,120000)), but it leaves a lot to be desired.  It is 2012.  Facebook and LinkedIn exist.  You should, before any job interview, have intimate knowledge of the target company.  Prospective peers within the company are one obvious way to get it.  So are ex-employees, folks who’ve had dealings with them professionally, etc.  Key things you want to learn:

• What do they value?
• Who do they value within the company?  (Roles?  Titles?  Groups?)
• What does the career path look like for successful people within the company?
• Roughly speaking, how generous are they with regard to axes that you care about?
• Do they have any compensation levers which are anomalously easy to operate?  (For example, if you asked around, you might hear a few people say that a particular firm pushes back modestly on out-of-band increases in salary but they’ll give in-the-money option grants like candy.)
• All the fuzzy stuff: what’s the corporate culture like?  Yadda yadda.

You can even bring a lot of these questions to the job interview, which is (again) prior to the negotiation.  (Maybe not “So are you guys tightwads?” but culture-esque questions like “What are the projects this company thinks are really key to its future and how would a motivated person go about getting on them?” are both a) totally fair game and b) will win you brownie points just for asking.  Similarly, a lot of employees will, out of company loyalty, attempt to sell you on taking the job with the company by trading you very useful information.)

The more you know, the more options you have when doing negotiation, because you’ll have more things and more motivational things which you can offer in exchange for things you want.  It will also help you avoid making mistakes like e.g. getting into a rigid classification system where the classification you’re aiming at will make forward advancement towards your goals very difficult.  (Example: there are some companies where Product and QA are run like separate fiefdoms which haven’t forgotten the most recent war, and in those companies getting hired as an engineer may not be a career enhancing move if you like making things for a living.  There are other companies where people cross-function in those responsibilities all the time and applying for a job advertising as “Support Engineer” makes lateral moves onto customer-facing projects trivial.  You can find which one you’re applying to by taking any engineer out for coffee.)

New Information Is Valuable And Can Be Traded For Things You Want

There was a post recently on Hacker News about someone’s experience with a job offer from Google.  They wanted more money.  The recruiters offered to think it over, and came back with the reply that Google’s food benefit was worth a significant amount of money, with a calculation to back it up.  That is a pretty brilliant reply.  Google’s food benefit is about as old as the company.  Approximately all people wanting to work at Google are aware of its existence.  However, the explicit calculation of what it is worth is new, so if you bring up that calculation, by implication you’re offering newly found value to the negotiation.  This successfully convinces people that they didn’t really need that extra money.  It is so successful at this that Google recruiters apparently have this entire interaction scripted, since multiple people report having the exact same experience.

You should steal this tactic.  You are an expert in your own skill set, life story, and (ideally) value you can create for the company.  However, the person you are talking to is not.  If they ever resist about something which you want, consider reaching into the treasure chest that they are buying mostly blind and revealing one of the many glittering jewels inside.  They are going to get them all anyhow if they buy the chest, but each one you bring out decreases the perceived risk of buying it and therefor increases its perceived value.

Company: We can’t see our way to $88,000.

Applicant: Well, I know you do a significant amount of business with your online store.  At my last company, I increased sales by 3% by $YADDA_YADDA.  What would a 1% increase in sales be worth to you?

Company: Well, I don’t have that figure in front of me, but…

Applicant: Would it be safe to say “millions of dollars”?

Company: That sounds about right, yeah.

Applicant: Great, I can’t wait to get started.  Getting me that extra $4,000 would make this a much easier decision.  Considering that this is conceivably worth millions to you, we’d be silly not to do business with each other.

Company: I’ll see what I can do.

Applicant: Let me help give you some options!  [See below.]

(This hypothetical applicant is doing well on the negotiation but apparently needs to do more research on what conversion optimization specialists can get away with charging these days.  Here, let me help: six figure salary with all the usual perks as an employee, “senior engineer project rates” through “you might not believe me if I told you” as a consultant.)

Anyhow, simply by bringing attention to something which was hopefully already in bold print on their resume, they just increased their perceived value to the company, thus justifying the company moving a lever which (again) the company isn’t really sensitive to at the end of the day.

You Have A Multi-Dimensional Preference Set.  Use It.

Don’t overly focus on your salary number.  It is important (of course), but there are many parts of your compensation package, and many more things that you value.  Should you and the other party reach an impasse on any part of it, offer to table that part of the discussion (to be returned to later) and bring up a different topic.  You can then trade improvements for concessions (or apparent concessions) on the earlier topic.

Employer: “We were thinking $80,000.”

Applicant: “$80,000 is interesting (*) but not quite where we need to be to get this done.  Do you have any flexibility on that number?”

Employer: “I think I can convince HR to approve $84,000 but that is the best I can do.”

Applicant: “I appreciate that.  $84,000, huh.  Well, it isn’t quite what I had in mind, but the right package offer could make that attractive.  How much vacation comes with the package?”

Employer: “20 days a year.”

Applicant: “If you could do 24 days a year, I could compromise on $84,000.”

Employer: “I think I can do that.”

For those keeping score at home: the applicant never gives up anything but the employer will walk away feeling he got a good deal.

* Micro-tip: “Interesting” is a wonderful word: it is positive and non-commital at the same time.  If they tell you a number, tell them it is an “interesting” number, not a “wonderful” number.

Hoping around the offer also helps you defuse common negotiating tactics like “I have to go to $EXTERNAL_AUTHORITY to get approval of that.”  (This is in the negotiation playbook, because it works well: it injects an automatic delay in the process, and gives you a scapegoat for refusing a request while not being guilty of the refusal yourself.  You should strongly consider having an $EXTERNAL_AUTHORITY of your own.  Significant others work well.  Note that in the US your would-be employer is legally prohibited from breathing about the subject of your marital status, so something like “We’ll have to talk that over” or “That sounds reasonable, but I’ll have to run it by the family” has the dual virtues of being a socially acceptable reason to delay any major decision while also being equally available to unattached young’uns.  I talk shop with my family all the time.  I’ll certainly continue discussing employment with my family after it includes my fiancee, too.)

Anyhow, say your decisionmaker says that approving deviations from the company’s salary structure is outside of his discretion and those evil ogres in HR will likely deny his request.  That’s fine.  Express sympathy with him, because he just said he wants to give you more but can’t, then refocus the discussion on things which are within his personal authority.  (Vacation days, work hours, project assignments, travel opportunities, professional development opportunities, and the like are good areas to probe at.)  You can then use the unspent “You wanted to do something nice for me” obligation which he just acknowledged on one of the things which he has authority to grant you.

For Your Further Perusal

I’m deeply indebted to a few buddies of mine, principally Thomas at Matasano and Ramit Sethi, for teaching me to be less of a doormat in terms of negotiation.  Thomas has forgotten more than I’ll ever know about doing negotiations with clients.  Check out HNSearch.com with [tptacek negotiation] for some good advice, or (if you’re in Chicago) take him out to coffee.  By the way, if you’re an engineer and want to practice salary negotiation, Matasano is hiring and a great place to work for.

Ramit is extraordinarily persuasive about how psychology influences how people act in negotiations, and how a deeper understanding of this gives you better preparation than trying to focus on particular tactics.  When I told him that I was finally going to blog about one of our dinner conversations (you’re reading it), he said he’d give my readers a free video on negotiation tactics and the psychology underlying them in return for your email address.  The video is good and elaborates on some subjects that I haven’t covered, plus I know some people find them easier to learn from than lots of text, so that strikes me as a very fair offer. You’ll also get Ramit’s emails, which have never wasted my time.

The Facebook messaging group was at one point titled “Harvard memes for horny bourgeois teens.”

It began when about 100 members of Harvard College’s incoming freshman class contacted each other through the university’s official Class of 2021 Facebook group. They created a messaging group where students could share memes about popular culture — a growing trend on the Internet among students at elite colleges.

But then, the exchanges took a dark turn, according to an article published in the Harvard Crimson on Sunday. Some of the group’s members decided to form an offshoot group in which students could share obscene, “R-rated” memes, a student told the Crimson. The founders of the messaging group demanded that students post provocative memes in the main group chat to gain admittance to the smaller group.

The students in the spinoff group exchanged memes and images “mocking sexual assault, the Holocaust and the deaths of children,” sometimes directing jokes at specific ethnic or racial groups, the Crimson reported. One message “called the hypothetical hanging of a Mexican child ‘piñata time’” while other messages quipped that “abusing children was sexually arousing,” according to images of the chat described by the Crimson.

Then, university officials caught on. And in mid-April, after administrators discovered the offensive, racially charged meme exchanges, at least 10 incoming students who participated in the chat received letters informing them that their offers of admission had been revoked.

In an email to The Washington Post Sunday night, Rachael Dane, a Harvard spokeswoman, said “we do not comment publicly on the admissions status of individual applicants.”

But according to the Harvard Crimson article, written by Harvard student Hannah Natanson, representatives from the admissions office emailed the implicated students asking them to reveal every picture they sent in the group.

“The Admissions Committee was disappointed to learn that several students in a private group chat for the Class of 2021 were sending messages that contained offensive messages and graphics,” read a copy of the Admissions Office’s email obtained by the Crimson. “As we understand you were among the members contributing such material to this chat, we are asking that you submit a statement by tomorrow at noon to explain your contributions and actions for discussion with the Admissions Committee.”

“It is unfortunate that I have to reach out about this situation,” the email continued.

According to Harvard college admissions policies, the university reserves the right to withdraw an offer of admission if the admitted student “engages or has engaged in behavior that brings into question their honesty, maturity or moral character,” among other conditions, Dane told The Post.

The Harvard College Class of 2021 official Facebook group — like similar groups for incoming students at other universities — allows admitted students to meet classmates, ask questions and prepare for their first semester. The group is managed by the Admissions and Financial Aid Office and states in its description it is “not responsible for any unofficial groups, chats, or the content within.”

“As a reminder, Harvard College reserves the right to withdraw an offer of admission under various conditions including if an admitted student engages in behavior that brings into question his or her honesty, maturity, or moral character,” the group description states.

Cassandra Luca, an incoming student who joined the first meme group but not the second, told the Harvard Crimson the “dark” group chat was a “just-because-we-got-into-Harvard-doesn’t-mean-we-can’t-have-fun kind of thing.” Luca’s admission offer was not revoked, she told the student newspaper.

This spring, 2,056 students were invited to join Harvard’s incoming freshman class, drawing from a record number of applications — 39,506, according to a university news release. Nearly 84 percent of the admitted students eventually chose to enroll at Harvard — the highest yield rate in several decades.

The university’s decision to rescind the students’ acceptance to Harvard underscores the dangers of social media posts — public or private — among prospective college students. According to Kaplan Test Prep, which surveyed more than 350 college admissions officers, 35 percent of admissions officers said they check social media sites like Facebook, Twitter and Instagram to learn more about applicants. About 42 percent of those officials said what they found had a negative impact on prospective students.

“For better or worse, social media has become an established factor in college admissions, and it’s more important than ever for applicants to make wise decisions,” Yariv Alpher, executive director of research at Kaplan Test Prep said.

The repercussions spurred both praise and criticism from Harvard students, alumni and others at a time when university campuses across the country are in the midst of clashes over free speech. Some felt the decision was justified, while others expressed a belief that admissions officers crossed a line by judging students for their private conversations.

Erica Goldberg, an assistant professor at Ohio Northern Law School who calls herself a “free speech enthusiast,” wrote in a blog post that by “ferreting out” the members of the private chat group and revoking their acceptances Harvard “has proven that there is an oppressive force to transgress.”

Goldberg, who said she taught at Harvard Law School for three years, compared the dark humor used by the Harvard students to the popular “unabashedly irreverent” game Cards Against Humanity, “whose purpose is to be as cleverly offensive as possible.

“Even many good liberals love the game, precisely because the humor is so wrong, so contrary to our values,” Goldberg wrote. She called on Harvard to reconsider its decision.

“Harvard should not teach its students to be afraid to joke in private, among people willing to joke back,” Goldberg wrote. “Harvard should not teach students to turn on each other for speech.”

This was not the first time Harvard administrators addressed controversial messages exchanged among incoming students. Last year, after connecting on the university’s official Facebook page for the Class of 2020, incoming students joined a private unofficial chat on the GroupMe messaging app. In it, some students exchanged images that included racially charged jokes and at least one message that mocked feminists.

Though the exchanges prompted a controversy among members of that incoming class, administrators did not discipline the students who sent the messages, according to the Harvard Crimson. Thomas A. Dingman, then the interim dean of student life, said in an interview at the time that the individuals were “not matriculated students at this point.”

[Yale dean placed on leave after calling people ‘white trash’ on Yelp]

In recent months, college meme groups on Facebook have become institutions among Ivy League students; some even refer to the craze as “college meme wars.” The groups have been popping up at the campuses of Harvard, Columbia, Princeton, Penn, Yale and Dartmouth, as well as the University of California Berkeley and others.

Students use the groups to share memes picking fun at college cliches, inside jokes and even standard student topics, such as textbook prices. Old-school college rivalries often play out in the groups, as Mic pointed on in a story titled “Inside the elite meme wars of America’s most exclusive colleges.”

These groups have become so popular that many now have more members than the schools have students. In early February, a Harvard freshman started a Facebook group titled “Harvard Memes for Elitist 1% Tweens,” modeling it after two similar university-based groups: “UCLA Memes for Sick AF Tweens” and “UC Berkeley Memes for Edgy Teens,” according to an article in the Harvard Crimson magazine, Fifteen Minutes.

By early March, there were more members of the Harvard meme group than Harvard undergraduate students. The group now has nearly 30,000 members — including “pharmabro” Martin Shkreli, the former Turing Pharmaceuticals executive who became known as “Pharma Bro” after he dramatically boosted the price of a drug.

According to the Harvard meme Facebook group description, all memes must be Harvard-specific. “If the meme could apply to any group of wealthy, pretentious pseudo-intellectuals, at least Photoshop a Harvard logo in there somewhere,” the description states.

As the group’s popularity swelled, so did disputes and controversies that played out in exchanges between its members.

“Most of these fights fell in line with a discourse familiar to contemporary college campuses,” Tarpley Hitt wrote in the article in the student magazine, “with one side calling for increased moderation of posts which played into racial stereotypes or targeted marginalized groups, while the other championed freedom of expression.”

One participant in the Harvard meme group posted a link Sunday to the news about the obscene meme exchange.

“When the memes get too real,” the post read.

One Twitter user who shared the story surmised: “Are these the first casualties of the college meme wars?”

More from Morning Mix

Why ‘Over the Rainbow’ was the perfect closer for Ariana Grande’s Manchester tribute

‘Completely heartless’: Man charged with stealing Portland hero’s wedding ring as he lay dying

‘Wonder Woman’ soars at the box office, breaks superhero glass ceiling

‘He broke me’: A defiant, tearful Kathy Griffin slams attacks by Trump and his family

It’s been more than 20 years since Microsoft used the Workstation branding in its Windows 4.0 Workstation edition, but it appears the company is ready to bring it back. Twitter users @AndItsTito and @GrandMofongo have discovered references to a new edition of Windows 10 in a build Microsoft accidentally released to testers last week. Described as “Windows 10 Pro for Advanced PCs,” the new version appears to cater for significant hardware demands.

In a leaked slide, Microsoft describes the edition as “Windows 10 Pro for Workstation” with four main capabilities:

• Workstation mode: Microsoft plans to optimize the OS by identifying “typical compute and graphics intensive workloads” to provide peak performance and reliability when Workstation mode is enabled.
• Resilient file system: Microsoft’s file system successor to NTFS, dubbed ReFS, is enabled in this new version, with support for fault-tolerance, optimized for large data volumes, and auto-correcting.
• Faster file handling: As workstation machines are typically used for large data volumes across networks, Microsoft is including the SMBDirect protocol for file sharing and high throughput, low latency, and low CPU utilization when accessing network shares.
• Expanded hardware support: Microsoft is also planning to allow Windows 10 Pro for Workstation on machines with up to 4 CPUs and a memory limit of 6TB. Windows 10 Pro currently only supports 2 CPUs.

Microsoft is clearly targeting this new edition of Windows 10 to power users who are processing mass amounts of data each day. This will likely be used mostly in enterprise scenarios, and the software giant appears to still be working on the exact feature set and naming.

Imagine walking into a roomful of strangers. Perhaps you’ve travelled to a new city. You don’t know anyone, and no one knows you. You’re free to do anything or go anywhere or talk to anyone. How do you feel?

Perhaps you feel free of the judgment and scrutiny from acquaintances or associates. Perhaps you feel energised that you can use this opportunity to experience life on your terms, at your own speed. But whatever your feelings would be, you would at least safely assume that you can enter this isolated situation without being monitored or tracked by a far-flung company or individual – right?

Wrong. What you’re experiencing as you walk into that room is anonymity: a sociocultural phenomenon that’s afforded privacy and freedom. But in the year 2017, it’s pretty much all but dead. It’s emerging as one of the major challenges of our age: how should we go about both ensuring national security and enhancing our lives through technology, whilst also maintaining a basic right to privacy that feels like it has existed since the beginning of human history?”

The internet made us stop caring

Anonymity, which is Greek for “no name,” is a uniquely human psychological experience: it’s the idea that we all have identities to present to the world, but under certain circumstances, can switch the identity off and operate in total secrecy.

“We need a public self to navigate the social world of family, friends, peers and co-workers,” says John Suler, professor of psychology at Rider University in New Jersey, and author of The Psychology of Cyberspace. “But we also need a private self – an internal space where we can reflect on our own thoughts and feelings apart from outside influence, where we can just be with our own psyche. Our identity is formed by both. Without one or the other, our wellbeing can easily become disrupted.”

Being anonymous allows us to try new things or express ideas without being judged. In 2013, researchers at Carnegie Mellon University in Pennsylvania published a study in which they conducted in-depth interviews with dozens of internet users on four continents. One interviewee, for instance, created an anonymous online community for English learners to practise their language skills. Anonymity helped them better manage certain spheres of their lives. One participant said that he frequented message boards to help people solve technical problems, but sought to avoid unwanted commitments through the detached nature of the internet. Plus, being anonymous in an environment like the internet can help safeguard personal safety.

“Our results show that people from all walks of life had reason, at one time or another, to seek anonymity,” the researchers wrote of the 44 interviewees.

But according to a 2013 study from the Pew Research Center, while most internet users would like to remain anonymous, most don’t think it’s entirely possible. The study found that 59% of American internet users believe it is impossible to completely hide your identity online.

And while some people are taking basic steps to preserve anonymity, like deleting their browsing history, many users who say they value anonymity aren’t really walking the walk.

Earlier this year, a meta-analysis published in the Journal of Communication explored something called the “privacy paradox”: the idea that, while people value privacy, they do little in practice to preserve it. Think about it: when was the last time you actually read one of those many, lengthy privacy policy updates before clicking “I agree”? Our attitude toward privacy has become increasingly blasé.

One could even argue it’s even detrimental not to divulge at least some info. Career coaches worldwide trumpet the professional importance of having a fleshed-out public LinkedIn photo complete with full name, headshot, full work history and more.

Perhaps this is more of a cultural thawing toward previously uptight attitudes. I remember getting on the internet for the first time. It was the 1990s and on my father’s work computer. In those days, internet service providers went to great, paranoid lengths to discourage users from divulging even basic tidbits in their public profiles, like first name, city, even gender.

Today? Personal info flies freely and wildly across the web, often on our volition: Instagrammed selfies of ourselves and loved ones, complete with geotagged locations. Social media users engaging in political spats and horrible insults, despite the fact that the target of their harassment could click on their real names and real photos and see who they actually are.

People tend to think of cyberspace as some kind of imaginary space without true boundaries, a space not to be taken too seriously – John Suler

“People tend to think of cyberspace as some kind of imaginary space without true boundaries, a space not to be taken too seriously – not subject to the same rules and standards as the ‘real’ world,” says Suler. In just the span of a few short years, people’s comfort level with the internet has risen to the point where information-sharing can be careless or reckless.

Call it privacy fatigue, but our increased interdependence on our smart devices and social media has given some of us a largely lazy attitude toward staying totally anonymous.

But what if you’re one of those people who eschews Facebook, has no social media presence, and goes to great lengths to leave a fleeting digital footprint? Sorry – your anonymity is at risk too.

Going off the grid is no fix

While skipping a Facebook profile is a good way to disconnect, there are still ways people can sleuth out your identity.

Paul Ohm, a law professor at Georgetown University in Washington, DC, says there’s “intentional anonymity” and “inferential anonymity”: the former being what we choose to keep close to the vest, and the latter referring to the data that a Google-savvy sleuth can “infer” from you online – that is, dig up loads of personal information about you using a single fact as a starting point.

“It’s become increasingly clear that it’s a losing game,” Ohm says on achieving total anonymity in 2017. “As long as someone knows something about you, they can probably find other things about you, and do it really successfully – more than they have in the past.”

If you’re a social media party pooper, that might mean old flames or long-lost classmates can’t track you down. But that doesn’t mean you’re anonymous from big entities, like corporations or the government.

“It’s much harder to be anonymous than it was 20 years ago, at least from the biggest companies and the government,” says Peter Swire, professor of law and ethics at Georgia Institute of Technology, and who served on US President Barack Obama’s Review Group on Intelligence and Communications Technology.

Advertisers track your internet habits across your devices – phone, tablet, laptop – to know where you habitually go, shop, and what kind of websites you visit, and there has been growing controversy about what internet companies should be allowed to track and sell to third parties.

Earlier this year, US President Donald Trump signed a law that repealed requirements for internet service providers to get permission from customers before gathering and sharing their personal data, like your web history and what apps you use.

Swire says we’re living in a “golden age of surveillance”: If you’re a person of interest in an investigation, looking up details like financial records, medical records, web history or call history is a breeze. And that hints at a larger, serious privacy concern in the age of cybersecurity breaches and digital services that keep your bank information and home addresses on record. It’s hard to go undetected these days.

What’s more? Ohm says we’re approaching the “next great frontier in advertising”: your location.

Sure, websites can tweak adverts to zero in on your interests based on the web searches you’ve made on the same device, or sites visited. But companies and advertisers are chasing technology and business deals that pinpoint your exact whereabouts in real-time for ‘personalised’ advertising. For example, an advert could flash on your mobile phone’s screen offering a coupon for a store you’re half a mile away from.

Unless you’re willing to live without the internet or without any smart device, it’s practically impossible to go completely off the grid.

“This is a bad time to be a spy,” Swire says. In other words, even for people whose job it is to be anonymous, it’s hard to be anonymous.

Still, there are plenty of instances in which anonymity is problematic, even dangerous. Is its demise actually a blessing for society?

Is the death of anonymity good?

Swire says that anonymity is a relatively new construct, and that the rise of cities gave rise to it. So, we’ve spent far more time living without it than living with it.

“Anonymity didn’t exist in small towns in the days of yore,” Swire says, where everybody knew everybody’s business. “To some extent, urban living created anonymity. The difference today is that even in a big city, each of us leaves breadcrumbs that an investigator can follow.”

Anonymity also has a dark side. In that same Carnegie Mellon study, 53% of interviewees admitted to malicious activities, like hacking or harassing other internet users, or engaging in “socially undesirable activities", like visiting sites that depicted violence or pornography, or downloading files illegally.

There may be signs that, while most people certainly want to keep sensitive information like bank accounts and medical records safe, others may not care about sacrificing true anonymity for a perceived greater good.

In a 2015 Pew study, Americans who were surveyed felt torn between maintaining privacy rights and ensuring national security: 56% surveyed said that they were more concerned that the government’s anti-terrorism policies hadn’t gone far enough to protect citizens, even if that meant sacrificing some civil liberties, like online privacy.

Meanwhile, YouGov, an internet market research firm, found in a survey last year that about nearly half of Britons contacted said that “more should be done to help the security forces combat terrorism, even if this means the privacy of ordinary people suffers.”

In any case, efforts to completely anonymise our activities are more or less futile: With the rise of the internet of things, more and more of the devices we use every day will require our personal information to function, and the more they’ll be integrated into our lives.

“There is this huge disconnect,” Ohm says. “Do we believe what people say when an interviewer asks them [about privacy], or do we believe their purchasing habits?”

Waning anonymity sounds inevitable. Still, if you do want to protect your privacy as best you can, the experts do offer a few tips.

Best practices you can use

Earlier this year, Pew found that most Americans don’t trust big institutions like the government or social media sites to protect their personal information – and yet, ironically, most Americans don’t follow best practices to protect their identities online.

What are some of those best practices? Keeping your passwords under lock and key, making a different one for each service, and making them hard to guess. But if you’re more concerned about your reputation than hackers, a little common sense goes a long way.

Follow the front page test: Don’t put comments down in texts or emails that would bother you if they were on the front page of the newspaper

“Follow the front page test,” Swire suggests. “Don’t put comments down in texts or emails that would bother you if they were on the front page of the newspaper. I give that advice to intelligence agencies, and I give that advice to ordinary people.” Because while some people may not care about third parties or governments tracking their purchasing habits, people will definitely care more about being anonymous when it involves people they interact with on a daily basis.

“You might not care if a busy bureaucrat or internet company can access those gossipy emails, but you’d really care if your boss sees them instead,” Swire says. Using encrypted messaging apps like Signal and WhatsApp make your messages more private and more difficult to trace.

But if we’re going to reassign real cultural value to anonymity; to secure it as a basic right people are entitled to, it’s going to take a lot more than just individual action, and a lot more than encryption apps you can load up your phone with.

It’s going to take sweeping societal change. It’s going to take governments, advertisers, and tech corporations worldwide to agree on a baseline system of ethics. It’s not just about customers opting out of digital services – it’s about the choice to temporarily opt out of their public-facing identities, as well.

“All of us need to keep some private space where our deepest dreams and darkest fantasies are hidden away from other people – it gives us room to develop as humans, to try out different thoughts and different sides of ourselves,” says Swire. “That doesn’t change because of the internet.”

--

Bryan Lufkin is the editor of Future Now for BBC Future. Follow him on Twitter at @bryan_lufkin.

Join 800,000+ Future fans by liking us on Facebook, or follow us on Twitter.

If you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Earth, Culture, Capital, and Travel, delivered to your inbox every Friday.

Namespaces and control groups (cgroups) are responsible for the magic behind Linux containers. The support for namespaces initially appeared in 2.4.19 kernels (mount point/file system isolation), but there are now six different types of namespace abstractions in the mainline of the contemporary kernels. From the kernel’s perspective, container is just another process with it’s own set of resources - file descriptors, process address space and processor’s state. For instance, a containerized nginx web server exposes external PIDs for it’s master and worker processes:

[nedo@archrabbit]$ ps  -eo pid,comm,cmd | grep nginx4971 nginx           nginx: master process nginx -g daemon off;4987 nginx           nginx: worker process

From the output of the ps command it’s hard to differentiate containerized processes from other processes running on the host. We can provide another option to the ps command so it also shows the cgroup to which the process is bounded.

[nedo@archrabbit]$ ps  -eo pid,comm,cmd,cgroup | grep nginx4971 nginx           nginx: master process nginx      9:blkio:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
8:net_cls:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
7:devices:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
6:pids:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
5:cpu,cpuacct:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
4:freezer:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
3:cpuset:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
2:memory:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
1:name=systemd:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe7513264987 nginx           nginx: worker process       9:blkio:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
8:net_cls:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
7:devices:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
6:pids:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
5:cpu,cpuacct:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
4:freezer:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
3:cpuset:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
2:memory:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326,
1:name=systemd:/docker/91dfd31b99a29c145b2f183970fc9c197261c8381463330aef8c262abe751326

The output is much more verbose now, but we can see processes are attached to the cgroups of the docker hierarchy, and so assume those are running inside container.

Surprisingly, the life of a container starts by issuing a clone system call which creates a new process descriptor. The newborn process can share a number of resources with its parent process depending on the value of the flags argument. Typically, both child and parent share the same memory address space until one of them decides to write a new memory page, when a copy of that page is moved to the address space of the process which requested the write operation (this optimization technique is commonly known as CoW). Besides sharing the memory address space, they often share file descriptor table and file system information. However, the child process may ask for a separate system resource, including an isolated namespace by providing one of the following flags - CLONE_NEWNS, CLONE_NEWPID, CLONE_NEWUSER, CLONE_NEWUTS, CLONE_NEWIPC, CLONE_NEWNET. For each of them a new namespace is created and the child process becomes the member of that namespace. The perception of the process is to have its own instance of the system resource that’s only visible to the members of the same namespace. That could be analogous to how kernel provides an illusion to a process through processor virtualization and virtual memory that it’s the only running process on the system (when it’s actually sharing the CPU cycles and physical memory with another processes).

The process can call unshare system call as an alternative mechanism to namespace creation, as well as it can join an existing namespace via setns syscall. The latter needs a file descriptor that identifies a namespace to which the process would like to join (the process can obtain the after-mentioned file descriptor from /proc/[pid]/ns).

File system isolation

Mount namespace isolates the set of mount tables. Thus, the collection of processes have a completely independent view of the file system hierarchy. The mount points are only visible to a group of processes of the same mount namespace and they don’t propagate to other mount namespaces, providing the ability to the process to have its own rootfs. To create a new mount namespace, we pass the CLONE_NEWFS flag to the clone syscall. In the example above, we allocate some memory for the child’s stack and pass the callback function that gets executed in the context of the child process. If the call to clone is successful, we’ll have a child process attached to a brand new mount namespace.

fnmain(){letstack=&mut[0;1024*1024];matchunsafe{clone(child_cb,stack.as_mut_ptr()as*mutc_void,CLONE_NEWNS,ptr::null_mut())}{-1=>panic!("unable to create child process"),_=>{}}}

However, because the child process inherits the copy of the parent’s mount namespace, an invocation to pivot_root is required to change the root file system of the process. One of the requirements of the pivot_root syscall enforces the file system directories that are about to be swapped, can’t share the same tree. Calling the mount function with MS_BIND flag get our way out.

NOTE:pivot_root may also fail if the root file system is mounted as shared. To workaround that, run this command: mount --make-rprivate /

fnpivot_root(rootfs:String)->Result<(),&'staticstr>{unsafe{ifmount(rootfs.as_ptr()as*consti8,rootfs.as_ptr()as*consti8,ptr::null(),MS_BIND,ptr::null())!=0{returnErr("unable to mount rootfs");}letoldrootfs=String::from(format!("{}/.oldrootfs",rootfs.clone()));if!Path::new(&oldrootfs).exists(){create_dir(oldrootfs.clone());}ifsys_pivot_root(rootfs,oldrootfs)!=0{returnErr("unable to change rootfs");}// change to root directoryCommand::new("chdir").arg("/").spawn();Ok(())}}

PID and IPC namespaces

Recall we mentioned the containerized nginx’s instance exposes its external PIDs on the host system. Besides that, every process inside container has an internal PID. This correspondes to a numeric value of 1 for the first process inside container which acts as the init process (waits for and reaps orphaned child processes). The isolation of the PID number space is the guarantee for different PID namespaces to be able to have processes with same PIDs. The PID namespace is created by passing the CLONE_NEWPID flag to the clone or unshare system calls.IPC namespace (which it’s name implies) isolate interprocess communication mechanisms such as POSIX message queues or System V IPC objects. Passing CLONE_NEWIPC flag to the clone system call creates an isolated IPC namespace.

Here is the full source code that illustrates the creation of new mount, PID and IPC namespaces. Note the code has been reduced to the minimum for simplicity reasons. It makes use of the libc crate to invoke the system calls through standard C library. If you are looking for a high level abstraction, check out nix.

externcratelibc;uselibc::{c_void,c_int,c_long,clone,mount,syscall,MS_BIND,CLONE_NEWNS,CLONE_NEWPID,CLONE_NEWIPC};usestd::ptr;usestd::path::Path;usestd::fs::create_dir;usestd::process::Command;staticSYSPIVOTROOT:c_long=155;extern"C"fnchild_cb(args:*mutc_void)->c_int{matchpivot_root(String::from(rootfs)){Ok(())=>{// we are now inside container// execute any command of your choiceprintln!("{:?}",Command::new("cat").arg("/etc/issue").output());},Err(e)=>{println!("error: {}",e);}};0}fnpivot_root(rootfs:String)->Result<(),&'staticstr>{unsafe{ifmount(rootfs.as_ptr()as*consti8,rootfs.as_ptr()as*consti8,ptr::null(),MS_BIND,ptr::null())!=0{returnErr("unable to mount rootfs");}letoldrootfs=String::from(format!("{}/.oldrootfs",rootfs.clone()));if!Path::new(&oldrootfs).exists(){create_dir(oldrootfs.clone());}ifsys_pivot_root(rootfs,oldrootfs)!=0{returnErr("unable to change rootfs");}// change to root directoryCommand::new("chdir").arg("/").spawn();Ok(())}}fnsys_pivot_root(root:String,oldroot:String)->c_long{unsafe{syscall(SYSPIVOTROOT,root.as_ptr(),oldroot.as_ptr())}}fnmain(){letstack=&mut[0;1024*1024];matchunsafe{clone(child_cb,stack.as_ptr()as*mutc_void,CLONE_NEWNS|CLONE_NEWPID|CLONE_NEWIPC,ptr::null_mut())}{-1=>panic!("unable to create child process"),_=>{}}}

Your standard citizen today is an avid, savvy, and therefore extremely demanding consumer of technology. Wearables, smartphones and social media dominate their lives and purchases. This has created the perfect storm for fluid Agile methods like the Scrum framework to take over the IT world because it now demands the speed that heavier and more mechanical waterfall approaches are not capable of delivering.

Agile has became the reference approach for modern product development in the last few decades. However, we still see a lot of misinterpretation and abuse of its core values. This is so common that there is a name for it. The experienced Agile community likes to call this unwanted byproduct frAgile. If you don’t handle it with care, it is sure to shatter your product and loudly crash your entire team.

The Unbearable Lightness of Being frAgile

Agile is all about being lightweight, nimble and flexible—that’s why it can quickly adapt to change. However, frAgile arises when the attempt to adopt Agile is clumsily undertaken. It’s like trying to run in a track meet with your shoelaces untied.

With this in mind, I’m going to walk you through a few examples of how Agile can indeed become frail, and how all of them are inevitably linked to frAgile anti-patterns. And I mean walk, because you have to learn how to walk before you can (speed) run.

On Your Mark…Get Set…Scrum

Before we get there, I must first explain how I got to a point where I am able to quickly identify this phenomenon. I have been in two quite distinct scrum environments. First, as a product owner for a software house that built a widespread B2C product from scratch; later on, as an engagement manager at OutSystems.

At OutSystems, I’m helping deliver one of our largest digital transformation initiatives. As a result, I now have a set of common and proven core best practices that are key for setting anyone up to have the smoothest demo weeks. Ever.

Without further ado, here are 4 hints that what you thought was Agile might actually have become frAgile. And, of course, my focus is a specific Agile practice: Scrum.

1. Your Team Consistently Fails to Meet the Plan, Carrying Work Over to the Next Sprint

DILBERT © 2003 Scott Adams. Used By permission of ANDREWS MCMEEL SYNDICATION. All rights reserved.

Working software over comprehensive documentation: Teams often become so obsessed with the “working software” part of this core Agile value that they neglect user stories, which are vital. Team members then are discussing and agreeing on the sprint’s scope while the sprint is already running—or walking because we’re not putting ourselves in anyone’s untied shoes—instead of using this time for speedy and objective development work.

Adding to this, these late definition activities normally linger on and have little-to-no support from business stakeholders.

The previous estimates you used for your sprint planning will have no basis in reality; your team will work daily through the unknown until the Day of Reckoning comes: the sprint demo.

Without taking the time to develop all originally “planned” user stories, predictability becomes an issue. The business feels they’re not getting what they expected and that they never know what sprint velocity to expect. This is the most recurrent and harmful frAgile anti-pattern I’ve seen around.

The point is that this Agile core value is not an excuse to overlook the value of collaborative and well-defined user stories, which you should do ahead of the sprint’s start so they can be used as a basis for proper planning. Yes, you should not be compiling the mile-long technical or functional specs document of old. But good user stories in no way resemble these behemoths. Instead, these straight-to-the-point descriptions of a feature capture the perspective of whoever wants a feature or capability (it’s Scrum, so that’s probably a customer) so you can plan what needs to be done.

2. Your Product’s Technical Debt is Untamable But Your Team Works Around the Clock

DILBERT © 2010 Scott Adams. Used By permission of ANDREWS MCMEEL SYNDICATION. All rights reserved.

Constant in-sprint changes and new stuff keep popping up. Your development team is continuously restarting its work and working against the clock because they need to make sure that their delivery meets expectations at the end of the sprint. Does this ring a bell?

If you allow this dynamic to go on for too long, it becomes the norm. On one hand it actually relieves business stakeholders from committing to a fairly defined and stable sprint scope—which can be addictive. On the other hand, the development team will keep piling up the most gruesome technical debt known to man. Add to it a huge list of defects, thanks to the fact that they don’t have the necessary stability to nurture their code base, and you’ll be set for disaster.

Responding to change over following a plan: Since Agile is mostly about embracing change and Scrum allows for requirements volatility, it kind of feels like you are doing what you are supposed to do. Well… not really, I have to tell you! Even though you should have a healthy degree of flexibility while you are developing, the change that you should welcome and embrace is not one that impacts your sprints, day in and day out. Bigger change should be introduced from sprint to sprint, ideally driven by end user feedback.

3. Your [Product Owner/Team Captain/Senior Developer] Is Gone and Now the Rest of the Team Trembles at the Sight of Its Next Release

DILBERT © 2007 Scott Adams. Used By permission of ANDREWS MCMEEL SYNDICATION. All rights reserved.

Think about one of those big or complex projects that live off unstructured processes for managing requirements, for instance. Now, imagine one or two key members leave your team. Given the job rotation that is the norm these days in the IT world, this is not uncommon, right?

When your local heroes are gone, none of the standing team members will have the same version of what the team should deliver. No one will be able to describe the existing system behavior with complete accuracy, either.

Testers won’t be able to perform their role satisfactorily because they won’t have proper requirements to test the software against. The go-to person isn’t around anymore, so your team won’t be willing or able to determine how sound the product will be for its next release.

Before long, delivery will stall, deadlines will pass unmet and quality standards will be a dream, not reality.

Individuals and interactions over processes and tools: The trick here, as with all other Agile values, is to find the right balance. In this case, having neglected how the work to be done is documented and managed has led your team to a very delicate position after the loss of a key player. Be sure to carefully and thoroughly assess what may indeed be deadweight holding your team back versus what actually provides the necessary support for the fast-paced of world-class engineering.

4. Your Project Is Near Completion and You Just Realized that Your Customer Doesn’t Really Get Agile

There are too many organizations out there that still can’t grasp what this Agile thing is all about. They tend to appreciate the flexibility the Agile methodology provides, but, upon closer inspection, you will notice those at the top of their hierarchy still have a waterfall mindset. They talk a good game by throwing around terms like sprint, scrum and iterative, but they don’t really know what they are or do.

In this kind of setup, agreeing on the scope of delivery will put you in delicate situations. Plans tend to change quickly—and this is one of the main reasons IT chooses Agile — but the contract’s feature list usually does not.

It’s not uncommon for the commercial relationship to then deteriorate. Your customer may decide to litigate, stop paying its bills… the works. All due to a lack of organic adoption of the Agile values and principles.

Customer collaboration over contract negotiation: Make sure that you assess how Agile your customer’s organization really is in its multiple hierarchical levels.

Set very clear communication flow from day 0, because establishing the Agile ins and outs is an absolute must. Plan for ongoing coaching to be part of your day job, as well. If a feature list needs to be met, make sure that any necessary changes to the scope you agree on are duly managed internally in your customer’s “home.”

Main Takeaways

To round it all up, here are the key takeaways for avoiding frAgility in a scrum environment:

1. Invest in proper user stories ahead of sprint planning. This will more than pay back the investment made in predictability and guaranteed business alignment. Remember: garbage in –> garbage out. User stories are to Scrum what oil is to your car engine!
2. When developing in your sprint, accept only manageable changes to your scope. Keep trigger-happy changes at a distance. This will give your code base the stability it needs for steady performance and easy workability so that you keep on delivering further value quickly.
3. Nurture your team and discard the perceived deadweight with care. Most IT processes are perhaps older than you and I. They exist for a reason, so be cautious when you’re simply discarding what you perceive as deadweight. Otherwise, you will be in danger of losing the structure your team needs in order to thrive.
4. Be a proactive Agile evangelist. Never assume that your customer is as educated and enabled as you expect them to be, even if they tell you they use Agile methods. They might even say they use Scrum when they don’t. Remember that all long-lasting relationships always start from a very clear alignment of expectations.

More to Come for Agile and Scrum

There is more that can be said in regards to Agile vs. frAgile. The selections for this article pertain to the most recurrent and hazardous pitfalls. No form of Agile is one-size-fits-all. Nevertheless, there are clear indicators, common to any context, that will signal if you are doing it right. End users are happy. The quality of what is delivered is spot on and the project team is healthy and eager.

This article is the first in a series that describes multiple best practices for Agile and Scrum. Some are already industry standards, which will pretty much just be pinpointed in the orchestration of our proposed method, while others derive directly from experience. All in all, we will be providing you with recipes to cater to key aspects of the two most important activities that you have, as either a product owner or engagement manager: Knowing what to build and how to document it, so that you can then properly manage your backlog for true agile development. Stay tuned! In the meantime, check out what Andy Hunt, a co-author of the Agile Manifesto, says about enterprise software and Agile in this blog or this podcast.

There was plenty to blame: the car wreck that broke his back. The job pouring concrete that shattered his spine a second time. The way he tore up his insides with cigarettes, booze, cocaine, and opioids.

It all amounted to this: Carl White was in pain. All the time. And nothing helped — not the multiple surgeries, nor the self-medication, not the wife and daughter who supported him and relied on him.

Then White enrolled in a pain management clinic that taught him some of his physical torment was in his head — and he could train his brain to control it. It’s a philosophy that dates back decades, to the 1970s or even earlier. It fell out of vogue when new generations of potent pain pills came on the market; they were cheaper, worked faster, felt more modern.

advertisement

But the opioid epidemic has soured many patients and doctors on the quick fix. And interest is again surging in a treatment method called biopsychosocial pain management, which trains patients to manage chronic pain with tools ranging from physical therapy to biofeedback to meditation. It helped Carl White, a 43-year-old social worker from Leroy, Minn.

The catch? It can take weeks and cost tens of thousands of dollars — and thus remains out of reach for most patients with chronic pain.

“We’ve been banging our heads on the wall, and banging our fists on the door, trying to get insurers to pay for this,” said Dr. Bob Twillman, executive director for the Academy of Integrative Pain Management. “For the most part, they will not.”

When pain is not just physical

Chronic pain affects nearly 50 million Americans, according to the American Pain Foundation. The largest drivers include migraines, arthritis, and nerve damage — but in many cases, emotional trauma also contributes to the sense of misery.

“We have a lot of people in this country who are unhappy, isolated, and hurting,” said Jeannie Sperry, a psychologist who co-chairs the division of addictions, transplant, and pain at Mayo Clinic. “Depression hurts. Anxiety hurts. It’s rare for people to have chronic pain without one of these co-morbidities.”

Indeed, chronic pain has a substantial psychological element: Being in pain often leads to self-imposed isolation. That loss of a social network then leads to anxiety, depression, and a tendency to catastrophize the pain — so that it’s all a patient can think about.

So even if the initial cause of the pain is treated medically — with opioids, surgery, steroid injections, or physical therapy — it’s unlikely to go away entirely. That may be why just 58 percent of patients who routinely take prescription painkillers say the drugs are effective in managing their chronic pain, according to the American Academy of Pain Medicine.

“In the past, pain was viewed just as a physical issue,” said Robert Gatchel, a pain management researcher and professor at University of Texas, Arlington. “The thought was, if you cut something out, the pain will go away — but lo and behold, it doesn’t in many cases, and sometimes [the pain] gets worse.”

Acute pain is generated in the peripheral nervous system, which conducts danger signals to the brain. From there, the brain determines whether it’ll experience the pain signals or ignore them, Sperry said. “In the case of chronic pain, that system has gone awry,” Sperry said. “Without training your brain to turn down the alarm system, the alarm keeps going off all the time.”

So although the pain may have originated in the foot, patients end up with headaches, chronic nausea, chronic fatigue, and back pain — developing a host of other symptoms as the brain short-circuits.

“Focusing solely on a pain generator in the body,” like a herniated disc or nerve damage, “utterly and completely misses the chronic, complex, changing nature of chronic pain” as it’s processed and experienced in the brain over time, said Dr. Tracy Jackson, an associate professor of anesthesiology and a pain specialist at Vanderbilt University.

Bracing for ‘New Age-y nonsense’

The second time White broke his back, he decided to shift gears: He left the cement pouring business and got first his high school diploma, and then a college degree. He began working at Mayo Clinic as a neurosurgery coordinator — but still relied on alcohol and painkillers to get through the day.

It’s tough to estimate how many people with chronic pain develop a dependence on medications, but in 2014, about 2.5 million adults had opioid addictions — and many of those addictions started with a prescription for potent pain pills.

White soon found that unless he distracted himself with intense physical labor, he focused obsessively on the pain, and his thoughts spiraled into darkness. “If it’s a 1-to-10 pain scale, a chronic pain patient will say, ‘Mine’s at a 12 or 13,’” White said. “A 1-to-10 scale isn’t even sufficient. When I was at my worst, all I know is that my pain was on my mind 24/7.”

By then, White had run out of medical options. No one would operate on him, and the pills were making him feel worse, not better. Finally, in 2009, Mayo referred him to its pain management clinic.

White went in with a great deal of skepticism. He recalls sitting in the back of the meeting room, arms crossed, waiting for the pain rehabilitation specialists to tell him some “New Age-y nonsense.”

Instead, in that short, intensive program, White learned tools that worked for him — some of them very simple instructions, like how to lift heavy objects with his legs instead of his back. The program also helped him process some traumas from his childhood, White said, such as when he was orphaned at age 10 when he found his abusive, alcoholic father dead.

The psychological component also helped White learn to be kinder to himself — easing a great deal of self-imposed pressure to overexert himself.

These days, White’s baseline pain stays at a 4 — meaning, it’s always there, but it’s manageable. On the bad days, he considers small things, like getting out of bed in the morning, to be victories. “Instead of lying in bed, I gotta get my butt out of bed — otherwise it becomes a tomb, and the ‘stinkin’ thinkin’’ comes back,” White said.

He’s now working at the Minnesota Adult and Teen Challenge, a faith-based addiction therapy program in Rochester. And he’s working on launching a pain management program for his clients.

Plumbing psychology to deal with pain

There used to be hundreds of integrated pain management centers all across the country. But in the ’90s, the insurance market shifted; more patients joined managed care plans that limited them to a narrow network of doctors. Then in 1996, the powerful opioid OxyContin hit the market. It quickly became the tool of choice for controlling pain.

Soon, there were just four major integrated pain management centers left: Mayo Clinic, Johns Hopkins University, Cleveland Clinic, and Stanford University.

The Mayo Clinic’s outpatient pain program runs for three weeks, and keeps patients busy from 8 a.m. and 5 p.m.

They do physical and occupational therapy — learning, for instance, how to go shopping or do yard work in ways that won’t aggregate their pain. And the program includes four to five hours of lessons each day on how to understand pain. Patients learn to relax, breathe slowly, and meditate to mitigate some of the anxiety-related pain flare-ups. Entire sessions are dedicated to understanding the psychological underpinnings of their own pain.

“By the time people get here, they have a lot of functional disability,” said Sperry, who helps run the program. “They’re fearful, because they’re getting such strong signals in the brain — so we offer a very structured increase in activity, where we’re retraining the brain to soothe the central nervous system to not process these signals as danger.”

The program is not cheap. It costs $37,000 to $42,000 for three weeks. Gatchel, who is affiliated with several pain management programs in Texas, said that price is an “exception to the norm,” and less intense versions of the treatment can run between $4,000 and $10,000.

Studies suggest the programs can be quite effective. The Brooks Rehabilitation pain program in Jacksonville, Fla., reports that six months after intensive treatment, 9 out of 10 patients reported improvements in quality of life and 3 out of 4 felt decreased levels of pain. The center reported that just 6 percent of patients who were weaned off of opioids during the program resumed taking them afterwards.

Another study, published in the journal Pain, followed 339 patients who completed a three-week program at the Mayo Clinic Pain Rehabilitation Center. Some 70 percent responded to questionnaires six months after their treatment, and reported significant improvements in pain severity, depression, social functioning, and other metrics. The gains held both for patients who had been taking opioids before entering the center and for those who had not.

A renewed interest in alternative treatments

Such results, and anecdotal reports from patients like White, have spurred renewed interest in biopsychosocial pain management.

Insurance coverage is still a battle: Many plans will pay for medical treatments such as surgeries, pills, and steroid injections that can run $2,000 apiece. They’re not as keen to cover therapy, massage, and meditation. “It’s much more efficient for insurers to pay for a pill in a 15-minute office visit,” Twillman said, “instead of a pill, plus a psychologist, plus a chiropractor, plus acupuncture, plus yoga and massage.”

Slowly though, that’s changing, in large part because of the opioid crisis.

The Food and Drug Administration just changed its provider education guidelines to urge doctors to learn about alternative strategies for managing pain.

Some insurers are open to new approaches, too. Oregon’s Medicaid system, for instance, recently began covering more physical therapy and chiropractor visits for people with back pain, so as to help them avoid painkillers and surgery. Cigna, too, has increased its coverage for back pain physical therapy.

Veterans Affairs, meanwhile, is taking steps to reach out early to chronic pain patients, often through their primary care physicians, to coax them into increasing physical activity, sitting through cognitive behavior therapy, and meditating.

Sperry has appealed to Congress to accelerate the shift with more funding — not just for chronic pain and addiction treatment, but also for medical education. She conducts a workshop at Mayo Clinic to teach medical students how to say no to patients who ask for opioid refills — and how to help them instead train their brains to manage their chronic pain.

“We need a cultural shift,” Sperry said. “There’s an implication that there’s a pill for everything — and that’s not accurate. It’s very dangerous.”

Amazon might be taking another shot at building its own smartphones, according to a new report from NDTV’s Gadget 360. And unlike the company’s failed Fire Phone, the new smartphones — allegedly branded “Ice” — would have access to the full line of Google services and apps, including the Play Store.

The report notes that the Ice line would be targeted at emerging markets like India, instead of the more US-facing focus of the original Fire Phone. As such, the rumored specs for one of the Ice phones fall in line with that goal: a screen between 5.2 and 5.5 inches, 2GB of RAM, 16GB of storage, a rear-mounted fingerprint sensor, and a Snapdragon 435 processor. Gadget 360 claims that this specific model would retail for roughly $93 at launch, although that price could change before launch.

Per Gadget 360’s source, the Ice phone doesn’t have Alexa support, but, like the price, details aren’t finalized which means software, too, could change in the future.

The original Fire Phone was a failure for a multitude of reasons, including a focus on gimmicky features, an expensive price tag, exclusivity to AT&T, and a bizarre operating system. It sold extremely poorly, even with Amazon slashing prices only weeks after launch. But perhaps the biggest failure was the dearth of apps by not using Google Mobile Services, in essence leaving customers with a confusingly skinned Android phone that didn’t really run any Android apps. It sounds like Amazon is taking a different approach with the Ice line, by embracing Google’s services and targeting a different slice of the smartphone market than the Apple-and-Samsung-dominated high-end segment.

If the Ice rumors are true, Amazon could have a big opportunity in emerging market places. Google has made no secret of its desire to put Android phones in the hands of “the next billion users” with initiatives like Android Go and Android One; an Amazon hardware push would certainly go a long way toward getting Google there, while also boosting Amazon’s own user base of its own products.

Bitcoin is really cool. Sure, there's arguments to be made about whether it's a useful technology, whether we're currently in a cryptocurrency bubble or if the governance problems that it's currently facing will ever be resolved.. But on a purely technical level, the mystical Satoshi Nakamoto created an impressive technology.

Unfortunately, while there's a lot of resources out there that give high level explanations of how Bitcoin works (one such resource I'd highly recommend is Anders Brownworth's fantastic blockchain visual 101 video), there isn't a whole heap of information at a lower level and, in my opinion, there's only so much you can properly grok if you're looking at the 10000 ft view.

As someone relatively new to the space, I found myself hungry to understand the mechanics of how Bitcoin works. Luckily, because Bitcoin is decentralised and peer to peer by its nature, anyone is able to develop a client that conforms to the protocol. In order to get a greater appreciation of how Bitcoin works, I decided to write my own small toy Bitcoin client that was able to publish a transaction to the Bitcoin blockchain.

This post walks through the process of creating a minimally viable Bitcoin client that can create a transaction and submit it to the Bitcoin peer to peer network so that it is included in the Blockchain. If you'd rather just read the raw code, feel free to check out my Github repo.

Address generation

In order to be part of the Bitcoin network, it's necessary to have an address from which you can send and receive funds. Bitcoin uses public key cryptography and an address is basically a hashed version of a public key that has been derived from a secret private key. Surprisingly, and unlike most public key cryptography, the public key is also kept secret until funds are sent from the address - but more on that later.

A quick aside on terminology: In Bitcoin, the term "wallet" is used by clients to mean a collection of addresses. There's no concept of wallets at a protocol level, only addresses.

Bitcoin uses elliptic curve public-key cryptography for its addresses. At an ultra high level, elliptic curve cryptography is used to generate a public key from a private key, in the same way RSA would but with a lower footprint. If you're interested in learning a bit about the mathematics behind how this works, Cloudflare's primer is a fantastic resource.

Starting with a 256 bit private key, the process of generating a Bitcoin address is shown below:

In Python, I use the ecsda library to do the heavy lifting for the elliptic curve cryptography. The following snippet gets a public key for the highly memorable (and highly insecure) private key 0xFEEDB0BDEADBEEF (front padded with enough zeros to make it 64 hex chars long, or 256 bits). You'd want a more secure method of generating private keys than this if you wanted to store any real value in an address!

As an amusing aside, I originally created an address using the private key 0xFACEBEEF and sent it 0.0005 BTC.. 1 month later and someone had stolen my 0.0005 BTC! I guess people must occasionally trawl through addresses with simple/common private keys. You really should use proper key derivation techniques!

fromecdsaimportSECP256k1,SigningKeydefget_private_key(hex_string):returnbytes.fromhex(hex_string.zfill(64))# pad the hex string to the required 64 charactersdefget_public_key(private_key):# this returns the concatenated x and y coordinates for the supplied private address# the prepended 04 is used to signify that it's uncompressedreturn(bytes.fromhex("04")+SigningKey.from_string(private_key,curve=SECP256k1).verifying_key.to_string())private_key=get_private_key("FEEDB0BDEADBEEF")public_key=get_public_key(private_key)

Running this code gets the private key (in hex) of

0000000000000000000000000000000000000000000000000feedb0bdeadbeef

And the public key (in hex) of

04d077e18fd45c031e0d256d75dfa8c3c21c589a861c4c33b99e64cf613113fcff9fc9d90a9d81346bcac64d3c01e6e0ef0828543edad73c0e257b845812cc8d28

The 0x04 that prepends the public key signifies that this is an uncompressed public key, meaning that the x and y coordinates from the ECDSA are simply concatenated. Because of the way ECSDA works, if you know the x value, the y value can only take two values, one even and one odd. Using this information, it is possible to express a public key using only one of x and the polarity of y. This reduces the public key size from 65 bits to 33 bits and the key (and subsequent computed address) are referred to as compressed. For compressed public keys, the prepended value will be 0x02 or 0x03 depending on the polarity of y. Uncompressed public keys are most commonly used in Bitcoin, so that's what I'll use here too.

From here, to generate the Bitcoin address from the public key, the public key is sha256 hashed and then ripemd160 hashed. This double hashing provides an extra layer of security and a ripemd160 hash provides a 160 bit hash of sha256's 256 hash, shortening the length of the address. An interesting result of this is that it is possible for two different public keys to hash to the same address! However, with 2^160 different addresses, this isn't likely to happen any time soon.

importhashlibdefget_public_address(public_key):address=hashlib.sha256(public_key).digest()h=hashlib.new('ripemd160')h.update(address)address=h.digest()returnaddresspublic_address=get_public_address(public_key)

This generates a public address of c8db639c24f6dc026378225e40459ba8a9e54d1a. This is sometimes referred to as the hash 160 address.

As alluded to before, an interesting point is that both the conversion from private key to public key and the conversion from public key to public address are one way conversions. If you have an address, the only way to work backwards to find the associated public key is to solve a SHA256 hash. This is a little different to most public key cryptography, where your public key is published and your private key hidden. In this case, both public and private keys are hidden and the address (hashed public key) is published.

Public keys are hidden for good reason. Although it is normally infeasible to compute a private key from the corresponding public key, if the method of generating private keys has been compromised then it having access to the public key makes it a lot easier to deduce the private key. In 2013, this infamously occurred for Android Bitcoin wallets. Android had a critical weakness generating random numbers, which opened a vector for attackers to find private keys from public keys. This is also why address reuse in Bitcoin is encouraged as to sign a transaction you need to reveal your public key. If you don't reuse an address after sending a transaction from the address, you don't need worry about the private key of that address being exposed.

That standard way of expressing a Bitcoin address is to use the Base58Check encoding of it. This encoding is only a representation of an address (and so can be decoded/reversed). It generates addresses of the form 1661HxZpSy5jhcJ2k6av2dxuspa8aafDac. The Base58Check encoding provides a shorter address to express and also has an inbuilt checksum, that allows detection of mistyped address. In just about every Bitcoin client, the Base58Check encoding of your address is the address that you'll see. A Base58Check also includes a version number, which I'm setting to 0 in the code below - this represents that the address is a pubkey hash.

# 58 character alphabet usedBASE58_ALPHABET='123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz'defbase58_encode(version,public_address):"""    Gets a Base58Check string    See https://en.bitcoin.it/wiki/base58Check_encoding    """version=bytes.fromhex(version)checksum=hashlib.sha256(hashlib.sha256(version+public_address).digest()).digest()[:4]payload=version+public_address+checksumresult=int.from_bytes(payload,byteorder="big")print(result)# count the leading 0spadding=len(payload)-len(payload.lstrip(b'\0'))encoded=[]whileresult!=0:result,remainder=divmod(result,58)encoded.append(BASE58_ALPHABET[remainder])returnpadding*"1"+"".join(encoded)[::-1]bitcoin_address=base58_encode("00",public_address)

After all of that, starting with my private key of feedb0bdeadbeef (front padded with zeros), I've arrived with a Bitcoin address of 1KK2xni6gmTtdnSGRiuAf94jciFgRjDj7W!

With an address, it's now possible to get some Bitcoin! To get some Bitcoin into my address, I bought 0.0045 BTC (at the time of writing, around $11 USD) from btcmarkets using Australian dollars. Using btcmarket's trading portal, I transferred it to the above address, losing 0.0005 BTC to transaction fees in the process. You can see this transaction on the blockchain in transaction 95855ba9f46c6936d7b5ee6733c81e715ac92199938ce30ac3e1214b8c2cd8d7.

Connecting to the p2p network

Now that I have an address with some Bitcoin in it, things get more interesting. If I want to send that Bitcoin somewhere else, it's necessary to connect to the Bitcoin peer to peer network.

Bootstrapping

One of the sticking points I had when first learning about Bitcoin was, given the decentralised nature of the network, how do peers of the network find other peers? Without a centralised authority, how does a Bitcoin client know how to bootstrap and start talking to the rest of the network?

As it turns out, idealism submits to practicality and there is the slightest amount of centralisation in the initial peer discovery process. The principle way for a new peer to find peers to connect is to use a DNS lookup to any number of "DNS seed" servers that are maintained by members of the Bitcoin community.

It turns out DNS is well suited to this purpose of bootstrapping clients as the DNS protocol, which runs over UDP and is lightweight, is hard to DDoS. IRC was used as a previous bootstrapping method but was discontinued to its weakness to DDoS attacks.

The seeds are hardcoded into Bitcoin core's source code but are subject to change by the core developers.

The Python code below connects to a DNS seed and prints out a list of peers that I'm able to connect to. Using the socket library, this basically performs a nslookup and returns the ipv4 address of the first result on running a query against the seed node seed.bitcoin.sipa.be.

importsocket# use a dns request to a seed bitcoin DNS server to find a nodenodes=socket.getaddrinfo("seed.bitcoin.sipa.be",None)# arbitrarily choose the first nodenode=nodes[0][4][0]

The address is 208.67.251.126 which is a friendly peer that I can connect to!

Saying hi to my new peer friend

Bitcoin connections between peers are through TCP. Upon connecting to a peer, the beginning handshake message of the Bitcoin protocol is a Version message. Until peers swap Version messages, no other messages will be accepted.

Bitcoin protocol messages are well documented in the Bitcoin developer reference. Using the developer reference as guide, the version message can be constructed in Python as the snippet below shows. Most of the data is fairly uninteresting, administrative data used to open a connection to the peer. If you're interested in more details than the attached comments provide, check the developer reference.

version=70014services=1# not a full node, cant provide any datatimestamp=int(time.time())addr_recvservices=1addr_recvipaddress=socket.inet_pton(socket.AF_INET6,"::ffff:127.0.0.1")#ip address of receiving node in big endianaddr_recvport=8333addr_transservices=1addr_transipaddress=socket.inet_pton(socket.AF_INET6,"::ffff:127.0.0.1")addr_transport=8333nonce=0user_agentbytes=0start_height=329167relay=0

Using Python's struct library the version payload data is packed into the right format, paying special attention to endianness and byte widths of the data. Packing the data into the right format is important, or else the receiving peer won't be able to understand the raw bytes that it receives.

payload=struct.pack("<I",version)payload+=struct.pack("<Q",services)payload+=struct.pack("<Q",timestamp)payload+=struct.pack("<Q",addr_recvservices)payload+=struct.pack("16s",addr_recvipaddress)payload+=struct.pack(">H",addr_recvport)payload+=struct.pack("<Q",addr_transservices)payload+=struct.pack("16s",addr_transipaddress)payload+=struct.pack(">H",addr_transport)payload+=struct.pack("<Q",nonce)payload+=struct.pack("<H",user_agentbytes)payload+=struct.pack("<I",start_height)

Again, the way in which this data can be found is available in the developer reference. Finally, each payload transmitted on the Bitcoin network needs to be prepended with a header, that contains the length of the payload, a checksum and the type of message the payload is. The header also contains the magic constant 0xF9BEB4D9 which is set for all mainnet Bitcoin messages. The following function gets a Bitcoin message with header attached.

defget_bitcoin_message(message_type,payload):header=struct.pack(">L",0xF9BEB4D9)header+=struct.pack("12s",bytes(message_type,'utf-8'))header+=struct.pack("<L",len(payload))header+=hashlib.sha256(hashlib.sha256(payload).digest()).digest()[:4]returnheader+payload

With the data packed into the right format, and the header attached, it can be sent off to our peer!

s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)s.connect((node,8333))s.send(get_bitcoin_message("version",payload))print(s.recv(1024))

The Bitcoin protocol mandates that on receiving a version message, a peer should respond with a Verack acknowledgement message. Because I'm building a tiny "for fun" client, and because peers won't treat me differently if I don't, I'll disregard their version message and not send them the acknowledgement. Sending the Version message as I connect is enough to allow me to later send more meaningful messages.

Running the above snippet prints out the following. It certainly looks promising - "Satoshi" and "Verack" are good words to see in the dump out! If my version message had been malformed, the peer would not have responded at all.

b'\xf9\xbe\xb4\xd9version\x00\x00\x00\x00\x00f\x00\x00\x00\xf8\xdd\x9aL\x7f\x11\x01\x00\r\x00\x00\x00\x00\x00\x00\x00\xddR1Y\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xcb\xce\x1d\xfc\xe9j\r\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\xb8>*\x88@I\x8e\x10/Satoshi:0.14.0/t)\x07\x00\x01\xf9\xbe\xb4\xd9verack\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00]\xf6\xe0\xe2'

Bitcoin transactions

To transfer Bitcoin it's necessary to broadcast a transaction to the Bitcoin network.

Critically, the most important idea to understand is that the balance of a Bitcoin address is comprised solely by the number of "Unspent Transaction Outputs" (UTXO) that an address can spend. When Bob sends a Bitcoin to Alice, he's really just creating a UTXO that Alice (and only Alice) can later use to create another UTXO and send that Bitcoin on. The balance of a Bitcoin address is therefore defined by the amount of Bitcoin it is able to transfer to another address, rather than the amount of Bitcoin it has.

To emphasize, when someone says that they have X bitcoin, they're really saying that all of the UTXOs that they can spend sum to X bitcoin of value. The distinction is subtle but important, the balance of a Bitcoin address isn't recorded anywhere directly but rather can be found by summing the UTXOs that it can spend. When I came to this realisation it was definitely a "oh, that's how it works!" moment.

A side effect of this is that transaction outputs can either be unspent (UTXO) or completely spent. It isn't possible to only spend half of an output that someone has spent to you and then spend the rest at a later time. If you do want to spend a fraction of an output that you've received, you instead can send the fraction that you want to send while sending the rest back to yourself. A simplified version of this is shown in the diagram below.

When a transaction output is created, it is created with a locking condition that will allow someone in the future to spend it, through what are called transaction scripts. Most commonly, this locking condition is: "to spend this output, you need to prove that you own the private key corresponding to a particular public address". This is called a "Pay-to-Public-Key-Hash" script. However, through Bitcoin script other types of conditions are possible. For example, an output could created that could be spent by anyone that could solve a certain hash or a transaction could be created that anyone could spend.

Through Script, it's possible to create simple contract based transactions. Script is a basic stack based language with number of operations centred around checking equality of hashes and verifying signatures. Script is not Turing complete and does not have the ability to have any loops. The competing cryptocurrency Ethereum built on this to be able to have "smart contracts", which does have a Turing complete language. There's much debate about the utility, necessity and security of including a Turing complete language in cryptocurrencies but I'll leave that debate to others!

In standard terminology, a Bitcoin transaction is made up of inputs and outputs. An input is a UTXO (that is now being spent) and an output is a new UTXO. There can be multiple outputs for a single input but an input needs to be completely spent in a transaction. Any part of an input leftover is claimed by miners as a mining fee.

For my toy client I want to be able to send on the Bitcoin previously transferred from an exchange to my FEEDB0BDEADBEEF address. Using the same process as before, I generated another address using the private key (before padding of) BADCAFEFABC0FFEE. This generated the address 1QGNXLzGXhWTKF3HTSjuBMpQyUYFkWfgVC.

Creating a raw transaction

Creating a transaction is a matter of first packing a "raw transaction" and then signing the raw transaction. Again, the developer reference has a description of what goes into a transaction. What makes up a transaction is shown below but a few notes first:

• Common Bitcoin parlance uses the terms signature script and pubkey script which I find a little confusing. The signature script is used to meet the conditions of the UTXO that we want to use in the transaction and the pubkey script is used to set the conditions that need to be met to spend the UTXO we are creating. A better name for the signature script might be a unlocking script and a better name for the pubkey script might be a locking script.

• Bitcoin transaction values are specified in Satoshis. A Satoshi represents the smallest divisible part of a Bitcoin and represents one hundred millionth of a Bitcoin.

For simplicity, what is shown below is for a transaction for one output and one input. More complex transactions, with multiple inputs and outputs are possible to create in the same way.

Ignoring the signature script and pubkey script for now, it's quite easy to see what should go in the other fields of the raw transaction. To send the funds in my FEEDB0BDEADBEEF address to my BADCAFEFABC0FFEE address, I look at the transaction that was created by the exchange. This gives me:

• The transaction id is 95855ba9f46c6936d7b5ee6733c81e715ac92199938ce30ac3e1214b8c2cd8d7.
• The output that was sent to my address was the second output, output 1 (output numbers are 0 indexed).
• Number of outputs is 1, as I want to send everything in FEEDB0BDEADBEEF to BADCAFEFABC0FFEE.
• Value can be at most 400,000 Satoshis. It must be less than this to allow some fees. I'll allow 20,000 Satoshi to be taken as a fee so will set value to 380,000.
• Lock time will be set to 0, this allows the transaction to be included at any time or block.

For the Pubkey script of our transaction, we use a Pay to Pubkey hash (or p2pk) script. The script ensures that only the person that holds the public key that hashes to the provided Bitcoin address is able to spend the created output and that a supplied signature has been generated by the person that holds the corresponding private key to the public key.

To unlock a transaction that has been locked by a p2pk script, the user provides their public key and a signature of the hash of the raw transaction. The public key is hashed and compared to the address that the script was created with and the signature is verified for the supplied public key. If the hash of the public key and the address are equal, and the signature is verified, the output can be spent.

In Bitcoin script operands, the p2pk script looks as follows.

OP_DUP
OP_HASH160<Length of address in bytes><Bitcoin address>
OP_EQUALVERIFY
OP_CHECKSIG

Converting the operands to their values (these can be found on the wiki) and inputting the public address (before it has been Base58Check encoded) gives the following script in hex:

0x76
0xA9
0x14
0xFF33195EC053D6E58D5FD3CC67747D3E1C71B280
0x88
0xAC

The address is found using the earlier shown code for deriving an address from a private key, for the private key we're sending to, 0xBADCAFEFABC0FFEE.

Signing the transaction

There are two separate, but somewhat related, uses for the signature script in a (p2pk) transaction:

• The script verifies (unlocks) the UTXO that we are are trying to spend, by providing our public key that hashes to the address that the UTXO has been sent
• The script also signs the transaction that we are submitting to the network, such that nobody is able to modify the transaction without invalidating the signature

However, the raw transaction contains the signature script which should contain a signature of the raw transaction! This chicken and egg problem is solved by placing the Pubkey script of the UTXO we're using in the signature script slot prior to signing the raw transaction. As far as I could tell, there doesn't seem to be any good reason for using the Pubkey as the placeholder, it could really be any arbitrary data.

Before the raw transaction is hashed, it also needs to have a Hashtype value appended. The most common hashtype value is SIGHASH_ALL, which signs the whole structure such that no inputs or outputs can be modified. The linked wiki page lists other hash types, which can allow combinations of inputs and outputs to be modified after the transaction has been signed.

The below functions put together a python dictionary of raw transaction values.

defget_p2pkh_script(pub_key):"""    This is the standard 'pay to pubkey hash' script    """# OP_DUP then OP_HASH160 then 20 bytes (pub address length)script=bytes.fromhex("76a914")# The address to pay toscript+=pub_key# OP_EQUALVERIFY then OP_CHECKSIGscript+=bytes.fromhex("88ac")returnscriptdefget_raw_transaction(from_addr,to_addr,transaction_hash,output_index,satoshis_spend):"""    Gets a raw transaction for a one input to one output transaction    """transaction={}transaction["version"]=1transaction["num_inputs"]=1# transaction byte order should be reversed:# https://bitcoin.org/en/developer-reference#hash-byte-ordertransaction["transaction_hash"]=bytes.fromhex(transaction_hash)[::-1]transaction["output_index"]=output_index# temporarily make the signature script the old pubkey script# this will later be replaced. I'm assuming here that the previous# pubkey script was a p2pkh script heretransaction["sig_script_length"]=25transaction["sig_script"]=get_p2pkh_script(from_addr)transaction["sequence"]=0xfffffffftransaction["num_outputs"]=1transaction["satoshis"]=satoshis_spendtransaction["pubkey_length"]=25transaction["pubkey_script"]=get_p2pkh_script(to_addr)transaction["lock_time"]=0transaction["hash_code_type"]=1returntransaction

Calling the code with the following values creates the raw transaction that I'm interested in making.

private_key=address_utils.get_private_key("FEEDB0BDEADBEEF")public_key=address_utils.get_public_key(private_key)from_address=address_utils.get_public_address(public_key)to_address=address_utils.get_public_address(address_utils.get_public_key(address_utils.get_private_key("BADCAFEFABC0FFEE")))transaction_id="95855ba9f46c6936d7b5ee6733c81e715ac92199938ce30ac3e1214b8c2cd8d7"satoshis=380000output_index=1raw=get_raw_transaction(from_address,to_address,transaction_id,output_index,satoshis)

It might be confusing to see that I'm using a private key to generate the to_address. This is really only done for convenience and to show how the to_address is found. If you were making a transaction to someone else, you'd ask them for their public address and transfer to that, you wouldn't need to know their private key.

In order to be able to sign, and eventually transmit the transaction to the network, the raw transaction needs to be packed appropriately. This is implemented in the get_packed_transaction function which I won't replicate here, as it's essentially just more struct packing code. If you're interested you can find it in the bitcoin_transaction_utils.py Python file in my Github repo.

This allows me to define a function that will produce the signature script. Once the signature script is generated, it should replace the placeholder signature script.

defget_transaction_signature(transaction,private_key):"""    Gets the sigscript of a raw transaction    private_key should be in bytes form    """packed_raw_transaction=get_packed_transaction(transaction)hash=hashlib.sha256(hashlib.sha256(packed_raw_transaction).digest()).digest()public_key=address_utils.get_public_key(private_key)key=SigningKey.from_string(private_key,curve=SECP256k1)signature=key.sign_digest(hash,sigencode=util.sigencode_der)signature+=bytes.fromhex("01")#hash code typesigscript=struct.pack("<B",len(signature))sigscript+=signaturesigscript+=struct.pack("<B",len(public_key))sigscript+=public_keyreturnsigscript

Essentially, the signature script is provided as an input to the pubkey script of the previous transaction I'm trying to use, so that I can prove I am allowed to spend the output that I'm now using as an input. The mechanics of how this works is shown below, which is taken from the Bitcoin wiki. Working from top to bottom, each row is a one iteration of the script. This is for a pay to pubkey hash pubkey script, which, as mentioned earlier is the most common script. It is also the script that both the transaction I'm creating and the transaction I'm redeeming use.

This script will fail if the provided public key doesn't hash to the public key hash in the script or if the provided signature doesn't match the provided public key. This ensures that only the person that holds the private key for the address in the pubkey script is able to spend the output.

You can see that here is the first time I have needed to provide my public key anywhere. Up until this point, only my public address has been published. It's necessary to provide the public key here as it is allows verification of the signature that the transaction has been signed with.

Using the get_transaction_signature function, we can now sign and pack our transaction ready for transmission! This involves replacing the placeholder signature script with the real signature script and removing the hash_code_type from the transaction as shown below.

signature=get_transaction_signature(raw,private_key)raw["sig_script_length"]=len(signature)raw["sig_script"]=signaturedelraw["hash_code_type"]transaction=get_packed_transaction(raw)

Publishing the transaction

With the transaction packed and signed, it's a matter of telling the network about it. Using a few functions previously defined in this article in bitcoin_p2p_message_utils.py, the below piece of code puts the Bitcoin message header on the transmission and transmits it to a peer. As mentioned earlier, it's first necessary to send a version message to the peer so that it accepts subsequent messages.

s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)s.connect((get_bitcoin_peer(),8333))s.send(get_bitcoin_message("version",get_version_payload())s.send(get_bitcoin_message("tx",transaction)

Sending the transaction was the most annoying part of getting this to work. If I submitted a transaction that was incorrectly structured or signed, the peer often just dropped the connection or, in a slightly better case, sent cryptic error messages back. One such (very annoying) error message was "S value is unnecessarily high" which was caused by signing the transaction hash using the ECSDA encoding method of sigencode_der. Despite the signature being valid, apparently Bitcoin miners don't like ECSDA signatures formatted in such a way that allows spam in the network. The solution was to use the sigencode_der_canonize function which takes care to format the signatures in the other format. A simple, but extraordinarily hard to debug, issue!

In any case, I eventually got it to work and was very excited when I saw that my transaction made its way into the blockchain!! It was a great feeling of accomplishment knowing that my small, dinky, hand crafted transaction will now forever be a part of Bitcoin's ledger.

When I submitted the transaction, my transaction fee was actually quite low compared to the median (I used the bitcoin fees website to check) and as such it took around 5 hours for a miner to decide to include it in a block. I checked this by looking at the number of confirmations the transaction had - this is a measure of how many blocks deep the block with the transaction is in. At the time of writing this was at 190 confirmations.. meaning that after the block my transaction is in, there's another 190 blocks. This can be pretty safely considered confirmed, as it would take an impressive attack on the network to rewrite 190 blocks to remove my transaction.

Conclusion

I hope you've gained some appreciation of how Bitcoin works through reading this article, I know I certainly did during the months it took me to put all of this together! While most of the information presented here isn't too practicably applicable - you'd normally just use a client that does it all for you - I think having a greater understanding of how things work gives you a better appreciation of what's happening under the covers and gives makes you a more confident user of the technology.

If you'd like to peruse the code, or play around further with the toy examples, please check out my associated Github repo. There's a lot of room to explore further in the Bitcoin world, I've only really looked at a very common use case of Bitcoin. There's certainly room out there to do cooler feats than just transferring value between two addresses! I also didn't touch how mining, the process of adding transactions to the blockchain, works.. which is another rabbit hole all together.

If you've read this far you might have realised that the 380000 Satoshi (or 0.0038 BTC) that I transferred into 1QGNXLzGXhWTKF3HTSjuBMpQyUYFkWfgVC can, with enough smarts, be taken by anyone.. as the private key for the address exists within this article. I'm very interested to see how long it takes to be transferred away and hope that whoever takes it has the decency to do so using some of the techniques I've detailed here! It'd be pretty lame if you just loaded the private key into a wallet app to take it, but I guess I can't stop you! At the time of writing this amount is worth about $10 USD, but if Bitcoin "goes to the moon" who knows how much it might be worth!

And just in case you're looking for an address to send Bitcoin to when you're playing around with this stuff, or if you think this post was valuable enough to warrant a tip - my address of 18uKa5c9S84tkN1ktuG568CR23vmeU7F5H is happy to take any small donations! Alternatively, if you want to yell at me about getting anything wrong, I'd love to hear it.

Further resources

If you found this article interesting, some further resources to check out:

• Mastering Bitcoin is a book that explains the technical details of Bitcoins. I haven't read this completely, but on a skim it looks like it is a wealth of good information.
• Ken Sheriff's blog article is a great source of information that covers a lot of the same topics as this article that I unfortunately only found this when I was nearly finished writing this article. If you didn't understand anything in this article, reading Ken's excellent post would be a good place to start.
• Mentioned earlier, but Anders Brownworth's fantastic blockchain visual 101 video is an excellent top level view of how blockchain technologies work.
• Unless you're a masochist for pain, I'd also recommend not doing everything from scratch unless you're interested in doing so for learning purposes. The pycoin library is a Python Bitcoin library that will save you a few headaches.
• To also save yourself pain, it's probably advisable to use the Bitcoin testnet to play around with, rather than using the mainnet like I did. That said, it's more fun when the risk of your code being wrong is losing real money!
• Lastly, it is probably worth repeating that the accompanying code for this article can be found in my Github repo.

Looking for a manual online? ManualsLib is here to help you save time spent on searching.

Our database consists of more than 2613321 pdf files and becomes bigger every day! Just enter the keywords in the search field and find what you are looking for! Search results include manual name, description, size and number of pages. You can either read manual online or download it to your computer. Moreover, documents can be shared on social networks. Welcome!

Nobody wants to deliver late, over-budget software. I don’t know a single software developer who wakes up in the morning and thinks “I’d like to do a rubbish job today. How can I cost my employer more money?” And yet, so many software projects don’t go well. And with every new project, there seems to be more and more pressure to go faster. So, if we’re in the business of making software, what do we do? How do we go faster without compromising quality?

Despite more than 50 years of history and countless methodologies, advice and books, IT projects keep failing.

—Susan Moore [1]

Now, I’m not writing here as some kind of expert. I have never run my own software company. I am not delivering wisdom distilled from copious academic studies or controlled experiments. I am writing this to organise my own thoughts as I try to make sense of what I see going on around me.

To think this through properly, we need to start with why. What is the point of all this software production? Why are we even making software in the first place? Let’s leave open source as the elephant in the room for the moment and talk about commercial software. Let’s start with business.

Business is about reducing customer pain.

As I understand it, to run a successful business, we first find something that causes people pain. It might be a metaphorical or literal pain (though usually, it is metaphorical). Then, we offer a way to reduce that pain in exchange for money. For example, people find it difficult (painful) to learn to code. So, there is a market for books and classes that teach programming. Some people dislike their physical appearance. So, there are whole industries in exercise, cosmetics, beauty therapy, and so on. A business delivers value to customers to the degree that they reduce that customer pain (or the perception of it). And if people are confident that we can reduce their pain, then they will be glad to pay us money. 

In a software product business, software is the thing we offer to reduce customer pain. In this type of business, software development is the key activity delivering value. Customers buy (or subscribe to) the product, and software development creates it. Of course, this applies only to product businesses. If we are selling consulting services, or IT as a support function, then things are different. But where the core business is a software product, then development is what gets it done.

This is not to say that development is the only activity that adds value. For example, if nobody knows our product exists, then it may as well not exist. So sales and marketing activities are essential. We also have to make sure that our product actually addresses real customer pain points. If not, we’re wasting our time. So market research (whether formal or ad-hoc) is also vital. We also need user experience (UX) and graphic design activities to reduce friction. Friction in our product gets in the way of solving our customers’ problems. All these activities (marketing, sales, market research, UX, design) are important. And if you squint a little, then they all start to look similar. They’re like facets of the same core activity: understanding people. But, in the end, all these activities deliver only plans and promises for customer value. It is software development that turns the plans and promises into a product.[2]

It all works better when you embrace the idea that “product,” “design,” and “engineering” are just different perspectives on the same thing.

— Greg Veen [3]

Minimising lead time to business impact

If we’re doing all this “understanding people” stuff right, then it’s an ongoing activity. As we go, we learn more about the problems we’re trying to solve. So we start devising better solutions. So we need the software product we’re creating to change as well. For this to work, we need a nimble development team. A team that is able to deliver value rapidly, and able to respond quickly to change. This is the core goal of software development practice. As Dan North puts it:

“The goal of software development is sustainably minimising lead time to business impact”

—Dan North [4]

So, having a nimble development team is important. But how do you get a nimble development team? Do you:

• Pay your developers like kings?
• Buy them super-fast, expensive computers?
• Send them to whatever crazy tech conferences they want to attend?

We could make a good argument for any of these things. If you want to keep your nimble development team, then give serious thought to each one. Fast computers and good tech conferences will improve developer performance. This investment will pay off over time. But these things are more relevant to retaining good developers. We want to think about building a nimble team.

So if the answer is not giving developers whatever they want, what do we do then? The short answer is, ask the developers. But, ask them at the right time, in the right way. The thing to understand about developers is that they tend to be natural problem solvers. Good developers like their jobs. They like their jobs because they get to solve interesting complex puzzles all day and get paid for it. Good developers revel in taking complex challenges and finding elegant solutions. So they should be able to come up with great ideas for becoming more nimble. But many organisations encourage developers to focus on the wrong problems. This encouragement may be neither deliberate nor conscious, but it happens nonetheless.

Focussing on the wrong problems

How does this happen? How do we end up asking developers to focus on the wrong problems, without even knowing we’re doing it? This happens because we distance developers from the customers. As soon as a project gets to be any reasonable size, we bring in project managers and business analysts.[5] And we bring these people in for a very good reason—developers can’t do everything. Software projects are complicated. The code is complicated enough, but on top of that, there’s all the work of deciding what to build, planning the development phases, structuring the roll-out and deployment plans, liaising with customers… the list goes on. The developers have enough to worry about with the code. So we need these extra people to help.

But, what happens is, these extras become the developers’ interface to the world. The project manager and business analyst mediate communication with external stakeholders. The project manager, in particular, cares about delivering the project. Project managers report to management. And management cares about:

• How much is it going to cost?
• How long is it going to take?
• Why is it costing so much?
• Why is the project so late?
• Why isn’t it finished already?
• My goodness, we’re burning through how much per day on this late project?!

It’s understandable then, that project managers become focussed on predictability. They want plans, structure, estimates. They want to know what is happening and when. Predictability and measurement allow them to sound competent when they’re reporting to management. So they talk to developers about estimates and reports and deadlines. So then, developers begin to focus on deadlines and reports and estimates. They focus on estimation and predictability to keep the project manager happy.

But there’s an unfortunate issue with this. The issue is that estimation and predictability are impossible problems to solve. Every time a developer starts a new task they face an uncomfortable reality. Any given task may or may not contain an enormous sinkhole of hidden complexity. We hope the task is simple. But it might not be. You never know. And then Hofstadter’s law comes into play:

Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law.

—Douglas Hofstadter [6]

Consider this scenario: A project manager asks an inexperienced developer for an estimate. The inexperienced developer gives an estimate they think is reasonable. And then the project manager goes away and turns that into a deadline and a plan. A good project manager will even add a little ‘fat,’ to be on the safe side. But then the inevitable happens—the project falls behind. So the developer starts working longer hours to meet the deadline. But working longer hours means the developer gets tired. They start making more mistakes. And it’s still not enough. The project is still behind. The project manager is demanding to know what’s taking so long. So the harried developer starts cutting corners. And in the process, they start shipping bugs. So now the product is not only late, but also buggy.

This situation delivers negative customer value. Sure the late, buggy product might still solve some amount of customer pain. But the bugs introduce new pain and take time to fix. The customer loses confidence in our ability to help them. This makes them less inclined to pay us money. Everybody loses.

Experienced developers know the estimates game is rigged, so they try hard not to play it. Imagine, a project manager comes to an experienced developer asking for estimates. What they do is give a number that is large enough to sound ridiculous. But also small enough that the project won’t be cancelled immediately. Next, the project manager (or salesperson) comes back to challenge this ridiculous figure. “That estimate seems a bit larger than we were hoping for. Is there any chance we could maybe squeeze things a little and bring that number down?” At this point, the experienced developer asks: “What kind of figure do we need to get down to?” The salesperson gives a number back. Then the experienced developer rubs her chin and says, “Well, it will be tight, but we’ll see what we can do. We’ll have to take some requirements out and deliver only the most basic features.” Then she estimates how little they can promise to deliver without appearing incompetent. And she commits to only that. This way, when she delivers much more than promised, everyone is happy. But even in this scenario, Hofstadter’s Law will still raise its ugly head. And soon enough we’re back to scrambling to meet deadlines and shipping buggy code.

Estimates are typically a necessary evil in software development. Unfortunately, people tend to assume that writing new software is like building a house or fixing a car, and that as such the contractor or mechanic involved should be perfectly capable of providing a reliable estimate for the work to be done in advance of the customer approving the work. […] With custom software, however, a great deal of the system is being built from scratch, and usually how it’s put together, how it ultimately works, and what exactly it’s supposed to do when it’s done are all moving targets. It’s hard to know when you’ll finish when usually the path you’ll take and the destination are both unknown at the start of the journey.

—Steve Smith [7]

My point here is not to complain about software estimation. Everyone knows it’s a necessary evil. But it’s a necessary evil that ends in a vicious cycle. We cut corners and ship poor quality code to meet the deadline. And all the while we assure each other that we’ll come back and fix it later. But ‘later’ never comes. We’re already behind on the next phase because we had to go back and fix those bugs. And we’re now building on top of brittle, hacked-together code that isn’t built for rapid change. And once locked in this cycle a developer’s focus shifts away from solving customer pain. Instead, they focus on problems like these:

• What’s the quickest possible way we can mark this feature ‘done’ and get the project manager off my back?
• How can I touch this brittle, fragile code as little as possible? Because the more I touch, the more likely it is to break.
• How can I eke out one tiny piece of code that I’m proud of amongst this giant steamy pile of technical debt?
• How can I get better at justifying my decisions to people who don’t have a clue about what I do or how complicated it is?
• How can I blame someone else when the customer starts complaining about the bugs I didn’t have time to fix?
• How can I get some good buzzwords on my CV so I can get another job somewhere where everything isn’t such a mess?

Now, no developer I’ve ever met wants to deliver late, buggy software. But we pressure developers to give short estimates because we want it sooner.[8] The developers comply because they want to please. But then they’re stuck because the estimates are always wrong. So they’re now under pressure to deliver. They want to please so they work longer hours, and cut corners. They compromise on quality because everyone is asking them ‘Is it done yet?’. But nobody is happy. The software is still late and buggy.

So, most developers I know, are doing the best they can. But they’re stuck. They’re too busy trying to catch up to even think about going ‘faster’.  And so, they focus on the wrong problems. They’re focussed on survival. It’s hard to focus on saving for retirement when you’re about to die of starvation. It’s also hard to work out how to work smarter when you’re working seven days a week on a late project. So the first step is to acknowledge that going faster requires investment. And if things are bad, it will need both a financial/time investment and an emotional investment.

Disrupting the cycle

Earlier, I suggested asking the developers how to reduce lead time to business impact. But, when developers are in ‘catch up’ mode we’re unlikely to get great responses from them. When we come into this environment and say, “How can we go faster?” we’ll likely get one of two types of response:

1. Burn it with fire.“We need to go away for two years and rewrite everything from scratch.” This happens when developers are completely overwhelmed by technical debt. So much so that they feel only way out is to declare bankruptcy. They may have a point, too. But at the same time, we may not have the budget to do that, and the market definitely won’t sit still while we rebuild.
2. Outrage.“We are going faster. I can’t believe you think you can fix this complicated issue with a half-hour brainstorm! How dare you?!” This happens when developers feel forced to ship poor quality code.  And then they feel they receive the blame when customers complain about bugs. And they may well be justified in their outrage. Developers in this mindset won’t help us at all until we can show that we hear them. They need to know we understand their concerns. We also need to show that we’re serious about changing things.

In both cases, the developer’s concerns are valid, but they are inward-focussed. We want to create a situation where everyone is working to minimise lead time to business impact. It won’t happen while developers are stuck in this mindset. Step zero is to show that we’re serious about changing things. That will usually involve finding some way to reduce pressure. Even if it’s only temporary.

But even then, unless something changes, developers will still be inward-focussed. They will have plenty of ideas on how to improve what they’re doing. Some of them might be great ideas. But there’s a lot of risk. We need the developers to focus on minimising lead time to business impact. We need to get their focus off dealing with internal pressures. We need to expose them to customer pain.

Exposing developers to customer pain

So, how then do you expose developers to customer pain? Plenty of other people have written at length on this, so I will only skim the surface. Here are three ideas in order of least-effective to most-effective:

1. Get developers to use the product they’re making as part of their day-to-day work. In the industry, this is known as drinking your own champagne or eating your own dog food. The advantage of doing this is that it turns developers into users of the product. So any glaring bugs or issues will now cause pain for the developers too. The problem with this approach is that developers aren’t typical users (most of the time). The way developers use software is often different from most customers. So, while this may help developers fix major bugs, it may not provide great insight into typical use cases. Also, this is not always practical. For example, imagine we’re producing a SaaS product for dental hygienists. It may be difficult to for developers to integrate thisinto their daily workflow.
2. Get developers to do rotations on support teams. A better approach is to encourage developers to take part in some kind of support roster for the product. (They may need quite strong encouragement.) This way developers get to experience customer pain first-hand. So, as they answer phone calls and email (or tweets, or whatever) customers tell them about problems. If developers do this long enough then they will also start to observe patterns of common issues. They’ll see things that come up over and over again. Not having to hear that same complaint again makes a good motivator to fix usability issues. Unfortunately, people rarely contact support to tell you what’s working well. So the feedback is somewhat biased.
3. Get developers to sit with and watch people using the software on a regular basis. This is the most inconvenient option as it requires the most organisation. But it is also likely to bring the best results. With this approach, developers get to see how real people use the software in real life to do real stuff. They get to see the good, the bad, and the ugly.

Doing these kinds of things with consistency is hard work. It takes effort and organisation. And most developers will have a natural disinclination for it. I feel awkward writing this because I don’t do this as often as I ought. But I believe it’s worth the effort.

Exposing developers to customer pain is an exercise of deliberate effort to overcome cognitive bias. Which is a long way of saying “it’s a way to learn some humility.” We developers are prone to think we’re clever. And many developers are clever. But we don’t know everything. Maybe I’ve finally figured out how monadic bind operations relate to functional composition. That’s great, but it doesn’t mean I know a thing about what our customers are facing as they use our software every day. Exposing myself to customer pain reminds me of how little I really know.

In my experience, the more isolated the developers, the worse the resulting end product. It doesn’t help that most teams have a layer of business analysts who feel it is their job it to shield developers from users, and vice-versa. It’s dangerous to create an environment where developers have no idea who the users are.

—Jeff Atwood [9]

Now, there is a glaring problem with all this customer-facing warm-fuzziness. In short, it doesn’t make developers go faster. In fact, it takes time away from coding, so it arguably slows them down. So why would I be arguing for it then? The short answer is that going faster doesn’t do a lick of good if you’re running in the wrong direction. Exposing developers to customer pain is about direction rather than speed.

Ask the developers

We want to sustainably minimise lead time to business impact. My hypothesis is that if you point developers in the right direction, then you can ask them for ideas on how to do that. If we then empower them to implement those ideas then we should start to see results.

Ideally, this is an ongoing process. We ask developers if they have any ideas on how to go faster. And then we try them out. Then come back after a couple of weeks and ask how that went. Then ask them again. And keep on asking them until every time you approach their work area you don’t even have to ask them. They start saying things like: “That refactor we did of the routeing engine is really paying off. But, I think we’ll be able to rip things in and out more quickly if we move some of that logic back out into the microservices layer.” You may have no idea what that means, but if we see fewer bugs and happier customers then everyone wins.

How you ask your particular team of developers is up to you. Some people like brainstorming workshops, while others prefer surveys or one-on-one interviews. Each approach will have different strengths and weaknesses. But whichever approach you choose, be sure to make any constraints clear. If you have only a very small budget, then say so. If there’s no flexibility to push out any deadlines, let the developers know. Assuming you have smart, capable developers, they can take these things into account. And if they don’t get it, even after you’ve explained it many times, then you’ve learned something…

Do be careful when communicating constraints though. If we tell developers that there’s no budget; deadlines are fixed; and there’s no wiggle room at all… then they will rightly reply that they can’t help. You have to be very careful in this situation. Quality software, produced fast, costs money. The developers need to see that we are willing to invest in them and their tools. If there is no budget; no room to move on deadlines; and no sign that this will ever change… then a smart developer will look elsewhere. And I would applaud them for it. That is a no-win situation. This is where emotional investment comes in. Show the developers that we care and we’re willing to invest in them in the future. And explain that we are severely resource constrained right now. Then, they may be willing to come up with some creative solutions to get us out of the current pickle.

Assumptions

I’m making a somewhat large assumption here. My assumption here is that your developer are smart enough to understand the constraints when you explain them. The biggest and most obvious constraint is that we do not have an infinite bucket of money to play with. Making software costs a lot of money. More than most people expect or realise. And good software developers are not cheap to employ. My big assumption here is that you have at least one or two smart developers who are capable of understanding this.

The sad fact is that some developers just don’t get it. What do you do then? Well, there’s no simple answer. But I suspect that the reason some developers don’t get it is because they have never been exposed to the bigger picture. They’ve just been asked for unrealistic estimates and told to go faster. They don’t often see things from the customer’s point of view or from the point of view of the person who pays their salary. And the only way they will start to get it is if someone shows them.

The other big assumption I am making is that the developers can be trusted not to embarrass the company if we put them in front of clients. And yes, I have been in plenty of meetings with clients where developers have said stupid things or vented their frustrations in front of clients. Not everyone is ready to be put in front of a powerpoint deck and asked to give a sales pitch. But if a developer can be trusted to just politely shake a hand and say hello, then surely they can at least sit in a corner and quietly watch people use software? [10] Maybe they need someone to go with them at first. But how else will someone learn to be a good ambassador for the organisation if they are never given a chance?

But, I digress. Back to going faster…

Safety belts and engine upgrades

Let’s assume your team is full of clever developers. When you ask them for ideas, they may come up with some things that seem counter-intuitive at first. Things like:

• Test Driven Development (TDD)
• Continuous Integration
• Pair programming or mob programming
• Code reviews

All these techniques will slow development down… at first. TDD looks a lot like writing double the amount of code to produce the same outcome. Pair programming sounds like taking two productive developers and halving their output. I can understand some scepticism. But these are not just trendy buzzwords (and most of these techniques have been around for decades). There are good reasons for all these things.

Let me try to explain with an analogy. When you drive a car, you wear a seatbelt. And these days, we expect our cars to have airbags and crumple zones. But, when you want to drive really fast, you wear a racing harness and helmet and fireproof clothing. To the car, we add a roll cage and a spoiler and sticky tires. It’s not a perfect analogy but hopefully, you see what I’m getting at. At first, things like TDD and code reviews appear to slow you down. They can be awkward and difficult to get used to. But these are the very things that enable the team to go faster safely.

What is pretty certain is that TDD saves time & money as maintenance costs get factored in — a lot of time & money.

—Eric Elliott [11]

Techniques like TDD and continuous integration are about improving software quality. This means releasing fewer bugs into production. Catching bugs before release means less re-work, less embarrassment and happier customers. They are usually quicker (and less expensive) to fix. Over time, the time not spent fixing bugs adds up. What’s more, these techniques also tend to produce code that is more flexible. Code that’s easier to change or re-use. This means that we spend less time fighting against a brittle codebase. And we can spend more time adding new features or modifying functionality. The net result is better software, faster.

Tightening Feedback Loops

The point of all this is to shorten the time between writing some code and getting it into the hands of customers. Once there, then the developers can observe how well this new code is reducing customer pain. Armed with this feedback, they can then improve the code further… and so on. We create a virtuous cycle.

What has been transformative for us is the massive reduction in the amount of time to get feedback from real users.

—Phil Wills [12]

If you’ve been following IT trends over the last few years, then this virtuous cycle will sound familiar. It sounds very much like continuous delivery. But the buzzword isn’t the point. Continuous delivery is just a label for a set of practices. Together, these practices provide tight feedback loops. The feedback loops allow us to decrease risk while increasing velocity.

There is a good reason for this. The environment in which we build software is not just complicated, it is complex. A complicated system has many parts. So many parts in fact, that it takes an expert to understand how everything fits together. But in a complex system, there are not just many parts, but all the parts are connected and react to each other. So, when you change one small thing, then the entire system might change in response. A classic example of this is the cobra effect:

The British government was concerned about the number of venomous cobra snakes in Delhi. The government therefore offered a bounty for every dead cobra. Initially this was a successful strategy as large numbers of snakes were killed for the reward. Eventually, however, enterprising people began to breed cobras for the income. When the government became aware of this, the reward program was scrapped, causing the cobra breeders to set the now-worthless snakes free. As a result, the wild cobra population further increased. [13]

With complex systems, it is very difficult to predict what the effect of a given change might be. This is because making the same change twice may have completely different consequences. The first change causes the system to respond in such a way that it reacts very differently the next time. This can lead to unintended consequences and makes planning and estimating problematic.

[The] way to understand complexity is that acting in the space causes the space to change, and cause and effect can only be understood in retrospect.

—Liz Keogh [14]

How then do we manage to get anything done in a complex environment? What the experts suggest is to “probe, sense and respond.” In other words, create tight feedback loops to gauge whether something is working or not. Then we iterate as quickly as possible. We keep the changes small and the cycle short. As a result, the risk associated with failures is also kept small, and it is cheaper to recover. We make lots of small experiments, keep the ones which work, and revert the ones that fail.

In a complex environment, you probe, sense and respond. You do something that can fail, safely, and it tells you things about the environment which you respond to, changing the environment. This is the land of high-feedback, risk and innovation.

—Liz Keogh [15]

Conclusion

We cannot build a high-performing development team simply by applying ‘best practice’. Unfortunately, there are very few silver bullets in software development. But there are patterns that work well when we have the humility to admit we don’t know everything. Exposing the developers to customer pain closes a feedback loop. This allows us to make sure that if we are going fast, we’re going fast in the right direction. Once this is in place, we can work on continual improvement in a way that suits our given circumstances.