A couple of years ago a friend of mine, a newspaper editor, was firewatching with some factory workers. They fell to talking about his newspaper, which most of them read and approved of, but when he asked them what they thought of the literary section, the answer he got was: “You don't suppose we read that stuff, do you? Why, half the time you're talking about books that cost twelve and sixpence! Chaps like us couldn't spend twelve and sixpence on a book.” These, he said, were men who thought nothing of spending several pounds on a day trip to Blackpool.

This idea that the buying, or even the reading, of books is an expensive hobby and beyond the reach of the average person is so widespread that it deserves some detailed examination. Exactly what reading costs, reckoned in terms of pence per hour, is difficult to estimate, but I have made a start by inventorying my own books and adding up their total price. After allowing for various other expenses, I can make a fairly good guess at my expenditure over the last fifteen years.

The books that I have counted and priced are the ones I have here, in my flat. I have about an equal number stored in another place, so that I shall double the final figure in order to arrive at the complete amount. I have not counted oddments such as proof copies, defaced volumes, cheap paper-covered editions, pamphlets, or magazines, unless bound up into book form. Nor have I counted the kind of junky books-old school text-books and so forth — that accumulate in the bottoms of cupboards. I have counted only those books which I have acquired voluntarily, or else would have acquired voluntarily, and which I intend to keep. In this category I find that I have 442 books, acquired in the following ways:

Now as to the method of pricing. Those books that I have bought I have listed at their full price, as closely as I can determine it. I have also listed at their full price the books that have been given to me, and those that I have temporarily borrowed, or borrowed and kept. This is because book-giving, book-borrowing and bookstealing more or less even out. I possess books that do not strictly speaking belong to me, but many other people also have books of mine: so that the books I have not paid for can be taken as balancing others which I have paid for but no longer possess. On the other hand I have listed the review and complimentary copies at half-price. That is about what I would have paid for them second-hand, and they are mostly books that I would only have bought second-hand, if at all. For the prices I have sometimes had to rely on guesswork, but my figures will not be far out. The costs were as follows(*):

Adding the other batch of books that I have elsewhere, it seems that I possess altogether nearly 900 books, at a cost of £165 15s. This is the accumulation of about fifteen years — actually more, since some of these books date from my childhood: but call it fifteen years. This works out at £11 1s. a year, but there are other charges that must be added in order to estimate my full reading expenses. The biggest will be for newspapers and periodicals, and for this I think £8 a year would be a reasonable figure. Eight pounds a year covers the cost of two daily papers, one evening paper, two Sunday papers, one weekly review and one or two monthly magazines. This brings the figure up to £19 1s, but to arrive at the grand total one has to make a guess. Obviously one often spends money on books without afterwards having anything to show for it. There are library subscriptions, and there are also the books, chiefly Penguins and other cheap editions, which one buys and then loses or throws away. However, on the basis of my other figures, it looks as though £6 a year would be quite enough to add for expenditure of this kind. So my total reading expenses over the past fifteen years have been in the neighbourhood of £25 a year.

Twenty-five pounds a year sounds quite a lot until you begin to measure it against other kinds of expenditure. It is nearly 9s 9d a week, and at present 9s 9d is the equivalent of about 83 cigarettes (Players): even before the war it would have bought you less than 200 cigarettes. With prices as they now are, I am spending far more on tobacco than I do on books. I smoke six ounces a week, at half-a-crown an ounce, making nearly £40 a year. Even before the war when the same tobacco cost 8d an ounce, I was spending over £10 a year on it: and if I also averaged a pint of beer a day, at sixpence, these two items together will have cost me close on £20 a year. This was probably not much above the national average. In 1938 the people of this country spent nearly £10 per head per annum on alcohol and tobacco: however, 20 per cent of the population were children under fifteen and another 40 per cent were women, so that the average smoker and drinker must have been spending much more than £10. In 1944, the annual expenditure per head on these items was no less than £23. Allow for the women and children as before, and £40 is a reasonable individual figure. Forty pounds a year would just about pay for a packet of Woodbines every day and half a pint of mild six days a week — not a magnificent allowance. Of course, all prices are now inflated, including the price of books: still, it looks as though the cost of reading, even if you buy books instead of borrowing them and take in a fairly large number of periodicals, does not amount to more than the combined cost of smoking and drinking.

It is difficult to establish any relationship between the price of books and the value one gets out of them. “Books” includes novels, poetry, text books, works of reference, sociological treatises and much else, and length and price do not correspond to one another, especially if one habitually buys books second-hand. You may spend ten shillings on a poem of 500 lines, and you may spend sixpence on a dictionary which you consult at odd moments over a period of twenty years. There are books that one reads over and over again, books that become part of the furniture of one's mind and alter one's whole attitude to life, books that one dips into but never reads through, books that one reads at a single sitting and forgets a week later: and the cost, in terms of money, may be the same in each case. But if one regards reading simply as a recreation, like going to the pictures, then it is possible to make a rough estimate of what it costs. If you read nothing but novels and “light” literature, and bought every book that you read, you would be spending-allowing eight shillings as the price of a book, and four hours as the time spent in reading it-two shillings an hour. This is about what it costs to sit in one of the more expensive seats in the cinema. If you concentrated on more serious books, and still bought everything that you read, your expenses would be about the same. The books would cost more but they would take longer to read. In either case you would still possess the books after you had read them, and they would be saleable at about a third of their purchase price. If you bought only second-hand books, your reading expenses would, of course, be much less: perhaps sixpence an hour would be a fair estimate. And on the other hand if you don't buy books, but merely borrow them from the lending library, reading costs you round about a halfpenny an hour: if you borrow them from the public library, it costs you next door to nothing.

I have said enough to show that reading is one of the cheaper recreations: after listening to the radio probably THE cheapest. Meanwhile, what is the actual amount that the British public spends on books? I cannot discover any figures, though no doubt they exist. But I do know that before the war this country was publishing annually about 15,000 books, which included reprints and school books. If as many as 10,000 copies of each book were sold — and even allowing for the school books, this is probably a high estimate-the average person was only buying, directly or indirectly, about three books a year. These three books taken together might cost £1, or probably less.

These figures are guesswork, and I should be interested if someone would correct them for me. But if my estimate is anywhere near right, it is not a proud record for a country which is nearly 100 per cent literate and where the ordinary man spends more on cigarettes than an Indian peasant has for his whole livelihood. And if our book consumption remains as low as it has been, at least let us admit that it is because reading is a less exciting pastime than going to the dogs, the pictures or the pub, and not because books, whether bought or borrowed, are too expensive.

_____

*) In Orwell time one pound (£) was equal to 20 shillings (s.) and one shilling was equal to 12 pence (d.). England converted their currency system to decimal one (1 pound = 100 pence) between July 1967 and January 1971 (in several stages). — Comment by Dag.[back]

Over the last few months, the Search Infrastructure engineering team at Dropbox has been busy releasing a new full-text search engine called Nautilus, as a replacement for our previous search engine.

Search presents a unique challenge when it comes to Dropbox due to our massive scale—with hundreds of billions of pieces of content—and also due to the need for providing a personalized search experience to each of our 500M+ registered users. It’s personalized in multiple ways: not only does each user have access to a different set of documents, but users also have different preferences and behaviors in how they search. This is in contrast to web search engines, where the focus on personalization is almost entirely on the latter aspect, but over a corpus of documents that are largely the same for each user (localities aside).

In addition, some of the content that we’re indexing for search changes quite often. For example, think about a user (or several users) working on a report or a presentation. They will save multiple versions over time, each of which might change the search terms that the document should be retrievable by.

More generally, we want to help users find the most relevant documents for a given query—at this particular moment in time—in the most efficient way possible. This requires being able to leverage machine intelligence at several stages in the search pipeline, from content-specific machine learning (such as image understanding systems) to learning systems that can better rank search results to suit each user’s preferences.

The Nautilus team worked with our machine intelligence platform to scale our search ranking and content understanding models. These kind of systems require a lot of iteration to get right, and so it is crucial to be able to experiment with different algorithms and subsystems, and gradually improve the system over time, piece-by-piece. Thus, the primary objectives we set for ourselves when starting the Nautilus project were to:

• Deliver best-in-class performance, scalability, and reliability to deal with the scale of our data
• Provide a foundation for implementing intelligent document ranking and retrieval features
• Build a flexible system that would allow our engineers to easily customize the document-indexing and query-processing pipelines for running experiments
• And, as with any system that manages our users’ content, our search system needed to deliver on these objectives quickly, reliably, and with strong safeguards to preserve the privacy of our users’ data

In this blog post we describe the architecture of the Nautilus system and its key characteristics, provide details about the choices we made in terms of technologies and approaches we chose for the design, and explain how we make use of machine learning (ML) at various stages of the system.

Nautilus consists of two mostly-independent sub-systems: indexing and serving.

The role of the indexing pipeline is to process file and user activity, extract content and metadata out of it, and create a search index. The serving system then uses this search index to return a set of results in response to user queries. Together, these systems span several geographically-distributed Dropbox data centers, running tens of thousands of processes on more than a thousand physical hosts.

The simplest way to build an index would be to periodically iterate through all files in Dropbox, add them to the index, and then allow the serving system to answer requests. However, such a system wouldn’t be able to keep up with changes to documents in anything close to real-time, as we need to be able to do. So we follow a hybrid approach which is fairly common for search systems at large scale:

• We generate “offline” builds of the search index on a regular basis (every 3 days, on average)
• As users interact with files and each other, such as editing files or sharing them with other users, we generate “index mutations” that are then applied to both the live index and to a persistent document store, in near real-time (on the order of a few seconds).

Some other key pieces of the system that we’ll talk about are how to index different kinds of content, including using ML for document understanding and how to rank retrieved search results (including from other search indexes) using an ML-based ranking service.

Data sharding

Before we talk about specific subsystems in Nautilus, let’s briefly discuss how we can achieve the level of scale we need. With hundreds of billions of pieces of content, we have an enormous amount of data that we need to index. We split, or “shard,” this data across multiple machines. To do this, we need to decide how to shard files such that search requests for each user complete quickly, while also balancing load relatively evenly across our machines.

At Dropbox, we already have such a schema for grouping files, called a “namespace,” which can be thought of as a folder that one or more users have access to. One of the benefits of this approach is it allows us to only see search results from files that they have access to, and it is how we allow for shared folders. For example: the folder becomes a new namespace that both sharer and share recipient have access to. The set of files a Dropbox user can access is fully defined by the set of underlying namespaces she was granted access to. Given the above properties of namespaces, when a user searches for a term, we need to search all of the namespaces that they have access to and combine the results from all matches. This also means that by passing the namespaces to the search system we only search content that the querying user can access at the time the search is executed.

We group a number of namespaces into a “partition,” which is the logical unit over which we store, index, and serve the data. We use a partitioning scheme that allows us to easily repartition namespaces in the future, as our needs change.

Document extraction and understanding

What are the kinds of things users would like to search by? Of course there is the content of each document, i.e., the text in the file. But there are also numerous other types of data and metadata that are relevant.

We designed Nautilus to flexibly handle all of these and more, through the ability to define a set of “extractors” each of which extracts some sort of output from the input file and writes to a column in our “document store.” The underlying technology has extra custom built layers that provide access control and data encryption. It contains one row per file, with each column containing the output from a particular extractor. One significant advantage of this schema is that we can easily update multiple columns on a row in parallel without worrying about changes from one extractor interfering with those from others.

For most documents, we rely on Apache Tika to transform the original document into a canonical HTML representation, which then gets parsed in order to extract a list of “tokens” (i.e. words) and their “attributes” (i.e. formatting, position, etc…).

After we extract the tokens, we can augment the data in various ways using a “Doc Understanding” pipeline, which is well suited for experimenting with extraction of optional metadata and signals. As input it takes the data extracted from the document itself and outputs a set of additional data which we call “annotations.” Pluggable modules called “annotators” are in charge of generating the annotations. An example of a simple annotator is the stemming module which generates stemmed tokens based on raw tokens. Another example is converting tokens to embeddings for more flexible search.

Offline build

The document store contains the entire search corpus, but it is not well-suited for running searches. This is because it stores extracted content mapped by document id. For search, we need an inverted index: a mapping from search term to list of documents. The offline build system is in charge of periodically re-building this search index from the document store. It runs the equivalent of a MapReduce job on our document store in order to build up a search index that can be queried extremely fast. Each partition ends up with a set of index files that are stored in an “index store.”

By separating the document extraction process from the indexing process, we gain a lot of flexibility for experiments:

• Modification of the internal format of the index itself, including the ability to experiment on a new index format that improves retrieval performance or reduces storage costs.
• Applying a new document annotator to the entire corpus. Once an annotator has demonstrated benefits when applied to the stream of fresh documents flowing in the instant indexing pipeline, it can be applied to the entire corpus of documents within a couple days by simply adding it to the offline build pipeline. This increases experimentation velocity compared to having to run large backfill scripts for updating the data corpus in the document store.
• Improving the heuristics used for filtering the data that gets indexed. Not surprisingly, when dealing with hundreds of billions of pieces of content, we have to protect the system from edge cases that could cause accuracy or performance degradations, e.g., some extremely large documents or documents that were incorrectly parsed and generate garbled tokens. We have several heuristics for filtering out such documents from the index, and we can easily update these heuristics over time, without having to reprocess the source documents every time.
• Ability to mitigate an unforeseen issue caused by a new experiment. If there is some error in the indexing process, we can simply rollback to a previous version of the index. This safeguard translates to a higher tolerance for risk and speed of iteration when experimenting.
  # Serving

The serving system is comprised of a front-end, which accepts and forwards user search queries; a retrieval engine which retrieves a large list of matching documents for each query; and a ranking system named Octopus that ranks results from multiple back-ends using machine learning. We’ll focus here on the latter two, as the front-end is a fairly straightforward set of APIs that all our clients use (web, desktop, and mobile).

Retrieval engine

The retrieval engine is a distributed system which fetches documents that match a search query. The engine is optimized for performance and high recall—it aims to return the largest set of candidates possible in the given allocated time budget. These results will then be ranked by Octopus, our search orchestration layer, to achieve high precision, i.e., ensure that the most relevant results are highest in the list. The retrieval engine is divided into a set of “leaves” and a “root”:

• The root is primarily in charge of fanning out incoming queries to the set of leaves holding the data, and then receiving and merging results from the leaves, before returning them to Octopus.
  • The root also includes a “query understanding” pipeline which is very similar to the doc understanding pipeline we discussed above. The purpose of this is to transform or annotate a query to improve retrieval results.
• Each leaf handles the actual document lookup for a group of namespaces. It manages the inverted and forward document index. The index is periodically seeded by downloading a build from the offline build process, and is then continuously updated by applying mutations consumed from Kafka queues.

Search Orchestrator

Our Search Orchestration layer is called Octopus. Upon receiving a query from a user, the first task performed by Octopus is to call Dropbox’s access-control service to determine the exact set of namespaces the user has read access to. This set defines the “scope” of the query that will be performed by the downstream retrieval engine, ensuring that only content accessible to the user will be searched.

Besides fetching results from the Nautilus retrieval engine, we have to do a couple things before we can return a final set of results to the user:

• Federation: In addition to our primary document store and retrieval engine (described above), we also have a few separate auxiliary backends that need to be queried for specific types of content. One example of this is Dropbox Paper documents, which currently run on a separate stack. Octopus provides the flexibility to send search queries to and merge results from multiple backend search engines.
• Shadow engines: The ability to serve results from multiple backends is also extremely useful for testing updates to our primary retrieval engine backend. During the validation phase, we can send search queries to both the production system and the new system being tested. This is often referred to as “shadow” traffic. Only results from the production system are returned to the user, but data from both systems is logged for further analysis, such as comparing search results or measuring performance differences.
• Ranking: After collecting the list of candidate documents from the search backends, Octopus fetches additional signals and metadata as needed, before sending that information to a separate ranking service, which in turn computes the scores to select the final list of results returned to the user.
• Access Control (ACL) checks: In addition to the retrieval engine restricting the search to the set of namespaces defined in the query scope, an additional layer of protection is added at the Octopus layer by double checking that each result returned by the retrieval engine can be accessed by the querying user before returning them.

Note that all of these steps have to happen very fast—we target a budget of 500ms for the 95th percentile search (i.e., only 5% of searches should ever take longer than 500ms). In a future blog post, we will describe how we make that happen.

Machine learning powered ranking

As mentioned earlier, we tune our retrieval engine to return a large set of matching documents, without worrying too much about how relevant each document is to the user. The ranking step is where we focus on the opposite end of the spectrum: picking the documents that the user is most likely to want right now. (In technical terms, the retrieval engine is tuned for recall, while the ranker is tuned for precision.)

The ranking engine is powered by a ML model that outputs a score for each document based on a variety of signals. Some signals measure the relevance of the document to the query (e.g., BM25), while others measure the relevance of the document to the user at the current moment in time (e.g., who the user has been interacting with, or what types of files the user has been working on).

The model is trained using anonymized “click” data from our front-end, which excludes any personally identifiable data. Given searches in the past and which results were clicked on, we can learn general patterns of relevance. In addition, the model is retrained or updated frequently, adapting and learning from general users’ behaviors over time.

The main advantage of using an ML-based solution for ranking is that we can use a large number of signals, as well as deal with new signals automatically. For example, you could imagine manually defining an “importance” for each type of signal we have available to us, such as which documents the user interacted with recently, or how many times the document contains the search terms. This might be doable if you only have a handful of signals, but as you add tens or hundreds or even thousands of signals, this becomes impossible to do in an optimal way. This is exactly where ML shines: it can automatically learn the right set of “importance weights” to use for ranking documents, such that the most relevant ones are shown to the user. For example, by experimentation, we determined that freshness-related signals contribute significantly to more relevant results.

After a period of qualification where Nautilus was running in shadow mode, it is currently the primary search engine at Dropbox. We’ve already seen significant improvements to the time-to-index new and updated content, and there’s much more to come.

Now that we have solid foundations in place, our team is busy building on top of the Nautilus platform to add new features and improve search quality. We’re exploring new capabilities, such as augmenting the existing posting-list-retrieval-algorithm with distance-based retrieval in an embedding space; unlocking search for image, video, and audio files; improving personalization using additional user activity signals; and much more.

Nautilus is a prime example of the type of large scale projects involving data retrieval and machine learning that engineers at Dropbox tackle. If you are interested in these kinds of problems, we would love to have you on our team.

Thanks to: Adam Faulkner, Adhiraj Somani, Alan Shieh, Annie Zhou, Braeden Kepner, Elliott Jin, Franck Chastagnol, Han Lee, Harald Schiöberg, Ivan Traus, Kelly Liu, Michael Mi, Peng Wang, Rajesh Venkataraman, Ross Semenov**,and Sammy Steele.

In 1864 British computer pioneer Charles Babbage described the first key-value store. It was meant to be part of his Analytical Engine. Sadly, the Analytical Engine, which would have been the first programmable computer, was never built. But Babbage lays out clearly the design for his key-value store in his autobiography. He imagined a read-only store implemented as punched cards. He referred to these as Tables:

I explained that the Tables to be used must, of course, be computed and punched on cards by the machine, in which case they would undoubtedly be correct. I then added that when the machine wanted a tabular number, say the logarithm of a given number, that it would ring a bell and then stop itself. On this, the attendant would look at a certain part of the machine, and find that it wanted the logarithm of a given number, say of 2303. The attendant would then go to the drawer containing the pasteboard cards representing its table of logarithms. From amongst these he would take the required logarithmic card, and place it in the machine.

Punched card illustration from Babbage’s autobiography showing an integer key (2303) and value representing the decimal part of log10(2303) (0.3622939)

Upon this the engine would first ascertain whether the assistant had or had not given him the correct logarithm of the number; if so, it would use it and continue its work. But if the engine found the attendant had given him a wrong logarithm, it would then ring a louder bell, and stop itself. On the attendant again examining the engine, he would observe the words, "Wrong tabular number," and then discover that he really had given the wrong logarithm, and of course he would have to replace it by the right one.

So, a key-value store (in this case mapping integers to their logarithm) implemented as external storage on punched cards with a human assistant as data bus. We’ve come a long way but key-value stores are as useful today as they were 150 years ago.

Today we’re announcing a native key-value store for Cloudflare Workers. We’re calling this Workers KV and this functionality is just the start of a sequence of announcements around storage and databases on the edge.

Values are written into Workers KV via the standard Cloudflare API and they are available within seconds at every one of Cloudflare’s 150+ global PoPs. Stephen and Zack’s technical post goes into more detail about how to use Workers KV. The values written are encrypted while at rest, in transit, and on local disk; they are only decrypted as needed.

Values can also be written from inside a Cloudflare Worker. Cloudflare takes care of synchronizing keys and values across our entire network.

So, what can you do with Workers KV?

Kenton Varda has taken the entire Cap’n Proto web site and implemented it using Workers and Workers KV. The entire site is served from a Worker that accesses static assets stored using Workers KV. That’s pretty neat… his site is entirely ‘serverless’ or rather ‘originless’

Others might not want to go quite so far but here are some use cases:

• Shopping cart storage: an e-commerce site can store a user’s entire shopping cart in a key/value pair using Workers KV. The e-commerce back end software can call the Cloudflare API to save (and retrieve, if necessary) the shopping cart contents. A Worker running on Cloudflare’s network can access the shopping cart from Workers KV and use it to show the visitor its contents and get them to checkout.

  Storing the cart using Workers KV is more stable than trying to store it in the browser’s local storage (and can’t be cleared by the user) and completely offloads the storage from the e-commerce site’s engine.

• A/B testing: a site can make a per-visitor decision about which profile to use for A/B testing and then store that profile information using Workers KV. This has significant speed advantages (since the A/B decision isn’t delayed until the page load in the browser).
• Authentication verification can be handled in a Worker. For example, a user can log into a web site and details of the login (including the associated authentication token) can be pushed into a Worker KV value and then checked within a Worker. This offloads checking tokens from the backend server meaning that unauthenticated requests can be rejected quickly and authenticated requests passed through.

  In fact, the entire authentication flow can be done in a Worker. A Worker can check credentials against an authentication service via an async API call, and then update a Worker KV value that gets replicated globally automatically.

• Page construction can be performed in a Worker. A Worker can retrieve a template from a backend server or the cache and then fill in values on the page from Worker KV values. Those values can be updated rapidly by backend applications. This eliminates the need for solutions like Edge-Side Includes.

With the addition of Workers KV, Cloudflare Workers moves closer to being a complete compute platform embedded inside the Internet.

The Third Place for Code

Until the advent of services like Cloudflare Workers there were really two places to run code: on a server (perhaps rented from a cloud provider) and on the end-user’s client (which could be an IoT device, mobile phone or computer).

Servers and clients have very different properties. Server software is quick to update, if you want to change functionality it can be done quickly because you control when server software is modified. On the other hand client software can be slow to update: it might require a firmware flash on an IoT device, a download on a mobile app or reinstallation on a computer. Even web-based software can be slow to update if the user keeps a session open for a long time.

Client software does have a massive latency advantage over server software: anything the user does with the client will be fast because because the latency between the eyeball and CPU is very low. Server’s tend to have large latency to the client, and latency that can vary widely from moment to moment and location to location.

Cloudflare Workers and similar services take the best of server software and client software. Workers are fast to update (10s of seconds for a global change), yet close to the end user (hence low latency). This gives developers a new way to think about building software: what should go on the server, what on the client and what in the Internet?

FaaScinating

The rise of Functions-as-a-Service (FaaS) has caused some confusion because anyone who has spent time with the lambda-calculus or purely functional languages will know that keeping state around turns out to be very, very useful. Truly writing code in a functional style is a big commitment and FaaS services really require state storage to fit in with the majority, imperative style of programming.

Of course, FaaS software can store state in external services and use API calls to get and update it. But that’s likely slow and error prone. Cloudflare Workers KV introduces FaaS-native storage in the form of a key-value store. Workers KV can be written and read externally via the Cloudflare API, or internally directly from a Cloudflare Worker.

We can’t wait to see what people build with Cloudflare Workers and Cloudflare Workers KV. Get hacking! If you want to get started today sign up here.

P.S.—  And if you’d like to replicate Babbage’s log table (without the need for a bell and an assistant) here’s how I built a Worker that retrieves the base-10 logarithm of an integer from a Worker KV. If the logarithm was missing it updates the Worker KV store with the computed log.

Step 1: Create a Namespace

Here CF_ACCT contains the Cloudflare account ID, CF_EMAIL the email address associated with the Cloudflare account and CF_KEY the API key. This call creates a namespace called logcard and returns its unique ID (which is needed later)

$ curl "https://api.cloudflare.com/client/v4/accounts/$CF_ACCT/workers/namespaces" \
-X POST \
-H "X-Auth-Email: $CF_EMAIL" \
-H "X-Auth-Key: $CF_KEY" \
-H "Content-Type: application/json" \
--data '{"title": "logcard"}'

{
  "result": {
    "id":    "<REDACTED NAMESPACE ID>", 
    "title": "logcard"
  },
  "success":  true,
  "errors":   [],
  "messages": []
}

Step 2: Upload Code and Bind a Variable to the Namespace

This curl uploads the script babbage.js containing the Worker and names it logcard. It binds the namespace created above to the variable LOGCARD. That variable can be used within the Worker to access the Workers KV namespace created above.

curl "https://api.cloudflare.com/client/v4/accounts/$CF_ACCT/workers/scripts/logcard" \
-X PUT \
-H "X-Auth-Email: $CF_EMAIL" \
-H "X-Auth-Key: $CF_KEY" \
-F 'script=@-;type=application/javascript' -F 'metadata={"body_part": "script", "bindings": [{"name": "LOGCARD", "type": "kv_namespace", "namespace_id": "<REDACTED NAMESPACE ID>"}]};type=application/json' < babbage.js

{
  "result": {
    "id":          "logcard",
    "etag":        "<REDACTED ETAG>",
    "size":        2254,
    "modified_on": "2018-09-28T07:33:01.109336Z"
  },
  "success":  true,
  "errors":   [],
  "messages": []
}

The code accesses the namespace and retrieves values with const retrieved = await LOGCARD.get(paddedInt).  The full code for the Worker can be found here.

I've bound this Worker to the route /logcard on the site https://glowbeamtechnologies.com/. The Worker takes a value between 1 and 9999 in the URI parameter int and outputs a representation of one of Babbage's punched cards containing the decimal part of the base-10 log of int.

It first tries to find the value in the Workers KV, if not found it calculates the value and stores it in the KV for later reuse. You can try it here: https://glowbeamtechnologies.com/logcard?int=2303.

Subscribe to the blog for daily updates on all of our announcements.

Image copyrightGetty Images

Judges in Seattle have decided to quash convictions for marijuana possession for anyone prosecuted in the city between 1996 and 2010.

City Attorney Pete Homes asked the court to take the step "to right the injustices of a drug war that has primarily targeted people of colour."

Possession of marijuana became legal in the state of Washington in 2012.

Officials estimate that more than 542 people could have their convictions dismissed by mid-November.

Mr Holmes said the city should "take a moment to recognise the significance" of the court's ruling.

"We've come a long way, and I hope this action inspires other jurisdictions to follow suit," he said.

Mayor Jenny Durkan also welcomed the ruling, which she said would offer residents a "clean slate."

"For too many who call Seattle home, a misdemeanour marijuana conviction or charge has created barriers to opportunity - good jobs, housing, loans and education," she said.

A total of 30 states in the US now allow the use of marijuana for medical purposes. In Washington state, it has been legal for medical use since 1998.

Currently, nine states (as well as Washington DC) have also legalised marijuana for recreational use, subject to regulations about growing and selling the drug.

Across the US, marijuana trafficking arrests (for selling or distributing the drug in violation of bans or restrictions) are falling, according to the US government's own figures.

In the past few decades, researchers have opened up the extraordinary world of microbes living on and within the human body, linking their influence to everything from rheumatoid arthritis to healthy brain function. Yet we know comparatively little about the rich broth of microbes and chemicals in the air around us, even though we inhale them with every breath. 

This struck Stanford University genomics researcher Michael Snyder as a major knowledge gap, as he pursued long-term research that involved using biological markers to understand and predict the development of disease in human test subjects. “The one thing that was missing was their exposure” to microbes and chemicals in the air, Snyder says. “Human health is clearly dependent not just on the genome or the microbiome, but on the environment. And sampling the environment was the big hole.”

In a new study published September 20 in Cell, Snyder and his co-authors describe their efforts to fix that. They aim to do so with a wearable device that monitors an individual’s daily exposure to airborne bacteria, viruses, protozoa, fungi and chemicals—the so-called exposome. Similar studies in the past have largely relied on a few fixed sampling stations; the Stanford researchers instead modified an existing monitoring device, the size of a big kitchen matchbox, to be strapped to a person’s upper arm or kept nearby. The device continually sipped the air around the 15 test subjects at home, at work and on the road, passing the air through separate filters to collect both biological and chemical compounds. DNA sequencing, together with comparisons of the results against a reference genome database of 40,000 species, indicated that the 15 participants had been exposed to 2,560 biological species—more than a thousand of those after wearing the device for just three months. 

Mass spectrometry analysis added 3,300 chemical signals to the mix. The researchers were able to identify fewer than a third of them—but they note that all the chemicals had passed through a pore-sized filter for the biological assay and could thus potentially reach deep into a person’s lower respiratory tract. Almost all of these samples contained diethylene glycol, used in products from brake fluid to skin cream. The insect repellant DEET also popped up everywhere, apparently an artifact of sampling mainly during the San Francisco area’s insect-friendly springtime. The study calls this “a previously unrecognized type of potentially hazardous exposure,” noting that no government agency “has evaluated possible health risks associated with inhalation” of such compounds.

Snyder and his co-authors suggest that any given person’s exposome is a product of two separate but interacting “clouds”—a term redolent of the dusty shadow around the character Pig-Pen in the Peanuts cartoon. One cloud is environmental and shared with immediate neighbors; the other is more personal, consisting of human- and pet-centric bacteria, fungi, parasites and protozoa. The test subjects lived scattered across the Bay Area, but the interaction of these two clouds meant that individual exposomes were often strikingly different.

Snyder’s exposome, for instance, showed relatively low exposure to pyridine, apparently because the paint in his house lacks this common antifungal additive. Thus, Snyder also lives with a rich fungal flora as a result. By comparing the timing of a mild allergy with his exposome results, he also discovered that the pollen causing his symptoms came not from the pine trees in his yard, as he had suspected, but from a eucalyptus. 

Beyond human health, says the study’s first author Chao Jiang, a postdoctoral researcher in Snyder’s lab, the exposome monitor “is a great tool for studying the evolution and ecology of the things living around us, for exploring the diversity of life.” It can even provide insights into our own hidden emotions. Many of the study’s samples, for instance, contained geosmin—the chemical compound responsible for the earthy smell that causes us to inhale deeply, and often with a hint of joy, when rain comes after a drought. “It’s not just about things that can kill us,” Jiang says. 

David Relman, a Stanford microbiologist who was not involved in the study, described the new research as “fascinating for the set of questions it raises, more so than for anything it might answer.” Environmental health studies often cannot explain the puzzling variability in individual health among people living in the same area and exposed to the same pollutants, he says. The conventional explanation is a kind of epidemiological shrug: Maybe some people experienced a different dosage? Maybe they were genetically predisposed to be vulnerable? 

Exposome monitoring, on the other hand, “allows for massively parallel collection of data,” Relman says. “And that means we can look for combinations of environmental factors that are present at the same place or time, and that might have synergistic or even antagonistic effects for human health.” Imagine a group of people is exposed to the same toxic chemical, he suggests. It might turn out that severe symptoms occur only in those victims also exposed to a certain fungal spore at the same time, for example, and not in people who do not encounter that particular fungus.  

“The technology is great. The ability to collect both microbial and chemical components you are exposed to is brilliant,” adds University of Chicago microbiome researcher Jack Gilbert, who also was not involved in the study. “I want the technology. I want it for my own research.”

But Gilbert adds that “the general design of the study was not ideal,” mainly because he believes there were too few test subjects and too few replications in comparable circumstances. A better design, he suggests, might involve comparing the daily exposomes over a month for 10 or 20 health care workers in the hospital, versus an equal number based at home. Gilbert adds that Snyder “has been very open about the fact that the amount of data in the study design wasn’t perfect,” largely for budgetary reasons.

Each monitoring device cost $2,700 before modification, according to the study, and testing of samples was even more expensive. But Snyder says that his team now hopes to deploy a miniaturized version of the monitor—this one about the size of a small pocket matchbox—on 1,000 test subjects a year from now, with the ultimate goal of commercializing a smartwatch version of the exposome monitor.

Snyder adds that he is currently wearing eight different monitoring devices, including three smartwatches, either to collect data or to evaluate new technologies. The first version of the exposome monitor was “a bit clunky,” he admits. “And as they get older they hum more. My wife notices it. There are times when she says, ‘Do you have to have that thing?’” But he adds, in the course of an interview, that it is on his desk—still monitoring as he speaks.

Earlier this year, researchers at the University of Tokyo accidentally created the strongest controllable magnetic field in history and blew the doors of their lab in the process.

As detailed in a paper recently published in the Review of Scientific Instruments, the researchers produced the magnetic field to test the material properties of a new generator system. They were expecting to reach peak magnetic field intensities of around 700 Teslas, but the machine instead produced a peak of 1,200 Teslas. (For the sake of comparison, a refrigerator magnet has about 0.01 Tesla)

This is the strongest magnetic field ever generated in a controlled, indoor environment, but it’s not the strongest magnetic field produced in history. This honor belongs to some Russian researchers who created a 2,800 Tesla magnetic field in 2001.

In both the Japanese and Russian experiments, the magnetic fields were generated using a technique called electromagnetic flux-compression. This technique causes a brief spike in the strength of the magnetic field by rapidly “squeezing” it to a smaller size. This technique has been around since the 1940s, but in the early days it relied on using large amounts of TNT to generate an explosion powerful enough to compress the magnetic field. The downside of this technique was that it could only be done once since the explosion destroyed the equipment. Furthermore, it was difficult to reproduce and control the explosion.

Instead of using TNT to generate their magnetic field, the Japanese researchers dumped a massive amount of energy—3.2 megajoules—into the generator to cause a weak magnetic field produced by a small coil to rapidly compress at a speed of about 20,000 miles per hour. This involves feeding 4 million amps of current through the generator, which is several thousand times more than a lightning bolt. When this coil is compressed as small as it will go, it bounces back. This produces a powerful shockwave that destroyed the coil and much of the generator.

To protect themselves from the shockwave, the Japanese researchers built an iron cage for the generator. However they only built it to withstand about 700 Teslas, so the shockwave from the 1,200 Teslas ended up blowing out the door to the enclosure.

“I didn’t expect it to be so high,” Shojiro Takeyama, a physicist at the University of Tokyo, told IEEE Spectrum. “Next time, I’ll make [the enclosure] stronger.”

According to the researchers, this experiment not only provides insight into how different materials respond to strong magnetic fields, but may also find applications in the quest to produce fusion energy. Many leading fusion reactor designs require the application of strong magnetic fields on the order of thousands of Teslas for short periods of time, a requirement that the researchers said was “tantalizingly similar” to what they had produced.

Read More: We Were Promised Fusion Energy

After making some adjustments to the generator and rebuilding the iron cage, the researchers plan to pump 5 megajoules of energy into the generator next time, which should around 1,500 Teslas.

“Only 40 years ago, magnetic fields of the order of 1000 Tesla were only reported in extremely complicated and sometimes unreliable explosively-driven systems, without any of the sophisticated level of control,” the researchers concluded in their report. “Therefore, one can say without any doubt, that the present results represent the beginning of a new era in the quest of producing and using ultrahigh magnetic fields for solid state studies as well as for plasma fusion related experiments.”

Looking for a start-up adventure? Google is the leading designer of next-generation search engine technology. We are rapidly hiring talented people to bring the latest and greatest technology to the web.

Opportunities (San Francisco Bay Area-Mountain View)

Engineering

Vice President of Engineering
Software Test Engineers
Systems/Software Engineers
Senior Operations Analyst
Junior Operations Analyst
Program Manager

Marketing

Vice President of Marketing
Director of Marketing
Product Marketing Manager
Marketing Manager (Online focus)
Marketing Manager (Offline focus)
Marketing Administrator

Other

Director of Human Resources
Staffing Manager
College Recruiting Program Manager
Facilities Specialist
Head Chef

Reasons to work at Google!

1. Hot technology
2. Cool technology
3. Intelligent, fun, talented, hard-working, high-energy teammates
4. In the center of the Silicon Valley
5. Excellent benefits
6. Stock options
7. Casual dress atmosphere
8. Free snacks and drinks
9. An exciting place to work! Your ideas can make a difference
10. Millions of people will use and appreciate your software

Google seeks to hire only the best, and conducts business following the spirit and the intent of the equal opportunity laws. This means we base employment decisions exclusively on our current business needs and the given merit of a candidate. We encourage excellence at all levels in our organization, and are not influenced by race, color, gender, sexual orientation, age, handicap, religion, or any other factor irrelevant to doing a great job.

Vice President of Engineering

Google is seeking a Vice President of Engineering to lead development of the core technologies and operational resources that will further establish Google as the world leader in search engine technology. This position will direct systems operations and product development. This position will also build the structure to allow Google to continue to attract and benefit from the research efforts of the world's top scientists.

Requirements:

• Ph.D. in Computer Science, and 10+ years of relevant experience
• Evidence of high scholastic achievement and initiative
• Experience developing/ designing large software systems
• Proven ability to build and grow a world-class engineering organization
• Experience managing a team of 50+ people

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. Please include the position for which you are applying in the subject field.

Software Test Engineers

Google is looking for software test engineers to help ensure the quality of our software. You will work closely with the engineering team to expand our test suite and isolate new problems. Here's an excellent opportunity for you to use your talent while leading the SQA aspects of products that will change the web experience of millions of users. We have openings at all levels of experience.

Requirements:

• Testing experience in any setting
• Knowledge of web technologies is a big plus
• BSCS or MSCS
• Team-oriented person

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. If you recently graduated, please attach an unofficial transcript. Please include the position for which you are applying in the subject field.

Systems/Software Engineers

Google is looking to expand its operations and needs talented engineers to develop the next generation search engine. If you have a need to bring order to a chaotic web, contact us.

Requirements:

• Several years of industry or hobby-based experience
• BS or MS in Computer Science or equivalent (Ph.D. a plus)
• Extensive experience programming in C or C++
• Extensive experience programming in the UNIX environment
• Knowledge of TCP/IP and network programming
• Experience developing/designing large software systems
• Experience programming in Python a plus

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. If you recently graduated, please attach an unofficial transcript. Please include the position for which you are applying in the subject field.

Senior Operations Analyst

You will monitor and analyze Google's Production Systems and manage process failures through resolution. In addition, you will act as a lead to solve process failures, report on historical trends for systems, and be a resource to senior development engineers.

Requirements:

• 3+ years systems administration-level UNIX experience
• Knowledge of scripiting languages (Korn Shell, Perl and/or Python)
• Familiarity with networking
• Ability to monitor and analyze production systems
• Ability to work in a high performance team environment
• Ability to work independently to solve problems
• Strong attention to detail, technical aptitude, dependability, and
• flexibility with hours to monitor from offsite
• BA/BS degree or equivalent

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. If you recently graduated, please attach an unofficial transcript. Please include the position for which you are applying in the subject field.

Junior Operations Analyst

You will monitor Google's Production Systems and escalate process failures through resolution. In addition, you will report on historical trends for systems, and be a resource to senior operations analysts.

Requirements:

• Knowledge of UNIX as a user
• Scripting experience a plus
• Ability to monitor production systems with strong attention to detail
• Strong technical aptitude, dependability, and flexibility with hours to monitor from offsite
• BA/BS degree or equivalent

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. If you recently graduated, please attach an unofficial transcript. Please include the position for which you are applying in the subject field.

Program Manager

Google is looking for project managers to help us coordinate our software development efforts. You are responsible for defining project schedules in cooperation with our technical leads, and for tracking these schedules to monitor progress and anticipate problems. Must be self-motivated and detail-oriented with good organizational skills as well as demonstrated success in project and functional leadership roles. Knowledge of the full software development life cycle required. Demonstrated ability to work with engineers in a team environment. High-impact position; as a senior project manager you will report directly to the VP of Engineering. (We are also looking to fill junior project management positions.)

Requirements:

• BSCS or MSCS in CS or related field
• 5+ years of experience in project management (senior position)
• 2+ years of experience in project management (junior positions)

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. Please include the position for which you are applying in the subject field.

Vice President of Marketing

Do you have a proven track record of building and growing a world-class marketing organization? Google is seeking a Vice President of Marketing to build and expand the Google brand as the leading provider of next generation internet search technology.

The Vice President of Marketing will actively assume all internal and external marketing responsibilities, including brand and product positioning strategy, product introductions, viral marketing strategies, advertising, public relations, and corporate marketing communications. This position will assume a key leadership role in the company's strategic and business planning process, working closely with sales, business development, and engineering functions. This position will confidently evangelize the company and its products as the company moves toward the IPO process.

Requirements:

• 10+ years experience as a leader in technology and/or consumer brand marketing
• Proven ability to build and grow a world-class marketing organization

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. Please include the position for which you are applying in the subject field.

Director of Marketing

Are you a talented and experienced marketer with a vision on how to promote the best search technology on the web? Google is seeking a dynamic individual with experience in developing coordinated marketing strategies, integrating marketing communication messages, and providing strong creative and strategic direction to build the Google brand and drive traffic to its website. This individual will execute for maximum coordination and effectiveness of online and offline marketing messages, advertising, and promotions.

Requirements:

• BA/BS and MBA preferred
• Strong communication skills
• High-levels of creativity
• 7-10+ years of experience managing marketing teams and building brands, including 2+ years of online experience
• Experience in management of online and offline ad campaigns

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. Please include the position for which you are applying in the subject field.

Product Marketing Manager

Do you have ideas on how to keep making Google's search service even better? If so, the product marketing manager position may be right for you. Google is looking for a product marketing manager who will assist with product positioning strategy, develop data sheets/ collateral, lead product launches, define features/ requirement documents, coordinate between marketing and engineering, and assist with product advertising campaigns.

Requirements:

• BA/BS and MBA preferred
• Strong communication skills
• High-levels of creativity
• Proven quantitative skills
• 4 - 7+ years of marketing experience of technology products
• Internet experience preferred
• Experience working with engineering teams

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. Please include the position for which you are applying in the subject field.

Marketing Manager (Online focus)

Google is seeking a dynamic individual to help promote the Google brand online and drive traffic to its website. Responsibilities include management of Google's online advertising campaigns (design of creatives, media buying, analysis of click-throughs), direct e-mail campaigns, online promotions, viral marketing campaigns, affiliate programs, and more. The marketing manager will also participate in offline marketing activities to further the promotion of the Google brand. This position will work closely with public relations, product managers, and other marketing managers.

Requirements:

• BA/BS and MBA preferred
• Strong communication skills
• High-levels of creativity
• Proven quantitative skills
• 4 - 7+ years of marketing experience, including 2+ years of online experience

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. Please include the position for which you are applying in the subject field.

Marketing Manager (Offline focus)

Google is seeking a dynamic individual to help promote Google's brand and website. Responsibilities will include the design and implementation of advertising programs, consumer research, promotions, event marketing, integrated marketing communications, and other creative marketing ideas. This position will work closely with public relations, product managers, and other marketing managers.

Requirements:

• BA/BS and MBA preferred
• Strong communication skills
• High-levels of creativity
• Proven quantitative skills
• 4 - 7+ years of marketing experience in developing consumer brands, managing advertising campaigns, and designing other consumer promotions
• Technology and Internet experience preferred

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. Please include the position for which you are applying in the subject field.

Marketing Administrator

This position reports to the VP of Marketing and supports Brand Marketing, Public Relations and Product Marketing management teams. Your primary responsiblities will be general administrative duties which may include organizing marketing and customer meetings, answering phones, preparing presentation materials, managing travel and completing expense reports. You will also manage special projects as needed.

Requirements:

• 5+ years of administrative experience within a fast paced high technology company
• Excellent organizational skills
• Strong communication skills, both written and oral
• The ability to interface effectively with both employees and customers
• Strong PC skills, MS Office preferred
• Self-motivated individual with strong initiative
• BA Degree or equivalent experience

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. Please include the position for which you are applying in the subject field.

Director of Human Resources

Google is seeking a creative, dynamic, and hands-on Director of Human Resources to help build the finest organization in the world!

This position reports directly to the President and manages the growth of our HR infrastructure (organizational development, employee relations, recruiting, compensation and benefits). Responsibilities will include the design of programs and implementation of processes to better drive Google's rapid growth of people resources.

Requirements:

• 7+ years of progressive HR/ generalist experience, with 2+ years at the director level
• Experience working within a progressive, fast paced, high-tech start-up environment, where the employee growth rate is very high
• Demonstrated ability to build relationships with all levels and functions within the organization
• Strong communication and leadership/ management skills
• BA/ BS degree

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. Please include the position for which you are applying in the subject field.

Staffing Manager

Are you ready to build the team for the hottest company in the Silicon Valley? We are seeking a creative, self-motivated individual to identify, contact, and interview candidates to fill specific hiring needs within Google. This person is responsible for optimizing the entire recruiting and hiring process at Google, which includes:

• Working with schools to recruit qualified applicants
• Attending job fairs
• Posting openings in creative locations
• Screening resumes and employment applications
• Evaluating candidates using phone screens

Requirements:

• BA/BS, MS or Ph.D. degree
• 5+ years experience in technical recruiting
• 5+ years HR generalist experience
• Excellent interpersonal skills
• Intelligence to interview the brightest in the world

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. Please include the position for which you are applying in the subject field.

College Recruiting Program Manager

To keep pace with the growth of our recruitment efforts, Google is looking to expand our college relations function. We are seeking a high-energy College Recruiting Program Manager to oversee all aspects of our college recruitment initiative. In this position, you will be responsible for:

• Designing, developing, and implementing Google's college recruitment program
• Building campus relationships
• Coordinating and conducting on-campus and in-house interviews
• Attending job fairs and campus presentations
• Publicizing career opportunities
• Supporting the staffing function with management of online recruiting efforts, employee referral, and recruitment program initiatives

Requirements:

• Excellent communication and organizational and follow-through skills
• Strong written and interpersonal communication skills
• Strong presentation and project management skills
• 2+ years staffing experience within a high technology company
• Creative approach to problem solving
• Strong PC skills a must
• Some travel required
• BA/BS degree required

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. Please include the position for which you are applying in the subject field.

Facilities Specialist

In this position you will manage multiple facilities projects and act as the primary contact for facilities needs. In addition, you will:

• Work with vendors on pricing, and establish vendor contracts for building management and maintenance services.
• Provide input to budget planning process, monitor expenses and review monthly
• Orient new hires regarding office procedures and facilities
• Streamline workflow, reduce process redundancy
• Coordinate and assist in office moves
• Maintain inventory of all facilities capital assets
• Be responsible for safety and security plans, and setting up emergency response procedures

Requirements:

• Excellent communication and interaction skills, and the ability to work with both employees and vendors effectively
• Ability to handle multiple requests and work in a fast paced office environment
• Strong PC Skills, such as Word and Excel, and provide analysis
• Strong analytical, judgement and problem solving skills
• Customer satisfaction orientation, ability to follow through
• At least 2 years of experience in a project management role
• Ability to lift and move heavy objects in excess of 35lbs
• BS Degree or equivalent

Send a text (ASCII) or HTML version of your resume to jobs@google.com or fax to (650) 618-1499. Please include the position for which you are applying in the subject field.

Head Chef

The Googlers are hungry!!

One of Silicon Valley's hottest and fastest growing internet companies is looking for an experienced and innovative gourmet Chef to manage all aspects of Google's onsite Cafe. In this position you will be responsible for managing the Cafe, from menu planning to final presentation. The experienced Chef of choice should be creative and healthy in planning menus for Googlers. Here's a group of people with well traveled refined palates with a craving for epicurial delights.

The only Chef job with stock options!

Home | About | Jobs@Google | Contact Us

©1999 Google Inc.

We've confirmed through multiple sources that Intel is fabbing its new H310C chipset on its 22nm process. That means the chip-making giant has taken a step back to an older process for the H310C chipset as it struggles with its ongoing shortage of 14nm processors. Contrary to recent reports, our sources confirmed Intel manufactures these chips and not TSMC (which has been reported in recent weeks), though that could be subject to change in the future.

H310C Credit: mydrivers.comThe shift in Intel's strategy comes as the company struggles with the fallout from its chronically delayed 10nm process. Now the company is dealing with an increasingly loud chorus of reports that Intel's 14nm shortage is now impacting its server, desktop and mobile chips.

The worrying lack of motherboards with the H310 chipset, which began back in March, served as the first sign of an impending shortage of Intel's 14nm silicon. In May, reports surfaced that Intel had suspended production of the chipset, and in July, the company finally acknowledged a much larger issue with 14nm production capacity.

Intel typically produces chipsets on a larger node than its current-gen processors, but the delayed 10nm production has found both chipsets and chips on the same 14nm node, creating a manufacturing bottleneck as the company experiences record demand for 14nm processors.

H310 Credit: mydrivers.com

Word of a new H310C chipset surfaced last month. Leaked images of the new H310C on mydrivers.com revealed that the new H310C, which measures 10 x 7mm, is much larger than the 14nm H310, which measures 8.5 x 6.5mm.

However, the increased physical size alone doesn't confirm that Intel is fabbing the new chipset on a larger process, so we reached out to several contacts and confirmed the change.

Intel hasn't commented on the matter, saying it doesn't comment on unreleased products. However, motherboards with the new chipset are already moving out into the supply chain, meaning that Intel will soon publish a formal specification sheet, much like the H310 specs, that confirms the change.

Our sources indicate that vanilla H310 motherboards will continue to be offered at retail locations, but they fully expect the H310C motherboards, which will be branded with either an H310C or H310 R2.0 branding, to replace the existing SKUs eventually. The new chipsets will also support Windows 7, as reported by our sister site AnandTech, which may signal that Intel will restore compatibility with the older OS on its newer motherboards, such as the forthcoming Z390 lineup (EDIT: we've learned this support comes as a byproduct of motherboard drivers and is not borne of an innate characteristic of the chipset).

Currently, the 14nm chipsets are exacerbating manufacturing challenges in Intel's 14nm foundries. In most cases, the company has to create one chipset for each processor, so easing that production load would free Intel up to produce more 14nm Coffee Lake processors. For Intel, it makes a lot of sense to move back to the 22nm process for its chipsets: performance and power consumption also isn't as much of a concern with the low-end chipsets, and, most importantly, the tiny chips generate very little margin. That means Intel stands to lose little from going back to an older node that may cost slightly more to produce.

We don't expect Intel to move its desktop or server processors back to the 22nm node, but it is possible that it may migrate other chipsets back to 22nm. It's also possible for Intel to move other low-margin parts back to an older process, or to explore outsourcing some of its production in the future.

According to Bloomberg, the self-proclaimed bug bounty hunter is a minor celebrity in Taiwan who's appeared on talk shows and was reportedly sued by a local bus operator after breaching their systems to nab a ticket for just NT$1 (3 cents). Earlier this month, Chi-yuan shared a screenshot showing an Apple Pay loophole he'd found that allowed him to pay NT$1 for 500 iPhones. His other claims include cyber-attacks on Apple and Tesla, and he's also listed on Japanese messaging giant Line's 2016 bug-hunters' hall of fame.

"I don't want to be a proper hacker, and I don't even want to be a hacker at all," Chang said in a recent post. "I'm just bored and try to dabble so that I can earn some money." Facebook, like other tech giants, dishes out cash to cyber-security experts who point out flaws in its system as part of a bug bounty program. But considering all the setbacks it's currently facing -- including the departure of Instagram's founders and its ongoing fake news crisis -- it probably doesn't want to deal with a high-profile hack right now.

Neither is the company keen on paying out bounties to people who test vulnerabilities against real users. Back in 2013, it refused to reward Khalil Shreateh -- a systems information expert from Palestine -- for hacking into Zuck's account and posting an Enrique Iglesias video on the wall of one of his college friends. The Facebook founder has also previously had his Twitter and Pinterest accounts breached by notorious hacker group OurMine.

Offentliggjort den 27. sep. 2018

Download & Documentation

Zig is an open-source programming language designed for robustness,optimality, and clarity. Zig is aggressively pursuing its goal of overthrowing C as the de facto language for system programming. Zig intends to be so practical that people find themselves using it even if they dislike it.

This is a massive release, featuring 6 months of work and changes from 36 different contributors.

I tried to give credit where credit is due, but it's inevitable I missed some contributions as I had to go through 1,345 commits to type up these release notes. I apologize in advance for any mistakes.

Special thanks to my patrons who provide financial support. You're making Zig sustainable.

Stack Traces on All Targets

Zig uses LLVM's debug info API to emit native debugging information on all targets. This means that you can use native debugging tools on Zig code, for example:

• MSVC on Windows
• lldb on MacOS
• gdb and valgrind on Linux

In addition, Zig's standard library can read its own native debug information. This means that crashes produce stack traces, and errors produceError Return Traces.

MacOS

This implementation is able to look at the executable's own memory to find out where the .o files are, which have the DWARF info.

Windows

Thanks to Sahnvour for implementing the PE parsing and starting the effort to PDB parsing. I picked up where he left off and finished Windows stack traces.

Thanks to Zachary Turner from the LLVM project for helping me understand the PDB format. I still owe LLVM some PDB documentation patches in return.

Similar to on MacOS, a Windows executable in memory has location information pointing to a .pdb file which contains debug information.

Linux

Linux stack traces worked in 0.2.0. However now std.debug.dumpStackTrace& friends use ArenaAllocator backed byDirectAllocator. This has the downside of failing to print a stack trace when the system is out of memory, but for the more common use case when the system is not out of memory, but the debug info cannot fit in std.debug.global_allocator, now stack traces will work. This is the case for the self hosted compiler. There is a proposal tommap() debug info rather than using read().

See also Compatibility with Valgrind.

zig fmt

Thanks to Jimmi Holst Christensen's dilligent work, the Zig standard library now supports parsing Zig code. This API is used to implement zig fmt, a tool that reformats code to fit the canonical style.

As an example, zig fmt will change this code:

test"fmt" {const a = []u8{1, 2, 3,4, 5,6,7 };switch (0) { 0 => {}, 1 => unreachable, 2,3 => {}, 4...7 => {}, 1 + 4 * 3 + 22 => {}, else => { const a = 1; const b = a; }, }

    foo(a, b, c, d, e, f, g,);
}

...into this code:

test"fmt" {const a = []u8{1, 2,3, 4, 5, 6,7,
    };switch (0) {0 => {},1 => unreachable,2, 3 => {},4...7 => {},1 + 4 * 3 + 22 => {},else => {const a = 1;const b = a;
        },
    }

    foo(
        a,
        b,
        c,
        d,
        e,
        f,
        g,
    );
}

It does not make any decisions about line widths. That is left up to the user. However, it follows certain cues about when to line break. For example, it will put the same number of array items in a line as there are in the first one. And it will put a function call all on one line if there is no trailing comma, but break every parameter into its own line if there is a trailing comma.

Thanks to Marc Tiehuis, there are currently two editor plugins that integrate with zig fmt:

zig fmt is only implemented in the self-hosted compiler, which is not finished yet, so in order to use it one must follow the README instructions to build the self-hosted compiler from source.

Theimplementation of the self-hosted parser is an interesting case study of avoiding recursion by using an explicit stack. It is essentially a hand-written recursive descent parser, but with heap allocations instead of recursion. When Jimmi originally implemented the code, we thought that we could not solve the unbounded stack growth problem of recursion. However, since then, I prototyped several solutions that provide the ability to have recursive function calls without giving up statically known upper bound stack growth. SeeRecursion Status for more details.

Automatic formatting can be disabled in source files with a comment like this:

test"this is left alone"  {   }

zig fmt is written using the standard library's event-based I/O abstractions andasync/await syntax, which means that it is multi-threaded with non-blocking I/O. A debug build of zig fmt on my laptop formats the entire Zig standard library in 2.1 seconds, which is 75,516 lines per second. See Concurrency Status for more details.

zig run

zig run file.zig can now be used to execute a file directly.

Thanks to Marc Tiehuis for the initial implementation of this feature. Marc writes:

On a POSIX system, a shebang can be used to run a zig file directly. An example shebang would be #!/usr/bin/zig run. You may not be able pass extra compile arguments currently as part of the shebang. Linux for example treats all arguments after the first as a single argument which will result in an 'invalid command'.

Note: there is a proposal to change this to zig file.zig to match the interface of other languages, as well as enable the common pattern #!/usr/bin/env zig.

Zig caches the binary generated by zig run so that subsequent invocations have low startup cost. See Build Artifact Caching for more details.

Automated Static Linux x86_64 Builds of Master Branch

Zig now supports building statically against musl libc.

On every master branch push, the continuous integration server creates a static Linux build of zig and updates the URL https://ziglang.org/builds/zig-linux-x86_64-master.tar.xz to redirect to it.

In addition, Zig now looks for libc and Zig std lib at runtime. This makes static builds the easiest and most reliable way to start using the latest version of Zig immediately.

Windows has automated static builds of master branchvia AppVeyor.

MacOS static CI builds arein progress and should be available soon.

Pointer Reform

During this release cycle, two design flaws were fixed, which led to a chain reaction of changes that I called Pointer Reform, resulting in a more consistent syntax with simpler semantics.

The first design flaw was that the syntax for pointers was ambiguous if the pointed to type was a type. Consider this 0.2.0 code:

const assert = @import("std").debug.assert;comptime {var a: i32 = 1;const b = &a;@compileLog(@typeOf(b));
    *b = 2;
    assert(a == 2);
}

This works fine. The value printed from the@compileLog statement is &i32. This makes sense because b is a pointer to a.

Now let's do it with a type:

const assert = @import("std").debug.assert;comptime {var a: type = i32;const b = &a;@compileLog(b);
    *b = f32;
    assert(a == f32);
}

$ zig build-obj test.zig 
| &i32test.zig:6:5:error:found compile log statement
    @compileLog(b);^test.zig:7:5:error:attempt to dereference non-pointer type 'type'
    *b = f32;^

It doesn't work in 0.2.0, because the & operator worked differently fortype than other types. Here, b is the type &i32 instead of a pointer to a type which is how we wanted to use it.

This prevented other things from working too; for example if you had a[]type{i32, u8, f64} and you tried to use a for loop, it crashed the compiler because internally a for loop uses the & operator on the array element.

The only reasonable solution to this is to have different syntax for the address-of operator and the pointer operator, rather than them both being &.

So pointer syntax becomes *T, matching syntax from most other languages such as C. Address-of syntax remains &foo, again matching common address-of syntax such as in C. This leaves one problem though.

With this modification, the syntax *foo becomes ambiguous with the syntax for dereferencing. And so dereferencing syntax is changed to a postfix operator: foo.*. This matches post-fix indexing syntax: foo[0], and in practice ends up harmonizing nicely with other postfix operators.

The other design flaw is a problem that has plagued C since its creation: the pointer type doesn't tell you how many items there are at the address. This is now fixed by having two kinds of pointers in Zig:

• *T - pointer to exactly one item.
  • Supports deref syntax: ptr.*
• [*]T - pointer to unknown number of items.
  • Supports index syntax: ptr[i]
  • Supports slice syntax: ptr[start..end]
  • T must have a known size, which means that it cannot bec_void or any other @OpaqueType().

Note that this causes pointers to arrays to fall into place, as a single-item pointer to an array acts as a pointer to a compile-time known number of items:

• *[N]T - pointer to N items, same as single-item pointer to array.
  • Supports index syntax: array_ptr[i]
  • Supports slice syntax: array_ptr[start..end]
  • Supports len property: array_ptr.len

Consider how slices fit into this picture:

• []T - pointer to runtime-known number of items.
  • Supports index syntax: slice[i]
  • Supports slice syntax: slice[start..end]
  • Supports len property: slice.len

This makes Zig pointers significantly less error prone. For example, it fixed issue #386, which demonstrates how a pointer to an array in Zig 0.2.0 is a footgun when passed as a parameter. Meanwhile in 0.3.0, equivalent code is nearly impossible to get wrong.

For consistency with the postfix pointer dereference operator, optional unwrapping syntax is now postfix as well:

0.2.0: ??x

0.3.0: x.?

And finally, to remove the last inconsistency of optional syntax, the ?? operator is now the keyword orelse. This means that Zig now has the property thatall control flow occurs exclusively via keywords.

There is a plan for one more pointer type, which is a pointer that has a null-terminated number of items. This would be the type of the parameter to strlen for example. Although this will make the language bigger by adding a new type, it allows Zig to delete a feature in exchange, since it will makeC string literals unnecessary. String literals will both have a compile-time known length and be null-terminated; therefore they will implicitly cast to slices as well as null-terminated pointers.

There is one new issue caused by Pointer Reform. Because C does not have the concept of single-item pointers or unknown-length pointers (or non-null pointers), Zig must translate all C pointers as ?[*]T. That is, a pointer to an unknown number of items that might be null. This can cause some friction when using C APIs, which is unfortunate because Zig's types are perfectly compatible with C's types, but .h files are unable to adequately describe pointers. Although it would be much safer to translate .h files offline and fix their prototypes, there isa proposal to add a C pointer type. This new pointer type should never be used on purpose, but would be used when auto-translating C code. It would simply have C pointer semantics, which means it would be just as much of a footgun as C pointers are. The upside is that it would make interaction with C APIs go back to being perfectly seamless.

Default Float Mode is now Strict

In response to an overwhelming consensus, floating point operations use Strict mode by default. Code can use @setFloatMode to override the mode on a per-scope basis.

Thanks to Marc Tiehuis for implementing the change.

Remove this

this was always a weird language feature. An identifier which referred to the thing in the most immediate scope, which could be a module, a type, a function, or even a block of code.

The main use case for it was for anonymous structs to refer to themselves. This use case is solved with a new builtin function, @This(), which always returns the innermost struct or union that the builtin call is inside.

The "block of code" type is removed from Zig, and the first argument of@setFloatMode is removed.@setFloatMode now always refers to the current scope.

Remove Explicit Casting Syntax

Previously, these two lines would have different meanings:

exportfnfoo(x: u32) void {const a: u8 = x;const b = u8(x);
}

The assignment to a would give error: expected type 'u8', found 'u32', because not all values of u32 can fit in a u8. But the assignment to b was "cast harder" syntax, and Zig would truncate bits, with a safety check to ensure that the mathematical meaning of the integer was preserved.

Now, both lines are identical in semantics. There is no more "cast harder" syntax. Both cause the compile error because implicit casts are only allowed when it is completely unambiguous how to get from one type to another, and the transformation is guaranteed to be safe. For other casts, Zig has builtin functions:

• @bitCast - change type but maintain bit representation
• @alignCast - make a pointer have more alignment
• @boolToInt - convert true to 1 and false to 0
• @bytesToSlice - convert a slice of bytes to a slice of another type
• @enumToInt - obtain the integer tag value of an enum or tagged union
• @errSetCast - convert to a smaller error set
• @errorToInt - obtain the integer value of an error code
• @floatCast - convert a larger float to a smaller float
• @floatToInt - obtain the integer part of a float value
• @intCast - convert between integer types
• @intToEnum - obtain an enum value based on its integer tag value
• @intToError - obtain an error code based on its integer value
• @intToFloat - convert an integer to a float value
• @intToPtr - convert an address to a pointer
• @ptrCast - convert between pointer types
• @ptrToInt - obtain the address of a pointer
• @sliceToBytes - convert a slice of anything to a slice of bytes
• @truncate - convert between integer types, chopping off bits

Some are safe; some are not. Some perform language-level assertions; some do not. Some are no-ops at runtime; some are not. Each casting function is documented independently.

Having explicit and fine-grained casting like this is a form of intentional redundancy. Casts are often the source of bugs, and therefore it is worth double-checking a cast to verify that it is still correct when the type of the operand changes. For example, imagine that we have the following code:

fnfoo(x: i32) void {var i = @intCast(usize, x);
}

Now consider what happens when the type of x changes to a pointer:

test.zig:2:29:error:expected integer type, found '*i32'
    var i = @intCast(usize, x);^

Direct Parameter Passing

Previously, it was illegal to pass structs and unions by value in non-extern functions. Instead one would have to have the function accept a const pointer parameter. This was to avoid the ambiguity that C programs face - having to make the decision about whether by-value or by-reference was better. However, there were some problems with this. For example, when the parameter type is inferred, Zig would automatically convert to a const pointer. This caused problems in generic code, which could not distinguish between a type which is a pointer, and a type which has been automatically converted to a pointer.

Now, parameters can be passed directly:

const assert = @import("std").debug.assert;const Foo = struct {
    x: i32,
    y: i32,
};fncallee(foo: Foo) void {
    assert(foo.y == 2);
}test"pass directly" {
    callee(Foo{ .x = 1, .y = 2 });
}

I have avoided using the term "by-value" because the semantics of this kind of parameter passing are different:

• Zig is free to pass the parameter by value - perhaps if it is smaller than some number of bytes - or pass it by reference.
• To the callee, the value appears to be a value and is immutable.
• The caller guarantees that the bytes of the parameter will not change for the duration of the call. This means that it is unsound to pass a global variable in this way if that global variable is mutated by the callee. There is anopen issue which explores adding runtime safety checks for this.

Because of these semantics, there's a clear flow chart for whether to accept a parameter asT or *const T:

• Use T, unless one of the following is true:
  • The function depends on the address of the parameter.
  • The parameter may alias another parameter or global variable.

Now that we have this kind of parameter passing, Zig's implicit cast from T to *const T is less important. One might even make the case that such a cast is dangerous. Therefore we havea proposal to remove it.

There is one more area that needs consideration with regards to direct parameter passing, and that is with coroutines. The problem is that if a reference to a stack variable is passed to a coroutine, it may become invalid after the coroutine suspends. This is a design flaw in Zig that will be addressed in a future version. See Concurrency Status for more details.

Note that extern functions are bound by the C ABI, and therefore none of this applies to them.

Rewrite Rand Functions

Marc Tiehuis writes:

We now use a generic Rand structure which abstracts the core functions from the backing engine.

The old Mersenne Twister engine is removed and replaced instead with three alternatives:

• Pcg32
• Xoroshiro128+
• Isaac64

These should provide sufficient coverage for most purposes, including a CSPRNG using Isaac64. Consumers of the library that do not care about the actual engine implementation should use DefaultPrng and DefaultCsprng.

Error Return Traces across async/await

One of the problems with non-blocking programming is that stack traces and exceptions are less useful, because the actual stack trace points back to the event loop code.

In Zig 0.3.0, Error Return Traces work across suspend points. This means you can use try as the main error handling strategy, and when an error bubbles up all the way, you'll still be able to find out where it came from:

const std = @import("std");const event = std.event;const fs = event.fs;test"unwrap error in async fn" {var da = std.heap.DirectAllocator.init();defer da.deinit();const allocator = &da.allocator;var loop: event.Loop = undefined;try loop.initMultiThreaded(allocator);defer loop.deinit();const handle = tryasync<allocator> openTheFile(&loop);defercancel handle;

    loop.run();
}

asyncfnopenTheFile(loop: *event.Loop) void {const future = (async fs.openRead(loop, "does_not_exist.txt") catchunreachable);const fd = (await future) catchunreachable;
}

$ zig test test.zig
Test 1/1 unwrap error in async fn...attempt to unwrap error: FileNotFoundstd/event/fs.zig:367:5: 0x22cb15 in ??? (test)
    return req_node.data.msg.Open.result;^std/event/fs.zig:374:13: 0x22e5fc in ??? (test)
            return await (async openPosix(loop, path, flags, os.File.default_mode) catch unreachable);^test.zig:22:31: 0x22f34b in ??? (test)
    const fd = (await future) catch unreachable;^std/event/loop.zig:664:25: 0x20c147 in ??? (test)
                        resume handle;^std/event/loop.zig:543:23: 0x206dee in ??? (test)
        self.workerRun();^test.zig:17:13: 0x206178 in ??? (test)
    loop.run();^

Tests failed. Use the following command to reproduce the failure:
zig-cache/test

Note that this output contains 3 components:

• An error message: attempt to unwrap error: FileNotFound
• An error return trace. The error was first returned at fs.zig:367:5 and then returned at fs.zig:374:13. You could go look at those source locations for more information.
• A stack trace. Once the error came back from openRead, the code tried tocatchunreachable which caused the panic. You can see that the stack trace does, in fact, go into the event loop as described above.

It is important to note in this example, that the error return trace survived despite the fact that the event loop is multi-threaded, and any one of those threads could be the worker thread that resumes an async function at the await point.

This feature is enabled by default for Debug and ReleaseSafe builds, and disabled for ReleaseFast and ReleaseSmall builds.

This is just the beginning of an exploration of what debugging non-blocking behavior could look like in the future of Zig. See Concurrency Status for more details.

New Async Call Syntax

Instead of async(allocator) call(), now it isasync<allocator> call().

This fixes syntax ambiguity when leaving off the allocator, and fixes parse failure when call is a field access.

This sets a precedent for using <> to pass arguments to a keyword. This will affect enum, union, fn, andalign (see #661).

ReleaseSmall Mode

Alexandros Naskos contributed a new build mode.

$ zig build-exe example.zig --release-small

• Medium runtime performance
• Safety checks disabled
• Slow compilation speed
• Small binary size

New builtins: @typeInfo and @field

Alexandros Naskos bravely dove head-first into the deepest, darkest parts of the Zig compiler and implemented an incredibly useful builtin function:@typeInfo.

This function accepts a type as a parameter, and returns a compile-time known value of this type:

pubconst TypeInfo = union(TypeId) {
    Type: void,
    Void: void,
    Bool: void,
    NoReturn: void,
    Int: Int,
    Float: Float,
    Pointer: Pointer,
    Array: Array,
    Struct: Struct,
    ComptimeFloat: void,
    ComptimeInt: void,
    Undefined: void,
    Null: void,
    Optional: Optional,
    ErrorUnion: ErrorUnion,
    ErrorSet: ErrorSet,
    Enum: Enum,
    Union: Union,
    Fn: Fn,
    Namespace: void,
    BoundFn: Fn,
    ArgTuple: void,
    Opaque: void,
    Promise: Promise,pubconst Int = struct {
        is_signed: bool,
        bits: u8,
    };pubconst Float = struct {
        bits: u8,
    };pubconst Pointer = struct {
        size: Size,
        is_const: bool,
        is_volatile: bool,
        alignment: u32,
        child: type,pubconst Size = enum {
            One,
            Many,
            Slice,
        };
    };pubconst Array = struct {
        len: usize,
        child: type,
    };pubconst ContainerLayout = enum {
        Auto,
        Extern,
        Packed,
    };pubconst StructField = struct {
        name: []constu8,
        offset: ?usize,
        field_type: type,
    };pubconst Struct = struct {
        layout: ContainerLayout,
        fields: []StructField,
        defs: []Definition,
    };pubconst Optional = struct {
        child: type,
    };pubconst ErrorUnion = struct {
        error_set: type,
        payload: type,
    };pubconst Error = struct {
        name: []constu8,
        value: usize,
    };pubconst ErrorSet = struct {
        errors: []Error,
    };pubconst EnumField = struct {
        name: []constu8,
        value: usize,
    };pubconst Enum = struct {
        layout: ContainerLayout,
        tag_type: type,
        fields: []EnumField,
        defs: []Definition,
    };pubconst UnionField = struct {
        name: []constu8,
        enum_field: ?EnumField,
        field_type: type,
    };pubconst Union = struct {
        layout: ContainerLayout,
        tag_type: ?type,
        fields: []UnionField,
        defs: []Definition,
    };pubconst CallingConvention = enum {
        Unspecified,
        C,
        Cold,
        Naked,
        Stdcall,
        Async,
    };pubconst FnArg = struct {
        is_generic: bool,
        is_noalias: bool,
        arg_type: ?type,
    };pubconst Fn = struct {
        calling_convention: CallingConvention,
        is_generic: bool,
        is_var_args: bool,
        return_type: ?type,
        async_allocator_type: ?type,
        args: []FnArg,
    };pubconst Promise = struct {
        child: ?type,
    };pubconst Definition = struct {
        name: []constu8,
        is_pub: bool,
        data: Data,pubconst Data = union(enum) {
            Type: type,
            Var: type,
            Fn: FnDef,pubconst FnDef = struct {
                fn_type: type,
                inline_type: Inline,
                calling_convention: CallingConvention,
                is_var_args: bool,
                is_extern: bool,
                is_export: bool,
                lib_name: ?[]constu8,
                return_type: type,
                arg_names: [][] constu8,pubconst Inline = enum {
                    Auto,
                    Always,
                    Never,
                };
            };
        };
    };
};

This kicks open the door for compile-time reflection, especially when combined with the fact that Jimmi Holst Christensen implemented @field, which performs field access with a compile-time known name:

const std = @import("std");const assert = std.debug.assert;test"@field" {const Foo = struct {
        one: i32,
        two: bool,
    };var f = Foo{
        .one = 42,
        .two = true,
    };const names = [][]constu8{ "two", "one" };

    assert(@field(f, names[0]) == true);
    assert(@field(f, names[1]) == 42);@field(f, "one") += 1;
    assert(@field(f, "on" ++ "e") == 43);
}

This has the potential to be abused, and so the feature should be used carefully.

After Jimmi implemented @field, he improved the implementation of @typeInfo and fixed several bugs. And now, the combination of these builtins is used to implement struct printing in userland:

const std = @import("std");const Foo = struct {
    one: i32,
    two: *u64,
    three: bool,
};pubfnmain() void {var x: u64 = 1234;var f = Foo{
        .one = 42,
        .two = &x,
        .three = false,
    };

    std.debug.warn("here it is: {}\n", f);
}

Output:

here it is: Foo{ .one = 42, .two = u64@7ffdda208cf0, .three = false }

See std/fmt/index.zig:15 for the implementation.

Now that we have @typeInfo, there is one more question to answer: should there be a function which accepts a TypeInfo value, and makes a type out of it?

This hypothetical feature is called @reify, and it's ahot topic. Although undeniably powerful and useful, there is concern that it would be too powerful, leading to complex meta-programming that goes against the spirit of simplicity that Zig stands for.

Improve cmpxchg

@cmpxchg is removed. @cmpxchgStrong and @cmpxchgWeak are added.

The functions have operand type as the first parameter.

The return type is ?T where T is the operand type.

New Type: f16

Ben Noordhuis implemented f16. This is guaranteed to be IEEE-754-2008 binary16 format, even on systems that have no hardware support, thanks to the additions to compiler_rt that Ben contributed. He also added support for f16 to std.math functions such as isnormal and fabs.

All Integer Sizes are Primitives

Zig 0.2.0 had primitive types for integer bit widths of 2-8, 16, 29, 32, 64, 128. Any number other than that, and you had to use @IntType to create the type. But you would get a compile error if you shadowed one of the above bit widths that already existed, for example with

constu29 = @IntType(false, 29); 

Needless to say, this situation was unnecessarily troublesome (#745). And so now arbitrary bit-width integers can be referenced by using an identifier ofi or u followed by digits. For example, the identifieri7 refers to a signed 7-bit integer.

u0 is a 0-bit type, which means:

• @sizeOf(u0) == 0
• No actual code is generated for loads and stores of this type of value.
• The value of a u0 as always the compile-time known value of 0.

i0doesn't make sense and will probably crash the compiler.

Although Zig defines arbitrary integer sizes to support all primitive operations, if you try to use, for example, multiplication on 256-bit integers:

test"large multiplication" {var x: u256 = 0xabcd;var y: u256 = 0xefef;var z = x * y;
}

Then you'll get an error like this:

LLVM ERROR: Unsupported library call operation!

Zig isn't supposed to be letting LLVM leak through here, but that's a separate issue. What's happening is that normally if a primitive operation such as multiplication of integers cannot be lowered to a machine instruction, LLVM will emit a library call to compiler_rt to perform the operation. This works for up to 128-bit multiplication, for example. However compiler_rt does not define an arbitrary precision multiplication library function, and so LLVM is not able to generate code.

It is planned to submit a patch to LLVM which adds the ability to emit a lib call for situations like this, and then Zig will include the arbitrary precision multiplication function in Zig's compiler_rt.

In addition to this, Zig 0.3.0 fixesa bug where@IntType was silently wrapping the bit count parameter if it was greater than pow(2, 32).

Improved f128 Support

Marc Tiehuis & Ben Noordhuis solved the various issues that prevented f128 from being generally useful.

• Fix hex-float parsing. -Marc Tiehuis (#495)
• Add compiler-rt functions to support f128. -Marc Tiehuis
  • __floatunditf
  • __floatunsitf
  • __floatunsitf
  • __floatuntitf
  • __floatuntisf
  • __trunctfdf2
  • __trunctfsf2
  • __floattitf
  • __floattidf
  • __floattisf
• Alignment fix and allow rudimentary f128 float printing. -Marc Tiehuis
• Fix f128 remainder division bug. The modulo operation computed rem(b+rem(a,b), b) which produces -1 for a=1 and b=2. Switch to a - b * trunc(a/b) which produces the expected result, 1. -Ben Noordhuis (#1137)

Build Artifact Caching

Zig now supports global build artifact caching. This feature is one of those things that you can generally ignore, because it "just works" without any babysitting.

By default, compilations are not cached. You can enable the global cache for a compilation by using --cache on:

andy@xps:~/tmp$ time zig build-exe hello.zig 

real	0m0.414s
user	0m0.369s
sys	0m0.049s
andy@xps:~/tmp$ time zig build-exe hello.zig --cache on
/home/andy/.local/share/zig/stage1/artifact/hkGO0PyaOKDrdg2wyhV1vRy0ATyTaT0s0ECa2BiHFJfsb9RDVKK_r3qwHI5gaEfv/hello

real	0m0.412s
user	0m0.377s
sys	0m0.038s
andy@xps:~/tmp$ time zig build-exe hello.zig --cache on
/home/andy/.local/share/zig/stage1/artifact/hkGO0PyaOKDrdg2wyhV1vRy0ATyTaT0s0ECa2BiHFJfsb9RDVKK_r3qwHI5gaEfv/hello

real	0m0.012s
user	0m0.009s
sys	0m0.003s

When the cache is on, the output is not written to the current directory. Instead, the output is kept in the cache directory, and the path to it is printed to stdout.

This is off by default, because this is an uncommon use case. The real benefit of build artifact caching comes in 3 places:

• zig run, where it is enabled by default:

  andy@xps:~/tmp$ time zig run hello.zig 
  Hello, world!
  
  real	0m0.553s
  user	0m0.500s
  sys	0m0.055s
  andy@xps:~/tmp$ time zig run hello.zig 
  Hello, world!
  
  real	0m0.013s
  user	0m0.007s
  sys	0m0.006s

• zig build, so that your build script only has to build once.
• When building an executable or shared library, Zig must build compiler_rt.o andbuiltin.o from source, for the given target. This usually only has to be done once ever, which is why other compilers such as gcc ship with these components already built. The problem with that strategy is that you have to build a special version of the compiler for cross-compiling. With Zig, you can always build for any target, on any target.

  So caching these artifacts provides a happy solution.

The cache is perfect; there are no false positives. You could even fix a bug in memcpy in the system's libc, and Zig will detect that its own code has (indirectly) been updated, and invalidate the cache entry.

If you use zig build-exe, Zig will still create a zig-cache directory in the current working directory in order to store an intermediate .o file. This is because on MacOS, the intermediate .o file stores the debug information, and therefore it needs to stick around somewhere sensible for Stack Traces to work.

Likewise, if you use zig test, Zig will put the test binary in the zig-cache directory in the current working directory. It's useful to leave the test binary here so that the programmer can use a debugger on it or otherwise inspect it.

The zig-cache directory is cleaner than before, however. For example, thebuiltin.zig file is no longer created there. It participates in the global caching system, just like compiler_rt.o. You can use zig builtin to see the contents of@import("builtin").

Compatibility with Valgrind

I noticed that valgrind does not see Zig's debug symbols (#896):

pubfnmain() void {
    foo().* += 1;
}fnfoo() *i32 {return@intToPtr(*i32, 10000000);
}

==24133== Invalid read of size 4
==24133==    at 0x2226D5: ??? (in /home/andy/downloads/zig/build/test)
==24133==    by 0x2226A8: ??? (in /home/andy/downloads/zig/build/test)
==24133==    by 0x222654: ??? (in /home/andy/downloads/zig/build/test)
==24133==    by 0x2224B7: ??? (in /home/andy/downloads/zig/build/test)
==24133==    by 0x22236F: ??? (in /home/andy/downloads/zig/build/test)

After digging around, I was able to reproduce the problem using only Clang and LLD:

static int *foo(void) {
    return (int *)10000000;
}

int main(void) {
    int *x = foo();
    *x += 1;
}

If this C code is built with Clang and linked with LLD, Valgrind has the same issue as with the Zig code.

I sent a message to the Valgrind mailing list, and theysuggested submitting a bug fix to Valgrind. That's a good idea. I'm a little busy with Zig development though - anybody else want to take a crack at it?

In the meantime, Zig now has a --no-rosegment flag, which works around the bug. It should only be used for this purpose; the flag will likely be removed once Valgrind fixes the issue upstream and enough time passes that the new version becomes generally available.

$ zig build-exe test.zig --no-rosegment
$ valgrind ./test
==24241== Invalid read of size 4
==24241==    at 0x221FE5: main (test.zig:2)

Zig is now on Godbolt Compiler Explorer

Marc Tiehuis added Zig support, and then worked with theCompiler Explorer team to get it merged upstream and deployed.

The command line API that Compiler Explorer uses is covered by Zig's main test suite to ensure that it continues working as the language evolves.

zig init-lib and init-exe

zig init-lib can be used to initialize azig build project in the current directory which will create a simple library:

$ zig init-lib
Created build.zig
Created src/main.zig

Next, try `zig build --help` or `zig build test`
$ zig build test
Test 1/1 basic add functionality...OK
All tests passed.

Likewise, zig init-exe initializes a simple application:

$ zig init-exe
Created build.zig
Created src/main.zig

Next, try `zig build --help` or `zig build run`
$ zig build run
All your base are belong to us.

The main Zig test suite tests this functionality so that it will not regress as Zig continues to evolve.

Concurrency Status

Concurrency is now solved. That is, there is a concrete plan for how concurrency will work in Zig, and now it's a matter of implementing all the pieces.

First and foremost, Zig supports low-level control over hardware. That means that it has atomic primitives:

...and it means that you can directly spawn kernel threads using standard library functions:

const std = @import("std");const assert = std.debug.assert;const builtin = @import("builtin");const AtomicRmwOp = builtin.AtomicRmwOp;const AtomicOrder = builtin.AtomicOrder;test"spawn threads" {var shared_ctx: i32 = 1;const thread1 = try std.os.spawnThread({}, start1);const thread2 = try std.os.spawnThread(&shared_ctx, start2);const thread3 = try std.os.spawnThread(&shared_ctx, start2);const thread4 = try std.os.spawnThread(&shared_ctx, start2);

    thread1.wait();
    thread2.wait();
    thread3.wait();
    thread4.wait();

    assert(shared_ctx == 4);
}fnstart1(ctx: void) u8 {return0;
}fnstart2(ctx: *i32) u8 {
    _ = @atomicRmw(i32, ctx, AtomicRmwOp.Add, 1, AtomicOrder.SeqCst);return0;
}

On POSIX targets, when you link against libc, the standard library uses pthreads; otherwise it uses its own lightweight kernel thread implementation.

You can use mutexes, signals, condition variables, and all those things. Anything you can accomplish in C, you can accomplish in Zig.

However, the standard library provides a higher level concurrency abstraction, designed for optimal performance, debuggability, and structuring code to closely model the problems that concurrency presents.

The abstraction is built on two language features: stackless coroutines andasync/await syntax. Everything else is implemented in userland.

std.event.Loop creates a kernel thread pool matching the number of logical CPUs. It can then be used for non-blocking I/O that will be dispatched across the thread pool, using the platform-native API:

This is a competitor to libuv, except multi-threaded.

Once you have an event loop, all of the std.event API becomes available to use:

• std.event.Channel - Many producer, many consumer, thread-safe, runtime configurable buffer size. When buffer is empty, consumers suspend and are resumed by producers. When buffer is full, producers suspend and are resumed by consumers.
• std.event.Future - A value that many consumers can await.
• std.event.Group - A way to await multiple async operations.
• std.event.Lock - Ensures only one thread gets access to a resource, without blocking a kernel thread.
• std.event.RwLock - Same as Lock except allows multiple readers to access data simultaneously.
• std.event.fs - File system operations based onasync/await syntax.
• std.event.tcp - Network operations based onasync/await syntax.

All of these abstractions provide convenient APIs based on async/await syntax, making it practical for API users to model their code with maximally efficient concurrency. None of these abstractions block or use mutexes; when an API user must suspend, control flow goes to the next coroutine waiting to run, if any. If no coroutines are waiting to run, the application will sit idly, waiting for an event from the respective platform-native API (e.g. epoll on Linux).

As an example, here is a snippet from a test in the standard library:

asyncfntestFsWatch(loop: *Loop) !void {const file_path = try os.path.join(loop.allocator, test_tmp_dir, "file.txt");defer loop.allocator.free(file_path);const contents =\\line 1\\line 2    ;const line2_offset = 7;tryawaittryasync fs.writeFile(loop, file_path, contents);const read_contents = tryawaittryasync fs.readFile(loop, file_path, 1024 * 1024);
    assert(mem.eql(u8, read_contents, contents));var watch = try fs.Watch(void).create(loop, 0);defer watch.destroy();

    assert((tryawaittryasync watch.addFile(file_path, {})) == null);const ev = tryasync watch.channel.get();var ev_consumed = false;deferif (!ev_consumed) cancel ev;const fd = tryawaittryasync fs.openReadWrite(loop, file_path, os.File.default_mode);
    {defer os.close(fd);tryawaittryasync fs.pwritev(loop, fd, [][]constu8{"lorem ipsum"}, line2_offset);
    }

    ev_consumed = true;switch ((tryawait ev).id) {
        WatchEventId.CloseWrite => {},
        WatchEventId.Delete => @panic("wrong event"),
    }const contents_updated = tryawaittryasync fs.readFile(loop, file_path, 1024 * 1024);
    assert(mem.eql(u8, contents_updated,\\line 1\\lorem ipsum    ));
}

You can see that even though Zig is a language with manual memory management that insists on handling every possible error, it manages to be quite high level using these event-based APIs.

Now, there are some problems to solve:

• The way that canceling a coroutine works is currently unsound. - I know how to fix this, but it'll have to be in 0.4.0. Unfortunately it's causing occasional test failures.
• Lack of a guarantee about whether an async function call allocates memory or not. - In theory, there are many cases where Zig should be able to guarantee that an async function call will not allocate memory for the coroutine frame. However in practice, using LLVM's coroutines API, it will always result in an allocation.
• LLVM's coroutines implementation is buggy - Right now Zig sadly is forced to disable optimizations for async functions because LLVM has a bug where Mem2Reg turns correct coroutine frame spills back into incorrect parameter references.
• LLVM's coroutines implementation is slow - When I analyzed the compilation speed of Zig, even with optimizations off, LLVM takes up over 80% of the time. And for the zig behavioral tests, even though coroutines are a tiny percent of the code, LLVM's coroutine splitting pass takes up 30% of that time.

And so, the plan is to rework coroutines, without using any of LLVM's coroutines API. Zig will implement coroutines in the frontend, and LLVM will see only functions and structs. This ishow Rust does it, and I think it was a strong choice.

The coroutine frame will be in a struct, and so Zig will know the size of it at compile-time, and it will solve the problem of guaranteeing allocation elision - the async callsite will simply have to provide the coroutine frame pointer in order to create the promise.

This will also be relevant for recursion; stackless function calls do not count against the static stack size upper bound calculation. See Recursion Status for more details.

Self-Hosted Compiler Status

The self-hosted compiler is well underway.Here's a 1 minute demo of the self-hosted compiler watching source files and rebuilding.

The self-hosted compiler cannot do much more than Hello World at the moment, but it's being constructed from the ground up to fully take advantage of multiple cores and in-memory caching. In addition, Zig's error system and other safety features are making it easy to write reliable, robust code. Between stack traces, error return traces, and runtime safety checks, I barely even need a debugger.

Marc Tiehuis contributed a Big Integer library, which the self-hosted compiler is using for integer literals and compile-time math operations.

Writing the self-hosted compiler code revealed to me how coroutines should work in Zig. All the little details and ergonomics are clear to me now. And so before I continue any further on the self-hosted compiler, I will use this knowledge to rework coroutines and solve the problems with them. See Concurrency Status for more details.

As a reminder, even when the self-hosted compiler is complete, Zig will forever be stuck with the stage1 C++ compiler code. See The Grand Bootstrapping Plan for more details.

The self-hosted compiler is successfully sharing some C++ code with the stage1 compiler. For example the libLLVM C++ API wrapper is built into a static library, which then exports a C API wrapper. The self-hosted compiler links against this static library in order to make libLLVM C++ API calls via the C API wrapper. In addition, the Microsoft Visual Studio detection code requires the Windows COM API, which is also C++, and so a similar strategy is used. I think it's pretty neat that the build system builds a static library once and then ends up linking against it twice - one for each of the two compiler stages!

Recursion Status

I've said before that recursion is one of the enemies of perfect software, because it represents a way that a program can fail with no foolproof way of preventing it. With recursion, pick any stack size and I'll give you an input that will crash your program. Embedded developers are all too familiar with this problem.

It's always possible to rewrite code using an explicit stack using heap allocations, and that's exactly what Jimmi did in the self-hosted parser.

On the other hand, when recursion fits the problem, it's significantly more clear and maintainable. It would be a real shame to have to give it up.

I researched different ways that Zig could keep recursion, even when we introduce statically known stack upper bound size. I came up with a proof of concept for@newStackCall, a builtin function that calls a function using an explicitly provided new stack. You can find a usage example in the documentation by following that link.

This works, and it does break call graph cycles, but it would be a little bit awkward to use. Because if you allocate an entire new stack, it has to be big enough for the rest of the stack upper bound size, but in a recursive call, which should be only one stack frame, it would overallocate every time.

So that's why I think that the actual solution to this problem is Zig's stackless coroutines. Because Zig's coroutines are stackless, they are the perfect solution for recursion (direct or indirect). With the reworking of coroutines, it will be possible to put the coroutine frame of an async function anywhere - in a struct, in the stack, in a global variable - as long as it outlives the duration of the coroutine. See Concurrency for more details.

So - although recursion is not yet solved, we know enough to know that recursion is OK to use in Zig. It does suffer from the stack overflow issue today, but in the future we will have a compile error to prevent call graph cycles. And then this hypothetical compile error will be solved by using@newStackCall or stackless functions (but probably stackless functions). Once recursion is solved, if stackless functions turn out to be the better solution, Zig will remove @newStackCall from the language, unless someone demonstrates a compelling use case for it.

For now, use recursion whenever you want; you'll know when it's time to update your code.

WebAssembly Status

The pieces for web assembly are starting to come together.

Ben Noordhuis fixed support for --target-arch wasm32 (#1094).

LLVM merged my patch to make WebAssembly a normal (non-experimental) target. But they didn't do it before the LLVM 7 release. So Zig 0.3.0 will not have WebAssembly support by default, but 0.4.0 will.

That being said, the static builds of Zig provided by ziglang.org have the WebAssembly target enabled.

Apart from this, there appears to bean issue with Zig's WebAssembly linker. Once this is solved, all that is left is to use WebAssembly in real life use cases, to work out the ergonomics, and solve the inevitable issues that arise.

Documentation

The language reference documentation now contains no JavaScript. The code blocks are pre-formatted with std.zig.Tokenizer. The same is true for these release notes.

The builtin.zig example code in the documentation is now automatically updated from the output of Zig, so the docs can't get out of date for this.

In addition to the above, the following improvements were made to the documentation:

Standard Library API Changes

• std.mem.SplitIterator is now public
• std.math.atan2 is now public
• std.os.linux now makes public all the syscall numbers and syscall functions
• std.math.cast handles signed integers
• added std.zig.parse
• added std.zig.parseStringLiteral
• added std.zig.render
• added std.zig.ast
• added std.zig.Token
• added std.zig.Tokenizer
• added std.io.readLine
• replace File.exists with File.access. -Marc Tiehuis
• rename std.rand.Rand to std.rand.Random
• added common hash/checksum functions. -Marc Tiehuis
  • SipHash64, SipHash128
  • Crc32 (fast + small variants)
  • Adler32
  • Fnv1a (32, 64 and 128 bit variants)
• Add Hmac function -Marc Tiehuis
• Added timestamp, high-perf. timer functions -tgschultz
  • std.os.time.sleep
  • std.os.time.posixSleep
  • std.os.time.timestamp
  • std.os.time.miliTimestamp
  • std.os.time.Timer
• Added complex number support. -Marc Tiehuis
  • std.math.complex.Complex
  • std.math.complex.abs
  • std.math.complex.acos
  • std.math.complex.acosh
  • std.math.complex.arg
  • std.math.complex.asin
  • std.math.complex.asinh
  • std.math.complex.atan
  • std.math.complex.atanh
  • std.math.complex.conj
  • std.math.complex.cos
  • std.math.complex.cosh
  • std.math.complex.exp
  • std.math.complex.log
  • std.math.complex.pow
  • std.math.complex.proj
  • std.math.complex.sinh
  • std.math.complex.sin
  • std.math.complex.sqrt
  • std.math.complex.tanh
  • std.math.complex.tan
  • std.math.complex.ldexp_cexp
• Added more slice manipulation functions. Thanks Braedon Wooding for the original PR. (#944)
  • std.mem.trimLeft
  • std.mem.trimRight
  • std.mem.trimRight
  • std.mem.lastIndexOfScalar
  • std.mem.lastIndexOfAny
  • std.mem.lastIndexOf
  • std.mem.endsWith
• Added std.atomic.Stack
• Added std.atomic.Queue
• Added std.os.spawnThread. It works on all targets. On Linux, when linking libc, it uses pthreads, and when not linking libc, it makes syscalls directly.
• Add JSON decoder. -Marc Tiehuis
  • std.json.Token
  • std.json.StreamingParser
  • std.json.TokenStream
  • std.json.validate
  • std.json.ValueTree
  • std.json.ObjectMap
  • std.json.Value
  • std.json.Parser - A non-stream JSON parser which constructs a tree of Value's.
• Added std.SegmentedList
• Removed functions from std.Buffer. Instead users should use std.io.BufferOutStream.
  • Removed std.Buffer.appendFormat
  • Removed std.Buffer.appendByte
  • Removed std.Buffer.appendByteNTimes
• Add arbitrary-precision integer to std. -Marc Tiehuis
  • std.math.big.Int
  • std.math.big.Limb
  • std.math.big.DoubleLimb
  • std.math.big.Log2Limb
• std.os.Dir gains Windows support.
• std.os.File.access no longer depends on shlwapi.dll on Windows.
• std.os.path.dirname returns null instead of empty slice when there is no directory component. This makes it harder to write bugs. (#1017)
• Reading from a file can return error.IsDir.
• Added std.math.floatMantissaBits and std.math.floatExponentBits -Marc Tiehuis
• std.mem.Allocator allows allocation of any 0 sized type, not just void. -Jimmi Holst Christensen.
• Added std.os.cpuCount
• Added std.sort.asc and std.sort.desc -Marc Tiehuis
• std.fmt.format add * for formatting things as pointers. (#1285)
• std.fmt.format add integer binary output format. -Marc Tiehuis (#1313)
• Added std.mem.secureZero. -Marc Tiehuis

  This is identical to mem.set(u8, slice, 0) except that it will never be optimized out by the compiler. Intended usage is for clearing secret data.

  The resulting assembly has been manually verified in --release-* modes.

  It would be valuable to test the 'never be optimized out' claim in tests but this is harder than initially expected due to how much Zig appears to know locally. May be doable with @intToPtr, @ptrToInt to get around known data dependencies but I could not work it out right now.

• std.fmt.format handles non-pointer struct/union/enums. Adds support for printing structs via reflection. (#1380)
• Many std.os file functions no longer require an allocator. They rely onPATH_MAX, because even Windows, Linux, and MacOS syscalls will fail for paths longer than PATH_MAX.
• Add std.crypto.chaCha20IETF and std.crypto.chaCha20With64BitNonce. -Shawn Landden & Marc Tiehuis
• Add poly1305 and x25519 crypto primitives. -Marc Tiehuis

  These are translated from monocypher which has fairly competitive performance while remaining quite simple.

  Initial performance comparision:
  

  Zig:
    Poly1305: 1423 MiB/s
    X25519:   8671 exchanges per second
  
  Monocypher:
    Poly1305: 1567 MiB/s
    X25519:   10539 exchanges per second

  There is room for improvement and no real effort has been made at all in optimization beyond a direct translation.
• Removed deprecated, unused Windows functions
  • std.os.windows.CryptAcquireContextA
  • std.os.windows.CryptReleaseContext
  • std.os.windows.CryptGenRandom

Thank you contributors!

• Tesla Ice Zhang fixed typos in the Zig grammar documentation and created The IntelliJ IDEA plugin for the Zig programming language
• Jay Weisskopf cleaned up the Zig documentation
• hellerve finished the Mac OS dir entry iterator code
• Raul Leal fixed an undeclared identifier error in readUntilDelimiterBuffer and incorrect number of parameters in readUntilDelimiterAlloc (#877)
• Wander Lairson Costa fixed the build process to find libxml2 and zlib correctly. (#847)
• tgschultz added more linux syscalls and constants to the std lib.
• tgschultz fixed compiler errors around Darwin code.
• Harry Eakins added readability improvements and a bug-fix to the standard library crypto throughput test.
• tgschultz Added DirectAllocator support for alignments bigger than os.page_size on posix systems. (#939)
• Braedon Wooding& Josh Wolfe Added UTF-8 encoding and decoding support. (#954)
• Alexandros Naskos Fixed a bug where comptime was being incorrectly applied across function definition boundaries. (#972)
• Braedon Wooding worked towards unifying the std.ArrayList and std.HashMap APIs regarding iteration. (#981)
• Braedon Wooding added documentation for arg types and error inference.
• tgschultz added custom formatter support to std.fmt.format.
• isaachier Fixed const-ness of buffer in std.Buffer.replaceContents method (#1065)
• isaachier Fixed error handling in std.Buffer.fromOwnedSlice. (#1082)
• Arthur Elliott Added std.ArrayList.setOrError so you can set a value without growing the underlying buffer, with range safety checks.
• marleck55 std/fmt: Use lowercase k for kilo in base 1000 (#1090)
• tgschultz added C string to fmt by using {s}. (#1092)
• Alexandros Naskos Fixed optional types of zero bit types. (#1110)
• Jay Weisskopf Made zig version compliant with SemVer with regards to the git revision metadata.
• Sahnvour fixed a compilation error on windows introduced by pointer reform.
• Bodie Solomon Fixed zig not finding std lib files on Darwin when the executable is a symlink. (#1117)
• Isaac Hier Fixed the increment operation for the comptime value -1.
• Isaac Hier Fixed the compiler's internal path joining function when the dirname is empty.
• tgschultz Fixed standard library regressions from updated syntax. (#1162)
• Isaac Hier Improved the compile error for when the RHS of a shift is too large for the LHS. (#1168)
• Jay Weisskopf Fixed version detection for out-of-source builds.
• Isaac Hier Fixed an assertion crash on enum switch values
• wilsonk Fixed a build error in the crypto throughput test (#1211)
• Bas van den Berg Fixed std.ArrayList.insert and added tests. (#1232)
• tgschultz Added std.ArrayList.swapRemove. (#1230)
• Eduardo Sánchez Muñoz fixed bad code generated when an extern function returns a small struct. (#1234)
• Bas van den Berg fixed aligned reallocation. (#1237)
• Bas van den Berg improved realloc on fixed buffer allocator. (#1238)
• Wink Saville gave ArrayList tests consistent names. (#1253)
• Wink Saville added std.ArrayList.swapRemoveOrError. (#1254)
• Jay Weisskopf Fixed minor documentation errors (#1256)
• kristopher tate Added more std.os.posix constants.
• kristopher tate Made tests skippable by returning error.SkipZigTest
• Nathan Sharp Added std.io.PeekStream and std.io.Slicestream.

  SliceStream is a read-only stream wrapper around a slice of bytes. It allows adapting algorithms which work on InStreams to in-memory data.

  PeekStream is a stream wrapper which allows "putting back" bytes into the stream so that they can be read again. This will help make look-ahead parsers easier to write.

• dbandstra added int writing functions to OutStream, and skipBytes function to InStream (#1300)
• dbandstra add SliceOutStream, rename SliceStream to SliceInStream (#1301)
• Matthew D. Steele added "Comments" section to language reference (#1309)
• kristopher tate Windows: Call RtlGenRandom() instead of CryptGetRandom() (#1319)
• kristopher tate Add builtin function @handle() (#1297)
• kristopher tate better support for `_` identifier (#1204, #1320)
• Matthew D. Steele Fix the start-less-than-end assertion in std.rand.Random.range (#1325)
• Matthew D. Steele Fix a type error in std.os.linux.getpid() (#1326)
• Matthew D. Steele Add thread ID support to std.os.Thread (#1316)
• Shawn Landdendoc: @addWithOverflow also returns if overflow occured
• Shawn Landdenadded a red-black tree implementation to std
• Wink Saville fixed @atomicRmw not type checking correctly.
• prazzb Fixed LLVM detection at build time for some linux distros. (#1378)
• tgschultz fixed handling of [*]u8 when no format specifier is set. (#1379)
• Shawn Landden do not use an allocator when we don't need to because of the existance of PATH_MAX
• Raul Leal Allow implicit cast from *[N]T to ?[*]T (#1398)
• kristopher tate Added a test for writing u64 integers (#1401)
• tgschultz Fixed compile error when passing enum to fmt
• tgschultz Implemented tagged union support in std.fmt.format (#1432)
• Raul Leal Allow implicit cast from *T and [*]T to ?*c_void
• kristopher tate correct version comparison for detecting msvc (fixes #1438)
• kristopher tate allow bytes to be printed-out as hex (#1358)
• Shawn Landden updated incorrect documentation comments (#1456)
• hfcc Added compilation error when a non-float is given to @floatToInt
• kristopher tate X25519: Fix createPublicKey signature and add test (#1480)
• Sahnvour Fixes a path corruption when compiling on windows. (#1488)
• Bas van den Berg Add capacity and appendAssumeCapacity to ArrayList
• emekoifixed WriteFile segfault
• kristopher tate fixed handling of file paths with spaces in the cache
• Wink Saville fixed build failures of FileOutStream/FileInStream from syntax changes
• emekoifixed compiling on mingw (#1542)
• Raul Leal added builtin functions: @byteOffsetOf and @bitOffsetOf.
• Christian Wesselhoeft fixed BufferOutStream import - it is defined in io.zig.
• Wink Saville fixed a typo in a doc comment
• Wink Saville fixed a build issue with GCC 8
• Wink Saville refactored some parsing code in the self-hosted compiler
• Jay Weisskopf improved the help output of the command line interface

Miscellaneous Improvements

• LLVM, Clang, and LLD dependencies are updated to 7.0.0.
• Greatly increased test coverage.
• std.os - getting dir entries works on Mac OS.
• allow integer and float literals to be passed to var params. See #623
• add @sqrt built-in function. #767
• The compiler exits with error code instead of abort() for file not found.
• Add @atomicLoad builtin.
• stage1 compiler defaults to installing in the build directory
• ability to use async function pointers
• Revise self-hosted command line interface
• Add exp/norm distributed random float generation. -Marc Tiehuis
• On linux, clock_gettime uses the VDSO optimization, even for static builds.
• Better error reporting for missing libc on Windows. (#931)
• Improved fmt float-printing. -Marc Tiehuis
  • Fix errors printing very small numbers
  • Add explicit scientific output mode
  • Add rounding based on a specific precision for both decimal/exp modes.
  • Test and confirm exp/decimal against libc for all f32 values. Various changes to better match libc.
• The crypto throughput test now uses the new std.os.time module. -Marc Tiehuis
• Added better support for unpure enums in tranlate C. -Jimmi Holst Christensen (#975)
• Made container methods that can be const, const. -Jimmi Holst Christensen
• Tagged union field access prioritizes members over enum tags. (#959)
• std.fmt.format supports {B} for human readable bytes using SI prefixes.
• Zig now knows the C integer sizes for OpenBSD. Thanks to Jan Schreib for this information. (#1016)
• Renamed integer literal type and float literal type to comptime_int and comptime_float. -Jimmi Holst Christensen
• @canImplicitCast is removed. Nobody will miss it.
• Allow access of array.len through a pointer. -Jimmi Holst Christensen
• Optional pointers follow const-casting rules. Any *T -> ?*T cast is allowed implicitly, even when it occurs deep inside the type, and the cast is a no-op at runtime.
• Add i128 compiler-rt div/mul support. -Marc Tiehuis
• Add target C int type information for msp430 target. #1125
• Add __extenddftf2 and __extendsftf2 to zig's compiler-rt.
• Add support for zig to compare comptime array values. -Jimmi Holst Christensen (#1167)
• Support --emit in test command. -Ben Noordhuis (#1175)
• Operators now throw a compiler error when operating on undefined values. -Jimmi Holst Christensen (#1185)
• Always link against compiler_rt even when linking libc. Sometimes libgcc is missing things we need, so we always link compiler_rt and rely on weak linkage to allow libgcc to override.
• Add compile error notes for where struct definitions are. (#1202)
• Add @popCount.
• Cleaner output from zig build when there are compile errors.
• new builder.addBuildOption API. -Josh Wolfe
• Add compile error for disallowed types in extern structs. (#1218)
• build system: add -Dskip-release option to test faster. -Andrew Kelley & Jimmi Holst Christensen
• allow == for comparing optional pointers. #658
• allow implicit cast of undefined to optional
• switch most windows calls to use W versions instead of A. (#534)
• Better anonymous struct naming. This makes anonymous structs inherit the name of the function they are in only when they are the return expression. Also document the behavior and provide examples. (#1243)
• compile error for @noInlineCall on an inline fn (#1133)
• stage1: use os_path_resolve instead of os_path_real to canonicalize imports. This means that softlinks can represent different files, but referencing the same absolute path different ways still references the same import.
• rename --enable-timing-info to -ftime-report to match clang, and have it print llvm's internal timing info.
• Binary releases now include the LICENSE file.
• Overhaul standard library api for getting random integers. -Josh Wolfe (#1578)

Bug Fixes

• fix incorrect compile error on inferred error set from async function #856
• fix promise->T syntax not parsed #857
• fix crash when compile error in analyzing @panic call
• fix compile time array concatenation for slices #866
• fix off-by-one error in all standard library crypto functions. -Marc Tiehuis
• fix use-after-free in BufMap.set() - Ben Noordhuis #879
• fix llvm assert on version string with git sha -Ben Noordhuis #898
• codegen: fix not putting llvm allocas together
• fix calling convention at callsite of zig-generated fns
• inline functions now must be stored in const or comptime var. #913
• fix linux implementation of self exe path #894
• Fixed looking for windows sdk when targeting linux. -Jimmi Holst Christensen
• Fixed incorrect exit code when build.zig cannot be created. -Ben Noordhuis
• Fix os.File.mode function. -Marc Tiehuis
• Fix OpqaueType usage in exported c functions. -Marc Tiehuis
• Added memmove to builtin.o. LLVM occasionally generates a dependency on this function.
• Fix std.BufMap logic. -Ben Noordhuis
• Fix undefined behavior triggered by fn inline test
• Build system supports LLVM_LIBDIRS and CLANG_LIBDIRS. -Ben Noordhuis
• The Zig compiler does exit(1) instead of abort() for file not found.
• Add compile error for invalid deref on switch target. (#945)
• Fix printing floats in release mode. -Marc Tiehuis (#564, #669, #928)
• Fix @shlWithOverflow producing incorrect results when used at comptime (#948)
• Fix labeled break causing defer in same block to fail compiling (#830)
• Fix compiler crash with functions with empty error sets. -Jimmi Holst Christensen (#762, #818)
• Fix returning literals from functions with inferred error sets. -Jimmi Holst Christensen (#852)
• Fix compiler crash for .ReturnType and @ArgType on unresolved types. -Jimmi Holst Christensen (#846)
• Fix compiler-rt ABI for x86_64 windows
• Fix extern enums having the wrong size. -Jimmi Holst Christensen (#970)
• Fix bigint multi-limb shift and masks. -Marc Tiehuis
• Fix bigint shift-right partial shift. -Marc Tiehuis
• translate-c: fix typedef duplicate definition of variable. (#998)
• fix comptime code modification of global const. (#1008)
• build: add flag to LLD to fix gcc 8 build. (#1013)
• fix AtomicFile for relative paths. (#1017)
• fix compiler assert when trying to unwrap return type type. -Jimmi Holst Christensen
• fix crash when evaluating return type has compile error. (#1058)
• Fix Log2Int type construction. -Marc Tiehuis
• fix std.os.windows.PathFileExists specified in the wrong DLL (#1066)
• Fix structs that contain types which require comptime. (#586)

  Now, if a struct has any fields which require comptime, such as type, then the struct is marked as requiring comptime as well. Same goes for unions.

  This means that a function will implicitly be called at comptime if the return type is a struct which contains a field of type type.

• fix assertion failure when debug printing comptime values
• fix @tagName handling specified enum values incorrectly. (#976, #1080)
• fix ability to call mutating methods on zero size structs. (#838)
• disallow implicit casts that break rules for optionals. (#1102)
• Fix windows x86_64 i128 ABI issue. -Marc Tiehuis
• Disallow opaque as a return type of function type syntax. (#1115)
• Fix compiler crash for invalid enums. (#1079, #1147)
• Fix crash for optional pointer to empty struct. (#1153)
• Fix comptime @tagName crashing sometimes. (#1118)
• Fix coroutine accessing freed memory. (#1164)
• Fix runtime libc detection on linux depending on locale. (#1165)
• Fix await on early return when return type is struct.
• Fix iterating over a void slice. (#1203)
• Fix crash on @ptrToInt of a *void (#1192)
• fix crash when calling comptime-known undefined function ptr. #880, #1212
• fix @setEvalBranchQuota not respected in generic fn calls. #1257
• Allow pointers to anything in extern/exported declarations (#1258) -Jimmi Holst Christensen
• Prevent non-export symbols from clobbering builtins. (#1263)
• fix generation of error defers for fns inside fns. (#878)
• Fixed windows getPos. -Jimmi Holst Christensen
• fix logic for determining whether param requires comptime (#778, #1213)
• Fixed bug in LLD crashing when linking twice in the same process. (#1289)
• fix assertion failure when some compile errors happen
• add compile error for non-inline for loop on comptime type
• add compile error for missing parameter name of generic function
• add compile error for ignoring return value of while loop bodies (#1049)
• fix tagged union initialization with a runtime void (#1328)
• translate-c: fix for loops and do while loops with empty body
• fix incorrectly generating an unused const fn global (#1277)
• Fix builtin alignment type. -Marc Tiehuis (#1235)
• fix handling multiple extern vars with the same name
• fix llvm assertion failure when building std lib tests for macos (#1417)
• fix false negative determining if function is generic
• fix @typeInfo unable to distinguish compile error vs no-payload (#1421, #1426)
• fix crash when var in inline loop has different types (#917, #845, #741, #740)
• add compile error for function prototype with no body (#1231)
• fix invalid switch expression parameter. (#604)
• Translate-c: Check for error before working on while loop body. -Jimmi Holst Christensen (#1445)
• use the sret attribute at the callsite when appropriate. Thanks to Shawn Landden for the original pull request. (#1450)
• ability to @ptrCast to *void. (#960)
• compile error instead of segfault for unimplemented feature. (#1103)
• fix incorrect value for inline loop. (#1436)
• compile errors instead of crashing for unimplemented minValue/maxValue builtins
• add compile error for comptime control flow inside runtime block (#834)
• update throughput test to new File API (#1468)
• fix compile error on gcc 7.3.0. Only set -Werror for debug builds, and only for zig itself, not for embedded LLD. (#1474)
• stage1: fix emit asm with explicit output file (#1473)
• stage1: fix crash when invalid type used in array type (#1186)
• stage1 compile error instead of crashing for unsupported comptime ptr cast (#955)
• stage1: fix tagged union with no payloads (#1478)
• Add compile error for using outer scoped runtime variables from a fn defined inside it. (#876)
• stage1: improve handling of generic fn proto type expr. (#902)
• stage1: compile error instead of incorrect code for unimplemented C ABI. (#1411, #1481)
• add support for partial C ABI compatibility on x86_64. (#1411, #1264)
• fix crash when var init has compile error and then the var is referenced (#1483)
• fix incorrect union const value generation (#1381)
• fix incorrect error union const value generation (#1442)
• fix tagged union with only 1 field tripping assertion (#1495)
• add compile error for merging non- error sets (#1509)
• fix assertion failure on compile-time @intToPtr of function
• fix tagged union with all void payloads but meaningful tag (#1322)
• fix alignment of structs. (#1248, #1052, #1154)
• fix crash when pointer casting a runtime extern function
• allow extern structs to have stdcallcc function pointers (#1536)
• add compile error for non-optional types compared against null (#1539)
• add compile error for @ptrCast 0 bit type to non-0 bit type
• fix codegen for @intCast to u0
• fix @bytesToSlice on a packed struct (#1551)
• fix implicit cast of packed struct field to const ptr (#966)
• implementation for bitcasting extern enum type to c_int (#1036)
• add compile error for slice.*.len (#1372)
• fix optional pointer to empty struct incorrectly being non-null (#1178)
• better string literal caching implementation

  We were caching the ConstExprValue of string literals, which works if you can never modify ConstExprValues. This premise is broken with `comptime var ...`.

  So I implemented an optimization in ConstExprValue arrays, where it stores a Buf * directly rather than an array of ConstExprValues for the elements, and then similar to array of undefined, it is expanded into the canonical form when necessary. However many operations can happen directly on theBuf *, which is faster.

  Furthermore, before a ConstExprValue array is expanded into canonical form, it removes itself from the string literal cache. This fixes the issue, because before an array element is modified it would have to be expanded.

  See #1076

• add compile error for casting const array to mutable slice (#1565)
• fix std.fmt.formatInt to handle upcasting to base int size
• fix comptime slice of pointer to array (#1565)
• fix comptime string concatenation ignoring slice bounds (#1362)
• stage1: unify 2 implementations of pointer deref. I found out there were accidentally two code paths in zig ir for pointer dereference. So this should fix a few bugs. (#1486)
• add compile error for slice of undefined slice (#1293)
• fix @compileLog having unintended side effects. (#1459)
• fix translate-c incorrectly translating negative enum init values (#1360)
• fix comptime bitwise operations with negative values (#1387, #1529)
• fix self reference through fn ptr field crash (#1208)
• fix crash on runtime index into slice of comptime type (#1435)
• fix implicit casting to *c_void (#1588)
• fix variables which are pointers to packed struct fields (#1121)
• fix crash when compile error evaluating return type of inferred error set. (#1591)
• fix zig-generated DLLs not properly exporting functions. (#1443)

This Release Contains Bugs

Zig has known bugs.

The first release that will ship with no known bugs will be 1.0.0.

Roadmap

• Redo coroutines without using LLVM Coroutines and rework the semantics. See #1363 and #1194.
• Tuples instead of var args. #208
• Well-defined copy-eliding semantics. #287
• Self-hosted compiler. #89
• Get to 100% documentation coverage of the language
• Auto generated documentation. #21
• Package manager. #943

Active External Projects Using Zig

Thank you financial supporters!

Special thanks to those who donate monthly. We're now at $1,349 of the $3,000 goal. I hope this release helps to show how much time I've been able to dedicate to the project thanks to your support.

• Lauren Chavis
• Raph Levien
• Stevie Hryciw
• Andrea Orru
• Harry Eakins
• Filippo Valsorda
• jeff kelley
• Martin Schwaighofer
• Brendon Scheinman
• Ali Anwar
• Adrian Sinclair
• David Joseph
• Ryan Worl
• Tanner Schultz
• Don Poor
• Jimmy Zelinskie
• Thomas Ballinger
• David Hayden
• Audun Wilhelmsen
• Tyler Bender
• Matthew
• Mirek Rusin
• Peter Ronnquist
• Josh Gentry
• Trenton Cronholm
• Champ Yen
• Robert Paul Herman
• Caius
• Kelly Wilson
• Steve Perkins
• Clement Rey
• Eduard Nicodei
• Christopher A. Butler
• Colleen Silva-Hayden
• Wesley Kelley
• Jordan Torbiak
• Mitch Small
• Josh McDonald
• Jeff
• Paul Merrill
• Rudi Angela
• Justin B Alexander
• Ville Tuulos
• shen xizhi
• Ross Cousens
• Lorenz Vandevelde
• Ivan
• Jay Weisskopf
• William L Sommers
• Gerdus van Zyl
• Anthony J. Benik
• Brian Glusman
• Furkan Mustafa
• Le Bach
• Jordan Guggenheim
• Tyler Philbrick
• Marko Mikulicic
• Brian Lewis
• Matt Whiteside
• Elizabeth Ryan
• Thomas Lopatic
• Patricio Villalobos
• joe ardent
• John Goen
• Luis Alfonso Higuera Gamboa
• Jason Merrill
• Andriy Tyurnikov
• Sanghyeon Seo
• Neil Henning
• aaronstgeorge@gmail.com
• Raymond Imber
• Artyom Kazak
• Brian Orr
• Frans van den Heuvel
• Jantzen Owens
• David Bremner
• Veit Heller
• Benoit Jauvin-Girard
• Chris Rabuse
• Jeremy Larkin
• Rasmus Rønn Nielsen
• Aharon sharim
• Stephen Oates
• Quetzal Bradley
• Wink Saville
• S.D.
• George K
• Jonathan Raphaelson
• Chad Russell
• Alexandra Gillis
• Pradeep Gowda
• david karapetyan
• Lewis
• stdev
• Wojciech Miłkowski
• Jonathan Wright
• Ernst Rohlicek
• Alexander Ellis
• bb010g
• Pau Fernández
• Krishna Aradhi
• occivink
• Adrian Hatch
• Deniz Kusefoglu
• Dan Boykis
• Hans Wennborg
• Matus Hamorsky
• Ben Morris
• Tim Hutt
• Gudmund Vatn
• Tobias Haegermarck
• Martin Angers
• Christoph Müller
• Johann Muszynski
• Fabio Utzig
• Eigil Skjæveland
• Harry
• moomeme
• xash
• bowman han
• Romain Beaumont
• Nate Dobbins
• Paul Anderson
• Jon Renner
• Karl Syvert Løland
• Stanley Zheng
• myfreeweb
• Dennis Furey
• Dana Davis
• Ansis Malins
• Drew Carmichael
• Doug Thayer
• Henryk Gerlach
• Dylan La Com
• David Pippenger
• Matthew Steele
• tumdum
• Alex Alex
• Andrew London
• Jirka Grunt
• Dillon A
• Yannik
• VilliHaukka
• Chris Castle
• Antonio D'souza
• Silicon
• Damien Dubé
• Dbzruler72
• McSpiros
• Francisco Vallarino
• Shawn Park
• Simon Kreienbaum
• Gregoire Picquot
• Silicas
• James Haggerty
• Falk Hüffner
• allan
• Ahmad Tolba
• jose maria gonzalez ondina
• Adrian Boyko
• Benedikt Mandelkow
• Will Cassella
• Michael Weber

Thank you Andrea Orru for sending me a giant box of Turkish Delight

On August 15th GitHub celebrated a major milestone: our main application is now running on the latest version of Rails: 5.2.1!

In total the project took a year and a half to upgrade from Rails 3.2 to Rails 5.2. Along the way we took time to clean up technical debt and improve the overall codebase while doing the upgrade. Below we’ll talk about how we upgraded Rails, lessons we learned and whether we’d do it again.

How did we do it?

Upgrading Rails on an application as large and as trafficked as GitHub is no small task. It takes careful planning, good organization, and patience. The upgrade started out as kind of a hobby; engineers would work on it when they had free time. There was no dedicated team. As we made progress and gained traction it became not only something we hoped we could do, but a priority.

Since GitHub is so important to our community, we can’t stop feature development or bug fixes in order to upgrade Rails.

Instead of using a long-running branch to upgrade Rails we added the ability to dual boot the application in multiple versions of Rails. We created two Gemfile.lock’s: one for the current version Gemfile.lock and one for the future version Gemfile_next.lock. The dual booting allows us to regularly deploy changes for the next version to GitHub without requiring long running branches or altering how production works. We do this by conditionally loading the code.

ifGitHub.rails_3_2?## 3.2 code (i.e. production a year and a half ago)elsifGitHub.rails_4_2?# 4.2 codeelse# all 5.0+ future code, ensuring we never accidentally# fall back into an old version going forwardend

Each time we got a minor version of Rails green we’d make the CI job required for all pushes to the GitHub application and start work on the next version. While we worked on Rails 4.1 a CI job would run on every push for 3.2 and 4.0. When 4.1 was green we’d swap out 4.1 for 4.0 and get to work on 4.2. This allowed us to prevent regressions once a version of Rails was green, and time for engineers to get used to writing code that worked in multiple versions of Rails.

The two versions that we deployed were 4.2 and 5.2. We deployed 4.2 because it was a huge milestone and was the first version of Rails that hadn’t been EOL’d yet (as an aside: we’d been backporting security fixes to 3.2 but not to 4.0+ so we couldn’t deploy 4.0 or 4.1. Rest assured your security is our top priority).

To roll out the Rails upgrade we created a careful and iterative process. We’d first deploy to our testing environment and requested volunteers from each team to click test their area of the codebase to find any regressions the test suite missed.

After fixing those regressions, we deployed in off-hours to a percentage of production servers. During each deploy we’d collect data about exceptions and performance of the site. With that information we’d fix bugs that came up and repeat those steps until the error rate was low enough be considered equal to the previous version. We merged the upgrade once we could deploy to full production for 30 minutes at peak traffic with no visible impact.

This process allowed us to deploy 4.2 and 5.2 with minimal customer impact and no down time.

Lessons Learned

The Rails upgrade took a year and a half. This was for a few reasons. First, Rails upgrades weren’t always smooth and some versions had major breaking changes. Rails improved the upgrade process for the 5 series so this meant that while 3.2 to 4.2 took 1 year, 4.2 to 5.2 only took 5 months.

Another reason is the GitHub codebase is 10 years old. Over the years technical debt builds and there’s bound to be gremlins lurking in the codebase. If you’re on an old version of Rails, your engineers will need to add more monkey patches or implement features that exist upstream.

Lastly, when we started it wasn’t clear what resources were needed to support the upgrade and since most of us had never done a Rails upgrade before, we were learning as we went. The project originally began with 1 full-time engineer and a small army of volunteers. We grew that team to 4 full-time engineers plus the volunteers. Each version bump meant we learned more and the next version went even faster.

Through this work we learned some important lessons that we hope make your next upgrade easier:

• Upgrade early and upgrade often. The closer you are to a new version of Rails, the easier upgrades will be. This encourages your team to fix bugs in Rails instead of monkey-patching the application or reinventing features that exist upstream.
• Keep upgrade infrastructure in place. There will always be a new version to upgrade to, so once you’re on a modern version of Rails add a build to run against the master branch. This will catch bugs in Rails and your application early, make upgrades easier, and increase your upstream contributions.
• Upstream your tooling instead of rolling your own. The more you push upstream to gems or Rails, the less logic you need in your application. Save your application code for what truly makes your company special (i.e. Pull Requests), instead of tools to make your application run smoothly (i.e. concurrent testing libraries)
• Avoid using private API’s in your frameworks. Rails has a lot of code that’s not private but isn’t documented on purpose. That code is subject to change without notice, so writing code that relies on private code can easily break in an upgrade.
• Address technical debt often. It’s easy to think “this is working, why mess with it”, but if no one knows how that code works, it can quickly become a bottleneck for upgrades. Try to prevent coupling your application logic too closely to your framework. Ensure that the line where your application ends and your framework begins is clear. You can do this by addressing technical debt before it becomes difficult to remove.
• Do incremental upgrades. Each minor version of Rails provides the deprecation warnings for the next version. By upgrading from 3.2 to 4.0, 4.0 to 4.1, etc we were able to identify problems in the next version early and define clear milestones.
• Keep up the momentum. Rails upgrades can seem daunting. Create ways in which your team can have quick wins to keep momentum going. Share the responsibility across teams so that everyone is familiar with the new version of the framework and prevent burnout. Once you’re on the newest version add a build to your app that periodically runs your suite against edge Rails so you can catch bugs in your code or your framework early.
• Expect things to break. Upgrades are hard and in an application as large as GitHub things are bound to break. While we didn’t take the site down during the upgrade we had issues with CI, local development, slow queries, and other problems that didn’t show up in our CI builds or click testing.

Was it worth it?

Absolutely.

Upgrading Rails has allowed us to address technical debt in our application. Our test suite is now closer to vanilla Rails, we were able to remove StateMachine in favor of Active Record enums, and start replacing our job runner with Active Job. And that’s just the beginning.

Rails upgrades are a lot of hard work and can be time-consuming, but they also open up a ton possibilities. We can push more of our tooling upstream to Rails, address areas of technical debt, and be one of the largest codebases running on the most recent version of Rails. Not only does this benefit us at GitHub, it benefits our customers and the open source community.

Mino Games is a mobile gaming studio that strives to build the best possible games. We produced the hit mobile game Mino Monsters, with over 15 million downloads. Our development studio is in Montreal, and we are building one of the top teams in the industry.

We are funded by the leading angel, institutional investors, and gaming companies from across the world. (Andreessen Horowitz, Y Combinator, Sybo Games).

We are rapidly growing our Montreal studio, and looking for world class talent to come join us.