Bill Gates made his largest single gift since the turn of the century, giving away Microsoft Corp. shares that accounted for 5 percent of his fortune, the world’s biggest.

The billionaire donated 64 million of the software maker’s shares to the Bill & Melinda Gates Foundation on June 6, according to Securities and Exchange Commission filings released Monday. The shares were valued at $4.6 billion at the time.

It’s the largest gift of Microsoft shares that Gates, 61, has made since 2000. He gave away $16 billion worth of the stock in 1999 and $5.1 billion a year later, according to calculations by Bloomberg.

"It’s a change in how he distributes that asset," said Ben Silverman, research director at InsiderScore, noting that Gates has been reducing his stake methodically for years through share sales to fund his foundation.

Spokesmen for Gates and Redmond, Washington-based Microsoft didn’t respond to requests for comment.

Historic Giving

Bill and Melinda Gates have given away about $35 billion of stock and cash since 1994, based on the value of the shares at the time of gifts, according to a review of Gates Foundation tax returns, annual reports and regulatory filings. Gates created the Giving Pledge in 2010 with billionaire investor Warren Buffett, and they have been joined by 168 others who promised to give the majority of their wealth to charity.

Gates probably has directed more than 700 million shares of Microsoft into the foundation, adjusting for stock splits, and he would be about $50 billion richer today had he kept them. The June gift represents 38 percent of his holding in the company and is the latest in a long line of Microsoft share disposals that have whittled his stake from 24 percent in 1996 to 1.3 percent now.

Watch Next: Bill Gates on Tropical Diseases, Trump and Brexit

Gates remains the richest person on earth after the donation with a fortune the Bloomberg Billionaires Index valued at $86 billion as of 10:40 a.m. Tuesday in New York. His donation once again puts Amazon.com Inc. founder Jeff Bezos close to the top spot, with a net worth of $84.5 billion.

Bezos, 53, whose fortune has surged 30 percent since Jan. 1, briefly leapfrogged Gates last month to become the world’s richest person on an intraday basis.

— With assistance by Brendan Coffey

There's been much focus on MongoDB, Elastic and Redis in terms of data exposure on the Internet due to their general popularity in the developer community. However, in terms of data volume it turns out that HDFS is the real juggernaut. To give you a better idea here's a quick comparison between MongoDB and HDFS:

Even though there are more MongoDB databases connected to the Internet without authentication in terms of data exposure it is dwarfed by HDFS clusters (25 TB vs 5 PB). Where are all these instances located?

Most of the HDFS NameNodes are located in the US (1,900) and China (1,426). And nearly all of the HDFS instances are hosted on the cloud with Amazon leading the charge (1,059) followed by Alibaba (507).

The ransomware attacks on databases that were widelypublicized earlier in the year are still happening. And they're impacting both MongoDB and HDFS deployments. For HDFS, Shodan has discovered roughly 207 clusters that have a message warning of the public exposure. And a quick glance at search results in Shodan reveals that most of the public MongoDB instances seem to be compromised. I've previously written on the reason behind these exposures but note that both products nowadays have extensive documentation on secure deployment.

Technical Details

If you'd like to replicate the above findings or perform your own investigations into data exposure, this is how I measured the above.

1. Download data using the Shodan command-line interface:

   shodan download --limit -1 hdfs-servers product:namenode

2. Write a Python script to measure the amount of exposed data (hdfs-exposure.py):

   from shodan.helpers import iterate_files, humanize_bytes
   from sys import argv, exit
   
   
   if len(argv) <=1 :
       print('Usage: {} <file1.json.gz> ...'.format(argv[0]))
       exit(1)
   
   
   datasize = 0
   clusters = {}
   
   
   # Loop over all the banners in the provided files
   for banner in iterate_files(argv[1:]):
       try:
           # Grab the HDFS information that Shodan gathers
           info = banner['opts']['hdfs-namenode']
           cid = info['ClusterId']
           # Skip clusters we've already counted
           if cid in clusters:
               continue
           datasize += info['Used']
           clusters[cid] = True
       except:
           pass
   
   
   print(humanize_bytes(datasize))
   

3. Run the Python script to get the amount of data exposed:

   $ python hdfs-exposure.py hdfs-data.json.gz
   5.0 PB

Imagine that you are a gubernatorial candidate who is making education and college preparedness a key facet of your campaign. Consider these two state average SAT scores.

                                    Quantitative            Verbal         Total

Connecticut                   450                       480             930

Mississippi                     530                       550             1080

Your data analysts assure you that this difference is statistically significant. You know that SAT scores are a strong overall metric for educational aptitude in general, and particularly that they are highly correlated with freshman year performance and overall college outcomes. Those who score higher on the test tend to receive higher college grades, are less likely to drop out in their freshman year, are more likely to complete their degrees in four or six years, and are more likely to gain full-time employment when they’re done.

You believe that making your state’s high school graduates more competitive in college admissions is a key aspect of improving the economy of the state. You also note that Connecticut has powerful teacher unions which represent almost all of the public teachers in the state, while Mississippi’s public schools are largely free of public teacher unions. You resolve to make opposing teacher unions in your state a key aspect of your educational platform, out of a conviction that getting rid of the unions will ultimately benefit your students based on this data.

Is this a reasonable course of action?

Anyone who follows major educational trends would likely be surprised at these SAT results. After all, Connecticut consistently places among the highest-achieving states in educational outcomes, Mississippi among the worst. In fact, on the National Assessment of Educational Progress (NAEP), widely considered the gold standard of American educational testing, Connecticut recently ranked as the second-best state for 4^th graders and the best for 8^th graders. Mississippi ranked second-to-worst for both 4^th graders and 8^th graders. So what’s going on?

The key is participation rate, or the percentage of eligible juniors and seniors taking the SAT, as this scatter plot shows.

As can be seen, there is a strong negative relationship between participation rate and average SAT score. Generally, the higher the percentage of students taking the test in a given state, the lower the average score. Why? Think about what it means for students in Mississippi, where the participation rate is 3%, to take the SAT. Those students are the ones who are most motivated to attend college and the ones who are most college-ready. In contrast, in Connecticut 88% of eligible juniors and seniors take the test. (Data.) This means that almost everyone of appropriate age takes the SAT in Connecticut, including many students who are not prepared for college or are only marginally prepared. Most Mississippi students self-select themselves out of the sample. The top performing quintile (20%) of Connecticut students handily outperform the top performing quintile of Mississippi students. Typically, the highest state average in the country is that of North Dakota—where only 2% of those eligible take the SAT at all.

In other words, what we might have perceived as a difference in education quality was really the product of systematic differences in how the considered populations were put together. The groups we considered had a hidden non-random distribution. This is selection bias.

*****

My hometown had three high schools – the local coed public high school (where I went), and both a boys and girls private Catholic high school. People involved with the private high schools liked to brag about the high scores their students scored on standardized tests – without bothering to mention that you had to score well on such a test to get into them in the first place. This is, as I’ve said before, akin to having a height requirement for your school and then bragging about how tall your student body is. And of course, there’s another set of screens involved here that also powerfully shape outcomes: private schools cost a lot of money, and so students who can’t afford to attend are screened out. Students from lower socioeconomic backgrounds have consistently lower performance on a broad variety of metrics, and so private schools are again advantaged in comparison to public. To draw conclusions about educational quality from student outcomes without rigorous attempts to control for differences in which students are sorted into which schools, programs, or pedagogies – without randomization – is to ensure that you’ll draw unjustified conclusions.

Here’s an image that I often use to illustrate a far broader set of realities in education. It’s a regression analysis showing institutional averages for the Collegiate Learning Assessment, a standardized test of college learning and the subject of my dissertation. Each dot is a college’s average score. The blue dots are average scores for freshmen; the red dots, for seniors. The gap between the red and blue dots shows the degree of learning going on in this data set, which is robust for essentially all institutions. The very strong relationship between SAT scores and CLA scores show the extent to which different incoming student populations – the inherent, powerful selection bias of the college admissions process – determine different test outcomes. (Note that very similar relationships are observed in similar tests such as ETS’s Proficiency Profile.) To blame educators at a school on the left hand side of the regression for failing to match the schools on the right hand side of the graphic is to punish them for differences in the prerequisite ability of their students.

Harvard students have remarkable post-collegiate outcomes, academically and professionally. But then, Harvard invests millions of dollars carefully managing their incoming student bodies. The truth is most Harvard students are going to be fine wherever they go, and so our assumptions about the quality of Harvard’s education itself are called into question. Or consider exclusive public high schools like New York’s Stuyvesant, a remarkably competitive institution where the city’s best and brightest students compete to enroll, thanks to the great educational benefits of attending. After all, the alumni of high schools such as Stuyvesant are a veritable Who’s Who of high achievers and success stories; those schools must be of unusually high quality. Except that attending those high schools simply doesn’t matter in terms of conventional educational outcomes. When you look at the edge cases – when you restrict your analysis to those students who are among the last let into such schools and those who are among the last left out – you find no statistically meaningful differences between them. Of course, when you have a mechanism in place to screen out all of the students with the biggest disadvantages, you end up with an impressive-looking set of alumni. The admissions procedures at these schools don’t determine which students get the benefit of a better education; the perception of a better education is itself an artifact of the admissions procedure. The screening mechanism is the educational mechanism.

Thinking about selection bias compels us to consider our perceptions of educational cause and effect in general. A common complaint of liberal education reformers is that students who face consistent achievement gaps, such as poor minority students, suffer because they are systematically excluded from the best schools, screened out by high housing prices in these affluent, white districts. But what if this confuses cause and effect? Isn’t it more likely that we perceive those districts to be the best precisely because they effectively exclude students who suffer under the burdens of racial discrimination and poverty? Of course schools look good when, through geography and policy, they are responsible for educating only those students who receive the greatest socioeconomic advantages our society provides. But this reversal of perceived cause and effect is almost entirely absent from education talk, in either liberal or conservative media.

Immigrant students in American schools outperform their domestic peers, and the reason is about culture and attitude, the immigrant’s willingness to strive and persevere, right? Nah. Selection bias. So-called alternative charters have helped struggling districts turn it around, right? Not really; they’ve just artificially created selection bias. At Purdue, where there is a large Chinese student population, I always chuckled to hear domestic students say “Chinese people are all so rich!” It didn’t seem to occur to them that attending a school that costs better than $40,000 a year for international students acted as a natural screen to exclude the vast number of Chinese people who live in deep poverty. And I had to remind myself that my 8:00 AM writing classes weren’t going so much better than my 2:00 PM classes because I was somehow a better teacher in the mornings, but because the students who would sign up for an 8:00 AM class were probably the most motivated and prepared. There’s plenty ofdetailed work by people who know more than I do about the actual statistical impact of these issues and how to correct for them. But we all need to be aware of how deeply unequal populations influence our perceptions of educational quality.

Selection bias hides everywhere in education. Sometimes, in fact, it is deliberately hidden in education. A few years ago, Reuters undertook an exhaustive investigation of the ways that charter schools deliberately exclude the hardest-to-educate students, despite the fact that most are ostensibly required to accept all kinds of students, as public schools are bound to. For all the talk of charters as some sort of revolution in effective public schooling, what we find is that charter administrators work feverishly to tip the scales, finding all kinds of crafty ways to ensure that they don’t have to educate the hardest students to educate. And even when we look past all of the dirty tricks they use – like, say, requiring parents to attend meetings held at specific times when most working parents can’t – there are all sorts of ways in which students are assigned to charter schools non-randomly and in ways that advantage those schools. Excluding students with cognitive and developmental disabilities is a notorious example. (Despite what many people presume, a majority of students with special needs take state-mandated standardized tests and are included in data like graduation rates, in most locales.) Simply the fact that parents typically have to opt in to charter school lotteries for their students to attend functions as a screening mechanism.

Large-scale studies of charter efficacy such as Stanford’s CREDO project argue confidently that they have controlled for the enormous number of potential screening mechanisms that hide in large-scale education research. These researchers are among the best in the world and I don’t mean to disparage their work. But given the enormity of the stakes and the truth of Campbell’s Law, I have to report that I remain skeptical that we have truly ever controlled effectively for all the ways that schools and their leaders cook the books and achieve non-random student populations. Given that random assignment to condition is the single most essential aspect of responsible social scientific study, I think caution is warranted. And as I’ll discuss in a post in the future, the observed impact of school quality on student outcomes in those cases where we have the most confidence in the truly random assignment to condition is not encouraging.

I find it’s nearly impossible to get people to think about selection bias when they consider schools and their quality. Parents look at a private school and say, look, all these kids are doing so well, I’ll send my troubled child and he’ll do well, too. They look at the army of strivers marching out of Stanford with their diplomas held high and say, boy, that’s a great school. And they look at the Harlem Children’s Zone schools and celebrate their outcome metrics, without pausing to consider that it’s a lot easier to get those outcomes when you’re constantly expelling the students most predisposed to fail. But we need to look deeper and recognize these dynamics if we want to evaluate the use of scarce educational resources fairly and effectively.

Tell me how your students are getting assigned to your school, and I can predict your outcomes – not perfectly, but well enough that it calls into question many of our core presumptions about how education works.

I’m a female engineer on the software team at YC. This is the sixth installment in a series where we ask female engineers questions and share their candid, anonymous responses. In this post, we did something a bit different: we asked engineers to answer questions about the recent memo by former Google engineer James Damore. The engineers responding to these questions were given pseudonyms, and those pseudonyms are consistent through the series. Edith, in this post, is the same person named Edith in previous posts.

When I read the memo, I thought Damore made some points worth discussing: that PC culture can be stifling, that echo chambers are unproductive, and that corporate diversity practices don’t always work well. But I also disagreed with a lot of what he wrote, particularly his arguments pointing to biological factors as a primary reason that there aren’t more female software engineers. This argument seems like a distraction from the point that there is an uneven playing field for women who work as software engineers. No one should have to battle preconceived opinions about what they will enjoy or what they will do well at.

There’s been a lot of anger on both sides, but I haven’t seen many constructive discussions between people who disagree on these issues. I and all the women who have contributed in this post feel that there’s forward progress to be made by finding common ground and discussing different viewpoints without yelling. I hope this will be a good forum for that.

Did you read the full memo by James Damore?

Frances: Yes, I read the full memo. I also watched the first interview he gave on YouTube post-firing.

Edith: I did read it all. I haven’t watched any of his taped interviews since, though.

Ida: I read the full memo, at least twice.

What do you agree with, or find to be interesting ideas worth discussing?

Edith: I do think Damore is right that there is a lot of tension in our society about what is “okay” free speech, what tolerance means, what political correctness means, and how this all links to political views. I’m extremely liberal myself (though I come from a family of conservatives) and I acknowledge that a lot of people feel they can’t share their beliefs, and that they feel marginalized in places that overwhelmingly trend toward one political view or the other. I think as a society right now, with an unprecedented number of ways to get our opinions out to the world, we’re all finding it hard to navigate that appropriately. There’s a need to understand what “free speech” actually means versus how to be respectful, non-threatening and productive.

I also agree that there are a lot of differences between men and women with respect to things they may be interested in. Please note that I’m breaking this down along a binary here the way that he has, with the recognition that gender and sex aren’t binary, and that these arguments about innate biological traits are complicated by trans, non-binary, and intersex folks.

Ida: I agree that there is an ideological echo chamber inside Google (I’m a SWE there). As someone who is generally on the “correct” side of this liberal echo chamber, it hasn’t affected me much, but I think the vehemence of the reaction to his document proves this point right.

I also think that society should have room for the discussion of ideas that are not in step with what is considered acceptable at a given time. There is, of course, a difference between an unfounded opinion and the pursuit of scientific truth, but logically, we should not avoid pursuing a scientific truth for fear that the answer will not be aligned with currently accepted dogmas.

I also agree that there are differences between the behavior of men and women, on average.

Frances: I do agree the current state of political correctness makes it hard for people with unpopular opinions to ask questions and discuss their viewpoints. I do think the left can be extreme and hostile toward more conservative views. The internet allows everyone to publish knee-jerk, couch-expert reactions to statements made by colleagues and strangers alike, which makes it challenging to have constructive, thoughtful debate. I do agree we should reconsider existing diversity practices and what their desired outcomes should be. I do agree striving to have 50% female representation in the tech workforce may not be the most effective goal and it’s worth re-examining our metrics. For example, I’d be more interested in metrics that track retention and employee satisfaction rather than focusing so heavily on quotas and KPIs related to hiring women.

What do you disagree with or find objectionable?

Edith: I disagree completely and utterly that the (yes, real) average differences between men and women map to being better or worse at certain jobs. Interest in certain jobs, certainly. And we know – and many of us have experienced – that interest levels are also heavily influenced by social and cultural factors. For example, students and professors I met in college that grew up in the USSR thought engineering was stereotypically women’s work. But ability to do those jobs? No research I’ve ever read – and I have read a lot, because since I was 15 and fell in love with engineering it’s been made very clear to me that a lot of people don’t believe women are suited for the work I like to do – has indicated that the differences are so significant as to suggest that men or women are better or worse on average at any job that relies on mental work.

I disagree that it’s possible to write what he did about general populations, then walk it back to say “but of course it doesn’t apply at an individual level.” A lot of people have used that argument in defense of what he wrote, as evidence that the memo was not harmful or hostile to the women he worked with. When I walk into my job at a tech company, how do I know which of my colleagues thinks I’m an outlier among women versus someone who was hired because I’m female that doesn’t deserve the job they have? How do I prove myself to people one way or another? The additional mental and emotional burden on me just to do my job is not negligible at all, and it’s also a pretty crappy way to start every day thinking: “Will the team/manager/VC I talk with today realize I’m qualified, or will they be making stereotypical assumptions about my abilities and therefore make it harder for me to do my job?” To me, that absolutely makes for a hostile work environment, and it’s an unequal burden my male coworkers don’t have to deal with every day.

Frances: I disagree with his use of science and data to convert opinions into facts. It seemed like he cherry-picked research that agreed with his views and didn’t seek dissenting research or opinions before sending the document to internal Google groups.

I think it’s far from clear why kids choose the interests that they choose. Growing up, I was more interested in activities that were dominated by boys. I don’t know why my interests turned out that way. My parents encouraged my interest in science and technology – did that play a role? I have no idea. But I was not convinced by the research he cited to suggest biology dictates those preferences. He did not address any counter arguments or research that opposes his views, or the validity of the studies he did cite and their reproducibility. He speaks about confirmation bias in the memo – and I wonder if, just like his footnote suggests might be the case, he fell victim to just that by focusing on views and research that proved his points and skipping the rest. Also, his skepticism of his own views deserves a much more prominent placement in the text than a footnote – had he led with this and made it clear he wasn’t sure whether he was correct and simply wanted to start a discussion (as he subsequently stated in a YouTube interview), he likely would not have been blasted the same way.

Ida: He claimed that Google’s diversity efforts represent a lowering of the bar. Google has stated many times that its efforts involve focusing more resources on searching for candidates in minority groups rather than lowering the bar for these groups. Such misrepresentation is harmful to those of us at Google who have to overcome the bias that we were hired based other factors beside our skills.

I don’t really see how it’s useful to have a discussion of general group traits in a work setting. Assuming that it’s true that women on average are more likely to have trait X, why should any woman have to overcome the additional barrier of proving that she’s not like other women, or that if she IS like other women, that the trait has no bearing on her job performance?

Some reactions to this answer will be along the lines of “but you’re not disagreeing with the substance of what he’s saying, you’re disagreeing with the form!” and that’s true. Nevertheless, I maintain that when I go to work, I go to work, and not to a debate club. Some people at Google reacted by saying “well if he’s so wrong, then why not refute him,” but that requires spending a significant amount of time building an argument against the claims in his document. On the other hand, if I remain silent, that silence could be mistaken for agreement. I should not be forced into that kind of debate at work.

Due in part to the aforementioned fear of my silence being misinterpreted as agreement, I’d like to mention that this is not a complete list of the items that I disagree with.

How do you feel about the way the internet reacted to his post?

Edith: Emotionally drained, mostly. The general advice of “don’t read the comments” and “don’t feed the trolls” is hard to remember when the takeaway from the memo is literally that the onus is on me to prove to men in tech that I’m not an “average” woman – it’s so hard to see so many defenses fly by full of inaccuracies and problematic claims that I know I shouldn’t spend the time and energy to respond to, but I feel like if I don’t, they “win”. I’ve been deeply disappointed to see a number of big names in tech defend this in ways I find really frustrating – like Paul Graham suggesting in a tweet that the strong reaction is due to “worry [the claims in the memo] might be true.” (No, I’m just exhausted by having this same damn argument over and over again since I was a teenager and the amount of time and energy I keep having to spend to counter it.) There is a whole spectrum of reaction that influential tech leaders could take on this, and I’m really bothered by the number I see that are on the far end of dismissal of the hurt and damage many of us are experiencing.

On the other hand, there have been some really fabulous responses, including many laying out a lot of research that counters what was in the memo, and I’ve found a lot of those informative and inspiring and encouraging.

Overall, I guess I am surprised that this has picked up the way it has – I realize Damore’s linking of his opinions about women in tech to his politics are hitting on two huge clickbait-y topics these days, but seeing the far-reaching reactions from people outside the tech industry is a little surprising to me. I guess I do see why it’s pushing so many peoples’ buttons, though.

Frances: I feel disappointed and tired. I’m disappointed that the left reacted with rage and sass and wasn’t able to calmly explain the failures of his text and have a constructive debate. I’m also disappointed that the men I know, including most of my male colleagues, remained silent on the topic. And the ones that did participate, either seemed to support Damore or demonstrated a fundamental lack of understanding for the issues women engineers are faced with and care about. As a female engineer, you don’t get to just love coding, or love problem solving and hacking on hardware – you also have to figure out how to navigate men who seem to demonstrate with words and actions that they don’t consider you an equal, that they consider you less smart or capable, or that they assume you don’t have as much expertise as your peers.

I wish more successful men in tech thought deeply about the advantages they’ve had – the situations in which they were more likely to be trusted, deemed competent, promoted, given raises, etc. as men than they would be as women. This exercise isn’t intended to place blame, but to inspire empathy toward those who feel the weight of their gender each day at work.

Many powerful men in Silicon Valley have huge bases of social media followers. By remaining silent on this topic or tweeting support for Damore, they are sending a message that philosophical arguments and principles take precedence over the lived experiences of many smart, talented female engineers and technical founders. I wish Damore, who is clearly intelligent and capable, had thought more about how his female co-workers would be impacted by his statements, how they might be experiencing the world. I wish he had talked to some of them and had considered incorporating some of their viewpoints into his own. I bet a lot of his female co-workers would have found some of his viewpoints interesting and perhaps even supported them; I know I might have.

Katherine: Twitter is the worst way to have this debate but that’s where it mostly was taking place (with a sprinkling of medium posts and malformed news pieces to complement it). Given that we all have different values and morals, the very foundation of the debate is already shaky. So when shoved into short form, we’ve set it up for complete failure. A lot of people are actually just talking past each other, arguing on different playing fields but yelling nevertheless, because well, we are all having our values and morals questioned.

Ida: The reaction of the internet has been completely unsurprising. I think it’s a shame, but also true, that we’ve reached a point where the first reaction to hearing something objectionable is to cry havoc and let slip the dogs of war, to become enraged, to accuse people of saying things that they aren’t actually saying, escalating all the way up to actual death threats. The discussion around this has followed the trajectory of most of the polarizing mass discussion in the last few years; everyone comes out the other side with their opinions more calcified than ever, and more convinced than before of the intractability of the other side.

We’ve also reached the point where truth doesn’t really matter – reactions on both sides have been rife with mischaracterizations, misstatements and overheated rhetoric. I don’t want this response to turn into a jeremiad, but I think it’s a real shame that we cannot disagree rationally and calmly, while sticking to what is actually true. If the goal is to change minds, then the current situation is not working, and it’s difficult to see how this will improve any time soon.

Do you have any thoughts on how his post could have been delivered in a way that caused less uproar and more meaningful conversation?

Edith: There’s a difference between “let’s have a discussion” and “let me tell you what’s up, all you wrong people.” An example I gave someone else of a better way to start a genuine discussion would be, “I find myself having trouble understanding what backs up Google’s diversity and inclusion programs, and that left me with a lot of questions about how the company manages this process while maintaining our very high standards. Can I see some of the research at the foundation of all this? Who might I talk to who could tell me about their experiences working within a system that is biased against them, so I can understand better? And also, I almost feel afraid to ask these questions because I worry I’ll just get shot down by the Google echochamber – I’d like people like me to feel more comfortable asking questions, so is there a way to ensure there’s a non-judgmental forum to do this with our VP D&I, especially for newer employees who have just gone through the training?”

Asking questions (versus making statements) is a great way to learn about the real experiences of people who aren’t like you, and tends to set people up to give you the benefit of the doubt in ways that will allow you to screw up occasionally without the wrath of god coming down on you. I also believe that conversations like this need to start with a much higher standard for engagement – this isn’t a political argument over tax rates or infrastructure policy. This is literally a discussion of whether half the human race is innately unsuited for a certain kind of work, work that is exciting, lucrative, prestigious, and actually very desirable to many of us. This is a discussion where I am not just defending my viewpoint, I am defending my right to do my chosen work without the emotional and mental tax of knowing some significant portion of my colleagues may not believe I can.

Even if the discussion is ostensibly brought about by purely objective logic and science (and that’s questionable, as many others have already pointed out in detail), the discussion itself cannot be divorced from the deeply personal and deeply human impact it has, and thus the standards for discussion must be very, very high. I get that the plural of anecdote is not data, so to some degree I understand his request to “de-emphasize empathy” and remove anecdotes from the equation, but that’s simply not possible in this sort of disagreement – you can’t remove the human element especially when there is a lot of acknowledged cultural baggage around it. You do need to find a balance between the data you have available (which is still often subject to bias, particularly in a lot of experiment design around human research and cognition) and the empathetic side, but you can’t discount one or the other completely and still have a respectful or meaningful conversation.

Frances: I thought the post showed a general lack of consideration for his female colleagues. If he had spoken with some of them individually and spent some time trying to better understand their views on the issues, I suspect he would have done a better job choosing words that would have inspired debate rather than hostility. If his intention was to persuade the left-leaning leaders at Google of concerns he had, he would need to understand how they think about diversity, how they would react to his words, and what the most effective way to persuade them might be. For example, he could have written an introduction stating that he had a lot of questions about diversity policies at Google and suspected it was an echo chamber, but he wanted to understand whether that was correct or not. That he’d spoken with many women before writing this piece, that he was proposing ideas and would like to talk about them and understand if they are reasonable. His tl;dr in particular was one of the most polarizing parts of what he wrote. I think he could have written it differently, so that people who chose not to read the whole 10-pages could have read the tl;dr and not immediately concluded he was sexist.

Ida: For one thing, I think that had he posted some variation on this in a non-work space, for example his blog, discussing diversity programs in general rather than at Google, it would have been a lot less of an issue. When you try to influence policy for a company, it is hardly surprising to see backlash inside the company. Perhaps he wasn’t expecting the vehemence of the backlash, but if he truly did not see that people would find aspects of his document objectionable, then he should find someone who can help him gauge such things in the future.

Which flows into my next point – presentation is important. This is an emotional topic and presenting one’s interpretations of research with an attitude of “this is a fait accompli because of this research I found, now just fall in line” is going to rub people the wrong way. Particularly since the inferences people draw from a descriptive social study may differ wildly. People’s hackles can be raised easily by taking an interpretation of research and presenting it as Truth.

Here is one example of someone who makes many of the same arguments, but posted them in a better venue and phrased them far less objectionably: http://slatestarcodex.com/2017/08/07/contra-grant-on-exaggerated-differences/

What’s your reaction to his firing?

Edith: I think it was the right thing for Google to do. There are consequences for things you say and do at work, for anyone and everyone, and had one of my coworkers circulated this I would absolutely perceive it as a hostile effort (even if it was not meant that way) to question me, my abilities, and my right to have my job the same as any of my other colleagues. As former Googler Yonatan Zunger wrote in his response, how on earth could he in good faith assign women to work with Damore after this? What does this publicity do to their ability to hire women and other underrepresented minorities in tech, or even to male allies who would look badly on Google for this? Not to mention the issues it presents to Google as a company given that they are under legal scrutiny for a gender-based wage gap. There are many ways Damore could have handled this very differently and possibly have gotten some traction with the people who could address his concerns or act on his suggestions without making himself a problem employee – he didn’t. Social skills are part of a professional skillset. It is important to learn how to handle difficult subjects in a workplace – we all have to do it. There are consequences for doing it in a way that causes problems for your employer, and I think in this case the consequences were appropriate. He was not fired for speaking truth to power, he was fired for mishandling a complex subject in a way that caused harm to his employer (and many of his colleagues).

Frances: I am conflicted about this and can see both sides of it. I think ideally there would have been a constructive internal discussion with him before the leak when he had shared the memo internally. Once the memo was leaked though, I understand how hard it would be for Google to keep him on staff and also support current and future female employees. It is really sad to watch him go from a confused, questionably informed kid, to a symbol of the alt-right, and to see his words polarize the issue even more.

Ida: I have mixed feelings about it. I can see that there are wide swaths of people who would refuse to work with him, and I am not sure that he can be unbiased when interviewing female candidates. Even if he does not believe that women are inherently worse than men at software engineering, I think it’s reasonable to be concerned that he might grade female candidates more harshly to counteract his perception of a lowered bar.

That being said, I can’t help but wonder if there was a useful discussion that could have been had. I think that variations on his opinion are held by people in the industry far more frequently than some (or many) people think, and it’s not productive to hide from that reality or shout down every expression of those opinions. That kind of opposition doesn’t change minds, and perhaps some minds could have been changed here.

Thanks to Aaron Stein, Michael Seibel, Kat Manalac, Adora Cheung, Sam Altman, Craig Cannon, Ramon Recuero, Scott Bell, and Daniel Gackle for reading drafts of this post.

When it comes to APIs, change isn’t popular. While software developers are used to iterating quickly and often, API developers lose that flexibility as soon as even one user starts consuming their interface. Many of us are familiar with how the Unix operating system evolved. In 1994, The Unix-Haters Handbook was published containing a long list of missives about the software—everything from overly-cryptic command names that were optimized for Teletype machines, to irreversible file deletion, to unintuitive programs with far too many options. Over twenty years later, an overwhelming majority of these complaints are still valid even across the dozens of modern derivatives. Unix had become so widely used that changing its behavior would have challenging implications. For better or worse, it established a contract with its users that defined how Unix interfaces behave.

Similarly, an API represents a contract for communication that can’t be changed without considerable cooperation and effort. Because so many businesses rely on Stripe as infrastructure, we’ve been thinking about these contracts since Stripe started. To date, we’ve maintained compatibility with every version of our API since the company’s inception in 2011. In this article, we’d like to share how we manage API versions at Stripe.

Code written to integrate with an API has certain inherent expectations built into it. If an endpoint returns a boolean field called verified to indicate the status of a bank account, a user might write code like this:

if bank_account[:verified]
  ...
else
  ...
end

If we later replaced the bank account’s verified boolean with a status field that might include the value verified (like we did back in 2014), the code will break because it depends on a field that no longer exists. This type of change is backwards-incompatible, and we avoid making them. Fields that were present before should stay present, and fields should always preserve their same type and name. Not all changes are backwards-incompatible though; for example, it’s safe to add a new API endpoint, or a new field to an existing API endpoint that was never present before.

With enough coordination, we might be able to keep users apprised of changes that we’re about to make and have them update their integrations, but even if that were possible, it wouldn’t be very user-friendly. Like a connected power grid or water supply, after hooking it up, an API should run without interruption for as long as possible.

Our mission at Stripe is to provide the economic infrastructure for the internet. Just like a power company shouldn’t change its voltage every two years, we believe that our users should be able to trust that a web API will be as stable as possible.

API versioning schemes

A common approach to allow forward progress in web APIs is to use versioning. Users specify a version when they make requests and API providers can make the changes they want for their next version while maintaining compatibility in the current one. As new versions are released, users can upgrade when it’s convenient for them.

This is often seen as a major versioning scheme with names like v1, v2, and v3 that are passed as a prefix to a URL (like /v1/widgets) or through an HTTP header like Accept. This can work, but has the major downside of changes between versions being so big and so impactful for users that it’s almost as painful as re-integrating from scratch. It’s also not a clear win because there will be a class of users that are unwilling or unable to upgrade and get trapped on old API versions. Providers then have to make the difficult choice between retiring API versions and by extension cutting those users off, or maintaining the old versions forever at considerable cost. While having providers maintain old versions might seem at first glance to be beneficial to users, they’re also paying indirectly in the form of reduced progress on improvements. Instead of working on new features, engineering time is diverted to maintaining old code.

At Stripe, we implement versioning with rolling versions that are named with the date they’re released (for example, 2017-05-24). Although backwards-incompatible, each one contains a small set of changes that make incremental upgrades relatively easy so that integrations can stay current.

The first time a user makes an API request, their account is automatically pinned to the most recent version available, and from then on, every API call they make is assigned that version implicitly. This approach guarantees that users don’t accidentally receive a breaking change and makes initial integration less painful by reducing the amount of necessary configuration. Users can override the version of any single request by manually setting the Stripe-Version header, or upgrade their account’s pinned version from Stripe’s dashboard.

Some readers might have already noticed that the Stripe API also defines major versions using a prefixed path (like /v1/charges). Although we reserve the right to make use of this at some point, it’s not likely to change for some time. As noted above, major version changes tend to make upgrades painful, and it’s hard for us to imagine an API redesign that’s important enough to justify this level of user impact. Our current approach has been sufficient for almost a hundred backwards-incompatible upgrades over the past six years.

Versioning under the hood

Versioning is always a compromise between improving developer experience and the additional burden of maintaining old versions. We strive to achieve the former while minimizing the cost of the latter, and have implemented a versioning system to help us with it. Let’s take a quick look at how it works. Every possible response from the Stripe API is codified by a class that we call an API resource. API resources define their possible fields using a DSL:

class ChargeAPIResource
  required :id, String
  required :amount, Integer
end

API resources are written so that the structure they describe is what we’d expect back from the current version of the API. When we need to make a backwards-incompatible change, we encapsulate it in a version change module which defines documentation about the change, a transformation, and the set of API resource types that are eligible to be modified:

class CollapseEventRequest  data[:request][:id])
    end
  end
end

Elsewhere, version changes are assigned to a corresponding API version in a master list:

class VersionChanges
  VERSIONS = {
    '2017-05-25' => [
      Change::AccountTypes,
      Change::CollapseEventRequest,
      Change::EventAccountToUserID
    ],
    '2017-04-06' => [Change::LegacyTransfers],
    '2017-02-14' => [
      Change::AutoexpandChargeDispute,
      Change::AutoexpandChargeRule
    ],
    '2017-01-27' => [Change::SourcedTransfersOnBts],
    ...
  }
end

Version changes are written so that they expect to be automatically applied backwards from the current API version and in order. Each version change assumes that although newer changes may exist in front of them, the data they receive will look the same as when they were originally written.

When generating a response, the API initially formats data by describing an API resource at the current version, then determines a target API version from one of:

• A Stripe-Version header if one was supplied.
• The version of an authorized OAuth application if the request is made on the user’s behalf.
• The user’s pinned version, which is set on their very first request to Stripe.

It then walks back through time and applies each version change module that finds along the way until that target version is reached.

Requests are processed by version change modules before returning a response.

Version change modules keep older API versions abstracted out of core code paths. Developers can largely avoid thinking about them while they’re building new products.

Changes with side effects

Most of our backwards-incompatible API changes will modify a response, but that’s not always the case. Sometimes a more complicated change is necessary which leaks out of the module that defines it. We assign these modules a has_side_effects annotation and the transformation they define becomes a no-op:

class LegacyTransfers 

Elsewhere in the code a check will be made to see whether they’re active:

VersionChanges.active?(LegacyTransfers)

This reduced encapsulation makes changes with side effects more complex to maintain, so we try to avoid them.

Declarative changes

One advantage of self-contained version change modules is that they can declare documentation describing what fields and resources they affect. We can also reuse this to rapidly provide more helpful information to our users. For example, our API changelog is programmatically generated and receives updates as soon as our services are deployed with a new version.

We also tailor our API reference documentation to specific users. It notices who is logged in and annotates fields based on their account API version. Here, we’re warning the developer that there’s been a backwards-incompatible change in the API since their pinned version. The request field of Event was previously a string, but is now a subobject that also contains an idempotency key (produced by the version change that we showed above):

Our documentation detects the user’s API version and presents relevant warnings.

Minimizing change

Providing extensive backwards compatibility isn’t free; every new version is more code to understand and maintain. We try to keep what we write as clean as possible, but given enough time dozens of checks on version changes that can’t be encapsulated cleanly will be littered throughout the project, making it slower, less readable, and more brittle. We take a few measures to try and avoid incurring this sort of expensive technical debt.

Even with our versioning system available, we do as much as we can to avoid using it by trying to get the design of our APIs right the first time. Outgoing changes are funneled through a lightweight API review process where they’re written up in a brief supporting document and submitted to a mailing list. This gives each proposed change broader visibility throughout the company, and improves the likelihood that we’ll catch errors and inconsistencies before they’re released.

We try to be mindful of balancing stagnation and leverage. Maintaining compatibility is important, but even so, we expect to eventually start retiring our older API versions. Helping users move to newer versions of the API gives them access to new features, and simplifies the foundation that we use to build new features.

Principles of change

The combination of rolling versions and an internal framework to support them has enabled us to onboard vast numbers of users, make enormous changes to our API—all while having minimal impact on existing integrations. The approach is driven by a few principles that we’ve picked up over the years. We think it’s important that API upgrades are:

• Lightweight. Make upgrades as cheap as possible (for users and for ourselves).
• First-class. Make versioning a first-class concept in your API so that it can be used to keep documentation and tooling accurate and up-to-date, and to generate a changelog automatically.
• Fixed-cost. Ensure that old versions add only minimal maintenance cost by tightly encapsulating them in version change modules. Put another way, the less thought that needs to be applied towards old behavior while writing new code, the better.

While we’re excited by the debate and developments around REST vs. GraphQL vs. gRPC, and—more broadly—what the future of web APIs will look like, we expect to continue supporting versioning schemes for a long time to come.

Like this post? Join the Stripe engineering team. View Openings

"Build Your Own Linux (From Scratch)" walks users through building a basic Linux distribution. Presented by Linux Academy, access the main Linux Academy website to view related course videos and other content.

Join the Linux Academy community for free to chat with thousands of like-minded Linux experts.

Section 1

Our Goal

What We are Building

This course walks through the creation of a 64-bit system based on the Linux kernel. Our goal is to produce a small, sleek system well-suited for hosting containers or being employed as a virtual machine.

Because we don't need every piece of functionality under the sun, we're not going to include every piece of software you might find in a typical distro. This distribution is intended to be minimal.

Here is what our end-result will look like:

• 64-bit Linux 4.8 Kernel with GCC 6.2 and glibc 2.24
• A system compatible with both EFI and BIOS hardware
• Bootable with GRUB2
• A VFAT formatted partition for GRUB/UEFI
• A boot partition
• A root partition

What We are Learning

This course provides step-by-step instructions in an effort to build the Linux kernel, the GNU C Standard Library implementation, GCC, and user-land binaries from source. The tasks are presented in linear order, and must be followed sequentially, as later tasks have dependencies on early tasks. Do not skip around.

Following this guide as intended will, in turn, enlighten you to many of the "hows" and "whys" of Linux, and assist in your ability to do tasks such as:

• Troubleshooting issues with the kernel
• Troubleshooting issues with user-land software
• Understanding the rationale behind various security systems and measures
• Performance tuning the kernel
• Performance tuning user-land binaries
• Building or "rolling" your own distribution
• Building user-land binaries from source

Back to top

Required Skills and Knowledge

We make extensive use of VirtualBox in this course. Working knowledge of VirtualBox and a solid foundation in Linux and Linux troubleshooting are essential. If you're not as familiar with VirtualBox as you would like, take a look at the "How to Install CentOS 7 with VirtualBox" lesson in the "Linux Essentials Certification" course. That course, as well, provides the foundational knowledge required for this course.

Back to top

Standards

As we progress through this course, we will adhere to the FHS (Filesystem Hierarchy Standard) specification, version 3.0. We will adhere (mostly) to the LSB (Linux Standard Base) specification, version 5.0. See the pertinent sections in this guide for more information on these two topics.

Back to top

Filesystem Hierarchy Standard

We follow the FHS 3.0 specification in this course. The FHS provides guidance as to how the filesystem should be structured in terms of directory structure, partition location, and directory use.

FHS 3.0 specifies four major file categories:

• Static OR variable
• Shareable OR unshareable

It would seem there are two categories above; however, there are not. "Static" and "variable" represent two mutually-exclusive categories, as do "shareable" and "unshareable."

A file must fit into one of these four categories; that is, it must be static or variable, shareable or unsharable or some combination thereof. All files fall into two of the four categories, without exception.

The following directories are required in the primary (or root) hierarchy; their use is as noted.

• bin :: Essential binaries
• boot :: Static boot-related files
• dev :: Device files
• etc :: Host-specific system configuration.
• lib :: Essential shared libraries and kernel modules
• media :: Mount point for removable media
• mnt :: Mount point for temporarily mounting a filesystem
• opt :: Add-on software
• run :: Data relevant to running processes; /var/run is used more frequently
• sbin :: Essential system binaries
• srv :: Data for services provided
• tmp :: Temporary files
• usr :: Secondary hierarchy; identical to primary (root) hierarchy
• var :: Variable (non-static) data

User home directories are located in /usr/home, which is linked to /home. This standard also specifies in detail which binaries are required; more information regarding this may be found athttp://refspecs.linuxfoundation.org/.

Back to top

Linux Standard Base

We follow the LSB Core Specification for the 64-bit x86 platform, as outlined athttps://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Core-AMD64/LSB-Core-AMD64/book1.html.

The LSB standard is expansive, and outside of the scope of this course. Adherence and deviance from the standard will be pointed out in the course where we feel it is important to do so.

Back to top

A Word on Linux

"Linux" as a term refers to two things: First, it refers specifically to the Linux kernel. Second, in a broader sense, it refers to the various packagings of the Linux kernel with other programs to provide the functionality required of a complete operating system.

Sound strange? It's not; it's one of the things that makes Linux so versatile. The kernel itself manages the hardware, memory, and the other parts of a computer system which are typically opaque to installed programs.

Programs installed to provide additional functionality are referred to as "user-land" or "the user-land." The combination of kernel and user-land constitute what are referred to as "distributions," many of which we are familiar: Red Hat Enterprise Linux, Ubuntu, Arch, Fedora, and so on.

In a broad sense, the term "Linux" refers to the operating systems created by the pairing of kernel and user-land, but the term is ambiguous. "Distribution," on the other hand, refers to the pairing of the kernel with a user-land built to some specification. Ubuntu, for instance, varies quite a bit from CentOS 6. Both of these are separate distributions of Linux.

Unlike operating systems, which are built in a monolithic fashion (where the user-land and kernel are tightly-coupled, such as FreeBSD, VMS, Windows, etc.), Linux allows for variations on theme which number into the thousands. The term "distro" better fits these variations, as each one is not entirely unique from the next (because of the shared kernel) but may differ substantially in terms of the user-land.

Back to top

Section 2

Prerequisites: Build System Specifications

Kernel/Distro Version

We use Fedora Core 24 in this course. You can run any distribution which uses a 4.x kernel, but be aware that doing so may introduce inconsistencies into the build process. Some distributions may package utilities using older or incompatible versions than what ia needed in this course.

Clean Install

We strongly recommend that you use a clean install for the build system.

Notice we undertake the whole of this course in a virtual machine running in VirtualBox; we do this to facilitate the building of Linux in a clean environment.

Any virtualization environment will do, provided you have access to the console, as it may be necessary at various points. Actual hardware is also acceptable, if those resources are readily available to you.

Build System Disk Partitions

The build system ” the virtual machine we use to build Linux ” uses the following disk layout. This output is from the parted print all command:

Number  Start   End     Size    Type     File system     Flags
1      1049kB  525MB   524MB   primary  ext4            boot
2      525MB   19.3GB  18.8GB  primary  ext4
3      19.3GB  21.5GB  2147MB  primary  linux-swap(v1)

Whilst we walk through the creation of the destination drive (where our newly-built distribution will be installed) in the videos, you should have the proficiency to install Linux and the necessary tools prior to undertaking this course. It is strongly recommended that you take the "Linux Essentials" course on LinuxAcademy.com if your Linux skills are not quite at this level.

Back to top

Development Tools

We'll need GCC, binutils, and other software packages installed with the "development tools" package group. You can select this during the installation process, or you can install using the group install option for yum or dnf. Note that the package group names may differ depending on distribution, but generally, we will need the "development tools" and "C development tools" groups installed.

Listing Package Groups with dnf

dnf group list -v

Listing Package Groups with yum

yum grouplist hidden

Installing Package Groups with dnf

dnf group install "C Development Tools and Libraries"
dnf group install "Development Tools"

Installing Package Groups with yum

yum groupinstall "Development Tools"

Texinfo

You may also have to install the "tex info" package to obtain the "make info" binary:

dnf install texinfo

MS-DOS Tools

Our first partition needs to be formatted as FAT12 or FAT16 to enable interoperability with GRUB and EFI. For this reason, you need the "dosfstools" package, or similar, installed on your system.

Back to top

Specific Software Packages and Required Versions

The nano editor is used throughout this course; amend commands with your preferred text editor as needed.

Some of the software packages installed require a specific or minimum version. Most often, the most recent version (with patches) will be sufficient.

• bash (/bin/sh must be a link to the bash binary)
• binutils
• bison (/usr/bin/yacc must be linked to or provided by bison)
• bzip2
• coreutils
• diffutils
• findutils
• gawk (/usr/sbin/awk must be a link to the gawk binary)
• gcc 6.2
• glibc 2.24
• grep
• gzip
• Linux kernel 4.x
• m4
• make
• patch
• perl 5.24
• sed
• tar
• texinfo
• xz

yacc and Bison

IMPORTANT

Note that Fedora Core 24 installs byacc by default. This means yacc is not a link to the bison executable. You can verify this with the following command:

rpm -qf `which yacc`

If the yacc binary is installed by the byacc package, execute the following:

Remove the byacc package:

dnf erase byacc

Re-install bison

dnf reinstall bison

Link the bison binary:

ln -s `which bison` /bin/yacc

Call yacc -V and make certain the output matches bison -V.

Hardware/Virtual Hardware Specifications

CPUs

Many of the packages we compile in this course can benefit from parallel make processes. The make command can, in many cases, compile multiple source files simultaneously, provided the build system has more than one CPU.

It is strongly recommended than you allocate at least two CPUs if your build system is virtualized. If you are using hardware, ideally you will have at least two physical CPU cores to speed up the build process.

In particular, build and test times for software like GCC can be reduced a great deal by having make execute processes in parallel. In places where you see -j2 indicated with the make command, it is perfectly acceptable to substitute a higher number, up to the number of CPUs allocated to the VM (or, if using hardware, the number of physical cores in the system). So, provided you have four CPU cores available, you are welcome to use -j4 instead of -j2.

RAM

Virtualized or otherwise, you will need at least two gigabytes of free RAM available. Do not include swap space in this consideration. Builds may fail if swap use becomes extant.

Build System Disks

In addition to to the system disk, the build system needs an additional block storage device available. We recommend attaching a 20 gigabyte drive to the second port of the SAS or SATA controller of your virtual machine. If using hardware, this device can be a USB drive, a second hard, etc.; however, it must be a local block device physically attached to the system.

Back to top

Users, Groups, and More

Don't Use root Unless Needed

The most important rule of this course parallels a general principle in *NIX in general: Do nothing as root unless it is required.

Particularly, when compiling code, it is very possible to crash your system or damage your installation. It is possible to even damage hardware in many cases. Beyond this, if you compile as root, you may find it next to impossible to work through the various steps of a process using anything but the root account. This sets us up for failure: Forget to set a single environment variable, and you've overwritten critical system libraries or binaries on the build system, only to find no binaries can be executed and the system cannot reboot.

At some stages of the build process, we must use the root account to execute commands. This will be noted when necessary.

Back to top

Creating Our User

For this course, we create a user and group to execute our builds. Make sure the user is part of wheel group (or equivalent).

First, add our byol group:

groupadd byol

Be sure to add the -k flag to the user add command to prevent files being copied to the home directory from /etc/skeleton:

useradd -s /bin/bash -g byol -m -k /dev/null byol

We do this to prevent environment variables that might otherwise be appropriate for a user account from being set in our build environment, as these can have unintended consequences.

Setting the Login Environment for the byol User

There should be two bash-related files in the home directory of the byol user: .bash_profile and .bashrc.

The .bash_profile file needs to have the following content:

exec env -i HOME=$HOME TERM=$TERM PS1='\u:\w\$ ' /bin/bash

While the .bashrc file needs to have the following content:

set +h
umask 022
LC_ALL=POSIX
PATH=/tools/bin:/bin:/usr/bin
export LC_ALL PATH

These files are read during login, and set up the shell environment for the user. We set the environment here specifically to avoid inheriting environment variables that might have an adverse effect on our efforts.

After altering these files, upon log out and log in, the output from the env command should return output similar to the following:

TERM=xterm-256color
LC_ALL=POSIX
PATH=/tools/bin:/bin:/usr/bin
PWD=/home/byol
PS1=\u:\w\$
SHLVL=1
HOME=/home/byol
_=/bin/env

Back to top

Destination Disk

Once you've configured your virtual machine, add an additional disk to it. 20 gigabytes should be sufficient; the disk type doesn't matter, but attaching to the SAS or SATA controller is recommended. We will refer to this newly attached disk as the "destination disk" from this point forward.

Partitioning the Destination Disk

By the conclusion of this section, the destination disk partition layout should look like this:

Number  Start   End     Size    File system     Name     Flags
 1      1049kB  105MB   104MB   fat16           primary  bios_grub
 2      105MB   551MB   446MB   ext4            primary
 3      551MB   19.3GB  18.7GB  ext4            primary
 4      19.3GB  21.5GB  2174MB  linux-swap(v1)  primary

The first partition begins at 1MB and extends to 104MB. This partition exists solely for the purposes of booting and is formatted as VFAT. We will refer to this partition as the EFI/GRUB boot partition henceforth.

The second partition begins at around 100MB and ends 400MB or so later. This is the boot partition.

The third partition begins at around 550MB and ends around 19GB. This is the root partition.

This last partition begins around 19MB and extends to the end of the disk. This is the swap parition.

The following command creates these partitions using parted:

parted --script /dev/sdb mklabel gpt mkpart primary 1MiB 100MiB mkpart primary 100MiB 525MiB mkpart primary 525MiB 19.3GB mkpart primary 19.3GB  100%

Setting Flags on the GRUB Boot Partition

The GRUB boot partition needs to have certain flags set so that BIOS-based systems can boot from it. We accomplish this using parted:

parted --script /dev/sdb set 1 bios_grub on

Creating Filesystems on the Destination Disk

Our partition table on our destination block storage device should look like this:

Number  Start   End     Size    File system     Name     Flags
 1      1049kB  105MB   104MB   fat16           primary  bios_grub
 2      105MB   551MB   446MB   ext4            primary
 3      551MB   19.3GB  18.7GB  ext4            primary
 4      19.3GB  21.5GB  2174MB  linux-swap(v1)  primary

Partition 1

mkfs.vfat /dev/sdb1

Partitions 2 and 3

The second and third partitions are our boot and root partitions. We will format these as "vanilla" ext4, using the mkfs.ext4 command; note that your device name may be different:

mkfs.ext4 /dev/sdb2
mkfs.ext4 /dev/sdb3

Partition 4

The last partition is our swap partition, which we will initialize using the mkswap command:

mkswap /dev/sdb4

Setting Partition and Filesystem Metadata

We're going to set some volume metadata on our filesystems using the tune2fs command.

Set the volume label of our second partition on our destination device to DESTBOOT:

 tune2fs -c 1 -L DESTBOOT /dev/sdb2

Set the volume label of our third partition to DESTROOT:

tune2fs -c 1 -L DESTROOT /dev/sdb3

Note that we're setting the mount count to 1, which means the consistency of these filesystems will be checked after they have been mounted once. This is optional, but can be helpful in the event you have to reset your VM or build system and the filesystems aren't cleanly unmounted.

Sanity Check

At this point, you should have:

• A complete build system with all of the necessary software installed to build Linux. Ideally, this system has two or more CPUs and two gigabytes or more of free RAM. Do not include available swap as free RAM.
• A byol user and group. This user should have access to root privileges via su or sudo.
• A second block storage device mounted on the build system, ideally on the same interface/controller as the system drive. This is our destination drive and should be partitioned and formatted as described above.

Measure twice, cut once: This is a good time to review all of the sections previous to this one and ensure your build system and destination disk are setup as recommended.

Mount Point(s)

Destination Device: Root and Boot Directories

We're going to mount our destination disk on /build, as follows, using the superuser account:

mkdir -v /build
export BROOT=/build
mount -t ext4 -L DESTROOT $BROOT
mkdir -v $BROOT/boot
mount -t ext4 -L DESTBOOT $BROOT/boot

Adding Our Swap Partition

swapon -v /dev/sdb4 

Exporting the BROOT Environment Variable

We're also going to add an export line to the .bashrc file of the byol account to export the BROOT variable. Our .bashrc file should look like this:

set +h
umask 022
LC_ALL=POSIX
PATH=/tools/bin:/bin:/usr/bin
BROOT=/build
export BROOT LC_ALL PATH

Back to top

Sanity Check

Measure twice, cut once: you should have the root and boot destination filesystems mounted under /build. Also, your swap partition should be active. We can check these with the mount and swapon commands:

mount | grep 'sdb'

This should return a listing similar to:

/dev/sdb3 on /build type ext4 (rw,relatime,data=ordered)
/dev/sdb2 on /build/boot type ext4 (rw,relatime,data=ordered) 

swapon | grep sdb should output similar:

/dev/sdb4 partition   2G   0B   -2

Keep in mind that your device names may be different.

Back to top

Building the Toolchain

We're now ready to build the requisite toolchain needed to build Linux from scratch.

Directories and Directory Permissions

First, let's create a directory in our destination root to hold the source files:

mkdir -v $BROOT/source

We want the /usr/src/toolchain directory to be sticky, which means that only the owner of a file can delete it. We also want the files in this directory to be writable for all users:

chmod -v 777 $BROOT/source

Additionally, we want to create a directory to contain our compiled tools, which we need to keep aside and apart from the same binaries on the build system:

mkdir -v $BROOT/tools

We want to link this directory to the build system. The following command looks somewhat strange, but is correct:

ln -sv $BROOT/tools /

The result should be a symlink from /tools in the build system root directory to the tools directory on the root of our destination device:

ln -sv $BROOT/tools /
'/tools' -> '/build/tools'

Source Code Files

A list of the software we need to install can be found at:http://linuxfromscratch.org/lfs/view/stable/wget-list

We can download all of these files using two wget commands:

wget http://linuxfromscratch.org/lfs/view/stable/wget-list
wget --input-file=wget-list --no-check-certificate --directory-prefix=$BROOT/source 

We do not want the source code files located in our tools directory.

Decompress and Extract the Downloaded Files

All of our source code files will have been compressed with zx, bzip2, or gzip. We can decompress these files by running the following three commands:

bunzip2 -d \*.bz2
gunzip \*.gz
xz -d \*.xz

Change Ownership of the Build Directory

Execute:

chown -v byol:byol $BROOT

Patch Files

You'll notice that some of the files listed in the above TXT file are patch files. These patches need to be applied to the source before we compile the relevant package; we touch on this where necessary.

A Word on Errors

Both binutils and GCC must compile without errors. If either built has errors, start over. Depending on the error type you encounter this might mean rebuilding just GCC or rebuilding binutils as well.

Back to top

Setting the Target Triplet

GCC and the rest of the build toolchain use a string called the "target triplet" to provide information as to the current CPU architecture, vendor, and operating system. The triplet itself is broken into three parts: machine-vender-os

So We could define our target triplet as follows:

x86_64-elf

However, we need to build GCC and the binutils toolchain without any external dependencies. To do that, our first build must be targeted for cross-compilation. This will ensure that all of the necessary dependencies are included and none of the shared libraries on the build system are relied upon. So we're going to use the triplet instead of the one above:

x86_64-lfs-linux-gnu

This target triplet indicates an x86_64 compatible machine and an elf binary compatible operating system. Since we need this value set in our environment, let's add this to the .bashrc file of the byol user. That file should now look like this:

set +h
umask 022
LC_ALL=POSIX
PATH=/tools/bin:/bin:/usr/bin
BROOT=/build
BTARGET=x86_64-lfs-linux-gnu
export BROOT BTARGET LC_ALL PATH   

The vendor field is optional in the target triplet; we've specified "lfs" here as this vendor code exists to facilitate building for purposes such as ours.

Back to top

Target Triplets: What's the Point?

The purpose of the target triplet is to provide a means for the system and compilers to determine specific information about any given binary. After cross referencing and disambiguation, the target triplet is actually resolved to three distinct values, which may be quite different from the triplet specified above:

• Build Platform :: Where the binary was built.
• Host Platform :: Where the binaries are intended to run.
• Target Platform :: In the case of compilers, this is for what the compiler will generate code.

So after disambiguation, target triplets are "resolved" into something like this:

x86_64-elf-gcc

In fact, we could specify this value as our target triplet. This would result, however, in subtle changes in GCC's handling of code, and the outcome may not be desirable. For example, changing the vendor to the value pc will result in the toolchain being cross-compiled with 32-bit compatibility enabled. This, in turn, will result in a different dynamic linker that will be used both at compile and runtime to line and find shared libraries, respectively. If GCC or glibc are configured incorrectly the result is often a toolchain that is broken such that problems do not manifest until the build process is nearly complete.

Back to top

Build Directory

For many of the builds, we cd into the source directory and create a build directory. Once changed into this directory, we proceed with the build. This is the recommended way to build most of the toolchain, as it keeps the original source files from being tampered with by the build process.

If a build directory is not specified, it is not necessary and won't be used.

Configuration

Once in the build directory, the "configure" script is called from the parent directory to determine the options for the tool being built. This also copies the relevant source and configuration files from the original source directory to the build directory created in the previous section.

Deleting Source Directories

For some builds (like GCC and binutils), it is perfectly fine to delete only the build directory after compilation and installation are executed. However, erring on the side of caution, it may be wise to delete the entire source directory to prevent misconfiguration down the road.

Back to top

Section 3

Stage 1: Building a Temporary Toolchain

The first stage of the build process entails building a toolchain capable of producing executables for our target platform. Because our target platform differs (in terms of the target triplet) from the platform on which we are building it (the build system), we need to first build a toolchain that is can be used in a "standalone" fashion. The ensures that we avoid any dependency on the build system environment or libraries.

Note that for Stage 1 builds, we don't execute the test suites for the software packages installed. The reason for this is simple ” the dependencies required to run the tests, such as TCL, have not yet been installed, and the tests will likely fail.

Running the tests ceases to be optional once we reach Stage 3, however.

Back to top

Compiling Binutils

Create the Build Directory

Change into the binutils source directory under $BROOT/source. Create a build directory:

mkdir -v build && cd build

Configure the Source

Call the "configure" script from within the build directory to configure binutils. We used the following parameters for configure:

../configure \
--prefix=/tools \
--with-sysroot=$BROOT \
--with-lib-path=/tools/lib \
--target=$BTARGET  \
--disable-nls \
--disable-werror

• prefix :: Tells the configure script where the compiled binaries should be installed.
• with-sysroot :: Tells the configure script to look in the specified directory for the target system libraries.
• with-lib-path :: Specifies which path the linker will be configured to use.
• target :: Because the triplet we've defined is slightly different than what will be determined by configure itself, the binutils source needs to be compiled to accommodate cross-linking. This is necessary because we want to ensure a "clean room" build, without any artifacts from the build system.
• disable-els :: Disables internationalization, which we don't need at this stage.
• disable-werror :: Keeps warnings from interrupting the compile process.

Compile the Source

make -j2

Create a Library Directory and Symlink

We need to create the /tools/lib directory and symlink /tools/lib64 to it. This ensures that the proper libraries can be found along both paths:

case $(uname -m) in
  x86_64) mkdir -v /tools/lib && ln -sv lib /tools/lib64 ;;
esac

Install binutils

make install

Delete the Build Directory

Be sure to delete the build directory once you've run make install.

Back to top

GCC

We are now going to build GCC. Because we're building GCC as a cross-compiler, it will not rely on libraries installed on the build system and instead prefers those packaged with it, or which we explicitly tell configure to look for.

Obtain the GCC Dependencies

First, we need to download the GCC dependencies. We can do that using the following commands to copy the relevant software packages into the GCC source directory. Once they're copied, make finds and builds these dependencies automatically. From the GCC source directory, execute:

tar -xf ../mpfr-3.1.4.tar
mv -v mpfr-3.1.4 mpfr
tar -xf ../gmp-6.1.1.tar
mv -v gmp-6.1.1 gmp
tar -xf ../mpc-1.0.3.tar
mv -v mpc-1.0.3 mpc

Note that the software versions may have changed since this guide was written.

Changing the Default Linker

We need to instruct GCC to use the linker previously installed by our binutils install. The following script (which you should be able to copy and paste to a terminal window) finds the relevant header files and changes the locations where make looks for certain libraries and files. Execute the following in the GCC source code directory:

for file in \
 $(find gcc/config -name linux64.h -o -name linux.h -o -name sysv4.h)
do
  cp -uv $file{,.orig}
  sed -e 's@/lib\(64\)\?\(32\)\?/ld@/tools&@g' \
  -e 's@/usr@/tools@g' $file.orig > $file
  echo '
#undef STANDARD_STARTFILE_PREFIX_1
#undef STANDARD_STARTFILE_PREFIX_2
#define STANDARD_STARTFILE_PREFIX_1 "/tools/lib/"
#define STANDARD_STARTFILE_PREFIX_2 ""' >> $file
  touch $file.orig
done

Alternatively, you can save this code in a file in GCC source directory and run it by calling that file.

A more detailed description of what this script does, per the 'Linux From Scratch' online version:

"If case the above seems hard to follow, let's break it down a bit. First we find all the files under the GCC/config directory that are named either linux.h, linux64.h or sysv4.h. For each file found, we copy it to a file of the same name but with an added suffix of .orig. Then the first sed expression prepends /tools to every instance of /lib/ld, /lib64/ld, or /lib32/ld, while the second one replaces hard-coded instances of /usr. Next, we add our define statements which alter the default startfile prefix to the end of the file. Note that the trailing / in /tools/lib/ is required. Finally, we use touch to update the timestamp on the copied files. When used in conjunction with cp -u, this prevents unexpected changes to the original files in case the commands are inadvertently run twice."

Configure

Create the build directory and change into it. Run the configure script with the following arguments:

../configure                                   \
--target=$BTARGET                              \
--prefix=/tools                                \
--with-glibc-version=2.24                      \
--with-sysroot=$BROOT                          \
--with-newlib                                  \
--without-headers                              \
--with-local-prefix=/tools                     \
--with-native-system-header-dir=/tools/include \
--disable-nls                                  \
--disable-shared                               \
--disable-multilib                             \
--disable-decimal-float                        \
--disable-threads                              \
--disable-libatomic                            \
--disable-libgomp                              \
--disable-libmpx                               \
--disable-libquadmath                          \
--disable-libssp                               \
--disable-libvtv                               \
--disable-libstdcxx                            \
--enable-languages=c,c++ 

• target :: Tells make to use the target triplet located in our environment variable.
• prefix :: Instructs make to use the specified path to prefix relative pathnames.kss
• with-glibc-version :: Indicates which version of glibc we're going to target.
• with-sysroot :: Specifies the system root and allows us to specify a different root path than that of the currently-running kernel.
• with-newlib :: Prevents any code which requires libc from being compiled (because we haven't built libc yet).
• without-headers :: If building a full-blown cross-compiler, GCC needs the header files compatible with the target system. This argument instructs configure and make not to look for them, as we don't need them for this stage.
• with-local-prefix :: This instructs configure and make to search the specified directory for include files.
• with-native-system-header-dir :: This changes the default include path for headers (which is normally /usr/include) to /tools/include. Without this switch, GCC will look in the default location for include files, and that will break our build since our header files are located in /tools/include.
• disable-shared :: Advises GCC to avoid performing static linking. This is a good idea simply because it avoids any conflicts that might arise if GCC were built to use shared libraries, as the system linker might attempt to link them with the libraries installed on the build system.
• disable-decimal :: This GCC extension is not compatible when building GCC for cross compilation.
• disable-float :: See disable-decimal.
• disable-threads :: See disable-decimal.
• disable-libatomic :: See disable-decimal.
• disable-libgomp :: See disable-decimal.
• disable-libmpx :: See disable-decimal.
• disable-libquadmath :: See disable-decimal.
• disable-libssp :: See disable-decimal.
• disable-libvtv :: See disable-decimal.
• disable-libstdcxx :: See disable-decimal.
• disable-multilib :: This functionality isn't supported on the x86_64 platform.
• enable-languages :: For this stage, we need GCC to compile only C and C++. This disables the compilation of compilers for other languages.

Make GCC

make -j2

This will take some time.

Install

make install

Do not delete the GCC source code directory just yet. Be sure to delete the build directory, however.

Back to top

The Linux kernel ships with a set of header files which provide programmers a software interface to the kernel. These are used primarily by libc (or the GNU implementation, glibc).

Change to the kernel source directory:

$BROOT/source/linux-4.x

Issue the following command:

make mrproper

We now call the make command, specifying the header file output path; we do this in two steps because make wipes any files from the destination directory during this step. First, we extract the files:

make INSTALL_HDR_PATH=dest headers_install

And second, we copy them to the proper location for libc:

cp -rv dest/include/* /tools/include

Back to top

Building glibc

Configure

Run the configure script with the following arguments. Note the exported variables; be sure to unset these when the build process is complete!

The libc_cv_forced_unwind variable impacts the handling of the ”force-unwind configuration parameter. The linker we installed when we compiled binutils is cross-compiled and cannot make use of the ”force-unwind option unless glibc is present. This variable disables this test.

The libc_cv_c_cleanup variables instruct configure to disable the test for this functionality, again for the same reasons given for libc\_cv\_forced\_unwind.

The commands for the configuration process should look like this:

export libc_cv_forced_unwind=yes
export libc_cv_c_cleanup=yes
../configure                         \
  --prefix=/tools                    \
  --host=$BTARGET                    \
  --build=$(../scripts/config.guess) \
  --enable-kernel=2.6.32             \
  --with-headers=/tools/include

• prefix :: Instructs make to use the specified path to prefix's relative pathnames.
• host :: Tells make to use the target triplet located in our environment variable.
• build :: Combined with the host flag, this instructs glibc's build system to configure itself to cross compile, using the cross-linker and compiler in /tools.
• enable-kernel :: Instructs glibc to use workarounds for this specific version (and later) of the kernel.
• with-headers :: Specifies the location of header files, so libc knows what features the kernel has and can configure itself accordingly.

Make

According to the Linux From Scratch website, glibc sometimes breaks during parallel makes. We're going to disable parallel compilation for glibc by executing make as follows:

make -j1
make -j1 install

Delete the Build Directory

Be sure to delete the build directory once you've run make install. Additionally, unset the environment variables defined during our configuration:

unset libc_cv_forced_unwind
unset libc_cv_c_cleanup

Back to top

Sanity Check

At this point, it is imperative that we stop and check our temporary toolchain to ensure that it has been built correctly. The online version of the "Linux From Scratch" website provides us a neat and sure method of testing that our toolchain is installed correctly.

Testing the Toolchain

We can test the temporary toolchain to check it compiles and links code and object code correctly with the following commands:

echo 'int main(){}' > dummy.c
$BTARGET-gcc dummy.c
readelf -l a.out | grep ': /tools'

Note that this should be done as the byol user.

If everything is working as it should be, the grep command should return this output:

[Requesting program interpreter: /tools/lib64/ld-linux-x86-64.so.2]

If the output is different ” particularly if the program interpreter is located on the build system's filesystem hierarchy (lib/ld-linux-x86-64.so.2, etc.) - then something has gone wrong.

Delete the build directories, check environment variables and start over. Continuing will result in a toolchain that is broken in odd ways which are not immediately apparent.

Be sure to delete all of the dummy.* files when you're done testing.

Back to top

Building libstdc++

libstdc++ is Part of GCC

This software is part of the GCC sources, so we'll need to be in the gcc source directory, in an empty build directory, before executing the following.

Configure

Create the build directory as per usual, and run the configure script with the following arguments:

../libstdc++-v3/configure       \
--host=$BTARGET                 \
--prefix=/tools                 \
--disable-multilib              \
--disable-nls                   \
--disable-libstdcxx-threads     \
--disable-libstdcxx-pch         \
--with-gxx-include-dir=/tools/$BTARGET/include/c++/6.2.0

Make

make -j2
make install

Back to top

Section 4

Stage 2: Building with The Temporary Toolchain

At this point, we've created a temporary toolchain capable of compiling and linking executables in a stand-alone sense. Now we need to make an additional compilation pass over these same tools so they are native to our target triplet.

In a sense, Stage 2 creates a "temporary" system, using minimal installations of several familiar programs and libraries like ncurses, bash, and more.

Back to top

Compiling Binutils - Native Build

Create the Build Directory

Change into the binutils source directory under $BROOT/source. Create a build directory:

mkdir -v build && cd build

Configure the Source

Because we want a native build of binutils, call the target-triplet-specific utilities installed in our first build of binutils:

export CC=$BTARGET-gcc				
export AR=$BTARGET-ar                 
export RANLIB=$BTARGET-ranlib         

Call the "configure" script from within the build directory to configure binutils. We used the following parameters for configure:

../configure                   \
--prefix=/tools            \
--disable-nls              \
--disable-werror           \
--with-lib-path=/tools/lib \
--with-sysroot

• with-sysroot :: Specifying no value here enables the linker to find shared libraries required by other objects. Without this flag, make may not be able to locate and link some required libraries.
• with-lib-path :: Here we're instructing make to use the specified directory explicitly.

Compile the Source

make -j2

Install binutils

make install

Prepare the Linker for Upcoming Adjustments

The following commands clean out the build directory for the ld utility, and then rebuilds the same. We specify the LIB_PATH to override the default value used by the temporary toolchain ” this is the default library search path.

make -C ld clean
make -C ld LIB_PATH=/usr/lib:/lib
cp -v ld/ld-new /tools/bin

Back to top

Building GCC Natively

GCC initially uses a self-provided limits.h file for the building of the temporary toolchain. Now, we want GCC to use a full set of definitions written in the limits headers. To ensure this, we concatenate several of GCC's internal files to create a single limits.h file:

cat gcc/limitx.h gcc/glimits.h gcc/limity.h > dirname $($BTARGET-gcc -print-libgcc-file-name)/include-fixed/limits.h

Changing the Default Linker (Again)

Once again, we want GCC to use the linker installed in our /tools directory:

for file in \
 $(find gcc/config -name linux64.h -o -name linux.h -o -name sysv4.h)
do
  cp -uv $file{,.orig}
  sed -e 's@/lib\(64\)\?\(32\)\?/ld@/tools&@g' \
  -e 's@/usr@/tools@g' $file.orig > $file
  echo '
undef STANDARD_STARTFILE_PREFIX_1
undef STANDARD_STARTFILE_PREFIX_2
define STANDARD_STARTFILE_PREFIX_1 "/tools/lib/"
define STANDARD_STARTFILE_PREFIX_2 ""' >> $file
  touch $file.orig
done

GCC Dependencies

These should already be in place, as this step was required for the Stage 1 build of GCC. However, if for some reason you've deleted the GCC source directory, re-trace the steps for installing the GCC dependencies as described previously.

Configure

Create the build directory and change into it. As with the build of binutils in the previous section, we define some environment variables before compiling:

export CC=$BTARGET-gcc                                    \
export CXX=$BTARGET-g++                                   \
export AR=$BTARGET-ar                                     \
export RANLIB=$BTARGET-ranlib                             \

Run the configure script with the following arguments:

../configure                                       \
--prefix=/tools                                \
--with-local-prefix=/tools                     \
--with-native-system-header-dir=/tools/include \
--enable-languages=c,c++                       \
--disable-libstdcxx-pch                        \
--disable-multilib                             \
--disable-bootstrap                            \
--disable-libgomp

Build GCC

Execute:

make -j2
make install 

Create the cc Symlink

Typically, scripts call cc instead cf gcc or clang to compile code. This allows developers to use different compilers without having to re-edit scripts or write scripts for each compile. The behavior is accomplished by creating a symlink to the compiler executable; in our case, cc should be a link to the gcc binary:

ln -sv gcc /tools/bin/cc

Cleanup

Be sure to unset the exported variables above. You may also logout and log back in as the byol user to reset the environment.

Back to top

Sanity Check

Once again, we need to stop and double-check our toolchain to make sure that it can compile and link executables properly.

Execute the following commands:

echo 'int main(){}' > dummy.c
$BTARGET-gcc dummy.c
readelf -l a.out | grep ': /tools'

Note that this should be done as the byol user.

If everything is working as it should be, the grep command should return this output:

[Requesting program interpreter: /tools/lib64/ld-linux-x86-64.so.2]

If the output is different ” particularly if the program interpreter is located on the build system's filesystem hierarchy (lib/ld-linux-x86-64.so.2, etc.), then something has gone wrong.

Return to the last sanity check and re-trace your steps. Continuing will result in a broken toolchain that will likely not compile or link correctly.

Back to top

Test Suite Dependencies

The test suite used for our toolchain requires a handful of dependencies. During the first and second compilation passes of binutils GCC, and Glibc, we didn't allow make to execute tests ” as the necessary programs were missing, there was no point in doing so.

At this stage, however, we most certainly do want to run the tests to make sure our toolchain is set up properly.

We downloaded the source code for the programs used in testing during Stage 1. Now we need to compile and install them. Unless it is otherwise noted, you'll need to extract the source code directories from the appropriate tar file in /sources, cd into it and execute the configure and make commands. It is assumed from this point forward that you have extracted the source and are in the resulting directory. Note that most of these utilities do not require the use or creation of a separate 'build' directory.

Regarding make and make install

These commands can typically run one after another.

make && make install

Sometimes, however, there is good reason to separate the commands into two steps, such as when header files or libraries need to be modified before being installed. Where you see the two commands one after another (as in the following example):

make
make install

You can replace the two commands with a single line:

make && make install

Be careful, though, to use this only where the make and make install commands are one after the other, and no additional steps are required.

TCL

Configure:

cd unix
./configure --prefix=/tools

Compile:

make -j2

Install:

make install

Change Permissions

chmod the installed library file so we can strip out debugging symbols later:

chmod -v u+w /tools/lib/libtcl8.6.so

make install-private-headers

Create Required Symlink

Be sure to replace the xxx with the appropriate minor version of TCL in the following command:

ln -sv tclsh8.xxx /tools/bin/tclsh

Expect

First, we want to force expect to use /bin/stty to open a terminal as opposed to /usr/local/bin/stty; the binary in /usr/local is located on the build system (as we don't have a /usr/local hierarchy on our destination disk). We don't want expect to depend on that binary.

To remedy this, execute:

cp -v configure{,.orig}
sed 's:/usr/local/bin:/bin:' configure.orig > configure

This uses sed to replace the path used by the configure script.

Configure

./configure --prefix=/tools --with-tcl=/tools/lib --with-tclinclude=/tools/include

Install

We include the SCRIPTS variable so make will skip including supplemental scripts.

make SCRIPTS="" install

DejaGNU

Configure

./configure --prefix=/tools

Compile and Install

make install

Check

Configure

We include an empty PKG_CONFIG variable here to prevent any pre-defined pkg-config options that may lead make to link against libraries on the build system.

PKG_CONFIG= ./configure --prefix=/tools

Compile and Install

make
make install

Ncurses

Ncurses is a library which provides terminal control routines. It allows for a basic graphical user interface to be used in text-only (e.g., terminal or console) environments.

Configure

We amend the source code here to ensure that the gawk command is found before awk:

sed -i s/mawk// configure
./configure --prefix=/tools \
            --with-shared   \
            --without-debug \
            --without-ada   \
            --enable-widec  \
            --enable-overwrite

Compile and Install

make
make install

Bash

Configure

./configure --prefix=/tools --without-bash-malloc

We pass the ”without-bash-malloc parameter to disable bash's internal malloc(), which is known to be somewhat buggy.

Compile and Install

make
make install

Symlink sh

In many distributions, the sh command is actually a symlink to the bash executable; ours is no different. Create the symlink with the following command:

ln -sv bash /tools/bin/sh

bzip

The bzip package does not use a configure script, only make.

Compile

make

Install

We pass the PREFIX variable here to ensure the resulting binaries are installed in the /tools hierarchy.

make PREFIX=/tools install

Coreutils

The coreutils package provides many of the basic user-land utilities we use in the shell to manipulate text, files and the environment.

Configure

Notice the --enable-install-program parameter. We pass this to configure to build the 'hostname' program, which is disabled by default.

./configure --prefix=/tools --enable-install-program=hostname

Compile and Install

make
make install

Diffutils, File, Findutils, and Gawk

These software packages use identical configuration and compilation/install steps. Execute these steps for each of these packages ” be sure not to skip any.

Configure

./configure --prefix=/tools

Compile and Install

make
make install

Gettext

We only need three programs from the gettext package at this point: msgfmt, msgmerge and xgettext; hence the unusual make commands.

Configure

Note that we pass an empty "EMACS" variable to configure; this prevents the detection of Lisp scripts used by emacs, which is known to hang some systems. Be certain to cd into the gettext-tools directory before running configure.

cd gettext-tools
EMACS="no" ./configure --prefix=/tools --disable-shared

Compile the Binaries

make -C gnulib-lib
make -C intl pluralx.c
make -C src msgfmt
make -C src msgmerge
make -C src xgettext

Install

Rather than using make to install our binaries for us, we copy them into place manually:

cp -v src/{msgfmt,msgmerge,xgettext} /tools/bin

Grep, Gzip, and M4

These software packages use identical configuration and compilation/install steps. Execute these steps for each of these packages ” be sure not to skip any.

Configure

./configure --prefix=/tools

Compile and Install

make
make install

Make

Configure

Here we pass the ”without-guile flag to configure. While these libraries may be available on the build system, they are not on our destination, and we don't want make to depend on them for that reason.

./configure --prefix=/tools --without-guile

Compile and Install

make
make install

Patch

Configure

./configure --prefix=/tools

Compile and Install

make
make install

Perl 5

Configure

The Perl source code doesn't use the standard configure script. Note the capital C in the configuration command:

sh Configure -des -Dprefix=/tools -Dlibs=-lm

Compile

make

Install

We only need a few of the libraries we've built, so we'll copy these to our destination manually:

cp -v perl cpan/podlators/scripts/pod2man /tools/bin
mkdir -pv /tools/lib/perl5/5.24.0
cp -Rv lib/* /tools/lib/perl5/5.24.0

sed, tar, and texinfo

Configure

./configure --prefix=/tools

Compile and Install

make
make install

Util-linux

Configure

We include an empty PKG_CONFIG variable here to prevent any pre-defined pkg-config options that may lead make to link against libraries on the build system.

./configure --prefix=/tools                \
            --without-python               \
            --disable-makeinstall-chown    \
            --without-systemdsystemunitdir \
            PKG_CONFIG=""

• without-python :: This disables the Python bindings, which we don't need at this point.
• disable-makeinstall-chown :: make tries to change the owner of the binaries after copying them into place. This requires root permissions, however, so we disable this.
• without-systemdsystemunitdir :: This instructs make to skip the installation of systemd-specific files.

Compile and Install

make
make install

xz

Configure

./configure --prefix=/tools

Compile and Install

make
make install

Back to top

Stripping (Optional)

Now that we've got a temporary system built and installed, we can save some space by removing the debug symbols from our binaries and our libraries.

This command will result in quit a "File format not recognized" warning messages. This is normal ” if you examine the file indicated in the message, you can note that the file in question is usually a script file, not a binary.

Please be very careful with this step. It is very easy to strip the wrong files and delete most of the libraries we've just built and installed.

THIS STEP IS OPTIONAL.

Strip the Debug Symbols from Our Libraries

strip --strip-debug /tools/lib/*

Strip the Debug Symbols from Our Binaries

DO NOT run this command against library files! This command uses the build system's strip binary to remove unneeded symbols. Running this against library files deletes them.

/usr/bin/strip --strip-unneeded /tools/{,s}bin/*

Back to top

Switching to Root

At this point, we've used the byol user for almost every activity, save the creation and mounting of our destination file system.

From this point forward, however, we will undertake all commands as root.

Environment Check

Execute su and run:

env | grep ^B 

Your output should look something like this:

BROOT=/build
BTARGET=x86_64-lfs-linux-gnu

While the "BROOT" environment variable is required, the "BTARGET" variable is not, and must not be set as we continue forward.

Unsetting the BTARGET Environment Variable

Unset this variable with the following command:

unset BTARGET

Check again to make sure that the variable has been unset:

env | grep ^B

"BTARGET" should not be anywhere in the output.

Setting the BROOT Environment Variables

If you're missing "BROOT", you can set it easily enough:

export BROOT="/build"

Back to top

Changing File Ownership

Now that we've installed our binaries and libraries, we need to change their ownership.

We've built and installed our binaries and libraries using the byol user so far. Our destination system, however, does not have any users defined. This poses a minor security risk ” a user could be created with a user ID identical to that of the byol user on the build system, which would then have access to all our binaries.

The simplest way to restrict access is to make root the owner of these files, as the superuser ID is identical across all systems:

chown -R root:root $BROOT/tools

If we examine these files, we should see that both user and group are set to root:

root:~# ls -la $BROOT/tools
total 68
drwxr-xr-x 13 root root  4096 Nov 30 18:20 .
drwxr-xr-x  7 byol byol  4096 Nov 29 00:57 ..
drwxr-xr-x  2 root root 12288 Nov 30 19:31 bin
drwxr-xr-x  2 root root  4096 Nov 30 17:25 etc
drwxr-xr-x 40 root root  4096 Nov 30 19:26 include
drwxr-xr-x 11 root root 12288 Nov 30 19:30 lib
lrwxrwxrwx  1 root root     3 Nov 30 17:00 lib64 -> lib
drwxr-xr-x  6 root root  4096 Nov 30 19:16 libexec
drwxr-xr-x  5 root root  4096 Nov 30 18:20 man
drwxr-xr-x  2 root root  4096 Nov 30 19:31 sbin
drwxr-xr-x 16 root root  4096 Nov 30 19:25 share
drwxr-xr-x  3 root root  4096 Nov 30 17:25 var
drwxr-xr-x  5 root root  4096 Nov 30 17:29 x86_64-lfs-linux-gnu
drwxr-xr-x  4 root root  4096 Nov 30 17:50 x86_64-pc-linux-gnu

Back to top

Backing Up

Now that we've got a complete (albeit temporary) system, it would be wise to backup our work. If you're using a virtual machine, exporting the VM as an OVA file is one way of ensuring you've got a patent backup.

You may choose to tar the /build directory ” whatever works. The point is that you'll be able to restore the environment to the state it is in now with minimal effort at some point in the future.

This is important, particularly as a time-saver: Everything we do from this point will alter the binaries and libraries we've built and installed, so there is a possibility things will break irreparably.

BE SURE TO DOUBLE CHECK YOUR ENVIRONMENT AS PER "Switching to Root" BEFORE CONTINUING.

Back to top

Section 5

Stage 3: Native (Full) System Build

Up to this point, we've used the byol user to build and install software, with a few notable exceptions. Be sure to check your environment as noted in "Switching to Root" in the previous section.

From this point forward, everything we do will be as the root user. After creating our virtual and pseudo filesystems, we'll make use of the chroot command to set up a working environment for our destination system.

To protect our build system, and to enable the use of virtual and pseudo filesystems on our destination, we undertake much of the Stage 3 process in a 'chroot jail.'

A Note Regarding Package Builds

After the nal GCC compile pass, it is necessary to compile software packages using fresh source les. This is to prevent any previous con guration from breaking the build. We strongly recommend you re-extract source code from the appropriate tar le before building and installing any of the userland binaries.

The packages are compiled and installed in the order given to ensure that all dependencies are met. They must, therefore, be installed in the order speci ed.

Back to top

Required Directories and Devices

Directories

We need to create some directories before mounting filesystems:

mkdir -pv $BROOT/{dev,proc,sys,run}

The output from this command should look like this:

mkdir: created directory '/build/dev'
mkdir: created directory '/build/proc'
mkdir: created directory '/build/sys'
mkdir: created directory '/build/run'

Device Nodes

In addition to the directories required for a fully-functioning system, we need to create some device nodes. These are expected to exist by the kernel and many programs. Execute:

mknod -m 600 $BROOT/dev/console c 5 1
mknod -m 666 $BROOT/dev/null c 1 3

Back to top

Mounting Device, Pseudo, and Virtual Filesystems

The kernel and many user-land programs expect these filesystems to be mounted; vmstat is one such example.

Mounting the dev Filesystem

To populate the dev filesystem, we first need to mount it. We do so by using a "bind" mount, which mounts $BROOT/dev on to the /dev mount of our build system. The result is that $BROOT/dev and /dev will be mounted to the same node, and display the same information.

Normally, the dev filesystem would be populated by dev at boot. However, since we haven't installed the udev package, and since we haven't booted our destination system, we need to populate this filesystem manually.

Execute the following to bind $BROOT/dev to /dev:

mount -v --bind /dev $BROOT/dev

Mounting Other Pseudo and Virtual Filesystems

The gid parameter below ensures that the mount is owned by the group whose ID is 5; we later assign this ID to the 'tty' group when we create our /etc/groups file. The 'mode' parameter sets the file mode (permissions) for the device in question.

mount -vt devpts devpts $BROOT/dev/pts -o gid=5,mode=620
mount -vt proc proc $BROOT/proc
mount -vt sysfs sysfs $BROOT/sys
mount -vt tmpfs tmpfs $BROOT/run

The Special Case of dev/shm

Some distributions link /dev/shm to /run/shm. Since we've created the "run" filesystem above, we only need to create the directory:

if [ -h $BROOT/dev/shm ]; then
   mkdir -pv $BROOT/$(readlink $BROOT/dev/shm)
fi

Back to top

Entering the chroot jail

We're now ready to enter the jail and continue the final build of our distribution.

IF YOU REBOOT, YOU MUST RE-POPULATE /dev AND RE-MOUNT THE PSEUDO/VIRTUAL FILESYSTEMS AS NOTED IN THE PREVIOUS SECTION.

Caveats

The jail provides limited functionality ” we've only installed the most basic of tools. At this stage, the goals are to simulate a running destination system and protect the build system from damage. This is why environment variables and filesystem mounts are so important.

Entering the Jail

You must be logged in as the superuser to execute the chroot command. Note that we specify the $BROOT for bash, which will adopt this location as the root directory of the jail. The "HOME", "TERM", "PATH" and prompt environment variables are passed in to set up our environment inside the jail. Other parameters are explained below:

chroot "$BROOT" /tools/bin/env -i HOME=/root TERM="$TERM" PS1='\u:\w\$ ' PATH=/bin:/usr/bin:/sbin:/usr/sbin:/tools/bin /tools/bin/bash --login +h

• -i :: Instructs env to clear all environment variables upon entering the jail.
• --login :: Notifies bash to provide an interactive login.
• +h :: Disables path-having by bash. This is important; note that the /tools/bin directory (where we've installed many of our binaries) is last on the path. This means that the tools we install at this stage will be found before tools installed at previous stages, which is the desired behavior.

Your output from this command should be something like:

I have no name!:/#

Back to top

Directory Structure

We now need to create the basic directory structure inside our jail, in keeping with the FHS standard.

Creating Directories

mkdir -pv /{bin,boot,etc/{opt,sysconfig},home,lib/firmware,mnt,opt}
mkdir -pv /{media/{floppy,cdrom},sbin,srv,var}
install -dv -m 0750 /root
install -dv -m 1777 /tmp /var/tmp
mkdir -pv /usr/{,local/}{bin,include,lib,sbin,src}
mkdir -pv /usr/{,local/}share/{color,dict,doc,info,locale,man}
mkdir -v  /usr/{,local/}share/{misc,terminfo,zoneinfo}
mkdir -v  /usr/libexec
mkdir -pv /usr/{,local/}share/man/man{1..8}

case $(uname -m) in
 x86_64) ln -sv lib /lib64
 ln -sv lib /usr/lib64
 ln -sv lib /usr/local/lib64 ;;
esac

mkdir -v /var/{log,mail,spool}
ln -sv /run /var/run
ln -sv /run/lock /var/lock
mkdir -pv /var/{opt,cache,lib/{color,misc,locate},local}

Take special note of the permissions assigned to the root home directory and the temporary directories.

Back to top

Required Files

Some user-land programs require the existence of specific files before they can be installed or used. We create these files (or links) here. These will be replaced as we build and install the user-land.

Creating Files and Links

Execute the following:

ln -sv /tools/bin/{bash,cat,echo,pwd,stty} /bin
ln -sv /tools/bin/perl /usr/bin
ln -sv /tools/lib/libgcc_s.so{,.1} /usr/lib
ln -sv /tools/lib/libstdc++.so{,.6} /usr/lib
sed 's/tools/usr/' /tools/lib/libstdc++.la > /usr/lib/libstdc++.la
ln -sv bash /bin/sh

The purpose of each is explained below:

• /bin/bash :: Quite a few scripts expect the bash binary to be located here. Creating this symlink keeps those scripts from breaking.
• /bin/cat :: Glibc hard-codes this pathname into its configure script.
• /bin/echo :: Used by Glibc's test-suite; this path is also hard-coded.
• /bin/pwd :: Used by Glibc; this path is hard-coded.
• /bin/stty :: This pathname is hard-coded by the expect package.
• /usr/bin/perl :: Many scripts expect to find the PERL binary at this location; this keeps them from breaking.
• /usr/lib/libgcc\_s.so{,.1} :: This is need by Glibc to enable POSIX threads.
• /usr/lib/libstdc++{,.6} :: Needed for C++ support by GMP and Glibc's test suite.
• /usr/lib/libstdc++.la :: This prevents GCC from referencing a previously built library of the same name in our tools directory.
• /bin/sh :: Some scripts hard-code this binary path.

mtab

The kernel historically exposes the list of mounted filesystems via the /etc/mtab file. Some user-land binaries expect this information available. To do this, execute:

ln -sv /proc/self/mounts /etc/mtab

/etc/passwd

The need for this file is obvious, without it, we can't set passwords or log in to our system once we boot it.

cat > /etc/passwd << "EOF"
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/dev/null:/bin/false
daemon:x:6:6:Daemon User:/dev/null:/bin/false
messagebus:x:18:18:D-Bus Message Daemon User:/var/run/dbus:/bin/false
nobody:x:99:99:Unprivileged User:/dev/null:/bin/false
EOF

/etc/group

This is another file we need to make use of file permissions properly.

cat > /etc/group << "EOF"
root:x:0:
bin:x:1:daemon
sys:x:2:
kmem:x:3:
tape:x:4:
tty:x:5:
daemon:x:6:
floppy:x:7:
disk:x:8:
lp:x:9:
dialout:x:10:
audio:x:11:
video:x:12:
utmp:x:13:
usb:x:14:
cdrom:x:15:
adm:x:16:
messagebus:x:18:
systemd-journal:x:23:
input:x:24:
mail:x:34:
nogroup:x:99:
users:x:999:
EOF

Re-executing Our Login Shell

The following command instructs bash to execute another login. This results in our bash prompt changing, as we've now installed the etc/passwd and /etc/group files.

exec /tools/bin/bash --login +h

Your login prompt should now reflect the user root.

Mandatory Log Files

We need to populate /var/log with files that are expected by a number of user-land utilities.

touch /var/log/{btmp,lastlog,faillog,wtmp}
chgrp -v utmp /var/log/lastlog
chmod -v 664  /var/log/lastlog
chmod -v 600  /var/log/btmp

Back to top

Just as during the second stage build, we need to extract the kernel headers for GCC and other programs that need them. Note that we're installing them in a different location this time.

Change into the kernel source directory: /sources/linux-4.x

Change into this directory and issue the following command:

make mrproper

That done, we now call the make command, specifying the header file output path; we do this in two steps because make wipes any files from the destination directory during this step. First, we extract the files:

make INSTALL_HDR_PATH=dest headers_install

Now we remove any files that are not specifically needed, which leaves us with only the headers which are intended to be exposed to the user-land:

find dest/include \( -name .install -o -name ..install.cmd \) -delete

And secondly, we copy them to the proper location:

cp -rv dest/include/* /usr/include

Installing man Pages

In the man-pages source directory, execute:

make install

Back to top

Building Glibc, Stage 3

This is our final build of glibc!

Patching the Source

To ensure compliance with the FHS, we need to patch the glibc source code. Execute the following commands to copy the patch file to the source directory and apply the patch:

cp ../glibc-2.24-fhs-1.patch .
patch -Np1 -i glibc-2.24-fhs-1.patch

Configure

../configure --prefix=/usr          \
             --enable-kernel=2.6.32 \
             --enable-obsolete-rpc

These options are identical to those used to configure glibc in our first stage.

Compile

make

Test

Unlike previous build stages, we now have all of the dependencies needed to run the tests against our binaries before we install them. This step is critical! Any problems with them build toolchain, the environment, library paths, etc., will likely reveal themselves at this point.

make check

Note that you will have some of the tests fail, in particular those related to the getaddrinfo functions. Overall, your output from make check should look something like this:

UNSUPPORTED: elf/tst-audit10
XPASS: elf/tst-protected1a
XPASS: elf/tst-protected1b
UNSUPPORTED: math/test-double-libmvec-alias-avx2
UNSUPPORTED: math/test-double-libmvec-alias-avx2-main
UNSUPPORTED: math/test-double-libmvec-alias-avx512
UNSUPPORTED: math/test-double-libmvec-alias-avx512-main
UNSUPPORTED: math/test-double-libmvec-sincos-avx2
UNSUPPORTED: math/test-double-libmvec-sincos-avx512
UNSUPPORTED: math/test-float-libmvec-alias-avx2
UNSUPPORTED: math/test-float-libmvec-alias-avx2-main
UNSUPPORTED: math/test-float-libmvec-alias-avx512
UNSUPPORTED: math/test-float-libmvec-alias-avx512-main
UNSUPPORTED: math/test-float-libmvec-sincosf-avx2
UNSUPPORTED: math/test-float-libmvec-sincosf-avx512
FAIL: posix/tst-getaddrinfo4
FAIL: posix/tst-getaddrinfo5
Summary of test results:
      2 FAIL
   2478 PASS
     13 UNSUPPORTED
     43 XFAIL
      2 XPASS
make[1]: *** [Makefile:331: tests] Error 1
make[1]: Leaving directory '/sources/glibc-2.24'
make: *** [Makefile:9: check] Error 2

Create /etc/ld.so.conf

make will throw an error if this file does not exist when the install is run.

touch /etc/ld.so.conf

Install

make install

Install the Configuration File and Runtime for nscd

 cp -v ../nscd/nscd.conf /etc/nscd.conf
 mkdir -pv /var/cache/nscd

Installing Locale Files

The following locales should be defined, if only to enable compliance with future tests.

Create The Locales Directory

mkdir -pv /usr/lib/locale

Install the Locale Definitions

localedef -i cs_CZ -f UTF-8 cs_CZ.UTF-8
localedef -i de_DE -f ISO-8859-1 de_DE
localedef -i de_DE@euro -f ISO-8859-15 de_DE@euro
localedef -i de_DE -f UTF-8 de_DE.UTF-8
localedef -i en_GB -f UTF-8 en_GB.UTF-8
localedef -i en_HK -f ISO-8859-1 en_HK
localedef -i en_PH -f ISO-8859-1 en_PH
localedef -i en_US -f ISO-8859-1 en_US
localedef -i en_US -f UTF-8 en_US.UTF-8
localedef -i es_MX -f ISO-8859-1 es_MX
localedef -i fa_IR -f UTF-8 fa_IR
localedef -i fr_FR -f ISO-8859-1 fr_FR
localedef -i fr_FR@euro -f ISO-8859-15 fr_FR@euro
localedef -i fr_FR -f UTF-8 fr_FR.UTF-8
localedef -i it_IT -f ISO-8859-1 it_IT
localedef -i it_IT -f UTF-8 it_IT.UTF-8
localedef -i ja_JP -f EUC-JP ja_JP
localedef -i ru_RU -f KOI8-R ru_RU.KOI8-R
localedef -i ru_RU -f UTF-8 ru_RU.UTF-8
localedef -i tr_TR -f UTF-8 tr_TR.UTF-8
localedef -i zh_CN -f GB18030 zh_CN.GB18030

Glibc Configuration (Post-install)

We undertake the following steps to configure glibc after installing.

Add /etc/nsswitch.conf

cat > /etc/nsswitch.conf << "EOF"
# Begin /etc/nsswitch.conf

passwd: files
group: files
shadow: files

hosts: files dns
networks: files

protocols: files
services: files
ethers: files
rpc: files

# End /etc/nsswitch.conf
EOF

Add Timezone Files and Configure Our Timezone

For this step, we stay in the glibc build directory, and unzip the timezone data files to our current location.

tar -xf ../../tzdata2016f.tar.gz

Now we need to configure our timezones:

export ZONEINFO=/usr/share/zoneinfo
mkdir -pv $ZONEINFO/{posix,right}
for tz in etcetera southamerica northamerica europe africa antarctica  \
          asia australasia backward pacificnew systemv; do
    zic -L /dev/null   -d $ZONEINFO       -y "sh yearistype.sh" ${tz}
    zic -L /dev/null   -d $ZONEINFO/posix -y "sh yearistype.sh" ${tz}
    zic -L leapseconds -d $ZONEINFO/right -y "sh yearistype.sh" ${tz}
done
cp -v zone.tab zone1970.tab iso3166.tab $ZONEINFO
zic -d $ZONEINFO -p Etc/GMT
zic -d $ZONEINFO -p Etc/UTC
unset ZONEINFO

Set the Local Timezone

cp -v /usr/share/zoneinfo/Etc/UTC /etc/localtime

Configuring the Dynamic Loader

The dynamic loader is what allows shared objects (libraries) to be called at run-time by any given binary. In this step, inform glibc what paths to search when configuring the dynamic loader:

cat >> /etc/ld.so.conf << "EOF"
# Add an include directory
include /etc/ld.so.conf.d/*.conf

EOF

And last, create the directory referenced in /etc/ld.so.conf:

mkdir -pv /etc/ld.so.conf.d

Back to top

Adjusting the Toolchain

Now that we've got a native glibc installed, we need to reconfigure our toolchain so that it looks for libraries and utilities inside the root filesystem hierarchy instead of looking in /tools.

Backup the Existing Linker

Let's backup the existing linker, and replace it with the one we've most recently built:

mv -v /tools/bin/{ld,ld-old}
mv -v /tools/$(uname -m)-pc-linux-gnu/bin/{ld,ld-old}
mv -v /tools/bin/{ld-new,ld}
ln -sv /tools/bin/ld /tools/$(uname -m)-pc-linux-gnu/bin/ld

Reconfigure GCC

We now need to alter the GCC so that it uses the new dynamic linker, headers and glibc libraries:

gcc -dumpspecs | sed -e 's@/tools@@g'                   \
    -e '/\*startfile_prefix_spec:/{n;s@.*@/usr/lib/ @}' \
    -e '/\*cpp:/{n;s@$@ -isystem /usr/include@}' >      \
    `dirname $(gcc --print-libgcc-file-name)`/specs 

Test the GCC Configuration

It is imperative that GCC use the proper linker and libraries when compiling. We can check that using the following commands:

echo 'int main(){}' > dummy.c
cc dummy.c -v -Wl,--verbose &> dummy.log
readelf -l a.out | grep ': /lib'

Your output should be something like this:

[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]

Be certain that the interpreter path does not include the string 'tools'. If it does, GCC has not been properly reconfigured and will break.

Additionally, we should check to ensure GCC is using the proper glibc startup files:

grep -o '/usr/lib.*/crt[1in].*succeeded' dummy.log

Your output should resemble something like this:

/usr/lib/../lib64/crt1.o succeeded
/usr/lib/../lib64/crti.o succeeded
/usr/lib/../lib64/crtn.o succeeded

Back to top

Installing binutils' Dependencies

The test suite included with binutils depends on the zlib and file packages. Let's install these before moving on to binutils itself.

Installing zlib

Configure

zlib does not make use of a configure script.

Compile

make

Make check

make check

Install

make install

Post-configuration

The shared library needs to be moved to /lib, and a symlink needs to be created in /usr/lib. This is to ensure compliance with FHS:

mv -v /usr/lib/libz.so.* /lib
rm /usr/lib/libz.so
ln -sfv /lib/libz.so.1.2.8 /usr/lib/libz.so

Installing file

This is another package upon which the binutils test suite depends.

Configure

./configure --prefix=/usr

Compile

make

Test

make check

Install

make install

Back to top

Installing binutils (Stage 3)

This is our third (and final) installation of binutils.

Start With Fresh Source

Firstly, delete any existing binutils source directory, an re-extract the source code from the tar archive: rm -rf binutils-2.27 tar xf binutils-2.27.tar

Check to Ensure ptys Work in the Jail

We need this to perform the tests against the binutils build. Execute: expect -c "spawn ls"

The output should be: spawn ls

Configure and Build

Create the build directory as per usual, and issue the following commands to configure, build, test, and install binutils: ../configure --prefix=/usr --enable-shared --disable-werror make tooldir=/usr make -k check make tooldir=/usr install

Back to top

Installing binutils Stage 3

This is our third (and final) installation of binutils.

Start with a Fresh Source

First, delete any existing binutils source directory, and re-extract the source code from the tar archive:

rm -rf binutils-2.27
tar xf binutils-2.27.tar

Check to Ensure ptys Work in the Jail

We need this to perform the tests against the binutils build. Execute:

expect -c "spawn ls"

The output should be:

spawn ls

Configure and Build

Create the build directory as usual, and issue the following commands to configure, build, test and install binutils:

../configure --prefix=/usr --enable-shared --disable-werror
make tooldir=/usr
make -k check
make tooldir=/usr install

Back to top

Installing GMP

The GMP package contains several math libraries. These are used mainly with precision arithmetic.

Unlike glibc and gcc, GMP does not require the use of a build directory.

Configure and Compile

The configure options are explained below. Execute:

./configure --prefix=/usr --enable-cxx --disable-static --docdir=/usr/share/doc/gmp-6.1.1

• prefix :: Tells the configure script where the compiled binaries should be installed.
• enable-cxx :: Enables C++ support.
• disable-static :: Disables the generation of static libraries.
• docdir :: Specifies the prefix for installation documentation.

Compile

make && make html

Test

make check 2>&1 | tee gmp-check-log

Ensure that all 190 of the tests have passed using the following command:

awk '/# PASS:/{total+=$3} ; END{print total}' gmp-check-log

Install

make install && make install-html

Back to top

Installing MPFR

Unlike glibc and gcc, MPFR does not require the use of a build directory.

Configure and Compile

The configure options are explained below. Execute:

./configure --prefix=/usr --disable-static  --enable-thread-safe --docdir=/usr/share/doc/mpfr-3.1.4

• prefix :: Tells the configure script where the compiled binaries should be installed.
• disable-static :: Disables the generation of static libraries.
• enable-thread-safe :: Enables thread-handling in the library.
• docdir :: Specifies the prefix for installation documentation.

Compile

make && make html

Test

make check 

Install

make install && make install-html

Back to top

Installing MPC

The MPC package installs libraries which handle rounding and high-precision numbers.

Unlike glibc and gcc, MPC does not require the use of a build directory.

Configure and Compile

./configure --prefix=/usr --disable-static --docdir=/usr/share/doc/mpc-1.0.3

Compile

make && make html

Test

make check 

Install

make install && make install-html

Back to top

Installing GCC

This is the fourth (and final) build of GCC. Make sure you've deleted the existing GCC directory, and re-extracted the source code from the tar file.

Configure and Compile

Create the build directory, as per usual. Note that the SED variable here prevents GCC from hard-coding a path to the sed binary.

Note as well that we do not copy the mpc, mpfr or gap source packages into the GCC source directory, as we've installed these previously on the system.

Execute the following to begin the build:

export SED=sed                               
../configure --prefix=/usr --enable-languages=c,c++ --disable-multilib --disable-bootstrap --with-system-zlib

• prefix :: Tells the configure script where the compiled binaries should be installed.
• enable-languages :: We only need C and C++ right now. Others can be added later.
• disable-multilib :: This functionality isn't supported on the x86_64 platform.
• disable-bootstrap :: We prevent GCC from performing a bootstrap build. As we've built GCC as a cross compiler, and then built GCC natively, we are now using the native build to compile GCC a third time. A bootstrapped build replicates these steps, which should be unnecessary if all of the testing has been executed as recommended.
• with-system-zib :: This instructs make to use the system zlib library instead of that bundled with GCC.

Compile

make -j2

Test

Before we run the tests, we need to make sure the stack size is large enough to accommodate the testing software. We increase the stack size using the ulimit command:

ulimit -s 32768

And now run our tests:

make -k check 

Check the Test Results

You should have very few errors reported.

../contrib/test_summary

Install

make install 

Add FHS-Mandated Symlink:

ln -sv ../usr/bin/cpp /lib

Add a cc Link to the GCC Binary

ln -sv gcc /usr/bin/cc

Add Compatibility Symlinks

install -v -dm755 /usr/lib/bfd-plugins
ln -sfv ../../libexec/gcc/$(gcc -dumpmachine)/6.2.0/liblto_plugin.so /usr/lib/bfd-plugins/

Sanity Check

Once again, we stop here to check the functionality of our toolchain. Execute the following commands:

echo 'int main(){}' > dummy.c
cc dummy.c -v -Wl,--verbose &> dummy.log
readelf -l a.out | grep ': /lib'

Check GCC Startup

Now we want to check that GCC is using the proper set of files when it starts:

grep -o '/usr/lib.*/crt[1in].*succeeded' dummy.log

Result:

/usr/lib/gcc/i686-pc-linux-gnu/6.2.0/../../../crt1.o succeeded
/usr/lib/gcc/i686-pc-linux-gnu/6.2.0/../../../crti.o succeeded
/usr/lib/gcc/i686-pc-linux-gnu/6.2.0/../../../crtn.o succeeded

grep -B4 '^ /usr/include' dummy.log

Results:

##include <...> search starts here:
 /usr/lib/gcc/i686-pc-linux-gnu/6.2.0/include
 /usr/local/include
 /usr/lib/gcc/i686-pc-linux-gnu/6.2.0/include-fixed
 /usr/include

Verify the Linker Uses Proper Search Paths

grep 'SEARCH.*/usr/lib' dummy.log |sed 's|; |\n|g'
SEARCH_DIR("/usr/x86_64-unknown-linux-gnu/lib64")
SEARCH_DIR("/usr/local/lib64")
SEARCH_DIR("/lib64")
SEARCH_DIR("/usr/lib64")
SEARCH_DIR("/usr/x86_64-unknown-linux-gnu/lib")
SEARCH_DIR("/usr/local/lib")
SEARCH_DIR("/lib")
SEARCH_DIR("/usr/lib");

Ensure We're Using the Proper Libc Implementation

grep "/lib.*/libc.so.6 " dummy.log

Result:

attempt to open /lib/libc.so.6 succeeded

Ensure GCC Uses the Correct Dynamic Linker

grep found dummy.log

Result:

found ld-linux.so.2 at /lib/ld-linux.so.2

Cleanup

rm -v dummy.c a.out dummy.log

Move a Misplaced File

Only execute this step if GCC has built correctly and all of the sanity checks are passed:

mkdir -pv /usr/share/gdb/auto-load/usr/lib
mv -v /usr/lib/*gdb.py /usr/share/gdb/auto-load/usr/lib

Back to top

Installing Bzip2

This package does not require the use of a build directory. Be sure to start with a fresh source directory extracted from the tar file.

Patch and Amend the Source

Change into the bzip2 source directory, and run the following command to patch the source:

patch -Np1 -i ../bzip2-1.0.6-install_docs-1.patch

Also, we need to amend the source code to ensure that symbolic links are installed using relative paths:

sed -i 's@\(ln -s -f \)$(PREFIX)/bin/@\1@' Makefile

This amendment ensures man page are installed to the proper location:

sed -i "s@(PREFIX)/man@(PREFIX)/share/man@g" Makefile

And now we need to reconstruct the make file. The following commands cause make to use a different file; the makefile we create here adds a libb2.s shared library, and links the bzip2 utilities against it. Execute:

make -f Makefile-libbz2_so
make clean

Compile and Install

make
make PREFIX=/usr install

Copy the Shared Binary into the Appropriate Directory and Make Symbolic Links:

cp -v bzip2-shared /bin/bzip2
cp -av libbz2.so* /lib
ln -sv ../../lib/libbz2.so.1.0 /usr/lib/libbz2.so
rm -v /usr/bin/{bunzip2,bzcat,bzip2}
ln -sv bzip2 /bin/bunzip2
ln -sv bzip2 /bin/bzcat

Back to top

Installing pkg-config

pkg-config does not require the use of a build directory.

Configure and Compile

The configure options are explained below. Execute:

./configure --prefix=/usr --with-internal-glib --disable-compile-warnings --disable-host-tool --docdir=/usr/share/doc/pkg-config-0.29.1

• prefix :: Tells the configure script where the compiled binaries should be installed.
• with-internal-glib :: Allows pkg-config to use an internal glibc, as a system version is not available.
• disable-compile-warnings :: Prevents the use of compilation flags which could result in build failure.
• disable-host-tool :: Disables the creation of a hard-link to the pkg-config binary.

Compile

make 

Test

make check 

Install

make install 

Back to top

Installing ncurses

This package does not require the use of a build directory. Be sure to start with a fresh source directory extracted from the tar file.

Amend the Source

This command prevents the installation of a static library otherwise not handled by configure:

sed -i '/LIBTOOL_INSTALL/d' c++/Makefile.in

Configure and Compile

The configure options are explained below. Execute:

./configure --prefix=/usr --mandir=/usr/share/man --with-shared --without-debug --without-normal --enable-pc-files --enable-widec

• widec :: Causes wide-character libraries to be installed. Wide character libraries are usable by both multibyte and traditional locales.
• enable-pc-file :: Generates .pc files for pkg-config.
• without-normal :: Disables the building of most static libraries.

Compile

make 

Test

make check 

Install

make install

Move the shared libraries to the /lib directory:

mv -v /usr/lib/libncursesw.so.6* /lib

Because we've moved the library files, there is now a symlink to a non-existent file. So we need to re-create it:

ln -sfv ../../lib/$(readlink /usr/lib/libncursesw.so) /usr/lib/libncursesw.so

These symlinks make the wide-character libraries accessible to binaries which expect to find the non-wide libraries:

for lib in ncurses form panel menu ; do
    rm -vf                    /usr/lib/lib${lib}.so
    echo "INPUT(-l${lib}w)" > /usr/lib/lib${lib}.so
    ln -sfv ${lib}w.pc        /usr/lib/pkgconfig/${lib}.pc
done

This snippet ensures that binaries which look for -lcurses at build time are still buildable:

rm -vf                     /usr/lib/libcursesw.so
echo "INPUT(-lncursesw)" > /usr/lib/libcursesw.so
ln -sfv libncurses.so      /usr/lib/libcurses.so

Install the documentation:

mkdir -v       /usr/share/doc/ncurses-6.0
cp -v -R doc/* /usr/share/doc/ncurses-6.0

Back to top

Installing attr

This package does not require the use of a build directory.

Amend the Source

Amend the location of the documentation directory so that it uses a version number:

sed -i -e 's|/@pkg_name@|&-@pkg_version@|' include/builddefs.in

Prevent make from overwriting man pages installed by the man-pages package:

sed -i -e "/SUBDIRS/s|man[25]||g" man/Makefile

Configure and Compile

Execute:

./configure --prefix=/usr --bindir=/bin --disable-static

Compile

make 

Test

make -j1 tests root-tests

Install

make install install-dev install-lib

Change the permissions on the shared library:

chmod -v 755 /usr/lib/libattr.so

Move the shared library to /lib:

mv -v /usr/lib/libattr.so.* /lib

Which results in a missing library in /usr/lib, which we recreate with a symlink:

ln -sfv ../../lib/$(readlink /usr/lib/libattr.so) /usr/lib/libattr.so

Back to top

Installing acl

This package does not require the use of a build directory.

Amend the Source

Amend the location of the documentation directory so that it uses a version number:

sed -i -e 's|/@pkg_name@|&-@pkg_version@|' include/builddefs.in

Fix some broken tests:

sed -i "s:| sed.*::g" test/{sbits-restore,cp,misc}.test

Fix a bug which causes getfacl -e to segfault on long group names:

sed -i -e "/TABS-1;/a if (x > (TABS-1)) x = (TABS-1);" libacl/__acl_to_any_text.c

Configure and Compile

Execute:

./configure --prefix=/usr --bindir=/bin --disable-static --libexecdir=/usr/lib

Compile

make 

Install

make install install-dev install-lib

Change the permissions on the shared library:

chmod -v 755 /usr/lib/libacl.so

Move the shared library to /lib:

mv -v /usr/lib/libacl.so.* /lib

Which results in a missing library in /usr/lib, which we recreate with a symlink:

ln -sfv ../../lib/$(readlink /usr/lib/libacl.so) /usr/lib/libacl.so

Back to top

Installing libpcap

This package does not require the use of a build directory.

Amend the Source

Prevent make from installing a static library:

sed -i '/install.*STALIBNAME/d' libcap/Makefile

Configure and Compile

The package doesn't make use of a configure script:

make

Install

The RAISE_SETFCAP variable defined on the command line prevents the use of setcap on the resulting library. This is necessary if the kernel or filesystem doesn't not support extended capabilities.

make RAISE_SETFCAP=no prefix=/usr install
chmod -v 755 /usr/lib/libcap.so

Move the shared library to /lib:

mv -v /usr/lib/libcap.so.* /lib

Which results in a missing library in /usr/lib, which we recreate with a symlink:

ln -sfv ../../lib/$(readlink /usr/lib/libcap.so) /usr/lib/libcap.so

Back to top

Installing sed

This package does not require the use of a build directory. Be sure to start with a fresh source directory extracted from the tar file.

Configure

./configure --prefix=/usr --bindir=/bin --htmldir=/usr/share/doc/sed-4.2.2

Compile

make
make html

Test

Note that the 'version' test may fail, resulting in a total of 5 failed tests of 66:

make check 

Install

make install
make -C doc install-html

Back to top

Installing shadow

This package does not require the use of a build directory.

Amend the Source

Prevent make from installing groups-related programs and related man pages. These will be installed by the coreutils package.

sed -i '/install.*STALIBNAME/d' libcap/Makefile
find man -name Makefile.in -exec sed -i 's/groups\.1 / /'   {} \;
find man -name Makefile.in -exec sed -i 's/getspnam\.3 / /' {} \;
find man -name Makefile.in -exec sed -i 's/passwd\.5 / /'   {} \;

Enable the use of the SHA-512 algorithm for password encryption:

sed -i -e 's@#ENCRYPT_METHOD DES@ENCRYPT_METHOD SHA512@' etc/login.defs

Change the location of user mailboxes from /var/spool/mail to /var/mail:

sed -i  -e 's@/var/spool/mail@/var/mail@' etc/login.defs

This amendment renders the 'user add' binary consistent with LFS:

sed -i 's/1000/999/' etc/useradd

Configure

./configure --sysconfdir=/etc --with-group-name-max-length=32

Compile and Install

make && make install

Move a misplaced binary to its proper location:

mv -v /usr/bin/passwd /bin

Post-install Configuration

To enable shadowed passwords, run the following command:

pwconv

To enable shadowed groups, execute this command:

grpconv

Back to top

Set the root Password

Now that we've installed the /etc/passwd file and the utilities to manage passwords, change the root password.

If you receive an error message about root not existing in /etc/passwd, there is likely a problem with the file.

Back to top

Installing psmisc

This package does not require the use of a build directory.

Configure

./configure --prefix=/usr

Compile

make 

Install

make install 

Move

Move the 'killall' and 'fuser' binaries to FHS-compliant locations:

mv -v /usr/bin/fuser   /bin
mv -v /usr/bin/killall /bin

Back to top

Installing IANA-etc

This package does not require the use of a build directory.

Configure

This package does not make use of a configuration script.

Compile

make 

Install

make install 

Back to top

Installing M4

This package does not require the use of a build directory.

Configure

./configure --prefix=/usr

Compile

make 

Test

M4 may fail the 'test-update-copyright' test; you check this by using cat to check the contents of tests/test-suite.log.

make check

Install

make install 

Back to top

Installing bison

This package does not require the use of a build directory.

Configure

./configure --prefix=/usr --docdir=/usr/share/doc/bison-3.0.4

Compile

make 

Install

make install 

Back to top

Installing flex

This package does not require the use of a build directory.

Configure

./configure --prefix=/usr --docdir=/usr/share/doc/flex-2.6.1

Compile

make 

Install

make install 

Compatibility Symlink

flex is predated by a software package by the name of lex. Some user-land binaries may be expecting lex to be installed, and may not call flex. However, this software package does provide for the emulation of lex using a symlink:

ln -sv flex /usr/bin/lex

Back to top

Installing grep

This package does not require the use of a build directory.

Configure

./configure --prefix=/usr --bindir=/bin

Compile

make 

Test

make check

Install

make install 

Back to top

Installing readline

This package does not require the use of a build directory. Be sure to start with a fresh source directory extracted from the tar file.

Patch and Amend the Source

Change to the readline source directory, and run the following command to patch the source:

patch -Np1 -i ../readline-6.3-upstream_fixes-3.patch

Reinstalling this package results in old libraries being moved, with the suffix .old attached to their filename. This can trigger bugs in ldconfig. We can avoid this behavior by removing the renaming:

sed -i '/MV.*old/d' Makefile.in
sed -i '/{OLDSUFF}/c:' support/shlib-install

Configure

./configure --prefix=/usr  --disable-static --docdir=/usr/share/doc/readline-6.3

Compile and Install

make SHLIB_LIBS=-lncurses
make SHLIB_LIBS=-lncurses install

Copy the Shared Binary into the Appropriate Directory and Make Symbolic Links:

mv -v /usr/lib/lib{readline,history}.so.* /lib
ln -sfv ../../lib/$(readlink /usr/lib/libreadline.so) /usr/lib/libreadline.so
ln -sfv ../../lib/$(readlink /usr/lib/libhistory.so ) /usr/lib/libhistory.so

Install the Documentation

install -v -m644 doc/*.{ps,pdf,html,dvi} /usr/share/doc/readline-6.3

Back to top

Installing bash

This package does not require the use of a build directory. Be sure to start with a fresh source directory extracted from the tar file.

Patch

patch -Np1 -i ../bash-4.3.30-upstream_fixes-3.patch

Configure

./configure --prefix=/usr --docdir=/usr/share/doc/bash-4.3.30 --without-bash-malloc --with-installed-readline

Compile

make

Test

First, we need to change the permissions on the source tree so that the nobody user can write:

chown -Rv nobody

Now, execute the tests as the nobody user:

su nobody -s /bin/bash -c "PATH=$PATH make tests"

Install

make install
mv -vf /usr/bin/bash /bin

To Use the Newly Installed bash:

exec /bin/bash --login +h

Back to top

Installing bc

This package does not require the use of a build directory.

Patch

patch -Np1 -i ../bc-1.06.95-memory_leak-1.patch

Configure

./configure --prefix=/usr --with-readline --mandir=/usr/share/man --infodir=/usr/share/info

Compile

make 

Test

echo "quit" | ./bc/bc -l Test/checklib.b

Install

make install 

Back to top

Installing lib tool

This package does not require the use of a build directory.

Configure

./configure --prefix=/usr 

Compile

make 

Test

Libtool will fail several tests, due to unmet dependencies on the automake package.

make check

Install

make install 

Back to top

Installing GBDM

This package does not require the use of a build directory.

Configure

./configure --prefix=/usr --disable-static --enable-libgdbm-compat

Compile

make 

Test

make check

Install

make install 

Back to top

Installing Gperf

This package does not require the use of a build directory.

Configure

./configure --prefix=/usr --docdir=/usr/share/doc/gperf-3.0.4

Compile

make 

Test

make -j1 check

Install

make install 

Back to top

Installing expat

This package does not require the use of a build directory.

Configure

./configure --prefix=/usr --disable-static

Compile

make 

Test

make check

Install

make install 
install -v -dm755 /usr/share/doc/expat-2.2.0
install -v -m644 doc/*.{html,png,css} /usr/share/doc/expat-2.2.0

Back to top

Installing inetutils

This package does not require the use of a build directory.

Configure

./configure --prefix=/usr --localstatedir=/var --disable-logger  --disable-whois --disable-rcp  --disable-rexec  --disable-rlogin --disable-rsh  --disable-servers

• disable-logger :: This prevents the installation of the logger program; this same program is provided (albeit, a more recent version) by the util-linux package.
• disable-whois :: Disables the building of the whois client.
• disable-rcp/rexec/rlogin/rsh :: Disables the building of obsolete programs whose functionality has been replaced by OpenSSH.
• disable-servers :: This disables the installation of a number of network services included in the package. Most of these are insecure.

Compile

make 

Test

make check

Install

make install 
mv -v /usr/bin/{hostname,ping,ping6,traceroute} /bin
mv -v /usr/bin/ifconfig /sbin

Back to top

Installing PERL

This package does not require the use of a build directory. Be sure to start with a fresh source directory extracted from the tar file.

Pre-configuration

Install the /etc/hosts file, which is required by PERL's configuration files:

echo -e "127.0.0.1\tlocalhost\t$(hostname)" > /etc/hosts

Export the following variables:

export BUILD_ZLIB=False
export BUILD_BZIP2=0

Configure

PERL uses a special configuration script, as we saw previously:

sh Configure -des -Dprefix=/usr -Dvendorprefix=/usr -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dpager="/usr/bin/less -isR" -Duseshrplib

• Dvendorprefix=/usr :: Notifies PERL where packages should install their PERL modules.
• Dpager="/usr/bin/less -isR" :: Instructs PERL to use less as a pager instead of more.
• Dman3dir=/usr/share/man/man3 :: PERL depends on groff, which is not installed yet. This directive instructs make to build man pages anyway.
• Duseshrplib :: Build a shared library.

Make

make

Test

make -k test

Install

make install

Cleanup

unset BUILD_ZLIB BUILD_BZIP2

Back to top

Installing XML::Parser

This is a PERL module which incorporates the functionality of the expat binary into PERL.

Configuration

perl Makefile.PL

Compile

make

Test

make test

Install

make install

Back to top

Installing intltool

This package does not require the use of a build directory.

Amend the Source

PERL version 5.22 and up throw a warning unless the source is amended as follows:

sed -i 's:\\\${:\\\$\\{:' intltool-update.in

Configuration

./configure --prefix=/usr

Compile

make

Test

make check

Install

make install
install -v -Dm644 doc/I18N-HOWTO /usr/share/doc/intltool-0.51.0/I18N-HOWTO

Back to top

Installing autoconf

This package does not require the use of a build directory.

Configuration

./configure --prefix=/usr

Compile

make

Test

make check

Install

make install

Back to top

Installing automake

This package does not require the use of a build directory.

Amend the Source Code

This fixes warnings thrown by PERL 5.22 and above.

sed -i 's:/\\\${:/\\\$\\{:' bin/automake.in

Configuration

./configure --prefix=/usr --docdir=/usr/share/doc/automake-1.15

Compile

make

Test

Some of the tests link to the wrong version of flex; we use the sed command to fix this:

sed -i "s:./configure:LEXLIB=/usr/lib/libfl.a &:" t/lex-{clean,depend}-cxx.sh
make -j2 check

Install

make install

Back to top

Installing xz

This package does not require the use of a build directory.

Amend the Source Code

This fixes warnings thrown by PERL 5.22 and above.

sed -e '/mf\.buffer = NULL/a next->coder->mf.size = 0;' -i src/liblzma/lz/lz_encoder.c

Configuration

./configure --prefix=/usr --disable-static --docdir=/usr/share/doc/xz-5.2.

Compile

make

Test

make check

Install

make install
mv -v   /usr/bin/{lzma,unlzma,lzcat,xz,unxz,xzcat} /bin
mv -v /usr/lib/liblzma.so.* /lib
ln -svf ../../lib/$(readlink /usr/lib/liblzma.so) /usr/lib/liblzma.so

Back to top

Installing kmod

This package does not require the use of a build directory.

Configuration

./configure --prefix=/usr --bindir=/bin --sysconfdir=/etc --with-rootlibdir=/lib --with-xz --with-zlib

Compile

make

Install

We create symlinks here, which provide backwards compatibility with the module-init tools, which previously provided loadable module support for the kernel.

make install
for target in depmod insmod lsmod modinfo modprobe rmmod; do
  ln -sfv ../bin/kmod /sbin/$target
done
ln -sfv kmod /bin/lsmod

Back to top

Installing gettext

This package does not require the use of a build directory. Be sure to start with a fresh source directory extracted from the tar file.

Configuration

./configure --prefix=/usr --disable-static --docdir=/usr/share/doc/gettext-0.19.8.1

Compile

make

Test

make check

Install

make install
chmod -v 0755 /usr/lib/preloadable_libintl.so

Back to top

Installing procps-ng

This package does not require the use of a build directory.

Configuration

./configure --prefix=/usr --exec-prefix= --libdir=/usr/lib --docdir=/usr/share/doc/procps-ng-3.3.12 --disable-static --disable-kill

Compile

make

Test

We need to modify the test code so that it does not rely on the presence of a tty device; otherwise, some tests will fail:

sed -i -r 's|(pmap_initname)\\\$|\1|' testsuite/pmap.test/pmap.exp
make check

Install

make install
mv -v /usr/lib/libprocps.so.* /lib
ln -sfv ../../lib/$(readlink /usr/lib/libprocps.so) /usr/lib/libprocps.so

Back to top

Installing e2fsprogs

This package does require the use of a build directory like binutils, GCC and glibc.

Amend the Source

We amend one of the test scripts here:

sed -i -e 's:\[\.-\]::' tests/filter.sed

Configure

The environment variables defined here instruct configure to look for libraries in the specified locations:

export LIBS=-L/tools/lib 
export CFLAGS=-I/tools/include 
export PKG_CONFIG_PATH=/tools/lib/pkgconfig
../configure --prefix=/usr --bindir=/bin --with-root-prefix="" --enable-elf-shlibs --disable-libblkid --disable-libuuid --disable-uuidd --disable-fsck

• bindir=/bin :: Ensure that the compiled binaries are installed in /lib and /sbin.
• enable-elf-shlibs :: Creates shared libraries used by the binaries installed as part of the package.
• disable :: Prevents the installation of the specified libraries, as more recent versions are installed by the util-linux package.

Compile

make

Test

For the tests to run properly, we need to link some shared libraries to a place where the tests can find them:

ln -sfv /tools/lib/lib{blk,uu}id.so.1 lib
make LD_LIBRARY_PATH=/tools/lib check

Install

make install
make install-libs
chmod -v u+w /usr/lib/{libcom_err,libe2p,libext2fs,libss}.a
gunzip -v /usr/share/info/libext2fs.info.gz
install-info --dir-file=/usr/share/info/dir /usr/share/info/libext2fs.info
makeinfo -o      doc/com_err.info ../lib/et/com_err.texinfo
install -v -m644 doc/com_err.info /usr/share/info
install-info --dir-file=/usr/share/info/dir /usr/share/info/com_err.info

Cleanup

DO NOT FORGET TO UNSET THESE ENVIRONMENT VARIABLES!

unset LIBS CFLAGS PKG_CONFIG_PATH

Back to top

Installing coreutils

This package does not require the use of a build directory. Be sure to start with a fresh source directory extracted from the tar file.

Patch the Source

patch -Np1 -i ../coreutils-8.25-i18n-2.patch

Configure

The variable defined here allows coreutils to be built as root:

FORCE_UNSAFE_CONFIGURE=1 ./configure --prefix=/usr --enable-no-install-program=kill,uptime

• enable-no-install-program :: Prevents the installation of binaries which will be installed by other packages.

Compile

FORCE_UNSAFE_CONFIGURE=1 make

Test

For the tests to run properly, we need to link some shared libraries to a place where the tests can find them:

make NON_ROOT_USERNAME=nobody check-root

Some tests must be run as the nobody user, and require the user be part of more than one group:

echo "dummy:x:1000:nobody" >> /etc/group

Change the permissions so the tests can be run by the nobody user:

chown -Rv nobody . 

And now we run the remainder of the tests:

su nobody -s /bin/bash  -c "PATH=$PATH make RUN_EXPENSIVE_TESTS=yes check"

Install

make install
mv -v /usr/bin/{cat,chgrp,chmod,chown,cp,date,dd,df,echo} /bin
mv -v /usr/bin/{false,ln,ls,mkdir,mknod,mv,pwd,rm} /bin
mv -v /usr/bin/{rmdir,stty,sync,true,uname} /bin
mv -v /usr/bin/chroot /usr/sbin
mv -v /usr/share/man/man1/chroot.1 /usr/share/man/man8/chroot.8
sed -i s/\"1\"/\"8\"/1 /usr/share/man/man8/chroot.8
mv -v /usr/bin/{head,sleep,nice,test,[} /bin

Cleanup

Remove the dummy group we created for testing:

sed -i '/dummy/d' /etc/group

Back to top

Installing Diffutils

This package does not require the use of a build directory. Be sure to start with a fresh source directory extracted from the tar file.

Amend the Source

We need to amend the makefiles so that the locale files are installed:

sed -i 's:= @mkdir_p@:= /bin/mkdir -p:' po/Makefile.in.in

Configure

./configure --prefix=/usr

Compile

make

Test

The test-update-copyright failure is expected.

make test

Install

make install

Back to top

Installing gawk

This package does not require the use of a build directory. Be sure to start with a fresh source directory extracted from the tar file.

Configure

./configure --prefix=/usr

Compile

make

Check

make check

Install

make install

Install Documentation

mkdir -v /usr/share/doc/gawk-4.1.3
cp    -v doc/{awkforai.txt,*.{eps,pdf,jpg}} /usr/share/doc/gawk-4.1.3

Back to top

Installing findutils

This package does not require the use of a build directory. Be sure to start with a fresh source directory extracted from the tar file.

Configure

./configure --prefix=/usr --localstatedir=/var/lib/locate

Compile

make

Test

make check

Install

make install
mv -v /usr/bin/find /bin
sed -i 's|find:=${BINDIR}|find:=/bin|' /usr/bin/updatedb

Back to top

Installing groff

Configuration

The gruff binary expects to find a "PAGE" variable in the environment which defines the default page size. This is normally stored in /etc/papersize; we provide a value on the command-line here.

export PAGE=letter
./configure --prefix=/usr

Build

make

Install

make install

Back to top

Installing GRUB

Configure

./configure --prefix=/usr --sbindir=/sbin --sysconfdir=/etc --disable-efiemu --disable-werror

• disable-efiemu :: Disables EFI emulation and testing.

Compile

make

Install

make install

Back to top

Installing less

This package does not require the use of a build directory. Be sure to start with a fresh source directory extracted from the tar file.

Configure

./configure --prefix=/usr --sysconfdir=/etc

Compile

make

Install

make install

Back to top

Installing gzip

This package does not require the use of a build directory. Be sure to start with a fresh source directory extracted from the tar file.

Configure

./configure --prefix=/usr

Compile

make

Test

make check

Install

make install
mv -v /usr/bin/gzip /bin

Back to top

Installing iproute2

This package does not require the use of a build directory.

Amend the Source

We amend the source here to remove a dependency on Berkeley DB, which is not installed.

sed -i /ARPD/d Makefile
sed -i 's/arpd.8//' man/man8/Makefile
rm -v doc/arpd.sgml

This command removes a dependency on iptables:

sed -i 's/m_ipt.o//' tc/Makefile

Configure

There is no separate configure step.

Compile

make

Install

make DOCDIR=/usr/share/doc/iproute2-4.7.0 install

Back to top

Installing kbd

This package does not require the use of a build directory.

Patch and Amend the Source

patch -Np1 -i ../kbd-2.0.3-backspace-1.patch

The following commands ensure the resizecons binary is not built:

sed -i 's/\(RESIZECONS_PROGS=\)yes/\1no/g' configure
sed -i 's/\(RESIZECONS_PROGS=\)yes/\1no/g' configure

Configure

PKG_CONFIG_PATH=/tools/lib/pkgconfig ./configure --prefix=/usr --disable-vlock

• disable-block :: Prevents the vlock utility from being built as it depends on the PAM library, which we have not installed.

Compile

make

Test

make check

Install

make install
mkdir -v       /usr/share/doc/kbd-2.0.3
cp -R -v docs/doc/* /usr/share/doc/kbd-2.0.3

Back to top

Installing libpipeline

This package does not require the use of a build directory.

Configure

PKG_CONFIG_PATH=/tools/lib/pkgconfig ./configure --prefix=/usr

Compile

make

Test

make check

Install

make install

Back to top

Installing make

This package does not require the use of a build directory.

Configure

./configure --prefix=/usr 

Compile

make

Test

make check

Install

make install

Back to top

Installing patch

This package does not require the use of a build directory. Be sure to start with a fresh source directory extracted from the tar file.

Configure

./configure --prefix=/usr 

Compile

make

Test

make check

Install

make install

Back to top

Installing sysklogd

This package does not require the use of a build directory.

Amend the Source

The following commands address problems that cause sysklogd to segment fault, as well as remove a reference to an obsolete data structure:

sed -i '/Error loading kernel symbols/{n;n;d}' ksym_mod.c
sed -i 's/union wait/int/' syslogd.c

Configure

This package does not make use of a configure script.

Compile

make

Install

make BINDIR=/sbin install

Post-install Configuration

Create /etc/syslog.conf with the following contents:

auth,authpriv.* -/var/log/auth.log
*.*;auth,authpriv.none -/var/log/sys.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log
*.emerg *

Back to top

Install sysvinit

This package does not require the use of a build directory.

Patch

patch -Np1 -i ../sysvinit-2.88dsf-consolidated-1.patch

Configure

This program does not make use of a configure script.

Compile

make -C src

Install

make -C src install

Back to top

Install eudev

This package does not require the use of a build directory.

Amend the Source

First, we need to amend one of the test scripts:

sed -r -i 's|/usr(/bin/test)|\1|' test/udev-test.pl

And now we prevent the /tools directory from being hard-coded as a library path into the binary; we do this by appending the following variables to the config.cache file:

cat > config.cache << "EOF"
HAVE_BLKID=1
BLKID_LIBS="-lblkid"
BLKID_CFLAGS="-I/tools/include"
EOF

Configure

./configure --prefix=/usr --bindir=/sbin --sbindir=/sbin --libdir=/usr/lib --sysconfdir=/etc --libexecdir=/lib --with-rootprefix= --with-rootlibdir=/lib --enable-manpages --disable-static --config-cache 

Compile

LIBRARY_PATH=/tools/lib make

Test

First, we need to create some directories required by the testing scripts:

mkdir -pv /lib/udev/rules.d
mkdir -pv /etc/udev/rules.d

And run the tests:

make LD_LIBRARY_PATH=/tools/lib check

Install

make LD_LIBRARY_PATH=/tools/lib install
tar -xvf ../udev-lfs-20140408.tar.bz2
make -f udev-lfs-20140408/Makefile.lfs install

Post-install Configuration

LD_LIBRARY_PATH=/tools/lib udevadm hwdb --update

Back to top

Installing util-linux

This package does not require the use of a build directory.

Configuration

As per FHS, we will use /var/lib/hwclock instead of /etc for our daytime file:

mkdir -pv /var/lib/hwclock

And now configure the package:

./configure ADJTIME_PATH=/var/lib/hwclock/adjtime --docdir=/usr/share/doc/util-linux-2.28.1 --disable-chfn-chsh --disable-login --disable-nologin --disable-su --disable-setpriv --disable-runuser --disable-pylibmount --disable-static --without-python --without-systemd --without-systemdsystemunitdir

Compile

make

Test

We do not run the tests for this package, as they will not run without specific kernel options. As well, some of these tests have been known to damage hardware.

Install

make install

Back to top

Installing man-DB

This package does not require the use of a build directory.

Configure

./configure --prefix=/usr --docdir=/usr/share/doc/man-db-2.7.5 --sysconfdir=/etc --disable-setuid --with-browser=/usr/bin/lynx --with-vgrind=/usr/bin/vgrind --with-grap=/usr/bin/grap

Compile

make

Test

make check

Install

make install

Post-install Configuration:

Remove a reference to a non-existent user:

sed -i "s:man root:root root:g" /usr/lib/tmpfiles.d/man-db.conf

Back to top

Installing tar

This package does not require the use of a build directory.

Configure

We set the FORCE_UNSAFE_CONFIGURE variable here to allow compilation using the root account.

FORCE_UNSAFE_CONFIGURE=1 ./configure --prefix=/usr --bindir=/bin

Compile

make

Test

make check

Install

make install
make -C doc install-html docdir=/usr/share/doc/tar-1.29

Back to top

Installing texinfo

This package does not require the use of a build directory.

Configure

./configure --prefix=/usr --disable-static

Compile

make

Test

make check

Install

make install
make TEXMF=/usr/share/texmf install-tex

Back to top

Installing vim

This package does not require the use of a build directory.

Amend the Source Code

We amend the source code so that the default location of the vimrc file is /etc:

echo '#define SYS_VIMRC_FILE "/etc/vimrc"' >> src/feature.h

Configure

./configure --prefix=/usr

Compile

make

Test

make -j1 test

Install

make install

Post-install Configuration

Here we install the symlinks for the vi binaries and man pages, so when people execute vi, they actually call the 'vim' binary.

ln -sv vim /usr/bin/vi
for L in  /usr/share/man/{,*/}man1/vim.1; do
    ln -sv vim.1 $(dirname $L)/vi.1
done

Here we symlink vim so that it appears versioned (in the interest of consistency):

ln -sv ../vim/vim74/doc /usr/share/doc/vim-7.4

Now we can create the vimrc file:

cat > /etc/vimrc << "EOF"
" Begin /etc/vimrc

set nocompatible
set backspace=2
syntax on
if (&term == "iterm") || (&term == "putty")
  set background=dark
endif

" End /etc/vimrc
EOF

Back to top

Stripping Binaries and Libraries

We can decrease the size of the compiled binaries and libraries by stripping out the debug symbols. This step is optional, but unless you intend to perform debugging, the symbols serve only to consume space.

This is a good time to backup your destination drive yet again ” a single typo here can destroy all of the work performed thus far.

Logout/Login

It is very important to note that none of the binaries we intend to strip are running. To accomplished this, we can exit the chroot jail and re-enter it with the following two commands:

logout
chroot $LFS /tools/bin/env -i HOME=/root TERM=$TERM PS1='\u:\w\$ ' PATH=/bin:/usr/bin:/sbin:/usr/sbin /tools/bin/bash --login

Strip

A large number of 'file format' errors will be reported by these commands. These can be ignored; the files in question are usually script files.

/tools/bin/find /usr/lib -type f -name \*.a \ -exec /tools/bin/strip --strip-debug {} ';'
/tools/bin/find /lib /usr/lib -type f -name \*.so* -exec /tools/bin/strip --strip-unneeded {} ';'
/tools/bin/find /{bin,sbin} /usr/{bin,sbin,libexec} -type f -exec /tools/bin/strip --strip-all {} ';'

Back to top

Section 6

Cleanup

Now we can clean some of the extra files left behind by various testing processes:

rm -rf /tmp/*

Cleanup Unused Libraries

There are also some library files used only for testing we can cleanup:

rm -f /usr/lib/lib{bfd,opcodes}.a
rm -f /usr/lib/libbz2.a
rm -f /usr/lib/lib{com_err,e2p,ext2fs,ss}.a
rm -f /usr/lib/libltdl.a
rm -f /usr/lib/libfl.a
rm -f /usr/lib/libfl_pic.a
rm -f /usr/lib/libz.a

Back to top

Entering the Jail

From this point forward, the jail can be re-entered after it has been exited by issuing the following command:

chroot "$LFS" /usr/bin/env -i              \
    HOME=/root TERM="$TERM" PS1='\u:\w\$ ' \
    PATH=/bin:/usr/bin:/sbin:/usr/sbin     \
    /bin/bash --login

Pseudo and Virtual Filesystems

These must be re-created and mounted each time you re-enter the jail.

Back to top

Installing the Bootscripts

These scripts are used to configure the system at boot and ensure an orderly shutdown.

LFS Bootscripts

We're going to make use of the "Linux From Scratch" bootscripts, which we downloaded along with the rest of our source code.

These install our rc scripts, as well as the scripts needed to bring the system up and down.

Find these in the /sources directory, un-tar them, and install them:

make install

Sysvinit Bootscripts

Once the kernel is loaded, the init program is usually executed. init needs to be given some configuration directives, which we place in /etc/inittab. Create this file now with the following content:

id:3:initdefault:

si::sysinit:/etc/rc.d/init.d/rc S

l0:0:wait:/etc/rc.d/init.d/rc 0
l1:S1:wait:/etc/rc.d/init.d/rc 1
l2:2:wait:/etc/rc.d/init.d/rc 2
l3:3:wait:/etc/rc.d/init.d/rc 3
l4:4:wait:/etc/rc.d/init.d/rc 4
l5:5:wait:/etc/rc.d/init.d/rc 5
l6:6:wait:/etc/rc.d/init.d/rc 6

ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now

su:S016:once:/sbin/sulogin

1:2345:respawn:/sbin/agetty --noclear tty1 9600
2:2345:respawn:/sbin/agetty tty2 9600
3:2345:respawn:/sbin/agetty tty3 9600
4:2345:respawn:/sbin/agetty tty4 9600
5:2345:respawn:/sbin/agetty tty5 9600
6:2345:respawn:/sbin/agetty tty6 9600

Back to top

Installing Network Devices

Network devices are managed using udev, which provides a script that generates the initial rules for the device. These can be generated by running:

bash /lib/udev/init-net-rules.sh

The rules file which creates network devices can be examined if needed:

cat /etc/udev/rules.d/70-persistent-net.rules

Note that you must have the appropriate kernel module loaded (or compiled into the kernel) for your network device to be recognized and used.

Configuring Network Devices

The following content must be placed in /etc/sysconfig/ifconfig.eth0 to configure the network device initially:

cd /etc/sysconfig/
cat > ifconfig.eth0 << "EOF"
ONBOOT=yes
IFACE=eth0
SERVICE=ipv4-static
IP=192.168.1.2
GATEWAY=192.168.1.1
PREFIX=24
BROADCAST=192.168.1.255
EOF

Configuring Domain Name Service (DNS) Resolution

By default, glibc looks in /etc/resolv.conf to determine where to send DNS queries. Create this file with the following contents:

domain localdomain
nameserver 8.8.8.8
nameserver 8.8.4.4

Configuring the Hostname

The hostname is stored in /etc/hostname. The hostname should not be a fully qualified domain name. Create this file with the hostname you prefer.

Configuring /etc/hosts

This file is where we provide IP address to hostname/FQDN mappings. Any number of hosts can be contained in this file, although it is usually better to rely on DNS.

We'll begin this file with the localhost mapping; create /etc/hosts with the following contents:

127.0.0.1 localhost localhost.localdomain

Back to top

Miscellaneous Files

rc.site

The /etc/rc.site file provides settings that are used by the System V scripts. We're going to create this file with the following content:

DISTRO="Linux From Scratch" # The distro name
DISTRO_CONTACT="lfs-dev@linuxfromscratch.org" # Bug report address
DISTRO_MINI="LFS" # Short name used in filenames for distro config

IPROMPT="yes" # Whether to display the interactive boot prompt
itime="3"    # The amount of time (in seconds) to display the prompt

wlen=$(echo "Welcome to ${DISTRO}" | wc -c )
welcome_message="Welcome to ${INFO}${DISTRO}${NORMAL}"

# Set scripts to skip the file system check on reboot
FASTBOOT=no

# Skip reading from the console
HEADLESS=no

# Write out fsck progress if yes
VERBOSE_FSCK=yes

# Speed up boot without waiting for settle in udev
OMIT_UDEV_SETTLE=n

# Speed up boot without waiting for settle in udev_retry
OMIT_UDEV_RETRY_SETTLE=no

# Skip cleaning /tmp if yes
SKIPTMPCLEAN=no

# For setclock
#UTC=1
#CLOCKPARAMS=

# For consolelog (Note that the default, 7=debug, is noisy)
#LOGLEVEL=7

# Delay between TERM and KILL signals at shutdown
KILLDELAY=3

/etc/profile

This file is read by various shells (notably bash) to set environment defaults. We're going to create this file with the following content:

if [ -x /usr/bin/id ]; then
    if [ -z "$EUID" ]; then
        # ksh workaround
        EUID=`id -u`
        UID=`id -ru`
    fi
    USER="`id -un`"
    LOGNAME=$USER
    MAIL="/var/spool/mail/$USER"
fi
HOSTNAME=/usr/bin/hostname 2>/dev/null
HISTSIZE=1000

PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/sbin:/bin

export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE

This file can be amended a number of ways to meet your needs; take a look at https://tiswww.case.edu/php/chet/bash/bashref.html#Modifying-Shell-Behavior for more information.

/etc/inputrc

This file controls keyboard mappings, and the "readline" library (on which bash and other shells rely) uses this file to determine how keyboard input is handled.

We're going to create a somewhat generic file suitable for just about everyone:

# Allow the command prompt to wrap to the next line
set horizontal-scroll-mode Off

# Enable 8bit input
set meta-flag On
set input-meta On

# Turns off 8th bit stripping
set convert-meta Off

# Keep the 8th bit for display
set output-meta On

# none, visible or audible
set bell-style none

# All of the following map the escape sequence of the value
# contained in the 1st argument to the readline specific functions
"\eOd": backward-word
"\eOc": forward-word

# for linux console
"\e[1~": beginning-of-line
"\e[4~": end-of-line
"\e[5~": beginning-of-history
"\e[6~": end-of-history
"\e[3~": delete-char
"\e[2~": quoted-insert

# for xterm
"\eOH": beginning-of-line
"\eOF": end-of-line

# for Konsole
"\e[H": beginning-of-line
"\e[F": end-of-line

/etc/shells

This file contains a list of shells which can be used for the purposes of login. Each shell binary should be on its own line; we'll create this file with the following content:

/bin/sh
/bin/bash

/etc/fstab

We'll create this file with the following entries. Note that your device names may be different than those listed here; be sure you indicate the proper device name to be used during the boot process, or your system won't boot.

For example, we've built our distribution on /dev/sdb for most of this course. When we boot, however, it will be from a virtual machine with a single hard drive ” so the device name will change to sda. Make sure your entries in fstab reflect the device names that will be present at boot.

/dev/sda3      /            ext4     defaults            1     1
/dev/sda4      swap         swap     pri=1               0     0
/dev/sda2      /boot        ext4     defaults            1     1
proc           /proc        proc     nosuid,noexec,nodev 0     0
sysfs          /sys         sysfs    nosuid,noexec,nodev 0     0
devpts         /dev/pts     devpts   gid=5,mode=620      0     0
tmpfs          /run         tmpfs    defaults            0     0
devtmpfs       /dev         devtmpfs mode=0755,nosuid    0     0 

Note that we do not mount /dev/sda1. That partition does not need to be mounted, and in fact, is best left unmounted.

Back to top

Building the Kernel

We downloaded the source for our kernel during stage 1. You may use a newer version of the kernel if it is available, but it is strongly recommended that you use source code from the stable branch.

For this course, we're using version 4.8.12 of the kernel source.

Configuring the Kernel

Kernel configuration is a subject worthy of its own course here at Linux Academy. We cannot cover all of the configuration options in this course but we do cover general configuration. Keep in mind that building the kernel is fairly straightforward once our distro is up and running, so re-building the kernel at a later time is not difficult.

Cleaning the Kernel Source Tree

Change into the kernel source directory and issue:

make mrproper

Entering the Kernel Configuration Utility

A text-based menu-driven configuration program is used to select the kernel options; we enter this system by executing:

make menuconfig

Setting Specific Kernel Options

Be sure to set the following options:

devtmpfs

Device Drivers ”\>
Generic Driver Options ”\>
\* Maintain a devtmpfs filesystem to mount at /dev

DHCP Support

Networking Support ”\>
Networking Options -\>
\<\*\> The IPv6 Protocol        

Building and Installing the Kernel

Once you've configured the kernel, save your configuration. Execute make to build the kernel:

make

Install the kernel modules:

make modules_install

NOTE: The /boot partition must be mounted before the following steps are undertaken. Be sure of this.

Now we need to copy the kernel image to the /boot directory:

cp -v arch/x86/boot/bzImage /boot/vmlinuz-4.8.12

The same goes for the kernel symbols, which are stored in System.map:

cp -v System.map /boot/System.map-4.8.12

It's a good idea, as well, to copy the kernel configuration file itself:

cp -v .config /boot/config-4.8.12

Install the documentation:

install -d /usr/share/doc/linux-4.8.12
cp -r Documentation/* /usr/share/doc/linux-4.8.12

Configuring Linux Modules

Now that we've got the kernel image installed, we need to tell the kernel how and in what order to load modules. For USB devices, specifically, we need to load modules in a specific order.

Create the /etc/modprobe.d directory:

install -v -m755 -d /etc/modprobe.d

And now, create a usb.conf file in that directory with the following contents:

install ohci_hcd /sbin/modprobe ehci_hcd ; /sbin/modprobe -i ohci_hcd ; true
install uhci_hcd /sbin/modprobe ehci_hcd ; /sbin/modprobe -i uhci_hcd ; true

Back to top

Installing GRUB

For this video, we employ GRUB as a bootloader. We strongly encourage you to look at the "Bootloading with GRUB" video on the LinuxAcademy.com website, even if you're familiar with GRUB. It is very easy to misconfigure GRUB and end up with an unbootable system.

Creating the GRUB Configuration File

We'll create this file with the following content. Note that the value of the set root command must be the /boot partition; the root= entry must specify the location of the / partition. Again, see the "Bootloading With GRUB" video if you do not understand what these things mean.

set default=0
set timeout=10 

insmod ext2 biosdisk 
set root=(hd0,2)

menuentry "Linux 4.8.12" {
        linux   /boot/vmlinuz-4.8.12 root=/dev/sda2 ro
}

Adding an Entry to an Existing GRUB Configuration

If you want to add the new distribution to an existing grub configuration, simply add the "menu entry" from the configuration file above. You may have to run grub-update or a similar command to actually update the bootloader.

Back to top

Logout and Reboot

We now want to reboot our system. First, let's logout:

logout

Unmount Virtual Filesystems

umount -v $LFS/dev/pts
umount -v $LFS/dev
umount -v $LFS/run
umount -v $LFS/proc
umount -v $LFS/sys

Umount Our Destination Drive

umount $BROOT/boot
umount $BROOT

Now Reboot

shutdown -rf now

The system should go down for reboot. If necessary, detach the destination drive and attach it to a new VM or re-configure your virtual machine (or hardware) as needed to boot.

Trouble

If your system doesn't boot, it's likely you have issues with GRUB. Remember to double-check your device names. Again, the "Bootloading With GRUB" video can be of great help when troubleshooting problems with the bootloader.

Back to top

Final Thoughts

Updates

Updates to both the kernel and installed packages are a frequent occurrence. You will need to check monthly, if not weekly, for updates to the source. Updating the packages makes use of the same process by which the packages were built (configure, make, make install) most of the time.

Adding Additional Software

Feel free to add additional software as needed. How you do this is entirely up to you ” building from source is one option, but keep in mind that constantly rebuilding software from source to stay abreast of security updates and the like is a time-consuming activity.

Consumers and experts alike have good reason to question the security claims of the VPN industry. Over the last few years, many less reputable VPN companies have abused users' trust by selling their bandwidth, their browsing data, offering poor security or even embedding malware.

Being within the industry, it’s been hard to watch. We knew TunnelBear was doing the right things. We were diligent about security. We deeply respected our users' privacy. While we can’t restore trust in the industry, we realized we could go further in demonstrating to our customers why they can, and should, have trust in TunnelBear.

TunnelBear has completed the consumer VPN industry's first 3rd party, public security audit.

Today, we’d like to announce TunnelBear has completed the Consumer VPN industry's first 3rd party, public security audit. Our auditor, Cure53, has published their findings on their website and we’re content with the results.

In late 2016, we hired Cure53, a respected security company, to do a complete audit of our servers, apps and infrastructure. Using a “white-box” approach, they were given full access to our systems and code. Our original plan was to use their findings internally to confirm we were delivering on our promise to secure your browsing and proactively identify vulnerabilities. However, the recent crisis of trust in the VPN industry showed us we needed to break the silence and share Cure53’s findings publicly. Today we’re sharing a complete public audit which contains both the results from last year and the results from the current audit.

As the auditor, Cure53’s opinions and findings are their own, with the results being published on their website. TunnelBear was given the opportunity to provide feedback on the report, before it was published, where we felt findings were inaccurate or irreproducible. As is the case of most security audits, Cure53 was paid for their work. We wouldn’t expect any cybersecurity company to spend a few hundred hours auditing our code for free.

What were the results?

If you’ve already looked at the results, you’ve seen that the 2016 audit found vulnerabilities in the Chrome extension that we weren’t proud of. It would have been nice to be stronger out of the gate, but this also reinforced our understanding of the value of having regular, independent testing. We want to proactively find vulnerabilities before they can be exploited. We hadn’t intended to publish the 2016 results. However, we’re hoping the security community has appreciation for our candid transparency in the 2016 report and for demonstrating our investment in security over time.

All findings discovered in the 2016 audit were promptly addressed by TunnelBear’s engineering team and verified to be fixed by Cure53.

TunnelBear deserves recognition for implementing a better level of security

In the June 2017 audit, we were more content with the results. All vulnerabilities represented low-risk findings. As Cure53 put it, “The results of the second audit clearly underline that TunnelBear deserves recognition for implementing a better level of security for both the servers and infrastructure as well as the clients and browser extensions for various platforms”.

All findings discovered in the 2017 audit have also been addressed by TunnelBear’s engineering team with only informational findings remaining.

You can read the full report on Cure53’s website.

Our ongoing commitment to security

Our plan is to earn trust and move the VPN industry in a new direction around transparency. While many VPN companies will continue to live in obscurity, with claims of protecting your security, it’s our hope that by completing the industry's first 3rd party, public security audit, experts and consumers alike can be sure that TunnelBear delivers on its security promises.

. . .good security needs constant reevaluation.

If we’ve learned anything from this audit, it’s that good security needs constant reevaluation. Annual public audits will become routine to help us quickly identify vulnerabilities and demonstrate transparency in an industry where trust is sorely lacking. In the coming months we’ll share more announcements, industry insights and how-tos to give you the information you need to make the right choices about your security.

Grizzly Regards

The TunnelBear Team

Press Inquiries
press@tunnelbear.com

There are many trends and cycles of computing that I’ve long observed or participated in as a technologist and investor — from the evolution of SaaS and the advance of cloud computing to the commoditization of infrastructure. Sometimes, however, the trends you watch combine in unexpected, and surprising, ways — and often, that combination is where the real innovation is at…

Take microservices, which are a more modular, independently deployable approach to developing software systems compared to monolithic architectures. Or take the rise of computing at the edge, which moves cloud computing closer to endpoints and users. And then take the rise of the developer and Git ecosystem over the past decade — with it, Github has become far more than a source code management/ version-control tool to a critical social and collaboration network for developers.

Each one of the above trends are important by themselves, and may therefore seem self-evident (and even obvious) to those studying the evolution of computing or watching the space. But when you combine these trends, they can help us move to yet another new and better way of doing things. Enter Netlify, a SaaS platform that enables web developers to build, publish, and maintain modern websites — more suited for our real-time, mobile-centric, distributed world today — through a “decoupled web” that isn’t hog-tied to legacy monolithic infrastructure. It’s ridiculous that the web is still clinging on to monolithic backends — with their high costs, slower speeds, and huge surface area for attacks — in an age of microservices.

By using microservices to help create faster, more secure, and more flexible websites/apps, we can bypass the static server-side approach. Netlify does this by “prebuilding” sites — abstracting to the visitor’s browser, which in turn deals directly with any needed microservices — instead of building them every single time there is a visitor. But the other genius of the approach is that developers can now choose from a growing list of third-party services delivered through APIs on the client side, enabling them to do, well, almost everything all in one place. They can just focus on the frontend and let Netlify do the rest.

Decoupling the frontend and backend in this way — moving the UI and any necessary real-time processing closer to the edge— gives applications the simplicity of the web with the fidelity of a mobile app. Back in the day, Sun Microsystems’ John Gage used to say “the network is where it’s at”; now, I’d argue, the “edge is where it’s at”. To me, it’s part of a broader trend of services at the edge.

Besides the performance advantages, though, Netlify makes web application development and collaboration so much easier and extends developer workflows overall by integrating with Git and Github. But the founders had yet another clever insight here: Using the Github database as a repository for holding content a eliminates the need for databases on websites altogether. Github becomes the de facto content management system or CMS. Take it a step further: when you store information this way, building and deploying applications involves just a few clicks without requiring developers to build everything themselves. The benefits of this cascade all the way from developers and the entire web ecosystem to all kinds of companies and users.

It’s these three things — using microservices and APIs to build more flexible, dynamic websites; deploying to the edge to create a more high-performance end-user experience; and integrating with Github to easily create and manage the application — that give Netlify co-founders Christian Bach and Mathias Christensen their “superpowers”. It’s also what gave me the “aha” moment when they presented to our team one Monday morning several months ago; I went from being merely interested to being blown away! While I’d seen or would have conceived of maybe one of these things, combining them all together and managing them all in one place was sheer genius. Until now, the industry has been suffering from fragmented workflows, difficulty accommodating infrastructure, and lack of a proper ecosystem of services — Netlify is the first such complete platform of its kind. It’s combinatorial innovation in action.

By the end of the meeting, I practically jumped out of my chair. Especially when I learned that the team already had 75,000 developers, growing at about 200 per day. Today, over 100,000 developers use Netlify, growing over 300 per day — running over 300,000 web projects and delivering 2.5 billion web requests a month. All this, on seed funding alone.

There are over 300M websites deployed every year; most of those sites and webapps could be on Netlify. That’s why I’m excited to lead the next stage of investment from Andreessen Horowitz. Netlify is creating a new, necessary, “hybrid” category in the evolution of computing and software as a service: It completely changes the way websites are built by providing both an infrastructure platform and an automation platform for building and deploying the modern open web. It’s literally the next generation, perhaps the holy grail, in web and application design.

Pretend for a moment that you’re walking through your neighborhood and notice a line of people wrapped around the block outside a newly opened restaurant.

Local food bloggers haven’t written about the venue, so you assume the trendy-looking crowd must be the result of contagious, word-of-mouth buzz.

There was a time when that may have been undoubtedly true — when you could trust that a crowd of people was, in fact, a naturally occurring mass of individuals.

But that time may be passing thanks to Surkus, an emerging app that allowed the restaurant to quickly manufacture its ideal crowd and pay the people to stand in place like extras on a movie set. They’ve even been hand-picked by a casting agent of sorts, an algorithmic one that selects each person according to age, location, style and Facebook “likes.”

They may look excited, but that could also be part of the production. Acting disengaged while they idle in line could tarnish their “reputation score,” an identifier that influences whether they’ll be “cast” again. Nobody is forcing the participants to stay, of course, but if they leave, they won’t be paid — their movements are being tracked with geolocation.

Welcome to the new world of “crowdcasting.”

Surkus raises new questions about the future of advertising and promotion. At a time when it has become commonplace for individuals to broadcast polished versions of their lives on social media, does Surkus give businesses a formidable tool to do the same, renting beautiful people and blending them with advertising in a way that makes reality nearly indiscernible? Or have marketers found a new tool that offers them a far more efficient way to link brands with potential customers, allowing individuals to turn themselves into living extensions of the share economy using a structured, mutually beneficial transaction?

The answer depends on whom you ask.

Stephen George, Surkus’s 30-year-old chief executive, said he considers his app an online matchmaker, one that pairs companies with the people who want to hear from them. If successful, he said, Surkus threatens to disrupt the expensive role that promoters and public relations firms have traditionally played in advertising and brand-building.

“So many companies know their core demographic, but they don’t know how to get a hold of those people,” George said.

“They hire promoters and marketers and PR agencies to connect, but it’s a one-sided interaction that involves blasting out a message to get people engaged, but they don’t necessarily know if that message is being received.”

Not everyone, however, is convinced that Surkus — which makes it easier for promoters and marketers to filter crowds according to people’s attractiveness — will improve that reception.

“I understand the need for quick results and attendance and that sometimes brands need people lined up at their door,” said Kerry O’Grady, a professor at New York University’s School of Professional Studies who teaches courses on public relations. 

“Okay, you have a bunch of pretty faces at a party, but what does that do?” O’Grady continued. “It’s not going to do anything if they just want to get paid to party and have no attachment to the brand itself.”

George, a Chicago native, got his start working with Groupon as a sophomore at DePaul University. He went on to make millions from the company’s stock before investing $250,000 in Surkus in 2015. 

The company’s tagline: “Go out. Have fun. Get paid.”

George said the company has amassed 150,000 members in Los Angeles, New York, Chicago, Miami and San Francisco. Anyone can download the app. The members are of all ages and backgrounds, George said, noting that people are drawn by the chance to be social and get paid.

After quietly launching two years ago, Surkus members have attended 4,200 events for 750 clients, including big-name brands, hospitality groups, live-ticketed shows, movie castings and everyday people who want to throw a party. George said users can be paid as little as $5 and as much as $100, though the average for most events is between $25 and $40. Prolific users, he said, can earn as much as $4,000 a year. And Surkus takes a portion of the client’s budget for each event.

The app is supposed to help with these transactions.

Once an event has been scheduled, Surkus’s algorithm sorts through users’ profiles using the client’s desired search criteria. For example: A gaming company throwing a launch party might ask Surkus to find men and women ages 18 to 32 who like comic books, day parties, dance music and the company’s product.

Once potential attendees have been identified from Surkus’s user profiles, the app sends “availability requests” to users’ phones.

Participants are paid within 24 hours via PayPal. During events, participants are asked to remain discreet about the origin of their invitations. Oftentimes, women are paid considerably more than men.

Caroline Thompson, 27, a contributing writer for Vice, said she downloaded Surkus and attended an event last year at a Chicago club full of “finance bros” on a Thursday night.

“It was a little weird that probably 80 percent of the women at the club were there because of the app,” she said.

Thompson said she was paid $40 to attend the event.

O’Grady called the system “scary” and said it raises ethical questions for companies that turn to the app for crowds.

“Good PR is all about transparency,” she said. “But in this case you’re telling people to be discreet, but you’re also telling us the events are organic and that people want to be there, and that’s not okay.”

George rejected the idea that Surkus allows brands to create fake events that manipulate consumers.

“We want to know as much as possible about you, so we can make sure we’re on target with your interests and what you love to do, so that you just can’t say no to an invitation,” he said. 

Entertainment companies aren’t the only ones turning to Surkus for crowdcasting. George said a research company recently used the app to find 750 people to fill movie theaters in Los Angeles and New York to compare how reactions varied from city to city.

In Los Angeles, Surkus has allowed one up-and-coming comedian to fill shows and refine material.

The comedian spoke on the condition that their name and gender not be used because of fear that using the app would tarnish their professional reputation. At many comedy clubs, filling seats is a prerequisite for performing.

The comedian used to be desperate, passing out fliers and asking homeless people to attend so that the performance could go on.

Now, the comedian turns to Surkus.

“Initially, I thought my jokes would appeal to educated white hipsters,” the comedian said. “Now I’ve realized, ‘Oh, wow, my act is appealing to Hispanic women and men in their 50s and old people — like, we’re talking people in their 70s.”

MORE READING: 

Tesla gets richer despite looming production challenge

Billionaire burn: Musk says Zuckerberg’s understanding of AI threat ‘is limited’

How doctors used virtual reality to save the lives of conjoined twin sisters

• Most of the table names were in French
• Database was not relational at all
• A significant amount of columns and tables were not in use anymore
• Data was partially corrupt, inconsistent or even unlinked

• Migrate every client database at same time into Postgres
• Provide consistency turning into a relational database
• Cleanup and repair corrupt, insistence and unlinked data
• Standardize table and column names using English as main language for it
• Minimize the migration time to reduce the time clients were not going to be able to use the product (downtime)
• Design a fail-over process to restore previous version back with the least consequences to final users.

Step1: Migrate database from MySQL to Psql

Beforehand this might look easy, basically we need to migrate design a mapping which converts previous database model to new one, redefining table names, column names, column types, indexes, constraints and adding foreign keys which makes our database relational. Therefore we decided to implement a easy JSON structure which defines every conversion rule, just if something was going to be manipulated during the process it will be written here. One of the examples of that was :

Moreover there are types which doesn’t exist in Postgres as MEDIUMINT, TINYINT.... apart of that there were additional problems coming from a bad database design as the use of TINYINT(1) or even SMALLINT(1) instead of BOOL, and on top of that Postgres use a different syntax for special cases as current_timestamp and so on. Then we defined a bunch of small rules which covered all those cases:

As a result from those rules we ended up with a intermediate JSON file which, finally, contains the whole database mapping, every table and columns and their new types, names, constrains, etc.

Only missing  thing would be to read the current db structure of MySQL databases from the INFORMATION SCHEMA and parse it using the above above JSON.

Migrate and fit data into new database

Derived from previous database conversion, we will need to apply change to data along with changes we applied to the model. That mean we won’t be able to just run a single mysqldump then convert couple of syntax issues and import it into Postgres, instead we will need to process every single row, every single column and value to validate it, convert it and even redefine it as it was the case of dates (because we were moving to UTC).

That was carried in two simple steps:

• Cleanup values: It was a bunch of rules which were checking one by one the former type of each value and its new one and converting them, a few example you have it in here:

• Cleanup rows: Due to the new relational database we would need to remove the rows which became over the time unlinked or inconsistent. For that it was enough with a battery of sql queries which were generated automatically and they were launched at the beginning of each table migration.

Maintaining former database structure

As it was said above, one of the main requirements is to avoid big changes in the code of the main app of Base7, due to it we were forced to find out a way to keep both versions running in parallel with the least effort.

Now is when one of the main reasons to take Postgres as our new database server. Postgres allows the use of Views(include link) supporting INSERT, UPDATE and DELETE meanwhile the relation between views and tables are one to one, ....isn’t it awesome?

Then another issue came up, some table were not renamed but their columns did and we cannot have views and table with the same name under same database, but Postgres also provides you with a solution for it, schemas(include link), schemas allow you to have separate database structure within same actual database, it is like a kind of namespace for database elements.

Here it is an example of how we created those views:

Notice we have at the end every column which are not longer needed, otherwise it won’t work because empty data values cannot be automatically parsed if they are in the middle.

Moreover we also implemented another small tool which can double check in the migrated data was moved without missing any single row, or getting wrong encoded text, or values were still matching. Thanks to this tool we realized of some conversion problems in booleans and encoding issues were the text was migrated in ASCII instead of UTF8.

If you are interested in knowing more about it or you are thinking on migrating your current database from MySQL to Postgres we open sourced the tool here: https://github.com/ggarri/mysql2psql

Step2: Validate and adjust product to use Psql

Once data was moved within Postgres lands, it was time to upgrade our main application to point to the new database in Postgres and see what happens. At first, and as it was expected after updating our config files and use Postgres connector nothing was working, it was time to monitor logs and fix errors.

To get the app loading it was as simple as update an insert sql statement to use “true or false” instead of “1 o 0”, then it was when I realized how tolerant is MySQL in compare with the Super Strict Postgres, at that time I realized it wasn’t going to be that easy as I thought to adjust the code to use Postgres. Here you have few of the issues you might run into if you decide to migrate from MySQL to Postgres:

• Postgres doesn’t support HAVING conditions using ALIASES
• Postgres forces the use of aggregation operation on every column which is not part of the GROUP BY
• Postgres doesn’t support operation between values of different types
• Postgres uses a different syntax, so IFNULL, CONCAT, DATE_ADD....many other are not available anymore

That was going to take time, but how was I going to find every glitch, by hand? one by one? Are we really going to test every single functionality under every single scenario to identify each of the lines of code we would need to modify? The answer was clear, NOT!!!  We need to automatize that, how? Python ! Time to develop another tool.

The tool we decided to implement was going to replay the traffic of every single user for a period of 2-3 days, outcoming same response outputs, including views, and ending with both version of database, mysql and postgres with same status. If we could achieve that we may highly ensure that our app is running decently with Postgres and it won’t cause major issues to our clients.

These were the steps we had to do to get that tool running:

• Save traffic of our clients
• Replay traffic across both app versions
• Compare outputs
• Compare final status of databases

Save traffic of our clients

There are many possible ways to get this, you could?

• Use third party tools as gor (https://github.com/buger/gor),
• Install external plugins to Nginx as HttpGzipModule or using Reverse Proxying
• Create a separate file where we log every request and all the information it will be required to replay the same request.

We took the third option because the other two options were quite intrusive and they might interfere at the normal use of our app because in both cases they were going to be in between our users request and our app, so the less harmest solution was logging the requests. Here we paste the nginx log format we used to after be able to replay our user requests:

```

log_format requestlog '$status $time_iso8601 $cookie_PHPSESSID $cookie_opcode "$request" "$request_body"';

```

• $status: Response status, it will be useful to know if the status remains the same, if one request had failed it has to still fail, if it was redirected it has to still do it.
• $time_iso8601: Time of the response, it will help us to know if a certain request drastically decreased its performance, so we would need to investigate.
•  $cookie_PHPSESSID: User cookie ID, it provides us information about the status of each user session, so we can simulate accurately the status of connection at the time each user does a request.
•  $cookie_opcode: Identifier code of each user, to know which database each request correspond to.
• $request: Path of the request
• $request_body: Params to be send so request is actually the same.

One day we stopped our app traffic for few minutes, setting up the new log_format into a new log file, creating a snapshot of the server which includes: databases backups(one per user) and client php session files, every file under /var/log/php. After two days we repeat the process to get the new backup of databases so we can use them to compare if our replaying process is successful or not.

Replay traffic across both app versions

Using logs obtained at the previous step, we replayed every request, first only replaying traffic from one client and once you get all its request a full match in the outputs then you replay for all of them at same time.

For that you have to keep in mind to bear the each user session in a separate process because requests can belong to different users, even if they all belong to the same client, so we stored each of the COOKIE_ID separately so sessions don’t get mess up, for that you can use CookieJar from the Cookielib of Python, which keep sessions up to date request after request and allows you to reproduce traffic with same flow than originally.

Compare outputs

During this step we faced a lot of issues, it wasn’t only a matter of reading every exception at the logs and going to straight to that and fix it. The main issue was how to obtain exactly the same output for the html views, and how to compare them and highlight only the differences. For that we needed to implement a separated tool based on BeautifulSoup library of Python and extending its normal use to take only differences. This was one of the outputs of that tool:

Compare final status of databases

For this step we reused the tool we implemented before to compare databases to see if everything is still in the same status on both version of the app, MySQL and Postgres. This step made us realized, in between many others, dates weren’t saved with the right time (UTF) even if the server was already running on UTC, at the end it was a project setting options related to the client timezones.

Step3: Speed up migration process

Now it was time to prepare the final launch of Postgres. So far everything went well but we were only testing few client databases and on a narrow scope from what it was going to be the real and final launching.

First of all, we hired a new dedicated server at the same company we have the main one, this new server was going to be become our new live server, becoming at same time the old one the next backup server.

On this new server we replicated the same settings we had in the live server, restoring the snapshots(backup database and php session files) taken during the prior phase. Apart of that we took advantage of the process to install a passwordless environment where every connection would be based on certificates, sockets and whitelisted ips. As last step we imported our implemented migration tools then it was time to run them on a full real environment.

These were the conclusions produced from running our implemented tools on a real environment:

• Migrate process was taking around 3-4 hours to complete.
• Validating if data was exported correctly, was endless.
• Replaying traffic for every client, after 24h running didn’t even reach even 10% of the total.

Derived from those facts we took the decision of parallelism every step which could ran in parallel for each of those processes, about all, we were concerned of the first of them, because we won’t be able to afford a 3-4 hours downtime in our servers, taking in account we were going to need to do some more maintenance tasks after migration was completed, on top of it, move cronjobs, redirect app domains and restart services, all of that should be within an hour. So it was time to work again.

As we just said above, our main challenge was improving the performance of  the migration script to get those 1200 databases into Postgres in the minimum amount of time, definitely it had to be less than 1 hour.

There are two concepts we had to apply here to improve performance:

• Multithreading: It is the ability of a central processing unit (CPU) or a single core to execute multiple processes or threads concurrently using shared memory for it.
• Multiprocessing: It is the use of two or more central processing units (CPUs) within a single computer system without shared memory.

According to the description above, we used Multiprocessing to split databases migrations, each database was going to run in a single and separated CPU. That was clear and easy to implement. Now we have to find out how to take advantage of the Multithreading, these were every step we were running to complete a single database migration, all of them were running sequentially and we needed to parallelism them.

Sequential Version:

Parallel Version:

As a result from previous improvement we were able to import every client database in less than 30 minutes, YUHU !!!! Then we got motivated from such a improvement and we decided to try go deeper in code and identify which were the heaviest computing pieces and work on them to improve it. Thanks to it we pointed few loops to where we were able to reduce the number of iterations and another few parsing methods were running unnecessarily and were consuming a lot of cycles. Moreover we came across a bottleneck at the disk I/O, so we decided to upgrade our server with a SSD.

Here there is an screenshot of the status of our server meanwhile we were running the script:

In conclusion from all previous improvements we reduced the computing time of our migration script from 3 hours to 9 minutes, that is a 6000% faster. We got kind of the same results on the rest of the tools. So now we were ready to the moment of truth.

Step4: The moment of Truth

After around 2 months working on it, we decided to move forward with the Postgres migration. This migration wasn’t going to involve only a new simple release, we were also going to migrate servers because we were setting up new domains, services and dependencies for Postgres in a server, that way we didn’t affect the normal use of our product and clients.

Aiming to have a easy backup plan in case things are not going as expected, we decided to prepare a plan B to restore original server in case migration is not going as expected. That plan B was the following one:

• Proxy traffic from former server to new server, instead of modifying DNS to point to the new server IP, that way we could always re-enable the previous version instantly.
• Log every single request in the replayable mode therefore we have the opportunity to reproduce the traffic onto former version of the product once we restore it.
• Create an snapshot of the server before shutting it down, so we restore it at the same point once we re-enable for original version of the app.

We informed our clients that we were going to have a maintenance downtime in the system for around an hour in 3th of July at 03.00AM(UTC+2) therefore we reduced the impact of normal use of our app to our clients, due to, even if we have clients from everywhere around the world, most of them are located in Europe.

During the day D, we all met around 16.00, we were doing a final testing, mostly manual testing, playing around the tool trying to find glitches derived from the migration, we found a bunch amount of them, luckily they were mostly UX issues, and easy to fix. At the same time we were preparing a set of scripts to run in sequence to cover the whole migration, that way we didn’t leave anything to the chance. The scripts we prepare were:

• Script 1: Cleans every testing data from new server and re-installing every db package from scratch.
• Script 2: Shuts down the live server, running a full back of every client and copying data to new server.
• Script 3: Triggers the migrations scripts, posteriori data validation and summarizing logs in easy readable way for us.
• Script 4: Enabling proxies to redirect traffic from former server to new server.

Time to release came and after a couple hours break we were ready to proceed with the migration, script_1 is launched, script_2 running during a long while....script_3 !!! critical moment just began, those 8 minutes were the slowest ones of my career as developer, few errors came up but not major issues we were going to check it later, then script_4 and time to monitor every application logs: proxies were performing well, traffic was coming into new server normally, a quick manual test on the application, everything was looking good, after checking the import logs we realized we got issues on one client but fortunately it was not active anymore, so nothing to worry. After few more minutes, till an hour couple of exceptions came up, we did couple of hotfixes and releases, but not major issues. WE DID HAVE GO IT!!! We went back home to sleep a bit we had got it.

For sure, nothing can go so well in real live. Three hours after we got into bed our phones started ringing. Application wasn’t accessible,we didn’t know what was happening, server was not reachable, nothing was working To sum up, the issue wasn’t part of our migration, at all, but we got bad luck but that will probably be part of another post.

As of today we are running with Postgres, we got a 30% average speed up in out time responses, we have postgres replication running across two servers what allows us to an alternative access in case of failover. We implemented two new schemas with triggers and procedures written in plv8(include link) whose performance allow us to have historical information about our client changes and many more things on the way running all across Postgres.

I hope you enjoyed the read and have learnt something from it.

Netflix Chief Content Officer Ted Sarandos

Now that Apple has seemingly lost its spark, Netflix has become the most talked-about company in the world – and for good reason.

The streaming innovator is in the middle of disrupting several industries, having completely revolutionized television and now poised to do the same with the film industry. The company’s commitment to innovation has put Netflix head and shoulders above its closest streaming competitors in terms of both subscriber totals and critical acclaim; Netflix dominated other streaming services in this year’s Emmy Award nominations, picking up a total of 91 different nominations, proving that their original series and films are true forces to be reckoned with.

While the internet freaks out about Disney ending its streaming agreement with Netflix, the company continues to forge ahead signing high-profile talent and throwing an enormous budget at its original programming. Just days after the Disney turmoil, Netflix’s visionary Chief Content Officer Ted Sarandos stated that the streaming leader plans to increase its budget by $1 billion dollars over the next year.

In an interview with Variety, Sarandos says that Netflix will spend over $7 billion on content in 2018, up from $6 billion in 2017 and $5 billion in 2016. As of now, Netflix currently has $15.7 billion in outstanding obligations in deals for new series and films over the next few years. With such an astronomically-large budget, media analysts are already beginning to wonder if Netflix is “rescuing” or “ruining” Hollywood by creating such a singular creator-producer-distributor model. Sarandos counters those claims, however, stating that Netflix is merely on the forefront of what’s already a growing trend throughout the media industries:

I would say that the relationship between studios and networks has always been that of a frenemy. Everyone is doing some version of it already. They just have to make a decision for their companies, their brands and their shareholders on how to best optimize the content. We started making original content five years ago, betting this would happen.

Netflix’s forward-thinking, big-spending strategy has been paying off in huge ways. The company’s stock has gone up over 5000% since Netflix first launched streaming video, and global subscribers now total over 100 million. Some analysts might balk at such a large budget and predict that Netflix is in the midst of a valuation bubble, but given how successful their original content has been for them over the last few years, Sarandos and company likely know exactly what they’re doing.

You’re walking at $HOME, minding your own business.

$ whoami> user

$ pwd
> /home/user

But something is bothering your feet. It’s like if a little rock has fallen into your shoe. You take it off, to see what’s going on.

$ ls -lah ./left-shoe
---------- 1 root root 4 May 30 13:20 little-rock

That’s odd. It’s there, but it doesn’t seem to be yours. It’s left there by root, the Rock Tamer, and only he can decide its fate.

# bash -c "echo 'You stay here' > /home/user/left-shoe/little-rock"
# chmod 0000 /home/user/left-shoe/little-rock

You reach into your pocket for your phone, to speed dial him with sudo. Suddenly, you feel powerful (from watching Gladiator last night), and decide to put back the phone, and try your luck.

$ rm -f ./left-shoe/little-rock
$ ls -lah ./left-shoe/little-rock
ls: cannot access little-rock: No such file or directory

You look down at your shaking hands, trying to figure out if this is the real world. It is. You did it. Without the Rock Tamer. But how?

The little rock in your shoe had absolutely no idea what’s coming. As seen from it’s incarnation, nobody had any permissions on it (--- --- ---). No reads, no writes, no throwing by anyone (owner, group, others).

The catch

What happened is, is that the Rock Tamer forgot that you are even more powerful than him, when you’re at $HOME. Let’s see why.

To be able to do anything with a file, the first step is to look it up in its directory. Listing a directory’s contents is controlled by the execute flag. If a user has execute permissions on a directory, he can see what’s inside it. Also, the execute flag on the directory gives access to its files’ inodes, which is crucial in this context, as the removal process unlinks the file.

Next, the removing part. Renaming or removing a file doesn’t involve the write() system call. Practically, we don’t need any permissions to remove the file, nor do we care about its owner. The only requirement is to have write permissions on the parent directory (and the execute flag on the parent directory).

The $HOME directory naturally fulfills both of these requirements from the user’s perspective.

The contra-catch

If the Rock Tamer, really didn’t want anyone to mess around with his rocks, he would’ve done:

# chattr +i /home/user/left-shoe/little-rock

This operation makes the file immutable, which among other things, prevents its removal. Excerpt from the man page:

A file with the 'i' attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file and no data can be written to the file. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.

Moonwalks away.

Cryptocurrencies and tokens are becoming increasingly popular. In this post, we’ll explore the context that triggered this frenzy, the new possibilities tokens unlock, and why people are excited about its future.

The Context

Decades after the web started and seventeen years after the dot com crash, we have finally entered the golden age of Internet Apps. Companies like Airbnb or Uber can be described as centralized entities governing a network that allocates a scarce resource i.e homes and rides. These communities are built on strong network effects – where, according to Metcalfe’s law, the value of the network is exponentially increased by the number of users who participate. In these cases, the accrued value produced by users is captured by the companies and their shareholders.

On the other hand, open source and decentralized projects that power the internet like TCP/IP, Wikipedia or Unix have generated huge amounts of value but historically struggle to capture most of it. Success in open source is measured by user adoption, not by revenue. For example, internet protocols like TCP/IP or Email (SMTP) have been massively underfunded despite their use by millions of people.

Companies heavily use these protocols while accruing the value in their own user facing layer, the end consumer application. The downside is that the lower protocols remain massively underfunded.

Bitcoin created the first “fair” peer-to-peer decentralized network. Bitcoin allows money to be transacted securely between parties without any central authority. Everyone verifies the transactions. Fairness is defined by rewarding good actors monetarily for contributing value to the network. In this case, actors are called miners. They contribute to the network by recording and securing all the transactions with cryptographic proofs.

The Spark

Ethereum took the Bitcoin concept one step further. It is a decentralized network of millions of computers that execute code at the same time. Efficiency is compromised in favour of security. When someone wants to send a transaction, all the nodes execute it and update the ledger. Unlike Bitcoin, where the capabilities are really restricted for security purposes, these Ethereum nodes can execute almost any program. Its code is defined in smart contracts. A smart contract is a set of promises, specified in digital form, including protocols within which the parties perform on these promises. One of the direct applications is the digitalization of complex financial instruments like derivatives.

More importantly, you can use a smart contract to build your own crypto-currency or token. Tokens allow decentralized protocols to capture the value of the network. A token is the basic economic unit in the ecosystem and should represent a scarce resource. Tokens are spent to use this resource and are earned by contributing to the network. For example, in Filecoin you earn tokens by lending your unused hard drive space. In summary, tokens coordinate efforts in the network and motivate responsible participation.

Excitement & Gold Rush

At the moment, an Initial Coin Offering or ICO is the main use case of Ethereum. An ICO is for a network what an IPO is for an startup, with some major differences. These ICOs provide a sizable improvement in terms of liquidity. Unlike traditional companies, you can represent the value of your assets in terms of tokens that are liquid and tradeable from day one. Besides, you can attract thousands of early adopters with skin in the game that believe and support your project from the beginning. The capital injection and initial speculation bring attention to developers, effectively bootstrapping the network.

Unfortunately, many startups that aren’t networks are are turning to ICOs to fundraise just to get quicker access to money at higher valuations. Only a few of these actually require a token to bootstrap and track the value of the network. Protocol tokens like Ethereum are examples of currencies with strong network effects. Their model simply wouldn’t work without this ‘token effect’. The token, and its ICO, is the only way the protocol can create and retain value from its utilization.

Looking ahead

The web took decades to mature. It went through periods of high speculation and subsequent crashes. It’s important to remember that even during those periods, great companies like Amazon or Google were founded. It is only reasonable to expect the Blockchain ecosystem to follow a similar timeline.

Cryptocurrencies and Tokens raised a record $1.27 billion in the first half of 2017 through Initial Coin Offerings. Most of them won’t survive but a chosen few will, laying down the foundation of a new age.

“Blockchains give us new ways to govern networks. For banking. For voting. For search. For social media. For phone and energy grids. Blockchain-based market networks will replace existing networks. Slowly, then suddenly. In one thing, then in many things.” – Naval Ravikant

Thanks to Cadran Cowansage, Stephanie Simon, Craig Cannon, and my friends outside YC Mariano Torrecilla and Raul San Narciso for reading the early drafts.

Notes
https://en.wikipedia.org/wiki/Metcalfe%27s_law↩
See Fat Protocols http://www.usv.com/blog/fat-protocols↩
Bitcoin Paper by Satoshi Nakamoto https://bitcoin.org/bitcoin.pdf↩
Nick Szabo Smart Contracts: Building Blocks for Digital Markets↩
ERC20 Token https://theethereum.wiki/w/index.php/ERC20_Token_Standard↩
CoinDesk ICO Tracker https://www.coindesk.com/ico-tracker/↩

If you are like most merchants you have at least considered using Google AMP on your website. If you are not yet familiar with AMP, it stands for Accelerated Mobile Pages. It is a Google service / project that allows you to create a mobile version of your site that is designed  to be quicker than a regular responsive mobile website. It does that by using a custom library of Google AMP JavaScripts and elements on the page. Sounds great, right? Not really from my point of view.

Google and proponents of AMP would like you to think it is important because it is simply a faster way to deliver web pages. A lot of the thought behind AMP is that it will improve usability by delivering pages faster, which will keep users on your site longer. But is this actually a fact? AMP itself does not help with engaging users in a page. The content is what engages users.

AMP is very limiting

To be shown as an AMP page in Google Mobile Search your AMP page has to perfectly validate. The tags for AMP and what the AMP project supports are very, very limiting. Things that you could see as detrimental to your website are generally not allowed on AMP websites. Some of those things are:

Chat Applications

A lot of sites use chat applications for customer service. With AMP these cannot be used, this could cause your site to lose potential sales from users not being able to get the help they need.

Payment Methods

AMP does not have any markup specific to checkouts, so users are redirected to your normal checkout. But on product page checkouts cannot be used. Checkouts such as Paypal Express, Amazon Payments, and Apple Pay, when used on a product page are disabled on the AMP version of the product pages.

Logging In

AMP does not allow for use of forms, so a user cannot log in on an AMP site. They are sent to your mobily responsive site, never to return to your AMP site again. At the same time AMP does not support social media logging in, which might be a deterrent to some of your users as well. One thing that is not apparent is that AMP pages are static for the most part. They really do not support a logged in state, or user preferences. Things like recommended products, or recently viewed products will not work with an AMP page. None of the personalization aspects like “Hi, Lesley” are done with AMP.

Filtering and Faceted Searching

This currently cannot be done with AMP either. The limited javascript that AMP works with does not allow these kinds of operations. SO if search and filtering are a large part of your site’s mobile navigation, AMP will be useless.

Analytics are limited

Google Analytics is not supported on AMP. A limited set of Analytics for AMP is available. It does not have as many tracking features or as much information as the standard Google Analytics. If you use a different suite of tracking such as Piwik or kissmetrics, they will not work with AMP.

Users Stay on Google

One feature of AMP is the caching. Google caches your pages and serves them from their cache, which is where a lot of the loading speed comes from. This means users never reach your site. They are on Google’s site still. A typical AMP url will look like the image below. Notice the URL in the browser, it is still on Google’s site.

Ad Revenue is Decreased

This is not generally a big issue for e-commerce sites, but it is something that is well-documented and worth mentioning.

Integrations are not Possible

Most sites run at least some third party integration. These just are not possible with AMP because of the way it caches the pages on Google. That means some of the tools you use to increase conversions or increase the order amount will not be used with AMP. Some of the common integrations we see that are not able to be used are:

• Yotpo or other review platforms, so reviews are not shown
• Trust seals, like SSL seals or payment gateway seals
• Product recommendation engines like Nosto or Segmentify do not work
• Search services like Algolia and others will not work
• Financing services like Time Payment do not work on AMP
• A/B testing is not supported

If your mobile site relies on any of these services or features, AMP will remove them all. Its starting to become clear how limited AMP is. Honestly, to me, it is a one trick pony. Is speed all it takes to close sales? I for one do not think so.

What About the Ranking Boost

If you have been told there is a ranking boost by using Google AMP, you have been lied to. The world of SEO sometimes seems like a mystical world, sometimes it is hard to understand how one thing will affect another that might affect rankings. But there are some clear things. When the people at Google talk, people in the SEO industry listen. Here is an official tweet on the matter that should put AMP as a ranking signal to rest.

As much as some developers and module sellers want you to think that AMP is a ranking factor, you have read from the horses mouth that it is not.

But the speed

AMP is geared around speed. It strips everything that can slow a page loading time out to make it faster, features in some cases are stripped. A lot of the speed actually comes from loading the site through Google’s CDN network. Because of how AMP is designed, it will detect a the browser and refer to the desktop version in most tests. This makes actually testing an AMP cached site for speed very difficult. After a lot testing around I was actually able to find a way to test AMP pages versus mobile sites for their speed. Using GTmetrix from the Vancouver testing location you can test with a mobile browser. Google does not redirect the page, so you are able to test the AMP speed directly.

The first series of tests are ran on ebay, since they are a well known early adopter of AMP. For the tests the I used the Vancouver location and the Chrome Android Galaxy Nexus settings for both tests. I tested them both unthrottled as well. Exact same tests the AMP cached page and the regular Ebay mobile page. One thing to note, you do need a free GTmetrix account to be able to change the testing to these settings.

You can see in the first test that ebay without AMP loads almost 2 seconds faster than with AMP. The page size for the AMP site is almost twice as big as well. Another thing to note, is that with AMP the scores are better for PageSpeed and YSlow. Does better scores negate the almost 2 seconds extra load time?

These results have to be an outlier, right? Why would people use AMP if it actually slowed things down? I felt the same way, so I tested some more site. These next tests are done from OverStock’s site on a leather recliner landing page.

Looking at the two different examples above does AMP seem worth it to you? Especially when you consider that some elements you might use for conversion factors will have to be disabled. I cannot say I would recommend using AMP by looking at these tests.

Conclusion

Taking everything into consideration, I would decline recommending AMP for e-commerce sites. AMP itself is too limited, even if the speed increases that are promised are there, I don’t think I could recommend it because of the limitations. Almost everything that is limited on AMP has an actual conversion value. So you would have to weight the speed against the value of the conversion addons. Would your site loading 1 second faster be worth turning off a one click buying option for mobile users? Which would result in more conversions? Would having category pages that cannot be filtered be worth having a speed increase? I just don’t think the benefits outweigh the disadvantages in using AMP for an e-commerce site.

Have a different opinion? Let me know below in the comments.

A dive into the mechanics that allow Postgres to provide strong atomic guarantees despite the chaotic entropy of production.

typedef struct PGXACT
{
    TransactionId xid;   /* id of top-level transaction currently being
                          * executed by this proc, if running and XID
                          * is assigned; else InvalidTransactionId */

    TransactionId xmin;  /* minimal running XID as it was when we were
                          * starting our xact, excluding LAZY VACUUM:
                          * vacuum must not remove tuples deleted by
                          * xid >= xmin ! */

    ...
} PGXACT;

typedef struct HeapTupleData
{
    uint32          t_len;         /* length of *t_data */
    ItemPointerData t_self;        /* SelfItemPointer */
    Oid             t_tableOid;    /* table the tuple came from */
    HeapTupleHeader t_data;        /* -> tuple header and data */
} HeapTupleData;

/* referenced by HeapTupleData */
struct HeapTupleHeaderData
{
    HeapTupleFields t_heap;

    ...
}

/* referenced by HeapTupleHeaderData */
typedef struct HeapTupleFields
{
    TransactionId t_xmin;        /* inserting xact ID */
    TransactionId t_xmax;        /* deleting or locking xact ID */

    ...
} HeapTupleFields;

# SELECT *, xmin, xmax FROM names;

 id |   name   | xmin  | xmax
----+----------+-------+-------
  1 | Hyperion | 27926 | 27928
  2 | Endymion | 27927 |     0

typedef struct SnapshotData
{
    /*
     * The remaining fields are used only for MVCC snapshots, and are normally
     * just zeroes in special snapshots.  (But xmin and xmax are used
     * specially by HeapTupleSatisfiesDirty.)
     *
     * An MVCC snapshot can never see the effects of XIDs >= xmax. It can see
     * the effects of all older XIDs except those listed in the snapshot. xmin
     * is stored as an optimization to avoid needing to search the XID arrays
     * for most tuples.
     */
    TransactionId xmin;            /* all XID < xmin are visible to me */
    TransactionId xmax;            /* all XID >= xmax are invisible to me */

    /*
     * For normal MVCC snapshot this contains the all xact IDs that are in
     * progress, unless the snapshot was taken during recovery in which case
     * it's empty. For historic MVCC snapshots, the meaning is inverted, i.e.
     * it contains *committed* transactions between xmin and xmax.
     *
     * note: all ids in xip[] satisfy xmin <= xip[i] < xmax
     */
    TransactionId *xip;
    uint32        xcnt; /* # of xact ids in xip[] */

    ...
}

static void
exec_simple_query(const char *query_string)
{
    ...

    /*
     * Set up a snapshot if parse analysis/planning will need one.
     */
    if (analyze_requires_snapshot(parsetree))
    {
        PushActiveSnapshot(GetTransactionSnapshot());
        snapshot_set = true;
    }

    ...
}

Snapshot
GetSnapshotData(Snapshot snapshot)
{
    /* xmax is always latestCompletedXid + 1 */
    xmax = ShmemVariableCache->latestCompletedXid;
    Assert(TransactionIdIsNormal(xmax));
    TransactionIdAdvance(xmax);

    ...

    snapshot->xmax = xmax;
}

typedef uint32 TransactionId;

#define InvalidTransactionId        ((TransactionId) 0)
#define BootstrapTransactionId      ((TransactionId) 1)
#define FrozenTransactionId         ((TransactionId) 2)
#define FirstNormalTransactionId    ((TransactionId) 3)

...

/* advance a transaction ID variable, handling wraparound correctly */
#define TransactionIdAdvance(dest)    \
    do { \
        (dest)++; \
        if ((dest) < FirstNormalTransactionId) \
            (dest) = FirstNormalTransactionId; \
    } while(0)

/*
 * Spin over procArray checking xid, xmin, and subxids.  The goal is
 * to gather all active xids, find the lowest xmin, and try to record
 * subxids.
 */
for (index = 0; index < numProcs; index++)
{
    volatile PGXACT *pgxact = &allPgXact[pgprocno];
    TransactionId xid;
    xid = pgxact->xmin; /* fetch just once */

    /*
     * If the transaction has no XID assigned, we can skip it; it
     * won't have sub-XIDs either.  If the XID is >= xmax, we can also
     * skip it; such transactions will be treated as running anyway
     * (and any sub-XIDs will also be >= xmax).
     */
    if (!TransactionIdIsNormal(xid)
        || !NormalTransactionIdPrecedes(xid, xmax))
        continue;

    if (NormalTransactionIdPrecedes(xid, xmin))
        xmin = xid;

    /* Add XID to snapshot. */
    snapshot->xip[count++] = xid;

    ...
}

...

snapshot->xmin = xmin;

static void
CommitTransaction(void)
{
    ...

    /*
     * We need to mark our XIDs as committed in pg_xact.  This is where we
     * durably commit.
     */
    latestXid = RecordTransactionCommit();

    /*
     * Let others know about no transaction in progress by me. Note that this
     * must be done _before_ releasing locks we hold and _after_
     * RecordTransactionCommit.
     */
    ProcArrayEndTransaction(MyProc, latestXid);

    ...
}

static TransactionId
RecordTransactionCommit(void)
{
    bool markXidCommitted = TransactionIdIsValid(xid);

    /*
     * If we haven't been assigned an XID yet, we neither can, nor do we want
     * to write a COMMIT record.
     */
    if (!markXidCommitted)
    {
        ...
    } else {
        XactLogCommitRecord(xactStopTimestamp,
                            nchildren, children, nrels, rels,
                            nmsgs, invalMessages,
                            RelcacheInitFileInval, forceSyncCommit,
                            MyXactFlags,
                            InvalidTransactionId /* plain commit */ );

        ....
    }

    if ((wrote_xlog && markXidCommitted &&
         synchronous_commit > SYNCHRONOUS_COMMIT_OFF) ||
        forceSyncCommit || nrels > 0)
    {
        XLogFlush(XactLastRecEnd);

        /*
         * Now we may update the CLOG, if we wrote a COMMIT record above
         */
        if (markXidCommitted)
            TransactionIdCommitTree(xid, nchildren, children);
    }

    ...
}

#define TRANSACTION_STATUS_IN_PROGRESS      0x00
#define TRANSACTION_STATUS_COMMITTED        0x01
#define TRANSACTION_STATUS_ABORTED          0x02
#define TRANSACTION_STATUS_SUB_COMMITTED    0x03

#define CLOG_BITS_PER_XACT  2
#define CLOG_XACTS_PER_BYTE 4
#define CLOG_XACTS_PER_PAGE (BLCKSZ * CLOG_XACTS_PER_BYTE)

/*
 * Record the final state of transaction entries in the commit log for
 * all entries on a single page.  Atomic only on this page.
 *
 * Otherwise API is same as TransactionIdSetTreeStatus()
 */
static void
TransactionIdSetPageStatus(TransactionId xid, int nsubxids,
                           TransactionId *subxids, XidStatus status,
                           XLogRecPtr lsn, int pageno)
{
    ...

    LWLockAcquire(CLogControlLock, LW_EXCLUSIVE);

    /*
     * Set the main transaction id, if any.
     *
     * If we update more than one xid on this page while it is being written
     * out, we might find that some of the bits go to disk and others don't.
     * If we are updating commits on the page with the top-level xid that
     * could break atomicity, so we subcommit the subxids first before we mark
     * the top-level commit.
     */
    if (TransactionIdIsValid(xid))
    {
        /* Subtransactions first, if needed ... */
        if (status == TRANSACTION_STATUS_COMMITTED)
        {
            for (i = 0; i < nsubxids; i++)
            {
                Assert(ClogCtl->shared->page_number[slotno] == TransactionIdToPage(subxids[i]));
                TransactionIdSetStatusBit(subxids[i],
                                          TRANSACTION_STATUS_SUB_COMMITTED,
                                          lsn, slotno);
            }
        }

        /* ... then the main transaction */
        TransactionIdSetStatusBit(xid, status, lsn, slotno);
    }

    ...

    LWLockRelease(CLogControlLock);
}

void
ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
{
    /*
     * We must lock ProcArrayLock while clearing our advertised XID, so
     * that we do not exit the set of "running" transactions while someone
     * else is taking a snapshot.  See discussion in
     * src/backend/access/transam/README.
     */
    if (LWLockConditionalAcquire(ProcArrayLock, LW_EXCLUSIVE))
    {
        ProcArrayEndTransactionInternal(proc, pgxact, latestXid);
        LWLockRelease(ProcArrayLock);
    }

    ...
}

static inline void
ProcArrayEndTransactionInternal(PGPROC *proc, PGXACT *pgxact,
                                TransactionId latestXid)
{
    ...

    /* Also advance global latestCompletedXid while holding the lock */
    if (TransactionIdPrecedes(ShmemVariableCache->latestCompletedXid,
                              latestXid))
        ShmemVariableCache->latestCompletedXid = latestXid;
}

static void
heapgettup(HeapScanDesc scan,
           ScanDirection dir,
           int nkeys,
           ScanKey key)
{
    ...

    /*
     * advance the scan until we find a qualifying tuple or run out of stuff
     * to scan
     */
    lpp = PageGetItemId(dp, lineoff);
    for (;;)
    {
        /*
         * if current tuple qualifies, return it.
         */
        valid = HeapTupleSatisfiesVisibility(tuple,
                                             snapshot,
                                             scan->rs_cbuf);

        if (valid)
        {
            return;
        }

        ++lpp;            /* move forward in this page's ItemId array */
        ++lineoff;
    }

    ...
}

bool
HeapTupleSatisfiesMVCC(HeapTuple htup, Snapshot snapshot,
                       Buffer buffer)
{
    ...

    else if (TransactionIdDidCommit(HeapTupleHeaderGetRawXmin(tuple)))
        SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED,
                    HeapTupleHeaderGetRawXmin(tuple));

    ...

    /* xmax transaction committed */

    return false;
}

bool /* true if given transaction committed */
TransactionIdDidCommit(TransactionId transactionId)
{
    XidStatus xidstatus;

    xidstatus = TransactionLogFetch(transactionId);

    /*
     * If it's marked committed, it's committed.
     */
    if (xidstatus == TRANSACTION_STATUS_COMMITTED)
        return true;

    ...
}

SetHintBits(tuple, buffer, HEAP_XMIN_COMMITTED,
            HeapTupleHeaderGetRawXmin(tuple));

BEGIN;

SELECT * FROM users 

INSERT INTO users (email) VALUES ('brandur@example.com')
RETURNING *;

COMMIT;

Over the last year, GitHub has gradually evolved the infrastructure that runs the Ruby on Rails application responsible for github.com and api.github.com. We reached a big milestone recently: all web and API requests are served by containers running in Kubernetes clusters deployed on our metal cloud. Moving a critical application to Kubernetes was a fun challenge, and we’re excited to share some of what we’ve learned with you today.

Why change?

Before this move, our main Ruby on Rails application (we call it github/github) was configured a lot like it was eight years ago: Unicorn processes managed by a Ruby process manager called God running on Puppet-managed servers. Similarly, our chatops deployment worked a lot like it did when it was first introduced: Capistrano established SSH connections to each frontend server, then updated the code in place and restarted application processes. When peak request load exceeded available frontend CPU capacity, GitHub Site Reliability Engineers would provision additional capacity and add it to the pool of active frontend servers.

While our basic production approach didn’t change much in those years, GitHub itself changed a lot: new features, larger software communities, more GitHubbers on staff, and way more requests per second. As we grew, this approach began to exhibit new problems. Many teams wanted to extract the functionality they were responsible for from this large application into a smaller service that could run and be deployed independently. As the number of services we ran increased, the SRE team began supporting similar configurations for dozens of other applications, increasing the percentage of our time we spent on server maintenance, provisioning, and other work not directly related to improving the overall GitHub experience. New services took days, weeks, or months to deploy depending on their complexity and the SRE team’s availability. Over time, it became clear that this approach did not provide our engineers the flexibility they needed to continue building a world-class service. Our engineers needed a self-service platform they could use to experiment, deploy, and scale new services. We also needed that same platform to fit the needs of our core Ruby on Rails application so that engineers and/or robots could respond to changes in demand by allocating additional compute resources in seconds instead of hours, days, or longer.

In response to those needs, the SRE, Platform, and Developer Experience teams began a joint project that led us from an initial evaluation of container orchestration platforms to where we are today: deploying the code that powers github.com and api.github.com to Kubernetes clusters dozens of times per day. This post aims to provide a high-level overview of the work involved in that journey.

Why Kubernetes?

As a part of evaluating the existing landscape of “platform as a service” tools, we took a closer look at Kubernetes, a project from Google that described itself at the time as an open-source system for automating deployment, scaling, and management of containerized applications. Several qualities of Kubernetes stood out from the other platforms we evaluated: the vibrant open source community supporting the project, the first run experience (which allowed us to deploy a small cluster and an application in the first few hours of our initial experiment), and a wealth of information available about the experience that motivated its design.

These experiments quickly grew in scope: a small project was assembled to build a Kubernetes cluster and deployment tooling in support of an upcoming hack week to gain some practical experience with the platform. Our experience with this project as well as the feedback from engineers who used it was overwhelmingly positive. It was time to expand our experiments, so we started planning a larger rollout.

Why start with github/github?

At the earliest stages of this project, we made a deliberate decision to target the migration of a critical workload: github/github. Many factors contributed to this decision, but a few stood out:

• We knew that the deep knowledge of this application throughout GitHub would be useful during the process of migration.
• We needed self-service capacity expansion tooling to handle continued growth.
• We wanted to make sure the habits and patterns we developed were suitable for large applications as well as smaller services.
• We wanted to better insulate the app from differences between development, staging, production, enterprise, and other environments.
• We knew that migrating a critical, high-visibility workload would encourage further Kubernetes adoption at GitHub.

Given the critical nature of the workload we chose to migrate, we needed to build a high level of operational confidence before serving any production traffic.

Rapid iteration and confidence building with a review lab

As a part of this migration, we designed, prototyped, and validated a replacement for the service currently provided by our frontend servers using Kubernetes primitives like Pods, Deployments, and Services. Some validation of this new design could be performed by running github/github’s existing test suites in a container rather than on a server configured similarly to frontend servers, but we also needed to observe how this container behaved as a part of a larger set of Kubernetes resources. It quickly became clear that an environment that supported exploratory testing of the combination of Kubernetes and the services we intended to run would be necessary during the validation phase.

Around the same time, we observed that our existing patterns for exploratory testing of github/github pull requests had begun to show signs of growing pains. As the rate of deploys increased along with the number of engineers working on the project, so did the utilization of the several additional deploy environments used as a part of the process of validating a pull request to github/github. The small number of fully-featured deploy environments were usually booked solid during peak working hours, which slowed the process of deploying a pull request. Engineers frequently requested the ability to test more of the various production subsystems on “branch lab.” While branch lab allowed concurrent deployment from many engineers, it only started a single Unicorn process for each, which meant it was only useful when testing API and UI changes. These needs overlapped substantially enough for us to combine the projects and start work on a new Kubernetes-powered deployment environment for github/github called “review lab.”

In the process of building review lab, we shipped a handful of sub-projects, each of which could likely be covered in their own blog post. Along the way, we shipped:

• A Kubernetes cluster running in an AWS VPC managed using a combination of Terraform and kops.
• A set of Bash integration tests that exercise ephemeral Kubernetes clusters, used heavily in the beginning of the project to gain confidence in Kubernetes.
• A Dockerfile for github/github.
• Enhancements to our internal CI platform to support building and publishing containers to a container registry.
• YAML representations of 50+ Kubernetes resources, checked into github/github.
• Enhancements to our internal deployment application to support deploying Kubernetes resources from a repository into a Kubernetes namespace, as well as the creation of Kubernetes secrets from our internal secret store.
• A service that combines haproxy and consul-template to route traffic from Unicorn pods to the existing services that publish service information there.
• A service that reads Kubernetes events and sends abnormal ones to our internal error tracking system.
• A chatops-rpc-compatible service called kube-me that exposes a limited set of kubectl commands to users via chat.

The end result is a chat-based interface for creating an isolated deployment of GitHub for any pull request. Once a pull request passed all required CI jobs, a user can deploy their pull request to review lab like so:

jnewland

.deploy https://github.com/github/github/pull/4815162342 to review-lab

Like branch lab before it, labs are cleaned up one day after their last deploy. As each lab is created in its own Kubernetes namespace, cleanup is as simple as deleting the namespace, which our deployment system performs automatically when necessary.

Review lab was a successful project with a number of positive outcomes. Before making this environment generally available to engineers, it served as an essential proving ground and prototyping environment for our Kubernetes cluster design as well as the design and configuration of the Kubernetes resources that now describe the github/github Unicorn workload. After release, it exposed a large number of engineers to a new style of deployment, helping us build confidence via feedback from interested engineers as well as continued use from engineers who didn’t notice any change. And just recently, we observed some engineers on our High Availability team use review lab to experiment with the interaction between Unicorn and the behavior of a new experimental subsystem by deploying it to a shared lab. We’re extremely pleased with the way that this environment empowers engineers to experiment and solve problems in a self-service manner.

With review lab shipped, our attention shifted to github.com. To satisfy the performance and reliability requirements of our flagship service - which depends on low-latency access to other data services - we needed to build out Kubernetes infrastructure that supported the metal cloud we run in our physical data centers and POPs. Again, nearly a dozen subprojects were involved in this effort:

• A timely and thorough post about container networking helped us select the Calico network provider, which provided the out-of-the box functionality we needed to ship a cluster quickly in ipip mode while giving us the flexibility to explore peering with our network infrastructure later.
• Following no less than a dozen reads of @kelseyhightower’s indispensable Kubernetes the hard way, we assembled a handful of manually provisioned servers into a temporary Kubernetes cluster that passed the same set of integration tests we used to exercise our AWS clusters.
• We built a small tool to generate the CA and configuration necessary for each cluster in a format that could be consumed by our internal Puppet and secret systems.
• We Puppetized the configuration of two instance roles - Kubernetes nodes and Kubernetes apiservers - in a fashion that allows a user to provide the name of an already-configured cluster to join at provision time.
• We built a small Go service to consume container logs, append metadata in key/value format to each line, and send them to the hosts’ local syslog endpoint.
• We enhanced GLB, our internal load balancing service, to support Kubernetes NodePort Services.

The combination of all of this hard work resulted in a cluster that passed our internal acceptance tests. Given that, we were fairly confident that the same set of inputs (the Kubernetes resources in use by review lab), the same set of data (the network services review lab connected to over a VPN), and same tools would create a similar result. In less than a week’s time - much of which was spent on internal communication and sequencing in the event the migration had significant impact - we were able to migrate this entire workload from a Kubernetes cluster running on AWS to one running inside one of our data centers.

Raising the confidence bar

With a successful and repeatable pattern for assembling Kubernetes clusters on our metal cloud, it was time to build confidence in the ability of our Unicorn deployment to replace the pool of current frontend servers. At GitHub, it is common practice for engineers and their teams to validate new functionality by creating a Flipper feature and then opting into it as soon as it is viable to do so. After enhancing our deployment system to deploy a new set of Kubernetes resources to a github-production namespace in parallel with our existing production servers and enhancing GLB to support routing staff requests to a different backend based on a Flipper-influenced cookie, we allowed staff to opt-in to the experimental Kubernetes backend with a button in our mission control bar:

The load from internal users helped us find problems, fix bugs, and start getting comfortable with Kubernetes in production. During this period, we worked to increase our confidence by simulating procedures we anticipated performing in the future, writing runbooks, and performing failure tests. We also routed small amounts of production traffic to this cluster to confirm our assumptions about performance and reliability under load, starting with 100 requests per second and expanding later to 10% of the requests to github.com and api.github.com. With several of these simulations under our belt, we paused briefly to re-evaluate the risk of a full migration.

Cluster Groups

Several of our failure tests produced results we didn’t expect. Particularly, a test that simulated the failure of a single apiserver node disrupted the cluster in a way that negatively impacted the availability of running workloads. Investigations into the results of these tests did not produce conclusive results, but helped us identify that the disruption was likely related to an interaction between the various clients that connect to the Kubernetes apiserver (like calico-agent, kubelet, kube-proxy, and kube-controller-manager) and our internal load balancer’s behavior during an apiserver node failure. Given that we had observed a Kubernetes cluster degrade in a way that might disrupt service, we started looking at running our flagship application on multiple clusters in each site and automating the process of diverting requests away from a unhealthy cluster to the other healthy ones.

Similar work was already on our roadmap to support deploying this application into multiple independently-operated sites, and other positive trade-offs of this approach - including presenting a viable story for low-disruption cluster upgrades and associating clusters with existing failure domains like shared network and power devices - influenced us to go down this route. We eventually settled on a design that uses our deployment system’s support for deploying to multiple “partitions” and enhanced it to support cluster-specific configuration via a custom Kubernetes resource annotation, forgoing the existing federation solutions for an approach that allowed us to use the business logic already present in our deployment system.

From 10% to 100%

With Cluster Groups in place, we gradually converted frontend servers into Kubernetes nodes and increased the percentage of traffic routed to Kubernetes. Alongside a number of other responsible engineering groups, we completed the frontend transition in just over a month while keeping performance and error rates within our targets.

During this migration, we encountered an issue that persists to this day: during times of high load and/or high rates of container churn, some of our Kubernetes nodes will kernel panic and reboot. While we’re not satisfied with this situation and are continuing to investigate it with high priority, we’re happy that Kubernetes is able to route around these failures automatically and continue serving traffic within our target error bounds. We’ve performed a handful of failure tests that simulated kernel panics with echo c > /proc/sysrq-trigger and have found this to be a useful addition to our failure testing patterns.

What’s next?

We’re inspired by our experience migrating this application to Kubernetes, and are looking forward to migrating more soon. While scope of our first migration was intentionally limited to stateless workloads, we’re excited about experimenting with patterns for running stateful services on Kubernetes.

During the last phase of this project, we also shipped a workflow for deploying new applications and services into a similar group of Kubernetes clusters. Over the last several months, engineers have already deployed dozens of applications to this cluster. Each of these applications would have previously required configuration management and provisioning support from SREs. With a self-service application provisioning workflow in place, SRE can devote more of our time to delivering infrastructure products to the rest of the engineering organization in support of our best practices, building toward a faster and more resilient GitHub experience for everyone.

Thanks

We’d like to extend our deep thanks to the entire Kubernetes team for their software, words, and guidance along the way. I’d also like to thank the following GitHubbers for their incredible work on this project: @samlambert, @jssjr, @keithduncan, @jbarnette, @sophaskins, @aaronbbrown, @rhettg, @bbasata, and @gamefiend.

Come work with us!

Want to help the GitHub SRE team solve interesting problems like this? We’d love for you to join us. Apply here!