One of my bucket list items since I read my first party report back in 1991 was to attend a european demoparty and compete in a compo.  Icompeted at NAID ’96 and placed there, which was awesome, but to compete with the best of the best, and win, has always been a dream of mine.  I’m happy to announce that after six months of hard work with good friends and extremely talented people, we achieved just that.  Our demo, 8088 MPH, won the Revision 2015 oldskool demo compo.  (A personal victory for me was having it shown last in the compo, which is a sign of respect that the organizers think it’s the best high to end a compo in.) As of April 7th 2015, there are no IBM PC emulators in the world that can run the demo properly; they hang or crash before the demo is finished, and the colors are wrong.  Same goes for anything that isn’t the target hardware (see below).  To see what 8088 MPH looks like, I direct you to the video+audio capture of the demo running on real hardware: Because there are so many technological world-firsts in the demo, and because we’re bending the hardware in ways that people have never thought to do so, it’s only fair that we try to explain exactly how this was achieved.  One of my roles was “organizer” for the demo, so I’ll break it down scene by scene, covering the basics of each trick.  For parts that I wrote, I go into some detail, but for a deep technical dive into certain areas, I’ll keep this blog entry updated to point to reenigne’s, VileR’s, and Scali’s blog posts about their parts.  It is our hope that these discussions will foster revived “old-school” interest in software development for the platform. After you read this summary post, please visit the following links by us that break down, in-depth, specific sections of the demo:

And for more general info:

Before going into each part, let’s define what the target system was for the demo:  A 1981 IBM 5150 (aka the very first “IBM PC”) with 640 KB RAM, a floppy drive, IBM CGA card, and internal speaker.  That setup consists of:

• 4.77 MHz 8088 CPU.  5 MHz seems like a lot compared to other 8-bit micros, but it takes the CPU 4 cycles to read a single byte.  So, compared to other 8-bit CPUs like the 6502 or 6809, which can read a byte in one clock cycle, the effective clock speed of the 8088 is more like (4.77 / 4) = 1.19 MHz.
• Video adapter that has a 9-pin RGBI interface and an RCA NTSC composite video interface.  Driven by a Motorola 6845 character generator.  No facilities for changing text characters; font is fixed.
• Internal speaker; no sound card like the Sound Blaster, or special sound hardware like the C64 SID.  Speaker can be driven by a timer pin to produce a square wave, or can be bit-banged directly via a port or a one-shot timer.

The 640KB RAM requirement seems steep, but not only was it possible to add that to the first IBM PCs, by 1985 it was quite common.  If you still want to cry foul, then please note the only effect that uses just about all of that RAM is the kefrens bars, so that the repeating pattern would take longer to repeat and be more pleasing to the eye.  We could have reduced it, but then you might have noticed the pattern repeating quicker.  With the kefrens bars effect, the demo uses 507 KB RAM; without it, the demo uses 349 KB.  Most effects use much less, and some are tiny, like the plasma which uses only 6KB (which includes the banner graphics) and the picture of the girl which uses 18K (2K more than the size of the raw picture data itself).  We intentionally traded size for speed, which was a deliberate decision to fit as many effects as we could into 8 minutes running time, the compo limit.  If we had a few more minutes running time, we probably could have fit the entire demo into 256 KB or even less, but you would have waited longer between effects. I should also note here that there were two different versions of IBM CGA produced, which differ mainly in how composite colors are generated.  We had equal numbers of both “old” and “new” style IBM CGA cards, so we chose to compose graphics for the “old” style.  If you have the “new” style CGA card, the demo will still run, but the colors will be off slightly.

Development tools used

• Turbo C
• Turbo Pascal
• Turbo Assembler
• Turbo Debugger
• Visual C++
• OpenWatcom
• NASM (and YASM)
• DOSBox
• A few real IBM 5160s (hardware equivalent to the 5150, but easier to find in the real world)

Any data files were directly included in the .exe/.com files themselves.  This kept everything together in the same binary which means the data could benefit from compression (see below). Most development cycles involved designing in wetware, coding on a modern system (or DOSBox running on a modern system), testing/debugging in DOSBox, and then transferring over to real hardware for a final test.  Once an effect grew so sophisticated it couldn’t run in an emulator any more, this cycle slowed down as testing could only be done on real hardware.  Various transfer methods were used to get code to real hardware:  Scali used a serial cable; I used an ethernet card running a packet driver and mTCP; at the party we used an 8-bit IDE ISA adapter (Silicon Valley ADP-50) connected to a CF-to-IDE adapter to make a CF card the hard drive, then used a USB CF card reader to move stuff back and forth.  The most intriguing method of all was reenigne’s method, who used a custom controller connected to the keyboard port that used the IBM BIOS factory test mode as a poor-man’s serial port.  (I hope Andrew writes up some details on that!)

Loader, API, and general structure

We all had different preferred development languages and environments, so it was decided early on to create an overseeing “loader” that would execute .EXE and .COM files, and then people could develop effects in whatever environment they wanted to.  This is not a new concept; the famous Second Reality demo did this for exactly the same reasons, and the same technique was used even earlier than that on numerous demos on other platforms.  (Before you ask: No, the Second Reality code was not copied; in fact, it wasn’t even consulted, as we had to write extremely tight code to minimize memory usage, and also have it work on an 8088 (the Second Reality code uses 80186 opcodes).  The loader API services assemble to about 450 bytes of code. The loader, as designed, would be responsible for:

• Playing music in the background
• Masking the load times and precalc times of various effects using “megademo”-style text
• Providing synchronization services (such as providing a vertical-retrace interrupt in software, and a user-definable countdown timer)

Running effects with the loader consisted of this workflow:

1. Print text on the screen and animate it using an interrupt and the 6845 start address register
2. Execute the effect
3. The effect would decompress, perform precalc, etc. and then signal the loader it is ready to start
4. The loader cleans up the moving onscreen text, then signal the effect it can start
5. Effect starts, magic occurs

Designing this correctly was extremely important, as any bugs would derail the entire thing.  It was designed fully before even a single line of code was written.  I’ve shared the design doc online for the curious.  (I wrote the loader.) The background music playback had to be as simple as possible so as to not interfere with any effects.  A single PC beep, changing (or silencing) once every frame, was the only thing that was practical, so 60Hz beeping is what the background music consists of.  The composition program used for generating the speaker timer values was MONOTONE.  Even though the code for playback is only 18 lines of assembler, it takes up two scanlines onscreen, so you can see how anything even slightly more complicated would have sucked much more CPU out of the system and some of the full-screen 60Hz effects simply would not have been possible.

Executable compression

Another decision early on was to see if executable compression was feasible, which means the following:

• Does it actually compress things small enough to be worthwhile?
• Is the decompression quick enough to avoid long pauses in the demo?
• Does the decompression routine affect the system while it decompresses? (ie. does it disable interrupts or something else equally horrible while it decompresses, which would trash the demo?)

I gathered most classic and modern executable compressors and ran tests against old programs that were representative of what we would be producing.  The results were welcome surprises.  The compression ratios were good enough that we could afford to include precalc’d data instead of calculating it on the fly, and the decompression was fast enough that the total end-to-end time loading a program from diskette was actually slightly faster than if it were to load uncompressed.  In the end, pklite emerged as the winner.  I’ve shared the comparison data online for comparison.  (If I missed any packers that hold substantial advantages over the ones in the data, please let me know what they are.  There were nearly 100 packers made for DOS, but unless they compress smaller than apack or upx, or decompress faster than pklite or lzexe — all while remaining compatible with 8088— then I don’t want to hear about them.)

Scene-by-scene breakdown

What follows is a screen-by-screen explanation of each effect.  As previously stated, I’ll only describe scenes in detail if I wrote them; it will be up to the others if they want to write a technical breakdown for their parts.  The explanation for each effect follows after the effect’s screenshot. The introduction was meant to serve two purposes:  To educate the audience on the system and explain at just how much of a disadvantage we were trying to make a world-class demo on such hardware, and also simultaneously shatter their expectations :-)  The text mode is obviously simulated; I essentially duplicated the basic BIOS functions for handling text mode but simulated in graphics mode.  The cursor blinking and text blinking are handled identically to how the 6845 does it, adding to the illusion. It is (nearly) impossible to change the display start address of graphics mode such that every single scanline comes from a different place, so the title screen unrolling was done brute force, by copying new scanlines into memory hidden by retrace.  The title screen goes away with a “fade” on the top edge by ANDing a mask on successive lines of the screen data. A lot of people think the title screen is the same picture demonstrated by VileR a few years ago.  It’s not!  He recomposed it for 16-color composite specifically for this demo, and changed it subtlety as well. The bobbing was achieved by creating a software vertical retrace interrupt that fired at the same place onscreen every time (just after the last displayed line) and then hooking it with a 6845 display start address change routine.  Flags were used to communicate to the interrupt if it was time to erase the letters, which was done by simply using REP STOSW to fill screen memory with black lines.  Because the 6845 displays two onscreen rows per “row”, the text could only move to even lines, which is why the movement isn’t as smooth as it could be.  Well, to be fair, it could be made to move to any line we wanted, but doing so would be CPU intensive, and the whole point of the loader is to use as little CPU as possible, so this was the compromise. The simulated vertical retrace interrupt was provided through loader API services for the rest of the effects to use as well.  Effects could disable it, re-initialize it, and hook/unhook their own routines to it. The moire (interference pattern) effect was achieved using a base of 40×25 text mode, the half-char block extended ASCII characters, and lots of unrolled code.  The circles were chosen to represent the classic effect, but in reality the effect can combine any two images.  reenigne’s effect. The rotozoomer is the same tired old routine I first rolled out in 1996 in the 8086 compo, but optimized to the hilt and sped up by only drawing every other line.  A miscommunication between me and VileR resulted in probably not the best texture to demonstrate the effect, but it still runs well enough.  There were plans to include a 60 Hz version of this effect, but we ran out of time. The core concept of the 1024-color mode is a serious abuse of 80×25 text mode with the NTSC colorburst turned on.  VileR made the first discovery with 512 colors, and reenigne was able to double this to 1024 with CRTC trickery. Some people thought the entire demo was in this mode.  It was not, because 80-column text mode suffers from the famous CGA “snow” defect when you write directly to CGA RAM in this mode.  This is unfortunately visible in the plasma effect (see below). BTW, when I saw this picture in 2013, that’s when I knew I had to get all these people together to make a demo.  I mean, geezus, look at it!  My jaw dropped when I saw it.  Had I never seen VileR’s collaboration with reenigne to make the above, 8088 MPH might never have existed. These stars were actually the result of unrolled code and a precalc’d table that, together, take a byte from one location and moves it to another position in video RAM.  While we had other patterns ready, such as a swirling display, we felt the starfield was most appropriate for a typical “oldskool” demo.  reenigne’s effect. The sprite part seems like black magic, but is the combination of using a sprite compiler written by Scali, and adjusting the screen vertically using the 6845 start address register.  CGA only has one screen’s worth of video memory, so moving the address down scrolls the screen up, with the data repeating across the boundary.  The data doesn’t repeat evenly across the boundary, however, requiring handling.  The timer was monitored to know when the screen line containing the last pixel of the sprite had been drawn, which prompted redrawing the sprite.  (In other words, re-drawing the sprite was an exercise in racing the beam.)  Timing was very tight to avoid screen/sprite tearing effects. Also part of the compiled sprite effect, this displays 30 vectorballs at 30 Hz.  We had an earlier display that used less balls to achieve 60 Hz, but Scali had the idea at the last minute to make them spell out something like “8088”, “IBM”, etc. and coded up the change at the party.  The update is done using double-buffering; the sprites only take up a small rectangular area onscreen, so the screen mode’s CRTC settings were reprogrammed to provide a video mode with a small area in the middle of the physical screen, using only half of available video memory.  This provided a true hidden page to draw/erase vectorballs to, which was then flipped to be visible using the 6845 display start address register. Using a 1024-color variant screen mode that could be updated using only the attribute byte (thereby limiting the number of colors to 256), this plasma had to perform writes only when the CRT beam was retracing horizontally or vertically.  Unfortunately, the timing required to get this right stopped working at the party for some reason (probably happened as we were rearranging effect order), and as a result you can see a line of noise along the left side of the screen, and a little bit of noise at the top.  This was my fault, as I wrote the effect using a somewhat lazy polling routine.  It’s a shame CGA snow exists, because without all the retrace handling to avoid it, this effect runs at 60fps.  In the demo with snow avoidance, it runs at only 20fps.  VileR may write more about how this screen mode and color system is constructed, and if so, I’ll update the links at the top of this article to point to the method. If we come out with a final version of the demo, fixing this is at the top of the priority list.  In fact, I’m betting reenigne could change this from a polling effect to a cycle-counting effect, which would not only fix the snow, but speed it up. The 1024-color mode reprograms the start address every two lines.  I took advantage of this behavior to create a simple “drip” effect for VileR’s amazing artwork.  Already you can posit that much more complicated effects are possible (thinking of the Copper demo here) but I ran out of time to make it more awesome. This classic Kefrens bars effect was done by reenigne in 320x200x4 mode.  It’s a cycle-counting effect, as there is simply no time to monitor for horizontal retrace.  To ensure the cycle counting was consistent, several things were done including changing the system default DRAM refresh from it’s default interval of 18 to 19, to get the DRAM refresh periods to line up with CRTC accesses. This was Scali’s effect and inspired by his 1991 demo which also featured a large torus.  There are several things going on here:

• Only changed portions of the screen are calculated and drawn, to minimize the amount of bandwidth needed to update the screen (this is the same “delta drawing” idea used in XDC).  This was done because CGA video memory has a wait state, so the less you need to write to it, the better.
• 320x200x4 mode is used with a background and palette combination that gives this specific composite color palette, which included many shades of blue.
• To help with the shading, dithering is applied during rasterization.

At the party, reenigne posited that it should be possible to restart the CRTC start address every single scanline.  This would result in a video mode that was only 100 lines high, and would give a 80×100 resolution 1024-color mode.  The above is the result of that coding, plus really extensive work done on a CGA NTSC composite signal modeling program done by reenigne months earlier to perform the image conversion.  (No, you can’t have it.  And before you ask, the “girl” and “CGA 1k” pictures were not stock conversions, but were hand-pixeled by VileR in Photoshop, and the 4-colors/16-colors/”Until Now” screens in a customized version of Pablodraw he created.) We didn’t have time to put text into this picture, so the people you see above are the same as in credits order:  Trixter, reenigne, Scali, VileR, Phoenix, and virt.  Apologies to coda and puppeh, but as you can see, any more squishing and the faces would have been unrecognizable.  Sorry! Finally, the coup de grâce:  A multichannel music engine for the PC speaker.  We didn’t want to just copy a ZX Spectrum engine, nor other engines such as the one used in Music Construction Set, but rather set the bar impossibly high by playing a protracker mod through the speaker.  Other modplayers for the speaker already exist, but they require a 10 MHz 80286, and can barely manage output at a 6KHz sampling rate.  Ours faithfully reproduces all protracker effects, mixing and outputting to the speaker realtime at 16.5 KHz, all on a 4.77 MHz CPU. This was reenigne’s baby, and is a truly stunning technical achievement that required unconventional thinking and considerable 8088 knowledge to pull off.  I’m sure he will write up a more detailed post on how it was done.  Until then, I can mention the following details:

• Preconversion of the module was necessary to align data structures and sample data to be favorable to how the 8088 indexes memory.  Sample data is also converted.
• Each sample must take exactly 288 cycles to calculate and output or else the sound goes completely pants.  This was very difficult to achieve.  4.77 MHz / 288 = 16572 Hz sample output.
• Audio output was done using traditional Pulse-Width Modulation (PWM) techniques, such as the kind made popular by Access’s Realsound.  PC speaker PWM is performed by tying the PC speaker input pin to the programmable interrupt timer’s (PIT) channel 2, then programming PIT 2 for byte value one-shot mode.  Any value sent to PIT 2 with the system configured like this will set the speaker HIGH and start a count, and when the count expires (ie. the sent value is reached), the speaker goes LOW again.  This results in an audible carrier wave “whine”, which was why the output needed to be fast (16.5 KHz) so that the carrier wave was above the range of human hearing.

Fun fact:  After preconversion of the song and being turned into a self-playing .exe, the final result is smaller after compression than the size of the original source module.

Party Sprint

At the party, we came with something that was 90% finished.  Prior to arriving at the party, we created what we thought was a decent entry, and created two “failsafe” videos, one that was a capture for the bigscreen and another that showed the demo running on real hardware as verification for the judges.  We were worried that the hardware we were bringing would get damaged in transit, so this was a precaution so that we could enter something if that happened.  Thankfully, reenigne’s and Scali’s IBM 5160s arrived unharmed (which was especially remarkable since reenigne had to bring his from the UK to Germany on a series of trains!).  We also brought two CGA cards, and two capture devices, and three different methods of exchanging new software bits from our laptops to the old hardware.  You can never be too prepared! Most of the coding time at the party was spent adding the kefrens and ending portrait picture, eliminating bugs from each part where possible, adding nice transitions where possible, shaving seconds off of each part to stay within the compo limit, and rearranging parts so that virt’s BTTF-inspired tune’s intro lined up with the sprite part.  We spent pretty much all our time before the compo coding, eating, or visiting the bathroom, and only had time to socialize after that. While we came mostly prepared for something that was worthy of entering the compo, the time spent at the party was invaluable for turning a rough draft into something that could really compete for first place.  Having all four of us at the same table meant we could collaborate instantly.  So, lesson learned:  There are rarely substitutes for working together in person!  One of the biggest improvements of “party collaborating” was the decision to change the credits from a variable-speed, text-only scrolling to a more evenly-paced, ANSI-style scrolling, which I think was the best implementation change compared to the bits we brought from home. To help save time (and to ensure the video was converted well — sorry, but most people don’t know how to deal with interlaced video properly), I offered to provide Gasman with a 720@60p video.  The NTSC output of CGA is slightly off; instead of 262.5 lines per field, it generates 262.  This means it generates 59.92 fields (29.96 frames) per second instead of the NTSC broadcast standard of 59.94 (29.97 fps).  This throws off most modern capture devices; Scali had access to a high-quality Blackmagic Intensity Shuttle, for example, but it couldn’t lock onto the signal.  I knew from experience that some cheap video capture devices, such as the Terratec Grabby or the Dazzle DVC100, have extra tolerance built into them as they were designed to be used with VCR sources, so I bought a few and sent one to reenigne for testing.  For the capture, we used a DVC100 with some slight proc amp adjustments so that the capture looked as close to the CRT monitor output as possible.  To further ensure better video capturing, we used VirtualDub for the capture software, which has an option to dynamically resample the input audio source to fit the capture framerate you are aiming for in case it’s slightly off, and the software and hardware combination worked very well.  For grabbing the audio, we initially tapped the speaker with alligator clips, but Scali brought his Sound Blaster which had a real PC speaker tap you could hook up with an internal cable, so we used that for the final capture.

After watching the demo and reading the above, you may be wondering if there is actually room for improvement.  Believe it or not, there is:  Alternative methods of sound generation and additional cycle-exact trickery are definitely possible.  We had more effects to put into the demo, but ran out of time:  We ran out of development time, and we also ran out of execution time, as the Revision compo limit was 8 minutes or less. I’ve known everyone who has worked on the demo collectively over 60 years.  It was an honor and a privilege to work with them all to produce this demo.  Will we work together again?  I’d say it’s definitely possible; the day after the compo, we threw around some ideas, such as making a game next instead of a demo.  Me personally, I’m burnt out and will be spending the next few weeks playing some games I’ve always wanted to finish, and working on my health.  I also have some other large projects I want to get kickstarted this summer, such as something the PC software preservation movement desperately needs, and an online sound card museum.  But hey, who knows.

Sacramento investigators tracked down East Area Rapist suspect Joseph James DeAngelo using genealogical websites that contained genetic information from a relative, the Sacramento County District Attorney's Office confirmed Thursday.

The effort was part of a painstaking process that began by using DNA from one of the crime scenes from years ago and comparing it to genetic profiles available online through various websites that cater to individuals wanting to know more about their family backgrounds by accepting DNA samples, said Chief Deputy District Attorney Steve Grippi.

The investigation was conducted over a long period of time as officials in Sacramento County District Attorney Anne Marie Schubert's office and crime lab explored online family trees that appeared to match DNA samples from the East Area Rapist's crimes, Grippi said. They then followed clues to individuals in the family trees to determine whether they were potential suspects.

The process finally came to fruition last Thursday, when the investigation focused on the possibility that DeAngelo might be a suspect, a suspicion bolstered by the fact that he had lived in areas where the attacks occurred and was in the right age range, Grippi said.

Schubert said in an interview at her office that the April 19 determination that DeAngelo might be a suspect set the investigation into high gear.

Sacramento County Sheriff Scott Jones' investigators set up surveillance on DeAngelo in his quiet Citrus Heights neighborhood and obtained his DNA from something he discarded.

The crime lab began testing the material and Schubert said she got a call last Friday night from Grippi while she was at a high school fundraiser telling her that the DeAngelo DNA matched that found at decades-old murder scenes in Ventura and Orange counties.

"I was at a dinner at Cristo Rey High School and Steve Grippi called me," she said. "And so I probably used a few words I wouldn't put in a newspaper, but basically said, 'You'd better not be lying to me.'"

The sample provided "overwhelming evidence that it was him," Schubert said, but she decided they wanted a second sample, which sheriff's officials recovered.

The results from testing that second sample came in while Schubert was in her office Monday night, she said.

"The second sample was astronomical evidence that it was him," Schubert said, adding, "There were a whole lot of holy s--- moments."

Authorities began moving quickly to plan the arrest.

For those that read my blog often you may have seen me talk about EV certificates before. From a technical perspective there are many things that I don't agree with about EV certificates and from a non-technical perspective there are many things I don't like about the way they're advertised and sold. There is however another problem, and it's what happens when your CA turns against you.

Extended Validation

Extended Validation, or 'EV', certificates give you a little piece of green lipstick in the browser when you visit a website. You can see an EV certificate in action on the PayPal site here:

The "PayPal, Inc. [US]" string to the left of the address is what EV gives you (most of the time) and that's it. If you want more details on the problems with EV you can read my blog Are EV certificates worth the paper they're written on? but in this article I want to focus on what happens with your certificate after it's issued.

Clearing up what EV 'is'

Before we start on the issue here, I think it's important to clarify exactly what an EV certificate 'is' and does. The best and only place to get this information is from the CA/Browser Forum, the governing body of Certificate Authorities if you will. They set out the Baseline Requirements which is the rule book by which all CAs must play and for EV certificates there are the additional EV SSL Certificate Guidelines too. The EV SSL Guidelines are where we need to focus our efforts today and at the time of writing v1.6.8 is current. Section 2 of the guidelines set out the purpose of EV certificates in general and section 2.1.1 gives us the following:

2.1.1. Primary Purposes
The primary purposes of an EV Certificate are to:
(1) Identify the legal entity that controls a Web site: Provide a reasonable assurance to the user of an Internet browser
that the Web site the user is accessing is controlled by a specific legal entity identified in the EV Certificate by name,
address of Place of Business, Jurisdiction of Incorporation or Registration and Registration Number or other
disambiguating information; and
(2) Enable encrypted communications with a Web site: Facilitate the exchange of encryption keys in order to enable
the encrypted communication of information over the Internet between the user of an Internet browser and a Web site

So, an EV certificate lets us know that a legally registered company applied for and obtained the certificate, and thus controls the site/domain, and to enable encrypted comms to that site, just like any other certificate would. Section 2.1.2 goes on to add some secondary purposes for EV certificates:

(1) Make it more difficult to mount phishing and other online identity fraud attacks using Certificates;
(2) Assist companies that may be the target of phishing attacks or online identity fraud by providing them with a tool to
better identify themselves to users; and
(3) Assist law enforcement organizations in their investigations of phishing and other online identity fraud, including
where appropriate, contacting, investigating, or taking legal action against the Subject.

The first point here is due to the cost and process involved in obtaining EV certificates but I think that's a really dangerous assertion to make and we should have already learned from this mistake. For years, decades even, we put too much faith in the DV indicator of a padlock because they were expensive and hard to get, and look where that left us. To make the same mistake again with EV would be foolish. The second point is the problem of relying on users to identify and behave according to the EV indicator and the third point is a bit of a stretch in my opinion but would hold true. Moving on to the next section, 2.1.3, we have the excluded purposes of EV certificates, things that they are not to be used for:

2.1.3. Excluded Purposes
EV Certificates focus only on the identity of the Subject named in the Certificate, and not on the behavior of the Subject.
As such, an EV Certificate is not intended to provide any assurances, or otherwise represent or warrant:
(1) That the Subject named in the EV Certificate is actively engaged in doing business;
(2) That the Subject named in the EV Certificate complies with applicable laws;
(3) That the Subject named in the EV Certificate is trustworthy, honest, or reputable in its business dealings; or
(4) That it is “safe” to do business with the Subject named in the EV Certificate.

Looking at point 1 you can easily agree with that, having an EV cert doesn't mean we're actively conducting business, but after this, things get interesting. Point 2 basically translates to 'EV certs don't mean a site is good' and points 3 and 4 also translate to exactly the same thing. A site with an EV cert can rip you off, steal your information, phish you and do all kinds of other nasty things. The point I want to make here is that EV certs offer assurance of nothing other than the fact that somewhere there is a legally registered company with the name shown in the EV indicator. That's it. It's really important that we understand the purpose and limitations of EV certificates if we're to understand the following problems.

Demonstrating weaknesses in EV

EV certificates rely absolutely and completely on the user for them to have any effectiveness. If the user doesn't know what the indicator is or means, they don't look for and verify the indicator or don't check for its presence on every page load, the EV certificate provided no value. My biggest problem with EV is this dependency on the user, I don't think it's fair to expect the user to participate in our security, but that aside, the user is unreliable.

In recent weeks I've been following and supporting Ian Carroll with some of his research to demonstrate problems with EV. He registered a company in the US called Stripe Inc. and applied for an EV certificate for his company. He meets the EV criteria in that he owns a legally registered company and controls the domain he was applying for a certificate for, and a certificate was issued to him. You can see his site here stripe.ian.sh but it doesn't currently have an EV certificate, which we'll talk about shortly:

Of course, he is demonstrating that his EV certificate looks very similar to the Stripe Inc. you're probably thinking of, the payment processor.

We need to be clear up front that Ian's certificate is absolutely valid and properly issued here. Legally registered company names do not provide any guarantee of being globally unique and he controls the domain stripe.ian.sh, there is nothing wrong here. The problem is that some CAs don't like it when you make them look bad, especially not when you're making their main revenue stream look bad. Comodo issued that certificate to Ian and his legally registered company but after it gained some attention in the wider community, Comodo were not happy.

Following various discussions and debates online the community agreed that this certificate was perfectly legitimate, it met all of the criteria to be issued and no action was required. Comodo did not agree. Comodo did not agree at all. In fact, Comodo disagreed so much that they took action against Ian and his certificate, they revoked it.

You can see the details of that revocation right here but yes, Comodo revoked Ian's certificate, despite him having done nothing wrong and having met all of the criteria to apply for and obtain it, which is why he got it in the first place. This represents a pretty significant problem because the CA has now taken his site offline. Without a valid certificate the website won't work so Comodo have essentially taken his site down. The poor excuse that Comodo came up with for this revocation was apparently that Ian was running a phishing site but I think we can all see this was because they looked bad. I prompted Ian to get another certificate but found that he didn't have the cash to burn on expensive EV certificates and he didn't get a refund for his now revoked certificate so he couldn't use that either (yes you did read that right, they revoked his cert and kept his money). Unhappy with the way this was playing out I offered to money for Ian to apply for an EV cert from another CA and he got one from GoDaddy.

The site was back and the proper EV indicator restored for the security and benefit of all! Except, it didn't stay online for long...

Yep, GoDaddy revoked the certificate and took Ian's site offline overnight. This is a really concerning event because the CA is wielding a great amount of power here and right now it's pretty apparent that they're abusing that power. Looking at the revocation the CA provided the reason 'cessationOfOperation' but Ian certainly hadn't stopped operating his site and most definitely did not ask for the revocation to take place.

If you want the details of this revocation then you can find those right here but again, we're in a situation where a CA has issued a certificate, verified all of the necessary information including the legally registered company and ownership of the domain name, charged money, issued the certificate and then at some point in the future changed their mind. They revoked the cert, took his site down and that's apparently the end of it. The only difference this time around is that after Ian spoke to GoDaddy they did refund his money, how very kind of them...

The larger issue

The bigger problem here is around how the CAs are wielding the power they have and abusing their position. Yes, this might not be a glorious example of how great EV is and it might not exactly make the CA look good, but the response here is chilling. We've seen similar issues in the security community for years where someone points out a problem with a system only to be met by overzealous reactions and an inappropriate response. In revoking the certificate issued to Ian they have taken his site down and potentially caused harm to him or his business. Both of these revocations came after the fact, because the CA was concerned about their reputation. This is not and should never be sufficient grounds to revoke a certificate. Just think about that, what happens if that continues to be acceptable as we move towards a 100% encrypted web?

When the entire web is encrypted, and we're making great progress on getting there, the CAs become powerful gatekeepers of content that can be put online. My Alexa Top 1 Million report from Feb 2018 shows that over 38% of sites in the top 1 million are now redirecting to HTTPS. We've also discussed the problem of a CA being some kind of 'content police' on the web before and it's really not a great idea. Over a year ago in March 2017 I published Let's Encrypt are enabling the bad guys, and why they should to address the constant reports of Let's Encrypt issuing certificates to phishing sites. The CA should not, must not and cannot be some or any kind of censor for which sites get certificates to go online. CAs have technical criteria set out in the Baseline Requirements and EV SSL Guidelines and they do not include any kind of 'are you a good person' metric, to suggest that they should is terrifying.

Going forwards

I will continue to fund Ian in his quest to obtain an EV certificate that a CA won't later turn around and revoke for many reasons, the main reason being that he should be able to get one according to all of the rules and guidelines set out by the CA/Browser Forum. Demonstrating a perceived flaw or weakness in the EV system should not give a CA the power revoke that certificate and there should have been consequences for those that have already. Once we start down the road that a CA can revoke a certificate for [reasons], where do we end up? DMCA takedowns, copyright, trolls, trademarks or even writing an article critical of a CA? What about paypal-class-action.com, should PayPal be able to have that revoked because it contains a substring match for PayPal? Certificates are issued once the technical criteria for issuance is met and the only time that a revocation should be exercised is when the host that obtained the certificate requests it or another party can prove ownership of the associated private key. To use revocation for any other purpose is a slippery slope and once we start down that slope, it's going to be hard to come back from our race to the bottom.

BOURNEMOUTH, England (Reuters) - Scientists have uncovered evidence of ancient humans engaged in a deadly face-off with a giant sloth, showing for the first time how our ancestors might have tackled such a formidable prey.

Standing over 2 meters tall, with forelegs tipped with claws, giant sloths lived until around 11,000 years ago. Most scientists believe over-hunting by humans eventually led to their extinction.

Fossilised footprints in the salt flats of White Sands National Monument, in the southwestern U.S. state of New Mexico, reveal humans walking in the exact footsteps of a giant sloth and then confronting it, possibly hurling spears.

“The story that we can read from the tracks is that the humans were stalking; following in the footsteps, precisely in the footsteps of the sloth,” said Matthew Bennett, one of a team of scientists behind the discovery.

“While it was being distracted and turning, somebody else would come across and try and deliver the killer blow. It’s an interesting story and it’s all written in the footprints,” said Bennett, a professor of environmental and geographical sciences at Bournemouth University in southern England.

At the White Sands National Monument, researchers identified what are known as “flailing circles” that show the rise of the sloth on its hind legs and the swing of its fore legs, likely in a defensive motion.

In addition to tracks of humans stalking the sloth, there are more human tracks further away. From this, scientists infer that the humans worked as a group, with a separate team distracting and misdirecting the sloth to outwit it.

The flailing circles are always associated with the presence of human tracks. Where there are no human footprints, the sloths walk in straight lines, but where human tracks are present the sloth trackways show evasion, with sudden changes in direction.

Thanks to new 3D modeling techniques, the fossilized footprints have been preserved using a system developed by Bennett. Using a standard digital camera to take images from 22 different angles, his computer algorithm builds up an ultra-precise 3D rendering of the footprint.

“What this evidence does is for the first time shows us how they might have tackled one of these big beasts and the fact that they were almost certainly doing it routinely is important,” Bennett told Reuters at his university.

“Getting two sets of fossil footprints that interact, that show you the behavioral ecology, is very, very rare,” he said.

Right now the market for educational software is a mess. It’s incredibly difficult for developers to get their products into schools, and it’s even harder for them to scale. School districts spend tons of money on learning applications, but they have no way of knowing if students are even using the apps they’re purchasing. Teachers know there’s great software out there, but relatively simple challenges like getting 30 students logged in at once make using it impossible.

At Clever, we’re working to change all that. We provide schools with a free API and single sign-on solution that makes using educational apps a breeze. We’ve grown fast: after five years, half of all schools in the US (and 90 of the 100 biggest districts) are using our platform. Our goals are much bigger than that, though. We want to be a full-cycle marketplace where schools can identify, purchase, integrate, and evaluate the hundreds of educational apps they're using.

We’re a team of about 110 (35 engineers) based in downtown SF, and we’re looking for engineers who enjoy working in (or would enjoy learning) Golang, Node and React. More generally, we want people who are sharp, adaptable, and passionate about improving the way education works for everyone.

Learn more at https://clever.com/about/jobs !

NOTE: We're also hosting a "Small Teams, Big Impact" meetup on Monday, April 30th at our office (1263 Mission St.) at 5:45 PM. Come check it out!

https://www.eventbrite.com/e/software-meetup-small-teams-big...

These release notes for Ubuntu 18.04 LTS (Bionic Beaver) provide an overview of the release and document the known issues with Ubuntu 18.04 LTS and its flavors

Support lifespan

The 'main' archive of Ubuntu 18.04 LTS will be supported for 5 years until April 2023. Ubuntu 18.04 LTS will be supported for 5 years for Ubuntu Desktop, Ubuntu Server, and Ubuntu Core. Ubuntu Studio 18.04 will be supported for 9 months. All other flavors will be supported for 3 years.

Official flavor release notes

Find the links to release notes for official flavors here.

Download Ubuntu 18.04 LTS

Images can be downloaded from a location near you.

You can download ISOs and flashable images from: http://releases.ubuntu.com/18.04/ (Ubuntu Desktop and Server)
http://cdimage.ubuntu.com/ubuntu/releases/18.04/release/ (Less Popular Ubuntu Images)
http://cloud-images.ubuntu.com/daily/server/bionic/current/ (Ubuntu Cloud Images)
http://cdimage.ubuntu.com/netboot/18.04/ (Ubuntu Netboot)
http://cdimage.ubuntu.com/kubuntu/releases/18.04/release/ (Kubuntu)
http://cdimage.ubuntu.com/lubuntu/releases/18.04/release/ (Lubuntu and Lubuntu Alternate)
http://cdimage.ubuntu.com/ubuntu-budgie/releases/18.04/release/ (Ubuntu Budgie)
http://cdimage.ubuntu.com/ubuntukylin/releases/18.04/release/ (Ubuntu Kylin)
https://ubuntu-mate.org/download/ (Ubuntu MATE)
http://cdimage.ubuntu.com/ubuntustudio/releases/18.04/release/ (Ubuntu Studio)
http://cdimage.ubuntu.com/xubuntu/releases/18.04/release/ (Xubuntu)

Upgrading from Ubuntu 16.04 LTS or 17.10

To upgrade on a desktop system:

To upgrade on a server system:

• Install the update-manager-core package if it is not already installed.

• Make sure the Prompt line in /etc/update-manager/release-upgrades is set to normal.

• Launch the upgrade tool with the command sudo do-release-upgrade -d.

• Follow the on-screen instructions.

Note that the server upgrade will use GNU screen and automatically re-attach in case of dropped connection problems.

There are no offline upgrade options for Ubuntu Desktop and Ubuntu Server. Please ensure you have network connectivity to one of the official mirrors or to a locally accessible mirror and follow the instructions above.

Updated Packages

Linux kernel 4.15

Ubuntu 18.04 ships with a v4.15 based Linux kernel, enabling the latest hardware and peripherals available from IBM, Intel, and others. The 18.04 kernel delivers new features inherited from upstream, including:

• CPU controller for the cgroup v2 interface
• AMD secure memory encryption support
• The latest MD driver with software RAID enhancements
• Improved power management for systems with SATA Link Power Management

We also see notable Ubuntu specific achievements with:

• Linux security module stacking support
• Support for signing of POWER host and NV kernels
• Backport improved support for IBM and Intel hardware from Linux 4.16

OpenJDK

As of 18.04 release, OpenJDK 10 is the default JRE/JDK. Once OpenJDK 11 reaches GA in September 2018, it will become the default in 18.04.

OpenJDK 8 has moved to universe and will remain available there for the life of 18.04, to provide migration time for packages, custom applications, or scripts that can't be build with OpenJDK 10 or 11. OpenJDK 8 will be updated in 18.04 until Ubuntu 16.04 LTS reaches EOL in April 2021.

Security Improvements

In Ubuntu 18.04 LTS, gcc is now set to default to compile applications as position independent executables (PIE) as well as with immediate binding, to make more effective use of Address Space Layout Randomization (ASLR). All packages in main have been rebuilt to take advantage of this, with a few exceptions.

Mitigations are in place to protect against Spectre and Meltdown. See the Spectre and Meltdown KnowledgeBase article for more details about the remediation and configuration options.

bolt and thunderbolt-tools have been promoted to main to provide security controls for Thunderbolt devices.

Default CIFS/SMB protocol version change in CIFS mounts

Since 17.10, the default SMB protocol used when mounting remote CIFS filesystems via mount.cifs (from the cifs-utils package) changed to 2.1 or higher, depending on what is negotiated with the server. If no version is specified when mounting such a remote share, the following will be logged:

No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3),from CIFS (SMB1). To use the less secure SMB1 dialect to access old servers which do not support SMB3(or SMB2.1) specify vers=1.0 on mount.

Should you encounter compatibility issues, like #1764778 or #1572132, please specify vers=1.0 when mounting the share and please file a bug if that fixes the problem for you.

Network configuration

New since 17.10

Teaming support with libteam is available in NetworkManager.

New since 16.04 LTS

The default DNS resolver is systemd-resolved.

ifupdown has been deprecated in favor of netplan.io and is no longer present on new installs. The installer will generate a configuration file for netplan.io in the /etc/netplan directory. This netplan.io configuration in turn renders backend-specific configuration via either systemd-networkd or NetworkManager. Desktop users will see their system fully managed via NetworkManager as it has been the case in previous releases. Server users will now see their network devices managed via systemd-networkd. This only applies to new installations.

Given that ifupdown is no longer installed by default, the commands: ifup and ifdown are also unavailable. Please use the ip command to achieve similar functionality, specifically ip link set $device up and ip link set $device down.

The networkctl command is now available for users to see a summary of network devices. networkctl status will display the current global state of IP addresses on the system. networkctl status $device displays details specific to a network device.

The ifupdown package remains available and supported in Ubuntu main for users that find netplan does not currently meet their networking needs.

For more information about netplan.io, please refer to the manual page using the man 5 netplan command or visit https://netplan.io/.

Scripts in /etc/network/ifup.d and /etc/network/ifdown.d no longer work in this new configuration. For the systemd-networkd backend, similar scripts can be added into subdirectories of /usr/lib/networkd-dispatcher (dormant.d, no-carrier.d, off.d, routable.d), if networkd-dispatcher is installed. Later on, custom scripts can be placed in /etc/networkd-dispatcher and potentially also override the ones in /usr/lib.

Other base system changes since 16.04 LTS

• The gpg binary is provided by gnupg2

• For new installs, a swap file will be used by default instead of a swap partition.

• Python 2 is no longer installed by default. Python 3 has been updated to 3.6. This is the last LTS release to include Python 2 in main.

• The installer no longer offers the encrypted home option using ecryptfs-utils. It is recommended to use full-disk encryption instead for this release. (1756840)

• OpenSSH now refuses to use RSA keys smaller than 1024 bits. ssh-keygen -l -f /path/to/key.pub can report the length of a key.

Ubuntu Desktop

New since 17.10

• X is the default display server. Wayland is provided as a Technical Preview and is expected to be the default display server in 20.04 LTS. To try it out, just choose Ubuntu on Wayland from the cog on the log in screen.

• The installer offers a minimal install option for a basic desktop environment with a web browser and core system utilities. Many official 18.04 desktop flavors are using this new feature too!

• Apps provided by GNOME have been updated to 3.28. For more details about GNOME 3.28, see their Release Notes.

• LibreOffice has been updated to 6.0.

• Emoji now show in color in most apps. Keyboard shortcuts for the emoji input chooser are Ctrl+. or Ctrl+;

• Calendar now supports weather forecasts.

• Some utilities have been switched to the snap format for new installs (Calculator, Characters, Logs, and System Monitor). Snap apps provide better isolation which allows them to be upgraded to new stable releases during the LTS lifecycle.

• The Characters app replaces the older Character Map by default.

• The Ubuntu Software app allows easy switching between different channels for Snap apps.

• The To Do app has been added to the default normal install.

• spice-vdagent is pre-installed for better performance for Spice clients such as the GNOME Boxes app.

• The right-click method for touchpads without physical buttons has changed to a two-finger click instead of clicking in the bottom right of the touchpad. You can use the GNOME Tweaks app (not installed by default) to change this setting.

• Although libinput is the default driver for mice and touchpads, it is now possible to use the synaptics driver with the Settings app. Support for the synaptics driver will be dropped in a future Ubuntu release.

• Computers will automatically suspend after 20 minutes of inactivity while on battery power.

• GNOME Shell now supports Thunderbolt 3.

Other highlights since 16.04 LTS

• 32-bit installer images are no longer provided for Ubuntu Desktop.

• The Ubuntu Desktop now uses GNOME instead of Unity.

• GDM has replaced LightDM as the default display manager. The login screen now uses virtual terminal 1 instead of virtual terminal 7.

• Window control buttons are back on the right.

• Driverless printing support is now available.

• GNOME's built-in screen keyboard is used instead of Onboard.

• Calendar has a Week View and supports recurring events.

• These apps have received major user interface redesigns: Disk Usage Analyzer, Files (nautilus), Remmina, Settings, and Ubuntu Software.

• System Log has been replaced by Logs, an app to view logs from the systemd journal.

• Many GNOME apps now have a Keyboard Shortcuts popup available in the app menu.

• gconf is no longer installed by default since it has long been superseded by gsettings. Note that statistics and preferences for the Aisleriot card games will be reset when upgrading from 16.04 LTS or 16.10. gconf will be removed from the Ubuntu package archives in a future Ubuntu release.

• The Ubuntu GNOME flavor has been discontinued. If you are using Ubuntu GNOME, you will be upgraded to Ubuntu. Choose the Ubuntu session from the cog on the login screen if you would like the default Ubuntu experience.

• Install gnome-session then restart your computer and choose GNOME (or GNOME on Wayland) from the cog on the login screen if you would like to try a more upstream version of GNOME. If you'd like to also install more core apps, install the vanilla-gnome-desktop metapackage.

Ubuntu Server

New since 17.10

Server installer

The next generation Subiquity server installer, brings the comfortable live session and speedy install of Ubuntu Desktop to server users at last.

N.B., If you require LVM, RAID, multipath, vlans, bonds, or the ability to re-using existing partitions, you will want to continue to use the alternate installer which can be downloaded from http://cdimage.ubuntu.com/releases/18.04/beta-2/

netplan.io

ifupdown has been deprecated in favor of netplan.io and is no longer present on new installs. Backend configuration on Ubuntu Server by default is provided by systemd-networkd.

Please see the Network configuration section of these release notes.

LXD 3.0

LXD is the system container manager that ships with all Ubuntu servers.

Ubuntu 18.04 includes the all new LXD 3.0 release, some of the highlights include:

• Clustering of LXD servers (one big virtual LXD)
• Support for NVIDIA runtime passthrough
• Remote transfer of custom storage volumes
• Extended /dev/lxd API inside the containers
• Support for port redirection
• Numerous improvements to the command line tools

To learn more about LXD 3.0, read the release announcement.

A new external tool called lxd-p2c is also available to turn existing systems into LXD containers.

QEMU 2.11.1

QEMU has been updated to the 2.11.1 release.

See the Changelog for major changes since Artful.

Among many other changes, fixes around Meltdown/Spectre are included. Since fully utilizing these mitigations needs more than just an upgrade, it is recommended to read details at the qemu.org blog post.

QEMU in Ubuntu 18.04 now has rdma support enabled as over the past year much unification in the rdma-core project has occured.

Migrations from former versions are supported just as usual. When upgrading it is always recommended to upgrade the machine types allowing guests to fully benefit from all the improvements and fixes of the most recent version.

libvirt 4.0

libvirt has been updated to version 4.0. See the upstream Changelogs for details since version 3.6 that was in Artful.

The packaging now builds libvirt storage drivers as pluggable libraries. This slims down the installation requirements but some drivers of less general interest will now be found in universe. (ex: gluster, sheepdog, zfs)

DPDK 17.11.x

Ubuntu includes 17.11.x the latest stable release branch of DPDK.

See the Release Notes for details.

By the new Stable Release exception for DPDK future stable updates to 17.11.x will be made available to Ubuntu 18.04

Open vSwitch 2.9

Open vSwitch has been updated to 2.9.

Please read the release notes for more detail.

Chrony

In Ubuntu 18.04 chrony will replace ntpd as the recommended server for the NTP protocol. See the upstream changelog for an overview of recent changes as well as the FAQ which will help for smooth conversions from NTP.

The comparison among ntp servers by the chrony maintainers may interest some users looking to see a high level reason why this change was made. It does lack the rather new and not yet completely ready ntpsec, but otherwise is a fair analysis.

For simple time sync needs the base system already comes with systemd-timesyncd. Chrony is only needed to act as a time server or if you want the advertised more accurate and efficient syncing.

Going along with this change, ntpd has been demoted from main to universe. ntpd will continue to work but will only receive best-effort security maintenance. When upgrading to Ubuntu 18.04 it is highly recommended to migrate to chrony if you had set up ntpd before.

cloud-init

The version was updated to 18.2. Notable new features include:

• VMware: support for 64-bit platforms and identifying OVF datasource provided
• GCE: Improvements and changes to ssh key behavior for default user.
• Azure pre-provisioning speed improvements
• NoCloudKVM and EC2 tests now run in continuous integration.

• New cloud support: IBMCloud and HetznerCloud now have official datasources and OpenTelekom is now recognized by cloud-id

• OpenNebula: Improve network configuration support.

• New cloud-init command-line tools available: status, analyze and clean
• New ubuntu cloud-config modules for managing snaps and ubuntu-advantage services

curtin

The version was updated to 18.1. Notable features include:

• Add experimental zpool and zfs filesystem support, including ZFS on root.

• Add support for installing remote sources that are a filesystem image. (1746348)

• Add pollinate user-agent configuration support.

• Improved device teardown of dirty devices to support re-deployment (1750519) (1743643) (1713537) (1722322) (1708052) (1718699)

• Default config now automatically tars curtin logs upon error using new curtin collect-logs command.

• storage: accept filesystem mount options (1709284)

• Extensive integration test coverage and improvements.

MAAS

The version was updated to 2.4b2. Notable features include:

• Add audit logging
• Add KVM pod support to create tags, select the storage pool, and compose machines with multiple storage pools.
• Add UI for DNS management
• Add the commissioning template framework for HBA management.
• Add the commissioning template framework for Firmware Upgrades.
• Improve UI performance by performance.
• Improve MAAS' backend performance and
• Improve the UI for the Settings.
• Add experimental support to configure zfs as the root filesystem.
• Switch to use Chrony instead of ntp.

For more information, please visit the MAAS website or review the 2.4 Release Notes.

SSSD

SSSD was updated to version 1.16.x and its secrets service is now enabled. Previously it was disabled because it required the http-parser library which lived in Universe, but a successful MIR brought it to main so SSSD could link with it.

Nginx

nginx was updated to version 1.14.0. New features include the mirror module, HTTP/2 push, and the gRPC proxy module.

PHP

PHP was updated to version 7.2.x. For upstream guidance on migrating from PHP 7.1 (Artful's version) to 7.2: http://php.net/manual/en/migration72.php. Also of relevance might be the 7.0 to 7.1 migration documentation: http://php.net/manual/en/migration71.php.

Apache

Apache was updated to version 2.4.29. Additionally, HTTP/2 support is now enabled in 18.04.

landscape-client

landscape-client has been ported to Python 3 and is now available to install on the default image.

ubuntu-advantage-tools

s390x-specific enhancements (since 17.10)

• improvements for IBM z14,z14 ZR1,LinuxONE Rockhopper II and LinuxONE Emporer II (1725260) (1736100)

• s390-tools major version upgrade to v2.2.0 (1735447)

• cryptsetup rebase and enhancements in support of dm-crypt (1724592)

• protected key support for dm-crypt (1741904)

• TLB enhancements (1732426) (1732452)

• TOD-Clock Epoch Extension Support (1732437) (1732691)

• DASD multi-queue (1732446) support and block layer discard support (1732440)

• Improved memory handling (1734120)

• support for new crypto hardware CEX6S (1735437)

• AP bus kernel API for KVM (1732449)

• CPU-MF/perf improvement (1735433)

• CPACF enhancements and acceleration for AES GCM (1735438) (1743750)

• HiperSocket connections enhacements (1735695)

• parted update for fdasd/vtoc (1737144)

• openssl-ibmca rebase (1747626)

• opencryptoki rebase for EP11 and ECC enhancement (1751272)

• lock optimization enhancement (1747877)

• libica upgrade for z14 and ECC support (1737159) and to use PRNO-TRNG to seed SHA512-DRBG (1754617)

• auto detect layer2 setting in qeth driver (1747639)

• Kernel support for STHYI/LPAR (1736093)

• rebase libpfm4 for z13/z13s CPU-MF hardware counters (1741905)

OpenStack Queens

Ubuntu 18.04 includes the latest OpenStack release, Queens, including the following components:

• OpenStack Identity - Keystone

• OpenStack Imaging - Glance

• OpenStack Block Storage - Cinder

• OpenStack Compute - Nova

• OpenStack Networking - Neutron

• OpenStack Telemetry - Ceilometer, Aodh, Gnocchi, and Panko

• OpenStack Orchestration - Heat

• OpenStack Dashboard - Horizon

• OpenStack Object Storage - Swift

• OpenStack Database as a Service - Trove

• OpenStack DNS - Designate

• OpenStack Bare-metal - Ironic

• OpenStack Filesystem - Manila

• OpenStack Key Manager - Barbican

Please refer to the OpenStack Queens release notes for full details of this release of OpenStack.

OpenStack Queens is also provided via the Ubuntu Cloud Archive for OpenStack Queens for Ubuntu 16.04 LTS users.

WARNING: Upgrading an OpenStack deployment is a non-trivial process and care should be taken to plan and test upgrade procedures which will be specific to each OpenStack deployment.

Make sure you read the OpenStack Charm Release Notes for more information about how to deploy Ubuntu OpenStack using Juju.

As is to be expected, with any release, there are some significant known bugs that users may run into with this release of Ubuntu 18.04. The ones we know about at this point (and some of the workarounds), are documented here so you don't need to spend time reporting these bugs again:

Desktop

• The computer suspends after 20 minutes of inactivity on battery power even if a user is logged in remotely. (GNOME:gnome-control-center#22)

• Bluetooth audio devices cannot be used in the Greeter. This will cause issues for people using the accessibility features such as screenreaders at the login screen. Once logged in everything should work as expected.

• Some admin utilities will not work with GNOME on Wayland since the apps have not been adapted to use PolicyKit to only use admin privileges for the specific functions needed. Also, some screenshot and screencast apps and all remote desktop server apps do not currently work on GNOME on Wayland. As a workaround, you can use the default Ubuntu session.

• Exiting the live session may get stuck with a "A start job is running for " error. You may need to forcefully power off the computer if you see this. (1706939)

• The Dock and Appindicator system extensions appear to be Off in tools like GNOME Tweaks. (They are on but cannot be disabled because they are system extensions for the Ubuntu session.) (1718850)

• Tracker is not installed by default. When installed, you must log out and log back in for the tracker service to start (1697769)

• Systems may fail to boot when connected over DisplayPort to an external screen, on NVidia graphics hardware such as the GTX970 chipset. (1723619)

• When an external monitor is connected to a laptop, the login screen is only displayed on the internal one and in some case is not visible (1723025)

• The warning dialog when a user force a UEFI installation does not respond to input event and the installation is then blocked at this stage (1724482) Avoid yourself some troubles and do not force a UEFI installation without a UEFI partition, grub-installer will fail anyway.

• Doing an "Entire disk" installation over an existing LVM installation will fail because the installer selects the wrong boot device (1724417) Use custom partitioning instead and manually select the right boot device in the combo box.

• The Files app remains at 3.26.

• Upgrading via the installer (Ubiquity) is deemed not safe due to bugs in apt-clone and so is no longer supported. (1756862) UIFE - remove ubiquity upgrade option.

• Setting a ulimit may cause segfaults in certain applications, especially those using webkit2gtk. Disabling the ulimit should restore normal functionality. More information in this Debian news entry: https://salsa.debian.org/webkit-team/webkit/blob/wk2/unstable/debian/NEWS

• Occasionally login may hang after an incorrect password (1766137). A workaround is to click cancel, click on your user and try again.

Server

• Partitioning step allows to configure LVM across multiple devices without requiring to setup a separate /boot partition. This may lead to failure to install the bootloader at the end of the installation, and failures to boot the resultant installations. (1680101)

• LVM configuration cannot be removed when volume groups with the same name are found during installation. Partitioner does not support installation when multiple conflicting/identical volume groups have been detected. For example reinstalling Ubuntu with LVM across multiple disk drives that had individual LVM installations of Ubuntu. As a workaround, please format disk drives prior to installation, or from the built in shell provided in the installer. (1679184)

• cio_ignore blacklist is no longer active after installation, because not all install-time parameters, like cio_ignore (s390x), are propagated to the installed system. Workaround is to edit /etc/zipl.conf to apply these and re-run sudo zipl to update the IPL. (1571561)

• Importing ssh keys from GitHub or Launchpad during the installation with the new subiquity server installer currently won't work. (#1766980)

The release notes for the official flavors can be found at the following links:

Reporting bugs

Your comments, bug reports, patches and suggestions will help fix bugs and improve the quality of future releases. Please report bugs using the tools provided.

If you want to help out with bugs, the Bug Squad is always looking for help.

Participate in Ubuntu

If you would like to help shape Ubuntu, take a look at the list of ways you can participate at

More about Ubuntu

You can find out more about Ubuntu on the Ubuntu website and Ubuntu wiki.

To sign up for future Ubuntu development announcements, please subscribe to Ubuntu's development announcement list at:

For the first time, scientists have identified the existence of a new DNA structure never before seen in living cells.

The discovery of what's described as a 'twisted knot' of DNA in living cells confirms our complex genetic code is crafted with more intricate symmetry than just the double helix structure everybody associates with DNA – and the forms these molecular variants take affect how our biology functions.

"When most of us think of DNA, we think of the double helix," says antibody therapeutics researcher Daniel Christ from the Garvan Institute of Medical Research in Australia.

"This new research reminds us that totally different DNA structures exist – and could well be important for our cells."

The new DNA component the team identified is called the intercalated motif (i-motif) structure, which was first discovered by researchers in the 1990s, but up until now had only ever been witnessed in vitro, not in living cells.

Now, thanks to Christ's team, we know the i-motif occurs naturally in human cells, meaning the structure's significance to cell biology – which has previously been called into question, given it had only been demonstrated in the lab – demands new attention from researchers.

(Zeraati et al., Nat Chem, 2018)

If your only familiarity with DNA shapes is the dual helical spirals made famous by Watson and Crick, the configuration of the intercalated motif could come as a surprise.

"The i-motif is a four-stranded 'knot' of DNA," explains genomicist Marcel Dinger, who co-led the research.

"In the knot structure, C [cytosine] letters on the same strand of DNA bind to each other – so this is very different from a double helix, where 'letters' on opposite strands recognise each other, and where Cs bind to Gs [guanines]."

According to Garvan's Mahdi Zeraati, the first author of the new study, the i-motif is only one of a number of DNA structures that don't take the double helix form – including A-DNA, Z-DNA, triplex DNA and Cruciform DNA – and which could also exist in our cells.

Another kind of DNA structure, called G-quadruplex (G4) DNA, was first visualised by researchers in human cells in 2013, who made use of an engineered antibody to reveal the G4 within cells.

In the new study, Zeraati and fellow researchers employed the same kind of technique, developing an antibody fragment (called iMab) that could specifically recognise and bind to i-motifs.

In doing so, it highlighted their location in the cell with an immunofluorescent glow.

Artist's impression, overlaid on imaging of the iMab antibody (green) in the nuclei of cells (Chris Hammang)

"What excited us most is that we could see the green spots – the i-motifs – appearing and disappearing over time, so we know that they are forming, dissolving and forming again," says Zeraati.

While there's still a lot to learn about how the i-motif structure functions, the findings indicate that transient i-motifs generally form late in a cell's 'life cycle' – specifically called the late G1 phase, when DNA is being actively 'read'.

The i-motifs also tend to appear in what are known as 'promoter' regions– areas of DNA that control whether genes are switched on or off – and in telomeres, genetic markers associated with ageing.

"We think the coming and going of the i-motifs is a clue to what they do," says Zeraati.

"It seems likely that they are there to help switch genes on or off, and to affect whether a gene is actively read or not."

Now that we definitively know this new form of DNA exists in cells, it'll give researchers a mandate to figure out just what these structures are doing inside our bodies.

As Zeraati explains, the answers could be really important – not just for the i-motif, but for A-DNA, Z-DNA, triplex DNA, and cruciform DNA too.

"These alternative DNA conformations might be important for proteins in the cell to recognise their cognate DNA sequence and exert their regulatory functions," Zeraati explained to ScienceAlert.

"Therefore, the formation of these structures might be of utmost importance for the cell to function normally. And, any aberration in these structures might have pathological consequences."

The findings are reported in Nature Chemistry.

Two more DRAM makers based in China, Innotron Memory and Fujian Jin Hua Integrated Circuit, are gearing up for volume production of computer memory in the coming month. Both manufacturers were founded with the help of the Chinese government, their output will initially be consumed locally.

Several years ago, the Chinese government announced plans to invest billions of U.S. dollars in the local semiconductor and adjacent high-tech industries and support appropriate companies directly and indirectly. Since the inception of the so-called Big Fund in 2014, not only over tens of billion were poured into various companies and initiatives (the plan is to invest $150 billion in the coming years), but the first actual results of their operations have started to show up. One of the industries that is developing fast in China is DRAM manufacturing. Earlier this year we reported about Xi'an UniIC Semiconductors, which started to sell its DDR4 chips developed and produced in-house. Apparently, there are at least two more DRAM makers in China ready to start shipments of their memory modules in the second half of 2018 and the first half of 2019: Fujian Jin Hua Integrated Circuit (JHICC) and Innotron Memory.

Fujian Jin Hua Integrated Circuit completed construction of its 300-mm fab in November and started to move in production tools late in 2017. According to a media report, the equipment move-in will be completed by early July and the first phase of the fab will be able to start volume production of DRAM in the third quarter. JHICC reportedly partnered with Taiwan-based UMC to develop its 22 nm DRAM manufacturing technologies, but at this point it is unclear which types of memory the company is gearing up to make. Some previous reports indicated that JHICC was looking at various specialized DRAM products (namely LPDDR4), but the actual plans might be different.

JHICC has already received the first phase of investment totaling $5.65 billion from the local authorities in Fujian, China. It is expected that aggregate investments in the Jin Hua DRAM fab will total $8 billion in the coming years. Manufacturing capacity of the JHICC’s memory fab is unknown, but the facility looks rather big on the picture.

Another China-based company that is getting ready to make DDR4 DRAM devices using its 22 nm fabrication process in the coming months is Innotron Memory (previously known as Hefei ChangXin and Hefei RuiLi). Innotron completed construction of its 300-mm fab in January and then started to move in production equipment. The company intends to start trial production of 8 Gb DRAM chips in late 2018 and then initiate volume production of these ICs sometimes in early 2019.

Innotron’s fab is not going to be very large: its initial capacity will be around 20,000 wafer starts per month in 2019, so in terms of volume it is not going to be a competitor for leading DRAM makers globally. Meanwhile, the firm intends to start building up the second phase of the fab in 2020 to boost its capacity. Furthermore, Innotron plans to complete development of its 17 nm process technology by 2021, the company announced recently.

Considering the facts that Innotron, Jin Hua, and Xi’an UniIC all had to start development of DRAM manufacturing technologies essentially from scratch, and their production capacities are not high, they are not expected to become viable competitors for companies like Micron, Samsung, and SK Hynix in terms of volumes, production efficiencies, or performance any time soon. In the meantime, there are more fabs incoming: in the recent years Tsinghua disclosed plans to invest $24 billion and $30 billion in memory chip production facilities near Wuhan and Nanjing, respectively. Furthermore, leading chip makers (Intel, Samsung, SK Hynix, etc.) already operate 3D NAND and DRAM fabs in China and invest in them to increase their capacities.

In the end, Chinese electronics industry will decrease their reliance on foreign volatile and non-volatile memory chips. Furthermore, it should be noted that it's questionable whether Chinese memory producers have all the patent/technology licenses they need to sell their products outside of China. Or for that matter with transnational semiconductor giants investing billions of U.S. dollars in China, whether they will be able to legally prohibit Chinese companies from building their presence on the foreign markets without facing problems from the Chinese authorities. The good news for the consumer is that with more 3D NAND and DRAM production in various parts of the world pricing of appropriate chips is going to decrease. The bad news is that without proper return-on-investments the progress of technologies may slow down.

Memory production is the first step towards establishment of a full-fledged self-sufficient semiconductor industry in China. Making memory ICs is relatively easy if you have appropriate process technologies, fabs, and clients to buy output. Designing competitive high-performance processors requires access to technology expertise and appropriate technology licenses from various companies. To boost its local SoC/CPU design industry, Tsinghua Unigroup acquired Spreadtrum and RDA in 2013 (which have a license to build mobile SoCs based on select Intel’s x86 cores), then signed an agreement with Intel to co-develop semi-custom server solutions. Meanwhile, Tianjin Haiguang Advanced Technology Investment Co., Ltd (THATIC) formed an x86 joint venture with AMD in 2016.

Related Reading:

Sources: DigiTimes (1, 2), EETimes, SemiEngineering, ElectronicDesign, TrendForce

This continues a translation of various parts of Lisp in Small Pieces into Clojure. For earlier chapters, see:

This time round I won’t say much about the contents of the chapter. The translation is an accompaniment to the book and the blog post is in no way a substitute. Buy the book if you don’t already have it!

Chapter 3

Chapter three ofLiSP begins by reviewing a range of control structures, historical and modern, that incorporate the notion of an escape - a transfer of control that is non-local but nevertheless less powerful than arbitrary ‘goto’. Unlike ‘goto’, you can only escape back to places you’ve already been.

• prog, return
• catch, throw
• block, return-from

These can be used for optimisation, for instance, for shortcutting out of a deeply nested search procedure or for error handling as per the near-ubiquitous try / catch constructs in popular OO languages.

Each pair provides a means of marking a continuation at which control will resume and a means of transferring control to that continuation.

In addition the book covers unwind-protect, a form that interacts with the various other escapes to provide an analogue of the familiarfinally construct, ensuring that a body of code is run when control escapes a block regardless of how the control escapes.

The notion of continuation that is gestured at by these control structures is famously provided as a first class object inScheme. call-with-current-continuation, a.k.a. call-cc, enables a continuation to be identified, named and stored (with indefinite extent) for later use.

Using call/cc, any of the other control flow primitives can be simulated.

Furthermore, by reifying this notion of continuation in our interpreter, using either the facilities in the underlying lisp (call/cc) or via a translation to continuation passing style (“CPS”), any of these control structures can be provided as primitives in our daughter lisp.

Chapter three describes both these methods although most of it is devoted to the strategy we need in Clojure - CPS.

The Translation

My translation is here.

It sticks closely to the strategy in the book with a few renamings here and there. It’s pretty generously commented so inspect the source for more detail.

Instead of directly mirroring the object orientation approach in the book, I’ve eradicated the few non-essential uses of implementation inheritance in the book (e.g. full-environment inheriting null-environment…) in favour of a more idiomatic protocol / record representation in Clojure. Further discussion of this below.

I’ve implemented call/cc, block / return-from and catch /throw. I’ve left out unwind-protect for now at least, despite its obvious utility. As our evaluator still lacks even such creature-comforts as an extensible global namespace, it’s hardly an pressing issue.

A Note on Object Modelling

Over years of dealing with Java and C++ and some pretty large codebases, I’ve become extremely wary of implementation inheritance. Not dogmatically - any language feature is fair game to developers striving for the simplest and cleanest expression of their intentions - but enough to acquire a firm conviction that inheritance is heavily overused in both ecosystems.

While deep inheritance hierarchies may seem like a great way of modelling your concepts when you have a blank slate, it’s a recipe for some extremely tight and non-obvious coupling that can deadlock refactoring attempts in later phases once those concepts have shifted and the original model is no longer a good fit. The problems can be particularly acute when the inheritance hierarchy has spread across several semi-autonomous modules or (worse) long-lived code branches.

Dynamic languages and languages with superior type inference can alleviate some of these difficulties - you can mitigate coupling by leaving things unsaid or allowing some flexibility in the meaning of what you have said. And various approaches to mixins and traits have been conceived in an attempt to reconcile the fidelity of modelling with evolvability.

Nevertheless, part of the problem is simply inherent. It seems that the more intricate a representation of a conceptual hierarchy, the harder it is to change. The issue is compounded by the scant effort that developers normally take to control the API they present to subclasses (the protected methods) and to take care of method visibility in general.

So in Java, I would generally start with a flat duality of interface and implementation these days, using composition and other approaches where possible and only indulging in implementation inheritance in cases that seem particularly benign.

Happily, this is the basic paradigm that Clojure’s protocols encourage too. However, in translating chapter three’s evaluator into Clojure I don’t think it’s appropriate to go to protocols / records in all cases.

The lisp implementation presented arranges matters like this:

(define-classcontinuationObject(k))(define-genericresume(kcontinuation)v)(define-classif-continuationcontinuation(etefr)(define-classbegin-continuationcontinuation(e*r)(define-genericinvoke(f)v*rk)

define-class sets up a true implementation inheritance relationship here. The definition of continuation contributes a k field to all subclasses and automatically defines accessor methods continuation-k and set-continuation-k! which are available on all subclasses.

This would correspond roughly to the following Java - where the generic functions have become methods defined inline within the class definitions.

/** * Unified call interface for functions, primitives and * continuations. */publicinterfaceInvokable{voidinvoke(Object[]vals,Environmentr,Continuationk);}publicclassContinuationimplementsInvokeable{privatefinalContinuationk;publicContinuation(Continuationk){this.k=k;}publicContinuationgetK(){returnk;}// Default, intended to be overriddenpublicvoidresume(Objectvalue){k.resume(value);}// This is a case of API adaptation, *not* to be overriddenpublicfinalvoidinvoke(Object[]vals,Environmentr,Continuationk){resume(vals[0])}}publicclassIfContinuationextendsContinuation{privatefinalFormet;privatefinalFormef;privatefinalEnvironmentr;publicContinuation(Formet,Formef,Environmentr){this.et=et;this.ef=ef;this.r=r;}publicFormgetET(){returnet;}publicFormgetEF(){returnef;}publicEnvironmentgetR(){returnr;}// override resume for custom behaviourpublicvoidresume(Objectvalue){...}}publicclassBeginContinuationextendsContinuation{...}

This illustrates three subtly different uses of implementation inheritance:

• getK - which provides access to the inner continuation, it would be dangerous for subclasses to override or alter this behaviour, they would very likely violate assumptions made in the class or elsewhere
• resume - which merely provides a sensible default that almost no subclasses will need
• invoke - which adapts the Continuation#resume API to a more generalised invoke API. This adaptation involves only the Invokable and Continuation abstractions and it would be incorrect to even want to override it elsewhere.

It’s tempting perhaps to define a Resumable interface and work where possible at a higher level of abstraction but the parts of the evaluate which work with continuations, the implementations of block and catch for instance, need to follow the chain of wrapped continuations via getK so the going the extra mail with interface segregation doesn’t buy us much in this case.

Both object systems provide for method implementation at the level of an intermediate base class, even if the Java approach is not as open to future extension.

By contrast, Clojure protocols and records do not provide for intermediate base classes at all. So a different approach again is appropriate:

(defprotocol Continuation(resume[selfv]))(defrecord IfContinuation[ketefr]Continuation(resume[selfv]...))(defrecord BeginContinuation[ke*r]Continuation(resume[selfv]...))(defmulti invoke(fn [fv*rk]))(defmethod invokeContinuation[fv*rk]...)

Each continuation is a separate record type and does not share a common implementation base class. Therefore each redefines the wrapped continuation, k.

From other code, access to the wrapped continuation is available via(:k cont) and available only because each and every continuation defines k. A more careful implementation might define a protocol for access to this information but in the spirit of simplicity and a preference for data over code (which we like in the Clojure world) a better implementation still would probably just rename the field to something sensible and leave it accessible to the full range of built in functions. k was chosen merely to stick close to the book’s implementation.

The Continuation protocol is actually the Resumable interface we considered earlier. No default implementation of resume is defined.

Invokeable is replaced by a multimethod which is a) closer to the original implementation and b) allows this sort of adaptation of at the abstraction level quite easily.

Despite protocols “solving the expression problem” for Clojure by allowing extension to arbitrary predefined types, they do not solve in and of themselves solve the further problem of protocol adaptation.

If we had realised Invokeable as a protocol we would then have had to extend Invokeable to each and every implementation ofContinuation to effect the adaptation. Or extend the protocol toObject and manage our own type-based method dispatch at that level.

In some circumstances extending protocols to an exhaustive enumeration of implementations might be reasonable (seeBlockLookup in the translation for instance) but in the case of protocol adaptation it is clearly not reasonable.

Other approaches exist (seeclojure protocol adapters for instance) but the multimethod approach is simple, flexible and powerful.

The fiddly bit though is the interaction if isa? which multimethod use to resolve method dispatch and the protocol extension relationship. If you’re dispatching on #(type %) you need to be very careful that you’re referring to the interface generated by the protocol rather than the protocol itself:

;; Continuation is a var reference the protocol, defined with defprotocol(isa?(type(BeginContinuation.nilnilnil))Continuation);; => false;; lispic.chapter3.cont.Continuation is the fully qualified class;; name of the corresponding interface(isa?(type(BeginContinuation.nilnilnil))lispic.chapter3.cont.Continuation);; => true

Other Stuff

There’s an extremely rich literature on continuations out there that I couldn’t even begin to cover. The book itself discusses delimited or composable continuations briefly and there are various approaches to these. See David Nolen’s delimc for an exprimental implementation in Clojure and a set of pointers for further reading.

Monadic implementations of continuations are also available intwoof the prominent Clojure monad libraries though monads and category theory are in the main orthogonal to the concerns of the book.

Author

Sean

Overview

This HOWTO describes a web server written for the day when even Yaws is not quick enough.

The web server presented is quite simple. Even so it is split into 5 modules. Some of these are dictated by the OTP framework, and others are split out for convenience. The 5 modules are:

• iserve - API for managing URIs and callbacks
• iserve_app - OTP Application behaviour
• iserve_sup - OTP Supervisor
• iserve_server - Gen_server to own the listening socket and create connections
• iserve_socket - Process to handle a single HTTP connection for its lifetime

This HOWTO presents code and descriptions for each of these as they arise.

TCP Server Framework

A web server needs to support lots of connections, so at it's heart it needs to be a multiple connection TCP/IP server. There are any number of ways to arrange a set of erlang processes into such a thing. My favourite method is to have a single gen_server which opens and owns the listen socket (the listening process). This spawns another process which waits in accept until a connection attempt is received. At this time this accepting process sends a message back to the listening process and goes on to handle the traffic. This avoids the need for gen_tcp:controlling_process/2 and associated complexity.

On receipt of the message from the accepting process the listening process spawns a new accepting process and so on.

The listening process also traps exits, and if it receives a non normal exit from the current accepting process it creates a new one. In this way the listening process supervises its acceptor.

The web server creates a #req{} record as it processes each request. This is used as part of the API into implementation callbacks and by the iserve_socket process. Here are the contents of iserve.hrl up front to get it out of the way:

% This record characterises the connection from the browser to our server
% it is intended to be a consistent view derived from a bunch of different headers
-record(req, {connection=keep_alive,	        % keep_alive | close
	      content_length,                   % Integer
	      vsn,                              % {Maj,Min}
	      method,                           % 'GET'|'POST'
	      uri,				% Truncated URI /index.html
	      args="",                          % Part of URI after ?
	      headers,				% [{Tag, Val}]
	      body = <<>>}).			% Content Body

Listening Process

Here is the code for the listening process. It is a very basic gen_server which models a single process:

-module(iserve_server).

-behaviour(gen_server).

-export([start_link/1, create/2]).

%% gen_server callbacks
-export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2,
         code_change/3]).

-record(state, {listen_socket,
                port,
                acceptor}).

%%--------------------------------------------------------------------
start_link(Port) when is_integer(Port) ->
    Name = list_to_atom(lists:flatten(io_lib:format("iserve_~w", [Port]))),
    gen_server:start_link({local, Name}, ?MODULE, Port, []).

%% Send message to cause a new acceptor to be created
create(ServerPid, Pid) ->
    gen_server:cast(ServerPid, {create, Pid}).


%% Called by gen_server framework at process startup. Create listening socket
init(Port) ->
    process_flag(trap_exit, true),
    case gen_tcp:listen(Port,[binary, {packet, http},
                              {reuseaddr, true},
                              {active, false},
                              {backlog, 30}]) of
	{ok, Listen_socket} -> %%Create first accepting process
	    Pid = iserve_socket:start_link(self(), Listen_socket, Port),
	    {ok, #state{listen_socket = Listen_socket,
                        port = Port,
			acceptor = Pid}};
	{error, Reason} ->
	    {stop, Reason}
    end.


handle_call(_Request, _From, State) ->
    Reply = ok,
    {reply, Reply, State}.

%% Called by gen_server framework when the cast message from create/2 is received
handle_cast({create, _Pid}, #state{listen_socket = Listen_socket} = State) ->
    New_pid = iserve_socket:start_link(self(), Listen_socket, State#state.port),
    {noreply, State#state{acceptor=New_pid}};

handle_cast(_Msg, State) ->
    {noreply, State}.


handle_info({'EXIT', Pid, normal}, #state{acceptor=Pid} = State) ->
    {noreply, State};

%% The current acceptor has died, wait a little and try again
handle_info({'EXIT', Pid, _Abnormal}, #state{acceptor=Pid} = State) ->
    timer:sleep(2000),
    iserve_socket:start_link(self(), State#state.listen_socket, State#state.port),
    {noreply, State};

handle_info(_Info, State) ->
    {noreply, State}.


terminate(Reason, State) ->
    gen_tcp:close(State#state.listen_socket),
    ok.


code_change(_OldVsn, State, _Extra) ->
    {ok, State}.

The notable thing about this code is the use of undocumented socket options to set up the initial state of connections made to the web server port.

• {backlog, 30} specifies the length of the OS accept queue.
• {packet, http} puts the socket into http mode. This makes the socket wait for a HTTP Request line, and if this is received to immediately switch to receiving HTTP header lines. The socket stays in header mode until the end of header marker is received (CR,NL,CR,NL), at which time it goes back to wait for a following HTTP Request line.

Acceptor/Socket process

It would be easy enough to create an abstraction of the Listen/Accept process structure and pass in the implementation function as another parameter. For this HOWTO however I'll stick with the most basic model - the acceptor process starts life as an acceptor and goes on to handle the traffic.

The acceptor process is implemented in a separate module iserve_socket. It is in two parts - the first part sets up a bunch of defines and exports and then does the accepting. Here is it is:

-module(iserve_socket).

-export([start_link/3]).

-export([init/1]).
-include("iserve.hrl").

-define(not_implemented_501, "HTTP/1.1 501 Not Implemented\r\n\r\n").
-define(forbidden_403, "HTTP/1.1 403 Forbidden\r\n\r\n").
-define(not_found_404, "HTTP/1.1 404 Not Found\r\n\r\n").

-record(c,  {sock,
             port,
             peer_addr,
             peer_port
	     }).

-define(server_idle_timeout, 30*1000).

start_link(ListenPid, ListenSocket, ListenPort) ->
    proc_lib:spawn_link(?MODULE, init, [{ListenPid, ListenSocket, ListenPort}]).

init({Listen_pid, Listen_socket, ListenPort}) ->
    case catch gen_tcp:accept(Listen_socket) of
	{ok, Socket} -> %% Send the cast message to the listener process to create a new acceptor
	    iserve_server:create(Listen_pid, self()),
	    {ok, {Addr, Port}} = inet:peername(Socket),
            C = #c{sock = Socket,
                   port = ListenPort,
                   peer_addr = Addr,
                   peer_port = Port},
	    request(C, #req{}); %% Jump to state 'request'
	Else ->
	    error_logger:error_report([{application, iserve},
				       "Accept failed error",
				       io_lib:format("~p",[Else])]),
	    exit({error, accept_failed})
    end.

Note here that the process is started via the proc_lib:spawn_link/3 call. This wraps the normal spawn_link/3 bif so that the same nice error reports are created as for gen_servers, but it allows for a totally unstructured process implementation.

Web server state machine

The rest of this module contains the web server code. It is structured as a state machine which follows the state changes of the http socket mode. A single function models each state, and state transitions are simply implemented as a call to the function which owns the next state.

The states are:

• request - wait for a HTTP Request line. Transition to state headers if one is received.
• headers - collect HTTP headers. After the end of header marker transition to body state.
• body - collect the body of the HTTP request if there is one, and lookup and call the implementation callback. Depending on whether the request is persistent transition back to state request to await the next request or exit.

The code for the state request is below. A blocking call is made to gen_tcp:recv/3 with a timeout. The http driver waits for a CRNL terminated line of the form GET / HTTP/1.0. If anything else is received an http_error indication is returned with the erroneous data.

Some broken clients include extra CR or CRNL sequences so these are skipped.

request(C, Req) ->
    case gen_tcp:recv(C#c.sock, 0, 30000) of
        {ok, {http_request, Method, Path, Version}} ->
            headers(C, Req#req{vsn = Version,
                               method = Method,
                               uri = Path}, []);
        {error, {http_error, "\r\n"}} ->
	    request(C, Req);
	{error, {http_error, "\n"}} ->
            request(C, Req);
	_Other ->
	    exit(normal)
    end.

The code for the state headers is below. After sending the HTTP request line the http driver automatically switches into header receive mode. The driver looks for values of the form Header-Val: value and sends them one by one after each call to recv.

The driver maintains a hash table of well known header values and if one of those is received from the network it returns the header value as an atom. Otherwise the header value is returned as a string. In both cases the driver takes care of case insensitivity and automatically capitalises the first letter of each hyphen separated word in the header name. The author clearly got a little carried away at this point!

This web server extracts the values of the 'Content-Length' and 'Connection' headers for its own purposes and simply accumulates the other headers in a list to be passed to the application callback.

At the end of the headers the driver returns {ok, http_eoh}. This is the cue for the web server to skip to body mode. The driver automatically switches to wait for a new request line at this point unless a subsequent call to inet:setops/2 is made.

headers(C, Req, H) ->
    case gen_tcp:recv(C#c.sock, 0, ?server_idle_timeout) of
        {ok, {http_header, _, 'Content-Length', _, Val}} ->
            Len = list_to_integer(Val),
            headers(C, Req#req{content_length = Len}, [{'Content-Length', Len}|H]);
        {ok, {http_header, _, 'Connection', _, Val}} ->
            Keep_alive = keep_alive(Req#req.vsn, Val),
            headers(C, Req#req{connection = Keep_alive}, [{'Connection', Val}|H]);
        {ok, {http_header, _, Header, _, Val}} ->
            headers(C, Req, [{Header, Val}|H]);
        {error, {http_error, "\r\n"}} ->
	    headers(C, Req, H);
	{error, {http_error, "\n"}} ->
            headers(C, Req, H);
        {ok, http_eoh} ->
            body(C, Req#req{headers = lists:reverse(H)});
	_Other ->
	    exit(normal)
    end.

%% Shall we keep the connection alive? 
%% Default case for HTTP/1.1 is yes, default for HTTP/1.0 is no.
%% Exercise for the reader - finish this so it does case insensitivity properly !
keep_alive({1,1}, "close")      -> close;
keep_alive({1,1}, "Close")      -> close;
keep_alive({1,1}, _)            -> keep_alive;
keep_alive({1,0}, "Keep-Alive") -> keep_alive;
keep_alive({1,0}, _)            -> close;
keep_alive({0,9}, _)            -> close;
keep_alive(Vsn, KA) ->
    io:format("Got = ~p~n",[{Vsn, KA}]),
    close.

The code for the state body is below. At this point we have everything required except the body in the case of a POST request. If present this is retrieved in a single chunk based on the content length supplied. Most web servers will implement some sort of size limit for POST requests. This is still needed in our case to avoid a single client taking all the memory of the Erlang Virtual machine with the subsequent crash. It should be simple to add.

Unless the connection is a keep-alive type the process terminates at the end of processing this function. All resources are cleared up at process exit including open sockets so we do not need to be too careful about explicitly tidying up.

body(#c{sock = Sock} = C, Req) ->
    case Req#req.method of
        'GET' ->
            Close = handle_get(C, Req),
            case Close of
                close ->
                    gen_tcp:close(Sock);
                keep_alive ->
                    inet:setopts(Sock, [{packet, http}]),
                    request(C, #req{})
            end;
        'POST' when is_integer(Req#req.content_length) ->
            inet:setopts(Sock, [{packet, raw}]),
            case gen_tcp:recv(Sock, Req#req.content_length, 60000) of
                {ok, Bin} ->
                    Close = handle_post(C, Req#req{body = Bin}),
                    case Close of
                        close ->
                            gen_tcp:close(Sock);
                        keep_alive ->
                            inet:setopts(Sock, [{packet, http}]),
                            request(C, #req{})
                    end;
                _Other ->
                    exit(normal)
            end;
        _Other ->
            send(C, ?not_implemented_501),
            exit(normal)
    end.

The rest of the iserve_socket module is below. There is not much left to do. The inet driver has already worked out for us what sort of URI is being used.

The call_mfa/4 function relies on the existence of an ets/mnesia table which converts the URI into a module and function dynamic callback. This must have been created at installation (see section later).

handle_get(C, #req{connection = Conn} = Req) ->
    case Req#req.uri of
        {abs_path, Path} ->
            {F, Args} = split_at_q_mark(Path, []),
            call_mfa(F, Args, C, Req),
            Conn;
        {absoluteURI, http, _Host, _, Path} ->
            {F, Args} = split_at_q_mark(Path, []),
            call_mfa(F, Args, C, Req),
            Conn;
        {absoluteURI, _Other_method, _Host, _, _Path} ->
            send(C, ?not_implemented_501),
            close;
        {scheme, _Scheme, _RequestString} ->
            send(C, ?not_implemented_501),
            close;
        _  ->
            send(C, ?forbidden_403),
            close
    end.

handle_post(C, #req{connection = Conn} = Req) ->
    case Req#req.uri of
        {abs_path, Path} ->
            call_mfa(Path, Req#req.body, C, Req),
            Conn;
        {absoluteURI, http, _Host, _, Path} ->
            call_mfa(Path, Req#req.body, C, Req),
            Conn;
        {absoluteURI, _Other_method, _Host, _, _Path} ->
            send(C, ?not_implemented_501),
            close;
        {scheme, _Scheme, _RequestString} ->
            send(C, ?not_implemented_501),
            close;
        _  ->
            send(C, ?forbidden_403),
            close
    end.

call_mfa(F, A, C, Req) ->
    case iserve:lookup(C#c.port, Req#req.method, F) of
        {ok, Mod, Func} ->
            case catch Mod:Func(Req, A) of
                {'EXIT', Reason} ->
                    io:format("Worker Crash = ~p~n",[Reason]),
                    exit(normal);
                {200, Headers0, Body} ->
                    Headers = add_content_length(Headers0, Body),
                    Enc_headers = enc_headers(Headers),
                    Resp = [<<"HTTP/1.1 200 OK\r\n">>,
                            Enc_headers,<<"\r\n">>,
                            Body],
                    send(C, Resp)
            end;
        {error, not_found} ->
            send(C, ?not_found_404)
    end.
       
add_content_length(Headers, Body) ->
    case lists:keysearch('Content-Length', 1, Headers) of
        {value, _} ->
            Headers;
        false ->
            [{'Content-Length', size(Body)}|Headers]
    end.


enc_headers([{Tag, Val}|T]) when is_atom(Tag) ->
    [atom_to_list(Tag), ": ", enc_header_val(Val), "\r\n"|enc_headers(T)];
enc_headers([{Tag, Val}|T]) when is_list(Tag) ->
    [Tag, ": ", enc_header_val(Val), "\r\n"|enc_headers(T)];
enc_headers([]) ->
    [].
enc_header_val(Val) when is_atom(Val) ->
    atom_to_list(Val);
enc_header_val(Val) when is_integer(Val) ->
    integer_to_list(Val);
enc_header_val(Val) ->
    Val.

%% Split the path at the ?. This would have to do all sorts of
%% horrible ../../ path checks and %C3 etc decoding if we wanted to
%% retrieve actual paths to real filesystem files. As it is we only
%% want to look it up as a key in mnesia/ets :)
split_at_q_mark([$?|T], Acc) ->
    {lists:reverse(Acc), T};
split_at_q_mark([H|T], Acc) ->
    split_at_q_mark(T, [H|Acc]);
split_at_q_mark([], Acc) ->
    {lists:reverse(Acc), []}.

  
send(#c{sock = Sock}, Data) ->
    case gen_tcp:send(Sock, Data) of
        ok ->
            ok;
        _ ->
            exit(normal)
    end.

Setting up the web server

The Web server requires two preparation steps. The port number the web server listens on is defined in a file called iserve.conf which must be located in the priv subdirectory of the iserve application. It must contain a line of the form:

{port, 8081}.

If this file is not present then the port number defaults to 8080.

The web server also uses an mnesia table to manage mappings between URLs and implementation callbacks. This may be created and managed with the iserve.erl module:

-module(iserve).
-export([create_table/1, 
         add_callback/5, delete_callback/3, 
         print_callbacks/0,lookup/3]).

-record(iserve_callback, {key,                 % {Port, 'GET'|'POST', Abs_path}
                          mf}).                % {Mod, Func}

create_table(Nodes) ->
    mnesia:create_table(iserve_callback,
                        [{attributes, record_info(fields, iserve_callback)},
                         {disc_copies, Nodes}]).

lookup(Port, Method, Path) ->
    case ets:lookup(iserve_callback, {Port, Method, Path}) of
        [#iserve_callback{mf = {Mod, Func}}] ->
            {ok, Mod, Func};
        [] ->
            {error, not_found}
    end.

add_callback(Port, Method, Path, Mod, Func) when ((Method == 'GET') or (Method == 'POST') and
                                                  is_list(Path) and is_atom(Mod) and
                                                  is_atom(Func) and is_integer(Port)) ->
    mnesia:dirty_write(iserve_callback, #iserve_callback{key = {Port, Method, Path},
                                                         mf = {Mod, Func}}).


delete_callback(Port, Method, Path) ->
    mnesia:dirty_delete(iserve_callback, {Port, Method, Path}).

print_callbacks() ->
    All = mnesia:dirty_match_object(#iserve_callback{_ = '_'}),
    io:format("Port\tMethod\tPath\tModule\tFunction~n"),
    lists:foreach(fun(#iserve_callback{key = {Port, Method, Path},
                                       mf = {Module, Function}}) ->
                          io:format("~p\t~p\t~p\t~p\t~p\r\n",[Port, Method, Path, Module, Function])
                  end, All).

iserve:create_table([node()]). must be called once at installation.

All Urls must be stored in this table with a module and function which will create the page. So for example the callback for the document root might be defined with:

iserve:add_callback(8081, 'GET', "/", test_iserve_app, do_get).

The callback for index.html could be:

iserve:add_callback(8081, 'GET', "/index.html", module, function2).

Building a web application

The simplest kind of iserve web application would be one to simply return a generated page. A function must be implemented which returns {200, Headers, Body} where Headers is a list of {Header-Atom, Val-String} and Body is a binary. For example:

-module(test_iserve_app).
-export([do_get/2]).
-include("iserve.hrl").

do_get(#req{} = Req, Args) ->
    {200, [], <<"<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\"><html><head><title>Welcome to iserve</title></head><body>
  Hello</body></html>">>}.

Obviously this is an extremely simple example. This is where you come in!

Supervisor and Application implementation

The web server only needs a little help to become a full blown OTP application. It needs an application behaviour, a supervisor behaviour, and a .app file.

These are presented below.

The Application:

-module(iserve_app).
-behaviour(application).
-export([
	 start/2,
	 stop/1
        ]).

start(_Type, _StartArgs) ->
    case iserve_sup:start_link() of
	{ok, Pid} -> 
	    alarm_handler:clear_alarm({application_stopped, iserve}),
	    {ok, Pid};
	Error ->
	    alarm_handler:set_alarm({{application_stopped, iserve}, []}),
	    Error
    end.

stop(_State) ->
    alarm_handler:set_alarm({{application_stopped, iserve}, []}),
    ok.

The Supervisor:

-module(iserve_sup).
-behaviour(supervisor).
-export([
	 start_link/0,
         init/1
        ]).

-define(SERVER, ?MODULE).

start_link() ->
    supervisor:start_link({local, ?SERVER}, ?MODULE, []).

init([]) ->
    Port = get_config(),
    Server = {iserve_server, {iserve_server, start_link, [Port]},
	     permanent, 2000, worker, [iserve_server]},
    {ok, {{one_for_one, 10, 1}, [Server]}}.

get_config() ->
    case file:consult(filename:join(code:priv_dir(iserve), "iserve.conf")) of
        [{port, Port}] ->
            Port;
        _ ->
            8080
    end.

The .app file.

A dependency on sasl is only included because of the calls to set and clear alarms in the application behaviour implementation:

{application, iserve,
        [{description, "Web Server"},
         {vsn, "%ISERVE_VSN%"},
         {modules, [    iserve_sup,
			iserve_app,
			iserve_server,
                        iserve_socket
			]},

         {registered, [	iserve_sup]},
         {applications, [kernel, stdlib, sasl]},
	 {mod, {iserve_app, []}}]}.

License

The code associated with this HOWTO is available under the BSD License

Disclaimer

The undocumented features presented in this HOWTO are undocumented because they are not supported by Ericsson. On the other hand they are used in commercially shipping systems.

PrivacyGuide: Towards an implementation of the EU GDPR on Internet privacy policy evaluation Tesfay et al., IWSPA’18

(Note: the above link takes you to the ACM Digital Library, where the paper should be accessible when accessed from the blog site. If you’re reading this via the email subscription and don’t have ACM DL access, please follow the link via my blog.)

…if a user was to read the privacy policy of every service she visits on the Internet, she would on average need 244 hours annually which is slightly more than half of the average time a user would spend on the Internet by then.

Studies have shown that only 1% or less of total users click on privacy policies, and those that do rarely actually read them. The GDPR requires clear succinct explanations and explicit consent (i.e., no burying your secrets on page 37 of a 70 page document), but that’s not the situation on the ground right now, and it’s hard to see that changing overnight on May 25th.

So we know that privacy is an important matter, and that a solution involving reading lengthy terms of service to determine the privacy implications of using a particular app/service/site is untenable. What can we do (beyond legislating for shorter ToS’s)? We could try either:

1. Crowdsourcing interpretations of privacy policies. For example, Terms of Service: Didn’t Read (ToS:DR) is a community based project to evaluate privacy policies by crowdsourcing, and provides a browser add-on. Unfortunately it lacks coverage (68 web sites covered since its launch in 2012, and only 11 of them given a final grade), and is nearly impossible to keep up to date since companies frequently update their policies.
2. Automating the interpretation of privacy policies to highlight the key information a prospective user really needs to know. That’s what PrivacyGuide does. It turns lengthy terms of service documents into this:

PrivacyGuide breaks down privacy policies into 11 standard categories, based on an analysis of the GDPR requirements. It presents a simple dashboard summary with green (low risk), orange (average risk) and red (high risk) indicators for each. You can drill down into the detail behind the icons to see a short description of the meaning and consequence of the risk class, and the closest original sentence or paragraph on which the assessment was performed. The idea is to present a consistent format across sites, similar to the way nutrition labels are standardised. This would be a big step forward. But even then, ‘The curious case of the pdf converter that liked Mozart’ study that we looked at last year showed that people still tend to just click-through. The dialogs developed in that study showing the privacy implications of permissions granted by a policy remains my favourite.

Understanding privacy policies

To start with, the authors conducted an extensive analysis of the GDPR with the support of legal experts. Comparing the privacy aspects of the GDPR with classification categories used in previous studies, resulted in the following set of 11 categories:

(Enlarge)

Within each category a group of privacy experts determined three risk levels. Here are examples for the third-party sharing and account deletion categories:

(Enlarge)

With this framework in hand, a privacy corpus was built using the privacy policies of the 45 most accessed websites in Europe according to the results provided by Alexa. (It’s a shame the authors don’t make this assessment available anywhere that I’m aware of…). 35 participants manually extracted on average 12,666 text passages from a single privacy policy, of which 433 ended up being assigned to a privacy category and classified with a risk level.

Classification

The next step is to train a multi-class classifier (11 privacy categories, and three risk levels per category) using this corpus. The team experimented with Naive Bayes, SVM, Decision Trees, and Random Forests: Naive Bayes turned out to work the best for this problem.

The PrivacyGuide workflow looks like this:

Content extraction is done using the Boilerpipe library. The resulting text is then split into individual sentences, and a set of keywords (obtained from the training data) used to identify informative sentences. Fragments not including any of these keywords are filtered out at this stage.

From the resulting set of sentences, stop word removal, tokenisation, and stemming are performed, and WEKA’s StringToWordVector filter is used to create sentence vectors. TF-IDF is also used to consider the most relevant words.

Using these features, classification is done in two steps.

1. The Privacy Aspect Prediction Engine (PAPE) predicts a privacy category, and is trained using 10-fold cross-validation.
2. Given a category, the Risk Prediction Engine (RiPE) predicts an associated risk class. In order to provide explanations to the user, the best representative sentence on which the risk classification is based is also captured.

Evaluation

The next 10 most accessed sites in Europe (according to Alexa) are then used for evaluation.

The table below shows the performance of the classifier on these sites.

Not perfect, but considering you weren’t going to read the policy otherwise and PrivacyGuide can produce its results in less than 2 seconds, it’s pretty good! Note the incidental discovery that none of the sites specified data breach notification plans in their policies.

U.S. President Donald Trump hailed Kim Jong Un’s effort to end his country’s seven-decade war with South Korea and pursue the “complete denuclearization” of the Korean Peninsula.

“KOREAN WAR TO END!” Trump tweeted Friday. “The United States, and all of its GREAT people, should be very proud of what is now taking place in Korea!”

Kim and South Korean President Moon Jae-in embraced after signing the deal during a historic meeting on their shared border, the first time a North Korean leader has set foot on the southern side. They announced plans to formally declare a resolution to the war and replace the 1953 armistice that ended open hostilities into a peace treaty by year’s end.

“We have agreed to share a firm determination to open a new era in which all Korean people enjoy prosperity and happiness on a peaceful land without wars,” Kim said, in his first remarks in front of the global press since taking power in 2011.

When Kim Met Moon: In Pictures

The two sides “confirmed the common goal of realizing, through complete denuclearization, a nuclear-free Korean Peninsula.” “South and North Korea agreed to actively seek the support and cooperation of the international community for the denuclearization of the Korean Peninsula,” according to the statement. It didn’t elaborate on what that would entail.

Trump had moments earlier expressed a mix of caution and optimism about the deal, “saying good things are happening, but only time will tell.”

“The commitment to ‘complete denuclearization’ is ambiguous, and subject to different interpretations,” said Youngshik Bong, a researcher at Yonsei University’s Institute for North Korean Studies in Seoul. “It can be interpreted as North Korea getting rid of all warheads, or North Korean demands on the U.S. military in South Korea.”

Rapid Thaw

The agreement follows a rapid thaw of tensions on the peninsula after a flurry of North Korean missile tests and a hydrogen bomb detonation last year. Kim plans to meet Trump soon, which would be the first summit between a North Korean leader and a sitting American president.

The question now is whether the commitment will lead to lasting change. Previous agreements have collapsed over inspections, weapons tests and disputes over economic aid.

Much of the agreement mirrors previous deals between North Korea and Moon’s liberal predecessors. It appeared aimed at restoring cooperation that had deteriorated over the past decade.

Kim’s official Korean Central News Agency issued a tersely worded commentary after the announcement urging the U.S. to respond “with sincerity.” “What is needed for the U.S. is to learn how to observe good manners and how to respect the party concerned, not resorting to high-handed practices and arrogance,” the piece said.

Reaction in markets was limited given the South Korean stock market had closed for the week when the deal was announced.

The cost of insuring South Korea’s sovereign bonds against non-payment fell 1.75 basis points to 44.75 basis points as of 6:10 p.m. in Seoul, Barclays Plc prices showed. The credit-default swap contracts, are on track for their lowest close since March 14, according to data provider CMA.

The stakes remain high, with Kim on the cusp of developing a missile capable of delivering one of his estimated 60 nuclear bombs to any city in the U.S. -- a step Trump has threatened war to stop.

Kim and Moon said they would hold military talks next month and seek a “phased disarmament,” without providing more details.

— With assistance by Heesu Lee, Sam Kim, and Kyungji Cho

When humankind entered North America, fifteen thousand years ago or more, it entered a world of giants—woolly mammoths, dire wolves, sabre-toothed cats, short-faced bears (effectively grizzlies on stilts). But, by the end of the Ice Age, about ten thousand years ago, the megafauna was mostly gone, driven extinct by the warming climate and, many scientists think, predation by us.

A paper in this week’s issue of Science Advances offers tantalizing evidence of that grim process in action. Archeologists working at White Sands National Monument, in New Mexico, found a series of fossilized footprints made by giant ground sloths, lumbering behemoths that once roamed North and South America. The tracks date to between ten thousand and fifteen thousand years ago, when the region was much wetter than it is today. The giant sloth was herbivorous but nonetheless a fearsome creature—eight feet tall when standing on its hind legs, with long arms, and long claws extending from its padded feet.

The researchers noticed something curious inside many of the sloth prints—human footprints, made not long after the sloth’s, the stride intentionally adjusted, as if the person were stalking the animal. The tracks of several sloths, and of several people, were found. The dating is imprecise, so the researchers can’t say with certainty whether the prints represent several events that took place during a span of time or just a single pursuit; perhaps the sloths were travelling as a group and were collectively harassed. In either case, a chase was under way. The sloth prints show sharp changes in direction, especially in areas where human footprints are numerous; they zigged, we zigged. In places, the sloth appears to have reared up, perhaps to fend off an attacker.

The researchers dismissed the possibility that the human track-maker was following the sloths simply to find an easier path through soft terrain. “The step length results in a long and uncomfortable human stride,” they note in the paper. Likewise, the interaction probably wasn’t “playful,” given the animal’s disposition. “A big animal like that would have come with huge amounts of risk,” Matthew Bennett, a professor of environmental and geographical sciences at Bournemouth University, in the U.K., and one of the paper’s authors, says in an accompanying video. “Going head to head with a sloth, the chances are that you might come off badly.” Many archeological sites offer only a static portrait of ancient life, but the team’s findings shed light on how we actually interacted with a now-extinct species. “We can begin to understand how they did it, how they actually stalked and attacked these large animals,” Bennett says. “And that gives us a better understanding of whether we, as humans, are guilty or not in the role of extinction.”

It’s a role that our species hasn’t outgrown. A separate paper in the same issue of Science Advances offers the most accurate count yet of the world’s great apes; the data are drawn from fieldwork done in fifty-nine countries by a global team of biologists and anthropologists. The results are initially reassuring: the researchers estimate that nearly three hundred and sixty-two thousand gorillas and a hundred and twenty-nine thousand chimps still live in Western Equatorial Africa, higher figures than in previous estimates. But eighty per cent of the populations of both species are in unprotected areas, outside the boundaries of parks or preserves, and all are under constant threat from poaching, disease, and the loss of habitat. The authors estimate that gorillas, already a critically endangered species, are declining in number by 2.7 per cent each year. Archeology continues to unfold around us, as one species after another slips forever into the fossil record. Ours is following a primitive track, heading we all know where.

Who we are

We're a world-class team that's changing how companies function. We've worked at Facebook, Snapchat, Palantir, Quora, and more.

Our team has built the simplest API for human intelligence. AWS changed the game by removing the need to run your own infrastructure, which enabled a new generation of software companies. We want to do the same by removing the need for you to run your own operations team.

AIY Voice Kit v2 includes Raspberry Pi Zero WH and pre-provisioned SD card

AIY Vision Kit v1.1 includes Raspberry Pi Zero WH, Raspberry Pi Cam 2 and pre-provisioned SD card

We're also introducing the AIY companion app for Android, available here in Google Play, to make wireless setup and configuration a snap. The kits still work with monitor, keyboard and mouse as an alternate path and we're working on iOS and Chrome companions which will be coming soon.

The AIY website has been refreshed with improved documentation, now easier for young makers to get started and learn as they build. It also includes a new AIY Models area, showcasing a collection of neural networks designed to work with AIY kits. While we've solved one barrier to entry for the STEM audience, we recognize that there are many other things that we can do to make our kits even more useful. We'll once again be at #MakerFaire events to gather feedback from our users and in June we'll be working with teachers from all over the world at the ISTE conference in Chicago.

The new AIY Voice Kit and Vision Kit have arrived at Target Stores and Target.com (US) this month and we're working to make them globally available through retailers worldwide. Sign up on our mailing list to be notified when our products become available.

We hope you'll pick up one of the new AIY kits and learn more about how to build your own smart devices. Be sure to share your recipes on Hackster.io and social media using #aiyprojects.