Quantcast
Channel: Hacker News
Viewing all 25817 articles
Browse latest View live

North American F-82 Twin Mustang

$
0
0
From Wikipedia, the free encyclopedia
P-82/F-82 Twin Mustang
North American XP-82 Twin Mustang 44-83887.Color.jpg
XP-82 prototype
RoleLong-range escort fighter and night fighter
ManufacturerNorth American Aviation
First flight15 June 1945
Introduction1946
Retired1953
Primary userUnited States Air Force
Number built272
Unit cost
US$215,154[1]
Developed fromNorth American P-51 Mustang

The North American F-82 Twin Mustang was the last American piston-engine fighter ordered into production by the United States Air Force. Based on the P-51 Mustang, the F-82 was originally designed as a long-range escort fighter in World War II. The war ended well before the first production units were operational.

In the postwar era, Strategic Air Command used the planes as a long-range escort fighter. Radar-equipped F-82s were used extensively by the Air Defense Command as replacements for the Northrop P-61 Black Widow as all-weather day/night interceptors. During the Korean War, Japan-based F-82s were among the first USAF aircraft to operate over Korea. The first three North Korean aircraft destroyed by U.S. forces were shot down by F-82s, the first being a North-Korean Yak-11 downed over Gimpo Airfield by the USAF 68th Fighter Squadron.

Design and development

Initially intended as a very long-range (VLR) escort fighter, the F-82 was designed to escort Boeing B-29 Superfortress bombers on missions exceeding 2,000 miles (3,200 km) from the Solomon Islands or Philippines to Tokyo, missions beyond the range of the Lockheed P-38 Lightning and conventional P-51 Mustangs. Such missions were part of the planned U.S. invasion of the Japanese home islands, which was forestalled by the surrender of Japan after the atomic bombings of Hiroshima and Nagasaki and the opening of Soviet attacks on Japanese-held territory in Manchuria.

The second prototype North American XP-82 Twin Mustang (44-83887) being flight-tested at Muroc Army Airfield, California.

In October 1943, the North American Aircraft design team began work on a fighter design that could travel over 2,000 mi (3,200 km) without refueling. It consisted of a twin-fuselage design, somewhat similar to the experimental German Messerschmitt Bf 109Z "Zwilling". Although based on the lightweight experimental XP-51F, which would later become the P-51H Mustang, it was actually a new design. North American Design Chief Edgar Schmued incorporated two P-51H Mustang fuselages lengthened by the addition of a 57 in (145 cm) fuselage plug located behind the cockpit where additional fuel tanks and equipment could be installed. These were mounted to a newly designed center wing section containing the same six .50 caliber M3 Browning machine guns as a single-engine Mustang, but with more concentrated fire. The first XP-82 prototype (s/n 44-83886) was equipped with a removable centerline gun pod housing eight additional .50 caliber M3 Brownings, but this did not feature on production aircraft. An even more powerful centerline gun pod containing a 40 mm cannon was considered, but was never built.[2] The outer wings were reinforced to allow the addition of hard points for carrying additional fuel or 1,000 pounds (450 kg) of ordnance. The two vertical tails were also from the XP-51F, but incorporated large dorsal fillets for added stability in case of an engine failure. The aircraft had a conventional landing gear with both wheels retracting into bays under each fuselage center section.

The XP-82 was to be powered by two Packard-built Rolls-Royce V-1650 Merlin engines. Initially, the left engine was a V-1650-23 with a gear reduction box to allow the left propeller to turn opposite to the right propeller, which was driven by the more conventional V-1650-25. In this arrangement both propellers would turn upward as they approached the center wing, which in theory would have allowed better single-engine control. This proved not to be the case when the aircraft refused to become airborne during its first flight attempt. After a month of work North American engineers finally discovered that rotating the propellers to meet in the center on their upward turn created sufficient drag to cancel out all lift from the center wing section, one quarter of the aircraft's total wing surface area. The engines and propellers were then exchanged, with their rotation meeting on the downward turn, and the problem was fully solved. The first XP-82 prototype (44-83886) was completed on 25 May 1945, and made the type's first successful flight on 26 June 1945. This aircraft was accepted by the Army Air Forces on 30 August 1945, whose officials were so impressed by the aircraft, while still in development, that they ordered the first production P-82Bs in March 1945, fully three months before its first flight.

Twin Mustangs and two jet engined FJ-1 Furys in production at North American, 1948.

The XP-82 prototypes, and production P-82Bs and P-82Es, retained both fully equipped cockpits so that pilots could fly the aircraft from either position, alternating control on long flights, while later night fighter versions kept the cockpit on the left side only, placing the radar operator in the right position.

Although some P-82B airframes were completed before the end of World War II, most remained at the North American factory in California waiting for engines until 1946. As a result, none saw service during the war.

Like several versions of the P-51 Mustang, the first two prototype XP-82s as well as the next 20 P-82B models were powered by British-designed Rolls-Royce Merlin engines, built under license by Packard. These provided the fighter with excellent range and performance; however, the Army had always wanted to give the Twin Mustang a purely American and stronger engine than the foreign-designed P-51's V-1650 (built at Packard plants, dismantled after the war). In addition, the licensing costs paid to Rolls-Royce for each V-1650 were being increased by Britain after the war. It therefore negotiated in August 1945 with the Allison Division of the General Motors Corporation for a new version of the Allison V-1710-100 engine.[1] This forced North American to switch subsequent production P-82C and later models to the lower-powered engines. It was found that Allison-powered P-82 models demonstrated a lower top speed and poorer high-altitude performance than the earlier Merlin-powered versions. The earlier P-82B models were designated as trainers, while the "C" and later models were employed as fighters, making the P-82 one of the few aircraft in U.S. military history to be faster in its trainer version than the fighter version.

In 1948, the 3200th Proof Test Group at Eglin AFB, Florida, fitted the 4th F-82B Twin Mustang (44-65163) with retractable pylons under the outer wings capable of mounting 10 High-Velocity Air Rockets (HVAR) each, which folded into the wing undersurface when not in use. This installation was not adopted on later models, the standard "tree" being used instead. The 13th aircraft (44-65171) was experimentally fitted with a center wing mounted pod housing an array of recon cameras, and was assigned to the 3200th Photo Test Squadron, being designated, unofficially, the RF-82B.[3]

Record-setting

P-82B-NA 44-65168"Betty Jo" taking off from Hickam Field, Hawaii, 27 February 1947.

On 27 February 1947, P-82B 44-65168, named Betty Jo and flown by ColonelRobert E. Thacker, made history when it flew nonstop from Hawaii to New York without refueling, a distance of 5,051 mi (8,129 km) in 14 hr 32 min. It averaged 347.5 miles per hour (559.2 km/h). This flight tested the P-82's range. The aircraft carried a full internal fuel tank of 576 US gallons (2,180 l; 480 imp gal), augmented by four 310 US gal (1,173 l; 258 imp gal) tanks for a total of 1,816 US gal (6,874 l; 1,512 imp gal). Also, Colonel Thacker forgot to drop three of his external tanks when their fuel was expended, landing with them in New York.[4]

To this day, it remains the longest nonstop flight ever made by a propeller-driven fighter, and the fastest such a distance has ever been covered in a piston-engined aircraft (the record for the longest unrefueled flight by a propeller-driven aircraft of any type is held by the Rutan Voyager). The aircraft chosen was an earlier "B" model powered by Rolls-Royce Merlin engines (see "Survivors" below).

Operational history

The Twin Mustang was developed at the end of the prop-driven fighter era and at the dawn of the jet age. Its designed role as a long-range fighter escort was eliminated by the atomic bombing of Japan and the sudden end of World War II. With the rapid draw-down of the armed forces after the war, the newly established United States Air Force had little money for new prop-driven aircraft, especially since jets, such as the Messerschmitt Me 262 and other Luftwaffe fighters, had been faster than P-51 Mustangs in the skies of Germany in late 1944. The completed airframes (less engines) of the P-82 pre-production aircraft already manufactured by North American went into storage, with an uncertain future.

However, during the 1947 Soviet Aviation Day display at Tushino Airport, a surprise appearance was put in by three Boeing B-29s, followed by a fourth four-engined long-range strategic bomber. It was an example of the Tupolev Tu-4, which was a bolt-for-bolt copy of the B-29 Superfortress, three examples of which were known to have been interned in the Soviet Union after having been forced to land there during bombing raids against Japan in WWII. Since the USSR was expected to have nuclear weapons sooner or later, the appearance of the Soviet Tu-4 was a shock to U.S. military planners, since it meant that the U.S. mainland might soon be vulnerable to a potential Soviet nuclear attack from the air.

Until jet interceptors could be developed and put into service, the Twin Mustangs already built were seen as an interim solution to SAC's fighter escort mission for its strategic bomber force and also as an all-weather air defense interceptor.

Early attempts to develop jet-powered all-weather fighters ran into a series of snags and delays. The Curtiss-Wright XF-87 Blackhawk had been ordered in December 1945, but it ran into developmental difficulties and the project was eventually totally abandoned in October 1948. The Northrop P-89 Scorpion was deemed to have greater promise, but it too ran into teething troubles and did not show promise of entering service until 1952 at the earliest. Due to the lack of any suitable jet-powered replacement, the wartime Northrop P-61 Black Widow night fighter was forced into this role, and in order to help fill in the gap until the Scorpion could be available, night fighter adaptations of the piston-engined North American P-82 Twin Mustang were developed and deployed.

On 11 June 1948, the newly formed United States Air Force eliminated the P-for-pursuit category and replaced it with F-for-fighter. Subsequently, all P-82s were re-designated F-82.

Strategic Air Command

27th FW North American F-82E Twin Mustangs, (Serial 46-354 identifiable) along with a Boeing B-29 Superfortress at Kearney AFB, Nebraska.

The F-82E was the first model to reach operational squadrons and its initial operational assignment was to the Strategic Air Command27th Fighter (later Fighter-Escort) Wing at Kearney Air Force Base, Nebraska in March 1948.

During World War II, the P-51 Mustang had escorted Boeing B-17 Flying Fortress and Consolidated B-24 Liberator bombers from bases in England and Southern Italy to targets throughout Nazi-occupied Europe. However, the Cold War brought on the challenge of possible B-29, Boeing B-50 or Convair B-36 bomber missions into the Soviet Union. The sheer size of the Soviet Union dictated that a bombing mission would be a 12-hour affair there and back from bases in Europe or Alaska, most of it over Soviet territory. Also the weather, which was bad enough in Western Europe, would make bombing missions impossible over the Soviet Union between October and May. With no long-range jet fighters yet available to perform escort missions for the strategic bomber force, the mission of the 27th FEW was to fly these long-range missions with their F-82Es.[5]

The F-82E had a range of over 1,400 mi (2,300 km), which meant that with external fuel tanks it could fly from London to Moscow, loiter for 30 minutes over the target, and return, the only American fighter which could do so. It also had an operational ceiling of 40,000 feet (12,200 m), where it could stay close to the bombers it was designed to protect. The first production F-82Es reached the 27th in early 1948, and almost immediately the group was deployed to McChord AFB, Washington, in June, where its squadrons stood on alert on a secondary air defense mission due to heightened tensions over the Berlin Airlift. It was also believed that the 27th would launch an escort mission, presumably to the Soviet Union, if conflict broke out in Europe. From McChord, the group flew its Twin Mustangs on weather reconnaissance missions over the northwest Pacific, but problems were encountered with their fuel tanks. Decommissioned F-61 Black Widow external tanks were found at Hamilton AFB, California, which could be modified for the F-82; fitted on the pylons of the Twin Mustang, these solved the problem. With a reduction in tension, the 27th returned to its home base in Nebraska during September, where the unit settled down to transition flying with their aircraft.[5]

Four F-82s were deployed from McChord to Alaska, where the pilots provided transition training to the 449th Fighter (All Weather) Squadron, which used Twin Mustangs in the air defense mission. They remained in Alaska for about 45 days, returning to rejoin the rest of the group at the beginning of November 1948.[5]

One of four F-82Es deployed by the 27th Fighter Escort Wing to Davis AFB, Aleutians in December 1948 to assist in the transition of the 449th Fighter (All-Weather) Squadron from P-61 Black Widows to the Twin Mustang.

In January 1949, Eighth Air Force planned a large celebration at Carswell Air Force Base. All of its assigned units were to participate in a coordinated flyover. Most of the Strategic Air Command's bombers were to participate, along with its only "Long Range" fighter group, the 27th. The weather in Nebraska was horrible, with most airports in the Midwest forced to close on the day of the display. Kearney Air Force Base was hit with a blizzard, and paths were cut though the snow allowing the F-82s to take off and rendezvous with the bombers. This was seen as proof of the F-82's capabilities in bad weather.[5]

In early 1949, the 27th began carrying out long-range escort profile missions. Flights to Puerto Rico, Mexico, the Bahamas and nonstop to Washington, D.C. were carried out. For President Truman's 1949 inauguration, the 27th FEW launched 48 aircraft to fly in review, along with several other fighter units, in formation down Pennsylvania Avenue. Another flyover of the newly dedicated Idlewild Airport in New York City soon followed, with the aircraft flying non-stop from Kearney AFB.[5]

In March 1949, Kearney AFB was closed, primarily for budgetary reasons, and the 27th FEW was reassigned to Bergstrom AFB, Texas. Other long-range missions were flown cross-country, and simulated "dogfights" with Lockheed F-80 Shooting Star units were carried out. The 27th FEW began transitioning to the Republic F-84 Thunderjet beginning in March 1950, and the F-82Es were largely declared surplus, with the last Twin Mustangs being phased out by September. A few were sent to Far East Air Forces for combat in Korea and a few were sent to Alaska to fly bomber escort missions over the Arctic from Ladd AFB until 1953. The majority were sent to reclamation and were gone by 1952.[5]

With the appearance of the MiG-15 over the skies of North Korea in late 1950, the B-29, as well as all of the propeller-driven bombers in the USAF inventory, were simply rendered obsolete as strategic offensive weapons. The straight-winged F-84Gs used in Korea as bomber escorts were ineffective against the MiG, and it took the swept-wing North American F-86 Sabre to clear the skies. It was clear that it would take a new generation of swept-wing jet bombers, able to fly higher and faster, to effectively defeat the defense of the jet propelled MiG-15 or subsequent Soviet-designed interceptors. Also, the era of large groups of bombers flying in formations to a strategic target ended after the Korean War. Strategic bombing evolved into a one-plane, one target affair, with the jet-equipped and nuclear-armed Boeing B-52 Stratofortress flying higher and faster than most enemy interceptors. The escort fighter concept became redundant, and by 1957 SAC had inactivated the last of its strategic fighter escort wings.

Air Defense Command

317th Fighter-All Weather Squadron North American F-82F Twin Mustang, AF Ser. No. 46-0415, Hamilton AFB, California, 1948. This aircraft was the first F-82F Twin Mustang delivered to Air Defense Command.

The appearance of the Soviet Tupolev Tu-4 'Bull' copy of the Boeing B-29 Superfortress in 1947 posed a new threat to postwar USAAF planners, and largely as a result of its appearance and the beginnings of the Cold War in 1948, led to the establishment of Air Defense Command (ADC) as part of the postwar United States Air Force.

In 1947, the embryonic ADC was not particularly effective. During a number of simulated raids on the United States carried out by Strategic Air Command during the period 1947–1949, defending F-51 Mustangs and F-47 Thunderbolts, being used as interceptors, repeatedly failed to find the incoming bombers, even when they knew they were on their way, and seldom got themselves into position to shoot them down. Furthermore, when they did so, the bombers (usually B-29 Superfortresses) had already made their runs over the intended targets of New York City, Los Angeles, Seattle, Chicago and St. Louis.[4]

It was found that the RADAR-equipped wartime Northrop P-61 Black Widow night fighter was effective in locating and attacking the incoming SAC bombers, and also had the range to attack the bombers far from their intended targets. In the late 1940s, there were no suitable jet U.S. interceptors yet developed, so the P-61 was transitioned into the ADC interceptor mission. The F-61s available, however, were largely war-weary and the night fighter F-82C/D models were modified into an all-weather interceptor to replace them.[4]

North American F-82F Twin Mustang 46-414, 52d Fighter Group (All Weather), Mitchel AFB, New York at the 1950 World Wide gunnery meet, Nellis AFB, Nevada on 26 March 1950. Marked as Group Commander's aircraft, flown by Colonel William Cellini.

The production interceptor versions of the Twin Mustang were designated the F-82F and F-82G; the distinguishing feature between the F and G models was largely the nacelle beneath the center-wing that housed radar equipment (F-82F's AN/APG-28 and F-82G's SCR-720C18). In addition, the interceptor version required numerous modifications. The right side cockpit was replaced with a radar operator's position without flight controls. A long radar pod, resembling a sausage and irreverently known as a "long dong", was attached to the underside of the center wing, below the six .50 inch machine guns and with its dish in front of the propellers to prevent signal interference. This unconventional arrangement was found not to affect the aircraft's performance seriously. Additionally the unit could be jettisoned in an emergency, or for belly landings – it was sometimes even lost during high-G maneuvers. F-82F models were designated for ADC units in the United States, while the F-82G models were deployed to Far East Air Forces for air defense of Japan and Okinawa. No F-82s were deployed to Europe.

The first F-82F models began to reach operational ADC squadrons in September 1948. By the end of September, ADC had 29 F-82Fs. By the middle of 1949 the F-82 was replacing Black Widows in service with Air Defense Command along the west coast at Hamilton AFB, California (317th FIS) and McChord AFB, Washington (318th FIS, 319th FIS). East coast defense was the mission of the Twin Mustangs assigned to McGuire AFB, New Jersey (2d FIS, 5th FIS).[1]

In addition to the forces in the Continental United States, the Caribbean Air Command 319th FIS at France Field in the PanamaCanal Zone received 15 F-82Fs in December 1948 to replace their Black Widows for the defense of the Panama Canal. The Twin Mustangs remained in Panama only briefly, being reassigned to McChord AFB in May 1949.[1]

The Twin Mustang was phased out of ADC during 1950–1951 by the Lockheed F-94 Starfire, the last being with the 318th FIS at McChord AFB in late 1951. Some were sent to Alaska, where they were modified into the F-82H configuration; most, however, were sent to reclamation.

Far East Air Forces

Farewell from family of Air Force Capt. Johnnie Gosnell, flying with the 68th All-Weather Fighter Interceptor Squadron at Itazuke AB Japan to fly on a combat mission over Korea.

After World War II, the Northrop P-61 Black Widow was the Far East Air Forces (FEAF) main interceptor. However, a lack of spare parts and maintenance issues made it difficult to keep the war-weary aircraft in the air. Until an all-weather jet interceptor became available, the F-82G Twin Mustang was seen as the interim solution. The last operational P-61s were sent by the 68th and 339th Fighter (All Weather) Squadrons to the reclamation yard at Tachikawa Air Base in February 1950, with Twin Mustangs arriving as replacement aircraft.[6]

In Far East Air Forces, there were three squadrons operating the Twin Mustang, consisting of 45 aircraft.[1] The 4th Fighter (All Weather) Squadron was attached to the 51st Fighter-Interceptor Wing at Naha Air Base, Okinawa, tasked with providing air defense within the Twentieth Air Force's area of responsibility during darkness and inclement weather. The 68th F(AW)S area would cover southern Japan from their base at Itazuke and the 8th Fighter Bomber Wing, with the 339th F(AW)S covering Tokyo and northern Japan from Johnson Air Base. FEAF had about 40 Twin Mustangs assigned to the command.[6]

With the outbreak of the Korean War in June 1950, the F-82 was pressed into combat duty. At 04:00 on 25 June 1950, 68th Fighter Squadron F-82 aircrews on alert at Itazuke Air Base were told that North Korea had crossed the 38th Parallel. They were ordered to fly to the area and report any activity on the main roads and railroads. They arrived to find overcast conditions, with cloud tops at 8,000 feet (2,400 m). The Twin Mustangs flew through the clouds using radar and broke out at 2,000 feet (610 m), heading for Kimpo Airfield near Seoul. The pilots observed huge convoys of North Korean trucks and other vehicles, including 58 tanks, which had crossed into South Korea. The crews flew back through the clouds to Itazuke Air Base, where they were debriefed by a U.S. Army colonel from General Douglas MacArthur's staff. This reconnaissance flight is considered to be the first United States air combat mission of the Korean War.[6]

With this information, along with other intelligence reports available to them, FEAF confirmed that the Korean People's Army had, indeed, launched a full-scale invasion of South Korea. FEAF's first priority, however, was to evacuate United States citizens. On the morning of 26 June, the nearby Norwegian freighter Reinholte was sent to Inchon harbor to evacuate non-military personnel from Seoul, which lay directly in the invasion route. A flight of Twin Mustangs from the 68th F(AW)S was dispatched to the area, arriving at dawn to provide air protection for the evacuation. Two of the F-82s were dispatched to fly over the road from Seoul, while others flew top cover over the Inchon docks. The patrol went without incident until about 1300, when a pair of Soviet-built aircraft (the exact aircraft type has never been determined) came out of the clouds. Orders given to the F-82 pilots prohibited any aggressive action; however, gun switches were activated when the enemy leader tightened up his turn and peeled off at the F-82s with his wing man in close tail. The F-82s dropped their external tanks, turned on combat power and started a climbing turn towards the North Korean aircraft. For some reason, the North Korean leader fired while too far away, with his bullets falling short of the F-82s which then pulled up into the clouds and above the overcast, putting them in a position to return fire if the North Koreans followed them. However, they did not, and no further contact was made for the rest of the day. The evacuation at Inchon was successfully carried out with a total of 682 civilians being transported to Sasebo, Japan.[6]

Flight of 339th FS F-82Gs (46-403, 46–390, 46–366, 46–394) heading to Korea in June 1950.

Once the bulk of US civilians had been evacuated out of South Korea by ship, the priority of FEAF changed to a military nature. The 339th F(AW)S received orders from Fifth Air Force to move all available aircraft, along with crews and equipment, to Itazuke Air Base to assist the 68th in providing air cover for the evacuation of Seoul. However, the 339th's complement of aircraft were scattered over several bases at the time. Seven F-82s at Yokota AB were flyable and two were in the hangar undergoing maintenance for major repairs. Four other F-82s were at Misawa AB on TDY. The three at Yokota were dispatched immediately to Itazuke, as well as the four at Misawa, making a total of seven combat ready F-82Gs present for duty on 27 June. The 68th F(AW)S had a total of twelve operational F-82Gs. This, combined with what the 339th could contribute, was insufficient to meet the combat needs thrust upon FEAF. The F-80 Shooting Star was available, but its thirsty jet engine meant it could only remain over the airfield for a few minutes before having to return to base and it could not reach the forward combat area from Japan. No USAF P-51 Mustangs were available, and the British Commonwealth had not yet made a decision to commit to combat in Korea, making the presence of Australian Mustangs in Japan academic at this critical situation. FEAF ordered Twentieth Air Force to send eight F-82s from its 4th F(AW)S from Okinawa to Itazuke, making a total of 27 F-82s available for combat duty. This was commendable, considering that on 31 May 1950 a total of 32 F-82s existed within FEAF. With these combined squadrons, the 347th Provisional Fighter Group (AW) was formed.[6]

Before dawn on 27 June, the 347th Provisional Group was up in the air over Korea, with a mission to provide cover for the Douglas C-54 Skymaster transports flying in and out of Kimpo Airfield as they moved the last civilians out. Fearing that the North Korean Air Force might try to shoot down the transport (a C-54 had been destroyed on the ground at Kimpo by North Korean fighters on 25 June), the Air Force requested air cover to protect the aircraft during takeoff. Fortunately, 339th Fighter All Weather Squadron (F(AW)S) with their F-82Gs were based at Yokota AB and the 68th F(AW)S was based at Itazuke AB Japan. With Lieutenant Colonel John F. Sharp in command, 27 F-82Gs of the 35 in Japan answered the call. Arriving in the early morning, they orbited Kimpo Airfield in three flights, each above the other. Suddenly, at 1150 hours, a mixed group of five North Korean fighters (Soviet-built Yak-9s, Yak-11s and La-7s) appeared, heading for the airfield. One of the Yak-11s immediately scored several hits on 68th F(AW)S pilot Lt. Charles Moran's vertical stabilizer. Moments later, Lt. William G. "Skeeter" Hudson, also of the 68th F(AW)S, initiated a high-G turn to engage the Yak, and soon was closing in on the Yak's tail. He then fired a short burst at close range, scoring hits with his six .50 inch machine guns. The Yak banked hard to the right, with the F-82G in close pursuit. A second burst hit the Yak's right wing, setting the gas tank on fire and knocking off the right flap and aileron. The North Korean pilot bailed out, but his observer, who was either dead or badly wounded, remained in the doomed aircraft. Parachuting down to Kimpo Airfield, the North Korean pilot was immediately surrounded by South Korean soldiers. Surprisingly, he pulled out a pistol and began firing at them. The South Korean soldiers returned fire, killing him. Moments later, Lt. Moran shot down an La-7 over the airfield, while a few miles away, Major James W. Little, commanding officer of the 339th F(AW)S, shot down another La-7. The C-54 was able to escape safely. Three of the five North Korean aircraft had been shot down, with pilot Lt. William G. "Skeeter" Hudson and radar operator Lt. Carl Fraiser scoring the first United States aerial "kill" of the Korean War.[6]

Considering these crews had not been extensively trained in air-to-air gunnery, they came out of combat looking very good. It is generally believed that the aircraft Hudson and Fraiser flew that day was an F-82G named "Bucket of Bolts" (46-383), as their usual aircraft was down for repairs. "Bucket of Bolts" would survive the Korean War and eventually be reassigned to escort duty in Alaska. It is believed to have been scrapped at Ladd AFB, Alaska in 1953.

46-371 of the 68th FS at Itazuke AB.

In the following weeks, the F-82 pilots would exceed all expectations in aerial combat. On 28 June, orders came down for all F-82s to be used in heavy ground support against any North Korean activity found between the front lines and the 38th Parallel. Every F-82 which could be made ready for flight was pressed into combat service. Although Fifth Air Force needed every available aircraft to slow down the North Korean invasion force, it was hard to justify the release of all F-82s from their defensive responsibilities for the many key bases in Japan. It was decided to release all F-82s for combat except for a flight which was deployed from the 4th F(AW)S in Okinawa to Japan and a full squadron of F-80s for air defense. On 30 June, FEAF requested HQ USAF for an additional 21 F-82 aircraft, which was denied. In addition, the projected level of support which could be provided at the level of combat usage FEAF was experiencing was no more than 60 days due to a shortage of parts. The fact was that, when F-82 production ended in April 1948, no provision had been made for an adequate supply of spare parts, as the aircraft was not expected to remain in operational service once jet-powered aircraft were available. Further, the Air Force simply did not have that many F-82s in the first place (182 total operational aircraft), and did not want to weaken the F-82 units committed to the Pacific Northwest or Atlantic coast, or to draw from the fourteen F-82Hs in Alaska.[1][6]

This was a heavy blow to FEAF planners, as they wanted to use the F-82 for escorting B-26 Invaders deep into North Korea and for searching out ground targets along the Han River. Making do with what they had was the order of the day, and maintenance crews were cannibalizing everything in sight in order to keep the maximum number of F-82s airborne. During the period 26–30 June, the 68th squadron flew 35 combat sorties, averaging five hours per sortie, with the 339th flying similar numbers of missions.[6]

During those early days, the stress and strain of combat put on the crews was intense. However, by early July the chances of F-82s engaging in air-to-air combat was significantly reduced, as the F-80 Shooting Stars had effectively stopped North Korea's air force from coming below the 38th parallel. The F-82s began flying strike and escort missions, along with night intruder sorties. Several F-82s took hits in their radar radomes, which were difficult to replace, and the radomes were removed, turning the aircraft into day fighters. In the ground support role, the F-82s could reach any part of the Korean battlefield with a total ordnance load of over 4,000 lb (1,800 kg). Each of the six .50 inch machine guns carried 400 rounds. This firepower was well-used against numerous ground targets. The escort missions flown with the B-26s took F-82s deep into North Korea. Flying with external fuel tanks, it was necessary on many occasions for the Twin Mustangs to drop tanks, owing to the risk of fire or explosion if enemy fire hit one of the empty tanks. On 10 July, F-82s from the 4th and 68th squadrons participated in one of the biggest strikes of the war against ground targets. Joined by B-26s and F-80s, the aircraft hit massive amounts of North Korean road traffic. An estimated 117 trucks, 38 tanks and seven personnel carriers were destroyed, along with a large number of enemy troops killed when the B-26s destroyed a bridge at Pyongtaek, causing a massive jam.[6]

46-363 and unidentified 68th FS F-82G on the alert ramp at Kimpo AB, South Korea.

On 5 July, the 339th Squadron was pulled out of combat and returned to Johnson AB. Shortly afterwards, the 4th Squadron returned to Okinawa, with the 347th Provisional Group being inactivated and control of the 68th Squadron being turned over to the 8th Fighter Group. The 339th had been in combat a total of 10 days (26 June–5 July), flying a total of 44 combat sorties for which they had been given no training. The 68th Squadron was left to carry on the battle. Throughout July and August 1950, F-82s from the 68th Squadron attacked enemy trains, vehicles, and numerous buildings, and constantly strafed North Korean troops on the roads. On the night of 27 August, an element of F-82s was patrolling over South Korea over a thick overcast when they received an urgent request for air support from some hard-pressed ground troops. Darkness was approaching when they reached the area and found UN ground troops pinned down by a concentration of mortars. The F-82 pilots made several passes to get set up with the ground controller, and as soon as the enemy target was pinpointed, the heavily armed aircraft commenced an attack that would last 45 minutes and use up all their ordnance. When the aircraft pulled up for the last time, the mortar positions were silent and ground forces later showed over 300 enemy dead.[6]

Beginning in October 1950, F-82s would start flying weather reconnaissance pre-dawn missions over North Korea. At the same time, the squadron would also be responsible for keeping at least three aircraft on alert on airfields in the Seoul area (K-13 (Suwon) and K-14 (Kimpo)) during the hours of darkness and bad weather. This would become the main mission for the F-82s for the balance of 1950, as the F-51s, F-80s and F-84s took on most of the combat ground attack missions which the F-82s had been pressed into at the beginning of the war. With the entry of the Chinese Communist forces into the war, the situation on the ground began to deteriorate rapidly. By late December, the 68th had begun flying two-aircraft missions during daylight and single-aircraft missions at night from Kimpo AB. On 7 January, FEAF ordered the 68th to start flying armed reconnaissance missions to check roads over southern North Korea as UN forces were rapidly withdrawing south before the Chinese onslaught. This was a nightmare as the Chinese were pouring south, and it appeared that the situation was becoming as it had been the previous June. On 26 January, the armed reconnaissance missions were discontinued and the F-82s were placed on continuous combat air patrols over Kandong Airfield near Pyongyang and over both of Pyongyang's main airfields, (K-23, Pyongyang and K-24, Pyongyang East) to monitor enemy air activity. This was essential as any Chinese aircraft operating out of these bases would be in easy range of the UN front lines. The 68th's efforts claimed 35 trucks destroyed, with damage to many others.[6]

As 1951 progressed, the F-82s of the 68th Squadron continued its mission of air defense over Seoul and flying weather reconnaissance flights; however, its combat duties became more and more limited. The end of the line was rapidly approaching for the F-82 in Korea. By the end of August 1951, there were only eight operational F-82s with the 68th, with its replacement, the Lockheed F-94 Starfire arriving in Japan, taking over missions previously flown by the Twin Mustangs. In March 1952, the Starfire-equipped 319th Fighter-Interceptor Squadron arrived from Moses Lake AFB, Washington and took up residence at K-13. On 28 March 1952, the last F-82G was sent for cold-weather modification, and then deployed to Alaska. By mid-April 1952, the F-82s in Okinawa were also sent to Japan for modification and were also sent to Alaska. All-weather responsibilities in FEAF were now in the hands of the F-94 and the jet era.[6]

F-82G operational losses during the Korean War[7]

  • 46-357 (6160th ABW, 68th FAWS) MIA 28 May 1951, 20 mi (32 km) N of 38th parallel.
  • 46-364 (6160th ABW, 68th FAWS) destroyed on ground at Suwon, Korea, 29 June 1950 by enemy aircraft.
  • 46-373 (6160th ABW, 68th FAWS) crashed 5 mi (8 km) NW of Brady AB, Japan, 12 February 1951.
  • 46-375 (6160th ABW, 68th FAWS) crashed 16 December 1950.
  • 46-378 (6160th ABW, 68th FAWS) shot down by AAA 3 July 1951.
  • 46-391 (51st FIG, 4th FAWS) in midair collision with F-80C 49-704 between Fukuoka and Ashiya AB, Japan, 29 September 1950.
  • 46-394 (6160th ABW, 68th FAWS) MIA 14 March 1951.
  • 46-399 (6160th ABW, 68th FAWS) MIA 27 January 1951.
  • 46–400 (6160th ABW, 68th FAWS) crashed near K-14 7 December 1950.
  • 46-402 (51st FIG, 68th FAWS) MIA 6 July 1950.

1951 was the last full year of F-82 operations in Korea, as they were gradually replaced by the jet-powered F-94 Starfire. Twin Mustangs destroyed 20 enemy aircraft, four in the air and 16 on the ground during the conflict.[1]

Alaskan Air Command

USAF operational F-82 Twin Mustang, F-82F AF Ser. No. 46-0415, on the ramp at Ladd AFB, just before going to salvage at Elmendorf AFB, May 1953.

Modified F-82F/Gs for cold weather were designated as F-82H. Six were assigned initially to the 449th F(AW)S at Adak Island in December 1948 to replace their P-61 Black Widows in the Alaska Air Defense mission. The Twin Mustang was well-suited for the air defense mission in Alaska due to its long-range flying ability. In March 1949, the squadron was reassigned to Ladd AFB, near Fairbanks, where an additional eight (14 total) arrived.[4]

In Alaska, the squadron would constantly patrol over wide areas of Alaska as the Soviets flying from airfields in Siberia would constantly test the air defenses. Beginning in August 1950, the 449th began receiving the F-94 Starfire jet interceptor, and the F-82s were assigned to a separate detachment within the squadron. With the outbreak of the Korean War, tensions were high in Alaska, as it was feared that it would become a "back door" to Soviet aircraft striking North America. The Soviets were constantly testing the Alaskan air defenses, with the F-94s responding when radar stations alerted them to incoming aircraft. Intercepts were rare, with only a few instances of eye-to-eye contact with Soviet aircraft. The slower F-82s had a longer range than the F-94s, and the Twin Mustangs constantly patrolled over many Alaskan villages and towns. Periodically, the F-82s were used for long-range visual reconnaissance near several known rough airstrips on the Chukchi Peninsula that the Soviets had used during World War II as landing fields for lend-lease aircraft and checking for any activity. In addition, flights around the Nome area and along the western Alaskan coastline were made. Squadron records show the Twin Mustangs were flown over some of the most remote areas of the Territory, along what today is known as the "North Slope" and over very rugged interior regions.[8]

The final flight crew and maintenance support personnel of F-82 46-415, May 1953.

The Twin Mustangs in Alaska operated further north than any other USAF aircraft at the time. In April 1952, they were flying escort duty for SAC B-36 Peacemaker bombers near Barter Island in the Beaufort Sea, near the most northerly tip of Alaska, about 1,000 mi (1,600 km) south of the North Pole.[8] The F-82H made a brief but memorable appearance in the movie "Top of the World" (1955).[9]

Another mission of the 449th was to act as a simulated invasion force during U.S. Army combat maneuvers during the summer months. Army ground units in Alaska were very limited in their movements due to the geography of the land. Most movements were up and down roads and paths and railroad rights-of-way. The F-82s would fly low along the terrain then pop up and initiate simulated strafing runs against them, causing the troops to take cover by hitting the muddy tundra. On occasions, the Twin Mustangs would also drop tear gas canisters, simulating gas attacks on the units.[8]

The 449th would also assist the local government by bombing ice jams on the Tanana, Kuskokwim and Yukon Rivers. This helped avert disastrous floods in the region, by dropping 500 lb (230 kg) bombs and firing 5 in (127 mm) rockets at the ice, thus allowing the rivers to keep flowing and not get clogged up.[8]

Two F-82Hs and one F-82G on the ramp at Ladd AFB, late summer 1952

Beginning in spring 1950, Strategic Air Command began replacing their F-82E bomber escorts and throughout the year, several were transferred to the 449th in Alaska. In February 1951, the Air Force ordered Alaskan Air Command that all the remaining F-82s in the Air Force inventory would all be transferred to Alaska. The Twin Mustangs would be used to support Army units in air-to-ground operations, and to use the F-94 Starfires for interceptor duty.[10]

During 1951 and 1952, F-82Fs from the Air Defense Command52d Fighter Group at McGuire AFB and 325th Fighter Group at McChord AFB were winterized and flown to Ladd AFB as replacements when those units transitioned to the F-94.[8]

F-82Gs from Far East Air Force were also sent to Alaska, as jets took over combat duties over the skies of Korea. The FEAF F-82s, however, arrived in a badly corroded condition. Also, many of these aircraft which were sent to the 449th had high time on their airframes from long bomber escort and air defense flights, as well as the stress from combat duty in Korea causing many of them to be difficult to maintain.[8] After four months at the 39th Air Depot Wing at Elmendorf, most were scrapped.[10]

All in all, this provided AAC with a total of 36 various E, F, G and H model F-82s. By 1952, attrition had claimed many of the Twin Mustangs which were assigned to the squadron. The occasional long-range reconnaissance flight over the Bering Sea was still flown, given that the range of the Twin Mustang was much greater than that of the F-94. The right seat of the aircraft was usually occupied by an experienced flight mechanic. With the long patrol flights with high-hour aircraft, pilots began having more and more mechanical problems that forced them to land on crude flight strips. The mechanic usually could repair the aircraft well enough to get airborne and head straight back to Ladd AFB.[8][10]

Retirement

The Twin Mustang had a very short operational life. About two years after its introduction to SAC, the F-82E was phased out of service in favor of the jet-powered Republic F-84E Thunderjet for bomber escort duties beginning in February 1950; the F-82Es were declared surplus by the end of the summer. Some were modified into F-82Gs and sent to Korea for combat as replacement aircraft, others were converted to F-82Hs and sent to Alaska, but most were sent to storage at Robins AFB, Georgia and ultimately reclamation.

F-82H Twin Mustang 46-377 at the end of the line, Elmendorf AFB, fall 1953

In the Pacific, the F-82Gs flown in combat over Korea were replaced by the Lockheed F-94A Starfire starting in April 1951 with the last being sent to the Tachikawa Air Depot in early 1952. There, they were either scrapped or sent to Alaska after being modified to the F-82H configuration in Japan for cold-weather use. Air Defense Command's F-82Fs began to be replaced by F-94As in June 1951, with most being declared excess by the end of the year and were sent to storage and ultimately reclamation at McChord AFB, Washington, although a few Twin Mustangs remained in ADC service towing aerial targets.

In Alaska, attrition and a lack of spare parts led to the withdrawal of the F-82 from the USAF inventory. Initially, 16 of the 36 aircraft became hangar queens and were the main source for spare parts to keep the others operational.[10] By the spring of 1953, the number of aircraft available had dwindled to a handful, with two or three operational aircraft kept flying by cannibalizing others which were incapable of being repaired.[10] Each aircraft were flown about 48 hours per month until it was impossible to keep them in the air on a reliable basis.

Ultimately, all were withdrawn from service when the lack of replacement parts made it impossible to keep the aging airframes flying. The last remaining F-82Hs were ultimately surplused and sent to Elmendorf AFB, Alaska for disposal in June 1953. The last Twin Mustang that remained in the operational inventory (46–377) was originally configured as an F-82G model that had served with two different squadrons in Okinawa and Japan. It was flown to Ladd AFB in 1952 and modified as a winterized F-82H. It was sent to Elmendorf AFB, being officially retired on 12 November 1953.

Some of the planes in storage at Robins and McChord were turned over to museums as static displays. Also after the end of their USAF service, the National Advisory Committee for Aeronautics (NACA) acquired four surplus F-82s for research work.

EF-82B Twin Mustang 1-NA 44-65168 "Betty Jo" with the National Advisory Committee for Aeronautics (NACA), Cleveland, Ohio, 1952.[N 1]
  • XP-82-NA 44-83886 (c/n 120-43742) (NACA-14). Operated by NACA from 6 June 1948 to 5 October 1955. This aircraft was ultimately sent to salvage at Langley AFB, Virginia.
  • XP-82-NA 44-83887 (c/n 120-43743). Operated from October 1947 to July 1950. Used for ram jet tests, it was damaged during July 1950. The damaged aircraft was sold to Walter Soplata and placed on his farm. In March 2008, it was sold to Tom Reilly and moved to his restoration facility in Douglas, Georgia.[11]
  • P-82B-NA 44-65168 (c/n 123-43754) (NACA-132). Operated from September 1950 to June 1957: This was the Betty Jo distance record holder aircraft and it was used for ram jet tests. Was turned over to the USAF Museum.
  • F-82E-NA 46-256 (c/n 144-38141) (NACA-133). Used for high-altitude icing tests beginning in January 1950, re-designated EF-82E. Sold to Walter Soplata for $1,600 in March 1954. Now being restored to flying condition by C&P Aviation in Anoka, Minnesota.

References for NACA F-82s:[7][12]

Variants

Production numbers
VariantProducedConversions
XP-822
XP-82A1
P-82B20
P-82C01
P-82D01
F-82E100
F-82F100
F-82G509
F-82H015
TOTAL273
NA-123
Basic Development design. The NA-123 design was presented by North American Aircraft to the USAAF in February 1944. The design for the new aircraft was for a long-range fighter to penetrate deep into enemy territory. Its immediate role would be to escort the B-29 Superfortress bombers used in the Pacific Theater of Operations against Japan. The USAAF endorsed it at once. A letter contract to construct and test four experimental XP-82 aircraft (P-82 designation) gave way in the same month to an order for 500 production models.
XP-82 / XP-82A
Prototype. The USAAF accepted the first XP-82 in August 1945 and a second one in September. Both were equipped with Packard Merlin V-1650-23 and −25 engines. The third experimental aircraft, designated XP-82A, had two Allison V-1710-119 engines. It was accepted in October 1945. There is no evidence that the XP-82A was ever actually flown, due to problems with the Allison engines. The fourth XP-82A prototype (44-83889) was canceled.
P-82B
Planned production version. With the end of World War II, production plans were cut back significantly. Against the 500 P-82Bs initially planned, overall procurement was finalized on 7 December 1945 at 270 P-82s. Included were 20 P-82Bs already on firm order and later allocated to testing as P-82Z. The USAAF accepted all P-82Zs in fiscal year 1947. Two aircraft were accepted in January 1946, four in February 1947, and 13 in March 1947. By December 1949, no P-82Bs (by then redesignated F-82Bs) remained in the Air Force inventory. These P-82Bs were basically similar to the XP-82, but differed in having provisions for underwing racks.
P-82C 44-65169 in black night fighter motif. Note the large bulge that carries the radar array under the wing
P-82C
Night fighter version. A P-82B, (44-65169) modified in late 1946, for testing as a night interceptor. The P-82C featured a new nacelle (under the center wing section) housing an SCR-720 radar. The SCR-720 was the same radar installation which was carried aboard the Northrop P-61 Black Widow, a considerably larger aircraft. The right-hand cockpit became the radar operator's position. The production version was designated P-82G.
P-82D
Night fighter version. Another P-82B (44-65170) modified with a different radar, the APS-4. The APS-4 was a much smaller set than the SCR-720, and operated in the 1.18 in (3 cm) waveband. As like the P-82C, the right-hand cockpit became the radar operator's position. The production version was designated P-82F.
F-82E
Escort fighter version. The F-82E followed the F-82B, which it so closely resembled. They were equipped with two counter-rotating Allison liquid-cooled engines, V-1710-143 and V-1710-145. The first four F-82Es were redesignated as F-82As and were allocated for engine testing. After production delays by engine problems and additional testing, F-82Es entered operational service in May 1948. The Air Force accepted 72 F-82Es in fiscal year 1948 (between January and June 1948), and 24 in fiscal year 1949 (22 in July 1948, one in October, and one in December).
North American F-82F Twin Mustang night fighter Serial 46-415
F-82F/G/H
Night fighter versions. A nacelle beneath the center-wing that housed radar equipment (F-82F's AN/APG28 and F-82G's SCR-720C18); automatic pilot; and a radar operator replacing the second pilot. When winterization was added to the F or G, it became an F-82H. Entered operational service in September 1948. One F-82G was accepted in fiscal year 1948 (February 1948), all other F-82s (F, G, and H models) in fiscal year 1949. The last F-82G and six winterized F-82Hs were received in March 1949.

Production totals

The Air Force accepted a total of 272 F-82s (including 22 prototype, test and early production aircraft). All examples were redesignated as F-82 in 1948. Specifically, the F-82 program consisted of two XF-82s, one XF-82A, 10 F-82Bs (known for a while as P-82Zs and all allocated to testing), four F-82As, 96 F-82Es, 91 F-82Fs, 45 F-82Gs and 14 F-82Hs. All models and variants of the P-82 were produced at North American's Inglewood, California manufacturing plant.[13]

Operators

 United States[6][14]
Notes for Korean War Service
  1. ^The 4th FAWS was reassigned to Twentieth Air Force and attached to the 51st Fighter-Interceptor Wing on 24 June 1950. It was subsequently attached to the provisional 6302d Air Base Group until the F-82s were replaced in 1951 with F-94 Starfires.
  2. ^The 68th FAWS was reassigned to Fifth Air Force and attached to the 347th Provisional Fighter Group (All Weather) on 27 June 1950 and then to the 8th Fighter-Bomber Wing on 5 July. It was lastly attached to the provisional 6160th Air Base Wing on 1 December 1950 until the F-82s were withdrawn in 1952.
  3. ^The 339th FAWS was reassigned to Fifth Air Force with five aircraft attached to the 347th Provisional Fighter Group (All Weather) on 27 June 1950. It was lastly attached to the provisional 6162d Air Base Wing on 1 December 1950 until the F-82s were withdrawn in 1951.

Survivors

F-82B Twin Mustang AF s/n 44-65162 at Midland International Airport
Wright-Patterson Air Force Base is home to two Twin Mustangs. Betty Jo, pictured here, is one of them.

Five F-82s are known to still exist.

  • F-82B Twin Mustang, AF s/n44-65162, is on display at the National Museum of the United States Air Force at Wright-Patterson AFB in Dayton, Ohio.[15] It had been a "gate guard" for many years outside Lackland AFB in Texas, was acquired by the Commemorative Air Force in 1966 and was operated for many years by its Midland, Texas squadron. That F-82B stalled while landing at Valley Regional Airport in Harlingen, Texas in 1987. The aircraft was restorable but its unique props and landing gear were destroyed in the crash and replacement parts could not be obtained. In 2002, the CAF included it with a crashed P-38 in a trade for a flyable P-38 from a private organization, NPA Holdings. The USAF stepped in and demanded the F-82 be returned since it was only loaned to the CAF on the condition that the CAF return the aircraft to the U.S. Air Force if it was no longer wanted. In a published opinion, the U.S. Sixth Circuit Court of Appeals ruled in December 2009 that the F-82B must be returned to the National Museum of the U.S. Air Force from NPA Holdings because the CAF violated the terms of the loan agreement with the U.S. Air Force.[16] This aircraft arrived at the National Museum of the United States Air Force in late 2009 and underwent an extensive restoration throughout 2010, modifying the aircraft to the appearance of an F-82G. In early 2011, the aircraft was placed on display in the museum's Korean War Gallery, modified and marked as the F-82G crewed by 1st Lt Charles Moran, pilot, and 1st Lt Fred Larkins, radar observer, 68th F(AW)S, when they shot down a North Korean La-7 on 27 June 1950, near Kimpo Air Base, South Korea. The aircraft is displayed without the F-82G's radar "long dong" pod, as none have been located.
  • F-82B Twin Mustang "Betty Jo", AF s/n 44-65168, is currently on display at the National Museum of the United States Air Force at Wright-Patterson AFB in Dayton, Ohio in its Cold War gallery.[17] The aircraft is displayed in the markings of its historic Hawaii to New York flight in 1947, along with clothing and artifacts used in that record-breaking flight. "Betty Jo" was delivered to the Museum on 21 June 1957.
  • XP-82 Twin Mustang, AF s/n 44-83887 and FAA tail number N887XP,[18] is currently under restoration to flying status by the B-25 Group, headed by aircraft restorer Tom Reilly, at Douglas Municipal Airport in Douglas, Georgia. A single fuselage and parts of the second XP-82 were located for many years on the farm of Walter Soplata in Newbury, Ohio; these were sold in April 2008 to Reilly and shipped to his facility in Douglas. With additional airframe parts sourced from Alaska, Colorado, and Florida; a left-turning engine found in Mexico City; control components from California; and other items fabricated at the Douglas facility, Reilly believes the aircraft can be restored to flying condition. During the restoration process, a number of written notes and graffiti by the original construction workers were found on internal surfaces. These have been preserved and will be reapplied to the interior of the finished aircraft.[19][20]
  • F-82E Twin Mustang, AF s/n 46-0256, NACA s/n 133 and FAA tail number N142AM,[21] is currently under restoration to flying status by James Harker in Anoka, Minnesota (previously in Birchwood, Wisconsin). It was an intact airframe formerly located at the Walter Soplata farm.[22]
  • F-82E Twin Mustang, AF s/n 46-0262, is currently on display as a "gate guard" at Lackland AFB in Texas as part of the USAF History and Traditions Museum in San Antonio, Texas.[23]

Specifications (F-82G)

Data from The Concise Guide to American Aircraft of World War II[24]

General characteristics

Performance

Armament

See also

Related development
Aircraft of comparable role, configuration and era
Related lists

References

Notes

  1. ^Note the "EFQ" Three-digit Buzz Code, the NACA symbol on the tail and the full serial number. Also the aircraft was still under the jurisdiction of the USAF while under NACA's control showing the USAF National Insignia and U.S. Air Force painted on the tail. Note the air-breathing ramjet under the wing. When retired in 1957, this F-82 was the last flyable Twin Mustang under USAF ownership.

Citations

  1. ^ abcdefgKnaack 1978, pp. 19–20.
  2. ^Carey, Alan C. (2014). Twin Mustang: The North American F-82 at War. Images of War. Pen and Sword Aviation. p. 20. ISBN 978-1-78346-221-6.
  3. ^Dorr & Donald 1990, p. 130.
  4. ^ abcdDean 1987, pp. 20–43.
  5. ^ abcdefPape 1977, pp. 48–63.
  6. ^ abcdefghijklmThompson 2001, pp. 22–37.
  7. ^ abBaugher, Joe; Kuris, Jeremy. "Aircraft Serial Number Search: F-82". RCN.com. Retrieved 23 January 2011.
  8. ^ abcdefgThompson 2003, pp. 28–41.
  9. ^"Top of the World (1955)". IMDb.com. Retrieved 26 August 2012.
  10. ^ abcdeCloe & Monaghan 1984, p. 188.
  11. ^Scott 2009, p. 12.
  12. ^O'Leary, Michael (August 1998). "Polar Acquires Twin Mustang". Air Classics. Challenge Publications.[dead link]
  13. ^Dean 2000.[page needed]
  14. ^Maurer 1969.
  15. ^"North American F-82G Twin Mustang". National Museum of the U.S. Air Force. 11 February 2011. Retrieved 17 October 2016.
  16. ^"Appeals Court: Historic Plane To Be Returned". WHIOtv.com. 30 December 2009. Retrieved 7 July 2011.
  17. ^"North American F-82B Twin Mustang". National Museum of the U.S. Air Force. 18 March 2010. Retrieved 17 October 2016.
  18. ^"FAA Registry: N887XP". FAA.gov. Retrieved 12 April 2012.
  19. ^Barendse, Tom; Barendse, Louisa (13 January 2011). "Rebuilding the Prototype XP-82 Twin Mustang". EAA.org. Archived from the original on 17 January 2011. Retrieved 30 September 2013.
  20. ^"The Restoration to Flying Condition of XP-82 Twin Mustang Prototype 44-83887". Warbirds News. 16 June 2013. Retrieved 30 September 2013.
  21. ^"FAA Registry: N142AM". FAA.gov. Retrieved 14 January 2013.
  22. ^"North American P-82/F-82 Twin Mustang: 144-38142". Aerial Visuals Airframe Dossier. Retrieved 14 January 2013.
  23. ^"North American P-82/F-82 Twin Mustang: 46-262". Aerial Visuals Airframe Dossier. Retrieved 6 March 2015.
  24. ^Mondey 1994, pp. 205–206.
  25. ^Dorr, Robert F.; Lake, Jon; Thompson, Warren (1995). "From Props to Jets". Korean War Aces. London: Osprey Aerospace. p. 9. ISBN 1-85532-501-2.

Bibliography

  • Cloe, John Haile; Michael F. Monaghan (1984). Top Cover for America: The Air Force in Alaska, 1920–1983. Missoula, Montana: Pictorial Histories. ISBN 0-933126-47-6.
  • Dean, Francis H (2000). America's Hundred Thousand: U.S. Production Fighters of World War II. Atglen, Pennsylvania: Schiffer. ISBN 0-7643-0072-5.
  • Dean, Jack (September 1987). "The Lonely Long Ranger". Airpower. Sentry Books. 17 (5).
  • Dorr, Robert F; David Donald (1990). Fighters of the United States Air Force. London: Temple. ISBN 0-600-55094-X.
  • Knaack, Marcelle Size (1978). Encyclopedia of US Air Force Aircraft and Missile Systems, Volume 1: Post-World War II Fighters, 1945–1973. Washington, D.C.: Office of Air Force History. pp. 13–21. OCLC 227502707. Accession no. ADA057002.
  • Maurer, Maurer (1969). Combat Squadrons of the Air Force, World War II. Washington, D.C.: Office of Air Force History. OCLC 72556. Accession no. ADA128026.
  • Mondey, David (1994) [1982]. "F-82 Twin Mustang". The Concise Guide to American Aircraft of World War II. Edison, NJ: Chartwell. ISBN 0-7858-0147-2.
  • Pape, Garry R (November 1977). "Escort Fighter: Flying the F-82 Twin Mustang in the Cold War". Wings. Vol. 7 no. 6. Sentry Books.
  • Scott, Phil (January 2009). "Tug of Warbirds". Air & Space/Smithsonian. Smithsonian Enterprises. 23 (5): 12.
  • Thompson, Warren E (March 2001). "Double Trouble, The F-82 Twin Mustang in Korea". Air Enthusiast. No. 93. Key Publishing.
  • Thompson, Warren E (October 2003). "Alaskan Twin Mustangs, North American F-82 Operations in Alaska". Wings. Sentry Books. 33 (5).

Further reading

External links


Designing a Tree Diff Algorithm Using Dynamic Programming and A*

$
0
0

17 June 2017

During my internship at Jane Street, one of my projects was a config editing tool that at first sounded straightforward but culminated in me designing a custom tree diffing algorithm using dynamic programming, relentlessly optimizing it and then transforming it into an A* accelerated path finding algorithm. This post is the story of the design and optimization of the algorithm, of interest to anyone who needs an algorithm for diffing trees, or who just wants an in-depth example of the process of solving a real-world problem with a custom non-trivial dynamic programming algorithm, and some tips on optimizing one while maintaining understandability, or anyone who just wants to read a cool programming story and maybe learn something.

Table of Contents

  • Background: Description of the problem I was solving.
  • The Heuristic Approach: My initial attempt at a simple algorithm, and why it was insufficient.
  • A Tree Diff Optimizer: Rethinking the problem as finding the optimal resulting tree diff.
  • Dynamic Programming: Background on using the correspondence between memoizing recursion and path finding on a grid to find solutions to problems like Levenshtein distance.
  • The Algorithm: Extending Levenshtein distance with two grids and recursion.
  • Profiling and Optimizing: Relentless profiling and optimizations to reduce run time.
  • Path Finding: Using A* to make the run time proportional to the difference rather than tree size.
  • Example Implementation: Analysis of the effectiveness and costs of A* on Levenshtein distance, sequence alignment and diffing problems, with open source example code.

Background

Jane Street has a lot of config files for their trade handling systems which use S-expressinons (basically a tree with strings as leaves). They often want to make changes that will only apply on certain days, for example days when markets will act differently than normal like elections and option expiry dates. To do this their config processor knows a special construct that is like a switch statement for dates, it looks something like this:

(thing-processor-config(:date-switch(case2017-04-07(speed5)(power7))(else(speed3)(power9))))

The semantics are that any :date-switch block has its branches checked against the current date and the children of the correct branch are used in place of the :date-switch in the config structure. Note that each branch can contain multiple sub-trees.

Now, just that small example took a while for me to type and get the indentation correct. People at Jane Street are frequently making edits where they just want to quickly change some numbers but have to make sure they get the syntax and indentation right and have everything still look nice. This is begging for automation!

So they asked me to write a tool that would allow people to just edit the file and run a command, and it would automatically scope their changes to the current day, or a date they specify, by detecting the differences between the current contents on disk and the committed contents in version control.

When I first heard the problem, it sounded pretty easy and I thought I’d be done within a few days, but I discovered a number of catches after starting and it ended up taking 5 weeks. This sounds like we maybe should have abandoned it when we discovered how complex it was, but given how much time expensive people were spending doing these edits, sometimes under time pressure when every extra minute had a high cost, it was worth it to get it right.

The first thing I discovered is that they had a library for parsing and serializing s-expressions while preserving indentation and comments, but it didn’t cleanly handle making structural modifications to the parse tree. Before even starting the rest of the project I had to write a library that provided a better data model for doing this and having the result be nicely indented.

Next, the real syntax for these date switch blocks is more complicated than my description and has a static constraint where the branches must cover every date in the current context and no more, including when nested inside other switches. I also didn’t want edits to :date-switch blocks to themselves be scoped to a date, since that would create invalid syntax. This required I parse my earlier style-preserving representation into one that computed date context and represented :date-switch blocks specially in an Abstract Syntax Tree (AST) as well as transform back from my representation to the one I could output.

Now finally I was ready to start on the actual algorithm. The basic task my script had to do was take two trees and wrap differing sub-trees in a :date-switch block with the old contents in one branch and the new contents in the other branch. The thing is, there are many ways to do this. The switch can be placed at many different levels and there may be multiple edits and it’s not specified how to group them. Technically it could just add a :date-switch at the top level with the old file in one branch and the new file in another, but that wouldn’t be very satisfying, just like how technically the diff command could just output the entire old file prefixed with - and then the new file prefixed with +, but then nobody would use it. I needed an algorithm that gave reasonable-looking and compact changes for real-world edits. It shouldn’t double the file size in an enormous :date-switch when only a single number changed.

The Heuristic Approach

If you just want to read about the optimal algorithm and not why one was necessary you can skip this section.

First, I came up with a simple algorithm that I thought would work in almost all real-world cases. I simply recursively walked down the tree until I got to either a leaf that was different or a node that had a different number of children, and then it would put a :date-switch at that level.

This didn’t produce the most satisfying results, it was okay for changes to leaves, but as soon as you added or removed a child from a list, it would duplicate most of the list when it could have taken advantage of the ability to have a different number of children in each branch.

It produced this:

(:date-switch(case2017-04-07(fooqux; ... 1000 more lines ...baz))(else(foo; ... 1000 more lines ...baz)))

When we really would have preferred:

(foo(:date-switch(case2017-04-07qux)(else)); ... 1000 more lines ...baz)

Luckily, this was easy enough to solve since Jane Street already had an OCaml library implementing the Patience Diff algorithm for arbitrary lists of comparable OCaml types. When I had two lists of differing length, I simply applied the Patience Diff algorithm and placed :date-switch blocks based on the diff.

This algorithm worked okay for cases of a single edit, but we wanted the tool to work with multiple edits, and for those it still often produced terrible results.

For example, because it stopped recursing and applied a diff as soon as it reached a list of changing length, it would do this:

(this(:date-switch(case2017-04-07(bar; ... 1000 more lines ...baz))(else(foo; ... 1000 more lines ...baz)))tests(:date-switch(case2017-04-07addingtoalistandmodifyingasub-tree)(else)))

It duplicates the long sub-tree instead of placing the :date-switch lower down in it, just because we made another edit at a higher level of the tree.

There were a number of other cases where it didn’t produce output as nice as a human would, but earlier on my mentor and I had sighed and accepted it. This though was the last straw, we needed a new approach…

A Tree Diff Optimizer

At this point I was fed up with constantly discovering failure modes of algorithms I thought should work on real-world cases, so I decided to design an algorithm that found the optimal solution.

I started by searching the Internet for tree diff algorithms, but every algorithm I found was either a different kind of diff than what I needed, or was complex enough that I wasn’t willing to spend the time understanding it (probably only to later find out it computed a different kind of diff than I needed).

Specifically, what I needed was mostly like a tree diff but I wasn’t optimizing for the same thing as other algorithms, what I wanted to optimize for was resulting file size, including indentation. This I thought represented what I wanted fairly well, and captured why previous results which duplicated large parts of the file were bad. As well as the character cost of the branches, each additional :date-switch block added more characters. Additionally, each switch construct could also contain multiple sub-trees in each branch, which I needed to model to account for overhead correctly.

Consider the case of (a b c) becoming (A b C). A human would write:

(:date-switch(case2017-04-07(AbC))(else(abc)))

Despite the fact that the b is duplicated, this is the smallest number of characters. However if we had something longer we’d want something different, so the optimal result even depends on the length of leaf nodes:

((:date-switch(case2017-04-07A)(elsea))this_is_a_super_long_identifier_we_do_not_want_duplicated_because_it_is_looong(:date-switch(case2017-04-07C)(elsec)))

A very common real-world example of why it is important for it to be able to batch together differences is when changing multiple values of a data structure. The config files often have lots of key-value pairs and edits often touch many nearby values:

(thing-processor-config(:date-switch(case2017-04-07(speed5)(size80)(power9001))(else(speed3)(size80)(power7))))

Even though size didn’t change, we duplicate it because it’s cleaner than having two :date-switch blocks.

Dynamic Programming

After 2+ days of research, discussing ideas with my mentor, and sitting down in a chair staring out at the nice view of London while thinking and sketching out cases of the algorithm in a notebook, I had something. It was a recursive dynamic programming algorithm that checked every possible way of placing the :by-date blocks and chose the best, but used memoization (that’s the dynamic programming part) so that it re-used sub-problems and had polynomial instead of exponential complexity.

The core of the tree diffing problem is similar to the Levenshtein Distance problem, you have two lists that have some number of insertions and deletions between them, and you want to find the best way to match them up. You can do this with a number of different cases based on the first elements of the lists with different costs, calculating those costs involves some constant plus recursively computing the cost for the rest of the lists. Then you compute the cost for each possible decision and take the minimum one.

For example if you have a function best_cost(old,new) to solve the Levenshtein distance problem, there’s three cases for the start of the lists: insert, delete and same. The simplest case is if the first two elements are the same, and characters that are the same cost 0, then the cost is just best_cost(old[1..], new[1..]). If a delete costs 1, then if the start of the list is a delete that means the character is in old but not new so the total cost is 1 + best_cost(old[1..], new). Insert is similar but the opposite direction. This recursion terminates with the base case of best_cost([],[]) = 0. The problem is that this leads to an exponential number of recursive calls.

But we can fix this by noticing that a lot of things are being computed redundantly and sharing the results by “memoizing” the function so that it stores the results for arguments it has been called with before. As seen in the diagram below, where the numbers on the nodes represent calls to best_cost(old[i..], new[j..]) as i,j:

Decision tree diagram

But in some cases it can be difficult to think about the problem as a recursive memoized decision tree. Luckily there’s a different way of thinking about it that lends itself very well to sketching out algorithms in a notebook or on a whiteboard. We can rotate the tree 45 degrees and notice that we can think about it as a grid:

Decision tree diagram rotated to be a grid

This is useful for memoization since it means we can store our results in an efficient-to-access 2D array, but also because we can now think of our problem as finding a path from the top left of a grid to the bottom right using a certain set of moves. Whenever our decisions are constrained we can annotate the grid where the input lists have a property, like two items being the same. In the example below, we have a diagram of trying to find the Levenshtein distance from “abcd” to “bcad”, with the best path bolded, and an alternative more costly path our algorithm might explore shown dashed.

Levenshtein distance paths

We can find the best path by testing all the paths and returning the best one. There are exponentially many paths, but we can notice that the best path from a point in the middle of the grid to the bottom right is always the same no matter what moves might have gotten us to that point.

One way to exploit this is to recursively search all paths from the top left, memoizing the best path at each point so we don’t compute it again. This corresponds to the memoizing of recursive functions mentioned earlier and is called the “top-down approach” to dynamic programming.

There’s also the “bottom up” approach where you start from the bottom right and fill in each intersection with the best path based on previously computed results or base cases by using an order where everything you need is always available. In this case it would be right to left, top to bottom, like reading a book from end to start.

Now we know that if we can restate our tree diffing problem as a problem of path-finding on a graph, we can turn that into an implementation using dynamic programming.

The Algorithm

The key differences between my problem and Levenshtein distances were the fact that it was a tree and not a list, and the fact that consecutive sequences of inserts/deletes were cheaper than separate ones (because consecutive edits could be combined in one :date-switch block). My cost function is also different in that I’m measuring the size of the resulting tree including :date-switch blocks, so my moves will need costs based on that.

I can extend the list algorithm to trees by adding a move I’ll call “recurse” that goes down and right and can be done on any square where both items are sub-trees (not leaves). The cost of the move is the cost of the resulting diff from running the entire tree-diff algorithm recursively on those two sub-trees. I don’t bother recursing if the two sub-trees are the same, since the “same” move has identical cost in that case, and is faster to compute.

We can handle the cheaper consecutive inserts and deletes by modeling entering and leaving a :date-switch block as moves. However now we have different move sets based on if we are in or out of a :date-switch block and different costs to get to the end from a given point. We can rectify this by splitting our problem into path-finding over two grids of the same size. One is our “outside” grid, where we can do the “same”, “recurse” and now also a “in” move which moves us to the same position on the other grid.

On the “inside” grid we can do “insert”, “delete” and “out” moves. But that won’t quite work because if “in” and “out” both don’t make forward progress, the graph has a cycle and our search algorithm will endlessly recurse over paths going “in” and “out” at the same point. We can solve this by splitting “out” into “insert out” and “delete out”. The first two are the same as insert and delete except they also move to the “outside” grid, we also have to make sure that we don’t use the “insert” and “delete” moves to go to the bottom right of the “inside” grid, because then we’d be stuck.

This gives us a set of moves that always make forward progress and share as much as possible, with this we can find the best path and that gives us an optimal diff. See the diagram below which also includes the cost of each move and an example path, although not necessarily the optimal one:

Tree diff diagram

Even this model is simplified, because in reality I had to handle input lists that both might have :date-switch blocks already in them, so there were a bunch more cases and contingencies for handling existing :date-switch blocks properly. But those aren’t very interesting and the core of the algorithm is the same.

So I implemented this algorithm on top of the AST manipulation framework I’d built by translating it to a memoized recursive algorithm operating on linked lists. Since the outer algorithm also involved recursion, this meant I had two kinds of recursion, which I structured using OCaml’s ability to define functions inside of other functions. I had an outer scope_diffs function that took two lists of s-expressions and produced a list of s-expressions with differences scoped by :date-switch blocks. Inside it, I allocated two tables to memoize the results in, and defined scope_suffix_diffs_inside and scope_suffix_diffs_outside functions that took indices of the start of the suffixes and mutually recursed and memoized into the tables based on the moves above.

Unlike the Levenshtein difference algorithm I wanted more than just the cost, so I stored the actual scoped s-expressions up to each point in the table directly, because I was using immutable linked lists in OCaml this was memory-efficient since each entry would share structure with the entries it was built from. This way I avoided the back-tracing path reconstruction step that is frequently used with dynamic programming. In order to make the lists share structure I did have to add to the front instead of the back, but I just reversed the best resulting list before I returned it.

Once I finished programming it and got it to compile, I think I only had to fix one place where I’d copy-pasted a +1 where I shouldn’t have and then it worked beautifully. Finally, unlike all my heuristic attempts, I couldn’t find a case where this produced a result significantly worse than what a human would do.

Side note: I used to expect lots of debugging time whenever I finished a bunch of complex algorithmic code, but to my surprise I’ve found that’s rarely the case when using languages with good type systems. The compiler catches almost all small implementation errors, and since I’ve usually spent a long time thinking about all the edge cases carefully while designing the algorithm, there’s usually no serious bugs left by the time it compiles. My tests usually fail a few times, but that’s normally because I wrote the tests wrong.

Unfortunately, while my new algorithm worked quite well for small files, it was very slow on large files. My mentor timed it on a few example files and fit a polynomial and discovered that it was empirically O(n^3) (or it might have been O(n^4) I forget) in the file size. This was unfortunate since some of the files were tens of thousands of lines long. I had to make it faster, luckily while I’d been thinking about and implementing the algorithm I’d accumulated quite a list of optimization ideas to try. But first, I decided to profile to see what the biggest costs were.

Profiling and Optimizing

Incremental cost computation

The first order of business was to discover why the empirical complexity was higher than we thought it should be. My mentor and I tried to come up with a proper analysis of what it should be, but given all the cases and the nested nature of trees there were just too many parameters and we couldn’t come up with anything precise. But, as far as we could tell the complexity of the underlying algorithm should have been about O(n^2*log(n)) in the length of real files.

I could have looked over the implementation carefully to find all the extra unnecessary work, but an easier method was just to use the Linux perf tool to profile it. I knew the work that caused it to be O(n^3) wasn’t at the outer levels of the algorithm, or I would have noticed easily, so it had to be an operation within that would show up in the profiles.

Sure enough, most of the program’s time was spent in the code that computed the length/cost of an s-expression. I had a function that walked a tree and computed its total length, and in the part of the algorithm where I had to choose the lowest-cost move I computed the cost of the entire path from that point, which added an extra O(n) inside the O(n^2) algorithm yielding O(n^3).

In order to fix this, I made sure every cost computation was constant time, which meant I had to construct the cost of a path incrementally as it was constructed, and also not repeatedly walk trees to determine their cost.

I solved this in three steps:

  1. Create a “costed list” type which was a linked list except each item included the cost of the suffix from that point. This had a constant-time prepend operation that just added the cost of the item being prepended to the cost field of the rest of the list.
  2. Modify the Abstract Syntax Tree (AST) data structure to include a cost field on every node, and to use a costed list for children. I also made all the AST builder functions compute the cost of their components by just adding the cost of their overhead with the costs of their child nodes or costed lists. Now both getting the cost of an AST subtree and constructing a node were constant time.
  3. When building the path/diff/result of my algorithm I used a costed list and constructed new :date-switch nodes using the constant-time builder API.

After I did this, our measurements of growth were consistent with the O(n^2*log(n)) we were expecting.

Skipping identical prefixes and suffixes

This was an easy but high-impact optimization I had written down earlier. By the properties of the cost of each move, if a prefix and/or suffix of the two lists was identical, the “same” move would always be the best for those parts of the path. This meant that I could find the longest prefix and suffix of the two lists that was the same and only run the dynamic programming algorithm on the middle part that wasn’t. This made the common case of edits being concentrated at a single point in the file very fast because now the running time was more like O(d^2*log(d)+n) with nbeing file size (large) and d being the size of the edited region (small).

Now almost all common uses of the tool would return instantly except edits in multiple places spread out through a large file. It was now a pretty useful tool, but users having to know which cases to avoid to make the tool not take forever wasn’t great. It would sometimes be used in high-pressure situations and often people did want to make edits in different places, manually batching the edits up and running the tool multiple times wasn’t ideal.

There was also only one or two weeks left in my internship, not much time to do another project, and I think my mentor was having fun challenging me to make the tool perfect and brainstorming how to do so with me. I was also enjoying the process, so the optimization would continue until performance improved!

Tree creation optimization

Profiling indicated a lot of time was spent creating :date-switch AST nodes.

First I wrote a fast-path method of creating and computing cost for :date-switch blocks the algorithm creates since they use a simpler format and a known indentation style than the more general AST construction builder uses.

Additionally, when exploring paths in the “inside :date-switch” table, I used to create a :date-switch node whenever I needed to know the cost so far to decide between moves. Instead, I switched to just adding the costs of the insert and delete branches (which were costed lists), to a known overhead of the :date-switch block. I only created the full node upon exiting to the “outside :date-switch” table.

But my mentor realized this could extend even further: The search for the best path only ever needs to know the cost of a resulting path/tree, we only need the full tree for the best path at the end of the search. So I added a “lazy :date-switch” AST node that has a stored cost computed by fast addition of the cost of the components plus a known overhead, but doesn’t actually create the node immediately and just stores an OCaml lazy thunk that creates it if we try to serialize the result.

More?

My tool was now instant in most common cases and difficult cases were over 100x faster. But, on the very largest 10,000+ line config files it would still take up to 5 minutes in the worst case if you made edits in multiple places. There were no longer any obvious hot spots in the profiles, I needed algorithmic improvements that let me search less possible paths.

I looked at my list of optimization ideas and there was only one left, which I had written down early on in the process when thinking about the correspondence between dynamic programming and path finding on a grid. It was just a few characters in a Markdown note that I had saved for if I was feeling ambitious and really needed it: “A*”.

Path Finding

Back when I was designing the algorithm by thinking about it as a path finding problem, I thought “hey wait, if this is a path finding problem, why not use an actual path finding algorithm?” The first thing I realized is that the memoized recursion approach I was planning on taking was just a depth first search, which can be a path finding algorithm, but not a particularly good one.

Could I use a better path finding algorithm? Breadth first search wouldn’t help much since the goal was near the maximum distance in bounds. However, A*, perhaps the most famous path finding algorithm, seemed like it might help, if only I could come up with a good heuristic. So I wrote it down without thinking too much and came back to it later after I had done all my other optimizations.

The last time I learned A* was when I read Algorithms in a Nutshell (good book) years ago and all I remembered was that it needed a priority queue and a good heuristic. I had a Heap for the priority queue, but I didn’t remember how to actually implement it or what a good heuristic was, so to Wikipedia I went!.

I learned that I needed a heuristic that never overestimated the remaining cost, and that ideally never decreased more than the cost of any move taken. One thing that satisfies those properties is the maximum of the costs of the two list suffixes from a location. This corresponds to the notion that a scoped config file that includes the contents of both config files can’t be smaller than either of the input files. This heuristic was easy to compute using the costed list representation I already had, which already has the cost of each suffix of the input lists.

With a heuristic and an understanding of A* in hand, I refactored the implementation of my algorithm to work by putting successor states in a priority queue based on their cost plus the heuristic remaining cost. This required changing each instance of recursion on my table into the creation of a State structure that encompassed if I was inside or outside of a :date-switch, and the current position.

I also had to make two changes to my data structures. First, since I was no longer using recursion to destructure my linked lists but was now indexing them, which is O(n), I created arrays of the tails of my input costed lists so that random access was fast. Next, my solution still used O(n^2) memory in all cases due to the 2D array memoization table, so I switched that to a hash table from position tuples.

Profiling now showed a lot of time was then spent in hash table lookups, so I experimented with dynamically using a 2D array for small input lists (like were often found on the lower levels of the tree) and a hash table for larger input lists, but further profiling showed it didn’t increase performance much, probably because most of the lookups were in the larger lists, so I stuck with plain hash tables.

After a little debugging of off-by-one errors, I ran the program on my largest test case and it finished instantly. It was so fast I was suspicious it was broken and just skipping all the work, but sure enough it worked perfectly in every case I threw at it. The cost was something like O(n * log(n) * e^2) where n is the file size and e is the number of edits. Me and my mentor managed to think of some edge case trees where it might revert to O(n^2) behavior, and it still only scaled to config files of tens of thousands of lines rather than hundreds of thousands, but it was nearly instant for all cases that might actually occur, so it was good enough™.

I spent the remaining 3 days of my internship polishing up the user interface of the tool, cleaning up the code and writing lots of doc comments explaining my algorithm. I also gave a presentation to a bunch of the other engineers telling a shorter version of the story I’ve written here. That marked the end of my internship with Jane Street and one of the most interesting algorithmic problems I’ve ever worked on, despite it being part of a tool for editing configuration files.

Example Implementation

I was curious about how the approach of turning a dynamic programming problem into an A* path finding problem scaled and how applicable it was to other problems. So, I developed an example implementation of this approach in Rust for the sequence alignment problem, which Levenshtein distance is a specific instance of. It’s structured for simplicity and I haven’t optimized it at all, but it’s good enough to demonstrate the asymptotic improvements.

The core code of the algorithm is fairly simple and is a good demonstration of the logic required to turn a dynamic programming algorithm into an A* path finding instance. It allows you to tune the weights of insertions/deletions, mismatches and matches of characters in the two strings, so that you can change it to be Levenshtein distance or some other instance of sequence alignment.

The heuristic it uses is based on splitting the remaining distance to the bottom right corner into two components: the minimum number of insertion/deletion moves necessary to get on the diagonal from the goal, and the minimum number of match moves necessary to get from that place on the diagonal to the goal. This represents the minimum possible cost required to reach the goal from any position without knowing what the actual best path is.

fndiag(pos:&Pos)->i32{(pos.1asi32)-(pos.0asi32)}fnheuristic(pos:&Pos,goal:&Pos)->Score{// find the distance to the diagonal the goal is onletgoal_diag=diag(goal);letour_diag=diag(pos);letindel_dist=(our_diag-goal_diag).abs();// find the distance left after moving to the diagonallettotal_dist=max(goal.0-pos.0,goal.1-pos.1)asi32;letmatch_dist=total_dist-indel_dist;return(indel_dist*INDEL)+(match_dist*MATCH);}

Heuristic diagram

Discoveries

Here’s some things I learned by fiddling with the program and timing it:

  • As expected, the algorithm only tends to explore states along the diagonal of the grid, with the width of the area explored proportional to the edit distance. This suggests the running time is something like O((a+b) * e^2) where a and b are the input lengths and e is the edit distance.
  • Running it on a 10,000 base pair gene sequence with an edit distance of 107 takes 0.26 seconds.
  • Running it on a 10 megabyte random text file with 1 edit near the beginning and 2 near the end takes 20 seconds and evaluates 40 million states. This is a case where with the O(n^2) algorithm just allocating and zeroing 4*10^14 bytes of memory for a table with the naive algorithm would take forever. Demonstrating that this optimization does in fact provide an asymptotic speed up.
  • It’s still way slower than specialized sequence alignment implementations like Edlib, probably asymptotically so. These implementations use all sorts of fancy tricks including fancy algorithmic tricks, SIMD and bit vectors to eke out maximum performance for bioinformatics applications. My implementation is at least way way simpler.
  • Plain Dijkstra’s algorithm (A* with a heuristic always returning 0) actually performs almost as well for Levenshtein distance because only edits have cost so it explores along the diagonal towards the goal along the path with less edits just because those have lower cost. However, if the problem has a cost for matching portions as well (like my original tree diffing problem) Dijkstra’s algorithm will explore in a blob expanding from the top left and be almost as slow as the naive algorithm.
  • With a heuristic, Levenshtein-distance like instances where only edits have cost take about the same time to run as instances like my tree-diffing problem where matching segments also have cost.

Conclusion

A* is an interesting technique that’s an easy way to accelerate a class of dynamic programming problems. It definitely works on any vaguely diff or edit-distance like problem, but it might extend to even more. If you want absolute peak performance on a simple algorithm there’s probably better techniques to use, I’d start by looking at what bioinformatics people do, but if you just want something easy and flexible this seems like a good technique, and it’s not one I’ve seen done before, and Google doesn’t turn up any results. It might even be novel, or it might just be that A* is a hard term to Google for, I’m interested to hear from anybody who’s seen something like this before.


Car Wash Hack Can Strike Vehicle, Trap Passengers, Douse Them with Water

$
0
0

The security problems found in internet-enabled medical equipment and cars in recent years have raised a lot of awareness about the public safety risks of connected devices. But it's not just life-saving implements and fast-moving vehicles that pose potential harm.

A group of security researchers have found vulnerabilities in internet-connected drive-through car washes that would let hackers remotely hijack the systems to physically attack vehicles and their occupants. The vulnerabilities would let an attacker open and close the bay doors on a car wash to trap vehicles inside the chamber, or strike them with the doors, damaging them and possibly injuring occupants.

"We believe this to be the first exploit of a connected device that causes the device to physically attack someone," Billy Rios, the founder of Whitescope security, told Motherboard. Rios conducted the research with Jonathan Butts of QED Secure Solutions. They plan to discuss their findings this week at the Black Hat security conference in Las Vegas.

Rios, working at times alone and with colleagues, has exposed many security problems over the years in drug-infusion pumps that deliver medicine to hospital patients; in airport x-ray machines designed to detect weapons; and in building systems that control electronic door locks, alarm systems, lights, elevators, and video surveillance cameras.

An attacker can send an instantaneous command to close one or both doors to trap the vehicle inside, or open and close one door repeatedly to strike the vehicle a number of times as a driver tries to flee.

This time his focus was on the PDQ LaserWash, a fully-automated, brushless, touchless car wash system that sprays water and wax through a mechanical arm that moves around a vehicle. PDQ car washes are popular throughout the US because they don't require attendants to operate. Many of the facilities have bay doors at the entrance and exit that can be programmed to automatically open and close at the start and end of a day, and a touchscreen menu that allows drivers to choose their cleaning package without interacting with any workers.

The systems run on Windows CE and have a built-in web server that lets technicians configure and monitor them over the internet. And herein lies the problem.

Rios says he became interested in the car washes after hearing from a friend about an accident that occurred years ago when technicians misconfigured one in a way that caused the mechanical arm to strike a minivan and douse the family inside with water. The driver damaged the vehicle and car wash as he accelerated quickly to escape.

A successful trip through the car wash. Researchers could not obtain permission to publish video of the hack from car wash owners.

Rios and McCorkle examined the PDQ software two years ago and presented their findings about vulnerabilities at the Kaspersky Security Summit in Mexico in 2015. Although they believed the vulnerabilities would allow them to hijack a system, they weren't able to test the theory against an actual car wash until this year when a facility in Washington state agreed to cooperate, using the researchers' own pickup truck as the victim.

Although the PDQ systems require a username and password to access them online, the default password is easily guessed, the researchers said. They also found a vulnerability in the implementation of the authentication process, making it possible to bypass it. Not all PDQ systems are online, but the researchers found more than 150 that were, using the Shodan search engine that searches for devices connected to the internet, such as webcams, printers, industrial control systems, and, in this case, car washes.

They could also manipulate the mechanical arm to hit the vehicle or spew water continuously, making it difficult for a trapped occupant to exit the car.

They wrote a fully automated attack script that bypasses authentication, monitors when a vehicle is getting ready to exit the wash chamber and cause the exit door to strike the vehicle at the appropriate time. All an attacker has to do is choose the IP address for the car wash they want to attack, then launch the script. The car wash's software tracks where a carwash is in its cycle, making it easy to know when the wash is about to end and a vehicle to exit. An attacker can send an instantaneous command to close one or both doors to trap the vehicle inside, or open and close one door repeatedly to strike the vehicle a number of times as a driver tries to flee.

Although infrared sensors detect when something is in a door's path to prevent this from happening, the researchers were able to cause the system to ignore the sensors. They could also manipulate the mechanical arm to hit the vehicle or spew water continuously, making it difficult for a trapped occupant to exit the car. They didn't try this during their live tests, however, to avoid damaging the arm.

A software-based safety mechanism prevents the arm from hitting a vehicle normally, but they were able to disable this, too.

"If you're relying purely on software safety, it's not going to work if there's an exploit in play," Rios said in an interview. "The only thing that's going to work [in this scenario] is hardware safety mechanisms."

Although the researchers filmed the tests with a mobile phone, the car wash owner won't let them publish the video.

This isn't the first time someone has hijacked a robotics system. In May, researchers at Trend Micro showed how they could recalibrate a robotic arm used in manufacturing plants to alter its movement. But the car wash attack has "broader potential impact to the masses," Rios said. "There aren't actually that many things … that are in the public space… and can [be made to] hit you."

The researchers reported their findings to the Department of Homeland Security and the vendor and are releasing a report this week in conjunction with their Black Hat talk.

A spokesperson for PDQ told Motherboard in an email that it is "aware" of the Black Hat talk and is working on investigating and fixing the security issues with the system.

"All systems—especially internet-connected ones—must be configured with security in mind," Gerald Hanrahan of PDQ wrote. "This includes ensuring that the systems are behind a network firewall, and ensuring that all default passwords have been changed. Our technical support team is standing ready to discuss these issues with any of our customers."

Get six of our favorite Motherboard stories every day by signing up for our newsletter.

Cooling the Earth with a cloud of small spacecraft near the inner Lagrange point

$
0
0
  1. Roger Angel*
  1. University of Arizona, Steward Observatory, 933 North Cherry Avenue, Tucson, AZ 85721
  1. Contributed by Roger Angel, September 18, 2006

Abstract

If it were to become apparent that dangerous changes in global climate were inevitable, despite greenhouse gas controls, active methods to cool the Earth on an emergency basis might be desirable. The concept considered here is to block 1.8% of the solar flux with a space sunshade orbited near the inner Lagrange point (L1), in-line between the Earth and sun. Following the work of J. Early [Early, JT (1989) J Br Interplanet Soc 42:567–569], transparent material would be used to deflect the sunlight, rather than to absorb it, to minimize the shift in balance out from L1 caused by radiation pressure. Three advances aimed at practical implementation are presented. First is an optical design for a very thin refractive screen with low reflectivity, leading to a total sunshade mass of ≈20 million tons. Second is a concept aimed at reducing transportation cost to $50/kg by using electromagnetic acceleration to escape Earth's gravity, followed by ion propulsion. Third is an implementation of the sunshade as a cloud of many spacecraft, autonomously stabilized by modulating solar radiation pressure. These meter-sized “flyers” would be assembled completely before launch, avoiding any need for construction or unfolding in space. They would weigh a gram each, be launched in stacks of 800,000, and remain for a projected lifetime of 50 years within a 100,000-km-long cloud. The concept builds on existing technologies. It seems feasible that it could be developed and deployed in ≈25 years at a cost of a few trillion dollars, <0.5% of world gross domestic product (GDP) over that time.

Projections by the Intergovernmental Panel on Climate Change are for global temperature to rise between 1.5 and 4.5°C by 2100 (1), but recent studies suggest a larger range of uncertainty. Increases as high as 11°C might be possible given CO2 stabilizing at twice preindustrial content (2). Holding to even this level of CO2 will require major use of alternative energy sources and improvements in efficiency (3). Unfortunately, global warming reasonably could be expected to take the form of abrupt and unpredictable changes, rather than a gradual increase (4). If it were to become apparent over the next decade or two that disastrous climate change driven by warming was in fact likely or even in progress, then a method to reduce the sun's heat input would become an emergency priority. A 1.8% reduction is projected to fully reverse the warming effect of a doubling of CO2 (5), although not the chemical effects.

One way known to reduce heat input, observed after volcanic eruptions, is to increase aerosol scattering in the stratosphere (6). Deployment of 3 to 5 million tons/year of sulfur would be needed to mitigate a doubling of CO2. This amount is not incompatible with a major reduction in the current atmospheric sulfur pollution of 55 million tons/year that goes mostly into the troposphere. The approach we examine here to reduce solar warming is to scatter away sunlight in space before it enters the Earth's atmosphere. The preferred location is near the Earth–sun inner Lagrange point (L1) in an orbit with the same 1-year period as the Earth, in-line with the sun at a distance ≥1.5 million km (Gm) (Fig. 1). From this distance, the penumbra shadow covers and thus cools the entire planet.

Fig. 1.

Shadowing geometry. (Left) Schematic. The L1 point and the common Earth–moon barycenter remain in-line as they both orbit the sun with a 1-year period (not to scale). (Right) Time-averaged view from Earth. The Earth wobbles with a 1-month period relative to the penumbral shadows cast from a sunshade at 1.5 and 2.4 Gm (dashed circles).

A major technical hurdle to be overcome is the instability of the orbit, which is at a saddle point. A cloud of scattering particles introduced there would dissipate in a few months. But a cloud of spacecraft holding their orbits by active station-keeping could have a lifetime of many decades. Stabilizing forces could be obtained by modulating solar radiation pressure, with no need for expendable propellants. The same controls could be used, if desired, to stop the cooling at any time by displacing the orbit slightly. In addition to longevity, space shading has the advantages that the composition of the atmosphere and ocean would not be altered further, beyond their loading with greenhouse gases, and because only a single parameter is modified, the flux of solar radiation, the results should be predictable.

Because of its enormous area and the mass required, shading from space has in the past been regarded as requiring manufacture in space from lunar or asteroid material and, thus, as rather futuristic. Here we explore quantitatively an approach aimed at a relatively near-term solution in which the sunshade would be manufactured completely and launched from Earth, and it would take the form of many small autonomous spacecraft (“flyers”).

Shading Efficiency and Radiation Pressure

Early (7) recognized that the orbit of a lightweight sunshade would be disturbed by radiation pressure. With the balance point moved farther away from L1 toward the sun, the area would need to be increased for a given flux reduction. This effect can be characterized by the blocking efficiency ε, defined as the fraction of the light blocked by a spacecraft that otherwise would have illuminated the Earth. It depends on the Earth's motion within the Earth–moon system as well as the orbital distance. Although the barycenter of the combined system and the L1 point sweep around the sun with uniform angular speed, the Earth's wobble in reaction to the moon can carry it partly out of the penumbral shadow (Fig. 1Right). The average efficiency taken over a month, allowing for solar limb darkening, is plotted as a function of distance in Fig. 2a. ε = 68% for L1 at distance 1.5 Gm, and it drops to 25% at 3 Gm. To reduce the solar flux by a fraction f, the total area A of sunlight that must be blocked by the spacecraft at given distance is given by A = fπRE2/ε, where RE is the Earth's radius. The sunshade area for our goal of f = 0.018 varies from 3.4 million km2 at 1.5 Gm distance to 9.4 million km2 at 3 Gm (Fig. 2a). The total mass of the sunshade is given by M = Aρs, where ρs is its average areal density. It is shown as a function of equilibrium orbital radius in Fig. 2b for densities from 2.5 to 40 g/m2.

Fig. 2.

Sunshade properties for 1.8% flux reduction. (a) Shadowing efficiency and total area. (b) Total mass for different reflectivities R and areal densities ρs in g/m2.

The equilibrium orbital radius depends on the strength of radiation pressure. It is convenient to define the effective reflectivityR as the ratio of the pressure experienced by the sunshade to the maximum possible for 100% specular reflection at normal incidence. For orbits with the same 1-year period as the Earth, the balance between gravity and radiation pressure is then given by:Formula Here, r is the orbital radius from the sun, ωE and rE are the angular frequency and radius of the Earth's orbit, MS and LS are the mass and luminosity of the sun, and ME is the Earth's mass. The equilibrium orbital radius depends on the ratio Rs, and Eq. 1 allows the determination of ρs (and hence total sunshade mass) as a function of R and orbital distance rEr. Fig. 2b shows results for f = 0.018, with reflectivities R from 3.2% to 100%. The figure can be used to determine any two of the quantities M, rrE, R, and ρs, given the other two.

In general, the total mass is reduced for sunshades with low areal density, but very low densities can be orbited near the L1 point only if they have very low reflectivity to minimize radiation pressure. For sunshades with density ≤40 g/m2, for any given reflectivity, the total mass is minimized at a distance of ≈2.5 Gm. Thus, for a high reflectivity (R∼ 1), the density required at this distance is 40 g/m2 and the mass is ≈270 million tons, marked “McInnes (1)” in Fig. 2b. Such a sunshade might be manufactured in space from an iron asteroid, which would have to be formed into ≈10-μm-thick foil (8). An opaque sunshade could be built with lower mass if its reflectivity were reduced by applying coatings that absorb light energy on the sunward side and reemit it as heat mostly on the Earthward side (8). Reflectivity as low as R = 0.3 might be achievable, given a sun-side coating with 90% solar absorption and 10% emissivity. The corresponding minimum mass at 2.5 Gm would be 80 million tons, marked “McInnes (2)” in Fig. 2b.

Further reduction of the overall mass will be crucially important for a sunshade that could be launched relatively soon from Earth. To achieve the required lower reflectivities, a transparent screen is needed that deflects the transmitted sunlight by a couple of degrees, enough to miss the Earth but not enough to transfer significant radiation pressure. Early (6) envisaged a 10-μm-thick glass Fresnel screen with dielectric reflectivity R = 8% and areal density 25 g/m2. Together with 5 g/m2 of supporting structure, ρs≈ 30 g/m2. The equilibrium distance is then 1.58 Gm, and for f = 1.8% the required area is 3.6 million km2. But, still, the mass is high at 100 million tons (marked “Early” in Fig. 2b).

A more efficient optical design is needed to deflect the light with a screen of lower areal density. For example, Fig. 2 shows that a sunshade with R = 10% and ρs = 5.6 g/m2 could be orbited at 2.25 Gm distance, where it would need area 5 million km2 and would weigh 27 million tons. A still lower mass of 11 million tons could be achieved with R = 3.2% and ρs = 2.5 g/m2. The next section explores the physical lower limits of density and reflectivity for refractive screens.

Optical Design to Minimize Mass and Reflectivity

The lightest refracting screen that can significantly reduce on-axis transmission is a very thin transparent film pierced with small holes making up half the total area. The thickness is chosen such that transmitted light at the peak wavelengthλp of sunlight is retarded by half a wave. Destructive interference then takes place for the directly transmitted beam at λp, with the energy deflected into diffraction peaks at angle θ = λp/2s, where s is the separation of the holes. Because from near L1 both the sun and Earth subtend 0.01 radians, θ ≥ 0.02. s thus must be ≤15 μm to diffract away light at λp = 0.6 μm. The axial component of radiation pressure for the transmitted light is reduced by a factor ≈θ2 and is thus negligible.

Such a simple screen is not ideal, because the on-axis transmission rises to 30% when averaged over the full solar spectrum. A screen with reduced transmission over a broad spectral band may be constructed at the expense of increased areal density by introducing steps in the film thickness. In general, if there are ml thickness levels of the same area (including one of zero thickness), with steps in optical depth of λ/ml, the on-axis transmission is given byFormula where tmax is the maximum thickness and n is the refractive index. Fig. 3a and b shows a design for a four-level screen (ml = 4) that yields on-axis transmission 〈T〉 = 10%, averaged over the solar spectrum.

Fig. 3.

Refractive screen. (a) Detail with holes are on 15-μm centers. (b) Detail of the antireflection coating, drawn to scale. The silicon nitride core with n = 2 shown in darker gray has thickness of 97,367 and 637 nm. The inner coating has index 1.587 and thickness 94 nm (medium gray), and the outer index is 1.26 and thickness 119 nm (light gray).

The thickness of screens of this type is proportional to (n− 1)−1 and is thus minimized by use of a high-index material. On the other hand, reflectivity of an uncoated surface increases strongly with index, as (n− 1)2/(n + 1)2. Thus a high-index, low-density material will be advantageous only if used with a thin, broad-band antireflection coating. This can be realized in a two-layer coating with the indices and thicknesses chosen so that the three reflections from the different interfaces have the same amplitude but interfere destructively at two wavelengths, one where the successive phase differences are at 120° and the other the half-wavelength where the phases are at ≈240°. An example of such a design, applied to the screen of Fig. 3a, is shown in Fig. 3b. The design is for a film of silicon nitride, chosen because it will not deteriorate in many decades of solar exposure and also is commonly manufactured in freestanding films <1-μm-thick. It has high index (2.0) and low density (3,100 kg/m3 in film form) and is also very stiff. The step-thicknesses of the silicon nitride shown in Fig. 3b were adjusted to allow for the path length increase of the coatings. The result is R = 2.62% and an average areal density ρs = 1.4 g/m2. This film is adopted as the baseline design for the remainder of this article.

An ideal sunshade with the above reflectivity and density would orbit at 2.2 Gm and, for 1.8% flux reduction, would require area 6 million km2 and would weigh ≈7 million tons (marked “screen material alone” in Fig. 2b). A practical sunshade will be heavier when structural and control elements are included. These additions are estimated to triple the average density of the complete flyer, to ρs = 4.2 g/m2, based on the discussion below. The reflectivity also will be higher. Our baseline design adds 0.9% for opaque structural elements and 1 ± 1% for reflecting control elements, for a total R = 4.5 ± 1%. With these parameters, the orbit would be 1.85 Gm from Earth. At this distance, the blocking efficiency ε is 54%, the total sunshade area, increased by 10% from Fig. 2a to allow for the on-axis screen transmission, is 4.7 × 106 km2, and the mass is 20 million tons total (marked “Baseline Design” in Fig. 2b).

From the Earth to L1

Is it at all realistic to transport a total payload mass of 20 million tons from Earth? If, for the sake of argument, we allow $1 trillion for the task, a transportation cost of $50/kg of payload would be needed. The present cost for multistage rocket transportation to high orbit is ≈$20,000/kg. For very high volume, it is reasonable to suppose that the cost might brought to a level approaching fuel cost, not unlike car and airline transportation. Thus, the cost to low-Earth orbit for a two-stage system using kerosene/liquid oxygen fuel might approach $100/kg (9), with additional costs to get to L1. Here, we explore the potential for still lower costs by using electromagnetic launch followed by ion propulsion.

In electromagnetic launch, the payload is driven by a current-carrying armature in a magnetic field. From the analysis below, it seems that there is no fundamental reason why launch from Earth by linear acceleration to escape velocity of 11.2 km/sec should not be possible, even allowing for atmospheric slowing and heating. Once the launch vehicle is clear of Earth's gravity, additional propulsion will be necessary to reach L1. If auxiliary rockets were used, the potential for large savings from the initial electromagnetic launch could not be fully realized. But ion propulsion is an ideally suited, low-cost alternative that adds only a small additional mass to the vehicle and is now space-proven by the SMART1 spacecraft to the moon.

The potential for very low transportation cost can be seen by consideration of launch energy cost. Kinetic energy at escape velocity is 63 MJ/kg = 17 kW·hr/kg (1 kW·hr = 3.6 ×106 J). Taking into account the mass of the armature and the ion-propulsion fuel, and the loss in conversion from electrical to kinetic energy, the energy for launch (as shown below) will be ≈10 times this final payload energy. At the current cost to industry of 5.3¢/kW·hr, the launch energy cost would be $9 per kg of payload. The additional major cost for energy storage is likely to be comparable, thus the $50/kg target for transportation is not unrealistic.

Atmospheric Drag and Heating.

On exiting the evacuated launch tube, the launch vehicle will be subject for about a second to strong drag and heating as it transits the atmosphere. Equating the loss of momentum of the vehicle to that gained by the displaced air, Δv/v = pδ/ρVg, where p is the atmospheric pressure, δ the drag coefficient, ρV the areal mass density of the vehicle, and v its velocity. Based on experience with space reentry vehicles designed for minimum drag, δ = 0.1 should be realizable. To minimize the energy loss, the launch would be vertical from a high site. A realistic goal would be an atmospheric entry point at 5.5 km elevation (18,000 feet) where p = 50 kPa, half that at sea level. Setting as a goal Δv/v = 1/8, an initial velocity of 12.8 km/sec would be needed for escape velocity of 11.2 km/sec above the atmosphere, and the vehicle will need an areal density ρV = 4 tons/m2.

The drag results in loss of 25% of the initial kinetic energy. Most will go into moving and heating the displaced air, but some will heat the vehicle itself. To prevent damage, an ablative shield must be used, as for space vehicles designed for atmospheric reentry. Based on past experience, it would seem that such a shield could be designed to weigh only a small fraction of the total vehicle mass. Measurements of a test vehicle with a low-drag (δ = 0.06) carbon nosecone entering the Earth's atmosphere at 6 km/sec showed an ablative loss of ≈0.1 kg, for a mass-loss to energy-loss ratio of 0.14 kg/GJ (10). A similar ratio of 0.25 kg/GJ was measured for the Galileo probe, which entered Jupiter's atmosphere at 47 km/sec and was brought to rest by a carbon ablation shield designed for high drag (11). In our case, a 4 ton/m2 vehicle losing 77 GJ/m2 would suffer an ablation loss of 20 kg/m2, if the loss rate were 0.25 kg/GJ. Even if the rate were twice as much, and the ablator including safety factor weighed 100 kg/m2, it would still make up only 2.5% of the vehicle total of 4,000 kg/m2. Based on the above considerations, it seems reasonable to suppose that atmospheric drag should not prevent Earth launch, but clearly modeling with codes such as those used for the Galileo heat shield needs to be undertaken. A full-scale test at 12.8 km/sec could be made with a rocket-propelled reentry vehicle (10).

Electromagnetic Launch to 12.8 km/sec.

Two types of electromagnetic launchers, rail and coil, have been studied over the years. In the rail type, the current in the armature is delivered by rails with sliding contact, and the driving magnetic field perpendicular to the armature current provided by a combination of the rail current and external coils. Laboratory experiments with rail systems have demonstrated acceleration of projectiles of a few grams to ≈8 km/sec and ≈1 kg to 2–3 km/sec (12). In the coil type, the armature is a cylinder with no contact, carrying a ring current maintained by magnetic induction. The magnetic field is provided by a long solenoid comprised of many short coils that are energized successively in synchronization with the armature accelerating along the axis. A 30-coil test system has been used in the laboratory to accelerate a 240-g armature to 1 km/sec with a comoving field of 30 T (13). The average accelerating pressure measured at 150 MPa reached nearly half the theoretical limit of B2/2μ0. For comparison, the same pressure applied to a 1-m-diameter armature would yield a thrust of 108 N, four times that of the Saturn V first stage.

Designs to harness such prodigious magnetic force to deliver payloads into orbit have been worked out for both launcher types but have never been attempted. The reasons are high up-front costs, the restriction to payloads able to survive very high acceleration, and the difficulty of launch into low-Earth orbits. Such orbits can be reached only by launch at low elevation angle, which incurs substantial aerodynamic drag, and with the addition of a supplemental rocket. However, these difficulties do not apply in our case, where a high volume is to be carried to very high orbit, and there is the possibility of ruggedizing the simple payloads to withstand high g force. The coil type is the better choice to survive a very large number of launches, given active control to prevent mechanical contact during launch. (Rail launchers inevitably suffer wear from the electrical connection required between the armature and rails.)

Previous designs for 6 km/sec coil-type space-launchers can serve as a starting point for a higher-velocity system. Table 1 gives the characteristics of two published designs, the first by Marder (14), based on simplified analytical expressions, and the second, fully optimized with a design code, by Lipinski et al. (15). Both envisage use fields of ≈24 T, similar track lengths of <1 km, and similar energy input of ≈65 GJ. The main difference is that Marder's design assumes as an input a conservative electrical to mechanical efficiency of 30%, whereas the Lipinski design finds a higher efficiency of 50% for the optimized configuration. Thus, the latter design can drive a substantially heavier launch vehicle (1,820 vs. 1,000 kg) for the same energy.

Table 1.

electromagnetic launcher designs

To reach higher velocity v, the pressure on the armature must be increased, as v2ρV/s, where ρV is the vehicle mass density and s is the length of the launch track. For systems of similar geometry and the same ratio of achieved to magnetic pressure B2/2μ0, the field thus must be increased as v√(ρV/s). To reach the desired 12.8 km/sec with acceptable field strength, a track length s = 2,000 m is baselined. Then, scaling from the optimized Lipinski design a field of 32 T is required for the targeted vehicle density of 4,000 kg/m2. Marder's analysis gives the same field if an energy efficiency of 45% is assumed.

An important issue for the higher-velocity launcher is to control Joule heating of the armature. During the launch intervalt, the temperature must not get high enough to cause the armature to yield under the high-magnetic-field pressure. The characteristic length for the depth of the eddy current and field penetration is the skin depth δ ∼ √(ηt0), where η is the armature resistivity (16). Both the 6- and 13-km/sec designs envisage use of aluminum armatures and have the same t≈ 0.35 sec. The skin depth δ ≈ 0.1 m is thus the same and is less than the armature diameter d. In this circumstance, the eddy current density will increase in proportion to the field strength and thus the temperature will rise as B2, a factor of 1.8 for the high-velocity launcher. This increase should be acceptable provided that the initial field rise is moderated to avoid high surface heating. Conductivity may be improved if necessary by precooling the armature with liquid nitrogen and possibly by incorporation of carbon nanotubes, if this is not too expensive. Computer models and subscale tests clearly are needed.

Ion Propulsion.

Going from a highly eccentric orbit with 2-month period and 1.5 Gm apogee to L1 requires changes in velocity totaling ≈1 km/sec. Given also some margin to correct for errors in launch velocity, a total of Δv = 2 km/sec is wanted. The propulsion force of ≈0.2 N available from ion propulsion will be sufficient, when applied over a few months. The mass of fuel needed is relatively low, because of its high ejection velocity, ≥20 km/sec. Thus, the Dawn spacecraft to the asteroids will carry 30% of its mass in xenon fuel to obtain a total Δv of 11 k/sec. For our task, a mass of ≈5% of the launch vehicle should be sufficient. Argon, which might be stored by adsorption in carbon, would be preferred to xenon to remove fuel as a significant factor in the transportation cost.

The Sunshade as a Cloud of Autonomous Spacecraft

Previous L1 concepts have envisaged very large space structures. The alternative described here has many free-flyers located randomly within a cloud elongated along the L1 axis. The cloud cross-section would be comparable to the size of the Earth and its length much greater, ≈100,000 km. This arrangement has many advantages. It would use small flyers in very large numbers, eliminating completely the need for on-orbit assembly or an unfolding mechanism. The requirements for station-keeping are reduced by removing the need for the flyers to be regularly arrayed or to transmit any signals.

The cross-sectional area of the cloud with random placement must be several times larger than the area of sunlight to be blocked, or the individual flyers will shadow one another and lose efficiency. On the other hand, if they are spread out too far off the axis, their penumbral shadows will move off the Earth. For randomly distributed flyers with the design parameters established above, namely a residual on-axis transmission of 10% and 1.85 Gm of distance, the optimum cloud cross-section size is a 6,200× 7,200-km ellipse. For this choice, the average off-axis shadowing efficiency is 51% (compared with 54% on-axis), and the loss from shadows overlapping is 6.5% (Fig. 4a). These two losses combined result in a 13% reduction in blocking, compared with the maximum achievable for the same number of elements in a tightly controlled, close-packed array, which would have a 7.6 times smaller cross-sectional area. The additional flyers needed to make up for the losses of the random configuration result in an increase in the total mass from 20 to 23 million tons, given the same areal density. In reality, the mass penalty may be smaller or even negative because small flyers will require lighter structural supports and simpler controls for station keeping.

Fig. 4.

Flyer configuration. (a) Projection of the full depth of the flyer cloud onto a plane transverse to the L1 axis (detail covering 5 × 15 m). The areal density shown is for the optimized case with flyers randomly distributed over 7.6 times their total area, resulting in 6.5% shadowing from overlaps. (b) A single 0.6-m-diameter flyer with the thin refracting disc faceted to improve stiffness. The three 100-μm-thick tabs have 2% of the disc area and contain the MEMS solar sails, tracker cameras, control electronics, and solar cells.

Position and Momentum Control.

The key requirements for autonomous control are to hold within the cloud envelope, to move slowly, and to keep facing the sun. The position must be actively controlled to prevent axial instability, which if left uncorrected will result in exponential increase in velocity with an e-folding time of 22 days. There is an independent need to control velocity, to minimize the chance of collisions between the randomly moving flyers, which even at low speed could set them spinning out of control. Control to ≤1 cm/sec, for example, will keep the collision probability to 10% per century per flyer.

To provide position and velocity information, special spacecraft with radio beacons in a global positioning system (GPS)-like system will be scattered through the cloud. Each flyer will incorporate a radio receiver to sense its velocity and position. In addition, it will carry two small tracker cameras mounted back-to-back to track the sun, Earth, and moon, to determine orientation.

Control of lateral and rotational motion will be accomplished by varying the radiation pressure on each flyer, with mirrors covering 2% of the flyer area and tiltable about an axis pointing to the flyer center. In the normal equilibrium configuration, half the mirrors would be turned so as to let the sunlight pass by and half would be set close to normal incidence to reflect back the sunlight. By appropriate rotations of the different mirrors, the lateral and angular acceleration in all six degrees of freedom can be set independently. From Eq. 1, the ±1% change in overall reflectivity of the flyers allows control of axial position to ±70,000 km and a maximum lateral acceleration (without changing the axial force) of ±5.10−6 m/sec2. Thus, flyers can easily be held within the elliptical envelope, requiring an outward acceleration of ≈8 × 10−7 m/sec2 5,000 km off the axis. Shadowing could be stopped temporarily if desired by placing the flyers into halo orbits about the L1 axis.

Flyer Size and Design for Launch at High Acceleration.

The preferred option is to eliminate completely construction, assembly, or unfurling in space by having rigid flyers completely fabricated on Earth and launched in stacks. A mechanism built into the launch vehicle would be used to deal the flyers off the stack, a steady process that could take around a year. This approach avoids any requirement for space rendezvous or infrastructure of any sort, except for the local beacon system.

Although aerodynamic considerations constrain the vehicle mass density to be ≥4,000 kg/m2, they do not favor a specific diameter. However, several factors argue for keeping the flyers small. To survive the high acceleration of launch, the smaller the flyers are, the less overhead will be needed for structural elements, and the easier it will be to make the sail-tilting mechanisms and to achieve high stacking density. A lower limit will be set ultimately by how small the control sensors and computer can be made, but a mass of no more than 0.1 g total seems reasonable. Based on these arguments, a flyer size of <1 m is adopted, to fit in a launch vehicle diameter of 1 m with cross-sectional area of 0.78 m2 and total mass of 3,100 kg.

As a specific example, consider flyers with optical screens 0.6 m in diameter. The solar sails adding 2% to the flyer area would be housed in three control ears sticking out 0.1 m, as shown in Fig. 4b. At an average areal density of 4.2 g/m2, each unit will weigh 1.2 g. The 1.4-μm-thick refractive film weighing itself 0.4 g would be supported by a 3.6-μm-thick, chicken-wire-like web of hexagonal cells, for a total thickness of 5 μm. The ears will be 100-μm-thick to accommodate solar cells, electronics, and optical trackers.

To pass the acceleration load directly down, the flyers would be tightly stacked for launch, with the webs lined up one above the other and in contact. The added thickness of the ears is allowed for by making their width 1/60 of the circumference and by clocking successive flyers in the stack by one tab width. In this way, the tabs will stack directly on their 20th nearest neighbors, also transmitting their acceleration load straight down. The tiltable mirrors to fit within the 100-μm ear thickness will be made by using MEMS (MicroElectro-Mechanical-Systems) technology and will be switched between open and closed positions by electrostatic force. By keeping the dimension of the mechanical elements very small in this way, the g force should not be a problem. Similarly, it should be possible to manufacture control electronics in the ears to survive 4,000 × g, as demonstrated by gun launch of a global positioning system.§ Once rugged flyer prototypes are developed, their operation with radiation pressure control would be tested in space. They would be taken to L1 initially by conventional rocket propulsion.

The mass of 3,100 kg for the launch vehicle will break down approximately as 1 ton for the flyers, 1 ton for the armature (scaled by area from the Lipinski design), and 1 ton for the structure and remaining items. To prevent the build up of very high loads, the flyers will be stowed in a number of short stacks, each supported by a shelf to transfer the local load to the outer cylindrical wall and thence down to the armature. Each 1,000-kg payload will contain 800,000 flyers. The payload height, set by the stacking separation of 5 μm, will be 4 m plus the thickness of the shelves. The remaining elements with 1,000-kg budget will include the structure and nonstructural items whose mass was already estimated, the ablation shield (≈80 kg), and the ion-propulsion fuel (≈150 kg) and motor, along with the mechanism to destack and release the flyers and vehicle spacecraft elements for communications and orientation.

Discussion

None of the technical issues explored above invalidate the space sunshade concept. To take it further, more analysis and experiments are needed, and the benefits and costs must be further explored, particularly in relation to Earth-based approaches. In making such a comparison, it will be important to understand flyer lifetime. Currently, spacecraft in high orbits such as communications satellites last for ≈20 years, failing in part from loss of solar power of 1% a year caused by cosmic rays. Lifetimes ≥50 years should be achievable for the much simpler flyers, provided that radiation damage is mitigated by derating the solar cells, and the control electronics is made highly redundant. The mirror mechanisms should not be a limitation, because lifetimes >1010 operations are achieved by MEMS mirrors in TV displays.

At the end of their life, the flyers will have to be replaced if atmospheric carbon levels remain dangerously high. The dead ones that find their way back to Earth could present a threat to Earth-orbiting spacecraft, but hopefully no greater than the annual flux of a million, 1-g micrometeorites, or the 30 million debris objects in low-Earth orbit that weigh ≈1 g. This issue needs to be analyzed. Similarly, the 20 million spent armatures would be directed into solar orbit or to the moon, but a small fraction might take up eccentric orbits and eventually reach the Earth intact. It seems, however, that this threat could be held to a level no more than that presented by the ≈100 1-ton natural objects that hit the Earth annually (17).

The total cost of the first full sunshade implementation will include development and ground operations, as well as the flyer production and transportation. Of these, transportation is the best understood at present, although a significant cost not yet addressed will be for storing the electrical energy for release during the short launch interval. Here, because of the large scale of the project, the key parameter is the cost per launch amortized over the lifetime of the storage medium. Capacitors of the type used to store 0.3 GJ at the National Ignition Facility would be suitable, if upgraded for million shot lifetime. Flywheel storage such as used currently to deliver ≈5 GJ to the JET torus at rates up to 800 MW (18) also could be adapted to supply high power over the 0.3-sec launch interval and should have potential for even longer lifetime. Batteries optimized for very fast discharge and long life are another possibility. A reasonable goal for cost of highly mass-produced storage with million cycle lifetime is 2¢/J. This corresponds to 7¢/kW·hr, comparable to the cost of the electrical energy itself.

To transport the total sunshade mass of 20 million tons, a total of 20 million launches will be needed, given flyer payloads of 1,000 kg. If it became necessary to complete the sunshade deployment in as little as 10 years, a number of launchers working in parallel would be needed. If each one were operated a million times on a 5-min cycle, in all, 20 would be required. To propel the 3.1-ton vehicles to escape velocity with 40% efficiency, each launcher will need 640 GJ of energy storage, which at 2¢/J will cost $13 billion. Allowing also $10 billion for the 2-km-high, largely underground launch structure, and another $6 billion for other costs such as for magnet wire and high-speed switches, then the total capital cost of each launcher would be ≈$30 billion. The first such launcher could serve not only to verify and start sunshade construction but also to test other systems requiring large mass in high orbit. (It could be used, for example, to transport a prototype space solar electric system weighing ≥100,000 tons to geosynchronous orbit, at a cost less than the National Research Council target for financial viability of $400/kg (19), or to deliver a similar mass of freight to the moon.) For all 20 million launchings the capital cost would be ≈$600 billion and the electrical energy cost $150 billion.

The environmental impact of launch must be considered in addition to its cost. In the worst case, if electrical energy were generated with coal, ≈30 kg would be required for each kg transported to L1. But each kilogram of the sunshade mitigates the warming effect of 30 tons of atmospheric carbon, a thousand times more. Note that if the launch were by rockets with kerosene/liquid oxygen fuel, the carbon consumed would be comparable. It takes ≈20 kg of kerosene to place 1 kg in low-Earth orbit with an efficient two-stage rocket (9), and likely twice this to escape the Earth. On the other hand, the fuel cost for rocket launch is much higher. Kerosene costs currently $0.73/kg, compared with ≈$0.02/kg for coal delivered to power stations. This difference underlies in part the economy of magnetic launch.

The production costs for the flyers as described here are unclear, as a completely unprecedented scale of mass-production is needed. An aggressive target would be the same $50 cost per kilogram as for launch, for $1 trillion total. To date, spacecraft have been mass-produced only in quantities ≤100. The Iridium satellites, for example (20), at $5 million each cost ≈$7,000/kg, an order of magnitude less than for one-off spacecraft but still over a hundred times too high. Strategies for completely automated production of 16 trillion flyers will have to draw on, but go far beyond, experience from the highest volume mass production in other fields. Some highly complex systems produced by the millions already come close to our cost target, for example, laptop computers at ≈$100/kg. At a volume a million time larger still, new economies of scale should further reduce cost, for example, mass-production of flyer mass-production lines themselves. Although further studies are needed, it seems that $50/kg for the flyers is not unreasonable. And if flyer construction and transportation costs each can be held in the region of $1 trillion total, then a project total including development and operations of <$5 trillion seems also possible. If the 50-year lifetime is achieved, the cost per year averages to $100 billion, (0.2% of current world gross domestic product) and would decrease after that when only flyer and energy storage renewal is needed.

In conclusion, it must be stressed that the value of the space sunshade is its potential to avert dangerous abrupt climate change found to be imminent or in progress. It would make no sense to plan on building and replenishing ever larger space sunshades to counter continuing and increasing use of fossil fuel. The same massive level of technology innovation and financial investment needed for the sunshade could, if also applied to renewable energy, surely yield better and permanent solutions. A number of technologies hold great promise, given appropriate investment (3).

Acknowledgments

I thank University of Arizona student Peng Su for help with the optical design, K. Eric Drexler and Maynard Cowan for valuable suggestions and comments on an earlier draft, and Pete Worden for providing some of the inspiration for this concept. This work is supported by National Aeronautics and Space Administration under Grant NAS5-03110.

Footnotes

  • *E-mail:rangel{at}as.arizona.edu
  • Author contributions: R.A. wrote the paper.

  • The author declares no conflict of interest.

  • Koppel, C. R., Marchandise, F., Estublier, D., Jolivet, L., 40th AIAA/ASME/SAE/ASEE Joint Propulsion Conference and Exhibit, Fort Lauderdale, FL, July 11–14, 2004, abstr. 3435.

  • Brophy, J. R., Marcucci, M. G., Ganapathi, G. B., Gates, J., Garner, C. E., Klatte, M., Lo, J., Nakazono, B., Pixler, G. (2005) 41st AIAA/ASME/SAE/ASEE Joint Propulsion Conference and Exhibit, Tucson, AZ, July 10–13, 2005, abstr. 4071.

  • § Dowdle, J. R., Throvaldsen, T. P., Kourepenis, A. S. (1997) AIAA Guidance, Navigation, and Control Conference, New Orleans, LA, August 11–13, 1997, abstr. 3694.

  • Abbreviations:
    L1,
    inner Lagrange point;
    Gm,
    million km.

Same day ACH: moving payments faster in the USA

$
0
0

New capabilities of Same Day ACH become effective over phases to allow financial institutions and businesses to acclimate to a faster processing environment, as well as to ease the implementation effort

. Beginning September 15, 2017, Same Day ACH will be available for debit entries, enabling the same-day processing of virtually any ACH payment. 

The Rule enables the option for same-day ACH payments through additional ACH Network functionality, without affecting previously available ACH schedules and capabilities:
 
Originating financial institutions (ODFIs) are able to submit files of same-day ACH payments through two additional clearing windows provided by the ACH Operators (Note: The actual ACH Operator schedules are not determined by the NACHA Operating Rules.):

  • A morning submission deadline at 10:30 AM ET, with settlement occurring at 1:00 PM.
  • An afternoon submission deadline at 2:45 PM ET, with settlement occurring at 5:00 PM.

Virtually all types of ACH payments, including both credits and debits, are eligible for same-day processing. Only international transactions (IATs) and high-value transactions above $25,000 are not eligible. Eligible transactions account for approximately 99 percent of current ACH Network volume.

All RDFIs are required to receive same-day ACH payments, thereby giving ODFIs and Originators the certainty of being able to send same-day ACH payments to accounts at all RDFIs.

Learn more about Same Day ACH with our Resource Center.

The Rule is based on a solid foundation of economic research on the use cases for Same Day ACH.

All consumers, businesses, government entities and financial institutions that use the ACH Network to move money between bank accounts will benefit from the option to move ACH payments faster.

NACHA projects that ACH Originators would generate approximately 1.4 billion same-day ACH payments annually as of ten years after full implementation and rollout, primarily for transactions that can be initiated before 2:45 PM ET on business days (not on weekends or holidays), and that do not require real-time functionality.

Using an expert, third-party economist, NACHA assessed 10 primary use cases for Same Day ACH. Significant use cases for Same Day ACH include: 

  1. Same-day payrolls, supporting business’ needs to pay hourly workers, and providing flexibility for late and emergency payrolls and missed deadlines; and enabling employees to have faster access to their pay in these cases;

  2. Business to-Business payments, enabling faster settlement of invoice payments between trading partners, and including remittance information with the payments;

  3. Expedited bill payments using both ACH credits and debits, enabling consumers to make on-time bill payments on due dates, and providing faster crediting for late payments; and,

  4. Account-to-account transfers, providing faster crediting for consumers who move money among various accounts they own.

The following changes to the NACHA Operating Rules will become effective on September 15, 2017.

  
SECTION 8.84 Same Day Entry
 
An Entry for which the Effective Entry Date is the same Banking Day as the date on which the Entry is Transmitted by the ODFI to its ACH Operator, and is Transmitted by the ACH Operator’s deadline for same-day processing and settlement. A Same Day Entry must be for an amount of $25,000 or less. An IAT or ENR Entry cannot be a Same Day Entry. For purposes of fulfilling its obligations under these Rules, an RDFI may rely on the Settlement Date of an Entry, regardless of the Effective Entry Date. An Entry with a stale or invalid Effective Entry Date will also be a Same Day Entry if it is Transmitted by the ODFI to its ACH Operator by the ACH Operator’s deadline for same-day processing and settlement, and is otherwise eligible for same-day processing and settlement.​

SUBSECTION 2.5.15.2 Authorization of TEL Entries
 
For an authorization related to a Single Entry TEL Entry, the Originator must either make an audio recording of the oral authorization, or provide the Receiver with written notice confirming the oral authorization prior to the settlement of the Entry.
  
SUBSECTION 2.8.3 Time Limitations on Initiation of Reversing Files
 
An Originator or the ODFI must Transmit each Reversing File and, when appropriate, a corresponding Correcting File, to the ACH Operator within five Banking Days after the Settlement Date of the Erroneous File. The Originator or ODFI must Transmit the Reversing File and any corresponding Correcting File to the ACH Operator within twenty-four hours of the discovery of the Erroneous File. Any debit Entry within the Reversing File must not contain an Effective Entry Date that is earlier than the Effective Entry Date of the credit Entry to which it relates.
 
SUBSECTION 2.9.1 General Rule for Reversing Entries
 

An Originator may initiate a Reversing Entry to correct an Erroneous Entry previously initiated to a Receiver’s account. The Reversing Entry must be Transmitted to the ACH Operator in such time as to be Transmitted or made available to the RDFI within five Banking Days following the Settlement Date of the Erroneous Entry.

 
A debit Reversing Entry must not contain an Effective Entry Date that is earlier than the Effective Entry Date of the credit Entry to which it relates.

For this Section 2.9 and Subsection 2.12.2 (ODFI Request for Return) only, an Erroneous Entry is defined as an Entry that:
 

  1.  is a duplicate of an Entry previously initiated by the Originator or ODFI;

  2.  orders payment to or from a Receiver different than the Receiver intended to be credited or debited by the Originator;

  3.  orders payment in a dollar amount different than was intended by the Originator; or

  4.  is a credit PPD Entry satisfying each of the following criteria:

  5. is a credit PPD Entry satisfying each of the following criteria:​
    1.  the credit PPD Entry is for funds related to a Receiver’s employment;

    2.  the value of the credit PPD Entry is fully included in the amount of a Check delivered to the same Receiver at or prior to the Receiver’s separation from employment; and
    3.  the credit PPD Entry was Transmitted by the Originator prior to the delivery of the Check to the Receiver.

The Originator must make a reasonable attempt to notify the Receiver of the Reversing Entry and the reason for the Reversing Entry no later than the Settlement Date of the Reversing Entry. For a credit PPD Entry satisfying the criteria of Subsection 2.9.1(d) above, the Originator must notify the Receiver of the Reversing Entry at the time the Check is delivered to the Receiver.
  
SUBSECTION 2.10.4 Timing Requirements for Reclamation Entries and Written Demands for Payment
 
An Originator or ODFI must originate a Reclamation Entry or written demand for payment within five Banking Days after the Originator receives notice of the death of the Receiver. If a Reclamation Entry is returned by the RDFI, the Originator may make a written demand for payment within fifteen Banking Days after it receives the returned Reclamation Entry. For this subsection, notice received by the Originator is considered to be effective from the time specified in Section 1-201(27) of the Uniform Commercial Code (1978 Official Text).
 
A Reclamation Entry must not contain an Effective Entry Date that is earlier than the Effective Entry Date of the credit Entry to which it relates.

SUBPART 3.2.2 Glossary of Data Elements
 
Effective Entry Date:  6 Positions – Company/Batch Header Record – Required (all batches)
 
The Effective Entry Date is the Banking Day specified by the Originator on which it intends a batch of Entries to be settled.
 
For credit Entries, the Effective Entry Date must be either the same Banking Day as the Banking Day of processing by the Originating ACH Operator (the processing date) for Same Day Entries, or one or two Banking Days following the Originating ACH Operator’s processing date for other Entries.  

For debit Entries, the Effective Entry Date must be either the same Banking Day as the processing date for Same Day Entries, or one Banking Day following the Originating ACH Operator’s processing date for other Entries.
 
A batch of Entries containing an Effective Entry Date beyond the designated number of days allowed is Rejected by the ACH Operator and returned to the ODFI. 
 
A batch of Entries containing an Effective Entry Date that is stale (i.e., is a Banking Day that is in the past, or is the current Banking Day but is not submitted timely to an ACH Operator for same-day processing and settlement) is settled at the next available settlement opportunity.
 
A batch of Entries containing invalid Effective Entry Date information (for example, if the field is blank or zero, partially blank or partially non-numeric, contains an incomplete date, contains day numbers higher than 31, or contains month numbers higher than 12) is settled at the next available settlement opportunity. The Originating ACH Operator inserts the Banking Day of processing or the next Banking Day as the Effective Entry Date, whichever corresponds to the next available settlement opportunity.
 
ENR: For Automated Enrollment Entries, this field must be space filled. Automated Enrollment Entries cannot be Same Day Entries.
 
Return Entries and Notifications of Change: The ACH Operator does not edit this field.
 
The scheduled Settlement Date is inserted by the Receiving ACH Operator. See the definition of “Settlement Date” in this Appendix Three.
 
Settlement Date: 3 Positions – Company/Batch Header Record – Inserted by Receiving ACH Operator (all batches)
 
The Settlement Date (a 3-digit Julian date) for a batch of Entries is inserted by the Receiving ACH Operator. This is the date on which the Participating DFI or its correspondent is scheduled to be debited or credited by the Federal Reserve.
 
The Settlement Date inserted by the Receiving ACH Operator is the same as the Effective Entry Date, except as noted below.

In the following situations, the Receiving ACH Operator will insert the Banking Day following the Banking Day of processing as the Settlement Date (i.e., the next Banking Day):
 

  1. The Effective Entry Date and the Originating ACH Operator’s processing date are the same, but the Entry is received by the Receiving ACH Operator after its deadline for same-day settlement. 

  2. The Effective Entry Date and the Originating ACH Operator’s processing date are the same, but the Entry is for an amount greater than $25,000.

  3. The Effective Entry Date and the Originating ACH Operator’s processing date are the same, but the Entry bears the IAT Standard Entry Class Code. 

  4. The Entry bears the ENR Standard Entry Class Code. 

Entries with invalid or stale Effective Entry Dates will be settled at the next available settlement opportunity.

Return Entries, dishonored Return Entries, and contested dishonored Return Entries are settled by the ACH Operator no earlier than the Effective Entry Date contained within the original Entry, as it appears in the Return Entry Company/Batch Header Record. The return of an Entry that contains an invalid or stale Effective Entry Date will be settled by the ACH Operator at the next available settlement opportunity (i.e., the Banking Day of processing or the next Banking Day).
 
Notifications of Change will be settled at the next available settlement opportunity, (i.e., the Banking Day of processing or the next Banking Day).

 

 
 

Same Day ACH Impacts by Party

ODFI Impacts
 

Offering Same Day ACH products and services is optional for ODFIs.  For those ODFIs that choose to offer Same Day products and services, the following impacts should be considered:

  • Discuss new file submission and delivery schedules with ACH Operator

  • Review and revise internal processing schedules and procedures to accommodate new windows for Same Day Entries.

  • Review downstream applications and systems that may be impacted by Same Day Entries.

  • Communicate with Originators and Third-Parties on Same Day Entries
    • Determine customers and business models to offer Same Day 
    • Discuss proper use of the Effective Entry Date and impacts of improper Effective Entry Dates. Discuss alternatives if an Originator is not able to properly use Effective Entry Dates. If desired, discuss use of an optional indicator for Same Day entries
    • Discuss eligibility requirements for Same Day Entries
  • Develop procedures for holding entries that are received from an Originator in time for a same day window but that is not the Originator’s desire

  • ODFIs will receive information from their ACH Operator regarding collection of the Same Day Entry fees. 

RDFI Impacts

Receipt of Same Day Entries is not optional for RDFIs.  RDFIs should consider the following potential impacts:

  • RDFIs should discuss new file delivery schedules with ACH Operator

  • RDFIs should review procedures and resources for processing files  in the new delivery windows

  • RDFIs should process entries based on the Settlement Date provided by the ACH Operator. RDFIs do not need to review Effective Entry Date or any other indicator to determine processing.

  • RDFIs do not need to determine same day eligibility for processing.

  • RDFIs should be prepared to also receive debit entries as same day entries.  

  • Determine whether to use the new same-day processing windows to send returns and NOCs. Returns will be allowed, but not required, to be processed on the same day. For a Same Day ACH entry, since settlement is one day sooner, the return deadline is also one day sooner.

  • Beginning in Phase 1, RDFIs should provide end of day funds availability for all same day credit entries received.  Phase 3 will require RDFIs to provide funds availability at 5:00 pm local time for same day credit entries. 

  • Review downstream applications and systems that may be impacted by Same Day Entries.

  • RDFIs will receive information from their ACH Operator regarding receipt of the Same Day Entry fees. 

Originators/Third-Parties

  • Talk to your FI to determine if Same Day is appropriate and cost effective
    • Discuss new file submission and exception resolution cut-off times

What are the major provisions of Same Day ACH?
The Rule will enable the option for same-day ACH payments through new ACH Network functionality, without affecting existing ACH schedules and capabilities:
 
Originating financial institutions (ODFIs) will be able to submit files of same-day ACH payments through two new clearing windows provided by the ACH Operators :
 

  • A morning submission deadline at 10:30 AM ET, with settlement occurring at 1:00 PM.
  • An afternoon submission deadline at 2:45 PM ET, with settlement occurring at 5:00 PM.
Virtually all types of ACH payments, including both credits and debits, will be eligible for same-day processing. Only international transactions (IATs) and high-value transactions above $25,000 will not be eligible. Eligible transactions account for approximately 99 percent of current ACH Network volume.

All RDFIs will be required to receive same-day ACH payments, thereby giving ODFIs and Originators the certainty of being able to send same-day ACH payments to accounts at all RDFIs.

RDFIs will be mandated to make funds available from same day ACH credits (such as payroll Direct Deposits) to their depositors by 5:00 PM at the RDFI’s local time.

To allow financial institutions and businesses to acclimate to a faster processing environment, as well as to ease the implementation effort, these new capabilities will become effective over three phases beginning in September 2016.

What are the Effective Dates?
The rule will become effective in three phases:

  • Phase 1 will become effective on September 23, 2016 (credit entries with funds available at end of RDFI processing day);
  • Phase 2 will become effective on September 15, 2017 (credit and debit entries with funds available at end of RDFI processing day);
  • Phase 3 will become effective on March 16, 2018 (credit and debit entries with funds available at 5:00 pm RDFI local time). 
Spreading implementation across three phases will ease the industry’s implementation effort, and allow the industry to acclimate to a faster processing environment with same-day ACH credits prior to processing same-day ACH debits.

How will Same Day ACH transactions be identified?
Same Day ACH transactions will be identified and processed by the ACH Operator using that day’s date in the Effective Entry Date field of the batch of Same Day ACH transactions and meeting the ACH Operator processing deadline. If the entry contains a stale or invalid date in the Effective Entry Date field, the Operator will process the entry in the next available window, which could be same day or next day.
 
ODFIs at their discretion may require their Originators to further show intent for same-day settlement using an optional, yet standardized, same-day indicator in the Company Descriptive Date field. The Company Descriptive Date field (5 record, field 8) is an optional field with 6 positions available (positions 64-69). Currently, the Rules provide that the “Originator establishes this field as the date it would like to see displayed to the Receiver for descriptive purposes.” NACHA recommends that, as desired, the content of this field be formatted using the convention “SDHHMM”, where the “SD” in positions 64- 65 denotes the intent for same-day settlement, and the hours and minutes in positions 66-69 denote the desired settlement time using a 24-hour clock. When electing to use this convention, the ODFI should validate that the field contains either “SD1300” for settlement desired at 1:00 PM ET, or “SD1700” for settlement desired at 5:00 PM ET.

Why are large dollar transactions (above $25,000) ineligible?
Faster velocity of payments is expected to introduce risks that must be managed and mitigated. A per- transaction dollar limit is one way to mitigate risk. Financial institutions and businesses will also have to adjust their end-of-day balance forecasting and funding, and the dollar limit is a way to limit the impact. Currently, more than 99% of ACH transactions are under $25,000, so the overall impact of the same-day limit is expected to be modest.
 
How will the $25,000 limit be enforced?
The ACH Operators will edit for the $25,000 limit for Same Day ACH transactions, and will assign next day settlement to any transactions above the limit.
 
Will Same Day ACH be available for all account types?
Any account that is able to receive ACH entries today could receive Same Day ACH entries.
 
What is the Same Day Entry Fee?
A Same Day Entry Fee of 5.2 cents will be paid from the ODFI to the RDFI for each Same Day ACH transaction. Collection and disbursement of Same Day Entry Fees for ODFIs and RDFIs will be handled by the ACH Operators through monthly billing. In billing for the Same Day Entry fees, the ACH Operators will round each financial institution’s monthly statement charge or credit to the nearest cent. For example, an ODFI that originates 13 same-day transactions in September 2016 would be billed $0.68 via its monthly ACH Operator billing statement (i.e., 13 same-day transactions * $0.052 = $0.676, which is rounded to $0.68).
 
Will non-monetary ACH transactions be eligible for Same Day processing?
For non-monetary transactions, there are no funds to settle. Generally, these transactions will benefit from same-day processing. Under this rule, most of these transactions will be eligible for same-day processing beginning with Phase 1 of implementation.

The eligible non-monetary transactions include:

  • Prenotifications for future credits (Prenotifications for future debits will become eligible for same-day processing in Phase 2)
  • Remittance information (using CCDs and CTXs)
  • Notifications of Change (NOCs) NOCs are transmitted from the RDFI to the ODFI and will not incur a Same Day Entry Fee

Automated Enrollment Entries (ENRs) will not be processed as same-day entries. ENRs do not use Effective Entry Dates, so there is no way to designate an ENR as a same-day transactions.

Why are IATs ineligible?
The receipt of an IAT triggers the burden of the OFAC screening requirements. Receiving IAT items late in the day could impose a significant burden on many financial institutions to complete the OFAC screening in a timely manner.​

How will TRC and TRX Entries be treated under the Same Day ACH rules?
Same Day ACH treats TRC and TRX the same as all other eligible SEC codes, where settlement will be based on the Effective Entry Date.​

How will returns be affected by Same Day ACH?
For ACH transactions that need to be returned, faster processing of the return is typically a “win-win” for all parties. ODFIs and Originators will learn about problems sooner and RDFIs will have exceptions processed and settled faster.
 
Same Day ACH will allow for the same-day processing of returns, regardless of whether the forward transaction is a same-day transaction or not. All returns are eligible to be sent and processed using the same-day processing schedules.  Returns that are settled on a same-day basis are not subject to the Same Day Entry Fee and are not subject to the eligibility limit on forward Same Day transactions (i.e., $25,000 per transaction; IAT).
 
An RDFI is not required to process returns on the same day an Entry is received, regardless of whether the Entry is a Same Day Entry.
 
A forward Same Day ACH entry that is returned by the ACH Operator (and does not reach the RDFI) will not be charged a Same Day Entry fee. 
 
How will reversals be affected by Same Day ACH?
Reversals are eligible for same-day processing using the same eligibility criteria as other forward transactions. A same-day reversal is limited to $25,000 per transaction, so any reversal to correct a forward entry over $25,000 (for example, to reverse an erroneous next-day credit of $100,000) would not be eligible to be sent as a Same Day ACH entry. Any reversal that is processed and settled same-day will be assessed the Same Day Entry Fee.
 
ODFIs/Originators must ensure that the Effective Entry Date of a reversing debit to correct an erroneous Credit Entry or a reversing file to correct an erroneous credit file is not earlier than the Effective Entry Date of the related credit(s). Since Phase 1 permits only same-day credits, a debit that reverses a credit Entry will not be eligible for same-day processing during Phase 1. Same-day debits become eligible in Phase 2, so at that time the debit to reverse a credit could be sent and settled the same-day. 

Will Same Day ACH transactions be able to carry remittance information?
CCD and CTX entries, as well as other entries that carry remittance, may be eligible to be sent as Same Day ACH. Requirements related to provision of remittance to Receivers will not change.
 
What is the difference between Same Day ACH and real-time payments?
Same Day ACH and real-time payments will offer complementary options for users, as they will have distinct functionality and thereby meet distinct needs of various use cases. Today’s ACH Network supports traditional ACH debit payments –bill payments, for example – as well as traditional credit payments like payroll. Approximately 60% of ACH transactions are debit payments. A real-time system would likely provide credit payments only; however, there is benefit to moving both credits and debits faster.
 
Will Federal government payments be eligible for Same Day processing?
The Federal government will not be participating in Phase One of Same Day ACH when it becomes effective on Sept. 23, 2016. According to the U.S. Treasury, any ACH transaction originated from, or received by, the U.S. government will not be eligible for same-day settlement, and will continue to settle on a future date regardless of submission date and time. This limitation applies only to Federal government payments, and not to state or local government agency payments.
 

For additional FAQs please see the Federal Reserve’s Same Day ACH Resource Center:
https://www.frbservices.org/resourcecenter/sameday_ach/index.html (link is external)

Writing a Formally-Verified Porn Browser in Coq and Haskell

$
0
0

It’s uncommon to use formal verification when developing software. Most people are unfamiliar with the tools and techniques, or assume they’re only for specialized use.

This article will show how to write a simple image browser with:

  • Core data structures and operations formally verified using the Coq theorem prover.
  • A Haskell web server that handles HTTP requests
  • An HTML/CSS/Javascript frontend

Definitions

We’re going to make an image browser. It’s an unspoken truth that most large image databases people have are actually porn collections, so we’ll be honest with ourselves and call it a porn browser.

We’ll have Coq-verified data structures and operations, which some Haskell will use later on to respond to HTTP requests. Specifically, we need the ability to:

  • Add new images to the database
  • Get a sorted list of images by date added, or category
  • Delete an image
  • Add a category to an image
  • Remove a category from an image
  • Add categories

Here are the major types:

RequireImportAscii.FromCoqRequireImportSets.EnsemblesNArith.BinNatZArith.IntFSets.FMapAVLFSets.FSetAVLStructures.OrdersExStructures.OrdersAltLists.ListStrings.String.ModuleN_as_OT:=Backport_OTN_as_OT.ModuleM:=FMapAVL.Make(N_as_OT).ModuleS:=FSetAVL.Make(N_as_OT).RecordImage:Set:=mkImage{id:N;filename:string;timestamp:N}.DefinitionImageId:=N.DefinitionCategoryId:=N.DefinitionIndex:=S.t.RecordImageDb:=mkDb{images:M.tImage;indices:M.tIndex;next_id:ImageId}.

FMapAVL and FSetAVL are finite maps and sets implemented using AVL trees. Our database is just a map of image ids to images, a map of category ids to sets of image ids, and an increasing id column incremented after every new addition.

It looks like the Coq developers are currently refactoring the equivalent of Haskell’s Ord typeclass, so I use Backport_OT N_as_OT to generate the older interface from the newer one.

This should be the Haskell equivalent:

importqualifiedData.MapasMimportqualifiedData.SetasStypeImageId=IntegertypeCategoryId=IntegerdataImage=Image{imageId::ImageId,filename::String,timestamp::Integer}typeIndex=S.SetImageIddataImageDb=ImageDb{images::M.MapImageIdImageindices::M.MapCategoryIdIndexnext_id::ImageId}

We’re eventually going to generate Haskell from our formally-verified Coq implementation. What are the generated Haskell data types?

ExtractionLanguageHaskell.ExtractionImage.ExtractionIndex.ExtractionImageDb.

Which prints

-- Extraction Image.dataImage=MkImageNPrelude.StringN-- Extraction Index.typeT=ZdataTree0=Leaf0|Node0TTree0NTree0typeT10=Tree0typeIndex=T10-- Extraction ImageDb.typeKey=NdataTreeelt=Leaf|Node(Treeelt)Keyelt(Treeelt)TtypeBstelt=TreeelttypeT8elt=BsteltdataImageDb=MkImageDb(T8Image)(T8Index)ImageId

which are essentially simple tree types hidden behind a few aliases. We also get a bunch of functions for manipulating the trees, which I left out. Our Haskell code will be using our own custom functions that manipulate these types, which will be formally-verified.

So what are these functions anyways? I’ll spec out their types using the “Axiom” command:

AxiomnewDb:ImageDb.Axiomcreate_image:forall(db:ImageDb)(img:Image),ImageDb.Axiomfind_categories:forall(db:ImageDb)(categories:listCategoryId),listImage.Axiomdelete_image:forall(db:ImageDb)(img:ImageId),ImageDb.Axiomtag_image:forall(db:ImageDb)(img:ImageId)(cat:CategoryId),ImageDb.Axiomuntag_image:forall(db:ImageDb)(img:ImageId)(cat:CategoryId),ImageDb.

This is similar to using undefined in Haskell for placeholders. In fact, that’s pretty much what the Extraction command generates:

newDb::ImageDbcreate_image::ImageDb->Image->ImageDbfind_categories::ImageDb->(ListCategoryId)->ListImagedelete_image::ImageDb->ImageId->ImageDbtag_image::ImageDb->ImageId->CategoryId->ImageDbuntag_image::ImageDb->ImageId->CategoryId->ImageDbnewDb=Prelude.error"AXIOM TO BE REALIZED"create_image=Prelude.error"AXIOM TO BE REALIZED"find_categories=Prelude.error"AXIOM TO BE REALIZED"delete_image=Prelude.error"AXIOM TO BE REALIZED"tag_image=Prelude.error"AXIOM TO BE REALIZED"untag_image=Prelude.error"AXIOM TO BE REALIZED"

In Haskell, our next step would be to jump right into the implementation. In Coq, we can think for a moment about what invariants should hold:

  • A new database has no images.
Axiomnum_images:forall(db:ImageDb),N.Axiomcount_empty_db:num_imagesnewDb=N.zero.

count_empty_db is the invariant here; num_images is a helper function we’ll define later.

  • Whenever we add an image to the database, the number of images should increase by 1:
Axiomsize_increases:forall(db:ImageDb)(img:Image),num_images(create_imagedbimg)=N.succ(num_imagesdb).
  • When we delete an image, either the image was in the database and the count of images decreases by 1, or it wasn’t and the count stays the same:
Axiommem_image:forall(db:ImageDb)(img:ImageId),bool.Axiomsize_decreases:forall(db:ImageDb)(img:ImageId),(Is_true(mem_imagedbimg)/\N.succ(num_images(delete_imagedbimg))=num_imagesdb)\/(~Is_true(mem_imagedbimg)/\num_images(delete_imagedbimg)=num_imagesdb).
  • When we tag or untag an image, the number of images stays the same:
Axiomsize_nochange:forall(db:ImageDb)(img:ImageId)(cat:CategoryId),num_images(tag_imagedbimgcat)=num_imagesdb/\num_images(untag_imagedbimgcat)=num_imagesdb.

I can think of others, but these 4 should be enough to demonstrate the technique.

Implementation

Here are the implementations, in order:

DefinitionnewDb:=mkDb(M.emptyImage)(M.emptyIndex)N.zero.(* OR *)DefinitionnewDb:={|images:=M.emptyImage;indices:=M.emptyIndex;next_id:=N.zero|}
Definitioncreate_image(db:ImageDb)(img:Image):=letnewImg:={|id:=next_iddb;filename:=filenameimg;timestamp:=timestampimg|}in{|images:=M.add(next_iddb)newImg(imagesdb);indices:=indicesdb;next_id:=N.succ(next_iddb)|}.

You can test it with the Compute command:

Computecreate_imagenewDb(mkImage0"testing"0).

which prints:

 = {|
   images := {|
             M.this := M.Raw.Node (M.Raw.Leaf Image) 0%N
                         {| id := 0; filename := "testing"; timestamp := 0 |}
                         (M.Raw.Leaf Image) 1%Z;
             M.is_bst := M.Raw.Proofs.add_bst 0%N
                           {| id := 0; filename := "testing"; timestamp := 0 |}
                           (M.Raw.Proofs.empty_bst Image) |};
   indices := {|
              M.this := M.Raw.Leaf Index;
              M.is_bst := M.Raw.Proofs.empty_bst S.MSet.t_ |};
   next_id := 1%N |}
 : ImageDb

It looks scarier than it is: It’s showing us the guts of the binary search trees it makes, while Haskell uses fromList in its Show instance to hide that:

create_imagenewdb(mkImage0"testing"0)-- PRINTSMkImage{images=fromList[(0,Image0"testing"0)],indices=fromList[],next_id=1}
Fixpointset_from_list(xs:listN):Index:=fold_rightS.addS.emptyxs.Fixpointlist_from_set(xs:Index):listN:=S.foldconsxsnil.Fixpointfind_categories_ids(db:ImageDb)(categories:listCategoryId):Index:=matchcategorieswith|nil=>set_from_list(mapfst(M.elements(imagesdb)))|conscatcats=>S.inter(find_category_idsdbcat)(find_categories_idsdbcats)end.Fixpointfind_imgs(db:ImageDb)(imgs:listImageId):=matchimgswith|nil=>nil|consiis=>matchM.findi(imagesdb)with|None=>find_imgsdbis|Someimg=>consimg(find_imgsdbis)endend.Fixpointfind_categories(db:ImageDb)(categories:listCategoryId):listImage:=find_imgsdb(list_from_set(find_categories_idsdbcategories)).

In Haskell, let bindings and top-level definitions are recursive by default. In Coq, we explicitly mark functions as Fixpoint to allow recursion.

Fixpointdelete_image(db:ImageDb)(img:ImageId):ImageDb:={|images:=M.removeimg(imagesdb);indices:=M.map(S.removeimg)(indicesdb);next_id:=next_iddb|}.
Fixpointtag_image(db:ImageDb)(img:ImageId)(cat:CategoryId):ImageDb:=letidxs:=matchM.findcat(indicesdb)with|None=>M.addcat(S.singletonimg)(indicesdb)|Someidx=>M.addcat(S.addimgidx)(indicesdb)endin{|images:=imagesdb;indices:=idxs;next_id:=next_iddb|}.
Fixpointuntag_image(db:ImageDb)(img:ImageId)(cat:CategoryId):ImageDb:=letidxs:=matchM.findcat(indicesdb)with|None=>indicesdb|Someidx=>M.addcat(S.removeimgidx)(indicesdb)endin{|images:=imagesdb;indices:=idxs;next_id:=next_iddb|}

Theorems

We’re almost ready to prove some theorems. Let’s implement those helper functions we deferred:

Fixpointnum_images(db:ImageDb):nat:=M.cardinal(imagesdb).Fixpointmem_image(db:ImageDb)(img:ImageId):bool:=M.memimg(imagesdb).

The first theorem we can do just by calculation:

Theoremcount_empty_db:num_imagesnewDb=0.

A lot of these proofs consist of a sequence of imperative mutations to a “proof context”. These mutations are called “tactics”. If you use a tool like Proof General, you can step through each tactic to see how it changes the proof context. I’ll add annotations above each tactic, showing the proof context.

Proof.(* 1 subgoal, subgoal 1 (ID 46)
   ============================
   num_images newDb = 0
*)compute.(* 1 subgoal, subgoal 1 (ID 47)
   ============================
   0 = 0
*)tauto.(* No more subgoals. *)Qed.

We start off with the statement we’re trying to prove as our “goal”. A theorem is proved when there are no remaining goals. We reduce the lefthand side using compute. The goal becomes 0 = 0, which is a tautology that the taut tactic can complete.

Not too difficult, right? The follow-up ones were involved enough that I’ll have to leave their detailed proofs on Github.

Now, you might be curious: “Coq lets us extract code to Haskell. What exactly does a theorem convert to?” Here’s your answer:

-- Extraction count_empty_db.count_empty_db::()count_empty_db=__

You can’t do it - Haskell’s type system isn’t powerful enough to handle theorem proving. Roughly, things that can be extracted into living, breathing computer code lie in Coq’s Set; while the world of theorems and proofs is Coq’s Prop. And the classic tripup for beginners is “bool vs. Prop”

A bool is decidably true or false. A bool when extracted is eventually going to live on the system heap. A bool has a bitstring representation.

A Prop is a statement in an intuitionist logic. In classical logic, you know for any proposition P, that P | ~P. You can actually add that axiom to Coq, but it’s fundamentally not constructive: Just because you’ve excluded the possibility of ~P doesn’t mean that you can physically instantiate a bitstring representing a specific P in memory somewhere.

Inductive types like bool are closed. There are only 2 constructors true and false, and if you’ve excluded true you know it’s false.

Propositions are open world: You can’t conclude a positive from a bunch of negatives.

Let’s try the next one:

Theoremsize_increases:forall(db:ImageDb)(img:Image),num_images(create_imagedbimg)=num_imagesdb+1.

This theorem is actually false! All of our functions should be safe, but anyone can call mkImageDb with a next_id that overlaps an existing image. Then, calling create_image would overwrite it, and num_images wouldn’t change.

Se we need to specify that this theorem only holds for InternallyConsistent databases, and that all of our operations preserve this property.

The only consistency check we’ll have for now is that next_id should not exist in the images map:

DefinitionInternallyConsistent(db:ImageDb):=forall(some_id:ImageId),M.Insome_id(imagesdb)->N.ltsome_id(next_iddb).

Now we’ll prove that newDb is internally consistent, using empty_in_iff.

Theorempreserves_consistency_1:InternallyConsistentnewDb.Proof.intro.intro.rewriteMF.empty_in_iffinH.contradiction.Qed.

That was easy because InternallyConsistent is a property of the indices in the images map. In newDb, the map is empty so the statement is vacuously true.

I’ll pick one more interesting one before moving on.

RequireImportCoq.FSets.FMapFacts(* Earlier: Module M := FMapAVL.Make(N_as_OT). *)ModuleMF:=FactsM.Theorempreserves_consistency_2:forall(db:ImageDb)(img:Image),InternallyConsistentdb->InternallyConsistent(create_imagedbimg).Proof.intros.intro.intro.simpl.applyN.lt_succ_r.simplinH0.applyMF.add_in_iffinH0.destructH0.applyN.le_lteq.right.symmetry.applyH0.applyHinH0.applyN.le_lteq.left.applyH0.Qed.

Recall that images is a finite map. We import all of the propositions relating to finite maps using the Facts functor, applied to the FMapAVL implementation we instantiated earlier.

Here’s the idea of the proof in words:

MF.add_in_iff says that “If you add an element to a map, any element in the new map was either already in the map or had the same key as the one we added.”. If it was already in the map, then the InternallyConsistent hypothesis already applies to it. Otherwise, the key we added was exactly next_id, and the definition of InternallyConsistent gives next_id db < succ (next_id db) after we increment it in create_image.

Full proofs for the theorems are in the Github repository.

Haskell Server

Coq allows us to map inductive types directly to Haskell. There are a few built-in modules for strings, and I’ll include examples for list and option also:

From Coq Require Import
     extraction.ExtrHaskellString
     extraction.ExtrHaskellNatInteger
     extraction.ExtrHaskellNatNum

Extract Inductive list    => "[]" ["[]" "(:)"].
Extract Inductive option => "Prelude.Maybe" ["Prelude.Just" "Prelude.Nothing"].
Extraction Language Haskell.
Extraction "Database" Database.

Once we set the extraction options and run the Extraction command, we’ll end up with a Database.hs file that can be imported by our Haskell code.

The routing part of the web server is here:

importNetwork.WaiasWimportNetwork.Wai.Handler.Warppv_port=1234db_file="images.db"collection_dir="images"dataServerState=ServerState{state_db::TVarDB.ImageDb,state_categories::TVar(M.MapStringDB.N)}typePageTa=(?state::ServerState,?req::Request,?respond::Response->IOResponseReceived)=>a->IOResponseReceivedtypePage=PageT()main::IO()main=dostate<-initializerunpv_port$\reqrespond->let?state=state?req=req?respond=respondindoputStrLn(showreq)case(pathInforeq,requestMethodreq)of([],"GET")->pageIndex()("static":xs,"GET")->pageStatic$T.unpack$T.intercalate"/"xs("images":imageId:_,"GET")->pageImage$param_nimageId("api":"listCategories":_,"GET")->pageListCategories()("api":"listImages":_,"GET")->pageListImages()("api":"tagImage":imageId:catName:_,"POST")->pageTagImage(param_nimageId,catName)("api":"untagImage":imageId:catName:_,"POST")->pageUntagImage(param_nimageId,catName)_->respond(W.responseLBSstatus404[]"???")

And here’s a fragment of Javascript for interacting with this API:

letmkFuncs=function(funcs){return{fetchJson:function(uri,options,onFetch){window.fetch(uri,options).then(function(response){if(!response.ok){console.log("Error: ",response);return;}response.json().then(onFetch);});},tagImage:function(imageId,cat,onTag){funcs.fetchJson("/api/tagImage/"+imageId+"/"+cat,{method:"POST"},onTag);},untagImage:function(imageId,cat,onUntag){funcs.fetchJson("/api/untagImage/"+imageId+"/"+cat,{method:"POST"},onUntag);}}};

In Haskell, a let binding creates a group of mutually-recursive functions. We can do that in Javascript with a Y combinator, or we can unroll it 4-5 times(whatever the maximum stack depth we want):

letfuncs;funcs=mkFuncs(funcs);funcs=mkFuncs(funcs);funcs=mkFuncs(funcs);funcs=mkFuncs(funcs);

Conclusion

Hopefully this example shows that there’s nothing really stopping anyone from using Coq in their Haskell programs today. You can see the full repository on Github.

Future Coq-related articles may cover:

  • An overview of the common tactics and techniques used in proofs
  • Defining custom LTAC commands to automate proofs
  • Writing a more robust Coq-Haskell compatibility layer.
  • Proving mathematical theorems using Coq
  • Implementing compilers or cryptographic primitives in Coq
  • Using GHCJS on the resulting Haskell to get formally-verified Javascript

Future formal verification articles may cover:

  • Using Liquid Haskell to add lightweight proofs to your existing Haskell code
  • Using Idris to develop e.g. a bitcoin exchange
  • Using randomized testing as a cheap substitute for a formal proof.

Let me know if you have a particular preference you’d like to see next.

Beginners Guide to Vulkan

$
0
0

Recently I asked the community for beginner-friendly resources on Vulkan, and I compiled a list of them that you can find below. 

For the beginners reading this, Vulkan is a new graphics API-- in other words, a way to communicate with your GPU and make it do things. It's managed by the Khronos Group, which means it's under multi-company governance - being managed by the industry for the industry. Anyone who wants to do work on GPUs (not restricted to graphics programmers!) should at least have a high level knowledge of what it is. 

What am I looking for in a beginner-friendly tutorial? It should provide a description of what Vulkan is, in words that anyone can understand. After that it can assume some programming knowledge. No graphics API knowledge is assumed.

I compiled this list because I saw many discussing how Vulkan was too hard, or even recommending that beginners master OpenGL first. It reminds me of a lot of the attitudes surrounding C++ when I was first learning it-- yes, it's more lines of code, but that doesn't mean it can't be a fine place for a beginner to start learning. Beginner tutorials are great for people who want to end up doing it professionally, people who want a taste of it to see if they want to bother diving deeper, and for people who want to have a high level knowledge of modern GPU programming in case it's useful in the future.

Dustin puts my feelings to words nicely in his tutorial:

So how did I finally break through that wall of understanding?  I'll give you a hint. It begins with a V and ends with ulkan. "But Vulkan is the hardz" I hear you saying. "Shouldn't I start with something easy like OpenGL?"  Emphatically I say NO.  

The point is that Vulkan removes the mystery.  It spells things out, plain as day. And for a programmer, this is awesome!  It's intellectually honest about what it needs, and what it's doing. It will teach you what it really means to program a GPU. Don't think of all the extra code as a mountain to climb, but instead as a clearly articulated set of instructions to help you get where you need to go.

Go ahead and try out the below tutorials. I've found that everyone has a different favorite depending on learning style, background, and goals-- so if you find yourself getting stuck on one, move on to another and see if that's a better fit.

  1. VULKAN TUTORIAL
  2. A SIMPLE VULKAN COMPUTE EXAMPLE
  3. VULKANO: SAFE RUST WRAPPER AROUND THE VULKAN API
  4. OCULUS VIRTUAL REALITY SDK: "OCULUS ROOM TINY" MINIMAL VULKAN SAMPLE CODE (VK/MAIN.CPP)
  5. MINIMAL EXAMPLE OF USING VULKAN FOR COMPUTE OPERATIONS
  6. I AM GRAPHICS AND SO CAN YOU
  7. API WITHOUT SECRETS: INTRODUCTION TO VULKAN
  8. VULKAN TUTORIAL IN THE LUNARG SDK

MTuner is a C/C++ memory profiler and memory leak finder

$
0
0

README.md

MTuner logo

Build statusLicense

MTuner is a C/C++ memory profiler and memory leak finder for Windows, PS4, PS3, etc.

MTuner utilizes a novel approach to memory profiling and analysis, keeping entire time-based history of memory operations. This gives an unique insight in memory related behavior of your software by making queries over the entire data set.

MTuner screenshot

You can get the latest source code by cloning it from github:

  git clone https://github.com/milostosic/MTuner.git 

Dependencies can be obtained by cloning the following repositories:

git clone https://github.com/milostosic/build.git
git clone https://github.com/milostosic/rbase.git
git clone https://github.com/milostosic/rdebug.git
git clone https://github.com/milostosic/rmem.git
git clone https://github.com/milostosic/rqt.git
git clone https://github.com/milostosic/MTunerCmd.git
git clone https://github.com/milostosic/MTunerDLL.git
git clone https://github.com/milostosic/MTunerInject.git

DIA (Debug Interface Access) SDK - Windows only

git clone https://github.com/milostosic/DIA.git 

To download the official release, please go to MTuner website
Documentation is available here

Consider donating to support the development and maintenance of MTuner

The author of MTuner is Milos Tosic

Copyright (c) 2017 Milos Tosic. All rights reserved.

https://github.com/milostosic/MTuner

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

   1. Redistributions of source code must retain the above copyright notice,
      this list of conditions and the following disclaimer.

   2. Redistributions in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY COPYRIGHT HOLDER ``AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
EVENT SHALL COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 

Things That Turbo Pascal Is Smaller Than (2011)

$
0
0

Turbo Pascal 3 for MS-DOS was released in September 1986. Being version 3, there were lesser releases prior to it and flashier ones after, but 3 was a solid representation of the Turbo Pascal experience: a full Pascal compiler, including extensions that it made it practical for commercial use, tightly integrated with an editor. And the whole thing was lightning fast, orders of magnitude faster at building projects than Microsoft's compilers.

The entire Turbo Pascal 3.02 executable--the compiler and IDE--was 39,731 bytes. How does that stack up in 2011 terms? Here are some things that Turbo Pascal is smaller than, as of October 30, 2011:

The minified version of jquery 1.6 (90,518 bytes).

The yahoo.com home page (219,583 bytes).

The image of the white iPhone 4S at apple.com (190,157 bytes).

zlib.h in the Mac OS X Lion SDK (80,504 bytes).

The touch command under OS X Lion (44,016 bytes).

Various vim quick reference cards as PDFs. (This one is 47,508 bytes.)

The compiled code for the Erlang R14B02 parser (erl_parse.beam, 286,324 bytes).

The Wikipedia page for C++ (214,251 bytes).

(If you liked this, you might like A Personal History of Compilation Speed.)

permalinkOctober 30, 2011

Destroy All Monsters: A Journey Deep into the Caverns of Dungeons and Dragons

$
0
0

If you are on a personal connection, like at home, you can run an anti-virus scan on your device to make sure it is not infected with malware.

If you are at an office or shared network, you can ask the network administrator to run a scan across the network looking for misconfigured or infected devices.

Catching the Great American Eclipse at 35,000 Feet

$
0
0

There’s nothing Alaska Airlines pilots like more than a challenge. As a company that started out flying between remote airfields deep in the Alaskan “bush,” safely navigating where other airlines can’t is in Alaska’s blood.

So, for Alaska, putting a flight in the path of the Great American Eclipse wasn’t really a question of if, but how.

Total solar eclipses aren’t rare – they come around every 16 months or so. Being lucky enough to get in their path is the hard part. Diehard eclipse chasers, called “umbraphiles,” go to drastic lengths to put themselves in the moon’s shadow. They plan years in advance, strategically choosing the best places to catch each and every eclipse, whether that means scaling a mountain, chartering an Arctic voyage… or planning an incredibly complicated flight path.

The math problem goes something like this: A plane leaves Portland flying 500 mph to catch a solar eclipse as the moon’s shadow decelerates from infinity in the instant it touches the Earth to approximately 2,400 mph approaching the coast of Oregon. Where in space and time does the plane need to be to give guests the ultimate eclipse experience?

It’s the kind of equation that takes a team of astronomers and aviators to crack.

Enter Alaska 737 Fleet Captain Brian Holm, Alaska Airlines Captain and eclipse project pilot Hal Andersen and astronomer Glenn Schneider. Both Holm and Andersen have decades of experience in the cockpit. Schneider is an astronomer at the University of Arizona’s Steward Observatory and one of the world’s most esteemed umbraphiles. Schneider has been in the moon’s umbral shadow for 33 total solar eclipses since 1970. In 1992, Schneider even developed his own software, EFLIGHT (which you can read about in detail on Schneider’s website), to calculate flight waypoints for viewing of total solar eclipses.

“It’s a fantastic group,” said Andersen, who captained the regularly-scheduled Alaska flight that intercepted last year’s solar eclipse over the Pacific. “This is a convergence of amazing people who dedicate their lives to chasing eclipses and an airline with a history of pioneering new aviation techniques and technologies.”

A match made in eclipse-chasing heaven. Together, they tackled the Great American Eclipse puzzle – an equation of three major parts that, if solved correctly, would give lucky guests the best view of the eclipse anywhere on the planet.

Part One: Where’s the Sun?

The first problem was determining where the plane needed to be to make the eclipse visible to guests at all.

“Getting a plane in the shadow is easy,” said Andersen. “That’s not the problem. We know the path the eclipse is going to follow – all you have to do is fly along and let the eclipse hit you. That’s all well and good in the cockpit, but if you’re a guest looking out the side window, you just see it get dark outside and then light again. You miss the whole show.”

The flight would have to catch the eclipse with the sun low in the sky for guests to see the celestial mechanics at work. How low is low? To find their answer, the team measured the 737’s cabin, and the dimensions and placement of the windows on a Boeing 737-900ER. They then estimated the average guest’s seated eye level and the distance from the windows to find the optimal viewing angle. Long story short, the flight needed to catch the sun at an angle of less than 30 degrees above the horizon.

Part Two: Where does the plane need to be?

Beyond catching the sun at the right level above the horizon, the flight would also need to perfectly time its run across the eclipse’s path, putting the sun directly in guests’ line-of-sight as the eclipse reached totality. It was time to map out the “waypoints” – exactly when and where to put the plane to make the whole plan come together.

That’s where Schneider took over. By feeding his EFLIGHT program variables including the required viewing angle, estimated aircraft speed, cruising altitude, anticipated wind speeds and more, the program produced three key waypoints and the precise times to hit them to view the eclipse.

According to Schneider, identifying those points is the easy part. As he nonchalantly puts it, “celestial mechanics are gonna do what they’re gonna do.”

But as it turned out, those points were nowhere near… anywhere. To catch the eclipse at the perfect place and time, Andersen and Holm would have to lead their merry eclipse chase out of Portland to a point a couple of hours off the U.S. west coast, before returning back to where they started. Perhaps Alaska’s longest-ever flight to nowhere.

Before sending the calculations to Alaska, Schneider showed them to fellow umbraphile Joe Rao, Schneider’s co-conspirator in detouring an Alaska flight to catch the 2016 total solar eclipse. Rao said, “they’re never going to go for this. We’re suggesting a five-hour flight. That’s a hell of a lot of jet fuel.”

But, much like Schneider’s celestial mechanics, an unstoppable plan was in motion. In less than 24 hours, Alaska’s leadership gave the OK to make the flight happen. Now it was a matter of planning, preparation and piloting skill.

Part three: How do we hit the waypoints at the right time?

Hitting Schneider’s three waypoints was easier said than done, subject to variables that he calls “ponderables, but unknowables.”

“If you get delayed on the ground or have an unfavorable wind or air traffic control denies you a corridor at the last minute, you’re in trouble,” said Schneider. “You just have to anticipate what the worst case may be and try to plan for it.”

As planning got underway, Captain Andersen outlined the problem to colleagues in football terms. “The eclipse is a pass from Peyton Manning… or Russell Wilson,” he quickly corrected, remembering who he was talking to (one of the perils of a 40-year career spent hopping between cities).

“We have to run down the field and decide exactly when to cut across to make the catch. And we have to do it exactly at the 50-yard line so everyone can see it. And do it facing the photographers so they get a perfect photo op.”

With the moon’s shadow screaming across the globe at thousands of miles per hour, making up for lost time would be almost impossible. So, Andersen and Holm built a make-up point into the flight plan, or a “get well point” as Andersen called it in his days piloting C-130s in the Air Force. In other words, an opportunity to cut the corner on the flight path before hitting Schneider’s eclipse viewing waypoints, closing the gap on a late start of up to five minutes.

“This only helps us if we’re only running a little late,” Andersen reminded his team as they reviewed the flight plan in the weeks leading up to the eclipse, the whiteboard behind him a maze of flight path possibilities. “Every second counts on this.”

If the flight ran more than five minutes behind, Andersen and Holm could attempt to catch the eclipse later in its journey as it neared the coast. The sun would be higher in the sky, negating the hard work to hit the perfect viewing angle, but allowing another few minutes of vital wiggle room.

And wiggle room is important when it means the difference between the experience of a lifetime and long, uncomfortable flight back to Portland with a planeload of very grumpy umbraphiles.

Putting it all together

As a guest on the flight, you’d never guess at the hundreds of emails, calculations, charts and contingency plans that transformed the Great American Eclipse Flight from a wild idea into a workable plan.

Unlike the usual flight’s mix of nappers, readers and phone fiddlers, guests buzzed with anticipation. Cheers erupted the moment the flight lifted off. When the fasten seatbelt sign went dark, the party started. Guests filled the aisle, reminiscing about past eclipse adventures, chatting with media, and collecting autographs from high-profile guests like NASA Astronaut Dr. Michael R. Barratt. Making up for his early pessimism at the five-hour flight plan, Joe Rao held court in row 9, waxing poetic about the views guests could expect as the moon eased in front of the sun.

The mood on the flight deck was a little different, more like frenetic kitchen staff working behind closed doors to serve a seven-course masterpiece. A steady stream of course corrections, wind calculations and communications with Oceanic Air Traffic Control. Despite the pressure, Andersen hopped on the PA to sing a few bars of Europe’s “The Final Countdown” while guests led by Rao shouted “TEN! NINE! EIGHT!” from the cabin. Seasoned umbraphiles and first-timers alike pressed their noses to the glass, ravenous for a 103 second glimpse of totality that had been months in the making.

And just like that, the moment was over. Sun and moon continuing on their way toward the coast, leaving behind an awestruck cast of eclipse chasers eager to touchdown in Portland and share their adventure with the world.

They landed back where they’d started five hours earlier, but with a memory to last a lifetime. Nearly two minutes in the totality of a solar eclipse at 38,000 feet. A moment free of clouds and haze, only the thinnest of atmosphere between them and a phenomenon that has been equal parts feared, celebrated and honored throughout human history.

Photos: Alaska Airlines Great American Eclipse Flight

Photos by Ingrid Barrentine/Alaska Airlines.

Fun vs. Computer Science (2016)

$
0
0

I've spent most of my career working on games, either programming or designing them or both. Games are weird, because everything comes down to this nebulous thing called fun, and there's a complete disconnect between fun and most technical decisions:

Does choosing C++14 over C++11 mean the resulting game is more fun?

Does using a stricter type system mean the game is more fun?

Does using a more modern programming language mean the game is more fun?

Does favoring composition over inheritance mean the game is more fun?

Now you could claim that some of this tech would be more fun for the developer. That's a reasonable, maybe even important point, but there's still a hazy at best connection between this kind of "developer fun" and "player fun."

A better argument is that some technologies may result in the game being more stable and reliable. Those two terms should be a prerequisite to fun, and even though people struggle along--and have fun with--buggy games (e.g., Pokemon Go), I'm not going to argue against the importance of reliability. Think about all the glitchiness and clunkiness you experience every day, from spinning cursors, to Java tricking you into installing the Ask toolbar, to an app jumping into the foreground so you click on the wrong thing. Now re-watch The Martian and pretend all the computers in the movie work like your desktop PC. RIP Matt Damon.

The one thing that does directly make a game more fun is decreased iteration time. Interactive tweaking beats a batch compile and re-launch every time, and great ideas can come from on the fly experimentation. The productivity win, given the right tools, is 10x or more, and I can't emphasize this enough.

And yet this more rapid iteration, which is so important to me, does not seem to be of generally great importance. It's not something that comes up in computer sciencey discussions of development technologies. There's much focus on sophisticated, and slow, code optimization, but turnaround time is much more important in my work. A certain circle of programmers puts type systems above all else, yet in Bret Victor's inspirational Inventing on Principle talk from 2012, he never mentioned type systems, not once, but oh that interactivity.

I realize that we're heading toward the ultimate software engineer dream of making a type-checked change that's run through a proven-correct compiler that does machine-learning driven, whole program optimization...but it's going the exact opposite of the direction I want. It's not helping me in my quest for creating fun.

For the record, I just picked those buzzwords out of my mind. I'm not criticizing static type checking or any of those things, or even saying that they preclude interactive iteration (see Swift's playgrounds, for example). They might make things harder though, if they necessitate building a new executable of the entire game for every little change.

Interactivity, I may have to grudgingly accept, is not trendy in computer science circles.

(If you liked this, you might enjoy You Don't Read Code, You Explore It.)

permalinkJuly 26, 2016

AdNauseam – Clicking Ads So You Don't Have To

$
0
0

As online advertising becomes ever more ubiquitous and unsanctioned, AdNauseam works to complete the cycle by automating Ad clicks universally and blindly on behalf of its users. Built atop uBlock Origin, AdNauseam quietly clicks on every blocked ad, registering a visit on ad networks' databases. As the collected data gathered shows an omnivorous click-stream, user tracking, targeting and surveillance become futile.

AdNauseam is a free browser extension designed to obfuscate browsing data and protect users from tracking by advertising networks. At the same time, AdNauseam serves as a means of amplifying users' discontent with advertising networks that disregard privacy and facilitate bulk surveillance agendas.

AdNauseam joins a broader class of technical systems that attempt to serve ethical, political, and expressive ends. In light of the industry'sfailure to self-regulate or otherwise address the excesses of network tracking, AdNauseam allows individual users to take matters into their own hands, fighting back against unilateral surveillance. Taken in this light, the software follows an approach similar to that of TrackMeNot, employing obfuscation as a strategy to shift the balance of power between the trackers and the tracked. For further information on this approach, please see this paper.

If the Waffle House is closed, it's Time to Panic (2016)

$
0
0

On a warm, cloudy morning in the first week of October, in an anonymous office park just outside Atlanta, operations analyst Matt Stark opened a computer program, ran through some data and looked thoughtfully at the results.

Out in the Atlantic Ocean, Hurricane Matthew was hurling winds of 115 miles an hour toward the coast of Florida. Hundreds of miles inland, in the headquarters of Waffle House Inc., Stark’s software predicted that 477 of the chain’s almost 1,900 restaurants might be affected by the onrushing storm.

This meant two things. First, as the storm made landfall, some locations of Waffle House — which boasts that every restaurant stays open 24 hours a day, 365 days a year — would probably have to close because of power loss or concerns for workers’ safety. And second, sometime after they did, someone would invoke the “Waffle House Index,” the slightly flippant measure of how bad a storm can get.

And Matthew brought on both those expected scenarios. Waffle House announced Oct. 6 that it was pre-emptively closing some restaurants on a 90-mile stretch of Interstate 95 between Fort Pierce and Titusville in Florida. (In the next few days, as the storm churned up the coast and flooded North Carolina, it would close 98 all told.) And as soon as the announcement went out, media tracking the storm, and customers on social media, invoked the closings as a sign of the apocalypse.

The Miami Herald: “When Waffle House surrenders to a hurricane, you know it’s bad.” The Washington Post: “Hurricane Matthew is so scary even the always-open eatery is evacuating.” A faithful customer on Twitter: “GOD IN HEAVEN THIS IS THE END!”

In those areas, the Waffle House Index had just gone to red.

Disaster responders pay attention to that index, which was created — in the midst of 2004’s devastating Hurricane Charley — by W. Craig Fugate, administrator of the Federal Emergency Management Agency since 2009. Fugate was director of emergency management for Florida when Charley slammed the state with unexpected force: Its winds strengthened abruptly and it went from a Category 2 to a Category 4, and the storm suddenly changed direction and struck the state’s Gulf Coast at Sanibel, 150 miles south of its predicted landfall. Tens of thousands of people were reportedly left homeless.

Fugate was in his office with state meteorologist Ben Nelson and members of the Florida National Guard, color-coding infrastructure loss on a map — green for operating, yellow for affected, and red for destroyed — and the group decided to take a look at some of the damage, and try to find a meal.

“They went to a Waffle House and noticed they had a limited menu, with nonperishable items,” Alexa Lopez, FEMA’s press secretary, told me. “The next day, they were driving around and they went to a different Waffle House, and the same thing happened, a limited menu.”

So, she said, the group was inspired first to rank Waffle Houses in the same way: green for fully operational, yellow for a limited menu and red for closed. “Which is pretty bad, because Waffle House is always open,” Lopez added. And, second, to use those observations as a proxy for how much a disaster disrupts a community. Fugate has since been quoted as saying: “If you get there and the Waffle House is closed? That’s really bad. That’s where you go to work.”

The concept of restaurant operations as an indicator of storm impact percolated slowly into emergency-management culture — the magazine Environment Health Safety Today wrote about it in July 2011— and broke out into the open around the time of Hurricane Irene in August 2011 (when, according to The Wall Street Journal, 22 Waffle Houses lost power but only one stayed closed longer than a day).

But the so-called index isn’t actually an official metric. FEMA doesn’t publish it anywhere; no one, except for Waffle House itself, counts how many restaurants are running lean or forced to close. But the company does give that count to the agency, and FEMA uses it — along with wind speeds and power outages and other objective measures — to judge a storm’s impact, and to figure out where its own crews and other emergency responders can get fed.

So far, so obvious: If a storm is bad enough to close restaurants that “never close,” we can agree that storm is bad. Hurricane Matthew was the strongest storm of the 2016 Atlantic hurricane season, which ended Nov. 30.

But the Waffle House Index also stands for something less obvious. It is an indicator of how complex and long supply chains are — for food, for fuel, for power — and of what it takes to plan around infrastructure that can be fragile in unexpected ways.

“The essence of the index is not just that the situation is bad,” Panos Kouvelis, director of the Boeing Center for Supply Chain Innovation at Washington University in St. Louis, told me. “Companies like Waffle House, and Wal-Mart and Home Depot, operate in areas that are frequently hit by disasters, where their operations may go down at the same time the demand for their services go up. So they have had to develop very well-defined playbooks for being prepared.”

But preparation is a complicated endeavor in an economy where few things are sold in the places where they were made. Vanilla comes from Madagascar. The United States imports most of its salt. The plywood needed to protect windows from storm debris might have been manufactured in China, and a storm in the Gulf of Mexico can force enough interruption on petroleum refining to choke off supply on the other side of the country.

“When I talk to companies about whether they understand their supply chains, they generally know one step up and back: who they buy from and who they sell to,” said Amy Kircher, director of the Food Protection and Defense Institute at the University of Minnesota. “They don’t understand that there are five or 10 steps in the chain before the product reaches them. Or that the alternate suppliers they consider their contingency plan are all buying from the same place.”

At Waffle House headquarters in November, Stark and his colleagues on the chain’s storm team — Vice Presidents Pat Warner and Will Mizell and Communications Director Kelly Thrasher-Bruner, who handles social media in disasters — walked me through how they prepare. As we talked, Stark pulled up an updated post-Matthew map. Of 200 restaurants that ended up in the storm’s path, just one, in inland North Carolina, gleamed red, for “still affected”; it had been flooded and needed cleaning out.

“It’s a big deal for us to shut down, because we’re not used to turning everything off and turning the lights off and closing the door,” said Warner, who estimates that he has worked “more than 10” hurricane responses in 17 years. “So our goal is to open up as quickly as possible afterward. The operations team works with the distributor to get food ready to go in. The construction team lines up generators. If you have generators you have to have fuel, so we line up that.”

On the edge of the predicted storm zone — which Stark monitors from a temporary “war room” assembled by putting mobile giant screens in a conference room — the company positions personnel who can swoop in: carpenters, electricians, IT specialists, a food-safety expert and someone to talk to local governments and law enforcement and soothe concerns about curfews. A little farther out, restaurants in other markets line up “jump teams”: spare personnel who volunteer to work in place of locals who might have evacuated or might need to repair their homes or care for family. In Hurricane Matthew, the company sent in an extra 250 people.

“We say we throw chaos at chaos,” Mizell said. “We just throw a lot of resources down there to get restaurants open. Our CEO will be there. In Matthew, our chairman was there, too.”

Before the carpenters and computer specialists or the replacement cooks and servers arrive, the company assesses how long it has been since supplies were delivered and — just as crucial — how long since the local trash removal company last emptied the dumpster. “Most restaurants get a delivery once a week,” Stark said. “If it got there Wednesday and the storm hits Thursday, they should have enough food, but if the storm hits Tuesday, we may have to hurry up and get some food there.”

Waffle House uses one main distributor, Illinois-based US Foods, which has depots scattered across the Southeast, where most Waffle Houses are concentrated, and where many hurricanes that strike the U.S. make landfall. So supplies don’t need to travel far in advance of a storm and are close by once roads are clear.

That model of staging supplies and personnel in layers outside an emergency, in order to swoop in quickly once it abates, isn’t unique to Waffle House. It’s also followed by the military and by the Centers for Disease Control and Prevention, which oversees the Strategic National Stockpile of emergency medications, vaccines and antidotes for natural disasters and biological attacks.

“We think about, What are the timelines in which people have to be provided medications?” Greg Burel, the stockpile’s director, told me. “Our primary stock is laid down in undisclosed locations around the country that give us access to large swaths of the population and that are accessible” by more than one type of transportation.

But the CDC also puts smaller, lighter assets — the equivalent of a temporary field hospital or a stash of medications that could be deployed immediately — “as far forward as we can if we have advance notice, but not so far forward that it becomes victim to that event,” Burel said.

The first rule of operating in a disaster, though, is keeping personnel safe. Every Waffle House employee gets a key fob with lists of relevant phone numbers. Local managers keep track of who has challenging home situations — disabled parents, children with special needs, single heads of household — and might need extra help. Every location has a wall-mounted “crisis response” flier that includes detachable wallet cards listing personal-preparedness steps for hurricanes, tornadoes and ice storms.

The cards also remind employees how needed they will be at work. “We will be very busy,” the current hurricane-preparedness card says, “and you will make lots of money!”

waffle_inline

Once employees arrive or return and a location can open, they work through the storm manual: a giant binder, a kind of choose-your-own-adventure game composed in eggs and grits. “Here’s how you run the restaurant if you’re without power, without gas for the grills, without water,” Mizell said. “We can cook on the grills even if we don’t have electricity. We can bring in bottled water and canned Coke products and serve on to-go plates. We can get porta-potties.”

The storm team examines what worked and didn’t after every disaster, each time learning new lessons. In Hurricane Katrina — when they closed 107 locations in Mississippi and on the north shore of Lake Ponchartrain — roads were so bad they stashed supplies in a refrigerated semi-trailer with armed guards, and fuel so scarce they brought in their own tank trucks. When Katrina struck in 2005, Twitter had not yet launched; now Bruner uses it not just to alert customers to closings, but to crowdsource whether roads are open and where the power is on.

Waffle House began working on storm response before Katrina — Warner said he thinks the first organized attempt was Hurricane Hugo, which hit Charleston, South Carolina, in September 1989 — but from then to now, the company’s biggest need in disasters hasn’t changed.

“People,” Warner said. “We can get the food there, by hook or by crook. It costs more, but we’re willing to spend it. But if we don’t have people, we can’t open up.”

Category Theory

$
0
0
First published Fri Dec 6, 1996; substantive revision Fri Oct 3, 2014

Category theory has come to occupy a central position in contemporary mathematics and theoretical computer science, and is also applied to mathematical physics. Roughly, it is a general mathematical theory of structures and of systems of structures. As category theory is still evolving, its functions are correspondingly developing, expanding and multiplying. At minimum, it is a powerful language, or conceptual framework, allowing us to see the universal components of a family of structures of a given kind, and how structures of different kinds are interrelated. Category theory is both an interesting object of philosophical study, and a potentially powerful formal tool for philosophical investigations of concepts such as space, system, and even truth. It can be applied to the study of logical systems in which case category theory is called “categorical doctrines” at the syntactic, proof-theoretic, and semantic levels. Category theory is an alternative to set theory as a foundation for mathematics. As such, it raises many issues about mathematical ontology and epistemology. Category theory thus affords philosophers and logicians much to use and reflect upon.

1. General Definitions, Examples and Applications

1.1 Definitions

Categories are algebraic structures with many complementary natures, e.g., geometric, logical, computational, combinatorial, just as groups are many-faceted algebraic structures. Eilenberg & Mac Lane (1945) introduced categories in a purely auxiliary fashion, as preparation for what they called functors and natural transformations. The very definition of a category evolved over time, according to the author's chosen goals and metamathematical framework. Eilenberg & Mac Lane at first gave a purely abstract definition of a category, along the lines of the axiomatic definition of a group. Others, starting with Grothendieck (1957) and Freyd (1964), elected for reasons of practicality to define categories in set-theoretic terms.

An alternative approach, that of Lawvere (1963, 1966), begins by characterizing the category of categories, and then stipulates that a category is an object of that universe. This approach, under active development by various mathematicians, logicians and mathematical physicists, lead to what are now called “higher-dimensional categories” (Baez 1997, Baez & Dolan 1998a, Batanin 1998, Leinster 2002, Hermida et al. 2000, 2001, 2002). The very definition of a category is not without philosophical importance, since one of the objections to category theory as a foundational framework is the claim that since categories are defined as sets, category theory cannot provide a philosophically enlightening foundation for mathematics. We will briefly go over some of these definitions, starting with Eilenberg's & Mac Lane's (1945) algebraic definition. However, before going any further, the following definition will be required.

Definition: A mapping e will be called anidentity if and only if the existence of any producteα or βe implies that eα = α and βe = β

Definition (Eilenberg & Mac Lane 1945): A category C is an aggregate Ob of abstract elements, called the objects of C, and abstract elements Map, called mappings of the category. The mappings are subject to the following five axioms:

(C1) Given three mappings α1, α2 and α3, the triple product α32α1) is defined if and only if (α3α21 is defined. When either is defined, the associative law
α32α1) = (α3α21

holds. This triple product is written α3α2α1.

(C2) The triple product α3α2α1 is defined whenever both products α3α2 and α2α1 are defined.

(C3) For each mapping α, there is at least one identity e1 such that αe1 is defined, and at least one identity e2 such thate2α is defined.

(C4) The mapping eX corresponding to each objectX is an identity.

(C5) For each identity e there is a unique objectX of C such that eX = e.

As Eilenberg & Mac Lane promptly remark, objects play a secondary role and could be entirely omitted from the definition. Doing so, however, would make the manipulation of the applications less convenient. It is practically suitable,and perhaps psychologically more simple to think in terms of mappings and objects. The term “aggregate” is used by Eilenberg & Mac Lane themselves, presumably so as to remain neutral with respect to the background set theory one wants to adopt.

Eilenberg & Mac Lane defined categories in 1945 for reasons of rigor. As they note:

It should be observed first that the whole concept of a category is essentially an auxiliary one; our basic concepts are essentially those of a functor and of natural transformation (…). The idea of a category is required only by the precept that every function should have a definite class as domain and a definite class as range, for the categories are provided as the domains and ranges of functors. Thus one could drop the category concept altogether and adopt an even more intuitive standpoint, in which a functor such as “Hom” is not defined over the category of “all” groups, but for each particular pair of groups which may be given. The standpoint would suffice for applications, inasmuch as none of our developments will involve elaborate constructions on the categories themselves. (1945, chap. 1, par. 6, p. 247)

Things changed in the following ten years, when categories started to be used in homology theory and homological algebra. Mac Lane, Buchsbaum, Grothendieck and Heller were considering categories in which the collections of morphisms between two fixed objects have an additional structure. More specifically, given any two objects X andY of a category C, the setHom(X, Y) of morphisms fromX to Y form an abelian group. Furthermore, for reasons related to the ways homology and cohomology theories are linked, the definition of a category had to satisfy an additional formal property (which we will leave aside for the moment): it had to be self-dual. These requirements lead to the following definition.

Definition: A category C can be described as a set Ob, whose members are the objects of C, satisfying the following three conditions:
Morphism : For every pair X,Y of objects, there is a set Hom(X,Y), called the morphisms from X toY in C. If f is a morphism from X to Y, we writef : XY.

Identity : For every object X, there exists a morphism idX inHom(X, X), called theidentity on X.

Composition : For every triple X,Y and Z of objects, there exists a partial binary operation from Hom(X, Y) × Hom(Y, Z) toHom(X, Z), called the composition of morphisms in C. Iff : XY andg : YZ, the composition of f and g is notated (gf ) : XZ.

Identity, morphisms, and composition satisfy two axioms:

Associativity : If f : XY,g : YZ andh : ZW, thenh○ (gf) = (hg)○ f.

Identity : If f : XY, then (idYf) = f and (f○ idX) = f.

This is the definition one finds in most textbooks of category theory. As such it explicitly relies on a set theoretical background and language. An alternative, suggested by Lawvere in the early sixties, is to develop an adequate language and background framework for a category of categories. We will not present the formal framework here, for it would take us too far from our main concern, but the basic idea is to define what are called weak n-categories (and weak ω-categories), and what had been called categories would then be called weak 1-categories (and sets would be weak 0-categories). (See, for instance, Baez 1997, Makkai 1998, Leinster 2004, Baez & May 2010, Simpson 2011.)

Also in the sixties, Lambek proposed to look at categories as deductive systems. This begins with the notion of a graph, consisting of two classes Arrows and Objects, and two mappings between them, s : Arrows → Objects andt : ArrowsObjects, namely the source and the target mappings. The arrows are usually called the “oriented edges” and the objects “nodes” or “vertices”. Following this, a deductive system is a graph with a specified arrow:

(R1) idX : XX,

and a binary operation on arrows:

(R2) Given f : XY and g : YZ, the composition of f andg is (gf) : XZ.

Of course, the objects of a deductive system are normally thought of as formulas, the arrows are thought of as proofs ordeductions, and operations on arrows are thought of asrules of inference. A category is then defined thus:

Definition (Lambek): A category is a deductive system in which the following equations hold between proofs: for all f : XY,g : YZ andh: ZW,

(E1) f○ idX =f,   idYf =f,   h○ (gf) = (hg)○ f.

Thus, by imposing an adequate equivalence relation upon proofs, any deductive system can be turned into a category. It is therefore legitimate to think of a category as an algebraic encoding of a deductive system. This phenomenon is already well-known to logicians, but probably not to its fullest extent. An example of such an algebraic encoding is the Lindenbaum-Tarski algebra, a Boolean algebra corresponding to classical propositional logic. Since a Boolean algebra is a poset, it is also a category. (Notice also that Boolean algebras with appropriate homomorphisms between them form another useful category in logic.) Thus far we have merely a change of vocabulary. Things become more interesting when first-order and higher-order logics are considered. The Lindenbaum-Tarski algebra for these systems, when properly carried out, yields categories, sometimes called “conceptual categories” or “syntactic categories” (Mac Lane & Moerdijk 1992, Makkai & Reyes 1977, Pitts 2000).

1.2 Examples

Almost every known example of a mathematical structure with the appropriate structure-preserving map yields a category.

  1. The category Set with objects sets and morphisms the usual functions. There are variants here: one can consider partial functions instead, or injective functions or again surjective functions. In each case, the category thus constructed is different
  2. The category Top with objects topological spaces and morphisms continuous functions. Again, one could restrict morphisms toopen continuous functions and obtain a different category.
  3. The category hoTop with objects topological spaces and morphisms equivalence classes of homotopic functions. This category is not only important in mathematical practice, it is at the core of algebraic topology, but it is also a fundamental example of a category in which morphisms are not structure preserving functions.
  4. The category Vec with objects vector spaces and morphisms linear maps.
  5. The category Diff with objects differential manifolds and morphisms smooth maps.
  6. The categories Pord and PoSet with objects preorders and posets, respectively, and morphisms monotone functions.
  7. The categories Lat and Bool with objects lattices and Boolean algebras, respectively, and morphisms structure preserving homomorphisms, i.e., (⊤, ⊥, ∧, ∨) homomorphisms.
  8. The category Heyt with objects Heyting algebras and (⊤, ⊥, ∧, ∨, →) homomorphisms.
  9. The category Mon with objects monoids and morphisms monoid homomorphisms.
  10. The category AbGrp with objects abelian groups and morphisms group homomorphisms, i.e. (1, ×, ?) homomorphisms
  11. The category Grp with objects groups and morphisms group homomorphisms, i.e. (1, ×, ?) homomorphisms
  12. The category Rings with objects rings (with unit) and morphisms ring homomorphisms, i.e. (0, 1, +, ×) homomorphisms.
  13. The category Fields with objects fields and morphisms fields homomorphisms, i.e. (0, 1, +, ×) homomorphisms.
  14. Any deductive system T with objects formulae and morphisms proofs.

These examples nicely illustrates how category theory treats the notion of structure in a uniform manner. Note that a category is characterized by its morphisms, and not by its objects. Thus the category of topological spaces with open maps differs from the category of topological spaces with continuous maps — or, more to the point, the categorical properties of the latter differ from those of the former.

We should underline again the fact that not all categories are made of structured sets with structure-preserving maps. Thus any preordered set is a category. For given two elements p, q of a preordered set, there is a morphismf : p → q if and only ifp ≤ q. Hence a preordered set is a category in which there is at most one morphism between any two objects. Any monoid (and thus any group) can be seen as a category: in this case the category has only one object, and its morphisms are the elements of the monoid. Composition of morphisms corresponds to multiplication of monoid elements. That the monoid axioms correspond to the category axioms is easily verified.

Hence the notion of category generalizes those of preorder and monoid. We should also point out that a groupoid has a very simple definition in a categorical context: it is a category in which every morphism is an isomorphism, that is for any morphism f : XY, there is a morphism g : YX such thatfg =idX andgf =idY.

1.3 Fundamental Concepts of the Theory

Category theory unifies mathematical structures in two different ways. First, as we have seen, almost every set theoretically defined mathematical structure with the appropriate notion of homomorphism yields a category. This is a unification provided within a set theoretical environment. Second, and perhaps even more important, once a type of structure has been defined, it is imperative to determine how new structures can be constructed out of the given one. For instance, given two sets A and B, set theory allows us to construct their Cartesian product A× B. It is also imperative to determine how given structures can be decomposed into more elementary substructures. For example, given a finite Abelian group, how can it be decomposed into a product of certain of its subgroups? In both cases, it is necessary to know how structures of a certain kind may combine. The nature of these combinations might appear to be considerably different when looked at from a purely set theoretical perspective.

Category theory reveals that many of these constructions are in fact certain objects in a category having a “universal property”. Indeed, from a categorical point of view, a Cartesian product in set theory, a direct product of groups (Abelian or otherwise), a product of topological spaces, and a conjunction of propositions in a deductive system are all instances of a categorical product characterized by a universal property. Formally, a product of two objectsX and Y in a category C is an objectZ of Ctogether with two morphisms, called the projections, p : ZX and q : ZY such that—and this is the universal property—for all objects W with morphisms f : WX and g : WY, there is a unique morphismh : WZ such thatph = f and qh =g.

Note that we have defined a product for X andY and not the product for X and Y. Indeed, products and other objects with a universal property are defined only up to a (unique) isomorphism. Thus in category theory, the nature of the elements constituting a certain construction is irrelevant. What matters is the way an object is related to the other objects of the category, that is, the morphisms going in and the morphisms going out, or, put differently, how certain structures can be mapped into a given object and how a given object can map its structure into other structures of the same kind.

Category theory reveals how different kinds of structures are related to one another. For instance, in algebraic topology, topological spaces are related to groups (and modules, rings, etc.) in various ways (such as homology, cohomology, homotopy, K-theory). As noted above, groups with group homomorphisms constitute a category. Eilenberg & Mac Lane invented category theory precisely in order to clarify and compare these connections. What matters are the morphisms between categories, given by functors. Informally, functors are structure-preserving maps between categories. Given two categories C and D, a functor F fromC to D sends objects ofC to objects of D, and morphisms ofC to morphisms of D, in such a way that composition of morphisms in C is preserved, i.e.,F(gf) =F(g)○ F(f), and identity morphisms are preserved, i.e.,F(idX) =idFX. It immediately follows that a functor preserves commutativity of diagrams between categories. Homology, cohomology, homotopy, K-theory are all example of functors.

A more direct example is provided by the power set operation, which yields two functors on the category of sets, depending on how one defines its action on functions. Thus given a set X, ℘(X) is the usual set of subsets of X, and given a function f : XY, ℘(f) : ℘(X) → ℘(Y) takes a subsetA of X and maps it to B =f(A), the image off restricted to A in X. It is easily verified that this defines a functor from the category of sets into itself.

In general, there are many functors between two given categories, and the question of how they are connected suggests itself. For instance, given a category C, there is always the identity functor fromC to C which sends every object/morphism of C to itself. In particular, there is the identity functor over the category of sets.

Now, the identity functor is related in a natural manner to the power set functor described above. Indeed, given a set X and its power set ℘(X), there is a functionhX which takes an elementx of X and sends it to the singleton set {x}, a subset of X, i.e., an element of ℘(X). This function in fact belongs to a family of functions indexed by the objects of the category of sets {hY : Y→ ℘(X) | Y in Ob(Set)}. Moreover, it satisfies the following commutativity condition: given any function f : XY, the identity functor yields the same functionId(f) : Id(X) → Id(Y). The commutativity condition thus becomes: hYId(f) = ℘(f)○ hX. Thus the family of functions h(-) relates the two functors in a natural manner. Such families of morphisms are callednatural transformations between functors. Similarly, natural transformations between models of a theory yield the usual homomorphisms of structures in the traditional set theoretical framework.

The above notions, while important, are not fundamental to category theory. The latter heading arguably include the notions of limit/colimit; in turn, these are special cases of what is certainly the cornerstone of category theory, the concept of adjoint functors, first defined by Daniel Kan in 1956 and published in 1958.

Adjoint functors can be thought of as being conceptual inverses. This is probably best illustrated by an example. Let U : GrpSet be the forgetful functor, that is, the functor that sends to each group G its underlying set of elements U(G), and to a group homomorphism f : GH the underlying set functionU(f) : U(G) → U(H). In other words, U forgets about the group structure and forgets the fact that morphisms are group homomorphisms. The categories Grp andSet are certainly not isomorphic, as categories, to one another. (A simple argument runs as follows: the categoryGrp has a zero object, whereas Set does not.) Thus, we certainly cannot find an inverse, in the usual algebraic sense, to the functor U. But there are many non-isomorphic ways to define a group structure on a given setX, and one might hope that among these constructions at least one is functorial and systematically related to the functor U. What is the conceptual inverse to the operation of forgetting all the group theoretical structure and obtaining a set? It is to construct a group from a set solely on the basis of the concept of group and nothing else, i.e., with no extraneous relation or data. Such a group is constructed “freely”; that is, with no restriction whatsoever except those imposed by the axioms of the theory. In other words, all that is remembered in the process of constructing a group from a given set is the fact that the resulting construction has to be a group. Such a construction exists; it is functorial and it yields what are called free groups. In other words, there is a functorF : SetGrp, which to any set X assigns the free groupF(X) on X, and to each functionf : XY, the group homomorphism F(f) : F(X) → F(Y), defined in the obvious manner. The situation can be described thusly: we have two conceptual contexts, a group theoretical context and a set theoretical context, and two functors moving systematically from one context to the other in opposite directions. One of these functors is elementary, namely the forgetful functor U. It is apparently trivial and uninformative. The other functor is mathematically significant and important. The surprising fact is that F is related toU by a simple rule and, in some sense, it arises fromU. One of the striking features of adjoint situations is precisely the fact that fundamental mathematical and logical constructions arise out of given and often elementary functors.

The fact that U and F are conceptual inverses expresses itself formally as follows: applying F first and then U does not yield the original set X, but there is a fundamental relationship between X andUF(X). Indeed, there is a function η : X → UF(X), called the unit of the adjunction, that simply sends each element of X to itself in UF(X) and this function satisfies the following universal property: given any functiong : XU(G), there is a unique group homomorphismh : F(X) → G such that U(h)○ η = g. In other words, UF(X) is the best possible solution to the problem of inserting elements ofX into a group (what is called “insertion of generators” in the mathematical jargon). Composing U and F in the opposite order, we get a morphism ξ : FU(G) → G, called the counit of the adjunction, satisfying the following universal property: for any group homomorphismg : F(X) → G, there is a unique functionh : XU(G) such that ξ ○ F(h) = gFU(G) constitutes the best possible solution to the problem of finding a representation of G as a quotient of a free group. IfU and F were simple algebraic inverses to one another, we would have the following identity: UF =ISet and FU =IGrp, whereISet denotes the identity functor on Set and IGrp the identity functor on Grp. As we have indicated, these identities certainly do not hold in this case. However, some identities do hold: they are best expressed with the help of the commutative diagrams:

Uη ○ U
UFU      FF○ η
FUF
U○ η      ξ ○ F
U      F

where the diagonal arrows denote the appropriate identity natural transformations.

This is but one case of a very common situation: every free construction can be described as arising from an appropriate forgetful functor between two adequately chosen categories. The number of mathematical constructions that can be described as adjoints is simply stunning. Although the details of each one of these constructions vary considerably, the fact that they can all be described using the same language illustrates the profound unity of mathematical concepts and mathematical thinking. Before we give more examples, a formal and abstract definition of adjoint functors is in order.

Definition: Let F : CD and G : DC be functors going in opposite directions. F is a left adjoint toG (G a right adjoint to F), denoted by FG, if there exists natural transformations η : ICGF and ξ : FG → ID, such that the composites
Gη ○ G
GFGG○ ξ
G

and

FF○ η
FGFξ ○ F
F

are the identity natural transformations. (For different but equivalent definitions, see Mac Lane 1971 or 1998, chap. IV.)

Here are some of the important facts regarding adjoint functors. Firstly, adjoints are unique up to isomorphism; that is any two left adjoints F and F' of a functor G are naturally isomorphic. Secondly, the notion of adjointness is formally equivalent to the notion of a universal morphism (or construction) and to that of representable functor. (See, for instance Mac Lane 1998, chap. IV.) Each and every one of these notions exhibit an aspect of a given situation. Thirdly, a left adjoint preserves all the colimits which exist in its domain, and, dually, a right adjoint preserves all the limits which exist in its domain.

We now give some examples of adjoint situations to illustrate the pervasiveness of the notion.

  1. Instead of having a forgetful functor going into the category of sets, in some cases only a part of the structure is forgotten. Here are two standard examples:
    • There is an obvious forgetful functor U : AbGrpAbMon from the category of abelian groups to the category of abelian monoids: U forgets about the inverse operation. The functor U has a left adjointF: AbMonAbGrp which, given an abelian monoid M, assigns to it the best possible abelian group F(M) such that M can be embedded inF(M) as a submonoid. For instance, if M is ℕ, then F(ℕ) “is” ℤ, that is, it is isomorphic to ℤ.
    • Similarly, there is an obvious forgetful functorU : HausTop from the category of Hausdorff topological spaces to the category of topological spaces which forgets the Hausdorff condition. Again, there is a functorF : TopHaus such thatFU. Given a topological space X,F(X) yields the best Hausdorff space constructed fromX: it is the quotient of X by the closure of the diagonalΔXX× X, which is an equivalence relation. In contrast with the previous example where we had an embedding, this time we get a quotient of the original structure.
  2. Consider now the category of compact Hausdorff spaceskHaus and the forgetful functor U : kHausTop, which forgets the compactness property and the separation property. The left adjoint to thisU is the Stone-Cech compactification.
  3. There is a forgetful functor U : ModRAbGrp from a category of R-modules to the category of abelian groups, where R is a commutative ring with unit. The functor U forgets the action of R on a group G. The functor U has both a left and a right adjoint. The left adjoint is R⊗ − : AbGrpModR which sends an abelian groupG to the tensor product RG and the right adjoint is given by the functorHom(R, −) : AbGrpModR which assigns to any groupG the modules of linear mappingsHom(R, G).
  4. The case where the categories C andD are posets deserves special attention here. Adjoint functors in this context are usually called Galois connections. Let C be a poset. Consider the diagonal functor Δ : CC× C, with Δ(X) = 〈X,X〉 and for f : XY, Δ(f) = 〈f,f〉 : 〈X,X〉 → 〈Y, Y〉. In this case, the left-adjoint to Δ is the coproduct, or the sup, and the right-adjoint to Δ is the product, or the inf. The adjoint situation can be described in the following special form:
    XYZ
    XZ, YZ
           
    ZXY
    ZY, ZX

    where the vertical double arrow can be interpreted as rules of inference going in both directions.

  5. Implication can also be introduced. Consider a functor with a parameter: (− ∧ X) : CC. It can easily be verified that when C is a poset, the function (− ∧ X) is order preserving and therefore a functor. A right adjoint to (− ∧ X) is a functor that yields the largest element of C such that its infimum with X is smaller than Z. This element is sometimes called the relative pseudocomplement of X or, more commonly, the implication. It is denoted by X ⇒ Z or byXZ. The adjunction can be presented as follows:
    YXZ
    YXZ
  6. The negation operator ¬X can be introduced from the last adjunction. Indeed, let Z be the bottom element ⊥ of the lattice. Then, since YX≤ ⊥ is always true, it follows that YX⇒ ⊥ is also always true. But since X ⇒ ⊥ ≤ X is always the case, we get at the numerator that X⇒ ⊥ ∧ X = ⊥. Hence, X⇒ ⊥ is the largest element disjoint from X. We can therefore put ¬X =defX⇒ ⊥.
  7. Limits, colimits, and all the fundamental constructions of category theory can be described as adjoints. Thus, products and coproducts are adjoints, as are equalizers, coequalizers, pullbacks and pushouts, etc. This is one of the reasons adjointness is central to category theory itself: because all the fundamental operations of category theory arise from adjoint situations.
  8. An equivalence of categories is a special case of adjointness. Indeed, if in the above triangular identities the arrows η : ICGF and ξ : FGID are naturalisomorphisms, then the functors F and G constitute an equivalence of categories. In practice, it is the notion of equivalence of categories that matters and not the notion of isomorphism of categories.

It is easy to prove certain facts about these operations directly from the adjunctions. Consider, for instance, implication. Let Z =X. Then we get at the numerator that YXX, which is always true in a poset (as is easily verified). Hence,YXX is also true for all Y and this is only possible if XX = ⊤, the top element of the lattice. Not only can logical operations be described as adjoints, but they naturally arise as adjoints to basic operations. In fact, adjoints can be used to define various structures, distributive lattices, Heyting algebras, Boolean algebras, etc. (See Wood, 2004.) It should be clear from the simple foregoing example how the formalism of adjointness can be used to give syntactic presentations of various logical theories. Furthermore, and this is a key element, the standard universal and existential quantifiers can be shown to be arising as adjoints to the operation of substitution. Thus, quantifiers are on a par with the other logical operations, in sharp contrast with the other algebraic approaches to logic. (See, for instance Awodey 1996 or Mac Lane & Moerdijk 1992.) More generally, Lawvere showed how syntax and semantics are related by adjoint functors. (See Lawvere 1969b.)

Dualities play an important role in mathematics and they can be described with the help of equivalences between categories. In other words, many important mathematical theorems can be translated as statements about the existence of adjoint functors, sometimes satisfying additional properties. This is sometimes taken as expressing the conceptual content of the theorem. Consider the following fundamental case: let C be the category whose objects are the locally compact abelian groups and the morphisms are the continuous group homomorphisms. Then, the Pontryagin duality theorem amounts to the claim that the category C is equivalent to the category C°, that is, to the opposite category. Of course, the precise statement requires that we describe the functors F : CC° and G : C° → C and prove that they constitute an equivalence of categories.

Another well known and important duality was discovered by Stone in the thirties and now bears his name. In one direction, an arbitrary Boolean algebra yields a topological space, and in the other direction, from a (compact Hausdorff and totally disconnected) topological space, one obtains a Boolean algebra. Moreover, this correspondence is functorial: any Boolean homomorphism is sent to a continuous map of topological spaces, and, conversely, any continuous map between the spaces is sent to a Boolean homomorphism. In other words, there is an equivalence of categories between the category of Boolean algebras and the dual of the category of Boolean spaces (also called Stone spaces). (See Johnstone 1982 for an excellent introduction and more developments.) The connection between a category of algebraic structures and the opposite of a category of topological structures established by Stone's theorem constitutes but one example of a general phenomenon that did attract and still attracts a great deal of attention from category theorists. Categorical study of duality theorems is still a very active and significant field, and is largely inspired by Stone's result. (For recent applications in logic, see, for instance Makkai 1987, Taylor 2000, 2002a, 2002b, Caramello 2011.)

2. Brief Historical Sketch

It is difficult to do justice to the short but intricate history of the field. In particular it is not possible to mention all those who have contributed to its rapid development. With this word of caution out of the way, we will look at some of the main historical threads.

Categories, functors, natural transformations, limits and colimits appeared almost out of nowhere in a paper by Eilenberg & Mac Lane (1945) entitled “General Theory of Natural Equivalences.” We say “almost,” because their earlier paper (1942) contains specific functors and natural transformations at work, limited to groups. A desire to clarify and abstract their 1942 results led Eilenberg & Mac Lane to devise category theory. The central notion at the time, as their title indicates, was that of natural transformation. In order to give a general definition of the latter, they defined functor, borrowing the term from Carnap, and in order to define functor, they borrowed the word ‘category’ from the philosophy of Aristotle, Kant, and C. S. Peirce, but redefining it mathematically.

After their 1945 paper, it was not clear that the concepts of category theory would amount to more than a convenient language; this indeed was the status quo for about fifteen years. Category theory was employed in this manner by Eilenberg & Steenrod (1952), in an influential book on the foundations of algebraic topology, and by Cartan & Eilenberg (1956), in a ground breaking book on homological algebra. (Curiously, although Eilenberg & Steenrod defined categories, Cartan & Eilenberg simply assumed them!) These books allowed new generations of mathematicians to learn algebraic topology and homological algebra directly in the categorical language, and to master the method of diagrams. Indeed, without the method of diagram chasing, many results in these two books seem inconceivable, or at the very least would have required a considerably more intricate presentation.

The situation changed radically with Grothendieck's (1957) landmark paper entitled “Sur quelques points d'algèbre homologique”, in which the author employed categories intrinsically to define and construct more general theories which he (Grothendieck 1957) then applied to specific fields, e.g., to algebraic geometry. Kan (1958) showed that adjoint functors subsume the important concepts of limits and colimits and could capture fundamental concepts in other areas (in his case, homotopy theory).

At this point, category theory became more than a convenient language, by virtue of two developments.

  1. Employing the axiomatic method and the language of categories, Grothendieck (1957) defined in an abstract fashion types of categories, e.g., additive and Abelian categories, showed how to perform various constructions in these categories, and proved various results about them. In a nutshell, Grothendieck showed how to develop part of homological algebra in an abstract setting of this sort. From then on, a specific category of structures, e.g., a category of sheaves over a topological space X, could be seen as a token of an abstract category of a certain type, e.g., an Abelian category. One could therefore immediately see how the methods of, e.g., homological algebra could be applied to, for instance, algebraic geometry. Furthermore, it made sense to look for other types of abstract categories, ones that would encapsulate the fundamental and formal aspects of various mathematical fields in the same way that Abelian categories encapsulated fundamental aspects of homological algebra.
  2. Thanks in large part to the efforts of Freyd and Lawvere, category theorists gradually came to see the pervasiveness of the concept of adjoint functors. Not only does the existence of adjoints to given functors permit definitions of abstract categories (and presumably those which are defined by such means have a privileged status) but as we mentioned earlier, many important theorems and even theories in various fields can be seen as equivalent to the existence of specific functors between particular categories. By the early 1970's, the concept of adjoint functors was seen as central to category theory.

With these developments, category theory became an autonomous field of research, and pure category theory could be developed. And indeed, it did grow rapidly as a discipline, but also in its applications, mainly in its source contexts, namely algebraic topology and homological algebra, but also in algebraic geometry and, after the appearance of Lawvere's Ph. D thesis, in universal algebra. This thesis also constitutes a landmark in this history of the field, for in it Lawvere proposed the category of categories as a foundation for category theory, set theory and, thus, the whole of mathematics, as well as using categories for the study of the logical aspects of mathematics.

Over the course of the 1960's, Lawvere outlined the basic framework for an entirely original approach to logic and the foundations of mathematics. He achieved the following:

  • Axiomatized the category of sets (Lawvere 1964) and of categories (Lawvere 1966);
  • Gave a categorical description of theories that was independent of syntactical choices and sketched how completeness theorems for logical systems could be obtained by categorical methods (Lawvere 1967);
  • Characterized Cartesian closed categories and showed their connections to logical systems and various logical paradoxes (Lawvere 1969);
  • Showed that the quantifiers and the comprehension schemes could be captured as adjoint functors to given elementary operations (Lawvere 1966, 1969, 1970, 1971);
  • Argued that adjoint functors should generally play a major foundational role through the notion of “categorical doctrines” (Lawvere 1969).

Meanwhile, Lambek (1968, 1969, 1972) described categories in terms of deductive systems and employed categorical methods for proof-theoretical purposes.

All this work culminated in another notion, thanks to Grothendieck and his school: that of a topos. Even though toposes appeared in the 1960's, in the context of algebraic geometry, again from the mind of Grothendieck, it was certainly Lawvere and Tierney's (1972) elementary axiomatization of a topos which gave impetus to its attaining foundational status. Very roughly, an elementary topos is a category possessing a logical structure sufficiently rich to develop most of “ordinary mathematics”, that is, most of what is taught to mathematics undergraduates. As such, an elementary topos can be thought of as a categorical theory of sets. But it is also a generalized topological space, thus providing a direct connection between logic and geometry. (For more on the history of categorical logic, see Marquis & Reyes 2012, Bell 2005.)

The 1970s saw the development and application of the topos concept in many different directions. The very first applications outside algebraic geometry were in set theory, where various independence results were recast in terms of topos (Tierney 1972, Bunge 1974, but also Blass & Scedrov 1989, Blass & Scedrov 1992, Freyd 1980, Mac Lane & Moerdijk 1992, Scedrov 1984). Connections with intuitionistic and, more generally constructive mathematics were noted early on, and toposes are still used to investigate models of various aspects of intuitionism and constructivism (Lambek & Scott 1986, Mac Lane & Moerdijk 1992, Van der Hoeven & Moerdijk 1984a, 1984b, 1984c, Moerdijk 1984, Moerdijk 1995a, Moerdijk 1998, Moerdijk & Palmgren 1997, Moerdijk & Palmgren 2002), Palmgren 2012. For more on the history of topos theory, see McLarty (1992).

More recently, topos theory has been employed to investigate various forms of constructive mathematics or set theory (Joyal & Moerdijk 1995, Taylor 1996, Awodey 2008), recursiveness, and models of higher-order type theories generally. The introduction of the so-called “effective topos” and the search for axioms for synthetic domain theory are worth mentioning (Hyland 1982, Hyland 1988, 1991, Hyland et al. 1990, McLarty 1992, Jacobs 1999, Van Oosten 2008, Van Oosten 2002 and the references therein). Lawvere's early motivation was to provide a new foundation for differential geometry, a lively research area which is now called “synthetic differential geometry” (Lawvere 2000, 2002, Kock 2006, Bell 1988, 1995, 1998, 2001, Moerdijk & Reyes 1991). This is only the tip of the iceberg; toposes could prove to be for the 21st century what Lie groups were to the 20th century.

From the 1980s to the present, category theory has found new applications. In theoretical computer science, category theory is now firmly rooted, and contributes, among other things, to the development of new logical systems and to the semantics of programming. (Pitts 2000, Plotkin 2000, Scott 2000, and the references therein). Its applications to mathematics are becoming more diverse, even touching on theoretical physics, which employs higher-dimensional category theory — which is to category theory what higher-dimensional geometry is to plane geometry — to study the so-called “quantum groups” and quantum field theory (Majid 1995, Baez & Dolan 2001 and other publications by these authors).

3. Philosophical Significance

Category theory challenges philosophers in two ways, which are not necessarily mutually exclusive. On the one hand, it is certainly the task of philosophy to clarify the general epistemological and ontological status of categories and categorical methods, both in the practice of mathematics and in the foundational landscape. On the other hand, philosophers and philosophical logicians can employ category theory and categorical logic to explore philosophical and logical problems. I now discuss these challenges, briefly, in turn.

Category theory is now a common tool in the mathematician's toolbox; that much is clear. It is also clear that category theory organizes and unifies much of mathematics. (See for instance Mac Lane 1971, 1998 or Pedicchio & Tholen 2004.) No one will deny these simple facts.

Doing mathematics in a categorical framework is almost always radically different from doing it in a set-theoretical framework (the exception being working with the internal language of a Boolean topos; whenever the topos is not Boolean, then the main difference lies in the fact that the logic is intuitionistic). Hence, as is often the case when a different conceptual framework is adopted, many basic issues regarding the nature of the objects studied, the nature of the knowledge involved, and the nature of the methods used have to be reevaluated. We will take up these three aspects in turn.

Two facets of the nature of mathematical objects within a categorical framework have to be emphasized. First, objects are always given in a category. An object exists in and depends upon an ambient category. Furthermore, an object is characterized by the morphisms going in it and/or the morphisms coming out of it. Second, objects are always characterized up to isomorphism (in the best cases, up to a unique isomorphism). There is no such thing, for instance, as the natural numbers. However, it can be argued that there is such a thing as the concept of natural numbers. Indeed, the concept of natural numbers can be given unambiguously, via the Dedekind-Peano-Lawvere axioms, but what this concept refers to in specific cases depends on the context in which it is interpreted, e.g., the category of sets or a topos of sheaves over a topological space. It is hard to resist the temptation to think that category theory embodies a form of structuralism, that it describes mathematical objects as structures since the latter, presumably, are always characterized up to isomorphism. Thus, the key here has to do with the kind of criterion of identity at work within a categorical framework and how it resembles any criterion given for objects which are thought of as forms in general. One of the standard objections presented against this view is that if objects are thought of as structures and only as abstract structures, meaning here that they are separated from any specific or concrete representation, then it is impossible to locate them within the mathematical universe. (See Hellman 2003 for a standard formulation of the objection, McLarty 1993, Awodey 2004, Landry & Marquis 2005, Shapiro 2005, Landry 2011, Linnebo & Pettigrew 2011, McLarty 2011 for relevant material on the issue.)

A slightly different way to make sense of the situation is to think of mathematical objects as types for which there are tokens given in different contexts. This is strikingly different from the situation one finds in set theory, in which mathematical objects are defined uniquely and their reference is given directly. Although one can make room for types within set theory via equivalence classes or isomorphism types in general, the basic criterion of identity within that framework is given by the axiom of extensionality and thus, ultimately, reference is made to specific sets. Furthermore, it can be argued that the relation between a type and its token isnot represented adequately by the membership relation. A token does not belong to a type, it is not an element of a type, but rather it is an instance of it. In a categorical framework, one always refers to a token of a type, and what the theory characterizes directly is the type, not the tokens. In this framework, one does not have to locate a type, but tokens of it are, at least in mathematics, epistemologically required. This is simply the reflection of the interaction between the abstract and the concrete in the epistemological sense (and not the ontological sense of these latter expressions.) (See Ellerman 1988, Marquis 2000, Marquis 2006, Marquis 2013.)

The history of category theory offers a rich source of information to explore and take into account for an historically sensitive epistemology of mathematics. It is hard to imagine, for instance, how algebraic geometry and algebraic topology could have become what they are now without categorical tools. (See, for instance, Carter 2008, Corfield 2003, Krömer 2007, Marquis 2009, McLarty 1994, McLarty 2006.) Category theory has lead to reconceptualizations of various areas of mathematics based on purely abstract foundations. Moreover, when developed in a categorical framework, traditional boundaries between disciplines are shattered and reconfigured; to mention but one important example, topos theory provides a direct bridge between algebraic geometry and logic, to the point where certain results in algebraic geometry are directly translated into logic and vice versa. Certain concepts that were geometrical in origin are more clearly seen as logical (for example, the notion of coherent topos). Algebraic topology also lurks in the background. On a different but important front, it can be argued that the distinction between mathematics and metamathematics cannot be articulated in the way it has been. All these issues have to be reconsidered and reevaluated.

Moving closer to mathematical practice, category theory allowed for the development of methods that have changed and continue to change the face of mathematics. It could be argued that category theory represents the culmination of one of deepest and most powerful tendencies in twentieth century mathematical thought: the search for the most general and abstract ingredients in a given situation. Category theory is, in this sense, the legitimate heir of the Dedekind-Hilbert-Noether-Bourbaki tradition, with its emphasis on the axiomatic method and algebraic structures. When used to characterize a specific mathematical domain, category theory reveals the frame upon which that area is built, the overall structure presiding to its stability, strength and coherence. The structure of this specific area, in a sense, might not need to rest on anything, that is, on some solid soil, for it might very well be just one part of a larger network that is without any Archimedean point, as if floating in space. To use a well-known metaphor: from a categorical point of view, Neurath's ship has become a spaceship.

Still, it remains to be seen whether category theory should be “on the same plane,” so to speak, as set theory, whether it should be taken as a serious alternative to set theory as a foundation for mathematics, or whether it is foundational in a different sense altogether. (That this very question applies even more forcefully to topos theory will not detain us.)

Lawvere from early on promoted the idea that a category of categories could be used as a foundational framework. (See Lawvere 1964, 1966.) This proposal now rests in part on the development of higher-dimensional categories, also called weak n-categories. (See, for instance Makkai 1998.) The advent of topos theory in the seventies brought new possibilities. Mac Lane has suggested that certain toposes be considered as a genuine foundation for mathematics. (See Mac Lane 1986.) Lambek proposed the so-called free topos as the best possible framework, in the sense that mathematicians with different philosophical outlooks might nonetheless agree to adopt it. (See Couture & Lambek 1991, 1992, Lambek 1994.) He has recently argued that there is no topos that can thoroughly satisfy a classical mathematician. (See Lambek 2004.) (For more on the various foundational views among category theorists, see Landry & Marquis 2005.)

Arguments have been advanced for and against category theory as a foundational framework. (Blass 1984 surveys the relationships between category theory and set theory. Feferman 1977, Bell 1981, and Hellman 2003 argue against category theory. See Marquis 1995 for a quick overview and proposal and McLarty 2004 and Awodey 2004 for replies to Hellman 2003.) This matter is further complicated by the fact that the foundations of category theory itself have yet to be clarified. For there may be many different ways to think of a universe of higher-dimensional categories as a foundations for mathematics. An adequate language for such a universe still has to be presented together with definite axioms for mathematics. (See Makkai 1998 for a short description of such a language. A different approach based on homotopy theory but with closed connections with higher-dimensional categories has been proposed by Voevodsky et al. and is being vigorously pursued. See the book Homotopy Type Theory, by Awodey et al. 2013.)

It is an established fact that category theory is employed to study logic and philosophy. Indeed, categorical logic, the study of logic by categorical means, has been under way for about 30 years now and still vigorous. Some of the philosophically relevant results obtained in categorical logic are:

  • The hierarchy of categorical doctrines: regular categories, coherent categories, Heyting categories and Boolean categories; all these correspond to well-defined logical systems, together with deductive systems and completeness theorems; they suggest that logical notions, including quantifiers, arise naturally in a specific order and are not haphazardly organized;
  • Joyal's generalization of Kripke-Beth semantics for intuitionistic logic to sheaf semantics (Lambek & Scott 1986, Mac Lane & Moerdijk 1992);
  • Coherent and geometric logic, so-called, whose practical and conceptual significance has yet to be explored (Makkai & Reyes 1977, Mac Lane & Moerdiejk 1992, Johnstone 2002, Caramello 2011b, 2012a);
  • The notions of generic model and classifying topos of a theory (Makkai & Reyes 1977, Boileau & Joyal 1981, Bell 1988, Mac Lane& Moerdijk 1992, Johnstone 2002, Caramello 2012b);
  • The notion of strong conceptual completeness and the associated theorems (Makkai & Reyes 1977, Butz & Moerdijk 1999, Makkai 1981, Pitts 1989, Johnstone 2002);
  • Geometric proofs of the independence of the continuum hypothesis and other strong axioms of set theory (Tierney 1972, Bunge 1974, Freyd 1980, 1987, Blass & Scedrov 1983, 1989, 1992, Mac Lane & Moerdijk 1992);
  • Models and development of constructive mathematics (see bibliography below);
  • Synthetic differential geometry, an alternative to standard and non-standard analysis (Kock 1981, Bell 1998, 2001, 2006);
  • The construction of the so-called effective topos, in which every function on the natural numbers is recursive (McLarty 1992, Hyland 1982, 1991, Van Oosten 2002, Van Oosten 2008);
  • Categorical models of linear logic, modal logic, fuzzy sets, and general higher-order type theories (Reyes 1991, Reyes & Zawadoski 1993, Reyes & Zolfaghari 1991, 1996, Makkai & Reyes 1995, Ghilardi & Zawadowski 2002, Rodabaugh & Klement 2003, Jacobs 1999, Taylor 1999, Johnstone 2002, Blute & Scott 2004, Awodey& Warren 2009, Awodey et. al. 2013);
  • A graphical syntax called “sketches” (Barr & Wells 1985, 1999, Makkai 1997a, 1997b, 1997c, Johnstone 2002).
  • Quantum logic, the foundations of quantum physics and quantum field theory (Abramsky & Duncan 2006, Heunen et. al. 2009, Baez& Stay 2010, Baez & Lauda 2011, Coecke 2011, Isham 2011, Döring 2011).

Categorical tools in logic offer considerable flexibility, as is illustrated by the fact that almost all the surprising results of constructive and intuitionistic mathematics can be modeled in a proper categorical setting. At the same time, the standard set-theoretic notions, e.g. Tarski's semantics, have found natural generalizations in categories. Thus, categorical logic has roots in logic as it was developed in the twentieth century, while at the same time providing a powerful and novel framework with numerous links to other parts of mathematics.

Category theory also bears on more general philosophical questions. From the foregoing disussion, it should be obvious that category theory and categorical logic ought to have an impact on almost all issues arising in philosophy of logic: from the nature of identity criteria to the question of alternative logics, category theory always sheds a new light on these topics. Similar remarks can be made when we turn to ontology, in particular formal ontology: the part/whole relation, boundaries of systems, ideas of space, etc. Ellerman (1988) has bravely attempted to show that category theory constitutes a theory of universals, one having properties radically different from set theory, which is also seen as a theory of universals. Moving from ontology to cognitive science, MacNamara & Reyes (1994) have tried to employ categorical logic to provide a different logic of reference. In particular, they have attempted to clarify the relationships between count nouns and mass terms. Other researchers are using category theory to study complex systems, cognitive neural networks, and analogies. (See, for instance, Ehresmann & Vanbremeersch 1987, 2007, Healy 2000, Healy & Caudell 2006, Arzi-Gonczarowski 1999, Brown & Porter 2006.) Finally, philosophers of science have turned to category theory to shed a new light on issues related to structuralism in science. (See, for instance, Brading & Landry 2006, Bain 2013, Lam & Wüthrich forthcoming.)

Category theory offers thus many philosophical challenges, challenges which will hopefully be taken up in years to come.

Bibliography

Readers may find the following useful:

Programmatic Reading Guide

The citations in this guide and in the text above can all be found in the list below.

  • Abramsky, S. & Duncan, R., 2006, “A Categorical Quantum Logic”, Mathematical Structures in Computer Science, 16 (3): 469–489.
  • Adamek, J. et al., 1990, Abstract and Concrete Categories: The Joy of Cats, New York: Wiley.
  • Adamek, J. et al., 1994, Locally Presentable and Accessible Categories, Cambridge: Cambridge University Press.
  • Arzi-Gonczaworski, Z., 1999, “Perceive This as That — Analogies, Artificial Perception, and Category Theory”, Annals of Mathematics and Artificial Intelligence, 26 (1): 215–252.
  • Awodey, S., 1996, “Structure in Mathematics and Logic: A Categorical Perspective”, Philosophia Mathematica, 3: 209–237.
  • –––, 2004, “An Answer to Hellman's Question: Does Category Theory Provide a Framework for Mathematical Structuralism”, Philosophia Mathematica, 12: 54–64.
  • –––, 2006, Category Theory, Oxford: Clarendon Press.
  • –––, 2007, “Relating First-Order Set Theories and Elementary Toposes”, The Bulletin of Symbolic, 13 (3): 340–358.
  • –––, 2008, “A Brief Introduction to Algebraic Set Theory”, The Bulletin of Symbolic, 14 (3): 281–298.
  • Awodey, S., et al., 2013, Homotopy Type Theory: Univalent Foundations of Mathematics, The Univalent Foundations Program.
  • Awodey, S. & Butz, C., 2000, “Topological Completeness for Higher Order Logic”, Journal of Symbolic Logic, 65 (3): 1168–1182.
  • Awodey, S. & Reck, E. R., 2002, “Completeness and Categoricity I. Nineteen-Century Axiomatics to Twentieth-Century Metalogic”, History and Philosophy of Logic, 23 (1): 1–30.
  • –––, 2002, “Completeness and Categoricity II. Twentieth-Century Metalogic to Twenty-first-Century Semantics”, History and Philosophy of Logic, 23 (2): 77–94.
  • Awodey, S. & Warren, M., 2009, “Homotopy theoretic Models of Identity Types”, Mathematical Proceedings of the Cambridge Philosophical Society, 146 (1): 45–55.
  • Baez, J., 1997, “An Introduction to n-Categories”, Category Theory and Computer Science, Lecture Notes in Computer Science (Volume 1290), Berlin: Springer-Verlag, 1–33.
  • Baez, J. & Dolan, J., 1998a, “Higher-Dimensional Algebra III. n-Categories and the Algebra of Opetopes”, Advances in Mathematics, 135: 145–206.
  • –––, 1998b, “Categorification”, Higher Category Theory (Contemporary Mathematics, Volume 230), Ezra Getzler and Mikhail Kapranov (eds.), Providence: AMS, 1–36.
  • –––, 2001, “From Finite Sets to Feynman Diagrams”, Mathematics Unlimited – 2001 and Beyond, Berlin: Springer, 29–50.
  • Baez, J. & Lauda, A.D., 2011, “A Pre-history of n-Categorical Physics”, Deep Beauty: Understanding the Quantum World Through Mathematical Innovation, H. Halvorson, ed., Cambridge: Cambridge University Press, 13–128.
  • Baez, J. & May, P. J., 2010, Towards Higher Category Theory, Berlin: Springer.
  • Baez, J. & Stay, M., 2010, “Physics, Topology, Logic and Computation: a Rosetta Stone”, New Structures for Physics (Lecture Notes in Physics 813), B. Coecke (ed.), New York, Springer: 95–172.
  • Baianu, I. C., 1987, “Computer Models and Automata Theory in Biology and Medecine”, in Witten, Matthew, Eds. Mathematical Modelling, Vol. 7, 1986, chapter 11, Pergamon Press, Ltd., 1513–1577.
  • Bain, J., 2013, “Category-theoretic Structure and Radical Ontic Structural Realism”, Synthese, 190: 1621–1635.
  • Barr, M. & Wells, C., 1985, Toposes, Triples and Theories, New York: Springer-Verlag.
  • –––, 1999, Category Theory for Computing Science, Montreal: CRM.
  • Batanin, M., 1998, “Monoidal Globular Categories as a Natural Environment for the Theory of Weak n-Categories”, Advances in Mathematics, 136: 39–103.
  • Bell, J. L., 1981, “Category Theory and the Foundations of Mathematics”, British Journal for the Philosophy of Science, 32: 349–358.
  • –––, 1982, “Categories, Toposes and Sets”, Synthese, 51 (3): 293–337.
  • –––, 1986, “From Absolute to Local Mathematics”, Synthese, 69 (3): 409–426.
  • –––, 1988, “Infinitesimals”, Synthese, 75 (3): 285–315.
  • –––, 1988, Toposes and Local Set Theories: An Introduction, Oxford: Oxford University Press.
  • –––, 1995, “Infinitesimals and the Continuum”, Mathematical Intelligencer, 17 (2): 55–57.
  • –––, 1998, A Primer of Infinitesimal Analysis, Cambridge: Cambridge University Press.
  • –––, 2001, “The Continuum in Smooth Infinitesimal Analysis”, Reuniting the Antipodes — Constructive and Nonstandard Views on the Continuum (Synthese Library, Volume 306), Dordrecht: Kluwer, 19–24.
  • –––, 2005, “The Development of Categorical Logic”, in Handbook of Philosophical Logic (Volume 12), 2nd ed., D.M. Gabbay, F. Guenthner (eds.), Dordrecht: Springer, pp. 279–362.
  • Birkoff, G. & Mac Lane, S., 1999, Algebra, 3rd ed., Providence: AMS.
  • Blass, A., 1984, “The Interaction Between Category Theory and Set Theory”, in Mathematical Applications of Category Theory (Volume 30), Providence: AMS, 5–29.
  • Blass, A. & Scedrov, A., 1983, “Classifying Topoi and Finite Forcing”, Journal of Pure and Applied Algebra, 28: 111–140.
  • –––, 1989, Freyd's Model for the Independence of the Axiom of Choice, Providence: AMS.
  • –––, 1992, “Complete Topoi Representing Models of Set Theory”, Annals of Pure and Applied Logic , 57 (1): 1–26.
  • Blute, R. & Scott, P., 2004, “Category Theory for Linear Logicians”, in Linear Logic in Computer Science, T. Ehrhard, P. Ruet, J-Y. Girard, P. Scott, eds., Cambridge: Cambridge University Press, 1–52.
  • Boileau, A. & Joyal, A., 1981, “La logique des topos”, Journal of Symbolic Logic, 46 (1): 6–16.
  • Borceux, F., 1994, Handbook of Categorical Algebra, 3 volumes, Cambridge: Cambridge University Press.
  • Brading, K. & Landry, E., 2006, “Scientific Structuralism: Presentation and Representation”, Philosophy of Science, 73: 571–581.
  • Brown, R. & Porter, T., 2006, “Category Theory: an abstract setting for analogy and comparison”, What is Category Theory?, G. Sica, ed., Monza: Polimetrica: 257–274.
  • Bunge, M., 1974, “Topos Theory and Souslin's Hypothesis”, Journal of Pure and Applied Algebra, 4: 159–187.
  • –––, 1984, “Toposes in Logic and Logic in Toposes”, Topoi, 3 (1): 13–22.
  • Caramello, O., 2011, “A Characterization Theorem for Geometric Logic”, Annals of Pure and Applied Logic,162, 4: 318–321.
  • –––, 2012a, “Universal Models and Definability”, Mathematical Proceedings of the Cambridge Philosophical Society, 152 (2): 279–302.
  • –––, 2012b, “Syntactic Characterizations of Properties of Classifying Toposes”, Theory and Applications of Categories, 26 (6): 176–193.
  • Carter, J., 2008, “Categories for the working mathematician: making the impossible possible”, Synthese, 162 (1): 1–13.
  • Cheng, E. & Lauda, A., 2004, Higher-Dimensional Categories: an illustrated guide book, available at: http://cheng.staff.shef.ac.uk/guidebook/index.html
  • Cockett, J. R. B. & Seely, R. A. G., 2001, “Finite Sum-product Logic”, Theory and Applications of Categories (electronic), 8: 63–99.
  • Coecke, B., 2011, “A Universe of Processes and Some of its Guises”, Deep Beauty: Understanding the Quantum World through Mathematical Innovation, Cambridge: Cambridge University Press: 129–186.
  • Couture, J. & Lambek, J., 1991, “Philosophical Reflections on the Foundations of Mathematics”, Erkenntnis, 34 (2): 187–209.
  • –––, 1992, “Erratum:”Philosophical Reflections on the Foundations of Mathematics“”, Erkenntnis, 36 (1): 134.
  • Crole, R. L., 1994, Categories for Types, Cambridge: Cambridge University Press.
  • Dieudonné, J. & Grothendieck, A., 1960 [1971], Éléments de Géométrie Algébrique, Berlin: Springer-Verlag.
  • Döring, A., 2011, “The Physical Interpretation of Daseinisation”, Deep Beauty: Understanding the Quantum World through Mathematical Innovation, Cambridge: Cambridge University Press: 207-238.
  • Ehresmann, A. & Vanbremeersch, J.-P., 2007, Memory Evolutive Systems: Hierarchy, Emergence, Cognition, Amsterdam: Elsevier
  • –––, 1987, “Hierarchical Evolutive Systems: a Mathematical Model for Complex Systems”, Bulletin of Mathematical Biology, 49 (1): 13–50.
  • Eilenberg, S. & Cartan, H., 1956, Homological Algebra, Princeton: Princeton University Press.
  • Eilenberg, S. & Mac Lane, S., 1942, “Group Extensions and Homology”, Annals of Mathematics, 43: 757–831.
  • –––, 1945, “General Theory of Natural Equivalences”, Transactions of the American Mathematical Society, 58: 231–294.
  • Eilenberg, S. & Steenrod, N., 1952, Foundations of Algebraic Topology, Princeton: Princeton University Press.
  • Ellerman, D., 1988, “Category Theory and Concrete Universals”, Erkenntnis, 28: 409–429.
  • Feferman, S., 1977, “Categorical Foundations and Foundations of Category Theory”, Logic, Foundations of Mathematics and Computability, R. Butts (ed.), Reidel, 149–169.
  • –––, 2004, “Typical Ambiguity: trying to have your cake and eat it too”, One Hundred Years of Russell's Paradox, G. Link (ed.), Berlin: De Gruyter, 135–151.
  • Freyd, P., 1964, Abelian Categories. An Introduction to the Theory of Functors, New York: Harper & Row.
  • –––, 1965, “The Theories of Functors and Models”. Theories of Models, Amsterdam: North Holland, 107–120.
  • –––, 1972, “Aspects of Topoi”, Bulletin of the Australian Mathematical Society, 7: 1–76.
  • –––, 1980, “The Axiom of Choice”, Journal of Pure and Applied Algebra, 19: 103–125.
  • –––, 1987, “Choice and Well-Ordering”, Annals of Pure and Applied Logic, 35 (2): 149–166.
  • –––, 1990, Categories, Allegories, Amsterdam: North Holland.
  • –––, 2002, “Cartesian Logic”, Theoretical Computer Science, 278 (1–2): 3–21.
  • Freyd, P., Friedman, H. & Scedrov, A., 1987, “Lindembaum Algebras of Intuitionistic Theories and Free Categories”, Annals of Pure and Applied Logic, 35 (2): 167–172.
  • Galli, A. & Reyes, G. & Sagastume, M., 2000, “Completeness Theorems via the Double Dual Functor”, Studia Logical, 64 (1): 61–81.
  • Ghilardi, S., 1989, “Presheaf Semantics and Independence Results for some Non-classical first-order logics”, Archive for Mathematical Logic, 29 (2): 125–136.
  • Ghilardi, S. & Zawadowski, M., 2002, Sheaves, Games & Model Completions: A Categorical Approach to Nonclassical Porpositional Logics, Dordrecht: Kluwer.
  • Goldblatt, R., 1979, Topoi: The Categorical Analysis of Logic, Studies in logic and the foundations of mathematics, Amsterdam: Elsevier.
  • Grothendieck, A., 1957, “Sur Quelques Points d'algèbre homologique”, Tohoku Mathematics Journal, 9: 119–221.
  • Grothendieck, A. et al., Séminaire de Géométrie Algébrique, Vol. 1–7, Berlin: Springer-Verlag.
  • Hatcher, W. S., 1982, The Logical Foundations of Mathematics, Oxford: Pergamon Press.
  • Healy, M. J., 2000, “Category Theory Applied to Neural Modeling and Graphical Representations”, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks: IJCNN200, Como, vol. 3, M. Gori, S-I. Amari, C. L. Giles, V. Piuri, eds., IEEE Computer Science Press, 35–40.
  • Healy, M. J., & Caudell, T. P., 2006, “Ontologies and Worlds in Category Theory: Implications for Neural Systems”,Axiomathes, 16 (1–2): 165–214.
  • Hellman, G., 2003, “Does Category Theory Provide a Framework for Mathematical Structuralism?”, Philosophia Mathematica, 11 (2): 129–157.
  • –––, 2006, “Mathematical Pluralism: the case of smooth infinitesimal analysis”, Journal of Philosophical Logic, 35 (6): 621–651.
  • Hermida, C. & Makkai, M. & Power, J., 2000, “On Weak Higher-dimensional Categories I”, Journal of Pure and Applied Algebra, 154 (1–3): 221–246.
  • –––, 2001, “On Weak Higher-dimensional Categories 2”, Journal of Pure and Applied Algebra, 157 (2–3): 247–277.
  • –––, 2002, “On Weak Higher-dimensional Categories 3”, Journal of Pure and Applied Algebra, 166 (1–2): 83–104.
  • Heunen, C. & Landsmann, N. & Spitters, B., 2009, “A Topos for Algebraic Quantum Theory”, Communications in Mathematical Physics, 291 (1): 63–110.
  • Hyland, J. M. E., 1982, “The Effective Topos”, Studies in Logic and the Foundations of Mathematics (Volume 110), Amsterdam: North Holland, 165–216.
  • –––, 1988, “A Small Complete Category”, Annals of Pure and Applied Logic, 40 (2): 135–165.
  • –––, 1991, “First Steps in Synthetic Domain Theory”, Category Theory (Como 1990) (Lecture Notes in Mathematics, Volume 1488), Berlin: Springer, 131–156.
  • –––, 2002, “Proof Theory in the Abstract”, Annals of Pure and Applied Logic, 114 (1–3): 43–78.
  • Hyland, J. M. E. & Robinson, E. P. & Rosolini, G., 1990, “The Discrete Objects in the Effective Topos”, Proceedings of the London Mathematical Society (3), 60 (1): 1–36.
  • Isham, C.J., 2011, “Topos Methods in the Foundations of Physics”, Deep Beauty: Understanding the Quantum World through Mathematical Innovation, Cambridge: Cambridge University Press: 187–206.
  • Jacobs, B., 1999, Categorical Logic and Type Theory, Amsterdam: North Holland.
  • Johnstone, P. T., 1977, Topos Theory, New York: Academic Press.
  • –––, 1979a, “Conditions Related to De Morgan's Law”, Applications of Sheaves (Lecture Notes in Mathematics, Volume 753), Berlin: Springer, 479–491.
  • –––, 1979b, “Another Condition Equivalent to De Morgan's Law”, Communications in Algebra, 7 (12): 1309–1312.
  • –––, 1981, “Tychonoff's Theorem without the Axiom of Choice”, Fundamenta Mathematicae, 113 (1): 21–35.
  • –––, 1982, Stone Spaces, Cambridge:Cambridge University Press.
  • –––, 1985, “How General is a Generalized Space?”, Aspects of Topology, Cambridge: Cambridge University Press, 77–111.
  • –––, 2001, “Elements of the History of Locale Theory”, Handbook of the History of General Topology, Vol. 3, Dordrecht: Kluwer, 835–851.
  • –––, 2002a, Sketches of an Elephant: a Topos Theory Compendium. Vol. 1 (Oxford Logic Guides, Volume 43), Oxford: Oxford University Press.
  • Joyal, A. & Moerdijk, I., 1995, Algebraic Set Theory, Cambridge: Cambridge University Press.
  • Kan, D. M., 1958, “Adjoint Functors”, Transactions of the American Mathematical Society, 87: 294–329.
  • Kock, A., 2006, Synthetic Differential Geometry (London Mathematical Society Lecture Note Series, Volume 51), Cambridge: Cambridge University Press, 2nd ed.
  • Krömer, R., 2007, Tool and Objects: A History and Philosophy of Category Theory, Basel: Birkhäuser.
  • La Palme Reyes, M., John Macnamara, Gonzalo E. Reyes, and Houman Zolfaghari, 1994, “The non-Boolean Logic of Natural Language Negation”, Philosophia Mathematica, 2 (1): 45–68.
  • –––, 1999, “Count Nouns, Mass Nouns, and their Transformations: a Unified Category-theoretic Semantics”, Language, Logic and Concepts, Cambridge: MIT Press, 427–452.
  • Lambek, J., 1968, “Deductive Systems and Categories I. Syntactic Calculus and Residuated Categories”, Mathematical Systems Theory, 2: 287–318.
  • –––, 1969, “Deductive Systems and Categories II. Standard Constructions and Closed Categories”, Category Theory, Homology Theory and their Applications I, Berlin: Springer, 76–122.
  • –––, 1972, “Deductive Systems and Categories III. Cartesian Closed Categories, Intuition≠istic Propositional Calculus, and Combinatory Logic”, Toposes, Algebraic Geometry and Logic (Lecture Notes in Mathematics, Volume 274), Berlin: Springer, 57–82.
  • –––, 1982, “The Influence of Heraclitus on Modern Mathematics”, Scientific Philosophy Today, J. Agassi and R.S. Cohen, eds., Dordrecht, Reidel, 111–122.
  • –––, 1986, “Cartesian Closed Categories and Typed lambda calculi”, Combinators and Functional Programming Languages (Lecture Notes in Computer Science, Volume 242), Berlin: Springer, 136–175.
  • –––, 1989a, “On Some Connections Between Logic and Category Theory”, Studia Logica, 48 (3): 269–278.
  • –––, 1989b, “On the Sheaf of Possible Worlds”, Categorical Topology and its relation to Analysis, Algebra and Combinatorics, Teaneck: World Scientific Publishing, 36–53.
  • –––, 1993, “Logic without Structural Rules”, Substructural Logics (Studies in Logic and Computation, Volume 2), Oxford: Oxford University Press, 179–206.
  • –––, 1994a, “Some Aspects of Categorical Logic”, Logic, Methodology and Philosophy of Science IX (Studies in Logic and the Foundations of Mathematics, Volume 134), Amsterdam: North Holland, 69–89.
  • –––, 1994b, “Are the Traditional Philosophies of Mathematics Really Incompatible?”, Mathematical Intelligencer, 16 (1): 56–62.
  • –––, 1994c, “What is a Deductive System?”, What is a Logical System? (Studies in Logic and Computation, Volume 4), Oxford: Oxford University Press, 141–159.
  • –––, 2004, “What is the world of Mathematics? Provinces of Logic Determined”, Annals of Pure and Applied Logic, 126: 1–3, 149–158.
  • Lambek, J. & Scott, P.J., 1981, “Intuitionistic Type Theory and Foundations”, Journal of Philosophical Logic, 10 (1): 101–115.
  • –––, 1983, “New Proofs of Some Intuitionistic Principles”, Zeitschrift für Mathematische Logik und Grundlagen der Mathematik, 29 (6): 493–504.
  • –––, 1986, Introduction to Higher Order Categorical Logic, Cambridge: Cambridge University Press.
  • Lam, V. & Wütrich, C., forthcoming, “No Categorical Support for Radical Ontic Structural Realism”, British Journal for the Philosophy of Science.
  • Landry, E., 1999, “Category Theory: the Language of Mathematics”, Philosophy of Science, 66 (3) (Supplement): S14–S27.
  • –––, 2001, “Logicism, Structuralism and Objectivity”, Topoi, 20 (1): 79–95.
  • –––, 2007, “Shared Structure need not be Shared Set-structure”, Synthese, 158 (1): 1–17.
  • –––, 2011, “How to be a Structuralist all the way down”, Synthese, 179: 435–454.
  • Landry, E. & Marquis, J.-P., 2005, “Categories in Context: Historical, Foundational and philosophical”, Philosophia Mathematica, 13: 1–43.
  • Lawvere, F. W., 1963, “Functorial Semantics of Algebraic Theories”, Proceedings of the National Academy of Sciences U.S.A., 50: 869–872.
  • –––, 1964, “An Elementary Theory of the Category of Sets”, Proceedings of the National Academy of Sciences U.S.A., 52: 1506–1511.
  • –––, 1965, “Algebraic Theories, Algebraic Categories, and Algebraic Functors”, Theory of Models, Amsterdam: North Holland, 413–418.
  • –––, 1966, “The Category of Categories as a Foundation for Mathematics”, Proceedings of the Conference on Categorical Algebra, La Jolla, New York: Springer-Verlag, 1–21.
  • –––, 1969a, “Diagonal Arguments and Cartesian Closed Categories”, Category Theory, Homology Theory, and their Applications II, Berlin: Springer, 134–145.
  • –––, 1969b, “Adjointness in Foundations”, Dialectica, 23: 281–295.
  • –––, 1970, “Equality in Hyper doctrines and Comprehension Schema as an Adjoint Functor”, Applications of Categorical Algebra, Providence: AMS, 1–14.
  • –––, 1971, “Quantifiers and Sheaves”, Actes du Congrès International des Mathématiciens, Tome 1, Paris: Gauthier-Villars, 329–334.
  • –––, 1972, “Introduction”, Toposes, Algebraic Geometry and Logic, Lecture Notes in Mathematics, 274, Springer-Verlag, 1–12.
  • –––, 1975, “Continuously Variable Sets: Algebraic Geometry = Geometric Logic”, Proceedings of the Logic Colloquium Bristol 1973, Amsterdam: North Holland, 135–153.
  • –––, 1976, “Variable Quantities and Variable Structures in Topoi”, Algebra, Topology, and Category Theory, New York: Academic Press, 101–131.
  • –––, 1992, “Categories of Space and of Quantity”, The Space of Mathematics, Foundations of Communication and Cognition, Berlin: De Gruyter, 14–30.
  • –––, 1994a, “Cohesive Toposes and Cantor's lauter Ensein ”, Philosophia Mathematica, 2 (1): 5–15.
  • –––, 1994b, “Tools for the Advancement of Objective Logic: Closed Categories and Toposes”, The Logical Foundations of Cognition (Vancouver Studies in Cognitive Science, Volume 4), Oxford: Oxford University Press, 43–56.
  • –––, 2000, “Comments on the Development of Topos Theory”, Development of Mathematics 1950–2000, Basel: Birkhäuser, 715–734.
  • –––, 2002, “Categorical Algebra for Continuum Micro Physics”, Journal of Pure and Applied Algebra, 175 (1–3): 267–287.
  • –––, 2003, “Foundations and Applications: Axiomatization and Education. New Programs and Open Problems in the Foundation of Mathematics”, Bulletin of Symbolic Logic, 9 (2): 213–224.
  • Lawvere, F. W. & Rosebrugh, R., 2003, Sets for Mathematics, Cambridge: Cambridge University Press.
  • Lawvere, F. W. & Schanuel, S., 1997, Conceptual Mathematics: A First Introduction to Categories, Cambridge: Cambridge University Press.
  • Leinster, T., 2002, “A Survey of Definitions of n-categories”, Theory and Applications of Categories, (electronic), 10: 1–70.
  • –––, 2004, Higher Operads, Higher Categories, London Mathematical Society Lecture Note Series, 298, Cambridge: Cambridge University Press.
  • Linnebo, O. & Pettigrew, R., 2011, “Category Theory as an Autonomous Foundation”, Philosophia Mathematica, 19 (3): 227–254.
  • Lurie, J., 2009, Higher Topos Theory, Princeton: Princeton University Press.
  • Mac Lane, S., 1950, “Dualities for Groups”, Bulletin of the American Mathematical Society, 56: 485–516.
  • –––, 1969, “Foundations for Categories and Sets”, Category Theory, Homology Theory and their Applications II, Berlin: Springer, 146–164.
  • –––, 1969, “One Universe as a Foundation for Category Theory”, Reports of the Midwest Category Seminar III, Berlin: Springer, 192–200.
  • –––, 1971, “Categorical algebra and Set-Theoretic Foundations”, Axiomatic Set Theory, Providence: AMS, 231–240.
  • –––, 1975, “Sets, Topoi, and Internal Logic in Categories”, Studies in Logic and the Foundations of Mathematics (Volumes 80), Amsterdam: North Holland, 119–134.
  • –––, 1981, “Mathematical Models: a Sketch for the Philosophy of Mathematics”, American Mathematical Monthly, 88 (7): 462–472.
  • –––, 1986, Mathematics, Form and Function, New York: Springer.
  • –––, 1988, “Concepts and Categories in Perspective”, A nCentury of Mathematics in America, Part I, Providence: AMS, 323–365.
  • –––, 1989, “The Development of Mathematical Ideas by Collision: the Case of Categories and Topos Theory”, Categorical Topology and its Relation to Analysis, Algebra and Combinatorics, Teaneck: World Scientific, 1–9.
  • –––, 1996, “Structure in Mathematics. Mathematical Structuralism”, Philosophia Mathematica, 4 (2): 174–183.
  • –––, 1997, “Categorical Foundations of the Protean Character of Mathematics”, Philosophy of Mathematics Today, Dordrecht: Kluwer, 117–122.
  • –––, 1998, Categories for the Working Mathematician, 2nd edition, New York: Springer-Verlag.
  • Mac Lane, S. & Moerdijk, I., 1992, Sheaves in Geometry and Logic, New York: Springer-Verlag.
  • MacNamara, J. & Reyes, G., (eds.), 1994, The Logical Foundation of Cognition, Oxford: Oxford University Press.
  • Majid, S., 1995, Foundations of Quantum Group Theory, Cambridge: Cambridge University Press.
  • Makkai, M., 1987, “Stone Duality for First-Order Logic”, Advances in Mathematics, 65 (2): 97–170.
  • –––, 1988, “Strong Conceptual Completeness for First Order Logic”, Annals of Pure and Applied Logic, 40: 167–215.
  • –––, 1997a, “Generalized Sketches as a Framework for Completeness Theorems I”, Journal of Pure and Applied Algebra, 115 (1): 49–79.
  • –––, 1997b, “Generalized Sketches as a Framework for Completeness Theorems II”, Journal of Pure and Applied Algebra, 115 (2): 179–212.
  • –––, 1997c, “Generalized Sketches as a Framework for Completeness Theorems III”, Journal of Pure and Applied Algebra, 115 (3): 241–274.
  • –––, 1998, “Towards a Categorical Foundation of Mathematics”, Lecture Notes in Logic (Volume 11), Berlin: Springer, 153–190.
  • –––, 1999, “On Structuralism in Mathematics”, Language, Logic and Concepts, Cambridge: MIT Press, 43–66.
  • Makkai, M. & Paré, R., 1989, Accessible Categories: the Foundations of Categorical Model Theory, Contemporary Mathematics 104, Providence: AMS.
  • Makkai, M. & Reyes, G., 1977, First-Order Categorical Logic, Springer Lecture Notes in Mathematics 611, New York: Springer.
  • –––, 1995, “Completeness Results for Intuitionistic and Modal Logic in a Categorical Setting”, Annals of Pure and Applied Logic, 72 (1): 25–101.
  • Marquis, J.-P., 1993, “Russell's Logicism and Categorical Logicisms”, Russell and Analytic Philosophy, A. D. Irvine & G. A. Wedekind, (eds.), Toronto, University of Toronto Press, 293–324.
  • –––, 1995, “Category Theory and the Foundations of Mathematics: Philosophical Excavations”, Synthese, 103: 421–447.
  • –––, 2000, “Three Kinds of Universals in Mathematics?”, in Logical Consequence: Rival Approaches and New Studies in Exact Philosophy: Logic, Mathematics and Science, Vol. II, B. Brown & J. Woods (eds.), Oxford: Hermes, 191–212.
  • –––, 2006, “Categories, Sets and the Nature of Mathematical Entities”, in The Age of Alternative Logics. Assessing philosophy of logic and mathematics today, J. van Benthem, G. Heinzmann, Ph. Nabonnand, M. Rebuschi, H. Visser (eds.), Springer, 181–192.
  • –––, 2009, From a Geometrical Point of View: A Study in the History and Philosophy of Category Theory, Berlin: Springer.
  • –––, 2013, “Mathematical Forms and Forms of Mathematics: leaving the shores of extensional mathematics”, Synthese, 190 (12): 2141–2164.
  • Marquis, J.-P. & Reyes, G., 2012, “The History of Categorical Logic: 1963–1977”, Handbook of the History of Logic, Vol. 6, D. Gabbay & J. Woods, eds., Amsterdam: Elsevier, 689-800.
  • McLarty, C., 1986, “Left Exact Logic”, Journal of Pure and Applied Algebra, 41 (1): 63–66.
  • –––, 1990, “Uses and Abuses of the History of Topos Theory”, British Journal for the Philosophy of Science, 41: 351–375.
  • –––, 1991, “Axiomatizing a Category of Categories”, Journal of Symbolic Logic, 56 (4): 1243–1260.
  • –––, 1992, Elementary Categories, Elementary Toposes, Oxford: Oxford University Press.
  • –––, 1993, “Numbers Can be Just What They Have to”, Noûs, 27: 487–498.
  • –––, 1994, “Category Theory in Real Time”, Philosophia Mathematica, 2 (1): 36–44.
  • –––, 2004, “Exploring Categorical Structuralism”, Philosophia Mathematica, 12: 37–53.
  • –––, 2005, “Learning from Questions on Categorical Foundations”, Philosophia Mathematica, 13 (1): 44–60.
  • –––, 2006, “Emmy Noether's set-theoretic topology: from Dedekind to the rise of functors”, The Architecture of Modern Mathematics, J.J. Gray & J. Ferreiros, Oxford: Oxford University Press, 187–208.
  • –––, 2011, “Recent Debate over Categorical Foundations”, in Foundational Theories of Classical and Constructive Mathematics, G. Sommaruga, ed., New York: Springer: 145–154.
  • Moerdijk, I., 1984, “Heine-Borel does not imply the Fan Theorem”, Journal of Symbolic Logic, 49 (2): 514–519.
  • –––, 1995a, “A Model for Intuitionistic Non-standard Arithmetic”, Annals of Pure and Applied Logic, 73 (1): 37–51.
  • –––, 1998, “Sets, Topoi and Intuitionism”, Philosophia Mathematica, 6 (2): 169–177.
  • Moerdijk, I. & Palmgren, E., 1997, “Minimal Models of Heyting Arithmetic”, Journal of Symbolic Logic, 62 (4): 1448–1460.
  • –––, 2002, “Type Theories, Toposes and Constructive Set Theory: Predicative Aspects of AST”, Annals of Pure and Applied Logic, 114 (1–3): 155–201.
  • Moerdijk, I. & Reyes, G., 1991, Models for Smooth Infinitesimal Analysis, New York: Springer-Verlag.
  • Palmgren, E., 2012, “Constructivist and Structuralist Foundations: Bishop's and Lawvere's Theories of Sets”, Annals of Pure and Applied Logic, 63: 1384–1399.
  • Pareigis, B., 1970, Categories and Functors, New York: Academic Press.
  • Pedicchio, M. C. & Tholen, W., 2004, Categorical Foundations, Cambridge: Cambridge University Press.
  • Peirce, B., 1991, Basic Category Theory for Computer Scientists, Cambridge: MIT Press.
  • Pitts, A. M., 1987, “Interpolation and Conceptual Completeness for Pretoposes via Category Theory”, Mathematical Logic and Theoretical Computer Science (Lecture Notes in Pure and Applied Mathematics, Volume 106), New York: Dekker, 301–327.
  • –––, 1989, “Conceptual Completeness for First-order Intuitionistic Logic: an Application of Categorical Logic”, Annals of Pure and Applied Logic, 41 (1): 33–81.
  • –––, 1992, “On an Interpretation of Second-order Quantification in First-order Propositional Intuitionistic Logic”, Journal of Symbolic Logic, 57 (1): 33–52.
  • –––, 2000, “Categorical Logic”, Handbook of Logic in Computer Science, Vol.5, Oxford: Oxford Unversity Press, 39–128.
  • Pitts, A. M. & Taylor, P., 1989, “A Note of Russell's Paradox in Locally Cartesian Closed Categories”, Studia Logica, 48 (3): 377–387.
  • Plotkin, B., 2000, “Algebra, Categories and Databases”, Handbook of Algebra, Vol. 2, Amsterdam: Elsevier, 79–148.
  • Porter, T., 2004, “Interpreted Systems and Kripke Models for Multiagent Systems from a Categorical Perspective”, Theoretical Computer Science, 323 (1–3): 235–266.
  • Reyes, G., 1991, “A Topos-theoretic Approach to Reference and Modality”, Notre Dame Journal of Formal Logic, 32 (3): 359–391.
  • –––, 1974, “From Sheaves to Logic”, in Studies in Algebraic Logic, A. Daigneault, ed., Providence: AMS.
  • Reyes, G. & Zawadowski, M., 1993, “Formal Systems for Modal Operators on Locales”, Studia Logica, 52 (4): 595–613.
  • Reyes, G. & Zolfaghari, H., 1991, “Topos-theoretic Approaches to Modality”, Category Theory (Como 1990) (Lecture Notes in Mathematics, Volume 1488), Berlin: Springer, 359–378.
  • –––, 1996, “Bi-Heyting Algebras, Toposes and Modalities”, Journal of Philosophical Logic, 25 (1): 25–43.
  • Rodabaugh, S. E. & Klement, E. P., eds., Topological and Algebraic Structures in Fuzzy Sets: A Handbook of Recent Developments in the Mathematics of Fuzzy Sets (Trends in Logic, Volume 20), Dordrecht: Kluwer.
  • Scedrov, A., 1984, Forcing and Classifying Topoi, Providence: AMS.
  • Scott, P.J., 2000, “Some Aspects of Categories in Computer Science”, Handbook of Algebra, Vol. 2, Amsterdam: North Holland, 3–77.
  • Seely, R. A. G., 1984, “Locally Cartesian Closed Categories and Type Theory”, Mathematical Proceedings of the Cambridge Mathematical Society, 95 (1): 33–48.
  • Shapiro, S., 2005, “Categories, Structures and the Frege-Hilbert Controversy: the Status of Metamathematics”, Philosophia Mathematica, 13 (1): 61–77.
  • Sica, G., ed., 2006, What is Category Theory?, Firenze: Polimetrica.
  • Simpson, C., 2011, Homotopy Theory of Higher Categories, Cambridge: Cambridge University Press.
  • Taylor, P., 1996, “Intuitionistic sets and Ordinals”, Journal of Symbolic Logic, 61: 705–744.
  • –––, 1999, Practical Foundations of Mathematics, Cambridge: Cambridge University Press.
  • Tierney, M., 1972, “Sheaf Theory and the Continuum Hypothesis”, Toposes, Algebraic Geometry and Logic, F.W. Lawvere (ed.), Springer Lecture Notes in Mathematics 274, 13–42.
  • Van Oosten, J., 2002, “Realizability: a Historical Essay”, Mathematical Structures in Computer Science, 12 (3): 239–263.
  • –––, 2008, Realizability: an Introduction to its Categorical Side (Studies in Logic and the Foundations of Mathematics, Volume 152), Amsterdam: Elsevier.
  • Van der Hoeven, G. & Moerdijk, I., 1984a, “Sheaf Models for Choice Sequences”, Annals of Pure and Applied Logic, 27 (1): 63–107.
  • –––, 1984b, “On Choice Sequences determined by Spreads”, Journal of Symbolic Logic, 49 (3): 908–916.
  • –––, 1984c, “Constructing Choice Sequences for Lawless Sequences of Neighbourhood Functions”, Models and Sets (Lecture Notes in Mathematics, Volume 1103), Berlin: Springer, 207–234.
  • Wood, R.J., 2004, “ Ordered Sets via Adjunctions”, Categorical Foundations, M. C. Pedicchio& W. Tholen, eds., Cambridge: Cambridge University Press.
  • Yanofski, N., 2003, “A Universal Approach to Self-Referential Paradoxes, Incompleteness and Fixed Points”, The Bulletin of Symbolic Logic, 9 (3): 362–386.

Copyright © 2014 by
Jean-Pierre Marquis<Jean-Pierre.Marquis@umontreal.ca>

Open access to the SEP is made possible by a world-wide funding initiative.
The Encyclopedia Now Needs Your Support
Please Read How You Can Help Keep the Encyclopedia Free

How it feels to learn JavaScript in 2017

$
0
0

Hey, I got this new web project, but to be honest I haven’t coded much web in a few years and from what I’ve been reading it looks like the landscape changed a lot. You are the most up-to-date web dev around here, right?

-I guess you could say that.

Cool. I need to create a page that displays the latest activity from the users, so I just need to get the data from the REST endpoint and display it in some sort of filterable table, and update it if anything changes in the server. I was thinking maybe using jQuery to fetch and display the data? I know there are newer frameworks out there, but the more I read about them the more confused I get.

-Wasn’t jQuery the reason you quit web dev a few years ago?

Yeah, well I think I was doing it wrong. I couldn’t figure out why my app kept getting into weird states. Maybe you can teach me to organize my jQuery code better so I don’t get myself into so much trouble.

-It happens to the best of us. With jQuery you change the DOM in response to various events, and sometimes the sequence of events is different from what you expect. How you got to a particular state definitely can be confusing.

You aren’t convincing me I should get back into web dev.

-Wait; hear me out. With modern web frameworks you simply write how the state of your data maps to the state of your web page. It’s way less confusing.

OK let me think about that…wouldn’t that mean it redraws the page every time your data changes? I guess that could work. My users are all on desktop so it won’t be a big deal. It sounds like it would be really slow on mobile browsers, though.

-It doesn’t redraw the page every time. Modern frameworks are smart enough to figure out what part of the DOM needs to change and only touch that part.

Interesting. Which one should I use? The big ones are React, Angular and Ember, right?

-They’re all good. If you want to write your front-end logic in Handlebars use Ember. If you want to write your front-end logic with HTML attributes use Angular or Vue. If you want to write your front-end logic with JavaScript use React, Mithril or Inferno.

JavaScript, I guess. But doesn’t React use this other thing…JSX?

-JSX is just a JavaScript syntax extension. It lets you use HTML tags where you might otherwise write code that generates DOM elements.

What’s wrong with just using JavaScript?

-Nothing’s necessarily wrong. Actually Mithril’s documentation is all straight JavaScript. I’ve just found I get better feedback from people who do HTML/CSS all the time when I show them JSX code than I do when I show them straight JavaScript.

All straight JavaScript? I’m glad to hear I’m not the only one who isn’t entirely comfortable with JSX. You just made me want to try Mithril. Is it popular?

-It’s popular enough that it’s not going away, but nowhere near as popular as the bigger frameworks. I’ve actually been writing a pretty ambitious web app in Ember lately. But come to think of it, Ember hides certain things I’d like you to explicitly see when you’re getting up to speed, so I’d be happy to show you how you’d implement your app using Mithril.

Great! Can we set up a few hours soon when you can show me how to set up all the libraries, scaffolding and boilerplate code? Which is better these days, webpack or browserify? I have to confess, the build setup is the most intimidating part of modern web development for me.

-We can skip all that for now. Once you get a feel for the main part of modern web dev you can just copy a build setup I made with nothing more than babel and rollup. The build system is really only a small part of developing a modern web app.

Skip all that? But I want to see my web app actually work.

-You will; I’ll show you. Let’s code up your app using Mithril now. You said it was a filterable table, right? Let’s make planets.html a filterable table of planets.

<!DOCTYPE html>
<html>
<body>
<div id="app">Loading...</div>
<script src="https://unpkg.com/mithril/mithril.js"></script>
<script src="https://unpkg.com/babel-standalone@6/babel.min.js"></script>
<script type="text/jsx" src="planets.jsx"></script>
</body>
</html>

OK, you told me what Mithril is. Another library? What’s Babel?

-Babel lets you to use modern JavaScript syntax that browsers don’t support it yet. You don’t absolutely need it, but it lets you forget what browsers do and don’t support and just code.

Oh wait, actually I think I read about that. Isn’t it bad to run a transpiler in your browser?

-Yes. It adds significant latency. But for learning, why not? It will be easy to change later. For now, let’s start writing your planets.jsx file by setting up the state of your app.

Just so you know, I have a meeting in 20 minutes. From what I’ve been reading, I know that when you say “state” things are about to get complicated. Maybe we should talk about Redux, Flux, etc. another time.

-We can talk about those when you’re writing an app that calls for them. With this app you just need two variables: your array of planets and a filtering function.

'use strict';
/** @jsx m */
let planets;
let planetFilter = planet => true;

Wait, shouldn’t that be var and not let?

-It’s the same, except let is block-scoped like variables in C or Java. No more hoisting.

Oh yeah, it’s been a while. I had forgotten about hoisting.

-You can go ahead and forget about it again. Just use let for variables you might reassign, and const for everything else.

You said that second one was a filtering function. Is that arrow just shorthand for the old-style function declaration?

-Yes, an arrow function is pretty much the same as the old function syntax with bind(this).

Oh yeah, I remember back when you helped me find that bug that we fixed by adding bind(this). I think I’m going to like these arrow functions.

-I bet you will. Now let’s make the top-level component for your app.

class PlanetApp {
view() {
return (
<section>
<PlanetFilters />
<PlanetTable planets={planets} />
</section>
);
}
}

That must be the new ES6 class syntax. I like the look of it. I’m not sure what to think of HTML mixed in with the JavaScript though.

-Don’t think of JSX as HTML mixed in with JavaScript. It’s equivalent to hyperscript, i.e. JavaScript for creating HTML elements. It’s important to understand that the JavaScript it compiles to doesn’t make a string; it makes an actual structure of elements. If, for example, your tags are unbalanced, it won’t compile.

OK, I’ll give it more time to see if I like it. Next can you show me the PlanetTable component?

-You bet. This is really the core of your app.

class PlanetTable {
view() {
return (
<table>
<tr>
<th>Name</th>
<th>Composition</th>
<th>Moons</th>
</tr>
{planets.filter(planetFilter).map(planet => <PlanetRow planet={planet} />)}
</table>
);
}
}

-Mostly it’s just static content, but on that one line you have a concise description of what your app does. It takes an array of planets, filters them to just the ones that should be shown, and that filtered array is mapped to HTML table rows.

Oh, I think I’m getting it now! That JSX syntax is just a JavaScript expression, so I can manipulate it however I want. I’m guessing the PlanetRow component is going to be really simple?

-Yes, possibly simpler than you think thanks to destructuring assignment.

class PlanetRow {
view(vnode) {
const { composition, name, moons } = vnode.attrs.planet;
return (
<tr>
<td>{name}</td>
<td>{composition}</td>
<td>{moons}</td>
</tr>
);
}
}

OK, so I guess vnode.attrs.planet is how you get that planet attribute that was passed in. There’s only one line with an equals sign so that must be…oh wow, destructuring assignment, where have you been all my life?

-I’m telling you, JavaScript is way more fun than it used to be. Here, let me show you how nice arrow functions are, even when you’re only considering their conciseness.

class PlanetFilters {
view(vnode) {
return (
<p>
<PlanetFilter
key="All"
func={planet => true} />
<PlanetFilter
key="Terrestrial"
func={planet => planet.composition === 'terrestrial'} />
<PlanetFilter
key="Gas Giant"
func={planet => planet.composition === 'gas giant'} />
<PlanetFilter
key="Ice Giant"
func={planet => planet.composition === 'ice giant'} />
</p>
);
}
}

OK, I see where you’re going with this. Those are filter functions. But I bet the event handlers you have to wrap around them can’t be that concise.

-They can with a little abstraction.

class PlanetFilter {
view(vnode) {
const { key, func } = vnode.attrs;
return (
<label>
<input type="radio" name="filter"
onchange={filterHandler(func)} /> {key}
</label>
);
}
}
function filterHandler(filterFunction) {
return function(event) {
if (event.target.checked) {
planetFilter = filterFunction;
}
};
}

-But you have a meeting to go to and you wanted to see this working. Here, let me toss some data into a separate planets.json file since you mentioned you needed to fetch data from the server. And now we just need code to fetch the data, put the data where your app can get it, and then mount the top-level component. Voila, it works.

m.request({url: 'planets.json'}).then(data => {
planets = data;
m.mount(document.getElementById('app'), PlanetApp);
});

Really, that’s it? Wow, last year this all seemed so intimidating! I have to run now, but I’m definitely looking forward to picking JavaScript up again. Thanks a bunch!

-Sure thing. I love talking about it. Any time.

Chinese gov't demands name verification for online commenting

$
0
0

China announces new rules, forbidding unidentified netizens from posting anything on internet platforms

China’s internet censorship is getting tougher and more comprehensive every day. On August 25, China’s top internet regulator announced new rules to manage internet forums and communities, forbidding unidentified netizens from posting anything on internet platforms. The new rules will become effective on October 1.

As The Diplomat has been following, since Chinese president Xi Jinping took office, China has been systematically increasing online control, and 2017 has witnessed the most fierce wave of  internet censorship yet:  Banning VPNs and independent multimedia contents, demanding international publishing houses such as Cambridge University Press remove specific content, punishing China’s top three internet giants for failing to manage their online platform properly, to name just a few.

The Cyberspace Administration of China, the top internet censor, just gave Chinese netizens further bad news. On August 25, the administration issued “Management Regulations on Internet Forum and Community,”  in order to “promote the healthy and orderly development of online community” and “safeguard national security and public interests.”

Enjoying this article? Click here to subscribe for full access. Just $5 a month.

The new regulations cover all online forums, communities and any other platforms that provide interactive communication.

According to the regulations, all internet companies and service providers are required to strictly manage all the content their registered users are going to post, and verify the identities of all users before they can post anything on their platforms. As for those users who refuse to provide their real names, the internet companies should not allow them to post anything at all.

An official of the Cyberspace Administration also specified the content that is banned from publishing or disseminating online:

The contents that (1) opposing the basic principles in the Constitution; (2) harming the national security, revealing state secrets, subverting state power and undermining national reunification; (3) damaging national honor and interests; (4) inciting national hatred, ethnic discrimination and undermining national unity; (5) undermining the state’s policies on religion or promoting cults and feudal superstitions; (6) spreading rumors or disrupting social order; (7) spreading obscenity, pornography, violence, terror or abetting the crime; (8) insulting or slandering others and infringing upon the lawful rights and interests of others; (9) violating any other laws and regulations.

Obviously, the forbidden items are so broad and vague that any criticism could be included in the categories.

In addition, the regulations also require all internet companies to fully record their users’ information and promptly report their illegal behavior to the regulators.

Germany's facial recognition pilot program divides public

$
0
0

German Interior Minister Thomas de Maizière paid a special visit on Thursday to Berlin's Südkreuz train station, one of the largest railway junctions in the country's capital. With local and long distance trains arriving every minute, thousands of people pass through this station every day. But the minister was not interested in the trains. De Maizière, who is responsible for domestic security in Germany, wants to find out how effective his facial recognition pilot project really is.

Critics of increased government surveillance also came out to Südkreuz station to protest de Maizière's visit. They have demanded the six-month field trial, launched in early August, be terminated immediately. Activist Paul Gerstenkorn from "Digitalcourage," a German privacy and digital rights organization, claims that the technology used in the tests would create more extensive profiles of the country's citizens than advocates of the project admit.

The 300 testers who volunteered for the project carry a transponder that apparently only transmits data on ambient temperature, battery status and signal strength, according to the project staff member in the Südkreuz station control room who explained the technology to de Maizière. But Gerstenkorn contends the angle and acceleration of the testers are recorded as well. With the help of his smartphone, Gerstenkorn easily detected the number of testers within a radius of 20 meters (66 feet), while de Maizière was fielding questions from journalists. The word "blukii," the name of the transponder used for the facial recognition field trial, appeared 10 times on Gerstenkorn's phone screen.

A phone showing the detection of the 'blukii' software (DW/M. Fürstenau)

Gerstenkorn was able to easily detect nearby participants in the pilot program

For German Data Protection Commissioner Andrea Vosshoff, the fact that active and not passive technology is being used is going too far. Unlike a passive chip, the transponder constantly transmits information that anyone can collect with the help of freeware available on the internet. Vosshoff says the police have not "sufficiently" informed the testers, and called for the project to be temporarily halted.

Surveillance as public safety

But Reinhard Thieme, a supporter of the program who was passing through Südkreuz station, has no reservations whatsoever. He considers every form of video surveillance to be useful for safety reasons and as evidence in criminal investigations.

De Maizière would be pleased to hear such views. The interior minister has vehemently defended the project, saying the technology is not being used to catch petty criminals such as shoplifters, but terrorists and serious offenders. Four weeks into the test phase, De Maizière has praised its "surprising accuracy" - specifically referring to people recognized by the software whose pictures are already stored in police databases. According to Germany's federal police force, pictures of all other passers-by captured by the surveillance cameras are "immediately deleted."

Berlin's Südkreuz station (picture-alliance/dpa/J. Carstensen)

Seventy-seven cameras have been set up at Berlin's Südkreuz station for the trial run

Fooling the software

After the six-month trial phase in Berlin, a decision will be made on whether automatic facial recognition will be implemented nationwide in Germany's train stations and other public spaces. "If this works out well, it would be an incredible increase in safety for the population," said de Maizière, who also cautioned there was still work to be done. "What happens when people put on sunglasses, a hat or a hood?"

The interior minister could have the answer sooner than expected, as many opponents of the facial detection project were riding up and down the escalators at Südkreuz station during his visit. They could be seen from the control room while he was listening to police explain the technology. The activists wore masks and wigs, or held newspapers in front of their faces. For the moment, it looks like there are still a few ways to escape facial recognition.

Breaking up the Container Monolith

$
0
0

Dan Walsh, of SELinux Coloring Book fame, presented on the work he and his team have been doing with containers. Dan has long been a technical leader in the container and SELinux spaces and is an amazing guy.

If you take a moment to think back to the PDF format, it was originally created by Adobe to solve representing a document in a consistent way. However, that is not what made it popular and useful. The power of the PDF format is the format is open and there are tons of tools to read and write them. This is what made the PDF format ubiquitous. Dan argues the same needs to be true of containers. They need to be every bit as generic as the PDF format, with multiple creation and runtime tools. Competition breeds better software and ultimately better standards.

If you look at containers, containers are just linux in a standard image format. So, what does OpenShift/Kubernetes need to run a container? It needs a standard container format. It needs a way to pull and push containers to/from registries. It needs a way to explode the container to disk. It needs a way to execute the container image. Finally and optionally, it needs a container management API. Red Hat has been working on these problems under the ‘Project Atomic’ flag. This is a collection of tools, all under one CLI command called atomic.

Standard Container Image Format

The Open Container Initiative (OCI) provides the OCI Image Format to standardize the container image format. This is what allows everything else to be developed, as now there is the “PDF” for containers.

Registry Interaction

Skopeo (‘remote viewing’ in Greek) is used by the Atomic CLI. Its purpose in life is to view container metadata at the registry without downloading the full image. It can now pull/push images as well. Red Hat worked with CoreOS to split it out into a go library and is now available on Github.

Explode Image Local Storage

A critical need for containers was a way to mount the container image without actually running it. There are many places where this is useful and required. For example, if you want to run an OpenSCAP scan of the container to make sure there are no critical vulnerabilities with it, you don’t want to actually execute the container. You need a way to mount the data in a standard way. This is now possible with the ‘atomic mount’ command. Of course, this project can be found on Github.

Execute the container

Executing the container was tackled by the full OCI standard, the OCI Runtime Specification. This standard describes what the container should look like on disk. runc is the default implementation of this standard. As of Docker 1.11, runc is the default backend. There are a couple other runtime implementations, runv and clear are alternatives; however, docker only supports runc as a backend. The runtime standard allows the PDF-like tool chain ecosystem.

These various components make up the bare minimum for running a container. That said, the container space also could use other concepts borrowed from full linux distributions.

Container Signing

The industry needs a way to sign containers in an open and scalable manner, similar to GPG web-of-trust. Strangely enough, containers can now be signed with GPG; moreover, the signatures can be detached from the image. They are also not specific to any registry. You can now trust containers produced by trusted organizations, as well as verify they have not been altered. You can find a signing example here and one for managing trust here.

System Containers

Red Hat has developed Atomic Host over the last several years.  Atomic Host is an OS based on RHEL but designed for running containers in a lightweight fashion. On Atomic Host, software is shipped as a container. Dan believes the future OS will be a very basic OS where system services and everything else will be installed as containers. Atomic CLI can now useSkopeo to pull down images, using OSTree to store image layers on disk, then creates a systemd unit file, then use runc to run the container. Note, there is no daemon involved here, which is important for boot processes. This allows etcd and flannel to start before the container runtime! Even open source docker can now run as a system container.

Standalone Containers

Standalone containers are a method to provide normal RHEL content as containers, rather than using RPM. Daemons run in containers with standard ports and volumes prepackaged for standard use-cases. Think of Apache when started at boot, listening on privileged ports, reading local content, etc.

Improved Storage

Read-only container images should be the default, but they are not currently. In production, images should really be immutable.  This would allow us to get rid of the COW filesystem, improving storage performance. Shared filesystems would also be a huge benefit here with read-only containers.  Currently, in order for a runtime in a cluster to run a container, that container has to be pulled down from a registry and stored on local storage.  Moving to an NFS storage model would dramatically improve performance and image management.  Rather than having to push new containers to a registry, just so all your OpenShift nodes can download it, wastes a ton of CPU, storage, and network resources. Dan’s team is working on this problem by supporting network-based storage.

Container Image Development Tools

Dockerfile sucks. It’s a fact. It has been four years and we still have to use Dockerfile, which is a horrible version of bash.  An image in a tarball and a json file at its core. We now have Ansible-Containers, which gives us another way to describe a containerized application; however, it still uses Dockerfile under the hood. We need a way to build images without a container runtime daemon. This brings us to Buildah, which is an open source tool for building containers without using Dockerfile at all.  This tool also allows us to build containers without all the container development tools inside it (yum, GCC, secrets, etc). Basically, Buildah produces clean containers, which means the container can be much, much smaller.  It even supports reading in a Dockerfile to produce a container, but that kind of defeats the point.

Container Management API

Red Hat started working on the CRI-O effort.  It is not a form of docker, rather it supports the Kubernetes container management API or to put it another way “it implements the Kubelet Container Runtime Interface (CRI) using OCI conformant runtimes”. Kubernetes communicates with CRI-O, which in turn executes the container.

Dan’s team at Red Hat has been focused on bringing open source development, principles and concepts to the container lifecycle. While containers to date are all open source, the container management system has been one large monolith. Breaking up the container into individual components allows for rapid innovation. These components each evolve, being developed, forked or even allowing for net-new tools to meet the needs of the container ecosystem. These efforts are now paying off and the container landscape is much better for it.


Whether you are new to Containers or have experience, downloading this cheat sheet can assist you when encountering tasks you haven’t done lately.


Join the Red Hat Developer Program (it’s free) and get access to related cheat sheets, books, and product downloads.

Introducing Puppet Support for Google Cloud Platform

$
0
0

The ability to control resources programmatically with tools they know and love can make a big difference for developers creating cloud-native applications. That’s why today, we released and open sourced a set of comprehensive modules to improve the ability for Puppet users to manage Google Cloud Platform (GCP) resources using the Puppet domain specific language, or DSL. The new modules follow Puppet’s object convergence model, allowing you to define the desired state of your GCP resources that our providers will enforce directly within the Puppet language.

The new modules support the following products:

These new modules are Puppet Approved, having passed the rigorous quality and review bar from Puppet Engineering team, and are open-source under the Apache-2.0 license, available from GCP’s Github repository.

We also released a unified authentication module that provides a single authentication mechanism for all the modules.

The modules have been tested on CentOS, Debian, Ubuntu, Windows and other OS’s. Refer to the operating system support matrix for compatibility details. They work with both Puppet Open Source and Puppet Enterprise.

To learn more, and see them in action, register for the webinar with me and Cody Herriges from Puppet on 13 September 2017.

The power of Puppet 

It’s important to note that Puppet is not a scripting language. Rather, it follows an object-convergence model, allowing you to define a desired state for your resource, which our providers make so by applying necessary changes.

In other words, with Puppet, you don’t say “run this list of commands to install Apache on my machine,” you say “Apache should be installed and configured.” There is some nuance here, but with the latter, Puppet handles verifying if Apache is installed, checks for the correct dependencies, upgrades it if it’s not at the correct version and — most importantly — does nothing if everything is good. Puppet already understands the implementation differences across operating system and will handle doing the right thing for your chosen distribution.

Following an object-convergence model has various benefits: It makes your resource manifest declarative, abstracting away various details (e.g., OS-specific actions); and it makes definitions simpler to read, modify and audit. The following manifest creates a full Google Container Engine cluster, in just 15 lines of code.

gauth_credential { 'mycred':provider => serviceaccount,path     => '/home/nelsona/my_account.json',scopes   => ['https://www.googleapis.com/auth/cloud-platform'],
}gcontainer_cluster { 'myapp-netes':ensure             => present,initial_node_count => 2,node_config        => {machine_type => 'n1-standard-4',# we want a 4-core machine for our clusterdisk_size_gb => 500,             # ... and a lot of disk space
  },zone               => 'us-central1-f',project            => 'google.com:graphite-playground',credential         => 'mycred',
}
For specific examples of how to use Puppet with the individual GCP modules, visit their respective Forge pages.

Getting started with Puppet on GCP 

To hit the ground running with Puppet and GCP, follow these basic steps:
Install the appropriate modules.
  1. Get a service account with privileges on the GCP resources you want to manage and enable the the APIs for each of the GCP services you intend to use. 
  2. Describe your GCP infrastructure in Puppet. 
  • Apply your manifest. 
  • Let’s discuss these steps in more detail.

    1. Install your modules 

    All Google modules for Puppet are available on Puppet Forge. We also provide a “bundle” module that installs every GCP module at once, so you can choose the granularity of the code you pull into your infrastructure.

    Note: Google modules requires neither administrator privileges nor special privileges/scopes on the machines being executed. It is safe to install the modules either as a regular user or in your Puppet master. Install on the master if you want it distributed to all clients.

    The authentication module depends on a few gems released by Google. As with everything related to system configuration, you can install the gems using Puppet itself.

    $ puppet apply <<EOF
    package { [
        'googleauth',
        'google-api-client',
      ]:
        ensure   =< present,
        provider =< gem,
    }
    EOF
    Here’s the command for Installing all the Puppet modules with a single command:
    puppet module install google/cloud
    Or, you can install only the modules for select products:
    puppet module install google/gcompute    # Google Compute Engine
    puppet module install google/gcontainer  # Google Container Engine
    puppet module install google/gdns        # Google Cloud DNS
    puppet module install google/gsql        # Google Cloud SQL
    puppet module install google/gstorage    # Google Cloud Storage
    Once installed, verify the modules’ health by running:
    puppet module list
    You should see an output similar to:
    $ puppet module list
    /home/nelsona/.puppetlabs/etc/code/modules
    ├── google-cloud (v0.1.0)
    ├── google-gauth (v0.1.0)
    ├── google-gcompute (v0.1.0)
    ├── google-gcontainer (v0.1.0)
    ├── google-gdns (v0.1.0)
    ├── google-gsql (v0.1.0)
    └── google-gstorage (v0.1.0)
    /opt/puppetlabs/puppet/modules (no modules installed)

    2. Get your service account credentials and enable APIs 

    To ensure maximum flexibility and portability, all authentication and authorization to your GCP resources must be done via service account credentials. Using service accounts allows you to restrict the privileges to the minimum necessary to perform the job.

    Note: Because service accounts are portable, you don’t need to run Puppet inside GCP. Our modules run on any computer with internet access, including on other cloud providers. You might, for example, execute deployments from within a CI/CD system pipeline such as Travis or Jenkins, or from your own development machine.

    Click here to learn more about service accounts, and how to create and enable them.

    Also make sure you have enabled the the APIs for each of the GCP services you intend to use.

    3a. Define authentication mechanism 

    Once you have your service account, add this block to your manifest to begin authenticating with it. The resource title, here 'mycred' is referenced in the objects in the credential parameter.
    gauth_credential { 'mycred':provider => serviceaccount,path     => '/home/nelsona/my_account.json',scopes   => ['https://www.googleapis.com/auth/cloud-platform'],
    }
    For further details on how to setup or customize authentication visit the Google Authentication documentation.

    3b. Define your resources 

    You can manage any resource for which we provide a type. The example below creates a Kubernetes cluster in Google Container Engine. For the full list of resources that you can manage, please refer to the respective module documentation link or to this aggregate summary view.
    gcontainer_cluster { 'myapp-netes':
      ensure             => present,
      initial_node_count => 2,
      node_config        => {
        machine_type => 'n1-standard-4', # we want a 4-core machine for our cluster
        disk_size_gb => 500,             # ... and a lot of disk space
      },
      project            => 'google.com:graphite-playground',
      credential         => 'mycred',
    }

    4. Apply your manifest 

    Next, tell Puppet to enforce and bring your resources into the state described in the manifest. For example:
    puppet apply <your-file.pp>

    Please note that you can apply the manifest standalone, one time, or periodically in the background using an agent.

    Next steps 

    You’re now ready to start managing your GCP resources with Puppet, and start reaping the benefits of cross-cloud configuration management. We will continue to improve the modules and add coverage to more Google products. We are also in the process of preparing the technology used to create these modules for release as open source. If you want have questions about this effort please visit Puppet on GCP Discussions forum, or reach out to us on puppet-on-gcp@google.com.

    Further Reading 

    Viewing all 25817 articles
    Browse latest View live


    <script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>