Gil's blog

Why did the United States invade Iraq?

Submitted by gil on
Kitty Saddam Hussein

There's a half-remembered folk history of the lead up to the invasion of Iraq that's shaped the war's modern understanding in the United States. Here, I'll lay out in brief the several contributing factors that made the invasion happen.[1]

They Thought It Would Work

This is sort of silly to bring up because it applies to most everything that has ever happened in history: people do things because they think they'll work out, the risks will outweigh the benefits, and in the end they'll be better off than how they started. But I want to discuss this here because the modern discourse isn't always interested in understanding the perspective of the perpetrators. And it's not just Bush and Cheney that thought it would work: the war happened because there were multiple constituencies or interest groups who thought it would work and these groups drew their own benefits, often different from each other, from the war. The Republican party, just like the Democratic party, is pluralist: there are multiple groups with their own interests and can have different motivations for the same action. That's mostly what we'll discuss here: who are those groups and what did they see in Iraq? Ultimately, what made the US invade Iraq was this confluence of interest groups all seeing an advantage to what Dubya was proposing.

The Same Team Already Did It Once

They thought it would work because the US already tried invading Iraq once before in the Gulf War and that went pretty well. Some of the same people came back to do it the second time, notably Dick Cheney (SecDef) and Colin Powell. Maybe this mostly reassured the administration itself but you can see how they might go into the war with the mindset that it'll work out just like the Gulf War.

Cultural Conservatism

Remember that Republicans are genuinely conservative: they hold conservative political beliefs, they think those beliefs are consistent and correct, they want to see a country and world oriented to their conservative views and they actively work towards those goals. I want to dispel the sometimes implicitly held idea that these beliefs are purely rhetorical, just a fig leaf over someone's genuine political motivations. Maybe this does not apply to everyone but does apply to lots of Republicans, especially once you get away from politicians and down towards the base. For the duration of the Cold War these conservatives had a consistent message: the US stands against Communism and authoritarian governments. An invasion of Iraq gives a renewed version of the branding: the US still stands against authoritarian and oppressive governments and is willing to use its industrial and military might to be a force for good in the world. It gives the masses something concrete to believe in and a movement to belong to. A war reaches all corners of the society, it gives an opportunity to rally around the flag in public, and it can even reach down into the smallest rural communities when their young men enlist, potentially giving a personal connection to the far-away conflict. The war becomes a way to build the conservative culture that you wish the country had.

Saddam Was A Bad Dude

The moral aspect of overthrowing Saddam resonated with the public because there was a sort of cosmic background radiation about the guy from the events of the 80s and 90s. Saddam had use chemical weapons against his own people before! He attempted genocide against his own citizens in 1998. Even Bernie Sanders, who famously opposed the 2003 invasion of Iraq was in favor of overthrowing Saddam in a speech in 1998. Bernie is not being inconsistent here: he believes the US should defend the oppressed people of the world, he just differs on the details of how it should be done. In the 1998 speech he's arguing the US should be arming an anti-Saddam insurgency and in the 2003 speech the US should operate within a UN coalition instead of unilaterally. Even when the arguments made by the Bush administration for the 2003 invasion flopped it wasn't enough to shake popular support of the invasion because the country was already pre-positioned to accept that overthrowing Saddam was just. 

Stovepiping

We now know that the evidence presented for Saddam's WMD program or collaboration with Al-Qaeda was wrong. Today there is a common belief, either stated or implicitly held, that the entire government intelligence community was behind this evidence and Iraq became either the first instance or the big reveal of the intelligence community's incompetence. In other words: they got it wrong with Iraq, why would you trust them on the issues of today? But even at the time there was widespread criticism of the Bush administration's evidence both outside and inside of the government: for example, the Plame affair can only happen as an act of revenge if someone is credibly and publicly criticizing the government's position. The administration had to turn to tricks like the Office of Special Plans, a whole institutional apparatus created just to shortcut the existing intelligence community to give Bush the evidence he wanted for the invasion. This stovepiping technique is insidious: it hijacks the credibility of the existing organization and uses it to sway those who trust the institutions with a work product ultimately not created by the institutions. Personally, I think stovepiping contributed to war support from a lot of people who "should have known better" like the Democratic leadership at the time: they were victim to an intentional plan to take advantage of institutional trust. It would have been better if they trusted institutions a bit further and leaned more into the non-stovepiped arguments that were made but unfortunately that didn't happen.

The End Of History

There was a well-timed argument made in 1989 that liberal democracy won out over authoritarian governments and that going forward the authoritarian governments of the world would eventually collapse out of the popular preference for democracy. Francis Fukuyama, the author of this "End of History" argument, eventually falls in with the like-minded 90s think tank Project for the New American Century. The PNAC becomes the spawning ground of neoconservative thought and Bush drafts many of its participants into his administration. That's the history, but I want to convey some of the mindset of these neoconservatives. Coming into the 90s they were greeted with a validation of their belief in liberal democracy and capitalism with the collapse of the USSR. Under George H.W. Bush the Gulf War was an incredible success, once again validating their faith in the US military. As the 90s dragged on the economy took off, eventually becoming the dot com bubble but genuinely transforming business and society for the better with improved computer and communication technology. The PNAC was high on its own supply, seemingly validated at every turn, and had no problem pitching aggressive and risky plans like the overthrow of Saddam Hussein. The neoconservatives were the constituency of fart-sniffing true believers.

Shortly after 9/11 George W. Bush gives a speech where he famously pitches that Al Qaeda "hates us for our freedoms". I want to bring this up because I think Bush is actually making an "End of History" argument here in a more digestible form for the public. The claim on its face is not true, 9/11 and Al Qaeda were motivated by Israel, but Bush's statement presupposes the End of History desire for liberal democracy and turns it into the motivating force for Al Qaeda. Here, the neoconservative ideology gives a way for the the administration to sell its ideas to the public. The same thing happens with "we will be greeted as liberators" - once Saddam is removed the End of History-driven desire guarantees the Iraqi people will self-organize a democracy.

Defense Contractors

A popular idea at the time was that defense contractors, looking for government war contracts, demanded the invasion. I think the military-industrial complex is certainly one of the groups in the coalition in favor of war but the popular conception that the defense industry dictates what it wants out of the government is backwards. I will elide some detail here for a later post but I think the appropriations and bidding processes keep civilian control over what contractors build and how much profit they can ultimately make. Ultimately, the Defense Production Act hovers over everyone's head as well: if things get real bad, the government can simply say "thank you for playing" and nationalize your business, wiping out your chance at profit.

Oil

Oil is important but I don't think the administration was motivated to invade Iraq for its oil. Iraq didn't have the oil, Kuwait did! That's why Iraq invaded Kuwait in the first Gulf War after all: Iraq's oil industry was economically uncompetitive. The US also started its invasion with the destruction of Iraq's oil industry infrastructure which is not what you'd do if you want to start exporting their oil. Dick Cheney, immediately before he was Vice President, was CEO of Halliburton, a company that originally provided oil infrastructure services but moved into defense contracting. Halliburton did secure oil and defense contracts from the US government during the war and Cheney and Halliburton's other stockholders profited from it. I don't think war profit was the motivation for many, though: it is just one of many causes that can help move some constituents to support the war. Cheney, in particular, I think is largely motivated by his neocon ideology even though he also stands to profit substantially from the war.

Revenge?

Iraq did attempt to assassinate George H.W. Bush, the father of George W. Bush, and some have floated the idea that the invasion of Iraq was motivated by GWB exacting revenge on Saddam for the attempt. Maybe? If could be in this big pile of motivations. But even if it was, it motivates only one person (albeit a very important person): GWB himself. Maybe it helped grease the wheels a bit but I don't think it contributed much.

Israel

Saddam Hussein had no way to strike the United States: he has no intercontinental bombers, no international navy and no ICBMs. Despite what the Bush administration suggested in the lead-in to the war Saddam had no capacity to attack the domestic United States. Israel, on the other hand, is a closer target and Iraqi-Israeli relations are not great. After the first Gulf War the UN imposed strict limitations on Saddam's missile program, limiting him to missiles unable to strike Israel, and these limits were enforced by UN inspections for some time. But it is not unreasonable to suspect that one day Saddam might acquire missile technology with the range he needs. Israel fears a war against all of its neighbors simultaneously (not that strange of a hypothetical, it's happened before!) and an Iraqi military with its population, relative military competence, and missiles that can strike from Iraq make it a serious threat. If Saddam does resurrect a chemical, biological or nuclear program he can launch dirty bombs at Israel. If he was willing to use dirty bombs against his own population of course he'd be willing to use them against Israel! You can see how Saddam Hussein and his military are a much larger threat to Israel than they are against the United States. A war that both overthrows Saddam, destroys his army and destroys any WMD labs motivates the pro-Israel constituency in the United States in favor of the war.

Kitty History

Everything I wrote above is bullshit. Here's why it really happened.


[1] Shoutout to Why Did We Invade Iraq? by Charles Davis which argues basically the same thing I am doing here but a bit more thoroughly and rigorously. I found this article while doing research for this one.

Copying my VPS to a new disk

Submitted by gil on

I've been a customer of TornadoVPS since 2011. The VPS came with an ext3 formatted filesystem which was at the time the default filesystem in the Debian installer. In the meantime the kernel's ext3 driver depreciated and eventually removed prompting me to do the in-place upgrade to ext4. However, at some point I wanted to get a new, clean filesystem with bigger inodes, fast_commit support and year 2038 fixes. Here's how I copied the filesystem over to a new ext4 partition.

The new disk

TornadoVPS support was willing to attach a second block device to my VPS to support my migration with the understanding that I would turn the original disk over to them to be deleted when I was done copying my data over. The new block device was hotplugged into the VM and was ready to be formatted right away:

fdisk /dev/xvdb
mkfs.ext4 /dev/xvdb1
tune2fs -O fast_commit /dev/xvdb1

There are plenty of useful ways to set up fdisk, just don't forget to have partitions aligned and give some space at the start of the disk for grub's data. (That is, have the start of the first partition not be right at the start of the disk.)

Backups

I rebooted into a rescue system to do the off-site backup of the original disk. You could use dd to dump the whole thing but there's a nifty e2image command that just dumps the blocks used by the filesystem. You can dump it over SSH easily:


ssh -i ~/.ssh/id_xyz root@the_rescue_system e2image -apr /dev/xvda1 - > backup_YYYYMMDD.img

To restore the image, you'd do it in reverse:

cat backup_YYYYMMDD.img | ssh -i ~/.ssh/id_xyz root@the_rescue_system e2image -apr - /dev/xvda1 backup_YYYYMMDD.img

Copying everything over

I rebooted into a system-provided rescue disk to do the copy although you may be able to do this from the initrd shell or single user mode. I didn't want to be running anything off of the disk, even mounted read-only, just to be safe.


mkdir /mnt/disk1
mkdir /mnt/disk2
mount /dev/xvda1 /mnt/disk1
mount /dev/xvdb1 /mnt/disk2
time rsync -ahHAXxS --info=progress2 --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found} /mnt/disk1/ /mnt/disk2

In brief, the flags to rsync are:

  • --info=progress2: instead of --verbose, this is a less spammy and more useful output for copying a large number of files.
  • --exclude: A sensible list of directories that don't need to make the jump.
  • -a: archive. (preserve timestamps and owenrship)
  • -h: file sizes in human-readable units
  • -H: preserve hard links
  • -A: preserve ACLs
  • -X: preserve xattrs
  • -x: don't cross file system boundaries. Unlikely to affect anything but a safe default.
  • -S: preserve sparse files

Fixing up the bootloader

First, you need to grab the UUID from the new filesystem.


tune2fs -l /dev/xvdb1 | grep 'Filesystem UUID'

Update the /etc/fstab on the new disk with the new filesystem's UUID.

Next, you want to start configuring the bootloader. First, change the GRUB_TIMEOUT in /etc/default/grub to -1 to let grub halt and let you choose the new disk. On the original disk's /boot/grub/grub.cfg I found the menuentry section for my current kernel, copied it over, updated it with the new UUID and renamed it. This was to support booting from the first disk onto the second as it is not possible to configure the VPS to boot from the second attached disk.

If you manage to get grub installed to the MBR of the second disk you could instead have the first disk chainload grub via a simpler menuentry:

menuentry "Boot from second disk" {
set root=(hd1)
drivemap -s hd0 hd1
chainloader +1
}

Once you're done editing the files you need to rebuild the grub configuration on the first disk. The easiest way to get a working chroot is to use the new systemd-nspawn tool, however this requires you to have the systemd-container package installed. It's a quick dependency to install and you should be able to do it with the package manager shipped in the rescue system.


systemd-nspawn -D disk1 -a --bind=/dev/xvda --bind=/dev/xvda1 --bind=/dev/xvdb --bind=/dev/xvdb1 --resolv-conf=off /bin/bash
(in the chroot) update-grub

Although you don't need the xvda / xvda1 bind mounted into the disk for this here is how you'd do it. You almost certainly want the --resolv-conf=off to avoid clobbering your resolv.conf. Note that you may be tempted to chroot into the second disk to run grub-install to the second disk (/dev/xvdb), however, the bind mounts were confusing grub and I had no luck with it.

Finally, outside of all the chroots, unmount your disks and restart.

umount /mnt/disk1
umount /mnt/disk2
shutdown -r now

Finishing up the new system

Upon reboot you should be able to choose your new menu option and boot off of the filesystem of the new disk. Check the dmesg after booting for the new disk's UUID to make sure you've come up on the right system! If everything is looking fine, you can install grub to the MBR of then new disk:


grub-install /dev/xvdb
update-grub

And then do a reboot to make sure everything's still working. if you added the chainloader menuoption to the first disk's grub.cfg above you can try that to make sure the MBR installation of grub is correct.

Post-install

I think the -S flag to rsync turned my on-filesystem swap file into a sparse file. Linux didn't like that and I had to create a new swap file and attach that.

Once everything was up and running on the new system I powered the system down and asked support to detach the old volume so I could do a clean start onto the new disk for the first time.

Conclusion

This process was easy enough I would not be opposed to re-creating filesystems after major kernel updates, especially if there are new filesystem features or flags that I can enable on the new code.

Taylor Swift albums as Linux distributions (Manjaro Linux will surprise you!)

Submitted by gil on

Taylor Swift s/t - Amazon Linux 2

Taylor's self-titled release came out in 2006. As an unknown talent she had to spend several years prior practicing her songwriting and slowly putting together what would be a chart-topping debut album. Everyone's familiar with it, most enjoy it, but it's rarely someone's favorite. Amazon Linux 2 is a fork of RHEL, a distribution with its own lengthy history, and is the culmination of effort in and outside of Amazon. Nearly everyone's worked with Amazon Linux at some point and sure, it gets the job done, but nobody really reaches for Amazon Linux outside of AWS or can say it's their favorite.

Fearless - Maemo

Taylor hit the typical sophomore slump with Fearless. It was time to bring something new to the table and she just didn't quite pull it off. The result was forgettable, and so is Maemo, Nokia's attempt to break the iPhone/Android duopoly. Did you know that after the Maemo/Moblin merge an open source community kept maintaining the original Maemo codebase? Nobody does because nobody used this stuff. It was a desperate attempt to stay relevant and a cash grab.

Fearless (Taylor's Version) - Tizen

Speaking of desperate cash grabs, Taylor Swift re-recorded and released Fearless to turn her one-time album purchases into ARR (Taylor's Version). Tizen was spun up from the ashes of MeeGo (the child of Maemo and Moblin) by Intel and Samsung. Intel wanted to get in on the smartphone revolution and Samsung wanted a knife in their back pocket should Google threaten their independence but in the end nobody really got anything out of it.

Speak Now - NixOS

Taylor's third release demanded creativity and our girl delivered on Speak Now with the start of her turn from country to straight pop music. NixOS also struck out for new territory. They've invented creative ways to package software and deliver on an immutable, declarative and transactional system but the usability of the system and the overgrown DSL used to package it is still an open question.

Speak Now (Taylor's Version) - Guix

Nobody requested a re-release of Speak Now but Taylor did it anyway. Nobody asked to wrap Nix with Guile Scheme and use it to build an entire distribution but someone at GNU did it anyway.

Red - Arch Linux

No, it's not Red Hat, even though she's literally wearing a red hat. Red is the fan's album, it doesn't introduce anything new but instead carves out a comfortable, intimate space for her listeners. The Swifties love it, fans often rank it as their favorite, but it has little weight out in the greater world. Arch Linux has passionate fans all over the web but at the end of the day it's all but missing from the server room.

Red (Taylor's Version) - Manjaro

It's a fork of Arch. Nobody asked for either of these and yet here we are.

1989 - Android

Taylor finished her pivot to pop music with 1989 and put out her most popular and accessible album yet. User friendliness and accessibility is a must if you want to capture the broadest audiences. Google created a Linux with working sound out of the box, a useful sandboxing model for user applications and a coherent user experience but only after throwing out the entire Linux userspace. As a reward for their efforts Android is the world's most popular software platform and Linux's world domination is complete.

Reputation - Ubuntu

Blaccent Taylor is an embarrassment and simply shouldn't exist. Ads in /etc/motd are an embarrassment and Ubuntu shouldn't exist. Bonus: both embraced bold, unconventional color schemes.

Lover - Red Hat Enterprise Linux

Lover is another pop album, doing a little bit of everything and has broad appeal. It turned into a sleeper hit years after its release when Cruel Summer started charting again in 2023. RHEL's been around forever, it's used for everything, it has its fans and it simply isn't going away.

Folklore - Slackware

Folklore is something new and stylistically unlike any of her previously releases but also her attempt at finding a nostalgia that could resonate with her now diverse fanbase. Slackware's an OG, older than everything on this list, but even back then it was nostalgic for the commercial UNIX workstations of the 80s.

Evermore - Gentoo

Evermore came quickly on the heels of Folklore and kept its theme, writing about the timeless characters of the past and the way things used to be. However it wore out its welcome and wasn't nearly as well-received as Folklore was. Gentoo had its moment in the sun in the 2000s but you don't seem to hear much about it anymore. The nostalgia for building everything from source, via shell scripts, on the dinky one and two core machines of yesteryear may have inspired ebuild but we've moved on.

Midnights - Debian

Taylor Swift focused on what really matters in Midnights: accessible pop music, TikTok-friendly hooks (nice!) and simple, reliable operating systems found on every server and in every container across the globe.

1989 (Taylor's Version) - ???

Who knows? Maybe it'll be your favorite distro, maybe it'll be one of those Android vendor kernels that hacked in massive security vulnerabilities to ship some questionable junkware feature.

The RESTRICT Act

Submitted by gil on

Banning or regulating TikTok has been discussed for years but now Congress may actually make a move towards it with the RESTRICT Act. Lawfare has the history of TikTok regulations and a brief summary of the RESTRICT Act and gives some useful analysis, although I wish they'd write more on it given the popular attention the bill's had so far. Vox also has a summary without a lot of analysis. I'd like to talk through my analysis of the issue and give some of my perspective.

The Story So Far

TikTok is different in important and interesting ways. Taking a step back in American history, and looking at the bigger picture of American media, we've always had media that is, at a fundamental level, at odds with the state. You don't sell newspapers with happy headlines. But at the same time, also at a fundamental level, all American media has also been aligned with the state's interests. This is a tension but not a contradiction. The state wants to propagate itself and preserve its influence. The media wants the same for itself, but it is also reliant on the state as they are part of a shared American society. Should the state's existence or influence be threatened the media's existence or influence is also threatened. Both the state and the media are, at some level, in alignment because they are both comprised of Americans sharing an American society and all the popular media we've seen so far in American history has shared this alignment.

There is also an overlooked view of the relationship of the state to its subjects, one that most often arises between the state and corporations. The primary relationship between the state its subjects is the one where the state gives direction, in the form of law, and enforces the law. It is the relationship where the state is superior to its subjects. In the case of the media, the First Amendment puts strong restrictions on the state's power in this relationship by restricting the laws the state can pass. But, in a case that is often overlooked, the relationship between the state and one of its subjects can also be one of equals. This typically looks like a business deal or any place where the state might sign a contract. In those situations the state and its subject are working together: both have their own agency, interests and goals and both are negotiating to find a mutually beneficial outcome. This is a form of soft power that the state can wield to accomplish its goals. Some people are uncomfortable with this soft power, they think the only way the state should ever exert power is in the explicit style of the first kind of relationship. But this soft power is inevitable and those who would wish it gone are naive. It arises because the the state and its subjects are not one and the same, a distinction that comes from liberalism itself and does not exist in authoritarian states. The state, enforcing a liberal order, has distinguished itself from its subjects, giving them agency, and this separation creates the opportunity for soft power when they happen to interact as equals.

The Issue At Hand

Those are some useful ideas but also abstract ones. They could apply to TikTok but they also apply to any number of apps, mass media platforms, or specialist products like SolarWinds. However, the topic at hand is much more concrete: how could TikTok be used to further the objectives of the Chinese state? Let's walk through some examples of the levers available to them through TikTok.

First, the most extreme and most unlikely thing China could do with TikTok is to push zero days to everyone's phones. This is unlikely because it is an act of war, and if it did ever happen we'd have more important things to worry about. Cybersecurity is still important, we should promote security practices in the personal, private and public spheres to protect us from realistic cybersecurity threats. But even in the context of a hot war a digital Pearl Harbor is not necessarily a great idea. It is most beneficial for China if the general population's view is "what did China ever do to us?" and not "those bastards hacked everyone's phones and screwed everything up!" This is a case where soft power can be more useful than hard power, an important idea I will come back to later.

Next is the collection of user data. This includes the data mediated by the phone's security model (the permissions you've granted the app) but also your own interactions with the app. For TikTok, that would include your interests (based on the videos you've watched and interacted with) and any social graph you've built in the app (any phone contacts you've imported, any friends you've added on the platform, any DMs you've sent to friends). That is a lot of very personal information! I am sympathetic to people who want to take a broad brush across the industry to reduce the total amount of information any app is allowed to collect on a user or reduce the legal uses of that information. If the current baseline of restrictions is unacceptable, let's raise that baseline.

Even if we find the baseline is acceptable the consumers of that data are different actors with different goals. The technical and legal baseline is just a baseline and it is not unreasonable to have different, more stringent standards for different users of the same data. Facebook and Google collect this data because they want to sell you ads, and TikTok does as well. But the Chinese state has its own interests and should they find TikTok or its data useful to promote them they have TikTok at their disposal to do so. TikTok is different, in ways we haven't seen before in American history, and it should not surprise us if it is subjected to a higher legal standard. We can't reason about it in the exact same way we'd reason about the American media and it is not unfair or illiberal to treat it different than the way we've treated the American media.

Getting to specifics, what is some of this soft power that China could exercise through TikTok in its role as part of the media? Remember that in 2020, a video of the execution of George Floyd went viral on the internet, reaching every part of American society. We saw race riots in several large cities, but don't overlook that we also saw civil rights protests happen basically everywhere. There were civil rights protests in Vidor, Texas - a place that is arguably still a sundown town! The simple ability to promote a strategically chosen video to the American public is a soft power. It is a tricky power to use but when the stars align it can have vast influence over the entire American society. TikTok has this soft power and unlike the media of the past it is not broadly aligned with the American state. TikTok is different.

As an aside, I am not saying anything about the specifics of the George Floyd video, but instead I think it is a familiar and modern example of the strength of the soft power of viral media. In a more concrete example, TikTok could promote English-language videos of the Chinese-picked leadership in Tibet and Tibetan Buddhism, popularizing the idea of an aligned Tibet and covering up China's colonization of Tibet. You and I may know that Tibet was invaded for geopolitical reasons but it's much easier for the public to digest videos of a happy and unified Tibet over content that explains snoozefest details like the early security objectives of the People's Republic of China. And even in this example it would only be under the scope of the RESTRICT Act if it was done to steer American decision-making on a topic.

This is also an opportunity for the state-and-media-as-equals soft power introduced earlier. Fundamentally, the First Amendment prevents the state from telling the media what to do. Coincidentally, if the colonization of Tibet results in unhappy headlines, the media might print it anyway. When the state and media interact as equals, the state now has an opportunity to bring attention to stories not covered by the media. But it could also bring forward its intelligence or analysis on what the media has or hasn't covered and the media, still acting with its own agency, can run or bury stories in its own response to what it's learned or negotiated.

The question at hand is not one of liberalism or to the degree that our government should set a baseline of liberalism for the media at large. Liberalism is what created this soft power and got us into this situation in the first place! We are instead looking at the paradox of freedom (see: the large body of philosophy on this topic) and the question is if American state should be allowed to defend itself and its interests against the narrow case of specific organizations that are fundamentally unaligned with itself. I think it does, and I hope you agree with me - liberalism doesn't uphold itself, it has to be defended. That is what the RESTRICT Act sets out to do, and I think it does an OK job at it.

What does the RESTRICT Act do?

If you really want to know, go read the bill or go read the Lawfare article. But in short, the RESTRICT Act doesn't call out TikTok by name. Instead, it seems to share the same ideological framework I've laid out here: it defines a category of organizations that are fundamentally unaligned with the American state, who at times could wield soft or hard power against the United States or its citizens, and it delegates identifying the specifics of the abuse of that power and the state's response to it to the executive branch. This sort of delegation is typical and expected under our Constitution, and this new power comes with additional protections: it can only be exercised against information technology systems and it limits that further to systems under the control of foreign adversaries. The scope is narrow enough to make it consistent with my understanding of liberalism and I'd go as far as saying it's consistent with all but the most naive conceptions of liberalism.

Once you get past the bill's scoping of covered transactions the powers delegated to the executive branch are intentionally broad. The purpose here is to give the government flexibility beyond prohibition ("you can no longer do this thing") as a response, which gives the government flexibility to scope its own exercise of power against the covered transaction. I am sympathetic to criticisms that the law could add explicit scoping of delegated powers here but understand that it may be hard to effectively draft it. The already explicitly-stated prohibition is the most impactful of powers the government could exercise and it seems like a necessary power to let this bill have any effect at all. There is little point in limiting the seemingly-broad "any other action" when prohibition is going to be a necessary part of the bill anyway. However, I admit that I could just lack imagination on a better solution.

Features I'd like in PostgreSQL

Submitted by gil on

I’ve put in my fair share of time with PostgreSQL. It has no shortage of functionality and I believe its expansive feature set drove a lot of its growing popularity in the 2010s. As my time with the database has worn on I’ve come up with a handful of features that I wish the database had. Some of these are ideas borrowed from other systems, some are inventions of my own. The real tragedy is that I simply don’t have the time to implement the patches myself.

–i-am-a-dummy mode

This idea comes from MySQL whose stock CLI client supports an –i-am-a-dummy flag. When enabled, MySQL identifies any UPDATE or DELETE that lacks a WHERE clause and fails them with an error. In other words, if you intended to DELETE FROM table1 WHERE pk = 10, started typing it into your prompt, and only got as far as DELETE FROM table1 before accidentally executing the query, the query errors out instead of having every row deleted. More modern versions of MySQL also use the flag to block queries that return excessively large result sizes, require the WHERE to reference a key column, or allow a LIMIT in place of a WHERE.

Update: User felixyz on lobste.rs points out that there is a pg_safeupdate server-side extension that blocks UPDATE/DELETE without WHERE.

Update: Andrey Borodin and Nikolay Samokhvalov of Postgres TV streamed themselves implementing this behind a GUC! They sent in the patch but it was rejected by Tom Lane.

Unit test mode (random result sorting)

The order of a query’s results is famously unspecified unless you nail it down with an ORDER BY. The flexibility is useful: sometimes you need a stable order for whatever code consumes the query results and sometimes you just don’t. But there is a secret, third thing: some code comes to accidentally depend on a coincidental ordering of the results. To help flush these out it would be neat if PostgreSQL could intentionally shuffle the order of query results if a given query did not specify an ORDER BY. I imagine this as a run-time configuration field for the database that you’d turn on when running automated tests or in a testing environment. The alternative, folding every query in your application into a subquery that randomly sorts the results, is ugly and invasive, although perhaps some ORMs could implement the feature at their level.

Update: User avianlyric on Hacker News pointed out that CLUSTER can can be used to reorder results in a pinch.

Query progress in psql

This is another feature from MariaDB. It’s possible in both databases to query the database to get progress information on several types of queries, but the ALTER TABLE, CREATE INDEX, DROP INDEX and LOAD DATA INFILE queries also automatically inform the CLI client on their progress. The CLI uses this to draw a nice animated status bar with some informational messages. PostgreSQL lets you query the system catalog for the progress of several query types but has no live update functionality in the psql client.

Pandas-like join validation

I am a true database expert who has never screwed up a join before. But there are many beginners out there and it would be neat to extend the SQL language to support validation features like Pandas has for its join function. When doing a join you’d specify what relation you expect to have unique keys and which one has potential duplicates and PostgreSQL could warn or panic when this assertion is triggered. Some hypothetical syntax:

SELECT x, y
FROM t1
JOIN t2 USING (key) VALIDATE 1:m

Which would have PostgreSQL check that t1.key has no duplicate rows and errors if it comes across any. This is another place where it would be useful to have a runtime option to disable the expensive uniqueness checks.

Pandas’ “suffixes” argument, which automatically renames overlapping columns in the output, and “indicator”, which emits a new column identifying the left or right table or both as the source of an output row, are also neat ideas that seem really helpful for relational newbies trying to troubleshoot their joins. Pandas is insanely popular, it’s taught to a lot of very inexperienced developers, and I think features like these were earned with newbie blood. I suspect PostgreSQL’s users and developers don’t get that sort of exposure to newbies and their typical problems but there’s no reason why they can’t learn from the hard-earned experience of Pandas and incorporate some good ideas.

Update: User richbell on Hacker News brought up DuckDB's Friendlier SQL which is a superset of sensible extensions to SQL. I endorse all of these ideas except for the trailing comma fix - join the dark side and use leading commas!

JIT support for CREATE INDEX

Today, the JIT support in PostgreSQL is closely tied to query planning and execution. It rests upon the object-oriented nature of the execution plan and emits LLVM IR for each plan node as it traverses the plan tree. It’s an elegant technical approach, and for its sweet spot of long-running analytical queries it can cut down on query time and free up CPU time.

DDL statements are not JITed, and for good reason - the vast majority of them are going to be I/O bound and not particularly interesting or useful to JIT. However, this is one exception: CREATE INDEX. Even though it does a lot of I/O (typically scanning the entire table) in my experience it is CPU-bound. Even moderately sized tables of tens or hundreds of gigabytes can take 10-30 minutes to index. And sometimes you just have to reindex everything, at once, like when you restore a backup with pg_restore. JIT compilation of the index creation code would be a huge and immediate benefit to just about every PostgreSQL user. And it would be a benefit: the PGCon 2017 presentation “JIT-Compiling SQL Queries in PostgreSQL Using LLVM“ which first explored JIT compilation found up to a 19% improvement in index creation time.

Reduce the memory usage of prepared queries

Like many databases, PostgreSQL has a PREPARE statement that takes an entire query with placeholder values and stores its query plan for future re-use. You can later invoke the stored query with EXECUTE, passing values in for the placeholders, and take advantage of all the precomputed query planning work. If your query is executed more than once, even if only a few times, this winds up being a measurable win. The database feature has been around for years and years and years, but over on the Python driver side, we’re finally starting to see database drivers automatically preparing and managing prepared queries for you (see psycopg3 and asyncpg). The rollout of these drivers is slow, even if they are production-ready, but over time we are only going to see more and more inadvertent users of prepared statements.

On the server side, prepared statements are per connection. They don’t take up a ton of memory, especially on the sort of beefy machine you’d use as a database host, but they take up more than you might expect them to do, to the point where it made sense for me to deallocate prepared plans when I knew they wouldn’t be used anymore. The per connection scope means that there’s no way to share query plans across connections. My first feature I’d like to see around PREPARE is a way to share prepared statements in some sort of global cache. A typical OLTP database is going to have a handful of long-lived connections, behind some sort of bouncer, and some number of hot queries, whether they are determined automatically by a driver or explicitly by the application developer. A global cache could easily be hydrated when a long-lived connection starts up or as queries are issued and easily gets some performance wins.

This issue was reported in 2019 by Daniel Migowski who provides some statistics. He happens to be working with some decently large queries which turn out to have 30-45 MB stored query plans, totalling 7 GB of RAM over his dataset of around 250 queries. It can get ugly! Under the hood, the PREPARE statement stores everything in struct CachedPlanSource, which includes the original query’s text, the parse tree for the query, the query tree for the query, and possibly the plan tree if the database decides to re-use a generic plan for each execution. All of this is potentially needed to execute a prepared statement in the future.

But I’d also like to draw your attention to an email by Tom Lane later in the thread where he suggests some techniques to reduce the memory usage of the individual query plan. In particular, one of the suggestions is:

A totally different idea is to make a variant version of copyObject that is intended to produce a compact form of a node tree, and does not create a separate palloc allocation for each node but just packs them as tightly as it can in larger palloc chunks.  [...] The stumbling block here is that nobody is gonna tolerate maintaining two versions of copyfuncs.c, so you'd have to find a way [...] to autogenerate the copy functions from annotated struct definitions; people have muttered about that for years but not done anything.

Some background on PostgreSQL’s memory management circus: there are a lot of tree data structures in the backend of the database. Originally, when the database was written in Lisp, the trees and their nodes were what I’d broadly call object-oriented giving them standard interfaces, constructors and destructors. When porting to C they created this copyObject function that can traverse trees, inspect each node and do the appropriate work to copy it. I’d still call it object-oriented but it is definitely C-flavored OO. This high-level interface over trees is the first of three legs of memory management in PostgreSQL.

The second leg is memory contexts. Each context has a malloc-like interface (palloc, as mentioned by Tom before) that allocates within a memory context. All memory contexts are organized into a hierarchy of contexts, and there is always a current context used by palloc() that gets set by the backend as it moves through the different stages of query execution. By scoping allocations with the same lifetime you reduce memory fragmentation but also make cleanup easier - when the user is finished with a query or it aborts for whatever error, you can tear down the relevant parent memory context and have all of its children recursively cleaned up.

The memory context API is also object-oriented in that it is a standard interface implemented by several sub-allocators. They make up third leg of memory management in the database. The most frequently used one is AllocSet, an arena allocator. This is the actual code that calls malloc() and free() to get whole free pages from the system. AllocSet maintains its own freelist to avoid too much round-tripping between PostgreSQL and the system allocator and groups smaller allocations of the same size onto the same page to reduce fragmentation.

There’s a lot of flexibility in this system but there’s also a decent amount of overhead as each step is allocating one or more structs for its own bookkeeping data. Avoiding this overhead is what Tom is proposing: there could be a special copyObject that figures out the full size of the tree it has to copy, makes a single allocation, and packs everything into it, avoiding most of the overhead from all the small data structures needed for the prepared query. This is the second feature I’d like to see attempted to save PREPARE memory usage.

An entirely new allocator?

AllocSet is a mature battle-tested, and pretty simple piece of code which has sat at the heart of PostgreSQL since 1999 with little change to its design since then. There is something to be said for the longevity of such a fundamental piece of code and I would not be surprised if any alternate implementations can’t beat its performance just because of its simplicity. I’d like to see someone try to slot one of the modern malloc implementations (e.g. jemalloc, mimalloc, tcmalloc) into PostgreSQL. Even if it isn’t a win I’d like to see the results. Modern implementations take advantage of everything we’ve learned about writing fast memory allocation on modern hardware and operating systems and have seen lots of real-world testing. I also don’t think the palloc()/memory context API needs to change at all, modern allocators typically have an arena allocator interface and could be slotted in to replace the memory context API without API breakage. Some concerns would be legal licensing or performance losses from synchronization (the PostgreSQL backend is single-threaded and thus any effort spent on locking or synchronization is wasted) but I am hopeful that there is a malloc that can fit the bill.

Batching query plans for joins

In the terminology of the PostgreSQL query planner, when you join two tables, the database picks one table to be the outer relation and one to be the inner relation. The outer relation is the table that gets scanned first and the matching rows it emits are checked one at a time against the inner relation to get the final, joined result row(s).

Most uses of a foreign data wrapper have you talking to another database system over a network. With that comes overhead from using the network and some per-query overhead in the foreign database. It would be bad to repeatedly query the foreign database in an inner loop as you'd pay this overhead each time. Unfortunately, if the database picks your foreign data wrapper to be the inner relation in a join you get one query for each row emitted by the upper relation and pay that overhead each time.

What I'd like to see is an execution technique (in PostgreSQL terminology, a path or upper relation) that is capable of batching rows from the outer relation, sending them as one query to the inner, and sorting out the results. I have a hunch this would be a win for typical FDW use cases but I have not tested it. This would be useful for the built-in PostgreSQL-to-PostgreSQL FDW (fdw_postgres) which has to emit one query per outer relation row but once implemented the same path could be used in other wrappers. It would be a useful addition to the core database but it's actually possible to implement your own paths in an extension via the little-known custom scan path API.

CIA Reorganization

Submitted by gil on

Ed. note - This is the transcription of a memo written to President Kennedy that explores the CIA's bureaucracy, explains its cultural problems, and suggests organizational reforms. It is dated June 30, 1961, not long after the infamous Bay of Pigs invasion in April of 1961. The author, Arthur M. Schlesinger Jr., was a "roving reporter" who reported directly to the President. This is a candid and well-written memo. Given Schlesinger's history with the CIA and his broad access to the entire executive branch I believe he paints an accurate picture.

Ultimately, the reforms suggested by Schlesinger did not happen. There was no dismemberment of the CIA's operational responsibilities, there appears to be no subjugation of the CIA to the State Department, and the Reagan-era creation of the Intelligence Community centralized intelligence coordination under the CIA instead of a pluralistic Joint Intelligence Board.


Memorandum for the President

Subject: CIA Reorganization

I submit the following views as one who worked in OSS during the war and served as a periodic CIA consultant in the years since.

On balance, CIA's record has probably been very good. In the nature of clandestine operations the triumphs of an intelligence agency are unknown[1], all the public hears about (or should hear about) are its errors. But, again in the nature of the case, an agency dedicated to clandestine activity can afford damned few visible errors. The important thing to recognize today, in my judgment, is that the CIA, as at present named and constituted, has about used up its quota. Its margin for future error is practically non-existent. One more CIA debacle will shake faith considerably in US policy, at home as well as abroad. And, until CIA is visibly reorganized, it will (as in the Algerian instance) be widely blamed for developments of which it is wholly innocent.

The argument of this memorandum is that CIA's trouble can be traced to the autonomy with which the agency has been permitted to operate and that this autonomy is due to three main causes: (1) an inadequate doctrine of clandestine operations; (2) an inadequate conception of the relationship between operations and policy; (3) an inadequate conception of the relationship between operations and intelligence. The memorandum also suggests ways in which some of these problems can perhaps be alleviated.

I. CIA Autonomy

CIA conduct three main forms of secret work: clandestine intelligence collection; covert political operations; and paramilitary activities. It carries on these functions with relative autonomy. The reasons for the autonomy are historical.

When CIA began, the State Department, still thinking too much in terms of its traditional missions in foreign affairs, looked on this new venture with suspicion and renounced the opportunity to seize firm control of CIA operations. It did not, for example, try to establish any effective system of clearance for CIA activities; and some ambassadors frankly preferred not to know what CIA was up to in their countries. After 1953 the fact that the Secretary of State and the Director of Central Intelligence were brothers further confirmed CIA's independence from supervision by State Department desks.

In the meantime, CIA grew in size and power. During the fifties it began in some areas to outstrip the State Department in the quality of its personnel. Partly because CIA paid higher salaries and even more perhaps because Allen Dulles gave his people courageous protection against McCarthyite attacks, CIA was able to attract and hold a large number of able and independent-minded men. The more it added brilliant activists to its staff, the more momentum its operation developed, and the greater role it played in the initiation of policy.

By the time that State had begun to be fully aware of the problems created by an active and autonomous CIA, it had long since missed the chance to establish its own ultimate authority. While State retained a nominal supervision over covert political operations, this was to some degree offset by CIA's tendency to present a proposed operation almost as a fait accompli; State never had title to control overt clandestine intelligence collection; and it had lost ground in such areas as overt political reporting and even in the maintenance of overt diplomatic contacts.

For its part, CIA had developed a whole series of functions paralleling already existing function of the State Department, and of the Defense Department as well. Today it has its own political desks and military staffs; it has in effect its own foreign service; it has (or had) its own combat forces; it even has its own air force. Its annual budget is about __ times that of the State Department. The contemporary CIA possesses many of the characteristics of a state within a state.

II. Doctrine

Though CIA's autonomy developed for historical reasons, it has been able to endure because there is no doctrine governing our conduct of clandestine operations. The problem of doctrine for CIA is the extent to which its various clandestine mission are compatible with a free and open society.

It is idle to argue that, because the Communists can do such-and-such, we are free to do it too. Communism ie a creed nurtured in conspiracy; and the whole point of Communist social and political organization is to make conspiracy effective. If 'fighting fire with fire' means contracting the freedoms traditionally enjoyed by Americans in order to give more freedom to CIA, no one seriously wishes to do that. Yet I do not feel that we have tried rigorously to think through the limits which the maintenance of an open society places on secret activity. Until this is done, CIA'a role will not be clearly defined and understood. The problem which must be faced is: what sort of secret activity is consistent with the preservation of a free social order?

We must begin, I believe, by accepting the fact that the United States will continue to be a nation in which politicians will ask questions and make speeches, reporters will dig out stories, newspapers will publish editorials, individuals, driven by promptings of conscience, will blurt out things harmful to the state, and so on. We do not wish to change these things and could not do so without violating the essence of our society. These things make up the framework in which CIA must operate. In short, they constitute the problem; and, as General Marshall used to say, "There's no point in fighting the problem."

There follows from [unreadable], I would think, the conclusion that secret activities are permissible so long as they do not corrupt the principles and practices of our society, and that they cease to be permissible when their effect is to corrupt those principles and practices.

Each form of secret activity presents its own problem. Clandestine intelligence collection has been a traditional function of the national state. Its rules and usages are fairly well established. In the main, espionage, if conducted with discretion, should present no great problem. However, when conducted with an open checkbook and a broad mandate even espionage can begin to push against the limits of secret activity in an open society. I consider later, for example, the question of the Controlled American Source (CAS) and whether the overdoing of CAS is not beginning to harm other [unreadable] of the government.

Covert political operations present a trickier problem. Occasions arise when it is necessary to subsidize newspapers, politicians and organizations in other countries. But corruption of the political life of another nation is not a responsibility to be lightly assumed. I wonder whether CIA has not done too much of this for the joy of it. Paramilitary activities create an even more difficult problem, if only because the problems of concealment are so much more difficult.

In general, I would suggest that any secret operation whose success is dependent on the suppression of news, on lying to Congressmen and journalists and on the deception of the electorate should be undertaken only when the crisis is so considerable that the gains really seem to outweigh the disadvantages. This suggests that the size of the proposed operation may be of crucial importance. Small operations can be done with a minimum of accompanying corruption. The greater the visibility of the operation, and more its success depends on thwarting the impulses and denying the values of an open society, the riskier it becomes, and the more urgent it is that an overwhelming case be made for its necessity.

In short, when confronted with a proposed operation, we must not only ask whether the operation is technically feasible in its own terms; we must ask whether its success requires our open society to be [unreadable] serious way false to its own principles.

III. Operations and Policy

Clandestine activities should be assessed not only to the context of their relationship to an open society but also in the context of their relationship to United States foreign policy.

CIA operations have not been held effectively subordinate to U.S. foreign policy.

1. Clandestine intelligence collection is, by charter, free from State Department control. This fact exposes American foreign policy to a multitude of embarrassments when CIA is discovered recruiting agents or developing sources in a friendly country. The recent Singapore case, when CIA, without notice to the Consul General, tried to subvert a member of the Special Branch of the Singapore Police provides an instructive example. After complications of ludicrous complexity, including an attempt to subject the recruit to a lie-detector test, it turned out that the recruit had long since informed his superiors of the CIA approach and was leading CIA into a trap, which, when sprung, produced considerable embarrassment to relations between the US and the Singapore government. There have been troubles of a comparable sort in Pakistan and in Japan (where a group of Chinese Nationals were smuggled into Sasebo[?] to work in a CIA-NSA operation).

CIA has said that, in such cases, neither the Embassy nor the Department in Washington is normally informed of this type of operation. In short, no one knows how many potential problems for US foreign policy -- and how much potential friction with friendly states -- are being created at this moment by CIA clandestine intelligence operations. Surely there is an argument for permitting State to decide whether the advantage to be gained by the operation (e.g. the information derivable from an agent within the Special Branch of the Singapore Police) outweighs the risk (e.g. exasperating the local government and shaking its confidence both in our purposes and in our sense). Clandestine intelligence operations should plainly be cleared both with the Department of State (and save for exceptional instances and on agreement of the Secretary of State) with the local U.S. Ambassador.

2. Covert political operations technically require State Department clearance. In practice, however, CIA has often been able to seize the initiative in ways which reduce State's role almost to that of a rubber stamp.

This has been partly the consequence of the superior drive and activism of CIA personnel, especially as compared with the diffidence of State Department personnel. For example, when men come to CIA with the assignment of developing covert political campaigns or organizing coups or preparing for paramilitary warfare, these men naturally fall to work with ingenuity and zeal; they probably feel that they are not earning their pay unless, say, they organize as many coups as possible. The concept of 'contingency planning' has legitimatized the concrete preparation of operations still presumably in a hypothetical stage; people did not at first notice that 'contingency planning,' when carried to the stage of recruitment and training of personnel, creates a vested interest which often transforms contingencies into apparent necessities. Thus, if a group is assembled and revved up on a contingency basis, then the failure to carry the project through (it is argued) will invite the disappointment and alienation of the group; so the pressure increases to follow through on what had started as a pure speculation. This was a central factor, of course, in the Cuban decision -- the fact that disbandment of the Cuban force in Guatemala just seemed to create too many problems and embarrassments. Having entered into relations with personalities in foreign lands, CIA has sometimes seemed to feel that we must thereafter do pretty much what they want -- or else they will blow security or even go over to the Communists. This approach has made us on occasion the prisoners of our own agents.[2]

It has meant too [unreadable] State Department, often apprised of an operation only in its later stages, is under great pressure to endorse the operation as already mounted because of the alleged evil consequences [unreadable] exercising a veto. I well remember Tom Mann's remark the day the decision was made to go ahead on Cuba: "I would never have favored imitating this operation; but, since it has gone as far as it has, I do not think we can risk calling it off." Moreover, at a time when there is increasing premium on activism, State, when it questions CIA operations or initiatives, casts itself in a prissy, sissy, negative role. The advocate of clandestine activities seems 'tough' and realistic; the opponent has to invoke such intangibles as the reputation of the United States, world public opinion, "What do we say in the United Nations?", etc., and seems hopelessly idealistic, legalistic and 'soft.'

The result of CIA's initiative in covert political operations has been to create situations which have forced policy on the State Department. This was not the original idea behind CIA. As Allen Dulles wrote in his 1947 memorandum to the Senate Armed Services Committee, "The Central Intelligence Agency should have nothing to do with policy." Yet, in the years since, CIA has, in effect, 'made' policy in many parts of the world. A number of governments still in power know that they even been targets of CIA attempts at overthrow -- not a state of mind calculated to stimulate friendly feelings towards the United States. Indonesia, of course, is a prime example.

This experience suggests that the present system by which CIA notifies State of a projected covert operation is inadequate to protect US interests. There must be some means by which State can be informed of such operations at an early enough stage to affect the conception and preliminary planning of the operation. Otherwise CIA will continue to confront State with propositions having potential impact on foreign policy but at too late a point to subject that impact to reasonable control.

3. The Controlled American Source (CAS) represents a particular aspect of CIA's encroachment on policy-making functions. [The rest of the section is redacted.]

4. Paramilitary warfare, I gather, is regarded in some quarters as a purely technical matter, easily detachable from policy and therefore a proper function of the Department of Defense. Yet there is almost no CIA function more peculiarly dependent on the political context than paramilitary warfare.

There are several reasons for this. For one thing, a paramilitary operation is in its nature a large and attributable operation and thereby, as suggested above, clashes with with the presupposition of our open society. (These considerations need not apply, however, to the training of, say the South Vietnamese in guerrilla tactics or to the support of already existing guerrilla activities.) For another, the moral and political price of direct paramilitary failure is acute for us. Communists, when they stimulate paramilitary activity, are doing what the world expects from them; when we do it we appear to betray our own professed principles and therefore cannot afford to compound delinquency by defeat. Moreover, as the recent Algerian episode showed, once we convince the world that ws are committed to a paramilitary endeavor, we will be blamed for all sorts of things. And, as the recent tractors-for-prisoners episode showed, when we do send men to possible death, we cannot lightly write them off and close the books. The Communists, on the other hand, have no scruples about liquidating a losing show.

Moreover, guerrilla warfare is fought, much more than military men ordinarily realize, in an ideological context. The guerrilla succeeds when his program enlists grassroots support; and it is very difficult indeed to beat him when the countryside is with him. Van Mook told General Marshall in 1946 not to worry about the guerrillas in Indonesia; "we are sending 50,000 men out there and will clean the situation up in the three months." Marshall, who had had experience fighting guerrillas in the Philippines, replied "It won't be so easy. You will find that they will bleed you to death." Cyprus and Algeria illustrate the difficulty of overcoming guerrilla resistance through sheer weight of military force.

Serious guerrilla movements hare been defeated only three times since the end of the Second World War: in Greece, because Tito's defection closed the northern border; in the Philippines; and in Malaya. In the last two cases the guerrilla resistance ended because of the combination of political and military countermeasures. The struggle in Malaya, as Field Marshal Templer understood (and said), was for the minds and hearts of the people; it therefore involved not only paramilitary operations but a vast educational program, organization of trade unions, political reform and an offer of national independence. If the guerrilla's power lies in his revolutionary program, the answer lies In part in meeting the needs which enable the guerrilla to rouse the countryside. "Without a political goal," wrote Mao Tse-tung, "guerrilla warfare must fail, as it must if its political objectives do not coincide with the aspirations of the people and if their sympathy, cooperation and assistance cannot be gained." He added, "Do we want the support of the masses? If we do, we must go among the masses; arouse them to activity; concern ourselves with their [unreadable] and woe."

For these reasons, paramilitary warfare cannot be considered as primarily a military weapon. It is primarily a political weapon and must therefore be subjected to close and careful political oversight. It probably should be retained in a reconstituted CIA rather than transferred to Defense. 

5. How to Establish Policy Control? Here I wonder whether the British experience might not be of value. The notable feature of the British Intelligence system is the determination to keep clandestine activity under strict Foreign Office control. This control is achieved in a number of ways: 

a) Secret Intelligence Service (MI-6) itself operates under the direction of the Joint Intelligence Committee, which has a Foreign Office chairman (until recently Sir Patrick Dean, who is now the British representative at the UN) and which includes the Service Intelligence directors and representatives of the Colonial Office and the Commonwealth Relations Office as as the Chief of [unreadable].

b) basic political warfare directives are originated, not by SIS, but by the Information Research Department in the Foreign Office, often in consultation with an interdepartmental Working Group on International Communist Fronts, and under the ultimate control of the Superintending Under-Secretary of the Permanent Under-Secretary's Department in the Foreign Office.

c) SIS covert political action campaigns must not only conform to Foreign Office directives but must be cleared with the appropriate Foreign Office geographical desks.

d) working groups under IRD chairmanship govern operations in special areas, such as, for example, Sino-Soviet relations or the World Youth Festival. 

e) a Foreign Office Staff Liaison Officer sits next to the SIS Chief, and Foreign Office officials serve tours of duty in SIS sections.

All these devices might be adapted for use by the State Department. Organizationally, this would mean that the intelligence agency would retain operational autonomy but that its operations would be at all points subject to State Department clearance -- a clearance to enforced by directive, by State Department chairmanship of working groups, and by the infiltration of State Department personnel into the intelligence agency.

Obviously this set-up would not succeed, however, unless the State Department itself were prepared to overcome its inbred habits of diffusion, negativism and delay and to take a firm and purposeful grip on the situation.

IV. Operations and intelligence

The relationship between operations and intelligence raises particularly perplexing questions. 'Intelligence' includes two separable activities: clandestine collection; and research, analysis and evaluation. The first deals with that small portion of raw intelligence procured by secret means (Mr. Dulles estimated in 1947 that "a proper analysis of the intelligence obtainable by ... overt, normal and aboveboard means would supply us with over 80 percent, I should estimate, of the information required for the guidance of our national policy"); the second involves the collation and interpretation of all forms of intelligence, however obtained and the production of estimates.

Under the British system, clandestine collection is entrusted to the Secret Intelligence Service. The research, analysis and estimating function is located in the Foreign Office Research Department.

Under the American system, CIA has responsibility for both clandestine collection and research and analysis. The State Department's Bureau of Intelligence and Research also has research and analysis responsibilities; but, in practice, CIA has established control over the machinery for producing national estimates in such a way as to reduce State's contribution to submitting its view to CIA for acceptance or rejection. In other words, where in Great Britain the Foreign Office plays the coordinating role in the intelligence field, in the United States that role has been assumed by CIA.

The argument against incorporating the research and estimate function in State was made by Dulles in 1947: "For the proper judging of the situation in any foreign country it is important that information should be processed by an agency whose duty it is to weigh facts, and to draw conclusions from those facts, without having either the facts or the conclusions warped by the inevitable and even proper prejudices of the men whose duty it is to determine policy and who, having once determined a policy, are too likely to be blind to any facts which might tend to prove the policy to be faulty."

Precisely the came argument can be used with equal effect against the incorporation of the research and estimate function in CIA -- i.e., if intelligence is too closely connected with operations, then those committed to a particular operation will tend to select out the intelligence which validates the operation.

Obviously both arguments conclude in a case for the establishment of a fairly independent research and estimate group. But, if the R & E group is too independent, one runs into the opposite danger: that is, that neither policy nor operations will be subjected to adequate intelligence checks.

The trouble with the Cuban operation, for example, was not that intelligence and operations were combined, but precisely that the Cuban operation evaded systematic intelligence judgment. The intelligence branch ([DDI?]) of CIA was never informed of the existence of the Cuban operation. The Office of National Estimates was never asked to comment on the assumption, for example, that discontent had reached the point in Cuba where a successful landing operation would provoke uprisings behind the lines and defections from the Militia. In December and February, the Office of National Estimates produced general appraisals of the Cuban situation, but these were holly independent of the Cuban operation.

I gather that, if its opinion had been invited, [DDI?] would have given quite a different estimate o t the state of opinion in Cuba from that on which the operation was based. There existed, in short, the ridiculous situation that knowledge of the Cuban operation, flaunted in Miami bars by any number of low-level agents in the operations branch of CIA, was denied to even the top officials of the intelligence branch. The Bureau of Intelligence and Research of the Department of State knew even less about the Cuban operation.

The problem, then, is to have an R & E group sufficiently independent of both policy and operations to resist the pressure to make the case for vested ideas and interests -- yet sufficiently close to be able to subject projected operations or policies to the most intense and searching scrutiny.

Where could this group be located? If the CIA were to become subordinate to State, as MI-6 is to the Foreign Office, then the R & E function might be vested in a coordinate subagency, somewhat independent of both State and CIA, yet closely connected with both in day to-day operations. The R & E subagency would receive intelligence from CIA and from State, as well as from the services and of course from public sources. It would represent, in effect, a fusion of CIA/DDI and State/ Intelligence and Research. It might also take over certain of the service functions now confided to CIA -- photographic interpretation, biographical data, foreign broadcast monitoring, overt collection, maps, etc. There might be in addition a Joint Intelligence Board with representatives from all the intelligence agencies and with a State Department or White House chairman.

V. Conclusion

The argument of this memorandum implies a fairly drastic rearrangement of our present intelligence set-up. It also implies the capacity of the State Department to assume command of the situation and to do so in an effective and purposeful way. If the State Department as at present staffed is not capable of assuming effective command, this is not, in my judgment, an argument against a rational reorganization of intelligence. It is an argument for a drastic overhaul of the State Department.

 The structure which would meet the criteria suggested in this memorandum would be as follows:
 
 1) The State Department would be granted general clearance authority over all clandestine activity. This might be effectuated on the British model by the appointment of a Deputy Undersecretary of State for Intelligence, who would act for the Secretary in these matters and who might serve as chairman of a Joint Intelligence Board.
 
 2) The Joint Intelligence Board would include representatives from all elements in the intelligence community and also from the White House.
 
 3) The operating branches of the present CIA would be reconstituted under some blameless title (the National Information Service). This new agency would be charged with responsibility for clandestine collection, for covert political operations and for paramilitary activities. It would submit projects to the Deputy Undersecretary of State for Intelligence for clearance. In general, the agency would bear somewhat the same relationship to State as the Disarmament Administration and ICA presently do.
 
 4) A second semi-independent agency would be set up, again bearing a blameless title (the Foreign Research Agency), coordinate with the operations agency. This agency would be charged with responsibility for collation and interpretation. It would include CIA/DDI, the Bureau of Intelligence and Research in State, and the various service functions now carried on by CIA (photographic interpretation, biographical files, foreign broadcast monitoring, scientific intelligence, maps, overt collection, etc.). It might well be located in the CIA building in McLean.

 Arthur Schlesinger, Jr.


[1] Or should be -- a gross and repeated CIA failing has been its occasional readiness to succumb to the temptations of favorable publicity. The Guatemalan and Iranian operations were almost nullified by the flood of self-congratulatory publicity which followed them (e.g., the articles by Richard and Gladys Harkness, "The Mysterious Doings of CIA," Saturday Evening Post, Oct. 30, NN. 6, 13, 1954).

[2] We become prisoners of our agents in another sense too. The Cuban episode leaves the strong impression that CIA is not able to control its own low-level operatives. While the CIA people in Washington are men of exceptionally high quality, the men attracted to field jobs are sometimes tough and even vicious people motivated by drives of their own and not necessarily in political or even moral sympathy with the purposes of the operation. Such actions as locking up the Revolutionary Council, such enterprises as Operation 40 in Miami where CIA agents reportedly trained Cubans in methods of torture, such episodes as the detention and third degree reportedly administered to Dr. Rodolfo Nodal Tarafa and 16 other democratic Cubans this February - all these suggest that things go on under CIA sponsorship which CIA in Washington is only dimly acquainted.

How do I make our data warehouse OSHA compliant?

Submitted by gil on

Ed. note: the cowards at the DBA Stackexchange deleted this post. It has been re-posted here for posterity.

How do I make our data warehouse OSHA compliant?

What are the safety techniques that we should follow in our data warehouse to guarantee OSHA compliance? The existing literature isn't very helpful. I bought yellow high visibility vests for all of my DBAs but they're refusing to put them on. - asked Apr 1, 2020 at 12:01


  • As with most questions on this network, you are encouraged to start with the official docs and update your post with specific regulations that concern you. Your General Counsel and HR should be made aware of any updates to your safety policy as well as any merges to the master branch. – Peter Vandivier Apr 1, 2020 at 12:20
  • Did you try switching the hi-vis vests to the dark mode? Many IT professionals prefer that for mental health reasons. – mustaccio Apr 1, 2020 at 12:56

I bought yellow high visibility vests for all of my DBAs but they're refusing to put them on.

Joint and severable liability of employer and employee for OSHA compliance is a longstanding dialogue1. For the time being, you are limited to internal HR procedures to discipline your employees for infractions.

Assuming your management troubles are sorted though, here's a few specific items you may find helpful to consider for workplace safety in your data warehouse

  • Employees should not lift more than 50 GB (48 GiB in Puerto Rico and USVI) of data unassisted. Those on reduced work orders are limited to 500 MB (450 MiB)
  • High-bandwidth cabling should be labelled and insulated. Any network pipe of capacity >1GB/sec must not be serviced unless two or more technicians are present.
  • Clearly labelled emergency stops should be easily accessible at all partition switching locations.

Stay safe out there.

– answered Apr 1, 2020 at 12:18 Peter Vandivier

Title research in Steele Creek, Mecklenburg County, North Carolina

Submitted by gil on

If you've ever purchased real estate you may remember the deed: a legal document describing the property and declaring that the grantor, the old owner of the property, is giving it to the new owner, the grantee. A copy of the deed is usually filed with the county. This leaves a chain of deeds on file going all the way back to whatever was the original grant for the property. In the original thirteen colonies of the United States this chain goes all the way back to the original grants from the English monarch.

Tracey and I recently purchased a home in Mecklenburg County, North Carolina whose register of deeds maintains a searchable database of deeds filed since 1990. But there are still more deeds: the county has records going back to 1763. Incredibly, the county has scanned in all of the old deeds as well and made both databases available for free! The old deeds don't have a text search, which is unfortunate but not surprising - deeds before 1920 or so are hand written in cursive. So my mission is now clear - can I trace the title of this land all the way back to the original grants from the Lords Proprietors of Carolina?

The county organizes its deeds into books, and the deed is indexed on its page in the book. These are written out like 36356-796, where the first number is the book number and the second is the page number. Yes, there are a lot of books - they've been counting up since 1763! That deed is the start of my search as it's the deed that transferred ownership to Tracey and myself.

Deeds are pretty formulaic. They are practical instruments, meant to cleanly transfer ownership of property to the grantee. In the original thirteen colonies of the United States they often use the traditional English legal practice of metes and bounds to describe the property as if you had walked around its boundary. The most recent deeds simply reference a specific lot on an official map, also filed with the county, instead of a metes and bounds description of the property, but once we get past the immediate subdivision history of the land we'll see metes and bounds used exclusively. The metes and bounds also put the boundary line in context with the property around it - for example, when a boundary runs along a public road or highway the road will be mentioned by name. This helps with locating the property on a modern map as many of our Mecklenburg County roads have not changed names or locations in over a hundred years. However, there is still some sleuthing required when names change, or when the deed only references nearby properties by owner's name at the time of sale, such as in 3032-203:

BEGINNING at an old iron, which iron is also the common northeast corner of Mrs. R. C. Mitchell and the southeast corner of the property of A. A. Porter and J. F. Neely estate, thence with the line of A. A. Porter and J. F. Neely estate North 31-59 West 376.62 feet to an old iron; thence with the line of Duke Power Company North 22-49 East 868.34 feet to an old iron located on the line of the property of Gambrell Investment Company (formerly Marshall Reid); thence with said line of Gambrell Investment Company South 20-47 East 1,834.59 feet to an iron located on the line of Mrs. L. E. Carothers, and which iron is also located South 66-17 West 356.06 feet from an old iron; thence with the line of Mrs. L. E. Carothers South 66-17 West 627 feet to an iron; thence continuing with the line of Mrs. L. E. Carothers, the line of the 1.11-acre tract conveyed by Mrs. Helen C. Carothers (Mrs. L. E. Carothers) to Gambrell Investment Company, and the line of Mrs. R. C. Mitchell North 14-24 West for a total of 874.03 feet to the point or place of BEGINNING, containing 20.33 acres as shown on a survey prepared by E. L. Faires, North Carolina Registered Surveyor, dated September, 1968. BEING the same property conveyed to Robert O. Brown and Wife, Doris H. Brown, by deed of Nancy Porter Kirby (Widow), and Clara Sledge Woods (Widow), dated __ ___, 1967, and recorded in Book 2881, at Page 442, of the Mecklenburg Public Registry.

I know that the deed is in Steele Creek township in Mecklenburg County as it is written elsewhere on the deed. The names are also familiar - the Carothers and Neelys come up many times in Steele Creek as I've done the title research. But where is this land actually located? There's no mention of county roads, postal addresses (although deeds never mention postal addresses) or natural features like creek crossings or confluences. There is no reference to any map (although sometimes one does get filed with the county) and the survey mentioned at the end is not filed with the county. At least it mentions the prior deed and I can look up that one in hopes it has more detail. But if that can't be found you have to find the neighboring properties mentioned by owner and hope you can triangulate the location from their context.

The "irons" in the description are metal stakes left in the ground by surveyors to designate the corners of the property. Yes, this comes with all the issues of accidentally or intentionally moved or misplaced markers. As you go back in time you start seeing trees, tree stumps, or the course of creeks and rivers in these descriptions making it more and more challenging to accurately draw a map of the boundaries of the property. The property may be formally specified and records kept for hundreds of years but the passing of time makes it a challenge to recover the actual property boundaries.

In a fascinating quirk of legal history in the US the complete history of the title has to be computed and double-checked to make sure that the land can be successfully sold. Should there be any problems in the past with the ownership or sale of the land you risk having older owners, their heirs, or an entire tribe of Indians sue to regain their rightful ownership over the land. This research is done by a title insurance company before they sell you a title insurance policy which pays out should you be subject to one of these lawsuits and lose ownership of your land.

The title search

Starting with the latest deed at 36356-796 the most recent history of the land is easy to trace. Each deed mentions the previous deed by book and page number making it easy to jump to the previous deed. In the year 2004 we reach deed 17821-200 where D. R. Horton, the homebuilder, sells the newly constructed home to its initial owners. A few months before that is deed 17203-933 where Provident Development Group sells the house to D. R. Horton. I am not sure what Provident Development Group is or does but I suspect they are closely related to D. R. Horton and heavily involved in the initial construction of the Huntington Forest subdivision.

In April of 2002 the Provident Development Group acquires the land from the Steele Creek (1997) Limited Partnership in deed 13509-753. This is the first time Sarah Belk Gambrell comes up in the title history. Sarah is a partner in the the partnership and mentioned by name in the deed. We'll also find that she's a major land owner in Steele Creek. But what is her story?

If you're from Charlotte the name Belk will jump out at you. It's the same Belk as the chain of department stores (think Macy's, but southern) and the John Belk Freeway in Uptown. The department store was founded by William Henry Belk in 1888 and his son John Belk was a former mayor of Charlotte. Sarah Belk was William's daughter and passed away in 2020 at the age of 102. Sarah worked in the family business, served as an officer in the corporation, held board seats and was a major shareholder. Belk was bought out for $3 billion dollars in 2015 and Sarah had a lifetime of salary, dividends and sales from the business. Her obituary mentions the extensive philanthropic support she gave to minority interests in Charlotte. But with that money she was also heavily invested in Charlotte itself as a substantial land owner, speculating on the growth and future development of Steele Creek.

The 13509-753 deed was Sarah's sale of part of her holdings in Steele Creek to create the Huntington Forest subdivision and some other nearby properties. However, at this point the deed trail runs cold. This deed does not mention the book and page of any prior deeds like we've seen so far. Luckily, this transaction is in the post-1990 online database which allows us to search on grantor/grantee name and try and figure out what happened here. I am not sure why this deed didn't mention the previous deed but there are a couple of clues. First, there is a lawsuit between the Steele Creek Limited Partnership and the city of Charlotte to acquire land for a sewer pipe. This lawsuit ends in default judgment a year later in favor of Charlotte, awarding them title to a narrow strip of land for the sewer. There's also some deeds giving the NC Department of Transportation extra room to widen the nearby NC 160 state highway and improve the NC 160 and NC 49 intersection. The existence of the lawsuit sounds dramatic - did the city have to FIGHT to get the land from Sarah? - but my understanding is that it is a lot more mundane than it sounds. The lawsuit is simply the legal process required for the city to acquire title or an easement for this utility work. Because it is a lawsuit, run by the courts, following their due process, it gives the land owner a fair chance to defend their ownership of the land. And because the trial expired in default judgment I believe both parties were in agreement from the get-go and a lawsuit with default judgment is the simplest and cheapest way to get it squared off. Sarah Belk Gambrell is a multi-millionaire practicing sophisticated land speculation, I don't think her lawyers would let a contentious lawsuit go unanswered. However, I also haven't dug into the trial beyond the court's documents ordered to be included in the deed book.

The title search also turns up deed 5399-284, from 1986, which is an antecedent to the prior deed. This is a transaction between a different Gambrell Investment Corporation to Sarah Belk Gambrell herself. I call it the mega deed: it does not feature the metes and bounds of any property, but instead references 38 different deeds, acquired between 1961 and 1973, and lists just the book and page number of each deed. Unfortunately there is no map here or any hints - I need to go through each one and figure out if Huntington Forest is in any of them.

As I dug into this I found that Sarah owned property close to what is now the intersection of I-485 and NC-160 on both sides of the interstate and also large parts of property along NC-160 south of that intersection. She owns the Huntington Forest area but also land around the NC-160 and NC-49 intersection, including what is now the RiverGate shopping mall in its southeastern corner. It took several hours, going deed by deed to figure out which ones were clearly not contributors to Huntington Forest and which ones were possible ancestors.

Modern map of Huntington Forest and the NC-49 and NC-160 intersection.
Modern map of Huntington Forest and the NC-49 and NC-160 intersection. 

The top third or so of Huntington Forest comes from deed 2881-596, the Ewart property. It is one of a few pieces of property carved out of the J. L. Milwee estate, which very helpfully had a map filed with the county.

J. L. Millwee Estate map

Tract "C" is the source of the Ewart property and others.
The map also gives us some concrete hints about land ownership and location in this area. First, you can see the modern Steele Creek Road (NC-160) and Erwin Road in their current alignments, making it easy to situate this on a modern map. The McClintock Presbyterian Church lot is also visible on the map. It was founded in 1865 as the first Presbyterian church for the newly freed Black slaves in Mecklenburg County and the deed to that property dates to 1867. The J.M. Corouthers property is also a name that comes up in many of these property descriptions. And finally, there is the Knox brothers property right below Tract "C". It is clear that Tract "C" is the top part of what is now Huntington Forest because it's nestled in the intersection of Steele Creek Road and Erwin Road. But to figure out if our property is in the former Tract "C" or the former Knox brothers land will require us to more precisely map out exactly how far north or south the Knox brothers line is.


After digging through the rest of the mega deed we find the Knox brothers property at 2881-477. Hooray! In August of 1967 the brothers Malcolm and Hugh Knox sold their family farm to the Gambrells. The deed is composed of three tracts. Tract C is further west on NC-49/Tryon St./York Road (as it was called then) and borders the Pleasant Hill Presbyterian Church that is still standing today. Tracts A and B both reference the NC-160/NC-49 intersection so they are clearly very close to Huntington Forest. But which one, if either, is going to be the ancestor of Huntington Forest? In order to find that I had to plot out the metes and bounds for each tract on a real map. Luckily, there is a nifty and free Plat Plotter website to help you with this.


To use Plat Plotter you mark the initial point on a slippy map and then type in the metes and bounds from the deed in a structured way. The description for Tract A and B both start with:

BEGINNING at a point in the center line of North Carolina Highway No. 49 (York Road), said point beginning N. 60-41-24 E. 456.00 feet from the intersection of the center line of N. C. Highway No. 49 and the center line of N. C. Highway No. 160 (Steele Creek Road) [...]

The modern intersection of NC-49 and NC-160 is slightly to the east of the historical intersection. Originally the two highways intersected as straight lines but an improvement to the intersection in the 1990s had NC-160 curve to the east before curving back to return to its former path. You can see the "hump" where NC-160 curves on the modern map above and also see the course of the original straight line path. On the Plat Plotter website I just marked the original intersection by eye.

The starting point for the metes and bounds is not at that intersection, though: it's at a very precise "N. 60-41-24 E. 456.00 feet" from it. What does that mean? That's how the surveyors encode the path around the property. "N. 60-41-24 E." is the bearing, the direction for the next segment in the boundary starting from your current location. It starts with N or S, meaning north or south, then the 60-41-24 is the degrees, minutes and seconds of the angle from north or south. The E or W at the end indicates if your bearing is turning towards the east (or clockwise from north) or turning towards the west (counterclockwise from north). To summarize, to find our property corner and the starting point for the metes and bounds you'd start in the center of the (former) intersection of NC-160/NC-49, you'd face north, turn 60 degrees clockwise, and march forward 456 feet.

Once you have the starting corner selected on the map you have the enviable task of transcribing all the legs from the deed into Plat Plotter so it can trace the boundary. Tract B, not pictured, covers a large portion of the RiverGate shopping mall south of Tryon St. Tract A, pictured below, is our jackpot. It clearly covers the bottom 2/3rds of Huntington Forest, including the house that we purchased!

Tract A of the Knox Brothers property
Tract A of the Knox brothers property

The parts not included in Tract A are also informative. It doesn't run all the way north to NC-160, which is what we saw on the Millwee Estate Tract "C". There's a missing piece to the west where the J. M. Corouthers land would be.  The boundary with Corouthers roughly follows the creek (Walker Branch) which would make for a natural boundary between farms. And it doesn't include a long, narrow strip of Huntington Forest along its east side, corresponding with the Tract "B" strip in the Millwee map that was also not part of the Knox brothers property. Plat Plotter also gives you the area of the property in acres and the gap between start and close of the boundary and they both came in perfectly. Mission accomplished!

There is still more title research to be done, though. The Knox brothers sold in 1967. Where did they get the land? Once again, the deed is less than helpful - it does not reference any prior deeds. The trail has gone cold. To get the search started again I turned to genealogy research. Luckily, the 1950 Census was recently released to the public. It was written entirely in longhand but the government also made an attempt to OCR the text and index it. I found the Knox family in enumeration district 60-16, called Steel Creek. There are five people in their household: Julia (65), Jessie (55), Malcolm (56), Hugh (57) and Paul (54). They are all siblings and have never married. My interest is piqued: who are these five adult children, never married, all living together on the farm?

For additional censuses and biographical information I turned to familysearch.org, a remarkably comprehensive database run by the Mormon church. I had never used it before and was blown away by the depth of data and level of integration. The Mormons have been hard at work, pulling every public data set they can find, scanning it and OCRing it, and doing a great job of linking sources to people and building family trees. The Knox family is in good shape.

Their father, Dr. John Knox, married in 1880 in Steele Creek. He was a well-known physician in Steele Creek and had eight children. Three married and moved away from Steele Creek, the remaining five stayed on the family farm. Hugh, Malcom, and Paul - the three men who stayed to work the farm - served in World War I. Julia and Jessie were homemakers for their entire lives. After Hugh and Malcolm sold the family farm to the Gambrells they appear to have moved to a nursing home in Pineville and passed away in 1978 and 1979. I am astounded at the level of detail familysearch.org has on this family. They have the original, hand-written World War I draft cards for all the children scanned in and OCRd. They have the military record file (really just a single 3x5 card) for everyone. They have everyone's death certificates scanned in and linked into their profiles.

Looking back at the registry of deeds you can paint a picture of the history of the family. Dr. John Knox purchased a small lot for his home in 1881 (shortly after marriage and right around the birth of his eldest) just north of the modern intersection of NC-160 and NC-49. His homestead is featured on a map of Mecklenburg County from 1911 where NC-160 (Wright's Ferry Road), NC-49 (Yorkville Road), the Pleasant Hill Presbyterian Church, and Walker Branch are all visible. The names of neighbors featured here often show up on the metes and bounds of Knox land painting a picture of a small community where farms are held in families for decades and generations.

1911 map of Steele Creek, Mecklenburg County, North Carolina

In 1885 Dr. John Knox purchased land on the eastern side of NC-160, including the land that becomes our property in Huntington Forest. You can see on the 1911 map that there are no houses on the east side of NC-160. It's prime, flat farming land and was reliably used for farming for decades until the whole area was developed. When Dr. Knox passes away in 1911 we see some deeds related to inheritance and dealings among the children but they keep referencing the original, hand-written deeds from the 1880s. This continues all the way up to deed 2703-277 in 1965 where the state pays the surviving brothers for the expansion of NC-160 and NC-49.

The Gambrell deed, made only a few years later, doesn't reference prior deeds by number or metes and bounds description. After almost 90 years the patchwork history of this land was getting a bit complex and could use some simplification but the surveys themselves also desperately needed to be redone. The original 1880s surveys were probably adequate for rural Mecklenburg County at the time but were quite inaccurate. Bearings on the metes and bounds were measured in whole degrees with no minutes and seconds, no doubt due to the technology available to those surveyors. The deeds referenced long gone roads, trees and other monuments such as a "stake on the branch with Sassafras pointers" from an 1885 deed. The more accurate Gambrell surveys translated into nice diagrams with PlatPlotter but the old surveys require substantial guesswork to figure out what they're trying to describe.

My best guess is that the southern half of Huntington Forest (the eventual Tract A in the Gambrell survey) was originally purchased by Dr. John Knox in 1885 in deed 44-10 from S. A. Carothers. The starting point is not clear from the description. I believe that when the surveyors refers to a "branch" they mean the Walker Branch creek and I also think they refer to the tiny creek behind the modern day Publix as a "drain". Even if the map location or the survey is inaccurate 44-10 seems the most likely candidate given its size, neighboring properties, and lack of other Knox tracts that could plausibly cover this amount of area.

Possible location of the 44-10 tract
Possible location of the 44-10 tract

S.A. Carothers didn't own the land for very long. He purchased the farm in 1881 from the estate of A.R. Erwin in deed 44-8. Things are getting fun now: the recording of the deed is in longhand and the scribe here has pretty good handwriting. As we get further on we'll find it harder to read handwriting and the quality of the scans gets worse. Note that the 44-8 deed is recorded along with the 1885 deeds. Modern land sales record the deed immediately but it is not a requirement. A valid sale of land can be done without recording the deed - the recording only helps establish the details of the transaction in case of a future dispute. If you lacked the money to record the deed or it was too inconvenient to get to the county seat for recording you might skip out on it for a while. In this case, after the death of Arthur Randolph Erwin, his children Margaret, Violet and Sarah sold the land to S.A. Carothers.

Arthur Erwin and his family resided in Steele Creek on that farm for decades, showing up in the 1850 and 1860 censuses. But now the deed trail goes cold - when did the Erwins get the land? There is at least one other Arthur Erwin in Steele Creek at the time which complicates the search. I see some likely deeds in the index but the scan quality doesn't let me read them. Unfortunately, my search comes to an end here. Perhaps I will have time to go in person some day to read a higher quality copy of the deed book.

Gil's LotD Announces Russian Sanctions

Submitted by gil on

CHARLOTTE, N.C. - In solidarity with the people of Ukraine, the Gil's LotD, Pro Football History.com and Omaha Poop Tracker websites have implemented economic sanctions against the Russian Federation in opposition to its war of aggression. The sanctions, a firewall on all IP connections originating from Russian IP origins, went into effect the morning of February 26th.

In 2022, Gil's LotD will be responsible for some part of the estimated $875 billion dollars in revenue from e-commerce in the United States. The sanctions intend to deny as much as possible of that revenue from Russian government officials and government-connected businesses who support the current government's invasion of Ukraine. Sanctions implemented by Gil's LotD are widely held as responsible[citation needed] for the collapse of Muammar Gaddafi's Libyan government in 2011 and the Taliban's rule over Afghanistan in 2001. Gil's LotD believes that its sanctions will pressure Vladimir Putin and his government to withdraw from Ukraine and recognize Ukraine's independence as a state.

Installing Windows 7 on an iMac 12,1 in 2022

Submitted by gil on

I've had this iMac 12,1 sitting around for a few years unused. The iMac was released in 2011 with a Sandy Bridge i5, a big, beautiful monitor, 12 GB of RAM or so, and an integrated, laptop-grade Radeon 6750M graphics card. I thought it might make a good web browsing appliance but I didn't realistically see it getting much use in this household. At the same time, though, I don't play games released in the past decade. So maybe I can throw Windows 7 on it, have a stable Windows machine around for once, and have an actual gaming PC? I went for it.

Backing up the old hard drive

The old hard drive had a still working but slow spinning disk drive full of personal documents and photos from my family. I use s3ql and Google Cloud Storage for my cloud backups and wanted to do a complete backup of this drive. It was easy to get the iMac to boot off of a Debian live USB, install s3ql and its dependencies on that, and do rsync to backup the whole drive. However, if you read the patch notes, s3ql had some serious sounding data corruption bugs and I needed to fsck the backup volume and also run a special data verification tool related to the data corruption issue. Furthermore, that tool worked by fetching the entire GCS blob to just to verify its size. I am not positive why they couldn't just get the size from the GCS metadata but it may have been related to the on-the-fly compression they do to your blobs. At any rate my volume is huge and it'll take forever to download and validate all these blobs so I have to spin up a one-off VM in the cloud just to do this fsck stuff and it still costs me like $40+ and a few days of VM time to do the entire fsck. s3ql's been an OK tool but its maintainership has not been great and it's had a few of these unfortunate bugs, enough to give me pause and question whether I should be using it for backups. Stay tuned, maybe I'll have the time to write my replacement for cloud backup some day.

Replacing the hard drive

To make the machine usable I wanted to replace the old spinning disk with a modern but cheap SSD. I think the machine was originally disused because the disk was just so slow, making the machine a chore to use. I grabbed a $60 internal SSD from a big box store and went to work.

This is one of those all-in-one iMacs where they've got the entire PC behind the LCD. It looks stunning now and was revolutionary when it first came out. The front of the iMac is actually a big slab of glass/plastic held in place with strong magnets. So to start any sort of internal repair you have to start with popping this plastic off without snapping it. Online videos recommended using suction cups like you'd do with a piece of sheet glass but I was able to get it with fingernails, shims and some patience.

Once the front plastic is off you can start with unscrewing the LCD which is sitting on top of everything. And it only gets weirder from there: the LCD is attached with several tiny cords, and in order to get to the spare drive bay (they have space for a spinning disk drive and a modern SSD) you have to disassemble nearly everything, as the spare drive bay is buried underneath the CD drive and against the back of the case. So I didn't do that. Instead, after sweating bullets just getting the monitor off and disassembled, I decided to just put the SSD in the space of the main drive bay and just sort of let it dangle there because I don't have any sort of tray or insert. It'll work out OK.

This machine is a work of art, inside and out. Apple really is the master of hardware design. But damn, this was one of the trickiest things I've had to do with electronics and it had me on edge the whole time. I wouldn't fault anyone for getting a third party to do this sort of maintenance for them.

Installing Windows 7 from a flash drive

Everything is put back together and it's time to install Windows. I have an ISO, I can just clone it to the flash drive with dd and tell the Mac to boot from it, right? That worked just fine with a Debian ISO on this exact same machine. Unfortunately Microsoft wasn't as thorough with their installer design as Debian is and the Mac wouldn't boot it. At this point, things start getting ugly:

  • I grab one of those free Windows VMs for testing Internet Explorer and use Rufus to create a NTFS-formatted Windows installer disk. Doesn't work.
  • Apple's Boot Camp Assistant supports creating Rufus-esque boot USB drives with the correct partitioning and drivers to boot on a Mac. I do this on my daily driver Mac, running macOS 10.15, and it doesn't work, because Apple has dropped support for Windows 7.
  • I create a High Sierra VM from scratch as that was the last release of MacOS X to support creating a Windows 7 boot USB drive. This does not work.
  • Thinking it was something about the virtual machine USB passthrough or something I copy the Boot Camp Assistant out of the High Sierra VM and run it under Catalina to format the USB drive. This does not work.
  • At this point I am desperate. I wasn't planning on putting MacOS on the new SSD, but while trying to boot all of these USB drives I notice that Apple's firmware has a "Network Recovery" option to download the MacOS X installer and boot without any operating system or installation CDs. I give that a shot, and it downloads the venerable MacOS X 10.7 Lion onto my SSD. That works fine.
  • Lion ships with a Boot Camp Assistant that supports Windows 7! Hooray! However, it doesn't support creating USB installers. You have to feed it an installation DVD, it formats the disk to leave a Windows partition and puts Apple drivers on it, and then boots off of the DVD. Luckily the iMac 12,1 shipped with a DVD burner and I have a very old stack of blank DVDs somewhere so I start burning my ISO to disk.
  • I have to burn two DVDs because they're old and rickety and a teeny bit scratched but eventually I get a good one. Boot Camp successfully partitions the disk and gets the Mac to boot into the Windows installer. Finally, we're making some progress!
  • Windows can't figure out what Boot Camp was trying to do with Windows partitions so I have to tell it to wipe the partition Apple made for it and install Windows into an empty filesystem. So, no Apple drivers on first boot. The DVD is also slightly damaged, preventing me from installing Windows 7 Enterprise, so I have to settle for Windows 7 Ultimate. But that installs and works fine.

Installing Windows drivers for the iMac 12,1

After the last installer reboot I am greeted by the Windows 7 desktop but with absolutely no drivers - no network card, no WiFi, no video card, no sound, nothing. I am so close but still so far from a functioning machine.

Boot Camp provides a standalone download of ~500 MB worth of Windows drivers for 64-bit Windows 7. This includes Apple-proprietary drivers for parts of the hardware but also the third-party stuff like the graphics card and network cards. I kept those on a USB drive and figured it would be one quick install to get everything working, right? Well, Apple's installers really didn't like my machine, for whatever reason. The installers didn't think I was running Windows on an iMac and wouldn't even attempt to install any drivers. I figured it was some sort of chicken and egg thing: because the drivers left by Boot Camp before Windows was installed got wiped, there must be some key drivers that I missed out on that allow the other driver installers to recognize that this was a Mac. Either way, I now have to bootstrap this entire machine's drivers from scratch.

First, I load up the driver for the Ethernet card. I figure with just that one I can use Window's driver discovery and update functionality to fetch the rest of the drivers for my hardware. I forget where I got this driver - it may have been shipped with Windows, it may have been pulled out of the Boot Camp package above. Boot Camp ships a top-level installer that was giving me errors, but that installer calls several sub-installers to handle each driver. Many of those sub-installers were also giving errors but I believe the Ethernet driver ran fine when ran independently of everything else and brought the card up right away.

Windows Update

Now that I have internet access I kick off Windows Update to download the inevitable patches but also hoping to get all of the drivers I need. However, Windows Update simply doesn't work. My Windows 7 ISO had SP1 slipstreamed into it, and it dates from 2012-2013 so it may even have more updates included. But at some point either a certificate expired or SSL settings changed or something and Windows 7 can't even communicate with Windows Update to get the list of available updates. Some quick Googling turns out I need a special KB update to get around this hitch. But even then I am having trouble - the old Internet Explorer lacks the certs/SSL support to view the installer page on Microsoft's website! I install modern Chrome, which of course ships with modern SSL and certificates, and Chrome will display the download page for me. But there's something about the installer download itself it doesn't like - another cert error or malware detection or something - and it won't allow you to download the installer and gives me no opportunity to override it. So I wind up having to browse to the update's web page in Chrome, copy the URL for the installer off of there, and paste it into Internet Explorer which will gladly let me download the installer. The patch works and we're back at it - Windows Update is working.

Upon reboot I am greeted with 163 urgent updates. This was the start of 3-4 hours of downloading, patching, and rebooting. There's detours in there to install some more KB updates to get around further issues with deprecated/changing security code (something about SHA2 hashes of update packages?) and fully download and install every single update it wants. The SSD is a lifesaver here but I'm still stuck on the wonderfully CPU-bound and single-threaded Windows Update.

I had hoped to be greeted by a fully functioning PC with drivers after finishing all of the updates. But that was not the case - Windows was unable to find any of the drivers for whatever reason, even when instructed to do so manually. It is now on me to find the rest of the drivers and bring them all up.

Drivers, part two

Although the Boot Camp driver installer isn't working I find a sub-installer at BootCamp\Drivers\Apple\BootCamp.msi. I believe I have to run this one from the command line to get around hardware checks. it installs most of Apple's Boot Camp drivers, especially the one-off ones provided by Apple directly, but I'm still left without a graphics card driver and WiFi driver, which are actually the only two I need to use this for gaming. Darn.

AMD still distributes old Radeon drivers for all of its graphics cards. I download a few versions but none of them are letting me install - they don't think I have a Radeon! This is troubling. Ultimately, I go to a somewhat-sketchy bootcampdrivers.com and get their patched driver which installs smoothly and brings up a working graphics card. I'm OK with leaving theirs installed as that driver is probably as modern as I'll get.

There's a Broadcom WiFi driver in the Boot Camp driver package and I am able to install it. However, it can't enable the WiFi chip once installed. I try using the generic Broadcom driver that ships with Windows (via the obscure "Install drivers on my own machine" dialog, which seems to be a relic from Windows 98 and incredibly can't be resized to show more than four of the hundreds of drivers available at a time) but this Broadcom driver does not work either. This is really unfortunate - I am incredibly close to a working system! Why won't the WiFi work, did I damage it or not reconnect it properly when putting the machine back together?

There's yet another sub-installer in the Boot Camp package that I try out, this one at BootCamp\Drivers\Apple\AppleSoftwareUpdate.msi. My hope is that this thing, once installed, will be able to detect my hardware and pull the appropriate drivers from Apple. And at first it seems promising - I think I did get some updated drivers, and I did see an installer package for WiFi support. But at the end of the day, even after installing Apple's own WiFi driver updates, I still didn't have a driver that could turn on my WiFi card!

Well, the answer turns out to be that I didn't have a Broadcom WiFi chip. After Googling the hardware identifier string I found that I had a Qualcomm Atheros AR938X. What the heck, man. I don't know if Apple even shipped drivers for this in the Boot Camp driver package. The best source for the driver online appears to be the incredibly sketchy ath-drivers.eu. Hopefully none of this stuff is compromised, but if it is - hey, it's a gaming PC.

Once that is installed the WiFi hardware comes up perfectly. Windows 7 does not yet support emoji and UTF-8-encoded WiFi network names so I have to join a mojibake SSID. This does not seem to impact the performance of the driver.

Conclusion

To test the hardware out I install Overwatch which is probably the most modern game I play. It defaults to the lowest graphics settings but appears to be getting a stable 30 FPS on the practice levels. It seems like my mission is a success!