Gil's blog

Title research in Steele Creek, Mecklenburg County, North Carolina

If you've ever purchased real estate you may remember the deed: a legal document describing the property and declaring that the grantor, the old owner of the property, is giving it to the new owner, the grantee. The deed is then kept on file in the county seat. This leaves a chain of deeds on file going all the way back to whatever was the original grant for the property. In the original 13 colonies of the United States this chain goes all the way back to the original grants from the English monarch.

Tracey and I recently purchased a home in Mecklenburg County, North Carolina whose register of deeds maintains a searchable database of deeds filed since 1990. But there are still more deeds: the county has records going back to 1763. Incredibly, the county has scanned in all of the old deeds as well and made both databases available for free! The old deeds don't have a text search, which is unfortunate but not surprising - deeds before 1920 or so are hand written in cursive. So my mission is now clear - can I trace the title of this land all the way back to the original grants from the Lords Proprietors of Carolina?

The county organizes its deeds into books, and the deed is indexed on its page in the book. These are written out like 36356-796, where the first number is the book number and the second is the page number. Yes, there are a lot of books - they've been counting up since 1763! That deed is the start of my search as it's the deed that transferred ownership to Tracey and myself.

Deeds are pretty formulaic. They are practical instruments, meant to cleanly transfer ownership of property to the grantee. In the original thirteen colonies of the United States they often use the traditional English legal practice of metes and bounds to describe the property as if you had walked around its boundary. The most recent deeds simply reference a specific lot on an official map, also filed with the county, instead of a metes and bounds description of the property, but once we get past the immediate subdivision history of the land we'll see metes and bounds used exclusively. The metes and bounds also put the boundary line in context with the property around it - for example, when a boundary runs along a public road or highway the road will be mentioned by name. This helps with locating the property on a modern map as many of our Mecklenburg County roads have not changed names or locations in over a hundred years. However, there is still some sleuthing required when names change, or when the deed only references nearby properties by owner's name at the time of sale, such as in 3032-203:

BEGINNING at an old iron, which iron is also the common northeast corner of Mrs. R. C. Mitchell and the southeast corner of the property of A. A. Porter and J. F. Neely estate, thence with the line of A. A. Porter and J. F. Neely estate North 31-59 West 376.62 feet to an old iron; thence with the line of Duke Power Company North 22-49 East 868.34 feet to an old iron located on the line of the property of Gambrell Investment Company (formerly Marshall Reid); thence with said line of Gambrell Investment Company South 20-47 East 1,834.59 feet to an iron located on the line of Mrs. L. E. Carothers, and which iron is also located South 66-17 West 356.06 feet from an old iron; thence with the line of Mrs. L. E. Carothers South 66-17 West 627 feet to an iron; thence continuing with the line of Mrs. L. E. Carothers, the line of the 1.11-acre tract conveyed by Mrs. Helen C. Carothers (Mrs. L. E. Carothers) to Gambrell Investment Company, and the line of Mrs. R. C. Mitchell North 14-24 West for a total of 874.03 feet to the point or place of BEGINNING, containing 20.33 acres as shown on a survey prepared by E. L. Faires, North Carolina Registered Surveyor, dated September, 1968. BEING the same property conveyed to Robert O. Brown and Wife, Doris H. Brown, by deed of Nancy Porter Kirby (Widow), and Clara Sledge Woods (Widow), dated __ ___, 1967, and recorded in Book 2881, at Page 442, of the Mecklenburg Public Registry.

I know that the deed is in Steele Creek township in Mecklenburg County as it is written elsewhere on the deed. The names are also familiar - the Carothers and Neelys come up many times in Steele Creek as I've done the title research. But where is this land actually located? There's no mention of county roads, postal addresses (although deeds never mention postal addresses) or natural features like creek crossings or confluences. There is no reference to any map (although sometimes one does get filed with the county) and the survey mentioned at the end is not filed with the county. At least it mentions the prior deed and I can look up that one in hopes it has more detail. But if that can't be found you have to find the neighboring properties mentioned by owner and hope you can triangulate the location from their context.

The "irons" in the description are metal stakes left in the ground by surveyors to designate the corners of the property. Yes, this comes with all the issues of accidentally or intentionally moved or misplaced markers. As you go back in time you start seeing trees, tree stumps, or the course of creeks and rivers in these descriptions making it more and more challenging to accurately draw a map of the boundaries of the property. The property may be formally specified and records kept for hundreds of years but the passing of time makes it a challenge to recover the actual property boundaries.

In a fascinating quirk of legal history in the US the complete history of the title has to be computed and double-checked to make sure that the land can be successfully sold. Should there be any problems in the past with the ownership or sale of the land you risk having older owners, their heirs, or an entire tribe of Indians sue to regain their rightful ownership over the land. This research is done by a title insurance company before they sell you a title insurance policy which pays out should you be subject to one of these lawsuits and lose ownership of your land.

The title search

Starting with the latest deed at 36356-796 the most recent history of the land is easy to trace. Each deed mentions the previous deed by book and page number making it easy to jump to the previous deed. In the year 2004 we reach deed 17821-200 where D. R. Horton, the homebuilder, sells the newly constructed home to its initial owners. A few months before that is deed 17203-933 where Provident Development Group sells the house to D. R. Horton. I am not sure what Provident Development Group is or does but I suspect they are closely related to D. R. Horton and heavily involved in the initial construction of the Huntington Forest subdivision.

In April of 2002 the Provident Development Group acquires the land from the Steele Creek (1997) Limited Partnership in deed 13509-753. This is the first time Sarah Belk Gambrell comes up in the title history. Sarah is a partner in the the partnership and mentioned by name in the deed. We'll also find that she's a major land owner in Steele Creek. But what is her story?

If you're from Charlotte the name Belk will jump out at you. It's the same Belk as the chain of department stores (think Macy's, but southern) and the John Belk Freeway in Uptown. The department store was founded by William Henry Belk in 1888 and his son John Belk was a former mayor of Charlotte. Sarah Belk was William's daughter and passed away in 2020 at the age of 102. Sarah worked in the family business, served as an officer in the corporation, held board seats and was a major shareholder. Belk was bought out for $3 billion dollars in 2015 and Sarah had a lifetime of salary, dividends and sales from the business. Her obituary mentions the extensive philanthropic support she gave to minority interests in Charlotte. But with that money she was also heavily invested in Charlotte itself as a substantial land owner, speculating on the growth and future development of Steele Creek.

The 13509-753 deed was Sarah's sale of part of her holdings in Steele Creek to create the Huntington Forest subdivision and some other nearby properties. However, at this point the deed trail runs cold. This deed does not mention the book and page of any prior deeds like we've seen so far. Luckily, this transaction is in the post-1990 online database which allows us to search on grantor/grantee name and try and figure out what happened here. I am not sure why this deed didn't mention the previous deed but there are a couple of clues. First, there is a lawsuit between the Steele Creek Limited Partnership and the city of Charlotte to acquire land for a sewer pipe. This lawsuit ends in default judgment a year later in favor of Charlotte, awarding them title to a narrow strip of land for the sewer. There's also some deeds giving the NC Department of Transportation extra room to widen the nearby NC 160 state highway and improve the NC 160 and NC 49 intersection. The existence of the lawsuit sounds dramatic - did the city have to FIGHT to get the land from Sarah? - but my understanding is that it is a lot more mundane than it sounds. The lawsuit is simply the legal process required for the city to acquire title or an easement for this utility work. Because it is a lawsuit, run by the courts, following their due process, it gives the land owner a fair chance to defend their ownership of the land. And because the trial expired in default judgment I believe both parties were in agreement from the get-go and a lawsuit with default judgment is the simplest and cheapest way to get it squared off. Sarah Belk Gambrell is a multi-millionaire practicing sophisticated land speculation, I don't think her lawyers would let a contentious lawsuit go unanswered. However, I also haven't dug into the trial beyond the court's documents ordered to be included in the deed book.

The title search also turns up deed 5399-284, from 1986, which is an antecedent to the prior deed. This is a transaction between a different Gambrell Investment Corporation to Sarah Belk Gambrell herself. I call it the mega deed: it does not feature the metes and bounds of any property, but instead references 38 different deeds, acquired between 1961 and 1973, and lists just the book and page number of each deed. Unfortunately there is no map here or any hints - I need to go through each one and figure out if Huntington Forest is in any of them.

As I dug into this I found that Sarah owned property close to what is now the intersection of I-485 and NC-160 on both sides of the interstate and also large parts of property along NC-160 south of that intersection. She owns the Huntington Forest area but also land around the NC-160 and NC-49 intersection, including what is now the RiverGate shopping mall in its southeastern corner. It took several hours, going deed by deed to figure out which ones were clearly not contributors to Huntington Forest and which ones were possible ancestors.

Modern map of Huntington Forest and the NC-49 and NC-160 intersection.
Modern map of Huntington Forest and the NC-49 and NC-160 intersection. 

The top third or so of Huntington Forest comes from deed 2881-596, the Ewart property. It is one of a few pieces of property carved out of the J. L. Milwee estate, which very helpfully had a map filed with the county.

J. L. Millwee Estate map

Tract "C" is the source of the Ewart property and others.
The map also gives us some concrete hints about land ownership and location in this area. First, you can see the modern Steele Creek Road (NC-160) and Erwin Road in their current alignments, making it easy to situate this on a modern map. The McClintock Presbyterian Church lot is also visible on the map. It was founded in 1865 as the first Presbyterian church for the newly freed Black slaves in Mecklenburg County and the deed to that property dates to 1867. The J.M. Corouthers property is also a name that comes up in many of these property descriptions. And finally, there is the Knox brothers property right below Tract "C". It is clear that Tract "C" is the top part of what is now Huntington Forest because it's nestled in the intersection of Steele Creek Road and Erwin Road. But to figure out if our property is in the former Tract "C" or the former Knox brothers land will require us to more precisely map out exactly how far north or south the Knox brothers line is.

After digging through the rest of the mega deed we find the Knox brothers property at 2881-477. Hooray! In August of 1967 the brothers Malcolm and Hugh Knox sold their family farm to the Gambrells. The deed is composed of three tracts. Tract C is further west on NC-49/Tryon St./York Road (as it was called then) and borders the Pleasant Hill Presbyterian Church that is still standing today. Tracts A and B both reference the NC-160/NC-49 intersection so they are clearly very close to Huntington Forest. But which one, if either, is going to be the ancestor of Huntington Forest? In order to find that I had to plot out the metes and bounds for each tract on a real map. Luckily, there is a nifty and free Plat Plotter website to help you with this.

To use Plat Plotter you mark the initial point on a slippy map and then type in the metes and bounds from the deed in a structured way. The description for Tract A and B both start with:

BEGINNING at a point in the center line of North Carolina Highway No. 49 (York Road), said point beginning N. 60-41-24 E. 456.00 feet from the intersection of the center line of N. C. Highway No. 49 and the center line of N. C. Highway No. 160 (Steele Creek Road) [...]

The modern intersection of NC-49 and NC-160 is slightly to the east of the historical intersection. Originally the two highways intersected as straight lines but an improvement to the intersection in the 1990s had NC-160 curve to the east before curving back to return to its former path. You can see the "hump" where NC-160 curves on the modern map above and also see the course of the original straight line path. On the Plat Plotter website I just marked the original intersection by eye.

The starting point for the metes and bounds is not at that intersection, though: it's at a very precise "N. 60-41-24 E. 456.00 feet" from it. What does that mean? That's how the surveyors encode the path around the property. "N. 60-41-24 E." is the bearing, the direction for the next segment in the boundary starting from your current location. It starts with N or S, meaning north or south, then the 60-41-24 is the degrees, minutes and seconds of the angle from north or south. The E or W at the end indicates if your bearing is turning towards the east (or clockwise from north) or turning towards the west (counterclockwise from north). To summarize, to find our property corner and the starting point for the metes and bounds you'd start in the center of the (former) intersection of NC-160/NC-49, you'd face north, turn 60 degrees clockwise, and march forward 456 feet.

Once you have the starting corner selected on the map you have the enviable task of transcribing all the legs from the deed into Plat Plotter so it can trace the boundary. Tract B, not pictured, covers a large portion of the RiverGate shopping mall south of Tryon St. Tract A, pictured below, is our jackpot. It clearly covers the bottom 2/3rds of Huntington Forest, including the house that we purchased!

Tract A of the Knox Brothers property
Tract A of the Knox brothers property

The parts not included in Tract A are also informative. It doesn't run all the way north to NC-160, which is what we saw on the Millwee Estate Tract "C". There's a missing piece to the west where the J. M. Corouthers land would be.  The boundary with Corouthers roughly follows the creek (Walker Branch) which would make for a natural boundary between farms. And it doesn't include a long, narrow strip of Huntington Forest along its east side, corresponding with the Tract "B" strip in the Millwee map that was also not part of the Knox brothers property. Plat Plotter also gives you the area of the property in acres and the gap between start and close of the boundary and they both came in perfectly. Mission accomplished!

There is still more title research to be done, though. The Knox brothers sold in 1967. Where did they get the land? Once again, the deed is less than helpful - it does not reference any prior deeds. The trail has gone cold. To get the search started again I turned to genealogy research. Luckily, the 1950 Census was recently released to the public. It was written entirely in longhand but the government also made an attempt to OCR the text and index it. I found the Knox family in enumeration district 60-16, called Steel Creek. There are five people in their household: Julia (65), Jessie (55), Malcolm (56), Hugh (57) and Paul (54). They are all siblings and have never married. My interest is piqued: who are these five adult children, never married, all living together on the farm?

For additional censuses and biographical information I turned to, a remarkably comprehensive database run by the Mormon church. I had never used it before and was blown away by the depth of data and level of integration. The Mormons have been hard at work, pulling every public data set they can find, scanning it and OCRing it, and doing a great job of linking sources to people and building family trees. The Knox family is in good shape.

Their father, Dr. John Knox, married in 1880 in Steele Creek. He was a well-known physician in Steele Creek and had eight children. Three married and moved away from Steele Creek, the remaining five stayed on the family farm. Hugh, Malcom, and Paul - the three men who stayed to work the farm - served in World War I. Julia and Jessie were homemakers for their entire lives. After Hugh and Malcolm sold the family farm to the Gambrells they appear to have moved to a nursing home in Pineville and passed away in 1978 and 1979. I am astounded at the level of detail has on this family. They have the original, hand-written World War I draft cards for all the children scanned in and OCRd. They have the military record file (really just a single 3x5 card) for everyone. They have everyone's death certificates scanned in and linked into their profiles.

Looking back at the registry of deeds you can paint a picture of the history of the family. Dr. John Knox purchased a small lot for his home in 1881 (shortly after marriage and right around the birth of his eldest) just north of the modern intersection of NC-160 and NC-49. His homestead is featured on a map of Mecklenburg County from 1911 where NC-160 (Wright's Ferry Road), NC-49 (Yorkville Road), the Pleasant Hill Presbyterian Church, and Walker Branch are all visible. The names of neighbors featured here often show up on the metes and bounds of Knox land painting a picture of a small community where farms are held in families for decades and generations.

1911 map of Steele Creek, Mecklenburg County, North Carolina

In 1885 Dr. John Knox purchased land on the eastern side of NC-160, including the land that becomes our property in Huntington Forest. You can see on the 1911 map that there are no houses on the east side of NC-160. It's prime, flat farming land and was reliably used for farming for decades until the whole area was developed. When Dr. Knox passes away in 1911 we see some deeds related to inheritance and dealings among the children but they keep referencing the original, hand-written deeds from the 1880s. This continues all the way up to deed 2703-277 in 1965 where the state pays the surviving brothers for the expansion of NC-160 and NC-49.

The Gambrell deed, made only a few years later, doesn't reference prior deeds by number or metes and bounds description. After almost 90 years the patchwork history of this land was getting a bit complex and could use some simplification but the surveys themselves also desperately needed to be redone. The original 1880s surveys were probably adequate for rural Mecklenburg County at the time but were quite inaccurate. Bearings on the metes and bounds were measured in whole degrees with no minutes and seconds, no doubt due to the technology available to those surveyors. The deeds referenced long gone roads, trees and other monuments such as a "stake on the branch with Sassafras pointers" from an 1885 deed. The more accurate Gambrell surveys translated into nice diagrams with PlatPlotter but the old surveys require substantial guesswork to figure out what they're trying to describe.

My best guess is that the southern half of Huntington Forest (the eventual Tract A in the Gambrell survey) was originally purchased by Dr. John Knox in 1885 in deed 44-10 from S. A. Carothers. The starting point is not clear from the description. I believe that when the surveyors refers to a "branch" they mean the Walker Branch creek and I also think they refer to the tiny creek behind the modern day Publix as a "drain". Even if the map location or the survey is inaccurate 44-10 seems the most likely candidate given its size, neighboring properties, and lack of other Knox tracts that could plausibly cover this amount of area.

Possible location of the 44-10 tract
Possible location of the 44-10 tract

S.A. Carothers didn't own the land for very long. He purchased the farm in 1881 from the estate of A.R. Erwin in deed 44-8. Things are getting fun now: the recording of the deed is in longhand and the scribe here has pretty good handwriting. As we get further on we'll find it harder to read handwriting and the quality of the scans gets worse. Note that the 44-8 deed is recorded along with the 1885 deeds. Modern land sales record the deed immediately but it is not a requirement. A valid sale of land can be done without recording the deed - the recording only helps establish the details of the transaction in case of a future dispute. If you lacked the money to record the deed or it was too inconvenient to get to the county seat for recording you might skip out on it for a while. In this case, after the death of Arthur Randolph Erwin, his children Margaret, Violet and Sarah sold the land to S.A. Carothers.

Arthur Erwin and his family resided in Steele Creek on that farm for decades, showing up in the 1850 and 1860 censuses. But now the deed trail goes cold - when did the Erwins get the land? There is at least one other Arthur Erwin in Steele Creek at the time which complicates the search. I see some likely deeds in the index but the scan quality doesn't let me read them. Unfortunately, my search comes to an end here. Perhaps I will have time to go in person some day to read a higher quality copy of the deed book.

Gil's LotD Announces Russian Sanctions

CHARLOTTE, N.C. - In solidarity with the people of Ukraine, the Gil's LotD, Pro Football and Omaha Poop Tracker websites have implemented economic sanctions against the Russian Federation in opposition to its war of aggression. The sanctions, a firewall on all IP connections originating from Russian IP origins, went into effect the morning of February 26th.

In 2022, Gil's LotD will be responsible for some part of the estimated $875 billion dollars in revenue from e-commerce in the United States. The sanctions intend to deny as much as possible of that revenue from Russian government officials and government-connected businesses who support the current government's invasion of Ukraine. Sanctions implemented by Gil's LotD are widely held as responsible[citation needed] for the collapse of Muammar Gaddafi's Libyan government in 2011 and the Taliban's rule over Afghanistan in 2001. Gil's LotD believes that its sanctions will pressure Vladimir Putin and his government to withdraw from Ukraine and recognize Ukraine's independence as a state.

Installing Windows 7 on an iMac 12,1 in 2022

I've had this iMac 12,1 sitting around for a few years unused. The iMac was released in 2011 with a Sandy Bridge i5, a big, beautiful monitor, 12 GB of RAM or so, and an integrated, laptop-grade Radeon 6750M graphics card. I thought it might make a good web browsing appliance but I didn't realistically see it getting much use in this household. At the same time, though, I don't play games released in the past decade. So maybe I can throw Windows 7 on it, have a stable Windows machine around for once, and have an actual gaming PC? I went for it.

Backing up the old hard drive

The old hard drive had a still working but slow spinning disk drive full of personal documents and photos from my family. I use s3ql and Google Cloud Storage for my cloud backups and wanted to do a complete backup of this drive. It was easy to get the iMac to boot off of a Debian live USB, install s3ql and its dependencies on that, and do rsync to backup the whole drive. However, if you read the patch notes, s3ql had some serious sounding data corruption bugs and I needed to fsck the backup volume and also run a special data verification tool related to the data corruption issue. Furthermore, that tool worked by fetching the entire GCS blob to just to verify its size. I am not positive why they couldn't just get the size from the GCS metadata but it may have been related to the on-the-fly compression they do to your blobs. At any rate my volume is huge and it'll take forever to download and validate all these blobs so I have to spin up a one-off VM in the cloud just to do this fsck stuff and it still costs me like $40+ and a few days of VM time to do the entire fsck. s3ql's been an OK tool but its maintainership has not been great and it's had a few of these unfortunate bugs, enough to give me pause and question whether I should be using it for backups. Stay tuned, maybe I'll have the time to write my replacement for cloud backup some day.

Replacing the hard drive

To make the machine usable I wanted to replace the old spinning disk with a modern but cheap SSD. I think the machine was originally disused because the disk was just so slow, making the machine a chore to use. I grabbed a $60 internal SSD from a big box store and went to work.

This is one of those all-in-one iMacs where they've got the entire PC behind the LCD. It looks stunning now and was revolutionary when it first came out. The front of the iMac is actually a big slab of glass/plastic held in place with strong magnets. So to start any sort of internal repair you have to start with popping this plastic off without snapping it. Online videos recommended using suction cups like you'd do with a piece of sheet glass but I was able to get it with fingernails, shims and some patience.

Once the front plastic is off you can start with unscrewing the LCD which is sitting on top of everything. And it only gets weirder from there: the LCD is attached with several tiny cords, and in order to get to the spare drive bay (they have space for a spinning disk drive and a modern SSD) you have to disassemble nearly everything, as the spare drive bay is buried underneath the CD drive and against the back of the case. So I didn't do that. Instead, after sweating bullets just getting the monitor off and disassembled, I decided to just put the SSD in the space of the main drive bay and just sort of let it dangle there because I don't have any sort of tray or insert. It'll work out OK.

This machine is a work of art, inside and out. Apple really is the master of hardware design. But damn, this was one of the trickiest things I've had to do with electronics and it had me on edge the whole time. I wouldn't fault anyone for getting a third party to do this sort of maintenance for them.

Installing Windows 7 from a flash drive

Everything is put back together and it's time to install Windows. I have an ISO, I can just clone it to the flash drive with dd and tell the Mac to boot from it, right? That worked just fine with a Debian ISO on this exact same machine. Unfortunately Microsoft wasn't as thorough with their installer design as Debian is and the Mac wouldn't boot it. At this point, things start getting ugly:

  • I grab one of those free Windows VMs for testing Internet Explorer and use Rufus to create a NTFS-formatted Windows installer disk. Doesn't work.
  • Apple's Boot Camp Assistant supports creating Rufus-esque boot USB drives with the correct partitioning and drivers to boot on a Mac. I do this on my daily driver Mac, running macOS 10.15, and it doesn't work, because Apple has dropped support for Windows 7.
  • I create a High Sierra VM from scratch as that was the last release of MacOS X to support creating a Windows 7 boot USB drive. This does not work.
  • Thinking it was something about the virtual machine USB passthrough or something I copy the Boot Camp Assistant out of the High Sierra VM and run it under Catalina to format the USB drive. This does not work.
  • At this point I am desperate. I wasn't planning on putting MacOS on the new SSD, but while trying to boot all of these USB drives I notice that Apple's firmware has a "Network Recovery" option to download the MacOS X installer and boot without any operating system or installation CDs. I give that a shot, and it downloads the venerable MacOS X 10.7 Lion onto my SSD. That works fine.
  • Lion ships with a Boot Camp Assistant that supports Windows 7! Hooray! However, it doesn't support creating USB installers. You have to feed it an installation DVD, it formats the disk to leave a Windows partition and puts Apple drivers on it, and then boots off of the DVD. Luckily the iMac 12,1 shipped with a DVD burner and I have a very old stack of blank DVDs somewhere so I start burning my ISO to disk.
  • I have to burn two DVDs because they're old and rickety and a teeny bit scratched but eventually I get a good one. Boot Camp successfully partitions the disk and gets the Mac to boot into the Windows installer. Finally, we're making some progress!
  • Windows can't figure out what Boot Camp was trying to do with Windows partitions so I have to tell it to wipe the partition Apple made for it and install Windows into an empty filesystem. So, no Apple drivers on first boot. The DVD is also slightly damaged, preventing me from installing Windows 7 Enterprise, so I have to settle for Windows 7 Ultimate. But that installs and works fine.

Installing Windows drivers for the iMac 12,1

After the last installer reboot I am greeted by the Windows 7 desktop but with absolutely no drivers - no network card, no WiFi, no video card, no sound, nothing. I am so close but still so far from a functioning machine.

Boot Camp provides a standalone download of ~500 MB worth of Windows drivers for 64-bit Windows 7. This includes Apple-proprietary drivers for parts of the hardware but also the third-party stuff like the graphics card and network cards. I kept those on a USB drive and figured it would be one quick install to get everything working, right? Well, Apple's installers really didn't like my machine, for whatever reason. The installers didn't think I was running Windows on an iMac and wouldn't even attempt to install any drivers. I figured it was some sort of chicken and egg thing: because the drivers left by Boot Camp before Windows was installed got wiped, there must be some key drivers that I missed out on that allow the other driver installers to recognize that this was a Mac. Either way, I now have to bootstrap this entire machine's drivers from scratch.

First, I load up the driver for the Ethernet card. I figure with just that one I can use Window's driver discovery and update functionality to fetch the rest of the drivers for my hardware. I forget where I got this driver - it may have been shipped with Windows, it may have been pulled out of the Boot Camp package above. Boot Camp ships a top-level installer that was giving me errors, but that installer calls several sub-installers to handle each driver. Many of those sub-installers were also giving errors but I believe the Ethernet driver ran fine when ran independently of everything else and brought the card up right away.

Windows Update

Now that I have internet access I kick off Windows Update to download the inevitable patches but also hoping to get all of the drivers I need. However, Windows Update simply doesn't work. My Windows 7 ISO had SP1 slipstreamed into it, and it dates from 2012-2013 so it may even have more updates included. But at some point either a certificate expired or SSL settings changed or something and Windows 7 can't even communicate with Windows Update to get the list of available updates. Some quick Googling turns out I need a special KB update to get around this hitch. But even then I am having trouble - the old Internet Explorer lacks the certs/SSL support to view the installer page on Microsoft's website! I install modern Chrome, which of course ships with modern SSL and certificates, and Chrome will display the download page for me. But there's something about the installer download itself it doesn't like - another cert error or malware detection or something - and it won't allow you to download the installer and gives me no opportunity to override it. So I wind up having to browse to the update's web page in Chrome, copy the URL for the installer off of there, and paste it into Internet Explorer which will gladly let me download the installer. The patch works and we're back at it - Windows Update is working.

Upon reboot I am greeted with 163 urgent updates. This was the start of 3-4 hours of downloading, patching, and rebooting. There's detours in there to install some more KB updates to get around further issues with deprecated/changing security code (something about SHA2 hashes of update packages?) and fully download and install every single update it wants. The SSD is a lifesaver here but I'm still stuck on the wonderfully CPU-bound and single-threaded Windows Update.

I had hoped to be greeted by a fully functioning PC with drivers after finishing all of the updates. But that was not the case - Windows was unable to find any of the drivers for whatever reason, even when instructed to do so manually. It is now on me to find the rest of the drivers and bring them all up.

Drivers, part two

Although the Boot Camp driver installer isn't working I find a sub-installer at BootCamp\Drivers\Apple\BootCamp.msi. I believe I have to run this one from the command line to get around hardware checks. it installs most of Apple's Boot Camp drivers, especially the one-off ones provided by Apple directly, but I'm still left without a graphics card driver and WiFi driver, which are actually the only two I need to use this for gaming. Darn.

AMD still distributes old Radeon drivers for all of its graphics cards. I download a few versions but none of them are letting me install - they don't think I have a Radeon! This is troubling. Ultimately, I go to a somewhat-sketchy and get their patched driver which installs smoothly and brings up a working graphics card. I'm OK with leaving theirs installed as that driver is probably as modern as I'll get.

There's a Broadcom WiFi driver in the Boot Camp driver package and I am able to install it. However, it can't enable the WiFi chip once installed. I try using the generic Broadcom driver that ships with Windows (via the obscure "Install drivers on my own machine" dialog, which seems to be a relic from Windows 98 and incredibly can't be resized to show more than four of the hundreds of drivers available at a time) but this Broadcom driver does not work either. This is really unfortunate - I am incredibly close to a working system! Why won't the WiFi work, did I damage it or not reconnect it properly when putting the machine back together?

There's yet another sub-installer in the Boot Camp package that I try out, this one at BootCamp\Drivers\Apple\AppleSoftwareUpdate.msi. My hope is that this thing, once installed, will be able to detect my hardware and pull the appropriate drivers from Apple. And at first it seems promising - I think I did get some updated drivers, and I did see an installer package for WiFi support. But at the end of the day, even after installing Apple's own WiFi driver updates, I still didn't have a driver that could turn on my WiFi card!

Well, the answer turns out to be that I didn't have a Broadcom WiFi chip. After Googling the hardware identifier string I found that I had a Qualcomm Atheros AR938X. What the heck, man. I don't know if Apple even shipped drivers for this in the Boot Camp driver package. The best source for the driver online appears to be the incredibly sketchy Hopefully none of this stuff is compromised, but if it is - hey, it's a gaming PC.

Once that is installed the WiFi hardware comes up perfectly. Windows 7 does not yet support emoji and UTF-8-encoded WiFi network names so I have to join a mojibake SSID. This does not seem to impact the performance of the driver.


To test the hardware out I install Overwatch which is probably the most modern game I play. It defaults to the lowest graphics settings but appears to be getting a stable 30 FPS on the practice levels. It seems like my mission is a success!

The Worst Employee Ever

This is archived from the original post still on the Internet Archive. I was bowled over by the lack of awareness here so I just had to archive it. There are several images in the original blog post that I haven't backed up. However, I did pull down the real whopper - an excerpt from the termination letter - and put that up.



I was employed by Hudson River Trading (HRT).

The company never provided due holidays to UK employees. The issue of holidays was brought up countless times by me and other employees, during the time I worked there. All requests were systematically ignored.

The problem could have been confirmed in 5 minutes of legal review. Looking at the employment contract (28 days) and looking at the holiday allowance in the absence tool, Namely (25 days).

The company simply explained to any employee raising a question about holidays, that they are not entitled to holidays and they’re wrong. When the UK office grew and the issue escalated further, because there is genuinely a problem that remained unaddressed (employees were risking losing their job or their bonus for missing days of work they don’t comprehend), Hudson River Trading responded by firing employees who questioned holidays.

I was fired for requesting holidays for UK employees. Upon termination, the company unilaterally refused to pay notice period, refused to pay non-compete and refused to pay outstanding holidays. I had to bring legal actions against the company, who largely ignored the requests from my lawyers and left me hanging with legal fees, the company required me to sign a very strict settlement agreement in order to get a reference letter and waiver of non-compete, or otherwise to see my career my livelihood and my family shattered. It was made abundantly clear that anybody who reports any concern will meet the same fate as me.

Hudson River Trading has zero regards for the law, their employment contracts, or financial regulations. I can’t comprehend the length they are willing to go, retaliating and silencing employees, just to prevent UK employees from having a few days of holidays. It’s absurd.

This article is a protected disclosure to highlight exceptional serious failures at Hudson River Trading.

I am going to cover my experience with Hudson River Trading in great details. This is a very long read, about 20 pages of text and 20 pages of legal documents. Hold on.


1) Let it be known that Hudson River Trading never granted due holidays to UK employees, the company only granted 25 days off (later 26), ignoring both employment law and their employment contracts which entitled employees to 28 days. The issue has been raised countless times and Hudson River Trading has resorted to terminate and silence employees who spoke up.

2) Let it be known Hudson River Trading does not pay notice period, does not pay non-compete clause, and does not pay outstanding holidays upon termination.

3) Let it be known that Roberta Yuan (HR and legal) and Prashant Lal (partners) are terminating and retaliating against employees who report concerns (aka. protected disclosures).

4) I am formally reporting Roberta Yuan to be formally investigated for exceptional wrongdoings and embezzlement. She runs both HR and legal in the company (extreme conflict of interest). She is using her position and drawing company funds to drive retaliations against employees and cover her own wrongdoings.

5) Prashant Lal is formally signing the papers as the director, which makes him personally responsible and liable for everything. Other than signing, his involvement is unclear.

6) HR and legal functions are compromised. I formally warn employees to NOT raise complaints/grievances, at the risk of their job and their personal safety. In fact the wiki page to raise HR questions is actually giving the lawyer as the point of contact.

7) I encourage current and past employees to come forward and raise concerns and documents directly to the FCA. The FCA can be reached by filling the contact form online Documents and attachments can be provided by email directly to whistle (at) with Case Number #53063 Hudson River Trading in the object.

Life at Hudson River Trading

First Contact

The first day I was off was 26th December, known as Boxing Day, a bank holiday in the UK.

The 27th of December I was awakened at 5am by HR (or whom I thought to be HR), to check where I was because I did not show up to work the previous day.

From their words, my manager was worried why I wasn’t in office and whether I was alright. He escalated to HR to contact me and check


The 26th of December was a bank holiday in the UK. The company was closed. It was formally written in the company policy that the company is closed on that day, see wiki page on Holidays. Oh boy I did not want to face disciplinary actions for being off on a day the company is closed. U_u

Notice that 5 AM London time is midnight in New York time. The employee had to stay all night just to wait to contact me?

Holidays Absence or Absence of Holidays

The next bank holiday was around April.

Same thing happened. I was off and the company was closed. Only to come the next morning and face my manager telling me I missed a day of work and he reported me. What?!

I had to explain again to my manager that it’s a bank holiday in the UK. It’s not a work day here. The same pattern repeated many times.

My manager insisted to formally take a day off if I am off, under HR instructions. I had to explain that doesn’t follow my employment contract. I wouldn’t be able to take holidays either way because I don’t have enough days off in Namely, the absence tracking tool. The manager referred to HR to sort out the problem. HR explained that I do not have holidays I thought I had. Well, what they stated did not match my employment contract.

The company only granted 25 days off per year to UK employees in Namely (the absence request tool). However UK employees should have 28 days off in addition to days the company is closed as per their employment contract.

Statutory Rights

Under British law, full time employees are entitled to 28 days off a year, including the bank holidays. (It’s possible to have 8 days other than the 8 bank holidays; the total has to be at least 28 days).

Contractual Rights

Most companies give more than the statutory minimum; it is set in the employment contract. It’s common in London to have between 25-30 days, in addition to bank holidays.

This is page 3 of my employment contract. I’ve asked other employees and they all had the same clause.

The contract means that UK employees are entitled to have 28 days off in addition to days the company is not open for business (typically bank holidays for UK businesses). It’s not clearly defined.

For reference, in British employment law, a “Working day” is formally defined: For the purposes of the Companies Act 2006, in relation to a company, a day that is not a Saturday or Sunday, Christmas Day, Good Friday or any day that is a bank holiday under the Banking and Financial Dealings Act 1971 in the part of the UK where the company. (For reference, there’s always 8 bank holidays a year and they are fall on Monday to Friday).

The firm had its own definition, that was shifting and unclear.

Holidays Absence or Absence of Holidays

I lost count how many times I found myself confronted to Human Resources and/or my manager for planning to be absent or for being absent on a closed day.

Hudson River Trading only granted 25 days off and expected employees to deduct bank holidays from that.

  • Under British employment law, employees are entitled to 28 days off per year (counting in the 8 bank holidays). Employees are given less than the statutory minimum (25 minus some bank holidays). It’s a breach of employment law.
  • Under employment contract, employees are entitled to 28 days off per year + days the office is closed. Employees are given less than 28. It’s a breach of contract.

I very specifically raised the request to Human Resources to review their UK employment contracts and correct the holiday entitlement to 28 days, multiple times, for myself and other UK employees. The request was always ignored. Sometimes they’d say they’d look into it and come back, but there was no follow up.

This was a general issue affecting ALL UK employees of Hudson River Trading. There had been countless complaints reported and it’s never been actioned. The issue had been ongoing for years I’ve been told, before I worked for the company.

In US English, this was ground for class action lawsuit against the company from every UK based employee against Hudson River Trading… if only class actions were a thing in the UK.

Unilateral Deduction

At some point end of April I noticed an automated email from the absence tracking tool Namely, a request had been submitted on my behalf to deduct a day of. Then much later a second email, the request had been approved.

I wasn’t informed that a day has been deducted except for the automated notification. I do not know who took the initiative to do that or who approved or why.

This was reducing the holiday entitlement further away from the statutory minimum, to a point where employees won’t have enough days even when taking most bank holidays off (which the company expect you to work). Shouldn’t do that.

Contractual Failure

Hudson River Trading doesn’t follow the contract or the law. What are they doing then?

One of the root causes of the holidays issue was the employment contract (or at least that’s what I thought at the time). A normal employment contract should state how many days are off, as in “The employee is entitled to X days off per year in addition to bank holidays” (most tech companies in London have X between 25 and 30).

The contract says employees are entitled 28 days off in addition to “days the company is not opened for business”. What are those? Nobody knows. The contract of employment for Hudson River Trading is unclear. It’s a notable mistake on the part of whoever drew and reviewed the contract.

The company tried to define closed days in a wiki, the wiki was unclear and it kept being modified (which is unlawful, it’s a contract and it cannot be modified unilaterally like a wiki).

Employees should have had 28 days off, the company only granted 25 days off. The company wanted to deduct (some) days the office is closed; however they have to be in addition as per the employment contract.

I raised these issues specifically I don’t know how many times. I’ve asked HR to formally review the contract with a lawyer. It should take 5 minutes to look at the contract and the Namely allowance, to realize that employees have never been granted due holidays. The lawyer should be able to work out what closed days is supposed to mean (and flag the lack of clarity as a problem).

One time I brought the issue of holidays to HR, she replied that she can adjust the policy, a wiki, to delete the extra holidays we’re not supposed to have (the company really doesn’t want employees to have days off). I had to explain that the company cannot delete holidays unilaterally. Employment terms form a contract in the UK and they cannot be edited unilaterally. This applies to holidays and most internal policies, they’re considered to be part of the employment contract in the UK.

What To Do?

The same pattern repeated over and over.

– Before or after a holiday, my manager would tell me I shouldn’t be off.

– I’d have to explain that the day is off because it’s written in my employment contract. And I can’t take the day off even if he wanted me to because I don’t have enough holidays allowance in Namely.

– My managed would report me to HR to sort out the problem of holidays, to who I would repeat the issue and ask for the holiday allowance and/or the contract to be corrected.

– HR would systematically dismiss the request and say there’s no holiday. Though once in a while, they’d say they’d look into it and come back to me. They never did!

– I was bleeding to death from the fear of losing half of my compensation or my job, at any minute.

April-May-June was supposed to be a chill period covered with bank holidays in the UK. Instead every single week was problematic and a source of unwarranted disciplinary actions for employees. Nobody understands what days employees are supposed to work or not.

Employee might be fired over this, to no fault of their own. Employees might face half of their compensation cut off suddenly. Hudson River Trading is a financial company with a large part of the compensation paid as discretionary bonus, confrontation is not how to increase bonus. Your mileage may vary, whether you try to take any day off and whether your manager or HR decide you cannot have that day. It’s quite inconsistent of course because nobody understands holidays.

There were new joiners in my team in April. They were supposed to start on a bank holidays. Had to explain that there won’t be people in the UK to welcome them because it’s a holidays, got directed to HR to sort out the problem, who insisted there are no holidays and we’re expected to work. New joiners thought they would have another day instead and they will have other bank holidays, well, nope. Expect a shock for employees coming from other companies and losing 5-10 days off compared to their previous role. Had to warn new joiners and bring them up to speed slowly about the company. The guy before me disappeared under very questionable circumstances. I might be next at this rate. Any of us might be next.

The general advice for developers in bad situation is to immediately leave. It’s almost became a meme on the internet, just quit. Well, it’s not that simple, notably because of claw-back clauses and non-compete.

Claw Back Clauses

I’ve looked at my options to leave the company. It is not possible to leave the company.

The employment contract has claw back clauses for 12 months, that’s pretty long. To leave the company an employee must repay their sign-on bonus if there was one (I had one) and part of their compensation. Assuming the clauses are enforceable, vary with circumstances.

I estimated that if I were to leave the company, either willingly or unwillingly, I’d have to (re)pay the company about £50k. The amount is increasing each month.

(Well, I estimated wrong but I only realized that months later, the claw back only applies to 4.1.b about the sign-on bonus, it doesn’t cover the discretionary compensation in 4.1.c, I missed the small (b) on the first reads. That’s really unfortunate for me. )

The company might try to claim a lot of money back in the event that they terminate employees. It was very scary. It made the problem of holidays much worse, waiting to explode in the face of the company, there is no easy way to leave willingly or unwillingly.

See contract section 4.2 for claw back and section 14.

Non-Competition Clause

Of course to leave, one must first get a job. It’s really difficult to get an interview, let alone a job. Financial companies are very few and very selective.

The abuse of non-compete clauses is rampant in the industry to prevent employees from moving. Other companies will simply not interview or cancel later stages because of non-compete. Non-compete clauses are an enormous problem to change job in the UK and they are considered very enforceable.

The employment contract with Hudson River Trading had a stringent non-compete for 6 months.

Between the non-compete and the claw-back clauses, it’s really difficult to leave Hudson River Trading. Prospective employees should be warned that when they join Hudson River Trading, there’s no leaving.

I thought, if something went horribly wrong (and things are not going great because of the holidays problem) the company will at least have to pay the non-compete for 6 months and I won’t be left to starve. Let it be known that Hudson River Trading unilaterally cancelled their non-compete.

Haemorrhaging Employees

I found out that the way my body reacts to extreme stress is to bleed. My nose just starts bleeding anytime for seemingly no reason. It’s stress!

It came to a point where I was bleeding 3 or 4 times a day, every day. This was my body warning me of life-threatening conditions. I’ve never been confronted to stress so extreme in my life than when I was working at Hudson River Trading. The worst is that there’s no way out between the claw back clauses and non-compete.

I had to turn off my webcam during most meetings because of the bleeding. If other employees noticed I stopped showing myself, I’m sorry, you should know that it’s because I was dying, I had to cover the blood.

Maybe I should have taken a doctor’s note and get off-sick, but how to get time off-sick when the problem that makes you sick is that you cannot get time off???

HR Announcement

After months of raising grievances, HR finally decided to take action to resolve the issue, by unilaterally reducing holidays. Wait, what?

Notice the first line, confirming that multiple employees have been raising issues for a while.

This email modified the holiday entitlement from 25 days off to 26 days off and made Boxing Day (26th December) a working day. (It might not be clear but that’s what happened after this email).

It effectively removed one day off (employee should have 28 days entitlement + closed days like Boxing days).

There is a wiki page, for the official company policy, specifying 26th as a day off, the wiki was edited out to remove the day. It’s unlawful, the company cannot unilaterally deduct and change holidays. This is part of employment contract and cannot be edited unilaterally. (Not that you could get away easily with the 26th anyway, I was off the last 26th December and was awakened the next day at 5 AM for not showing up to the office).

That announcement did not help. It was still unclear what days we are supposed to work or not this year. It did not match the employment contract. It did not grant statutory holidays. There were many more bank holidays coming up in May and June and it’s not clear when employees are supposed to work.

I can’t comprehend how did this announcement ever passed 5 minutes of legal review? It’s unlawful. It doesn’t address the problem.

Had to go back to HR once again to report all the points above and try to clarify holidays. The HR lady looked legitimately surprised that she cannot unilaterally cut holidays like that and said she would get back to me. There was no follow up.


Nothing happened despite more grievances. I could not go on like that not knowing when to work and constantly being under afraid, there was a genuine major issue, that was leading to get employees fired or bonus cut or lawsuit or worse.

I escalated the issue to the partners with UK employees in copy.

I received a reply from Jason, who’d said he’d personally look into the matter. No idea who Jason was but he seemed important. (Remember the name, it will come up later).

At about the same time, I was contacted by another individual to get me into a meeting, this was lead HR and my manager’s manager informing me I am suspended until they investigate the matter. I insisted again that the issue is real and if they took 5 minutes to show the employment contract and the allowance to any lawyer or the lawful, the company would be immediately told that they forgot to grant holidays to UK employees.

My laptop was remotely shutdown and wiped right after the meeting. (They told they’d cut access while investigating).

Was that really necessary to review a contract?


I was invited to join a phone call few days later, with lead HR (or whom I believed to be the lead HR) and my manager’s manager. Couldn’t even use my work laptop to join the meeting because the laptop had been murdered.

Oh boy I was naive to think that they would look into the matter.

Lead HR told me that I am fired effective immediately.

(☉_☉) That’s legally not possible, it is unlawful to terminate employee after making a protected disclosure, they can’t terminate me now.

HR informed me, precisely, that I am not being fired, the company is simply breaking my probation period, because I am still in probation period and I failed for poor performance.

(☉_☉) I am in utter shock and silent. I am NOT in probationary period or under a PIP. There is no issue with my performance. In fact my last conversation with my manager was that I am doing great and I will see my bonus increasing this quarter. The company is just making up reasons that do not exist.

HR repeated that I am terminated for performance issue. The manager added that I should never have sent this email to the partners, this is extremely inappropriate to escalate the issue like that and he can’t keep me on the team.

(☉_☉) Silence. Notice how HR is trying to make up non-existent issues about performance while the manager is formally telling me that I am being fired for making a protected disclosure. The reason for the termination is confirmed. I am now officially a whistle-blower. (That sucks, I don’t want to be a whistle-blower, what I wanted was my laptop unlocked, holiday allowance corrected to 28 days, and get back to work)

HR explained that the company is not paying the non-compete, the company is paying me 1 week to cover the notice period, but as an exceptional gesture the company is generously giving me 1 month of pay as severance to cover the troubles and the time to find a new job.

(☉_☉) There is no word for how shocked I am. The company cannot unilaterally decide to cancel the non-compete like that. I always thought if something went horribly wrong I would get the 6 months of non-compete as severance as a last resort. Nope. Hudson River Trading is unilaterally cancelling it. The notice period is actually 1 month. The company is refusing to pay the notice period and offering me to shut up for not paying my notice period. These people are insane. And they really think 1 month is enough to find a job?

I asked if they looked into holidays and if they will adjust the holidays for UK employees? and reviewed the employment contracts as requested?

HR responded that I am wrong and they will do nothing about holidays. Employees do not need more holidays.

(☉_☉) It’s extremely clear that the company hasn’t investigated anything. My access was terminated specifically to prevent me from contacting other employees and my laptop was wiped remotely in a deliberate attempt to destroy any evidence I may have had.

I insisted it’s not possible. If they had shown the employment contract to any lawyer in the UK and the holiday entitlement in namely (25 days before announcement or 26 after) and the employment contract (28 days expected), it would be pointed out immediately that the company has not granted due holidays to any UK employee.

HR insisted again that I am wrong about everything, HR added that non-disclosure clauses from my employment contract are in effect and if I talk about this matter anywhere to anyone, this is a breach of my confidentiality clause and she will pursue me on behalf of the company.

(☉_☉) Holy shit. She’s lying straight to me face about my rights and she’s plainly threatening to personally pursue me. For reference, I was just fired in retaliation for making a protected disclosure for the benefit of all UK employees, under British law all confidentiality clauses are waived and I am permitted to make a public disclosure. HR should be aware of that. (The lawyer would have pointed that too if they consulted one about holidays!)

I asked who is the legal contact for the company? I require the legal contact in order to bring legal action against the company. I am getting a lawyer, the company will be contacted by my lawyer.

HR replied herself. The HR person who fires employees is also the legal contact. (Odd?)

I received my formal termination letter shortly afterwards.

A Personal Note

There’s something that’s really difficult to capture in text. It how problems were constantly created out of thin air and escalated out of control at Hudson River Trading (and not just about holidays).

  • The first time you’re awakened at 5 am for being absent on a bank holidays. It’s painful, but it’s accidental, right? US folks don’t pay attention to UK holidays or time zones.
  • The third time you’re confronted by your manager or HR, it’s not accident, there’s really a problem. I checked with other UK employees and UK employees really cannot take holidays. I’ve worked for international companies; it’s a mess when they open new offices abroad, lots of little things to sort out. The issue here is not out of the ordinary, just need to adjust some holiday’s numbers and contracts.
  • Then suddenly you’re fired on the spot, your access is terminated on the spot and your laptop is remotely wiped in a deliberate attempt to destroy any evidence. The company refuses to pay your notice period and openly threatens you.

For me, there was sadly never an option to ignore the problem. My manager is a developer turned manager who is possibly the most pedantic person I’ve ever worked with. If I was (thinking of) missing any day or doubting any policy, it was systematically reported, like a bug. He actually wanted me to address the issue for other employees and new joiners of the team, which is fair, however it cannot be done if the company is not willing. It was a death march. It was bound to end up with the issue resolved or the team eliminated.

Lawyer Up

Time to get the lawyers involved. Not like there was a choice. The company refused to pay the notice period or the non-compete or to offer any severance.


First thing was to call ACAS and explain to get advice.

They replied the basics:

  • What you made was a protected disclosure.
  • The company terminated you following the protected disclosure.
  • Under British law, most protections for full time employees do not apply until 2 years of service, however if you are terminated for making a protected disclosure about a statutory right, which you were, protections apply regardless of time of service and you can sue for unfair dismissal.
  • There is no cap to the level of compensation (we’re talking UK, not US, the compensation is peanuts anyway)
  • You must get a lawyer ASAP.
  • There’s only 3 months to bring a claim to the employment tribunal, starting from the moment you were terminated.

On Normal Abnormal Termination

Let’s cover how a normal “abnormal” termination was supposed to happen. When the company terminates an employee, the company should offer a severance package and a settlement agreement.

  • reference letter of employment
  • non-competition pay AND/OR formal non-competition waiver
  • holidays pay
  • notice pay
  • severance pay
  • payslips (separately later)

Note: I ought to include payslips and tax forms last. They don’t go into the settlement agreement but better make sure that they are provided to employees soon enough and that the amounts due are paid. Without the document above, including payslips and letter of reference and non-compete, there are major troubles to move on.

Hudson River Trading didn’t do any single item out of that. Not a single one. In fact, from their perspective they simply broke my probation period (I wasn’t on probation).

The settlement agreement goes to layers for revision. The employee and the employer each gets a lawyer. You have to find a lawyer, who will get back to the company and negotiate the terms. Law firms usually offer a standard package to negotiate the terms of a settlement agreement.

In the UK, there is a time limit of 3 months from the date of termination to sue the company in employment tribunal. It’s really short. The employee has to decide what to do, quick, the timer is running.

Without anything on the table, I had to pay a lawyer thousands of pounds just to start the process. I had to pay the full legal fees myself because Hudson River Trading did not do their job. And they will refuse to cover legal fees that they have incurred on me.

Termination Letter

Here comes the termination letter.

The first page stated what was said during the call.

  • The company is breaking my probation period; I was not in probation period.
  • The company is giving me 1 week of pay to cover the 1 week notice; the notice period is actually 1 month.
  • The company is terminating me for poor performance and claiming I am under a PIP; there is no issue with my performance and there was no PIP.
  • The company is not paying the non-compete clause, unilaterally, and did not provide a formal waiver of non-compete to forward to prospective employers.
  • The company is firing me for making a protected disclosure; yes, it’s explained in writing

What’s interesting is the page 2, see below. It acknowledges in writing that there had been concerns reported about holidays for months, which remained unresolved. I resorted to making protected disclosure to partners and UK employees hoping to get the issue addressed. That made some individuals unable to cover their wrongdoing and they elected to terminate me.

This page alone should be enough to prove wrongdoing and retaliation from Hudson River Trading. That page alone should be enough for the company to terminate anybody who initiated that. If the document is escalated to regulators like FCA or SEC, there may be grounds to start an investigation and distribute fines.

The termination letter is signed by Prashant Lal, one of the partners. See page below. I’ve seen him few minutes during one induction for new joiners and seen him giving end of quarter presentations. Gotta admit, he did not strike me as a guy who would move heaven and earth to fire employees for few days of holidays.

Did Prashant actually read and sign this document for real? If yes, was Prashant explained the nature of this document, specifically meant to retaliate and silence employees making a legitimate protected disclosure. Was he explained that by signing this unlawful document he is personally liable and exposed to actions before courts and regulators?

On HR Announcement

I complained many times that the company doesn’t grant holidays and the HR announcement was unlawful.

Letter from my lawyer to the company, regarding the announcement and following termination:

Behold! The company doesn’t grant holidays and the HR announcement was unlawful.

That was determined in the first hour of meeting my lawyer, in the first minutes really, after being provided the employment contract and HR announcement.

Why was the company unable to figure that out in spite of getting recurring complaints about holidays for months?

On Holidays

What holidays are employees supposed to get? I’ve been trying to determine that for a long time.

As of 1st February 2021, the answer was 28 days in addition to the following days which were the closed days defined in a wiki:

  • Friday January 1st 2021, New Year Day
  • Friday April 2th 2021, Good Friday
  • Monday April 5th 2021, Easter Monday
  • Monday December 27th 2021, Christmas Day

(It got pretty hairy if one wants to look at the history, not that it matters. Employees were never granted 28 days off. The wiki specifying these days was edited to remove Christmas Day, which was unlawful. There was another wiki for the previous year, which was deleted, and showed something else.)

On Non-Competition

The non-competition is an enormous problem. It was practically impossible for me to leave the company willingly when I was employed, because of the non-compete and claw back clauses. (Not that I wanted to leave, I actually liked my work, I wanted to stay and keep my coworkers, which is exactly why I was trying to get holidays sorted out.)

It was extremely challenging to apply for positions after termination because of the non-competition clauses.

I tried to get some interviews with financial companies, which is incredibly difficult with how few and how selective companies are these days (note that the UK is a fraction of NYC finance job pool). They systematically asked if I had a non-compete and how long? They made a whole round of HR interviews just about that! It was horrifying.

I was falling apart and crying inside in interviews. I DON’T KNOW. I don’t know if I have a non-compete, what’s the status of the non-compete when the company illegally terminated me and refused to pay the non-compete and I am pursing legal action against the company to get it? Nobody knows. (Of course you can’t say that in an interview).

It was inhuman how much Hudson River Trading and their non-compete and their catastrophic handling of contractual matters is destroying my life and my career. And they did not bother to provide a formal waiver of non-compete when they decided to waive it unilaterally so I can’t show prospective employers that I do not have a non-compete. If a prospective employer asks to check my employment contract, which is a possibility for trading companies, they will see that I can’t work for them due to non-compete.

I wish I could say it’s unlawful to unilaterally cancel but it’s not that simple. There’s little general rule about non-compete in the UK, non-compete cases are decided on a case-by-case basis. Typically the non-compete will be considered by the court when deciding compensation.

As an employee my only recourse was to formally sue Hudson River Trading in court to enforce the non-compete against the company. It’s horrifying how non-compete are one-sided in the UK.

And there was a lot of money and details behind the non-compete, 6 month of salary, making it very important. Ideally this requires a lawyer who understands non-compete (who will say that it is ruled on a case-by-case-basis) as well as protected disclosures as well as financial regulations as well as general employment.

About Performance

The company claimed there were performance issue with me, it is false.

  • In my last weekly meeting with my manager on a Wednesday, my manager informed I was doing great and my bonus would increase this quarter.
  • On the following Monday I was terminated by HR for issues with my performance. It’s non-sense. There are zero issues with my performance.

The company claimed that I was under a performance improvement plan. It’s false. I was not.

There is something though that I ought to cover though. My very first week-on call there was an alert and an outage of the compute grid. Due to a bug that’s always been present, it just happened to trigger the first week I was on-call and took about an hour to resolve. The following Monday my manager asked me to join a video call. Had to wait a little while because I was in a meeting, can’t be everywhere. Unbeknownst to me, this was HR and my manager waiting, to inform me that I am being put on a performance improvement plan, following poor handling of the on-call issue, then HR immediately left because she had other events to attend to. (They couldn’t bother to arrange a time where people are free and make sure every is aware they need to be present?).

An email was sent to me afterward. I formally replied to contest placing new employees on a PIP on their first week on-call and asked for training and asked to change manager rather than getting fired.

I talked to manager and HR separately the next day. I was deadly afraid to be fired. They each separately assured me they are not planning to fire anyone, it’s a very normal procedure followed with numerous employees and they rarely fired people. (Understand why I am bleeding to my own death now? The combination of the company constantly pursuing people with no reason in combination with holidays issue and claw back clauses and non compete clauses, it’s horrifying and there’s no way out)

There was never any follow up of any sort to that event. It was a one-time rant. (HR actually did not realize performance improvement plan had a specific meaning in the UK, this isolated event wasn’t one. )

I talked to some coworkers about it. They warned me that this sort of circus happened before. The person before me disappeared under questionable circumstances; it was all extremely worrying for everyone. My role when there will be new joiners, soon, will be to warn them and ensure that they do not get canned by accident, just like other people warned me.

When Hudson River Trading started firing UK employees to cover up the holiday issue, Hudson River Trading tried to justify my termination by claiming that I was under a PIP for repeated poor performance and claimed that automatically extended my probation period. Well. It’s false and it’s unlawful. There is no performance issue. There is no PIP (that absurd rant that was never followed up isn’t one). The probation cannot be extended unilaterally without ever informing employees, if it can be extended at all.

Payslips and GDPR

There was an enormous problem after being terminated. I didn’t have payslips. (The company paid, it just did not send payslips). I was left unable to get any payslips.

There was a system to download payslips from, I tried to access it but I couldn’t access any of my payslip anymore. Access was terminated as part of the company shutting down all access?

I had to send a GDPR request to Hudson River Trading to request information, screenshot below. The company has 30 days to process the request.

The first line about HR profile was supposed to include payroll and tax forms, though that may not be clear. Communications are going through lawyers; it’s formal, slow and costing a fortune for each message.

Behold! The company simply ignored the GDPR request. No reply. Nothing at all.

It was an enormous problem because I need these documents to ask for any missing payment and to sue Hudson River Trading on various grounds. There is a 90 days delay to start legal action against the company and that required figuring missing payment and having these documents as soon as possible. The company not processing GDPR request timely or at all, was depriving me of the ability to seek legal recourse and to report wrongdoing to regulators. Also, it’s a breach of GDPR regulations.

The request got nowhere at all. Hudson River Trading will never provide anything because they know that all their communications and documents are damning evidence of illegal activities. Hudson River Trading was probably playing the watch too, because I might just miss the deadline for the employment tribunal if they stall forever.

The only thing the request achieved was the company openly threatening me. The company put a line into the later settlement agreement (which I had no choice but to sign) to retroactively cancel my GDPR request and prohibit me from sending any more GDPR request. If I send any request for any information of any sort, I will be immediately punished and requested to pay the company £35000 in damages (the sum referred in clause 3.2 is £35 000). See screenshot below, section 7.6 and 7.7.a.

They company also tried to waive my rights to make a subject access request or similar. See clauses 6.1 waiver of claim (see the settlement later in this article).

Was that legal? Can the company refuse to process GDPR requests and retaliate with punitive damages on the requester? Can the company waive rights to make GDPR requests?

There was supposed to be a data protection officer in the company, don’t know who it is, were they informed about the request and the retaliation?

The GDPR being fairly new, there is little information about what to do or the risks. I filled a report to the ICO, don’t see a case number. Sadly they are overwhelmed and only a tiny fraction of requests are processed. It’s unlikely it will ever be processed. 

Holiday Pay

I eventually managed to payslips after a while. That was difficult.

The company paid the notice period in full, after the notice from my lawyer that they are required by law to pay the notice. The company DID NOT pay outstanding holidays. There are thousands of pounds missing one way or another.

That took months to find out (took forever to get the payslips and the payment) and it’s seriously approaching the deadline for the tribunal. It’s bad.

For the avoidance of doubt, there may be debate about what holidays are due because holidays are a problem, that’s not the issue I am talking about there. It’s possible to count a few days more or less to calculate holidays pay. No matter the scenario, there are thousands of pounds missing. The company really did NOT paid outstanding holidays.

Had to send back a request through my lawyer for the company to pay outstanding holidays, there was a few backs and forth with the company lawyer to establish the amount. Then the company simply declined to pay, stating they “do not agree that holiday pay should be calculated in the manner that your client [me] suggested”.


It was not “my position” on calculating holidays. This was a legal request sent by my lawyer that’s costing me about £400 per email and that’s going to the lawyer of the company for review that’s costing I don’t know how much to the company. It took quite a bit of back and forth to establish the breakdown for the very specific circumstances here and with the text of laws given as reference.

The company is represented by counsel from Shearman and Sterling, a well-known silver circle law firm, that’s expensive and known to represent high profile financial firms.

Well, Hudson River Trading did not believe in their legal representations or in their obligations to pay outstanding holidays. (It’s consistent; the company did not believe in paying notice period or non-compete either!)

Seriously, are there actual people on the end of Hudson River Trading to process legal requests?


Yet another potential problem: Pensions.

Companies in the UK must provide a pension scheme and on board employees after they join.

With Hudson River Trading, I had some forms to fill and a zoom call with HR to cover benefits and pension, around the time I joined. Hudson River Trading made pensions opt-in and they discouraged employees to opt-in to the pension. That was an interesting call to say the least (good luck proving that in court!).

Took me a little while to find and get access to the pension:

  • Hudson River Trading paid the statutory amount, about £291 a month.
  • It took 115 days between the moment I joined the company and enrolment email from the pension.
  • That’s about 3 months of no pension contributions. No back pay.

There were legitimate questions about pensions. The company ought to make sure that employees are enrolled automatically and timely and NOT discourage employees to enroll. I think there is a legal delay of 90 days to on board or as soon as requested. How many employees missed the registration to the pension?

I wanted to bring this up formally but I was never able to, a few hours of lawyer is more expensive than the pension. At this stage, Hudson River Trading had made it pretty clear that they refuse to pay anything or refund legal fees they incurred on me.

It’s a good illustration of the catastrophic problem when dealing with Hudson River Trading. Every single thing done by the company is questionable when not outright unlawful. It’s reached a ridiculous point where it’s impossible to seek remediation because there’s too much to cover. Any employee who finds themselves in troubles will end up financially bankrupt in an endless list of claims, assuming one could get some lawyer(s) in the first place. It’s difficult to find legal representation, lawyers are specialized in a narrow field of law and the company is touching ten different things in a very complex case.

Final Settlement

Didn’t get anywhere reasonable. The company refused to pay outstanding holidays, refused to compensate for the non-compete, refused to refund legal fees they incurred upon me; the company plainly ignored the GDPR request.

The company simply considered that they’ve done nothing wrong; the company is in total denial about the existence of the law or the existence of contractual obligations or the existence of regulations.

That was two months in. There was a legal delay of 3 months to sue the company in employment court. Don’t know whether Hudson River Trading was playing the watch willingly or accidentally but that’s where things were. There is no money to cover going to courts and the company has made it abundantly clear that they will sabotage the process. I need a reference letter and waiver of non-compete NOW because it’s being an enormous problem to interview.

I was presented with a settlement agreement that gave me nothing. It was pages over pages of putting extreme liabilities over me in order to get a reference letter and a formal waiver of non-compete. I don’t think I’ve ever seen a contract in my life that was so one-sided. I can’t believe how they are shifting all the liabilities upon me, to cover their wrongdoings.

I cried for days.

I’ve tried to get interviews with some companies. I’ve had interview processes starting with a formal half hour phone call with HR to talk about the role and very specifically inquire about any non-compete obligations I may have. I want to cry. I will not be able to get a new job, not without the reference and waiver of non-compete. Background checks in finance go back 10 years; I will have to go back to Hudson River Trading to beg for a reference over the next 10 years, or risk watching my career burst in flame and go starving. The situation is catastrophic.

Sign or die.

Note: I consider that this contract was signed under duress and unenforceable. It was specifically meant to cover up wrongdoings from specific individuals, who are actively threatening me and using this contract as a mean to silence me.

See the settlement agreement.


Recognize the extreme waivers, followed by extremely strict NDA followed by extreme retaliation clauses all backed with extreme warranties and retaliation clauses. If I break any of these conditions, I will have to pay the company £35,000 immediately upon request, at any point in my life, or other worse punishment. Of course the conditions are impossible to satisfy.

See clauses 6.1 waiver that specifically named Roberta Yuan and Prashant Lal as off limits. Somehow the individuals involved got themselves off limits, they cannot be pursued by any means, if I intend any actions of any nature against them, including going to court or regulators, and they will personally pursue me and request me to pay damages. It’s waiving right to make a GDPR request too. I do not agree with any of that at all. I do not believe that this is lawful either.

See clause 8 about confidentiality obligations. It’s void and only intended to scare me (and it really does because the company has no regard for the law). Under British law, all confidentiality clauses are waived when wrongdoing is involved.

See clause 7 about warranties. An enormous list of warranties that cannot possibly be provided but if I don’t bend over and give willingly consent to be raped, the company will destroy my career and my livelihood.

It’s inhuman. I disagree with all these pages. I was so afraid when I received the documents from Hudson River Trading. I was asking them to offer some compensation and provide reference letter and waiver of non-compete, without which I cannot work again. The document they provided turned out to be pages over pages of legalese to kill me and my family whenever they wish to in the future. No human being should ever have to sign anything like this.

FCA violations

For reference, there were a few back and forth with the company. The corrections I managed to get from Hudson River Trading was to get a 10 year time limit on the non-disclosure agreement with an exclusion for “immediate family” (which doesn’t matter because confidentiality obligations are void) and remove some warranties clauses, screenshot below (which doesn’t matter because it’s void too but I was not aware of that).

Below, please find the original warranty section that was sent to me and I negotiated to have removed. I was scared to death because it’s impossible to satisfy. (Remember for example that the company does not honor their obligations toward pensions which I am unable to inform them because they refuse to cover legal fees required to inform them). I had no idea it was null and in violation of financial regulations.

See FCA handbook section 18.5.3 about making settlement agreements. “Firms must not request that workers enter into warranties which require them to disclose to the firm that: (a) they have made a protected disclosure; or (b) they know of no information which could form the basis of a protected disclosure.”

Post Settlement

There was one final email from Hudson River Trading on or about the time to conclude the settlement. They wanted to confirm that I have formally handed over and destroyed any [incriminating] documents I may have.

That was plainly asking for destruction of evidence which is required by courts and regulators. It was unlawful and I could not reasonably reply or confirm that request. It was in breach of FCA handbook for preventing employees from making a protected disclosure. It was potentially a federal crime to ask for deletion of evidence if the evidence covered a federal crime (didn’t know at the time but embezzlement and fraud are very much on the table).

I should have kept the email to be able to publish it, but I couldn’t. I cannot look at it. I and my partner just cried. Couldn’t sleep. Had to delete it. It’s inhuman was the company is doing.

Hudson River is simultaneously ordering me to destroy critical evidence and building a paper trail for (not) doing it to pursue me later. I am trapped either way. The company asked me to be consent-in-writing to be raped otherwise the settlement is off the table and they destroy my career and take away my roof (no reference letter and no waiver of non-compete); either way they reserve the right to pursue me and take away anything I have at any future moment of their choosing. 😦

The email went hand in hand with Clause 9.

Prashant Lal

The settlement agreement is signed by Prashant Lal, one of the partners. See page below.

Did Prashant actually read and sign this document for real? (The digital signature from DocuSign confirms yes)

If yes, was Prashant explained the nature of this document? Was he explained that by signing this unlawful document he is personally liable and exposed to actions before courts and regulators?

The legal communications (letters from my lawyers and GDPR requests) were exclusively sent to Roberta Yuan (or to the law firm) who terminated me and self-elevated to the only point of contact for the company. No idea if anybody else has a copy of any documents.

(I note that I willingly refrained from trying to contact employees because the company is retaliating against anybody who gets involved and I do not want to endanger other employees, it is not safe!).

There was supposed to be a financial regulator in the company representing the FCA and SEC, I think a lady from northern Europe, was she informed and provided with a copy of the documents above? Forgot the name and did not have any contact information.

I’ve many times wanted to contact Prashant to talk to him personally, trying to deescalate the situation, maybe try to reduce legal fees and get a more positive outcome for both sides. Maybe confirm and warn him about some stuff, there is a real possibility the signatures were forged or that he was unwittingly talked into signing these documents. Does he comprehend the liability this personally exposes him to? Would a director sign any document like that for no benefit to them? Would a director willingly sign a document that exposes the company to enormous fines and counter action from regulators?

Maybe the ultimately irony is that this agreement formally prohibits me to contact Prashant Lal or anyone. The only recourse left open is to make a public disclosure to the general public.

The company was very successful in isolated me and preventing any contact with employees. The sudden destruction of the work laptop and the sudden termination while pretending to investigate the matter were terribly effective in isolating me and destroying evidences.

Fear and Wonder

It’s really puzzling the length the company is willing to, just to avoid giving a few holidays that employees are entitled to. It was not an accident. The company is fundamentally opposed to granting holidays. And it is committed and willing to destroy life over it. What happened to me will happen to the next person.

What will happen next when Hudson River Trading will extend to other locations in Singapore, Tokyo, continental Europe… Hudson River Trading will fire and threaten all employees there for raising any questions about local laws or customs?

It’s absurd.

Part IV: Revelations

There was a news article published in Bloomberg some time later, that highlighted critical new information that I wasn’t aware of.

For reference, the article is right and the company is making over a billion dollar per quarter with about 500 employees. That should give you an idea of budget available to cover up and retaliation (it’s a shame the company forgot to put few pounds of budget aside to cover notice period and non-compete!).

Current and future employees of Hudson River should be afraid.

That should also give an idea how much money is on the table, personally for partners and high level employees on the other side, or employees who are committing embezzlement. A 7 or low 8 figures sum is peanuts for the company. People do get killed daily for much less than that.


The first new information that came up: Jason Caroll is the owner and founder of the company. I did not know that!

There are three partners in Hudson River Trading: Prashant Lal, Jason Caroll, and Oaz Nir.

  • Oaz Nir stopped actively running the company years ago, as far as I am aware. Never saw him.
  • Prashant came much later; he’s giving quarterly presentations and signed all the unlawful documents to retaliate against me.
  • No idea what was the involvement of Jason. Never saw him.

When I escalated the issue to the partners and UK employees, there was a global reply from a Jason but with a different surname. He said that he would personally look into that. No idea who that guy was until now, Jason is the owner and founder of the company. Obviously there was zero follow up on his part, employees got fired and threatened, and the holiday issue was never addressed.

I’ve many times wanted to talk to “whoever” runs Hudson River Trading. It’s ridiculous how every single thing the company has done is blatantly wrong and illegal. Surely the owner would not approve of that if he was aware.

@Jason: Your employees are stealing from you to threaten other employees and cover up their wrongdoings.


In Hudson River Trading, there were only two Human Resource employees L. and R. These are the only two employees to raise employment questions or concerns. (Note there are recruiters who do outreach and go to university job fairs, these are not Human Resources per se). The first part of the disclosure is mostly with L., the second part is mostly with R.

This is formally explained in a wiki page. If there is any kind soul left at Hudson River Trading, please find and take a full screenshot of the wiki page to raise HR question (whole screen including URL bar) and send to the FCA whistle (at) with subject “#53063 Hudson River Trading”.

Just found out after the Bloomberg article. R is not just Human Resources. She is a qualified lawyer in the New York bar. She is running both Human Resources AND is the lawyer of the company AND is the person executing all contracts on behalf of the company. That’s a lot of responsibility for a single individual, all with zero oversight. She never stated she was a lawyer and was representing the company. I always thought she was the manager/lead for Human Resources.

I believe this is an extreme form of conflict of interest. Everything is run by the same person. What would regulators think about that?

If employees read and follow the procedure to contact Human Resources, they may be contacting the lawyer instead, who may be preparing their termination. Is it normal for the lawyer of the company to be the contact for Human Resources?

Since R is not Human Resources, that means there is ONE SINGLE Human Resource employee, L, who is all alone. Is it normal for a company of 500+ employees to have only a single Human Resource person?

Is it reasonable for a regulated financial company with $4B in cash and a two digits percent of all equities trades in the US market, to have a human resource department consisting of a single new graduate? What would regulators think about that?

The entire time I thought I just needed to reach anybody who can order a contract review to correct holidays. There is nobody in the company who can process any of this. That makes sense now. There’s one single person controlling everything to cover her wrongdoing. She had me terminated and had my laptop erased the second I raised a concern she couldn’t bury. She personally threatened me for legal action if I were to disclose anything about what happened because that would highlight her wrongdoing and put her job in jeopardy.

See the elephant in the room?

This is a perfect setup to cover wrongdoing and embezzlement.

I am formally reporting Roberta Yuan, representing Human Resources and Legal of the company, to be formally investigated for embezzlement.

She’s terminated me and she personally threatened me to stay silent, or otherwise she would pursue me on behalf of the company. She personally contracted lawyers and executed contractual agreements on behalf of the company to retaliate and silence employees, her actions could not have been done on behalf of the company because it is illegal and exposes the company to enormous fines and regulatory actions, meaning it was done on her own initiative but using Hudson River Trading for the express purpose of covering her own wrongdoings. That ought to fall under the definition of embezzlement.

I sincerely believe that both Human Resources and Legal functions in Hudson River Trading are compromised.

It’s catastrophic. Only god knows what else may be covered up. Every contract from the company may have had similar treatment as those in this article. Any employee who report any concerns will be terminated and threatened the same way I was. Any next employee who joins Human Resources or legal will be faced with a walls of problems and risk their careers. For reference, there was a recruitment person who was fired suddenly in great fanfare earlier this year, I think she’d have to work with HR and legal closely as part of her work, what did she come across that pushed the company to instant fire?

The involvement of Prashant Lal remains unclear, except for signing all the documents and being responsible for everything. I never talked to him. (It’s extremely fishy. Why would anybody do that? There must be critical information that’s missing?).

This article is a protected disclosure for exceptional wrongdoing

Reporting Concerns to the FCA

I encourage all employees, both current and previous employees, to report concerns and any wrongdoings to the FCA. You can provide tips, conversations and documents to the FCA directly to whistle (at) subject #53063. There is no not need to be in the UK to assist the FCA. Employees from all over the world can contribute and make the world a safer place.

No human being should be treated and threatened like Hudson River Trading has done. It’s wrong. It has gone way too far.

I tried to but I can’t live with the constant fear over my head that the company will turn around any day and try to take anything I have. I can’t live knowing that they will do the exact same procedure to me to the next person. It’s not human.

Final Disclosure

Now is December 2021, the next bank Holidays will be soon on 27th December 2021, known as Boxing Day.

Boxing Day was a closed day, employees were not expected to work on that day and did not need to deduct one day from their allowance, it was formally written in the company policy, until the wiki was unilaterally edited to remove that day, an edit which was unlawful.

What’s the result? Are employees supposed to work or not? Who knows?

Please clarify holidays.

Please stop firing people for being off on bank holidays when no employee can comprehend what days they are required to work or not.

Zappa vs Mangum

Zappa and Mangum[1] are Python libraries that parse events from Amazon's API Gateway and feed them into a Python WSGI or ASGI application. This allows you to put API Gateway in front of a Lambda runtime, have Lambda execute the service requests that come in through API Gateway, and code your service logic in any Python web framework.

But what framework should you use for your next serverless REST API? I say Mangum. Both are mature products and Zappa appears to have been around longer and has more features implemented to show for it. But there are some key features, ranging from nice to have to increasingly required, that Mangum is delivering on and Zappa isn't that makes me recommend Mangum for future uses.

First, Zappa's maintainership hasn't been great. There is constant activity on Zappa, but a lot of low-hanging bug fixes and feature improvements, like the ones I am detailing here, haven't been implemented. Even if it is possible for Zappa to catch up and get these things knocked out the lack of responsible maintainership points to a dying project.

Zappa goes beyond being a shim library to handle WSGI requests and has grown up a small ecosystem of functionality, including the ability to package Lambda zip deployments on your behalf, deploy lambdas for you, create CloudFormation JSON to manage your configuration outside of Zappa, invoke one-off functionality in Lambda outside of the API Gateway request cycle (like a maintenance command), and tail Lambda logs in real time. This is all extremely useful stuff and makes Zappa much more appealing than the bare-bones Mangum. However, Zappa's functionality has slowed its ability to keep up with the latest AWS features. Zappa can only assemble Lambda deployment zip files for x86 Lambdas, not the new, cost-effective Graviton 64-bit ARM CPUs. It also only supports v1 of the API Gateway events, locking you out of a simpler and more cost-effective v2 deployment. Finally, Zappa does not have any way to pass the original Lambda event on to your application code and only extracts the minimum data from the event required to invoke a WSGI request (omitting things such as the IP address of the original request made to API Gateway). This is a really unfortunate one-two punch where Zappa isn't even letting you work around its shortcomings.

If you jump to Mangum you'll have a bit more of a pull ahead of you. There's no tooling to package and deploy a Lambda for you so you'll have to do some DevOps work to create that pipeline. But if you dig in and get it done yourself you could assemble a Docker image for Lambda on the the Graviton CPUs, something not available to you at all with Zappa. Using a Docker image lets you get around the 50 MB limit on zip files for the Lambda deployment. If you are using one of the many WSGI-only web frameworks you may be put off by Mangum's lack of support for WSGI. However, there is an open source WSGI-to-ASGI adapter that will let you connect any existing WSGI code into Mangum, so ASGI-only is not as limiting as it sounds. Mangum is also doing a better job at parsing the API Gateway event payloads, supports the latest version of API Gateway, and even lets you write WebSocket APIs.

Taking the tradeoffs into account, even though Mangum does not offer as much overall functionality as Zappa does right now, it still supports the core, important functionality that you'll increasingly need to use in the future, like 64-bit ARM Lambdas, support for the new AWS APIs, and support for ASGI frameworks. For those reasons I recommend using Mangum for new API Gateway projects with Python.

[1] Zappa is named after musician Frank Zappa who needs no introduction. Mangum is named after Jeff Mangum, of the band Neutral Milk Hotel, best known for an album about Anne Frank and "semen stains the mountaintops."

ngx_brotli for Debian bullseye and buster

Debian does not yet ship ngx_brotli (libnginx-mod-brotli) so I've compiled Ondřej Surý's work so it can be used by everyone on Debian bullseye (Debian 11) and buster (Debian 10).

To install, download the latest version of the .deb from the GitHub releases page and install it with:

sudo dpkg -i libnginx-mod-brotli*.deb

Next, you must enable the module:

sudo ln -s /usr/share/nginx/modules-available/mod-brotli.conf /etc/nginx/modules-enabled/50-mod-brotli.conf

Edit your nginx configuration in /etc/nginx to make use of ngx_brotli. Typically, this means adding brotli_static on; or brotli on; to the relevant parts of your configuration.

Finally, restart nginx to load the new configuration and modules:

sudo systemctl restart nginx

And that's it!

Frequently asked questions

Help! I'm trying to install a security update for nginx or update nginx on my system. Apt is forcing me to uninstall libnginx-mod-brotli because its dependencies are unmet. What do I do?

You must install the version of libnginx-mod-brotli that precisely matches the version of nginx on your system. Download a new version from the GitHub releases page and reinstall that. If a new version is not available yet please open an issue on the GitHub page and I will upload a new version as soon as I can. It may be prudent to uninstall libnginx-mod-brotli before updating nginx and re-install the new version of libnginx-mod-brotli immediately after updating nginx to avoid complications with apt.

Can you provide an apt repo with this package in it?

Maybe later!

Demand for computing power

I'm putting my prediction hat on again, let's come back in a year and see how well I've done.

It's well known that Intel has fallen behind TSMC's fab technology. I think this is the beginning of a trend that will drive up the cost for computing power across the market. Big buyers of semiconductors, who carefully forecast their own growth in demand for computing power, also estimate the lower costs and improved performance that come with each generation and use that to guide their budgets. When Intel starts stumbling and comes up short, with chips that offer less computation or more electricity draw than expected, these buyers suffer as they now have to increase the volume of their buys of chips that aren't as marginally effective as expected. It's a big capital demand up front and a higher total cost over the lifetime of the hardware. Increased demand from the biggest volume buyers means increased cost to the smaller buyers who are also suffering from the higher total cost and missed performance of those chips. Our old hardware doesn't go away, and the new supply of chips doesn't disappear. But this is a big enough supply chain disruption to drive up the cost of computing power for a decade.

So what does this mean for the software industry? The cost of self-hosting is going up but so is the cost of cloud computing. If capital costs are high expect businesses to lower headcounts and hire fewer engineers to do more. I expect that we'll see a move away from productive but computationally inefficient ecosystems like Python and Ruby and towards more computationally efficient ecosystems like Rust, Swift and C++. I don't know if Java, C# and Javascript - the languages with top-tier JITs - are in one category or the other, but they're probably more efficient than not. They might suffer more from the decrease in headcounts over their increased capital costs.

Disk caching in the year 2020

In the previous blog post I went over the design of storage systems and how they protect themselves from data loss in the face of disk failure or unclean shutdowns. If you’re putting together the hardware yourself modern Linux systems and modern hardware give you endless cost, performance and durability tradeoffs for whatever system you want. But 2020 also gives us the cloud, and storage systems in the cloud whose durability far exceeds whatever you could build yourself. Cloud providers also have the scale to amortize the insane costs of these exotic, durable, and yet still performant system designs across their broad user bases, bringing down their prices and making them accessible to us mortals. In the year 2020, if you’re in the cloud, durability just isn’t a concern to you so it is time to start focusing on performance.

Caching basics in 2020

2020 also brings affordable, reliable and speedy SSDs, and if you’re designing a system yourself you want to have some way to take advantage of SSD performance. Building everything out of SSDs isn’t an awful choice, it’ll be simple and probably won’t even cost that much. However, the expected usage of most storage systems will have some bulk of data rarely read or written giving an opportunity to reduce marginal costs with tiered storage. If you have a way to keep cold data on slower but cheaper magnetic disks and can use SSDs when appropriate this tiered storage design gives you the speed benefits of SSDs when you need them the most and the cost benefits of magnetic disks.

Tiered storage introduces complexity back into the storage system, increasing operational overhead and the risk of data loss. If you can implement tiered storage at the application layer you have some of the best tools available to you to tame that complexity. For example, a filer could keep some file shares known to be cold, like backups, directly on magnetic storage and others known to be hot directly on SSD storage. A database could have tablespaces for slow and fast storage and assign cold and hot tables to each. You could keep table indexes in fast storage with reduced reliability knowing that disk failures in the fast storage won’t cause unrecoverable data loss. These are all simple systems, with well-understood failure modes and controlled operational complexity. They also give you the most benefit from SSD performance because their simplicity skips the CPU and I/O overhead introduced by more complex systems.

But what if your application layer doesn’t give you tiered storage? What if you can’t afford the operational complexity required to maintain it yourself? Delegating where the data goes initially is easy, but moving it around is harder. And assumptions can change, with cold data becoming warm, and you need continuous feedback to make and question your storage decisions. Caching is the next choice, where the system makes decisions on where to keep data automatically. It’s usable anywhere and can bring its benefits to every storage system but it comes at the cost of potentially suboptimal usage of the fast storage and a potentially increased risk of data loss.

Caches typically implement multiple caching algorithms, giving you more tradeoffs between speed, durability and cost. Writeback caching has all data written to fast storage first and later copied asynchronously to the slow storage. It gives the best performance as cache reads can use data from the newest writes immediately and writes go at the speed of the fast pool. But should your fast storage system suffer data loss before the asynchronous copy finishes you’ll have permanent data loss. Your system performance also hinges on the intelligence of the writeback cache algorithm more than other algorithms. Imagine a system that does backups - if the writeback cache is unable to determine that backup writes are cold and shouldn’t fill up the cache you’ll wind up evicting all the useful data for extremely cold data. This logic can’t be simplified or formalized for general use and requires tuning and measurement.

Writearound caching has writes go directly to slow storage and only reads make use of the cache. It’s a blunt tool, unable to improve write speeds, and unable to cache recently written data, but it never has to implement the dark art algorithms like writeback does to get good performance against pathological workloads. You also won’t lose data if your cache disk dies. There is also writethrough caching where writes are done to both fast and slow disks at the same time, improving the cache hit rate of recently written data but still limiting your write speed to the speed of the slow disk.

The previous blog post introduced the RAID write hole and a few techniques to close it. If you’re building your own storage system you are probably considering RAID and using one of those techniques to improve the durability of the system. Journaling is a well-supported and reliable way to close the write hole and under the hood these journals are often just writeback/writethrough/writearound caches, with all of the benefits and drawbacks as described above. In other words, a RAID array of magnetic disks with large enough SSD journals is also a RAID array with SSD cache. Journal disk failure is still a threat, putting the less aggressive cache algorithms into strong consideration, because an unfortunate disk failure in a writeback journal results in losing all data on the entire array.

The storage available to you in the cloud shifts your priorities the other way. Durability is a given as discussed earlier and you’re now looking to squeeze what performance you can out of your design. More aggressive caching algorithms like the writeback cache are now in scope and should be seriously considered, even for ordinary cloud setups.

Finally, I want to cover the Linux buffer and page caches, just for completeness. Your application's I/O hits these first and all of the other cache systems later. If a cache hit is done here any other disk caching code is never run at all. A 2017 summary of the history, current state and future of these Linux caches is on LWN. The page cache carves system memory up into equally sized pages, typically 4 kilobytes, and maintains attributes on each page's uses. A page could cache a region of data on a disk, meaning that the page's contents directly correspond to 4 kilobytes on a disk somewhere. A read request fills that cached page and future reads can be serviced by that page. A write to the disk updates the cached page and marks it as dirty, signalling the kernel to rewrite the corresponding data on the disk at its leisure. The buffer cache is a legacy API, internal to the kernel, that is implemented on top of the page cache. This API works with smaller units of data, called blocks, that are mapped onto relevant page cache entries for each file. Reads and writes to blocks are passed through to the page cache subsystem. The buffer cache was only separate from the page cache in much older kernels. The modern API for I/O is built around I/O requests and the bio struct, instead of stateful buffer cache blocks, and allows for I/O requests to be batched, reordered, or executed in parallel.

Disk caching in 2020

So what cache options are available to us on a modern system? As mentioned in the previous section, you can always just buy a bunch of RAM and you’ll get some in-memory caching provided by the kernel's page cache out of the box. If you’re using a RAID array there are ways to use the array’s journal as a cache. Modern kernels come with more exotic tools, though, and I will cover a few of them.

bcache, introduced in 2010, ships with most kernels. To use bcache you format your backing (slow) disk or logical volume with bcache and create whatever filesystem you want on top of the resulting bcache device. You then enable caching on the backing device by formatting a cache device (your SSD) and attaching it to the backing disk via sysfs. bcache implements writeback, writethrough and writearound caching, with the ability to switch between cache algorithms at runtime, and has implemented its own code to identify read patterns that shouldn’t be cached. The filesystem also exposes plenty of statistics and tunables for those who need to measure the performance of their system. Failure of the cache disk in writeback cache mode results in data loss. However, you still get a saving throw with fsck as bcache keeps the filesystem data on the backing store in its correct order. I’m very optimistic about bcache, I think it’s a good option for those rolling their own hardware and in the cloud and part 3 of this blog series will go through a bcache setup in the cloud.

Linux’s LVM also surfaces a device mapper target called dm-cache that provides similar caching on top of logical volumes. It’s configured through the typical LVM command line tools and is shipped with most kernels. dm-cache offers writethrough and writeback modes and the ability to safely stop caching on a logical volume. For whatever reason it seems that lvmcache is not nearly as popular as bcache. It’s not that much more complex to get started with lvmcache over bcache but I suspect the overhead of having to wade through the massive LVM documentation and familiarize yourself with LVM terminology is enough to keep many away.

ZFS has ARC, its own sophisticated cache algorithm that heavily ties into the overall design of ZFS. Writes are initially kept in RAM (ARC) and asynchronously committed to disk. L2ARC periodically grabs the data most likely to be evicted from ARC and persists it to its own disk. Later reads that result in an ARC cache miss can then be read out of L2ARC if the data exists there. This is similar to writearound caching in that it’ll only cache previously read data and the failure of the L2ARC disk doesn’t risk data loss. Your workload only benefits from L2ARC when the working set of hot reads is larger than the size of your ARC cache - not because L2ARC is bad, but because you just don’t need it until ARC is exhausted.

Another ZFS feature that comes up when discussing ZFS performance is the ZFS Intent Log, or ZIL. It’s a journal used to guarantee the durability of writes of filesystem metadata and user data and is not a cache. To improve the overall speed of writes and reduce the I/O load on the main ZFS disks the ZIL is often moved to a fast SSD disk in a configuration called SLOG. Even in this configuration the ZIL is still not a cache: it is only ever read from when the filesystem has an unclean shutdown and has to be restored to a consistent state. It comes up because it is also relevant to users looking to improve their ZFS performance in the same way that caching is a way to improve ZFS performance. By adding a SLOG you get the improved write performance of writeback caching but only the improved write performance, and with L2ARC you get improved reads of cache hits but only for reads that were cache hits. To take advantage of SSD read and write performance in ZFS you have to add both SLOG and L2ARC to your filesystem, requiring at least one SSD for L2ARC and two for SLOG.

I get the feeling that ZFS was really intended for general purpose filesystems and filers with general purpose workloads. The design is optimized for write durability whenever it can (a good, safe default!) and does a good job getting there. A dedicated ZFS appliance can soak up all the RAM you can throw at it and can continue to scale with separate, additional disk clusters for the ZIL and the L2ARC. And even if you were to put databases or other applications that carefully micromanage their writes onto a ZFS filesystem you’ll still be able to get SSD write improvements from an SLOG on a SSD. But suppose you had a database on ZFS and you wanted to keep the entire index on fast storage. This seems like a good use case for L2ARC as the size of your indexes on a non-trivial database will exceed the size of RAM. Because L2ARC can only cache data previously read you’d have to come up with some sort of cache warming to get your index on the SSD, and only until very recently could L2ARC caches persist across clean system restarts. It makes more sense to put the indexes on their own tablespaces that are explicitly located on SSD disks. ZFS gives you the tools to do this but there is no real set it and forget it architecture like with a true writethrough SSD cache. The combination of ZFS and Linux also seems to be geared towards filers. The Linux kernel still maintains its normal page cache alongside ARC and the same data can be cached in both places redundantly. There is also the risk that Linux soaks up cache hits first in a large page cache and ARC rarely gets a chance to grow. Systems that exhibit either behavior have to set a large, minimum ARC size to keep the page cache small and force hits into ARC - something you can really only get away with on a device dedicated to storage like a filer.

Btrfs does not yet have its own caching functionality.

For the next and final blog post in this series, I’m going to walk through what bcache looks like in the cloud and try and get some benchmarks.