If, for whatever reason, you need a spreadsheet with all of the mailing addresses of every sheriff's office in North Carolina (that's 100 sheriff's offices for all 100 counties in North Carolina), I have put together a spreadsheet of mailing addresses for every North Carolina sheriff.
Here's the same thing for a spreadsheet of mailing addresses for every South Carolina sheriff.
In the previous blog post I went over the design of storage systems and how they protect themselves from data loss in the face of disk failure or unclean shutdowns. If you’re putting together the hardware yourself modern Linux systems and modern hardware give you endless cost, performance and durability tradeoffs for whatever system you want. But 2020 also gives us the cloud, and storage systems in the cloud whose durability far exceeds whatever you could build yourself. Cloud providers also have the scale to amortize the insane costs of these exotic, durable, and yet still performant system designs across their broad user bases, bringing down their prices and making them accessible to us mortals. In the year 2020, if you’re in the cloud, durability just isn’t a concern to you so it is time to start focusing on performance.
Caching basics in 2020
2020 also brings affordable, reliable and speedy SSDs, and if you’re designing a system yourself you want to have some way to take advantage of SSD performance. Building everything out of SSDs isn’t an awful choice, it’ll be simple and probably won’t even cost that much. However, the expected usage of most storage systems will have some bulk of data rarely read or written giving an opportunity to reduce marginal costs with tiered storage. If you have a way to keep cold data on slower but cheaper magnetic disks and can use SSDs when appropriate this tiered storage design gives you the speed benefits of SSDs when you need them the most and the cost benefits of magnetic disks.
Tiered storage introduces complexity back into the storage system, increasing operational overhead and the risk of data loss. If you can implement tiered storage at the application layer you have some of the best tools available to you to tame that complexity. For example, a filer could keep some file shares known to be cold, like backups, directly on magnetic storage and others known to be hot directly on SSD storage. A database could have tablespaces for slow and fast storage and assign cold and hot tables to each. You could keep table indexes in fast storage with reduced reliability knowing that disk failures in the fast storage won’t cause unrecoverable data loss. These are all simple systems, with well-understood failure modes and controlled operational complexity. They also give you the most benefit from SSD performance because their simplicity skips the CPU and I/O overhead introduced by more complex systems.
But what if your application layer doesn’t give you tiered storage? What if you can’t afford the operational complexity required to maintain it yourself? Delegating where the data goes initially is easy, but moving it around is harder. And assumptions can change, with cold data becoming warm, and you need continuous feedback to make and question your storage decisions. Caching is the next choice, where the system makes decisions on where to keep data automatically. It’s usable anywhere and can bring its benefits to every storage system but it comes at the cost of potentially suboptimal usage of the fast storage and a potentially increased risk of data loss.
Caches typically implement multiple caching algorithms, giving you more tradeoffs between speed, durability and cost. Writeback caching has all data written to fast storage first and later copied asynchronously to the slow storage. It gives the best performance as cache reads can use data from the newest writes immediately and writes go at the speed of the fast pool. But should your fast storage system suffer data loss before the asynchronous copy finishes you’ll have permanent data loss. Your system performance also hinges on the intelligence of the writeback cache algorithm more than other algorithms. Imagine a system that does backups - if the writeback cache is unable to determine that backup writes are cold and shouldn’t fill up the cache you’ll wind up evicting all the useful data for extremely cold data. This logic can’t be simplified or formalized for general use and requires tuning and measurement.
Writearound caching has writes go directly to slow storage and only reads make use of the cache. It’s a blunt tool, unable to improve write speeds, and unable to cache recently written data, but it never has to implement the dark art algorithms like writeback does to get good performance against pathological workloads. You also won’t lose data if your cache disk dies. There is also writethrough caching where writes are done to both fast and slow disks at the same time, improving the cache hit rate of recently written data but still limiting your write speed to the speed of the slow disk.
The previous blog post introduced the RAID write hole and a few techniques to close it. If you’re building your own storage system you are probably considering RAID and using one of those techniques to improve the durability of the system. Journaling is a well-supported and reliable way to close the write hole and under the hood these journals are often just writeback/writethrough/writearound caches, with all of the benefits and drawbacks as described above. In other words, a RAID array of magnetic disks with large enough SSD journals is also a RAID array with SSD cache. Journal disk failure is still a threat, putting the less aggressive cache algorithms into strong consideration, because an unfortunate disk failure in a writeback journal results in losing all data on the entire array.
The storage available to you in the cloud shifts your priorities the other way. Durability is a given as discussed earlier and you’re now looking to squeeze what performance you can out of your design. More aggressive caching algorithms like the writeback cache are now in scope and should be seriously considered, even for ordinary cloud setups.
Finally, I want to cover the Linux buffer and page caches, just for completeness. Your application's I/O hits these first and all of the other cache systems later. If a cache hit is done here any other disk caching code is never run at all. A 2017 summary of the history, current state and future of these Linux caches is on LWN. The page cache carves system memory up into equally sized pages, typically 4 kilobytes, and maintains attributes on each page's uses. A page could cache a region of data on a disk, meaning that the page's contents directly correspond to 4 kilobytes on a disk somewhere. A read request fills that cached page and future reads can be serviced by that page. A write to the disk updates the cached page and marks it as dirty, signalling the kernel to rewrite the corresponding data on the disk at its leisure. The buffer cache is a legacy API, internal to the kernel, that is implemented on top of the page cache. This API works with smaller units of data, called blocks, that are mapped onto relevant page cache entries for each file. Reads and writes to blocks are passed through to the page cache subsystem. The buffer cache was only separate from the page cache in much older kernels. The modern API for I/O is built around I/O requests and the bio struct, instead of stateful buffer cache blocks, and allows for I/O requests to be batched, reordered, or executed in parallel.
Disk caching in 2020
So what cache options are available to us on a modern system? As mentioned in the previous section, you can always just buy a bunch of RAM and you’ll get some in-memory caching provided by the kernel's page cache out of the box. If you’re using a RAID array there are ways to use the array’s journal as a cache. Modern kernels come with more exotic tools, though, and I will cover a few of them.
bcache, introduced in 2010, ships with most kernels. To use bcache you format your backing (slow) disk or logical volume with bcache and create whatever filesystem you want on top of the resulting bcache device. You then enable caching on the backing device by formatting a cache device (your SSD) and attaching it to the backing disk via sysfs. bcache implements writeback, writethrough and writearound caching, with the ability to switch between cache algorithms at runtime, and has implemented its own code to identify read patterns that shouldn’t be cached. The filesystem also exposes plenty of statistics and tunables for those who need to measure the performance of their system. Failure of the cache disk in writeback cache mode results in data loss. However, you still get a saving throw with fsck as bcache keeps the filesystem data on the backing store in its correct order. I’m very optimistic about bcache, I think it’s a good option for those rolling their own hardware and in the cloud and part 3 of this blog series will go through a bcache setup in the cloud.
Linux’s LVM also surfaces a device mapper target called dm-cache that provides similar caching on top of logical volumes. It’s configured through the typical LVM command line tools and is shipped with most kernels. dm-cache offers writethrough and writeback modes and the ability to safely stop caching on a logical volume. For whatever reason it seems that lvmcache is not nearly as popular as bcache. It’s not that much more complex to get started with lvmcache over bcache but I suspect the overhead of having to wade through the massive LVM documentation and familiarize yourself with LVM terminology is enough to keep many away.
ZFS has ARC, its own sophisticated cache algorithm that heavily ties into the overall design of ZFS. Writes are initially kept in RAM (ARC) and asynchronously committed to disk. L2ARC periodically grabs the data most likely to be evicted from ARC and persists it to its own disk. Later reads that result in an ARC cache miss can then be read out of L2ARC if the data exists there. This is similar to writearound caching in that it’ll only cache previously read data and the failure of the L2ARC disk doesn’t risk data loss. Your workload only benefits from L2ARC when the working set of hot reads is larger than the size of your ARC cache - not because L2ARC is bad, but because you just don’t need it until ARC is exhausted.
Another ZFS feature that comes up when discussing ZFS performance is the ZFS Intent Log, or ZIL. It’s a journal used to guarantee the durability of writes of filesystem metadata and user data and is not a cache. To improve the overall speed of writes and reduce the I/O load on the main ZFS disks the ZIL is often moved to a fast SSD disk in a configuration called SLOG. Even in this configuration the ZIL is still not a cache: it is only ever read from when the filesystem has an unclean shutdown and has to be restored to a consistent state. It comes up because it is also relevant to users looking to improve their ZFS performance in the same way that caching is a way to improve ZFS performance. By adding a SLOG you get the improved write performance of writeback caching but only the improved write performance, and with L2ARC you get improved reads of cache hits but only for reads that were cache hits. To take advantage of SSD read and write performance in ZFS you have to add both SLOG and L2ARC to your filesystem, requiring at least one SSD for L2ARC and two for SLOG.
I get the feeling that ZFS was really intended for general purpose filesystems and filers with general purpose workloads. The design is optimized for write durability whenever it can (a good, safe default!) and does a good job getting there. A dedicated ZFS appliance can soak up all the RAM you can throw at it and can continue to scale with separate, additional disk clusters for the ZIL and the L2ARC. And even if you were to put databases or other applications that carefully micromanage their writes onto a ZFS filesystem you’ll still be able to get SSD write improvements from an SLOG on a SSD. But suppose you had a database on ZFS and you wanted to keep the entire index on fast storage. This seems like a good use case for L2ARC as the size of your indexes on a non-trivial database will exceed the size of RAM. Because L2ARC can only cache data previously read you’d have to come up with some sort of cache warming to get your index on the SSD, and only until very recently could L2ARC caches persist across clean system restarts. It makes more sense to put the indexes on their own tablespaces that are explicitly located on SSD disks. ZFS gives you the tools to do this but there is no real set it and forget it architecture like with a true writethrough SSD cache. The combination of ZFS and Linux also seems to be geared towards filers. The Linux kernel still maintains its normal page cache alongside ARC and the same data can be cached in both places redundantly. There is also the risk that Linux soaks up cache hits first in a large page cache and ARC rarely gets a chance to grow. Systems that exhibit either behavior have to set a large, minimum ARC size to keep the page cache small and force hits into ARC - something you can really only get away with on a device dedicated to storage like a filer.
Btrfs does not yet have its own caching functionality.
For the next and final blog post in this series, I’m going to walk through what bcache looks like in the cloud and try and get some benchmarks.
It is the year 2020 and we still don’t have great answers to data durability in the face of unclean shutdowns. Unclean shutdowns are things like power outages, system faults and unlucky kernel panics and preventing data loss when they happen is a hard problem. I’m going to talk through a few ways these can cause data loss in this blog post but you can probably come up with new ones on your own - exotic failures that no system will ever handle correctly. In the year 2020, the conventional wisdom is still true. Always take backups, and RAID is no substitute for backups.
How do we deal with these unclean shutdowns? If minimizing data loss from unclean shutdown is in scope for your system you should put serious consideration into putting time and money into functionality that minimizes the chance of unclean shutdowns. If you fear power outages you invest in battery-powered uninterruptible power supplies. If you fear kernel panics you should run stable distro kernels with simple, widely tested software like ext4/xfs and mdraid.
Even in the face of these unclean shutdowns there are some systems that do a better job at dodging fate, getting lucky and persisting good data than others. There’s RAID level 1, a classic simple design: data is written to two disks simultaneously and writes are acknowledged when both disks finish writing. If the application layer is properly using write barriers and not progressing until the write completes you’ll not be committing transactions and losing data afterwards. But if you get unlucky with an unclean shutdown gone very wrong you could have some exotic data loss. The recovery mode for an unclean RAID 1 array as done by md is to copy the unclean blocks from the first available device in the array to the rest of the drives. Should the unclean shutdown cause any issues with the data written to that first drive you’ll have undetected corruption.
RAID levels 3, 4 and 5 introduce parity data that can validate disk content and recreate corrupted or missing data from bad or missing disks, avoiding some of the corruption problems with RAID level 1. But they have their own weakness - the famous write hole, where data previously written safely to the array is damaged because of an unclean shutdown during a later write. These RAID levels write data out as stripes, larger chunks of data that can potentially contain unrelated older data alongside the new data you’re trying to write. Any problems that happen while writing out a stripe risk glitching the entire stripe and result in data loss.
Linux’s md has picked up a few solutions to closing the write hole in recent years. It now supports a journal, where the array’s write status is continually written to a separate disk. If an unclean shutdown happens it is now possible to identify the stripes in the array that were mid-write and unwind or correct the array’s data by replaying the transaction log on the journal disk. And the journal doesn’t just close the write hole problem, it also can help speed up array writes by optionally using the journal as a writethrough cache. In this mode the actual data in the write request is first committed to the journal disk and only later flushed to the array. If you use a speedy SSD for your journal disk you now get much faster write speeds and the same resistance to individual disk failure of a RAID array.
But if you have to use a separate disk for the journal aren’t you back to square one? What happens if the journal's disk fails, either from normal operations or as part of an unclean shutdown? If the journal disk fails you will have data loss. RAID level 1 is certainly a solution here, and it seems like the failure modes for RAID 1 are slim enough, especially for modern SSDs, that it makes a real improvement to the overall reliability of the system. In recent times the improved software support and hardware availability of persistent memory (PMEM) means that you could build a system with a fast, reliable journal disk that survives power outages. On Linux, PMEM hardware is exposed to the system as an ordinary block device, can be put into a RAID array of its own, and used by mdadm as a journal device, giving you a journal resistant to the failure of any one PMEM DIMM.
md also gained the Partial Parity Log in recent years, a new technique for writing RAID level 5’s parity data. It incorporates parity data about the previous state of a stripe into the write done for the updated state of the stripe, allowing for the previously written data to be recovered in case of failure and closing the write hole. Extra data must be written and read, slowing array performance, and there is still an opportunity for data loss with newly written data as the PPL only calculates its extra parity data for old data in the stripe and not the new write. But on the flip side the PPL doesn’t require a separate journal disk as it writes the extra parity data directly into the array, sidestepping a potential I/O bottleneck in the journal disk and any durability issues with it. But this extra parity data computed by the PPL increases the I/O and CPU load of writes, lowering write performance overall.
The ZFS filesystem, known for its rampant layering violations, actually does a pretty good job at taking advantage of its situation and providing a durable filesystem in the face of unclean shutdowns. ZFS on Linux is mature, stable, and used by many. Its RAID-Z design closes the write hole by having variably-sized stripes, shrunk down to the size of each write. Previously written data won’t be included in a stripe and thus can’t be damaged during writes of new data. The filesystem’s metadata is also mutated with copy-on-write semantics instead of in place, so all unclean shutdowns can be rolled back to a previously good state. It’s just a really solid design and it delivers robustness without any of the elaborate workarounds described earlier. These design choices come with the tradeoff of decreased performance but often not large enough to turn use cases away from it. If you are going to be maintaining hardware yourself ZFS should be on your radar.
Finally, btrfs is still hanging around. I remember reading about it on lwn.net in 07, my freshman year of college, and thinking that this was the filesystem of the future, comparable with ZFS on features but GPLed and developed in-tree. But as the years went by the team never really pulled it together, despite corporate backing from a few places, and btrfs has left a trail of users burnt by bugs and big unfixed issues even in cutting-edge kernels. This mailing list post was posted in June of 2020 and lists a shocking number of workarounds for data loss bugs. I really do not trust btrfs to do well in any unclean shutdown.
Durability in the face of unclean shutdowns and other issues is an important thing to have, but the cost/performance/durability tradeoff can often lean in favor of reducing the probability of unclean shutdown with battery backup instead of complicating the design of the storage system. Many businesses and individuals also no longer manage their own hardware and operate in the cloud. The cloud block devices used for bulk storage are internally redundant and extremely durable. Unclean shutdowns rarely happen as cloud hardware runs in datacenters with uninterruptible power systems and live migration of server instances. In the cloud you have endless hardware available to you and it's extremely durable, swinging the system design pendulum away from durability and back towards performance. In the next post I’ll explore the performance tradeoffs of storage systems.
The big news of this year's WWDC is the announcement that Apple's moving desktop Macs onto Apple Silicon, ARM CPUs manufactured by Apple and the first real push at ARM desktops in the industry.
For Apple, there's a lot of risk here, but with that risk comes lots of opportunity. The transition could go very poorly, harming Apple's profits and market share. Rosetta 2, the binary translation layer Apple is shipping with macOS 11, is an unknown. It may not work well, it may have an unacceptable performance penalty for users. The transition to the new architecture could go poorly for developers, delaying the availability of native ARM software and further discouraging users from upgrading. There could be an unacceptable regression in CPU performance. And if there are issues, the longer Apple takes to get on top of them the more users they'll lose in the transition.
But there is lots of opportunity for Apple to do well here. Apple Silicon's performance, energy use, or the ratio of the two could beat Intel's hardware, giving Apple hardware a market advantage. It seems very likely that energy use will be greatly improved over Intel. Tight coupling of hardware and the operating system, possibly introducing more information about trusted execution contexts to the CPU, could open the door to innovative new fixes to Spectre-type bugs. Even if Apple's CPUs never beat the performance of Intel's CPUs a clever new design could turn into an advantage Intel can't replicate and narrow the performance gap between Intel and Apple.
Where do I see this going? I expect PC hardware manufactures to start experimenting with ARM desktop machines today. These machines wouldn't be usable for anything beyond Linux and bootstrapping efforts right now. Instead, they give the hardware manufacturers experience with designing and manufacturing these systems in the event that ARM does come to replace Intel generally across the industry. And if that does happen, we are in for exciting times. I would say it'll take about a year, starting today, for the first ARM desktop hardware to come out. Then we have to see if Apple's gamble paid off, which won't be clear for another year or two. Once that happens, the race is on to ditch Intel and I figure Microsoft is going to need two years to get something shippable. The world outside of Microsoft will take another year or two beyond that to start shipping ARM software. The exact timelines are up in the air but if the industry switches to ARM on the desktop you're looking at years of stagnation in the PC world while Apple has years of lead time on ARM.
As for the Hackintosh, the conventional wisdom is that Apple will stop shipping Intel versions of macOS in three to four years, the new ARM macOS will have hard requirements on Apple-only hardware and that will be the end of Hackintosh as a platform. I think you'll never have a fully compatible Hackintosh, there will definitely be Apple-specific hardware required for full functionality. But not all users require all functionality so as long as Apple hasn't wedded the operating system to Apple-exclusive hardware there is hope for a usable ARM Hackintosh. I don't think secure boot, signed code and the like are going to be a threat to Hackintosh. Those are useful parts of the security model on a real Mac, but on a Hackintosh where you control the boot process it'll be possible to bypass them without impacting system functionality. The real threat to Hackintosh might look like important hardware that can't be emulated efficiently or a toolchain that deeply integrates with Apple Silicon-specific functionality, requiring binary patching of the entire system, not just a few kernel drivers.
It's interesting that the Intel to ARM transition is going to be the reverse of the PPC to Intel transition in some ways. Before, Apple was coming onto a platform with extensive hardware, driver support and lots of understanding around how those all work. Because ARM on the desktop is so new there is a lack of all of that, complicating efforts at bringing up a Hackintosh system that isn't just a tech demo. Drivers are going to be all new, and graphics drivers are going to be a particularly sore point. Reliable, performant graphics drivers are required for most desktop PC use cases and none of that is going to exist for years. If we're going to see a stagnant period where the PC industry spends a few years transitioning to ARM the Hackintosh will have an even longer stagnant period.
Personally, I think we'll have some ARM Hackintoshes. Apple will treat the Hackintosh on ARM like they treat the Hackintosh on Intel: they'll mostly look the other way, not chase after the communities and not heavily integrate the operating system with Apple-specific hardware. I don't know if ARM will take over on PC desktops but I don't think it's an impossible outcome. Microsoft can pull it off, they'll have to make some platform users unhappy, it'll take them a while to pull off, but it can be done under threat of losing their entire PC business. But even if this all comes to pass I think we are definitely looking at a dark age of Hackintosh lasting as long as five years or more as people work on new hardware, drivers and hacks.
As for the official Apple hardware I don't think Apple Silicon will compete against AMD or Intel on performance right away, if ever. But I don't think the outcome where Apple gets a performance edge after applying creative shared execution environment mitigations is impossible. For most users the ARM laptops will be a nice improvement over past models: better battery life, roughly the same CPU performance, maybe some thinner hardware and an even wider gap between what you get from Apple hardware and what you get from PC hardware. And I think you will see Microsoft making an ARM "Windows", although who knows if it'll be a full port with backwards compatibility like Apple or a subset of today's Windows. I also don't know if they'll be very reactive (starting work now) or reluctantly reactive (starting work in a year or two when Apple sets the industry direction) but I think the forward-thinking leadership Microsoft has now means they're already taking this seriously.
bjoern is a WSGI-compatible web server for Python web applications. It supports both Python 2 and Python 3. Unfortunately the authors don't provide any binary packages but I've created compiled binary wheels for bjoern for Python 2 and 3 for Debian Jessie, Stretch and Buster and Ubuntu Xenial, Trusty and Bionic.
To install the package:
- Install libev with
sudo aptitude install libev4
- Find out the release you have installed with
source /etc/os-release; echo $PRETTY_NAME
This will output a line of text like
Debian GNU/Linux 9 (stretch). In this example
stretchis the name of the release. Write that down somewhere.
- Source your virtualenv. You're using a virtualenv, right?
- Browse to the bjoern releases page and find the wheels for your release. In my example, they would be
- Download the Python 2 (cp27) or Python 3 (cp3X) wheel for the version of Python your project uses.
- Rename the file to remove the release prefix. In my example,
pip installto install the wheel in your virtualenv. In my example, I would do
pip install bjoern-2.2.3-cp35-cp35m-linux_x86_64.whl
Note that the renaming step appears to be optional; pip will throw some errors when you try to install the wheel but the bjoern wheel appears to be installed correctly anyway.
netatalk hasn't been packaged with Debian for a few years now. If you don't want to get your hands dirty with compiling the package you can download compiled (binary) deb packages of netatalk version 3 from my GitHub releases. Right now I have packages for netatalk version 3.1.12 and 3.1.11 for for Debian buster, stretch and jessie.
To install the packages:
- Find out the release of Debian you have installed with
source /etc/os-release; echo $PRETTY_NAME
This will output a line of text like
Debian GNU/Linux 9 (stretch). In this example
stretchis the name of the Debian release. Write that down somewhere.
- Find the appropriate packages for your Debian release on the releases page. Out of the several packages for your release you'll need to download the
netatalkpackages. So for my example, I would download the files
- Download both .deb packages to your local machine through the browser or with
- Install both packages at the same time with
dpkg -i. In my example, I would do
sudo dpkg -i ./stretch_libatalk18_3.1.12-1_amd64.deb ./stretch_netatalk_3.1.12-1_amd64.deb
I had the desire to go to Japan, and especially Tokyo, because I wanted to spend my next trip in an urban area doing urban things. I felt that the last few vacations had a focus on exploring nature and natural wonders and I wanted to try going the other way. Tokyo is a massive urbanized area with the largest urban population in the world and endless things to do so it fit my goals well.
9/2, the afternoon:
I arrive in Narita. Japan is overcast and the foliage is definitely unfamiliar - trees are dark, dense and the foliage is a bit weedy. The air isn't North Carolina humidity but it isn't exactly comfortable. The airport takes a good chunk of time to navigate, both getting through passport control and eventually acquiring my luggage. But then I am set free into the mall-like interior of the airport. I immediately go to change some cash on hand to yen and get a pasmo so I can get out of here.
My first attempt at riding the train was a bit painful. I didn't find Hyperdia very useful, it was more helpful to use Google Maps with its GPS location tracking and address routing. I went off to Meguro and my first AirBNB apartment. I was able to get on the right train but I didn't understand how Google maps communicated trains/platforms so I wound up taking a local train and had to transfer someplace underneath downtown Tokyo. Navigating the station around Meguro was also painful, it was my first introduction to how disorienting it can be to come out of a train station in Japan if you don't have a mental map of the area already. But I eventually make it to my first place to stay. The apartment is small and cozy and it has two low beds pushed together. There is a coin operated laundry machine outside. The neighborhood is very residential and maybe even slightly suburban - the houses are all multi-family dwellings for the most part but it's still distinctly quiet and cosy.
After unpacking my initial stuff I then head out to tackle Ebisu. Turns out my apartment is pretty much right in between the start of Ebisu and Meguro so it's a quick walk over to the 7/11 where I get my first yen withdrawal from the ATM. I then start wandering the commercial district nearby, just poking around in the department stores. I've got a goal to pick up some toiletries but this place seems too upscale to get what I want. I also get my first chance to wander into the basement of the department store which is always full of food. I look at the things and eat a taiyaki, one of those bean paste-filled waffle fishes. It's quite good, the bean paste tastes half bean, half Nutella. It's delicious hot. Of course I pay with a 10000 yen bill for my 120 yen snack because that's all I have on hand. I keep walking, passing the "Americabashi Park", a big pole with a star on it built in St. Louis. There's a weird collection of people in old-fashioned dresses singing, forgot exactly what that was about. I keep going into the Ebisu station area where I find a craft beer bar and have some good beer and sausages. My first attempts at ordering in Japan aren't totally abortive and I'm getting the hang of it. After a few beers and a few hours I then head off to Shibuya to buy stuff at their Don Quixote. I'm aware now that I could probably have just kept hitting up convenience stores and smaller department stores in Ebisu to get this stuff but I wanted to give this a shot. It's a good thing that I did, because it turns out that Shibuya on a Sunday night is totally packed. People everywhere, the scramble is full, the area around Hachiko is packed. I keep going down the streets to the store and it is just completely full of people. I'm learning that Shibuya and Don Quixote both attract a young crowd and it seems like Sunday night is full of teenagers doing shopping and hanging out. I get my stuff - it's a bit pricey, but I get it and get out of there eventually. I'm also immediately struck by how plain the Japanese sense of fashion is. Uniqlo style plainness is everywhere, on everyone, and I am looking at these women with long, plain dresses and pants and it's reminding me of fundamentalist Mormons more than anything modern. Finally, I return to my home and get many solid hours of CPAP sleep.
Up and out to Ikebukuro. This was bumped earlier in the trip because the Gudetama cafe is there and was closing for the year in a few days. Thought I'd check it out. I make it onto the street in Ikebukuro and start wandering around. I find a cafe, a chain I believe, and have breakfast and coffee there while it's cloudy, foggy, wet and humid outside. I then wander around what I'd call "North Ikebukuro" for a little bit, it's a very seedy corner of the town. I see a ton of people lined up for a large, faceless building with a young woman service worker organizing the line and handing out tickets. Later, when I went past it I saw it's a Maruhan, a giant pachinko parlor. I'm amazed to see that the line was wrapping around the building while it was only 9:30 with an opening at 10:00. Not sure why the crowd didn't have work on a Monday morning but it was all men. Next, while wandering into the main part of Ikebukuro I wander into the Animate tower and spend some time going through every floor there. It's all content marketed towards women and I'm just about the only man in the store. There are lots of little collectibles and toys and trinkets and not so much figurines everywhere like I'd later see in Akihabara. I then wandered around the central commercial district in Ikebukuro for a while. I went through a SEGA arcade from top to bottom. The only game I played was Taiko no Tatsujin which was pretty fun but mostly I was there to check out all the different games. There was a Pokemon RPG that seemed to be pretty popular, a horse racing game where 20 or more people could all sit in armchairs around a big screen and race horses together, an entire floor of photoshop booths marketed towards women and a floor of music/rhythm games.
It was getting towards lunchtime so I went to the Ikebukuro Seibu, built on top of the train station, and had lunch at the Gudetama cafe. The food was OK, it was creative and cool but Japanese curry is a bit too much for me. I then spend a chunk of time wandering the Seibu, my first time really exploring one of these giant Japanese department stores. It's a lot of women's fashion so I don't spend a ton of time here but notably I see a $7000 wooden ping-pong table. I also make it down to the basement floors. This department store has one floor for gift-type food and the floor below that is a grocery store which I understand is pretty typical.
Finally, I'm back out on the streets, wandering Ikebukuro again and browsing more shops. I find the big store whose top floor is all Gundam models for sale which is neat. Make my first visit to a camera/electronics store. I stumbled into a video game store (pretty sure it was the Ikebukuro Super Potato). I didn't realize it was a landmark store - the entrance in Ikebukuro is a tiny door up to a 2nd floor shop. The store was packed with retro games, all individually shrinkwrapped, and I was impressed by the kind of resale market that must exist to support all this trade in used, vintage games. Also, I stick my head into a porn shop. It seemed to be a place where you rent the porn and a room and you got a space with a big TV, air conditioning, optionally a beer and had some time to relax, watch the video and have a wank. I didn't buy anything or rent anything but I was shocked at the obscene content right on display - a front-facing DVD on the shelves was literally just a close-up of a gaping asshole with nothing else on the box. There was also a sizable collection of poop videos.
I head across Ikebukuro back to the main, central part, browsing many stores along the way, and over to Sunshine Road. There is a bunch more female-friendly manga for sale here but I didn't spend a ton of time browsing it. It has the nickname of Otome Road - Virgin Road - a pun on the nerdy women who shop there. I head over to the large mall but find everything there is closed on this Monday for some reason I can't figure out. I am committed to coming back again later because it seems like there's a lot here and they have a Pokemon store I want to check out. On the way out of Sunshine Road I stop at the Mocha Lounge cat cafe - it's a very clean, professional place with a modern American interior design. The cats aren't super playful, they're mostly just sleeping, but one guy is running around and freaking out. There is lots of space to sit and read and an entire library of manga to borrow. I take off after 30 or so minutes and wander back to Ebisu.
In Ebisu, I go for ramen for dinner right off of the subway. There is a highly rated ramen place nearby called Afuri and it's very, very good. It has a salty broth and you can definitely get a bit of that salty taste in it. I guess it's a more modern, hip style of ramen. I had a Kirin beer with it, an incredibly tasty lager. As the trip goes on I find out that there are lagers available in most places and I wind up drinking one with most meals. Kirin beer is great, the Asahi stuff is good (another dominant soft drink manufacturer) and Sapporo is straight trash. I then start wandering Ebisu for bars or some place to spend the rest of the evening and I wander the neighborhood looking at stuff. At this point disaster strikes: I am walking (well, power walking) while looking at Google Maps and I plow into a foot and a half tall concrete wall that is sticking out into the sidewalk. I get a pretty big gash cut open on my leg, the old woman who sees me tumble is frightened and says something , and the phone in my hand goes straight into the ground, cracking the screen. Things are still functional, thankfully, but I can see that the phone is in a precarious position. At this point I'm pretty shaken up, I decide to head home, defeated, and go to bed.
When it comes to accessibility and safety Japan seems like America in the 50s: no handicap access anywhere, lots of environmental hazards, tiny steep stairways and smoking is allowed indoors. Interesting to see the differences in culture - in America people would be straight up offended if you didn't have handicap access, myself included.
Jet lag had me up super early, and I went down to Meguro to get breakfast. I went into Sukyia, a chain gyudon restaurant, and ate that. It was pretty good and served fast, glad to check it out. Would be interesting to try and reproduce the meal at home, it seemed very practical and tasty. I then went down to Shibuya at around 7 AM for reasons I don't understand (it was supposed to be a Shibuya day?) and hung around the Hachiko statue from 7 to 8 AMish standing under an umbrella and watching the commuters. The place was incredibly quiet, partly because people weren't up yet and partly because it was raining out. I have some insane photographs of the scramble, the Hachiko square and the station entrance that are almost completely devoid of people. I then went around and found a nice, quiet cafe and chilled out on the 2nd floor, watched the quiet and rainy street below and planned the day.
My first stop was halfway across town at the water treatment museum. It was decently interesting, and it set a bit of context for the city of Edo and how it was planned, built and developed. It was also pretty quiet, with me arriving right as the place opened. On the way over I wound up in Kanda by accident. I wandered around that neighborhood before realizing I was going to the wrong place but it was cool to explore it in the rain. It was a neighborhood with lots of small streets and izakayas and I'm sure it's a cool place to visit at night although I ultimately never had the chance.
Next on the schedule was the Edo museum, also on that side of town. I also realized that I was on the same side of town as Akihabara and it was bound to have plenty of places to repair my cell phone. It would also work out that I could go and walk through the museum while they spent a few hours on the repair of my phone. So I walked over to Akihabara and found the place, it was about $400 to get the screen replaced. They took my phone and I wandered off without the aid of any map to find the Edo museum. Akihabara was decently busy and had the largest number of white people I had seen so far on this trip. There are definitely different demographics for different neighborhoods in Tokyo and I'm starting to get a feel for that.
Before going into the museum I wander around it trying to find a restaurant for lunch. I didn't have my phone so this is the one meal where I was flying blind. It turns out that I actually wandered into a Chinese restaurant named 蜀食成都. I had to ask the waitress what to recommend, she pointed to the first thing on the menu and it was incredible. It was a ramen dish but with a creamy sauce and a delicious amount of heat from chili oil. It was garnished with finely ground beef. It turns out this is a hybrid Japanese - Szechuan meal called tantan-men (a form of the Chinese dandan noodles) and it was delicious. Someone left a Japanese language review and you can see it on tabelog along with pictures of the same tantan-men dish that I ate.
The Edo museum was cool. This was about the history of the city of Tokyo, formerly called Edo. The main part of the museum was historical artifacts from the Edo period which was when the shogun leaders of Japan ruled the country from Edo. There was a second exhibition which covered Tokyo and the Japanese way of life from WWII to the present. The first part of the museum was cool with lots of displays and artifacts from the giant castles that the shoguns had built for their rule. It also had lots of background on what life was like for the people of the time, how the government was organized and what services it provided to the people of the city. Fires were a big problem for the wooden city and there were these military-like fire crews organized all over town. They had these cool, big white standards they'd carry when charging in to fight a fire and there was a neat ukiyo-e which really captured the military aspect of these firefighter crews. There was a section of the exhibit full of ukiyo-e and other printed materials from the time period and the one thing that stuck in my mind was a ukiyo-e of a plain woven grass mat in a temple that had a single rice plant growing out of it. The museum explained that many of the printed artworks from the period were like this: elegant reproductions of slightly out of the ordinary but mostly unremarkable events like that one and the population at the time loved that sort of thing.
I also had time to reflect on Edo and the Japanese mindset. The city was mostly constructed by the shogun for the purpose of running his administration. Everything about the design and construction of this city had this sense of duty to it: we need to have a city that will serve the shogun's need for governance, and the people who live in the city need city services to live here. And I think that carries on to the present with how YIMBY, dense and urban the city has become. The most important priority is to have a city that functions. There is little consideration given to traditional NIMBY complaints, the city just keeps plowing forward and urbanizing. And because of that I don't think Tokyo's urbanization is a good goal for most other cities, it's uniquely urbanized because of its unique culture and history.
Also, it's interesting to think about how exocitized Edo era ukiyo-e and other artwork is done. We think of it as some ancient, beautiful artwork, but it was just ordinary printed artwork for newspaper-esque distribution. Compare that to anything printed or painted in antebellum North Carolina, roughly the same time period, and you wouldn't have that viewed with the same reverence.
The post-WWII part of the museum was also very interesting although I don't remember a whole lot about it, it was mostly an overview of the increasing consumerism in their society.
Made it back to get my phone, everything works. The aftermarket screen is a bit wonky but I'm functional again. On the way out I was idly standing in the street, getting my bearings again, and I found that I was standing next to a sign that advertises some sort of service where you get to watch a schoolgirl through a one-way mirror. Sketch.
Finally, I ride back to Shibuya, my original plan for the day. I'm on the subway and my feet are starting to hurt from all this walking. I notice now that virtually everyone on every train has a phone and they spend all their time on the train looking at it. It's only the very old people that don't have a phone and they're often reading a book.
Back in Shibuya, I walk around and eventually walk all the way down to find a craft beer bar - goodbeer faucets. It has a nice overlook of a busy street. They have several local beers and I have some good ones, including a weak but interesting and tasty lemon beer. The couple next to me looks like they're in high school, way too young to be drinking in a bar. I also had to get through Shibuya station in the mid afternoon which is full of schoolkids for whatever reason. By the time I head out of there I'm decently drunk and I'm ready to take on Shibuya. Wandered to find dinner and eventually found an izakaya that sells kushiage - a kind of fried, battered food on a stick, called 本場大阪串カツの店 串猿. It was nice, I was drunk. Finally, I wandered around a bit more, looking for another place to go and drink, and wound up in a chain bar called Kirin City. They have decent beer, it's got a sort of diner vibe, but they also have these frozen heads where the head is a beer slushy. It was pretty good. I am fading fast, drunk and tired from being up all day, and I head back to the room in Meguro to sleep.
I forget what I did for breakfast this day but I do remember looping around my Meguro block checking stuff out like the Algerian embassy which happens to be a street over. I was up late so I see all the little kids go to school in the opposite direction of where I am going. Lots of them have this little safari hat on, I guess to identify them as little school kids. Most are unaccompanied but a few have their parents with them. They're all really shy and avoid eye contact with me which is really cute! As I'm leaving the neighborhood two slightly older girls try to jaywalk and the smaller one grabs her sister and pulls her back to get her out of the way of an oncoming car and says something harsh. I wind up laughing, it was just a funny situation to see the older kid humbled by the younger one.
Headed over to Meguro in the morning and saw the graceful Meguro Sky Garden which is a highway spiral that connects an underground expressway with an elevated expressway. They built a small garden on top of the spiral, it's cool. I then walked down the Meguro river to the main part of Meguro. It was small and pleasant, the river is walled in on three sides and is shallow but there are lots of quiet trees and small shops along the way. It's also not super busy, most everyone is at work at this point of the day and the river isn't all that commercialized anyway. I eventually make it down to the Meguro Parasitological Museum which is small but somewhat interesting - I guess an influential scientist lived here and they decided to showcase his collected work.
Made it back to Ikebukuro with hopes of exploring the Sunshine City mall again. I found a place called 燕返し and had soba there, it was pretty good although I am not sure if I was eating it correctly. I then wandered the big mall, it was somewhat interesting, and bought a bunch of stuff at the Pokemon store. The original 151 and Pikachu are still the most popular pokemon.
Finally, I head over to Harajuku, my destination for the evening, and arrive around 4:00 PM. I don't know what it was about the place: maybe it was the gentle afternoon light, maybe it was the sun out after several days of rain, maybe it was the trees (the neighborhood's boulevards had lots of large trees growing along the sides of the street which is unique for Tokyo), maybe it was the energy of all the students hanging out there after getting out of school for the day, but for whatever reason I was just in great spirits when walking around in Harajuku. It was one of the few places in Tokyo where I walked slower than the people around me as I was looking at everything along the way. Most of the architecture in Tokyo was very samey and utilitarian, this was a neighborhood where everyone went out of their way to make everything visually interesting, even the small buildings and shops. Maybe it's a way of marketing to Western eyes but I enjoyed it. I was there until after nightfall and I got to see the busy side streets which were packed with school children and definitely had a carnival feel to them. I had a realization that $10 gets you a JR train across town and back with enough left over for a soft drink. With that kind of accessibility I can see how kids growing up in Tokyo could do some serious exploring. It made me really consider how raising a kid in a big city could be a hugely positive thing over the American ideal of a quiet, small town where everyone knows each other and nature is the only thing to explore. When I was a kid I loved what freedom I had to explore and I spent a ton of time doing it so I can see the appeal in it.
Finally, I went home - my feet are dying and the blisters are TERRIBLE. I am starting to buy more of the blister bandages and waterproof bandages and although my feet look bad these really help a lot. By the end of the trip I'm not feeling any blister pain and my feet are healing quite a bit despite many long days on my feet. When I went home I headed over to the local ramen shop - it turns out there's a tiny ramen shop in my neighborhood, just nestled up in the middle of all of these houses and I didn't notice it at all. I decided to head over and try and get some ramen. It was called ラーメン隊, Ramen Tai (Ramen Corps/Army?), and it seated about 10. There were three men in the corner, drinking and talking, a mom with her two children, one boy and one girl, and an older couple who ran the place. I take a stab at Google Translating the menu and the mom comes over to talk to me. Her name is Mika and she has pretty good English. She helps me order and I get a standard bowl of ramen and a shumai, a Chinese steamed bun. The bun is good, the ramen is only ok, unfortunately, with a weak and watery broth.
We talk about the city and my plans to go around and see all of these neighborhoods. We talk about the Kichijoji fall festival and she is very excited for it! She has only good things to say about Kichijoji, something echoed by everyone else who had to talk about it. She recommended i go to an izakaya (which I wasn't able to do, unfortunately) and the park there. We talked a bit about the matsuri and she confirmed that it would be happening. She also mentioned there was a festival happening in Meguro where they served free grilled fish on order of the shogun. She goes on to talk about her son - he is in a little league baseball league and her father and the men in the store are his coaches! They're gathering in the neighborhood ramen shop after practice to hang out, drink and smoke! She said the men took her son on a trip to hike up Mt Fuji in hopes the exercise would make him strong and successful at little league baseball. Eventually the ramen is served and I get a chance to eat it and she takes some pictures of me. It was a nice conversation and a nice, wholesome situation. I'm glad to have ran into her!
I then went back to the apartment, nursed my feet and did laundry.
Up early-ish, went to another chain - Yoshinoya - and got the extra green onions. Of course, that's what a real Yoshinoya veteran does. The waiter helpfully explains how the egg de-whiter strainer works and how to mix everything together. It's pretty good though. I make my way to the car rental place, along the way hoping to pick up more bandages. I wander into a Matsumoto Kiyoshi, but it is apparently before the opening hour and the smiley young clerk kicks me out. I go on to pick up the car anyway, not being entirely sure how they will handle me if I'm late. There are a group of teenagers there also renting a car to go out on some fun trip. I inspect the car and get off and go.
Driving is a pain. At first it's a bit tricky to drive on the left side of the road. The turn indicator and windshield wiper sticks are also mirrored in the car which takes a while to get used to. And of course the Google Maps directions are absolute garbage - there isn't good warning at all about when you're supposed to turn, and they spend a ton of time reading out Japanese language street names which are totally meaningless to me. Incredibly frustrating. I wind up missing many turns and spending my time winding along the Yamanote line and winding through traffic and some very narrow streets. After only 10 or 15 minutes of driving I manage to completely scrape up the side of the car. I have no idea how wide the body of the car is because I'm driving on the opposite side of the road and I drive too far to the left, scraping a street barrier along the entire length of the car and leaving a huge gouge. I'm mortified. Eventually I make it to the interstate and I'm driving through the docks and industrial areas - lots of cool, huge structures. The trip is worth it so far. I then plunge into the Aqua Line, the bridge tunnel across Tokyo Bay. The in-car driver pass is the voice of a perky young woman who pipes up as you drive through these gates and is kind of fun and novel.
The rest stop was nice. There were lots of fancy food gift shops. I liked the view over the bay, there were many ships and the sun was out and everything was pretty. It was a cool structure. I had some ice cream - my first mochi in Japan! It was ok. There are samples for many of the foods in the gift stores but I am too coward to try them and also because I know I won't buy them. But what a cultural difference between Japan and America: nobody would ever leave out unattended samples of store food like this. I see a bunch of Chiba-kun merchandise but I don't purchase it for whatever reason. I now regret this, it's a cool mascot and I wound up having a lot of fun in Chiba. I then go across the bridge half which is mostly unremarkable.
Driving across Chiba was nice, low traffic, nice long cruising stretches and plenty of sunshine. We're up against the Tokyo Bay so there are long marsh grasses and oceanside plants everywhere. I visit this strange platform art installation thing that overlooks the bay and climb to the top. I'm then back on the road, crawling along the coast of Chiba towards Mt Nokogiri. As you come down the coast and finally turn the corner to see the mountain it's pretty impressive. Most of the mountains are low and rounded but this one sits right up above everything and goes right up to the coast. I attempted to take the ropeway lift but it was inexplicably closed. Luckily the lobby was still open at the time and a lady was able to give me a map and some directions and I drove up to the parking area near the top. At the entrance to the trails a man gave me another map and waved me through, not charging me for whatever reason (maybe the fee to drive up the mountain covered me?).
Hiking across the mountain was nice. It wasn't a super long hike, maybe a bit over an hour, but it was quite humid. The view of the town below and the bay was neat. There is a buddha carved near the top in a stone quarry that was accessible. I zigzagged across the mountain to see the 1500 arhat sculptures. They came in a huge variety of sizes and faces although all were sitting with legs crossed. I made it to the main attraction, the big stone buddha, the largest in Japan. It's carved into the side of the mountain but has a large, grassy plain in front of it so it's a commanding presence. There was also a closed-off temple complex which seemed to be pretty big. I think this was the Nihon-ji complex but I am not sure.
While driving back down the mountain I came across a group of monkeys! There was about 20 or 30 of them and they were covered in brown fur with red, hairless faces. They ran away at the sight of my car although one monkey, possibly a mom, took its time as it had a little baby monkey on its back. I think there was also one angry monkey which charged my car although it went for the back of the vehicle and didn't catch up to me. I had no idea that there were wild monkeys in Japan and the whole thing caught me off guard. How incredibly fortunate.
I then keep going down through the sleepy towns of southern Chiba and eventually make it to Tateyama. At this point a storm is blowing in from the sea and the sun is starting to set. Google Maps sends me to a tiny road in the foothills and I hang out in someone's driveway, thinking it must be the place as Google Maps sent me here. The house doesn't look familiar and nobody is around. Pension ATCA isn't responding to my message on AirBNB. What's going on? Well, after camping out for some time at this person's house I start heading out to town to find a hotel, fearing I have to give up. Turns out that Airbnb had the wrong address (it sent me to the geographic center of the postal code) but my backup plan took me past the correct house by coincidence, what a relief.
Pension ATCA is my stay for the night. ATCA comes from "cAT CAt", it's supposed to mean you're in a place sandwiched between cats. It's owned by a husband and wife who have two daughters, age 3 and 5. There are six cats in the building. I didn't fully understand this before but a pension has a shared bathroom and separate bedrooms. The upper floor was all small rooms, sleeping two people each, and there were about eight of them on the upper floor. The shared bathroom and toilet on the bottom floor was nice and they also had this giant special bathroom with a hot tub like thing in it. I hoped to get in it - I had reserved some time for it - but that did not happen. There is also a front room and balcony that is painted bright white, takes in lots of sun and overlooks the ocean. There are lots of rocking chairs and the cats like to hang out there.
The owner introduces himself and the pension and we go through check-in. He is full of smiles and energy. We walk through this very professional and spiffy pamphlet with headshots and information on all six cats. I hand him the Mac Tabby sticker as a gift and explain to him the story of Charlotte's prized cat cafe. We walk through the pictures on Google maps and he says kawaii to every picture of a cat. He and his wife think the space is neat. We talk about many things and break out a good amount of Google Translate. I forget all of the details, but I think we cover their backgrounds - he was from Chiba city, she was from Tateyama (I think?), they lived elsewhere (she worked in the city in Tokyo) but eventually decided to buy the pension, move here and start a family. They were surprised at how young I was, they were both 35-40. At some point his wife comes out and she's just a fireball. She drills me on what I'm doing here, wondering if I'm into anime or a japanophile, and I explain I'm just here to tour the city. We talk a bit about driving as I drove down here. I talked about how precarious it was and she says she thought the same thing about driving in the US with our fast speeds. She also mentions that they're going to a cat convention in Tokyo and I get the details on that from them. My weekend is filling up! Finally, the sun has set and it's time to settle down and get food. They recommend I go to a sushi place in the nearby village so I head off to visit before it closes.
It's called 富鮨, Tomi Sushi, (wealthy sushi?). The town is a small fishing village so I expect the sushi to be pretty fresh. It's a small, cozy place decorated in a traditional Japanese style and I take a seat at the bar. There's a young couple there, on vacation, and later a middle-aged couple comes to sit around a traditional Japanese table in the other part of the store. The young man knows English and he offers some translation help which is nice. There's also the boss: this ancient dude, looks like a 90 year old Popeye, and he's just a riot. The Google reviews all call him 大将, military general. He's in disbelief at the sight of me, he's instantly shocked by my beard and is convinced I'll get rice stuck in it while eating. And the whole time they're having a blast laughing at my clueless ways - I have my utensils organized in front of me wrong and the general comes by and rearranges them. I eat the sushi with chopsticks and they're all perplexed - why aren't you using your hands? I don't know, I didn't know you should! I asked for a spoon to drink the miso soup and a shout goes out to the staff in the back for it and they come back with one. And of course later the English speaker points out that everyone was drinking the soup from the bowl with their hands. At one point I thank the general with "arigato", but he's hard of hearing or something or doesn't understand my accent. The English speaker explains to him - arigato, but in japanese - and everyone laughs. It's hard to convey but it was a festive atmosphere, people were enjoying the show. The sushi was great and was served with giant chunks of meat. Unfortunately I was so hungry after the day's events I was pounding it down quickly and in a few bites. They put little bits of seasoning under each bit of sushi, something spicy or savory, and I think that is a great way to serve it. They also had a weak soy sauce available which really complemented the fish. Typically I don't put any soy sauce on sushi but in this case I'm glad I did. I order namero made from yellowtail, a meal with finely chopped raw fish and tartar-esque spices pounded into a thin, flat slab. It's a Chiba specialty, it's actually pretty dang good and something I can't imagine I'd ever have a chance to eat in the United States. At the end of the meal I almost pay with a 10,000 yen note before realizing I have a 5,000 yen note and the chef gives me shit for it (I have this anecdote in my notes although I don't remember it) and he keeps teasing me about having food in my beard. It's all in good fun though. Wish I could have come during sunlight hours - apparently the place has a view of the ocean and even Mt. Fuji on clear days! I leave, smiling nonstop, but also feeling awkward I couldn't interact with these people and join in the fun in more meaningful ways.
On the way back I grab a canned Suntory highball cocktail from the 7/11. It's surprisingly good - Suntory is the well bourbon, but it has fantastic flavor. Cool stuff.
Back at Pension ATCA I start writing down the day's journal entries - a massive list. The wife comes out and we talk for an hour or two, mostly through Google Translate. We cover a bunch of topics about our lives and backgrounds. I show pictures of Snowball, Tracey, life back in Charlotte and American food. She's impressed with a plate of shrimp and grits and is surprised that it's not spicy (although maybe she meant more savory?). I explain how it's a very traditional Southern dish. She talks about her time in southern California and how she wants to go back again and also brings up the Venice Beach hotels whose clean, white design were the inspiration for Pension ATCA. She talks about how busy her life is here in Tateyama and how she's working every day with no holidays. Finally, I finish my drink, finish my notes, go to bed and pass out as soon as I hit the pillow.
Woke up and went for breakfast, which was served by the owners. They served a unique quiche that was delicious and had a fantastic crust. There were also little slabs of french toast that were delicious. Just a really great homemade breakfast. I went and showered and prepared to leave. We conversed a lot as they were glad to have met me and I was glad to have stayed here. They asked me about buying a Mac Tabby tshirt and I think there was some translation error here - I saw the wife get a money envelope but decided against it. I tried to ask them about buying it online although I don't know if that's possible. Either way I will be sure to send them one independently through the mail.
I start driving up the Tateyama coast. There are many beaches here, I can see how it's a resort town. The sun is out and there are cool breezes. The sea grasses remind me of North Carolina's Outer Banks. I realize this would have been a fun place to visit with Tracey. We would have fun on the beach, we could do some hiking, we could visit the little town on the beach and we could have some good sushi. For the last leg of the drive I take the toll road freeway through the mountains instead of the coastal road like I did on the way down. That was a good idea: it was a fun drive with little traffic, graceful curves, lots of tunnels and impressive mountain views as you're going in and out of these tunnels. Finally, I arrive for the ferry with a good 20 minutes or so of wiggle room. The ferry was short and pleasant, maybe only 10 cars in total and a quick 40 minute ride across the bay. I spent the trip walking all over the boat. It is surreal to sail under the Japanese flag.
There was also a motorcycle parked in the hold of the ship. It was decorated with several flags and had a drawing of an anime-style girl on the back of it along with a Twitter handle. I took a picture and checked it out upon returning home. Turns out this guy has a tiny, realistic doll of a fashionable young woman and he's doing a motorcycle tour of Japan and taking pictures along the way. I didn't get a chance to see him in person but I was able to check out his Twitter account and discovered all of this and all of his pictures with the "smartdoll". Apparently he quit college to do his trip around Japan and take photographs. He is from Chiba and showed his doll to his grandmother.
When I made it to the other side of the bay I had originally hoped to drive around and see things in that area, however, with the time I had left I pretty much could only get up to Tokyo without being late for the car drop-off. I then set off for the expressways and cruised up towards Tokyo. The highways were nice and easy, but navigating the interchanges was surprisingly challenging, once again because Google Maps was a pain. Once in Tokyo I drove the length of Meguro which was also a bit tricky as there was plenty of traffic on this road plus Google Maps kept being a pain in the ass. I managed to navigate the Japanese gas pumps and correctly fill up my car although it did take two tries. Eventually I returned my car and wasn't charged anything for the scratch, or at least they haven't billed me for it yet. It seems like the car was still driveable so they kept it rolling. We'll see. I wound up paying 4000 yen in road tolls which is a pretty impressive number for a day's trip. Finally, I set off for my hotel in Koenji, which was a pretty great deal - it's right on top of their subway station.
I then head over to spend the evening in Shimokitazawa, the hipster neighborhood. I spend a few hours on the north side of the station, wandering around, looking at all the "vintage"/used clothing stuff. It's all American, 90s style designs with big printed logos, stuff like a very 90s bright yellow Iowa Hawkeyes pullover sweatshirt. The people there are also dressing with a distinct hipster vibe, they're clearly all in their 20s or 30s, have stylish, carefully trimmed facial hair and are often wearing flannel. The whole town is urbanized but nothing is over 2 to 3 floors tall, it's got a cozy feel to it. And I stumble upon the find of the trip: a vintage Creighton Prep t-shirt. I can't place the timing or style, I wonder if it's a counterfeit. But at the same time it is a visibly worn in the collar Creighton Prep t-shirt sold for 4500 yen or so.
For the evening I wander to south side of the tracks where I get food in a place called the "Milk Bar". I have oxtail soup which is pretty good. On the way out I realize the place is full of women and I guess the whole name and branding was more woman-focused and I didn't pick up on it at all. It was just a pretty plain, slightly hipster restaurant to my eyes. I then explore the south side of town, it's not quite as big and doesn't take a whole lot of time. Finally, I turn back around to go to the north end in hopes of finding a bar for the evening - this is the hipster district, after all, and there are quite a few bars, izakayas and even craft beer bars. As I'm walking over to one I come across a place tucked back into the 2nd floor of a building called "Bar&Records Delmonico's". It's empty, except for the owner, and it's small, seating about 15 or so. One entire wall is packed with records, a huge collection, over 10,000 records. I suspect that it's the owner's personal collection that he transformed into a business. He spends the entire night playing DJ. It turns out he has a large collection of good 80s and 90s stuff and some modern Japanese idol music and I have a blast - he's a fantastic DJ, he plays great stuff all night long. He asks me about what I like and winds up playing Stereolab and shoegaze records and some really, really deep cuts too. Keep in mind that this is all on vinyl, to have a collection that includes a not-so-popular McCarty record really shows that you've got a giant collection. I have a great night. Some of the stuff he played:
Stereolab - Orgiastic (reminding me of Mitch's wedding!)
Deep cuts from Moose
Deep cuts from McCarthy
French films (???)
RYUTist, a J-Pop band that plays what I consider to be more EDM-type music sampling young, Anime-sounding women
This cover of Smells Like Teen Spirit that caught me off guard when it jumped into the Japanese lyrics.
Sandii and the Sunsetz - Heat Scale
Colored Music - Colored Music
The Art of Noise - In Visible Silence
Nick DeCaro - Italian Graffitti. One of the cocktails I had here was named Italian Graffiti after this. All the cocktails here were really good, the kind of stuff that would have people raving about them in America and this place wasn't even really a cocktail bar!
Funkaho - Villain Style
Goat - World Music (?)
Denim - Back in Denim
Sugar - Changes
The Boo Radleys - Lazy Day. One of my favorites, and I didn't request it! He owned and played it off of this Boo Radleys vinyl EP which appears to be super rare. Deep cuts, man. Later, I find out that the owner tweeted about my visit and what he played.
After this magical, serendipitous evening I took the train back to my home. I picked up a McDonalds hamburger late at night and had the teriyaki burger which was not great but also not bad. It's served with a thick teriyaki sauce and mayo.
Eventually made my way down to Kichijoji and walked around the shopping district north of the station. Lots more department stores, lots of the same, nothing really notable here. However, there is also the start of the matsuri! This was a pretty neat thing and I'm glad I made it to Kichijoji to see it. It's not a huge, famous festival: it was a bunch of neighborhood kids wanting to have fun and carry the mikoshi around. It was a bunch of shoppers gawking at the festival as it went past. It was a handful of parents and grandparents bringing their little kids to see the festival. It wasn't huge, so there was plenty of opportunity for me to get up close and see what is going on. They have several mikoshi, and they're all elaborately decorated. There is a little truck/float thing that follows the main mikoshi around and it's full of women in kimono and with fans. They're beating the drums and playing the music that keeps everyone marching forward on beat. The mikoshi is carried by a large group of teenage kids, some in their 20s, and they're wearing this loose fitting robe with short shorts. They're planning on sweating and today is an incredibly hot one. There are a few women carrying the mikoshi but I think they are just a little too short to get a shoulder on it. They're packed together carrying this thing and bouncing it up and down and chanting. Their route is a slow loop around the entire shopping district and I happen to catch them when they start near the main shrine in the middle. It's funny to see that the "inactive" mikoshi are abandoned by the side of the road without guard - in America, you'd never leave something so decorative and flashy unattended by the side of the road! There are also traditional Japanese festival games and food booths off on the side streets and those seem to be patronized by families with young kids.
I also see puppy shops in Kichijoji as well as a few other places on the trip. They stock incredibly tiny puppies and kittens that are all about two to three months old. They're pretty popular, full of people looking at them and some getting to play with them. The pets of course are incredibly expensive, thousands of dollars for a kitty and I think a purebred puppy comes to almost $10,000.
After an hour or so of watching the matsuri and walking around I head down to Inokashira Park. The most striking thing about it to my eyes is the lack of grass: there could be some sort of grass or other plant or surface throughout the park, but instead it's just big trees and dirt floors. After looping around I went back to Kichijoji and saw the tail end of the festival. There were three mikoshi in front of a stage right in front of the train station and there was some kind of announcer on the stage. One by one the mikoshi were brought up in front of the stage where they were jostled around and lifted to a 45 degree angle. Later, reading about matsuri on Wikipedia, I learn that jostling the mikoshi around and tilting it is a style only done in Tokyo and they do it to give the kami a more exciting ride. When one mikoshi takes another mikoshi's place in front of the stage the two groups march right at each other head on and only stop when the front people get squished into each other. I think that's also part of the excitement, they're faking out a collision between mikoshi.
I head back to my hotel to change out clothes - today wasn't super hot, but the humidity really struck a note. I was drenched in sweat. No idea what it is with Japan but not even North Carolina is like this. I'm stinky and disgusting.
I take off from Kichijoji and head to Hatagaya's Club Heavy Sick. It's a tiny basement venue that maybe packs 20 into a tiny room. Like everything in Japan it's smoke friendly too. The first band, which I believe was from Korea, was pure shoegaze and pretty enjoyable. I didn't see their whole set but I saw most of it. In between sets they were selling some kind of beef stew on rice as a fundraiser in the other room. It smelled great, I'm sure it was a pragmatic choice but I laugh at the idea of a beef stew salesman outside of an American show. The second was more of an 80s/90s influenced jangly pop band and the final group was pure 90s alternative rock. The kids are still making good music! It was a fun show. It seems like most of the attendees were friends of the bands playing, possibly because of the strange 3pm-8pm schedule of the show. The crowd was really getting into it, though, and I loved it.
The four bands performing that afternoon were “youthememory”, “Free, i do”, “tranquil life”, and “Kailios(京都)”. I believe I saw the last three in that list.
After the show I wandered back to the station and found a gyoza place, which is good because I had been looking for gyoza for the past few days and this was right in front of me next to the train station. The gyoza was good and they had some good sauces but it really shined when I got these tiny "one bite gyoza" that were a local specialty and were quite good. Wait staff was super busy, there was a group with some young bratty kids screaming the whole time. The mom looked trashy and the dad had tattoos - wondering what was up there? It was unique for Tokyo. I then went to my hotel room, pretty darn tired and defeated, and slept.
When I was planning this trip Japan rolled out new laws around AirBNB hosts called the minpaku laws. I had nearly all of my AirBNBs cancelled and some were cancelled several times. AirBNB tried to soften it with giving away hundreds of dollars of free bookings but it was still a huge pain in the ass. At the end of the day I had screwed up my dates for this hotel (a replacement for an AirBNB when I finally hit my limit of having places cancelled on me) and had to extend it for one day. Luckily they were able to do that, but if they weren't, man that would have been a huge pain in the ass. AirBNB is still a better platform for booking and discovering hotels than the competition with its endless dark patterns but this really grated on me. A bunch of the cancellations were done because AirBNB wasn't even vetting the minpaku registration that the hosts would provide until a week before the booking and I was rebooking places right up until I left for Japan.
The hotel had free breakfast at the Denny's below and this morning I went for the traditional Japanese breakfast with baked salmon and natto. The fish was dry and the natto wasn't pure death like I was expecting. The goopy texture was a little off-putting but mixing all the mustard and stuff into it made it pretty tasty. I think the bitterness of the natto is analogous to the bitterness of greek yogurt so I understand how it is popular as a breakfast food.
I then marched way down the Koenji commercial street in hopes of finding this speakeasy bakery down in the basement of a residential building. After hiking 15 minutes or so down there I found that they were closed for vacation for a month or so. Oh well. On the way back I was struck with the oddness of the English signs in Koenji, for whatever reason out of all the neighborhoods I visited this was the only one that had so many bizarrely named stores.
I took the train over to Asakusa. When I did all of my planning for this trip somehow I did not know about Asakusa at all and didn't include it in my planning. I was incredibly surprised to pop out of the subway and see the Skytree and the golden turd building. I was even more surprised to see the Sensoji, which I had no idea existed, and all the crowds around it. I was coming to Asakusa to visit this cat convention that the Pension ATCA owners were attending and told me about. Glad to "discover" this fun neighborhood. I went for the cat convention, it was full of people selling handmade cat-themed equipment and it was pretty busy. I spent quite a bit of money there just impulse buying all sorts of cat things. The gender ratio was heavily skewed towards women, and in fact when I was waiting in line outside of the convention it seemed like I was one of only one or two other men there.
After the cat convention I went up two floors to the Gothic fashion convention happening at the same time on the floor above the cat convention. There were lots of people dressed up in steampunk and gothic styles but really it was a grab bag of all sorts of stuff. For example, there was even a booth where all the guys were wearing latex gimp suits. I bought a wall scroll with a cat in a kimono from a guy in a giant furry helmet and a rice farmer hat. Later, I figure out his artist name and make an attempt to contact him through a 3rd party site but am unsuccessful. I suspect it is a hand painted scroll. There was also a bunch of women in lolita fashion. There was a grab bag of stuff for sale, most of it was fashion accessories but there were all kinds of things, like notably the girl who sold books of photographs of herself in bondage, with makeup bruises and in very rapey settings. The crowd also had lots of teenagers who were dressed up in steampunk or gothic fashion with their moms hot on their tails accompanying them to the convention which I thought was funny.
I wander around the temple area for a bit and eventually find a hole in the wall that serves me some hot, greasy yakisoba that was quite good. There are lots of little trinket and gift shops all around Sensoji so there's lots to walk around and look at. I get my first melonpan with matcha ice cream in it and it's pretty good. I also walk into this underground shopping mall near the subway station - it's tiny, cramped, dirty, and of course incredibly humid but it's kind of got a neat vibe. The area is also full of groups of people dressed up and the women were often wearing yukata - I guess the thing to do on the weekends is to get with your friends, dress up and go to the temple area and hang out.
The day's plan is to go back to Nakano Broadway - it's back on the other side of town, by my hotel. It's an indoor mall that's full of lots of small shops selling gear for nerd interests. The focus is on lots of gacha, old pop culture, toys and especially models. It's pretty busy, it's a Sunday and the place is packed. I load up on lots of gacha there, including a 300 yen porn DVD. Yes, you can buy porn anonymously from a gacha vending machine: you just crank the handle and it slowly squeezes out a DVD in an unmarked plain envelope.
I'm then back to Koenji and wandering the neighborhood to find some place to eat, drink and hang out. I don't find anything satisfying - not sure where all the izakayas are in Koenji. But that's ok, it works out - I wind up making it to the Mona Bar right when it opens. It's mostly empty, with two staffers, and I see one other dude who shows up, has a drink and leaves all before I do. The place has lots of Mona and SJIS-art themed things and the food and drinks are pretty good. They also have a cryptocurrency themed deal where you buy drinks and aquire Monacoins that you can spend on prizes. I go hard, spending $50 on several alcoholic drinks just so I can save up for a little button that says GOX. Worth.
I move to my hotel in Shinjuku, which was pretty nice and also situated directly over the Golden Gai. I was a bit apprehensive over Shinjuku for this whole trip: I knew in the back of my head it would be very touristy and I was a bit nervous - I didn't want to commit to spending tons of time to it and for it to be a letdown. It turned out OK though. I first went through the Kabuchiko and it was touristy as expected. There was a surprising number of people wandering through for 10 AM on a weekday when everything is closed. I went to the samurai museum which was also tourist trappy but hey, I enjoyed it, they had a tour guide and I learned stuff. I then walked through several large department stores, including a fancy men's fashion one. I had lunch at the top of one - the meal was thin slices of beef, decently rare, pretty tasty, but also this disgusting snotty soup thing. I thought you were maybe supposed to mix it with the rice so I did that, but it only ruined the rice. Oh well.
For the afternoon, I went to the western half of Shinjuku which is much more commercial. I go for a tour of Toto's toilet showroom - everything is nice and clean and formal and I sort of stand out as a tourist. They do have a row of toilets in one part of it - I remember walking past and having the motion sensor in one lift the seat to greet me as I go past - but it wasn't the largest user of showroom floor space here, even though Toto seems to be predominantly a toilet and washlet manufacturer. However, I wonder if I wasn't thinking it through - they're a toilet manufacturer, what if the real showroom was in the bathroom? I should have gone and checked it out.
I walk through more of the commercial buildings in Shinjuku, including going past the government office tower whose observation decks are closed on Monday. That's ok, I'll just go to the building next door, which also has observation floors - and they're all closed. Oh well. I retreat back to my hotel to check in and I actually have a room! This is a huge surprise to me. I had booked the place through this really seedy booking agent based in Turkey. They were a total fly-by-night operation and didn't even email me confirmations of my reservation. I was expecting them to have taken the money and run with it, but no, I had a legit hotel room, even though it was booked on the business floor. I chill out for a bit - it's been humid all day and now rainy, so I get to see Shinjuku with all of its lights in the rain.
The plan for the evening is Kagurazaka. It's a small neighborhood so I'm glad I was able to find stuff to do in Shinjuku during the day, there's no way I could have spent an entire day there. I think Mika also warned me about its size as well. I'm there in the rain, so I don't wander a whole lot, but the neighborhood is nice but small. There are lots of izakayas and shops along the main street. I spend a chunk of time in a bar, overlooking the street while it rains outside. At some point I leave and walk around in the rain, it's pleasant. I go past this small restaurant that has a window display where they're grinding flour in a mill right out next to the street. I don't go in there because I was looking for a more traditional izakaya but I record a video and later, when looking up the restaurant and its details, find out that it's a michelin star restaurant and I'm now committed to visiting it again. I eventually find an izakaya in a small, rainy alleyway and I get some good yakitori from it. Unfortunately it's pretty empty (maybe more of a late night party place) and I don't really talk with anyone. At any rate I'm pretty tired and head home. I pass by a store advertising a giant, 7000 yen gyoza that you have to order in advance and was maybe a foot long on the flat edge. I guess you can order it and eat it with a bunch of friends. I go home, passing through Shinjuku in the rain which is cool although I don't take any pictures because I'm tired.
At this point in the trip my legs are pretty well destroyed but a good night's sleep and stretching in the morning helps pick them back up again. I've also been consistently bandaging my toes and although the blisters look pretty nasty I'm not feeling any pain on them from walking. I go to Matsuya, another chain gyudon place for breakfast, then head back over to the Tokyo Metropolitan Government building to hang out before my bike tour. I spend a chunk of time up there which is nice as it just opened and was not as packed with tourists. I also reflected on the unintentional coincidence that I was spending the morning of September 11 at the top of a twin tower.
The bike tour was something I found on AirBNB: as a part of their apology for the minpaku cancellations they gave free coupons for people to try these hosted AirBNB events and I picked up this one. I think biking could be a cool way to explore Shinjuku and I wasn't leaping at the idea of walking all over the neighborhood given that I thought it would be very touristy. The bike ride was leisurely and fun and we managed to cover a lot more things than I would have done on foot so I think it was a win overall. Our group was made of four tourists: two American-educated college students, a couple, one who grew up in Japan and one who is going to grad school in Columbia to study AI/ML. The last guy was a webdev from Utah, a nerdy tall guy full of energy and a minor weeb. The group leader was a hardened city bicyclist and he brought around two friends as English translators to help the group. Of course, the first place we went to was back in the tower, but I didn't mind - it was fast up, and everyone got to walk around and look at everything and take pictures. The next stop was by an old-fashioned Japanese candy store where we all bought a little piece of candy or whatever. I got these plums in a red sauce, everyone described it as sour, but no - it was a sweet pickle flavor! The juice was disgusting, I choked it down but the taste stayed with me for a while. The plums were dense and inedible too. Weird-ass stuff. We then went on to the Meiji Shrine and they show us how to do the shrine thing, which is nice - I hadn't planned on visiting any shrines so I'm glad I got a chance to do one. It's also a ways away from Shinjuku Station and I probably wouldn't have walked to it by myself so doing it on bicycle was nice. Next, part of the trip includes lunch and we went to a gyodon restaurant which was pretty good. We talked about our lives back home and about our trips in Japan. By now we're getting later in the afternoon and the sidewalks are starting to fill up. Biking, which is still done on the sidewalk, is starting to get hectic, and I'm having a few sudden stops and close calls. Nobody in our group manages to hit anyone, though. Next we visit a taiyaki stand and have some fresh taiyaki, which is fantastically delicious. And finally we loop through the Golden Gai, Kabuchiko and that corner of Shinjuku before heading back to our origin. We put the bikes away into this cavernous basement bike locker that held hundreds and hundreds of bikes stacked on two layers. It even had a little bike elevator to help you get your bikes up out of the basement. Finally, I head home to shower and relax before going to the fancy soba place. I almost join up with the nerdy Utahn for dinner but he is dedicated to his plans to see Ueno Park and we split up.
When I finally make it back to the fancy Michelin-star soba place it's got a few seats open at the bar so I can get seated immediately. This is good, because otherwise I'd have a super long wait. It's called Kyourakutei Soba, 蕎楽亭. The bar overlooks the chef's working area so I'm seeing them go around and prepare everything. They're taking thin slices right off an octopus tentacle and eating it right there. I guess it was really good because the one dude cut a slice off for the boss to try. I see them hand cut and boil the soba which is nifty. I see them make all these carefully arranged plates of fried food and fresh sushi which all look incredible. I order the ordinary hot soba which is pretty tasty - the noodles have a distinct, rich flavor to them that's a bit bitter - and it's good, but i really wish I could stay there and keep ordering appetizers all night.
To close off the day I went back to the Golden Gai to find a bar. I find a place that is nice and I have a bit of conversation in English but eventually it turns to all Japanese. There's a big soccer game on the TV and when it's over the players are rushed off to pose for photographs with giant soft drink and beer can mascots. At the bartender's request, I try the Hakushu whiskey, it is pretty good. The Fuji-Sanroku whiskey is pretty unremarkable, and I have a plain Suntory and club soda which tastes OK. At this point I'm realizing that I'm probably not going to have some English conversation and I go back to my hotel room for the night.
I forgot the previous day was Subtember 11, but luckily it turns out not only is there a subway in Shinjuku but it's open for breakfast. It was still 9/11 in the New York time zone so I say it counts for a Subtember 11 sandwich. I ran over there and had something unremarkable, however it was noticeable that the "Subway smell" from the break was not present at all, this was an ordinary loaf of white bread and the restaurant didn't have the smell at all. I check out of the hotel, once again shocked that it was a real hotel and not a scam. I then head across town to Ueno to my AirBNB home for the rest of the trip.
I spend most of the day at the Tokyo National Museum, a gallery of art and historical artifacts. Because this is a tourist spot not just for foreigners but also Japanese I get to see a bunch of people taking photographs. I see one dude who is going around the entire museum, item by item, and photographing both the item and the museum's placard with what appears to be an expensive camera setup. I later visit another room where a woman in a dress and stilettos is taking lots of pictures of a reflecting pond and a tree from inside of a building - a shot that can't possibly be good - and then turn around to take a picture of the tasteful yet totally unremarkable chairs in the lobby of the museum. Photography is a strange hobby over here. The museum guards also sit completely motionless in chairs and stare straight ahead at all times. I have no idea if you're even supposed to talk with them under any circumstances.
The museum itself has many immensely cool artifacts, and it's lots of very, very old stuff, hardly anything from the Edo period. I see the oldest katana with writing in the metal. There are lots of artifacts from the almost-prehistoric early kingdoms of Japan around the 10th century and it's inspired me to read up lots on the history of ancient Japan. One memorable exhibit was an entire scroll laid out for people to read and thankfully they provided an English translation for the story. It's the story of a princess who meets a mouse at a temple and decides to marry him. It goes through the story of the mouse and the princess getting ready for their wedding, preparing themselves and putting together the feast. At the very end the bride peeks out of her room and sees that the mouse groom person is really a mouse and not a man and calls the wedding off. It was a neat little story and I found it engaging despite the vast cultural differences and I'm glad they put it on display. The artwork was also very cool and the drawing of the mouse as a human was pretty compelling. Unfortunately I can't find the name of the artwork otherwise I'd try and see if I could link it here.
I then spend a bit more time walking around Ueno Park, checking it out. There's some sort of military drill/competition type thing happening with firefighters right in the middle and lots of people are standing around and cheering. There's a whole group of JR Railway employees hanging out, watching this thing and cheering and they've got these green JR scarves like they're some kind of sports team. It's time to check in to the final AirBNB, so I hunt that down and run into the host still cleaning it out. I explain I just want to check my bag and leave and go back on my way. I head over to the Kaneiji temple which is back on the other side of the Ueno Park. It's in a quiet neighborhood but there are still a few tourists there. There's a bunch of ritual objects around it, including a giant bell that is hardly fenced off and I am oh so tempted to sneak up and ring it. I also see the insect shrine which is a small, worn black rock with writing carved into it. The interior of the temple is beautiful, peaceful and warm and it's heavily decorated with the Tokugawa family crest as this temple was built or managed by them and there are many things tied to that family in the Ueno area. I go into the temple, do the prayer thing and purchase the goshuin. The design is beautiful, and the book it comes in on is a tastefully packaged tiny notebook decorated with the Tokugawa family crest. It looks incredible. The calligraphy is done by a little old lady who seems to be running the place. I wander down to the other Tokugawa shrine, the Ueno Toshogu shrine. It was elaborately decorated and also had the family crest prominently placed all over it. There was a very cool railing with lots of animals carved all around the shrine and also a flame continually burning from the bombing of Hiroshima. Finally, there are many giant stone lanterns scattered all over the property around the shrine which were sent from all over Tokyo as gifts to Tokugawa Ieyasu from his vassals after his death. It was very cool to walk in this forest of lanterns as you walked around the grounds.
At this point I am pretty tired, I've spent a lot of the day on my feet. I walk through the park and see a few more things but I'm heading on to the Ueno business district for dinner. I wind up going to Hanamaru Udon which turns out to be the McDonalds of udon. I'm tired, though, and my feet are pretty beat so I take it. The food isn't bad at all, just not great. I'm feeling a bit rested so I take on the Ueno markets, eating a bit of ice cream and melonpan, walking around and seeing everyone shopping. The main drag of Ueno is a small road covered in retail shops with the tables spilling far out into the road and they're very busy with people picking up food on the commute home. I walked through a big toy department store and saw lots of models for sale. I also came across a place called 1kara that rents out solo rooms for people to do karaoke by themselves, I guess. Finally, I retreat home and lie on the floor of my apartment because my feet are ridiculously sore and I can't stand to put any weight on them again.
The apartment is in a weird state. The owner left a bunch of towels up in the shower dryer and I had to put them away and kill the dryer heater that was heating up the entire place. It has a washing machine which is this tiny futuristic thing lurking in the corner that I did get to use later. And the decoration for the main area was nonsensical with fratty vintage beer ad reproductions (all in English!) on the walls and a Japanese traditional bed on the floor with Hello Kitty sheets and pillowcases.
For breakfast I look for cafes in the area, and there's a nice one only a few blocks away called Coffee Katsura, 珈琲店 桂 台東区役所西横店. It's a very cozy place with light sandwiches and fantastic coffee, some of the best I've had in all of Tokyo so far. I have some time to kill before the Marutaro store opens at 11 so I walk around the Shinobazu pond in Ueno which is full of water lilies and giant carp. I slowly walk up the road towards Nezu and Maru's shop. There's a bunch of stuff in there, lots of calendars and tshirts and other things. I pick up some magnets and gatcha. I'm then sent on a lengthy subway trip across the city to the subway museum.
The subway museum is a little bit kiddy, I'd definitely say it's kid friendly, but there's lots of more technical information on the history and construction of the system. At the very end of the museum they have a bunch of simulators, including two different subway locomotive cars that have been converted so you can drive them and a pretty sophisticated PC where you can drive simulated trains along entire subway lines. There's an old man watching over the system, a kid on the PC and his dad watching him and we all laugh as the kid plows through a station at 20km/h. There were lots of moms and kids at the museum but also a good number of single guys wandering around and also a few guys who seemingly paid the entry fee so they could people watch through windows that overlooked the subway exit. I then have my first chance to take the bus in Tokyo and go a few hops down to the Kasai park. There are lots of cool structures and buildings here, including a large glass-walled building that I think was meant for watching the sunrise. The park has many trails through the forest and a large beach area including some manmade islands that stick out into the bay. It was a pretty cool park and worth the detour. I'm then taking trains back into the heart of the city to go to the Skytower, but first, I check out the salt and tobacco museum that is near the Skytower. It's a slick, modern presentation about salt and tobacco, which were at one point regulated by a single government monopoly after WWII until the eighties. The salt floor talks about the history of salt extraction in Japan and the tobacco floor is mostly an exhibit of smoking apparatus and carton designs. There is no mention of North Carolina at all in the tobacco museum! What an oversight.
I'm then on to the Skytree. The place is busy, but there isn't much of a line to get tickets or to go up to the top. And man, is the view incredible. The whole Skytree observation deck was clearly designed for viewing the city as it has these large windows that stretch far below foot level so you can easily look down at the city below. Absolutely one of the best places I've been on the trip. I loop around the observation deck, taking a bunch of pictures, and also look for some place to sit and watch the sun go down as it is only a little bit of time before the sunset begins. I don't see anything to sit on at the observation deck - later, I learn that if you go down a floor there are a bunch of couch things. Instead, I go and get some (probably pretty overpriced...) soft serve ice cream that has vinegar as a topping. Never heard of putting vinegar on ice cream but it was really good. The vinegar was cut with fruit juice so it was a little bit sweet but very fruity and still bitter. I lean on this standing table thing and chill out for half an hour or so, eating my ice cream and slowly watching the sun set. Because I'm in the same spot I decide to take a bunch of pictures and I get a nice series of photos of the city lights slowly turning on and the sky slowly darkening. While this is going on I'm suddenly greeted by a ton of noise and shouting - school kids! A few elevator loads came up and the place is now swarming with school kids, all there to see the city in the dusk and at night. It's instantly much more packed and much louder, however I am safely sheltered in my foot court area because there's no way a middle schooler is going to buy the overpriced ice cream here. Once things calm down a bit and the sun has completely set I then loop around again, taking pictures of everything in the dark and helping a group of schoolgirls take their group photo. I pay for the extra Where's Waldo floor which is ok but not really worth the price, you have a better view from the main observation deck anyway. I wind my way down, taking a chance to stand on the glass ledge and for whatever reason it wasn't really terrifying to me at all, possibly because it was night out. I get out of the Skytree and head back to Ueno.
I've been staying in the north side of Ueno, which is mostly residential and I later find out full of gay bars on the back streets. The market district is south of the station. I notice that there is a commercial west side of Ueno as well and I decide to walk over there tonight. Turns out the reason why I haven't heard much about this is because it's full of seedy sex shops: massage parlors, girl bars, dance clubs. The women hang out on the street corners to try and lure customers in which is interesting. I remember having to shoulder my way though a group of 10 or so women hanging out at the end of the alleyway to get onto the main street. I go into a random restaurant which is full of salarymen eating and drinking. The food in these places is all greasy comfort food: lots of fried things and lots of strange cuts of meat and organs which I guess they really like. My meal is a whole course of this sort of stuff and I finish most of it but there are just some organ bits I can't finish. I head home, pretty worn out, and go to bed.
Today is the day I tackle using the fancy washing machine. It features a delay and I try to figure out the buttons with Google Translate but it really doesn't work too well at all with these technical terms. It's also a little weird in that because I'm trying to set up an 8 hour delay I can't go off of seeing the device kick on right in front of me, I have to program in everything and hope it works properly at the end of the day when I'm not around. Luckily I did eventually figure it out and I came home to nicely washed clothes. Breakfast was at a chain bakery in the train station, but one cool thing about it is that you put all your baked items on a tray and it scans them in and uses some ML to figure out what to purchase. You have to make sure they're all spread out on the tray, the cashier will spread them out if you're not careful, but it's pretty fast and accurate.
I head down the train to the Nihonbashi which is a cool little bridge. The modern bridge is in stone, the original Edo period bridge was made of wood. I keep walking, making it to the Hakozaki Junction which is much more impressive in person than in any of the pictures I've seen. You get a much better sense of how tall this thing is and how many layers of ramps there are on it. I keep walking across this corner of town, walking along the river and through more neighborhoods. I eventually get back on the subway and emerge in the Tokyo station. It's a huge station and I'm also on what has to be the lowest subway platform within the thing. It's packed, even though it's mid-morning, and it feels like it takes forever to get all the way up and out of the station. Eventually I'm on the surface in downtown Tokyo and I head over to the imperial palace.
Unfortunately it's a Friday and the palace gardens are closed on Fridays. Oh well. I walk around for a little bit but there isn't a whole lot to see without being able to get into the palace. I then keep wandering around the downtown area, passing in front of the Diet building, the Prime Minister's residence and going through a nice park. I probably should have kept looping around, it looks like there are interesting things all around the imperial palace, but I decided to call it there and head down to Ginza.
Ginza is pretty busy and I decide to go for coffee and a chance to rest my feet, which have been pretty sore all day today. I find some swanky place in the basement of a building and chill out for some time there, the coffee is good and I have a waffle with gelato on it which was very good. I'm then off to wander the streets of Ginza, looking at all the different shops, but for the most part not going into any. I do step through the entirety of Itoya which is kind of interesting, they've got all kinds of different stuff for sale there, not just stationery and writing tools. I find a leather wallet for sale for a few hundred dollars, it's marketed as quality American leather. At some point I go past the main Uniqlo store but I decide against browsing it - yes, my feet are tired but more importantly I'm all hot and sweaty from walking everywhere all day and not really in shape to try out clothes. I also walk past a fashion boutique that had screen printed vaporwave-style T-shirts. I see why it's popular, but it's pretty incredible to think that a bunch of Internet posters managed to roll this meme into a legitimate movement that's brought their aesthetic as far as the shops of Ginza. I make a tour of a big toy shop (pretty focused on kids stuff - it's not like the nerd focused ones elsewhere) and head back to the subway. I zip up to Shibuya, as I read that a place sells vinegar ice cream there. Turns out I was a bit mixed up - the place I wanted was in Ginza but wasn't getting any search results for it, so I wound up going to the Shibuya one by accident. I was sent to a random department store in Shibuya that had a small shop that sold the vinegar. I did buy a bottle and tasted a sample but oh well, missed out on the ice cream. I will have to experiment with it back home. At this point I'm still pretty worn out and I get on the subway again, taking the Ginza line back from Shibuya to Ueno which is a pretty lengthy trip. That's ok, though - I saw a fair amount today and pushed myself to my limit while walking.
There's a place in Ueno just south of my apartment that is open 24 hours a day and has this big, set back, wooden framed design. The place is called GyuMaru, 上野産直飲食街, and it's always got a bunch of people in it when I go past, all sitting on benches around long wooden tables, talking and drinking. I decide to check it out for dinner. It's mostly shellfish of different types, not so much fish, and I wind up enjoying it. It's another comfort food place full of salarymen but the foot is a bit less weird. I have some oysters cooked in a chili sauce which is tasty but not as spicy as you'd expect. After that I order some kind of seafood gyoza that was pretty good. I wander down the market street again, going to the matcha ice cream place, and I realize that everything's in English, no Japanese at all, and the design is very modern and hip. I realize they're taking our basic bitch culture and marketing it to the Japanese! It's cultural appropriation.
I keep wandering around the market and eventually decide to spend the evening in a basement cocktail bar. It fills up a decent amount over time but at the beginning it was pretty empty. I don't really talk with anyone, even the bartenders, but it is kind of strange that when I do eventually get up to leave suddenly everyone's wanting to talk to me about this and that. There was a lady sitting next to me with what I believe she said was her husband and she spent the entire time talking his ear off in a very one-sided conversation. When I start getting up to leave suddenly she wants to talk to me in English and is asking me all kinds of questions about what I'm doing in Japan and what life is like back home. The drinks were OK. I started with an old fashioned, which was served with Old Overholt but it had undissolved granular sugar at the bottom which was unfortunate. I go on to their house drink menu, and order roughly because I don't know Japanese and the bartender can't help a whole lot. I have a gin and tonic that is made with some sort of pear that gets mashed up and stirred into the drink that is pretty good. I also order some kind of mojito (I think, it had loose, crushed mint in it) and they also prepared it with liquid nitrogen poured out of a big Dewar flask so it was pretty cool to watch, plus you get a drink with the nitrogen smoke pouring out of the top of it when you drink it. Finally, I got a picture of their distillery apparatus: they had this cool looking thing in the dark corner of the bar and with HDR the picture actually came out pretty good. They actually do use it to distill some natural extracts and I get a chance to smell them - I forget what they were but they were pretty strong smelling so I guess it worked. At the end of the evening I return home to find my clothes washed (the machine worked!) and hung them up in the shower. Japanese places don't have dryers, they have either clotheslines or heating vents that you turn on in the shower to blow a ton of hot air into the shower and evaporate dry them that way.
My last full day in Japan, with the entire day dedicated to Akihabara. Wander down to a random chain cafe in Ueno and order a breakfast grilled cheese that is oddly sweet. I'm then off to the trains and down to Akhibara, which is actually pretty close to Ueno, I probably could have just walked down the main street. I'm first checking out the stores close to the river right on the west side of the train station. I'm going into nearly everything I can just to browse. In the morning things are pretty empty as it's been raining but by the afternoon crowds start picking up. I also note that the crowd is mostly Japanese compared to the largely white crowd I saw when I walked through Akihabara at the start of my trip.
There's a building, a good 5 or so stories tall, entirely selling trading card games. Magic the Gathering still has a strong showing in Japan but there's also a very popular game where all of the cards appear to be different anime girls from different series. There are a few shelves of Pokemon cards too and I check them out.
Next is a porn store, which also turns out to be one of the largest that I visit and also one of the most diverse. I notice that there's a shelf for beastiality videos - must be legal over there. Most of the porn is plain and cookie cutter and I'm also noticing that the women in all of these videos fit a very narrow look. It's a big jarring difference from all the different women I've been seeing all around Tokyo for the past two weeks. There's an entire section or two of poop videos. And I also find a few sections with creeper/peeper/upskirt/candid camera videos. They are notably sold in a plain, undecorated case unlike all the other cases, probably to give the illusion that it's the illicit stuff that's actually recorded in public and secretly distributed. The patrons of the porn stores of Akihabara are all men, middle aged and about as sketchy as you'd expect. It's a distinct crowd.
I find an electronics mall, a tiny cramped space full of little booths selling all kinds of parts. This was the only thing of its kind that I could find in Akihabara and I'm glad I stumbled into it as it was a pretty unique experience to see this kind of diversity in electronic parts all available in one spot. They had microcontrollers dispensed from a gacha machine but I forgot to loop back around and get one. Another weird spot I pass by is a kind of garage sale for electronic goods in the garage space for some business. There was just tons of strange, assorted stuff in there, like boxes of laptop power supplies and a whole case of random install DVDs for various bits of hardware for $20 a pop or more! The garage sale was decently packed with shoppers too.
The maids from the maid cafe are starting to come out onto the street to get customers. I walk past one place, shangrila, which appears to have a theme of chubby maids with lots of homecooked food. I make it to the giant gachapon store where an entire floor is stacked to the ceiling with gacha. I've bought everything I want at this point and there are lots of repeats in this gallery but I'm glad to have browsed them.
There's also a giant cosplay and toy store called Lashinbang. At this point the toys are all blending together for me but the cosplay floors are pretty impressive with a crazy selection of all kinds of stuff.
My feet are getting tired again, it's getting into the afternoon and I've had a good few hours on my feet. I start looking for a maid cafe and wind up going to Pinafore, which is this incredibly sketchy looking single room. It's barely decorated and everyone sits in plain tables arranged in a big U around maids in the middle. But they're real maids and it's a real maid cafe, half of them are in sailor costumes and half in schoolgirl outfits with one or two in a pink and black maid apron. They all do the talking in sync thing when you come in and leave. I'm presented with a menu and order a mystery cocktail and the rights to take one picture with my waitress. None of them know much English and I drop the ball here by not breaking out the Google Translate to try and at least get some conversation done, but I make it through the event. My drink is a bright green and sweet alcoholic cocktail with a cherry in it. The schoolgirl who is my waiter talks to me in Japanese and I try to squeak past with my "hai" but she isn't fooled. She knows at least one word of English, though - magic spell. I accept the offer and she does some chanting in Japanese, it's all pretty fast and she's waving her hands around but I do catch a "moe, moe" when she makes a heart with her hands and bounces it over my drink. Getting a magic spell put on your cocktail by a maid - you can't get that kind of service in America! One of the other maids comes over and strikes up a conversation with me, which I guess is part of the package. However she knows even less English and I just have to "gomennasai" my way through the conversation and she's halfway between humored and frustrated with it, doing a little mimic of me with her hands folded in front of her and a small bow. I should have brought out the Google Translate but didn't, oh well. I pay my bill and I'm on my way out of there.
I make a tour of the big Mandarake building, it's pretty packed at this point. It's also more of the same models that I've been seeing everywhere, but I also get a chance to see the shiny Japanese Charizard Pokemon card and an entire aisle of dakimakura. For the rest of the afternoon I keep walking in and out of stores, seeing endless models and figurines. It seems like K-On! will never die, though, it's still popular and it's been a decade. I find another doujin store, it's not quite as big as the first but there are a lot more tankoubon. They're all wrapped in plastic to prevent readers so you have to look at a little photocopied sample page that's put in the back of the bag. I find a tankoubon in the porn section with a happy family - young son, mom, grandmother - posing together on the cover so that's got to turn out nice, right? No, of course it was some sort of twisted rape thing.
At some point my phone thinks my SIM is missing. I'm a bit frightened because I use it heavily to navigate, even in Akihabara, but I realize that i'm in Akihabara which is probably the best place in the entire city to get a SIM card poker tool and check it out myself. I walk maybe a block or two down the street and find a place that sells cell phones, gesture to the clerk and he points one out to me right away. I do some surgery, reseat the card and it works great! I am incredibly thankful to the clerk. It's cool that it worked out.
It's getting later in the evening and I go for food. I wander into Nikujumen Susumu which is a ramen place. It isn't bad but it isn't great, just greasy and fatty. I also get greedy with the egg yolk separator and it breaks after my second time through (I manage to have it slip through into the catch bowl and pick it up the first time). It's now dark and the streets are crowded with even more people and maids. I'm wandering through the last few shops and stores and finally make it to M's, the supposedly legendary sex toy shop but I find it cramped and its selection is tiny. A huge disappointment. I look at one more hobby shop full of models, toys and figurines and at this point my eyes are falling out of my head after looking at figurines for the past nine hours. I then make the decision to take the train across town and visit my favorite bar in Shimokitazawa for my last night in Tokyo.
I show up at Delmonico's right around when the bar opens and I'm the only one there. As the night goes on through there is a bit of a crowd, mostly older couples in contrast to the younger salarymen that were there last time. The owner has a freaking Stereolab inspired shirt referencing Peng!. Deep cuts. I sit and reflect on the Japan trip and it seems like the first half of the trip had most of the memorable events. Not sure if that's because I was packing more into it or if I was less tired. There was still some very neat stuff in the second half and in all it was a fantastic trip.
The DJ’s playlist for the night:
Television Personalities - She's Never Read My Poems
Denim - Novelty Rock
Elmore Judd - Angel Sound
Love Zombies - The Monochrome Set
Miaow - When it All Comes Down / Did She
The Popguns - Snog
Happy Daze - a late 80s/early 90s compilation album of some kind
The House of Love - Beatles and the Stones
Nav Katze - Oyzac
I take off after two hours or so because I'm a bit unsure of the trains but that was probably my mistaken understanding of the time table. As I'm walking through the dark streets of Shimokita it finally hits me that my Japan trip is over and I have to go home tomorrow. There was still so much I wanted to do, and even just walking through Shimokitazawa there were bars I still wanted to check out. I stumbled up on this place next to Delmonico's called BAR FORCE and appears to be Star Wars themed but wasn't open when I went past it earlier in the evening. To think that on the initial incarnation of the trip I was actually going to spend a few nights in Shimokitazawa! What a missed opportunity. I ride the Keio line back to Shibuya and ride the entire length of the Ginza line for one last time.
I pack and get out of there to get a breakfast of more baked goods from the place with the cool automated scanner. I head off to the bus, which is packed full of old ladies and standing room only. It's a long trip across town too, 30 or 40 minutes, because there isn't a whole lot of cross-town connectivity other than this bus between these two places. I get to St. Mary's Cathedral a bit after 10:00 AM, the service has already started and it's standing room only. The building is magnificent, much more impressive than any pictures. The walls are all gray concrete but have texture to them, it's somehow inviting. Each section is perfectly straight top to bottom but they all twist around and align so it looks like each one is twisty. It also seems very small for a cathedral. Sure, it was built in the 60s when Catholicism was no doubt very small in Japan but in Omaha it would be the size of a small parish. The Japanese spoken for the mass was very slow, slower than any I've heard all trip. None of the women in the pews were wearing veils but the offering was brought up by women and several grade school age girls who all wore white lacey veils and capes. The individual collections were done by the women, even the grade schoolers, and I had a tiny girl give me a deep bow after I put my 100 yen coin into her velvet bag. The sign of peace didn't have any handshakes, it was a deep bow to all your neighbors and they said something in Japanese. The mass was almost 1 hour 20 minutes in length, likely extended by a post-communion lecture, and there was an additional blessing of old married couples after the service. Afterwards I stick around with several other people who are all in awe of the building and all want to take pictures once it starts clearing out. I also note that there is only a tiny bowl of holy water at the entrance. Maybe the giant baptismal fonts in the entrance are an American trend but I also wonder if it's too reminiscent of the little fountain that Shinto shrines have in front for purification and they want to break up that association.
I grab a bulletin, which is a multi-page, glossy, full color thing and head down to the main street to find postage to mail it back to Omaha. Surprisingly the first Family Mart I find doesn't have any stamps for sale and I have to keep going to find a 7-11, which of course is only two blocks or so down the road. I buy 400 yen in postage for 430 yen total which I hope is enough to get it back to America. It did manage to make it back and arrived on September 28th, a twelve day journey. I keep going down the road and turn into a ramen shop for my last meal. It's a chain called 太陽のトマト麺 that sells ramen in a tomato soup broth with vegetables and parmesan in it. They definitely have a sort of healthy, organic slant to their marketing and the food is pretty good. I pack up and head for my last subway trip across town and to Narita which takes some time, even with an express train to Narita.
It seems like nobody wants to fly out of Japan on a Sunday evening and the airport feels oddly empty and spacious. Security is virtually empty and I get scanned and whisked right through. To kill an hour or so worth of time before the flight I watch this NHK program in the lobby in English that has these short shows on different professional craftsmen in Japan. There's a whole hour long program on a master wall plasterer that can't keep my attention but before that there was a thing on a person who produces organic indigo dyes in a traditional manner which was pretty neat. Eventually the time comes to board and as I get my ticket and passport scanned and thank the flight attendant with one last "arigato" I realize that this is my last interaction with the Japanese people and my trip is finally over.
Git, the command-line version control system, was initially released on April 7, 2005. Github launched to the public on April 10, 2008 and was the killer application that promoted Git beyond its initial audience of systems programmers. By 2010 it was clear to me that the industry was moving away from Subversion and Git was the new standard in version control. I took it upon myself to git with the times and learn it for myself. My learning project whose initial commit was on September 13, 2010 was a repository for my dotfiles and I've been using the same repo since then to manage my dotfiles. It was my first attempt at using Git and my first attempt at managing configuration in a rigorous way. Over the years I've hit several problems with my initial design and I've also learned a lot about configuration management and what is now called devops - a term whose first appearance on Hacker News was on September 3, 2010! I've wanted to replace my dotfiles setup for some time but have never really put it into action. Recently I discovered deadc0de6's dotdrop which seemed to be more or less what I was looking for: source-controlled, templated management of dotfiles. As I dug into the project I found several serious shortcomings with the code which I will explain in this blog post. The next blog post will describe the design of a not-yet-written system that incorporates what I feel are best practices for dotfile management that I hope to implement some day.
First, I want to applaud deadc0de6 for what they did right. Using a human-readable, declarative configuration to orchestrate the dotfile setup across all hosts is a good practice. YAML is a good fit for the heavily hierarchical configuration inherent to this domain and probably a better fit for the problem than the trendier TOML. Jinja2 is an excellent templating system flexible enough to find use outside of its original HTML templating purpose. Dotdrop's dotfile orchestration uses configuration profiles and inheritance to eliminate the repetitive configuration similarities between machines which is a good fit for the domain. I also like the author's suggestion to put dotdrop's upstream code directly in your dotfile repository as a git submodule. This is a sane way to pin the version of dotdrop and install it correctly and easily when setting up a new machine. It's also nice that dotdrop uses your machine's hostname as the default profile name although it would also be a nice enhancement to search for profiles that match your user and hostname.
There are also several design decisions that I don't agree with but I think that nothing in here is too serious that deadc0de6 coudn't release a workaround, even if it needs a compatibility-breaking dotdrop2 release. if these changes are made I would consider using dotdrop but otherwise I'm still planning on writing my own system at some point.
If you install dotdrop as a git submodule you have to use a dotdrop.sh driver script to set up the Python environment correctly. The driver script is a necessary workaround for the situation but the .sh suffix (to keep it from conflicting with the dotdrop submodule directory) seems like a hack. Couldn't you bundle a ~/dotfiles/bin or ~/bin along with your dotfiles and put the driver script in there along with any other shell scripts you might tote around with you? It would make sense to promote the practice of maintaining your own personal ~/bin in dotdrop's documentation.
The Python community has pushed virtualenvs for almost a decade now but the dotdrop documentation does not encourage their use with either of its installation methods. This is a big missed opportunity because dotdrop already requires a working directory to hold your dotfiles. It's the perfect place to stash a virtualenv and keep your dependencies isolated from the system Python. The current installation methods are not good Python practice.
Dotdrop lets you install dotfiles by placing a symlink in your home directory that points back to the dotfile kept under source control. I think this is a good practice: it's a good, easy-to-check way to see if a dotfile is managed by dotdrop. However, if your dotfile is a template the expanded dotfile must be written directly to its destination, there is no way to have dotdrop make a symlink. I think it would make sense to have dotdrop maintain a working directory for dotfiles and symlink into that working directory so all dotfiles, templated or not, can be visibly symlinked. I also think that in addition to using symlinks to identify managed dotfiles that all dotfiles should strongly be encouraged (possibly with a deploy-time check) to have a dotdrop-generated header that reminds readers that the file is managed by dotdrop and maybe lists some statistics like build date, git commit id and commit date. If a user insists on being able to review a dotfile before deploying it to their system there could be an optional way to disable symlinking and the deployment could be split into a second, independently run phase that does the copy of dotfiles out of the working directory.
Users that do not use symlinks in their dotdrop installations can use the dotdrop update command to install their updated dotfiles by copying them out of the dotdrop directory and into their proper location. However, if your dotfile is a template dotdrop will refuse to do this copy and ask you to edit the configuration file by hand. This is the opposite of what I would expect, I would want dotdrop to always copy and overwrite template-generated dotfiles. If dotdrop keeps a working directory for its dotfiles as described above the update command could always overwrite the dotfile as it is under dotdrop's control. Users who use symlinks to point into the working directory would instantly have the newly generated dotfile.
Finally, the template system's variable support is lacking. Variables must be sourced from the environment, there is no way to source them from the YAML configuration. This seems backwards. I would expect to only keep one or two NODE_ENV-esqe variables in an .env file. They would just define the configuration file's location and dotdrop's profile name, two things that likely won't be checked into source control but are easy to set up in a new installation. All of the rest of the configuration variables would be in dotdrop's source-controlled YAML configuration file where you can take advantage of profiles and inheritance. If you have a ~/bin driver shell script it can source this file and any virtualenvs needed to run dotdrop itself. The current design that pushes all variable state to a .env file keeps a large portion of your configuration outside of source control and duplicates work already done by dotdrop's profile system.
There is clearly a lot of effort put into dotdrop and many parts of it are polished. But at the same time I think it is clear that not as much thought has gone into the templating features of dotdrop. Templates are an important enough feature that I am willing to hold off until I can either write my own dotfile management or see if dotdrop will improve its feature set.
After a sudden reduction in force at the day job I inherited a crufty old reporting system from a team of DBAs. We're a company full of Linux and a few legacy Solaris systems but the DBAs who built this thing almost 15 years ago must have insisted on SQL Server 2005 so I was the proud owner of one of the few Windows servers on our side of the business. After three months of wrangling with the report I was able to get it off of the Microsoft stack just before the Windows sysadmins decided it was time to pull the plug on this long forgotten and neglected Windows Server 2003 system. I chose Python and sqlite3 as the replacements for a mix of SQL Server queries and jobs and Visual Basic macros embedded in Excel templates for a bunch of reasons that aren't relevant here but I do have some notes on problems I ran into while moving everything over:
The default collation for SQL Server 2005 is something called SQL_Latin1_General_CP1_CI_AS. It is case insensitive. SQLite is case sensitive and the only way to change this is by declaring every text column as COLLATE NOCASE. Welcome to hell.
Some background information: in DBA speak a clustered index/table/column refers to the order of data on disk. If a table is clustered on an index or column it means that the underlying table on disk has its data in the collation order for that index or column and updates to the table will keep that order. SQL Server supports clustered indexes in a normal way. In SQLite there is no way to cluster on a given column. The underlying storage of the table does whatever it wants.
Unfortunately my reporting database relied heavily on SQL Server’s clustered indexes. Most of the data was in two tables with around 50 sparsely populated columns each, a single clustered index on the leading 7 columns or so and no primary keys or UNIQUE constraints. This sounds strange but the report was synthesizing data across multiple internal company databases whose data was populated by devices from many different vendors so it needed the flexibility while it was normalizing data from all of these sources. A typical query might have a WHERE clause that matched on the first 4 or 5 indexed columns and the remaining data would be read in a partial table scan. Thanks to the on-disk ordering provided by the clustering this was sequential, predictable I/O and SQL Server did a pretty good job with it, often beating or tying SQLite which was running on a modern i7 with an SSD.
The first, naive port over to SQLite had quite a few performance problems which I will get into shortly. The original database’s reliance on its single, giant clustered index thrashed the performance of virtually all queries queries across the entire run so I wound up rewriting many queries to use SQLite’s hidden rowid feature. Internally SQLite keeps a b-tree index to store and retrieve the physical rows on disk and the key to this index is exposed as the hidden rowid column. It is very fast to read and write queries keyed on the rowid and I made extensive use of it. As an example, here’s what a typical SQL Server query looked like:
UPDATE big_table SET blade = tmp_table.blade, port = tmp_table.port FROM big_table INNER JOIN tmp_table ON cmts_name = tmp_table.deviceName AND snmp_index = tmp_table.IndexNum AND time_stamp = ?
This becomes two SQLite queries:
SELECT blade, port, rowid FROM big_table INNER JOIN tmp_table ON cmts_name = tmp_table.deviceName AND snmp_index = tmp_table.IndexNum AND time_stamp = ?
UPDATE big_table SET blade = ?, interface_port = ? WHERE rowid = ?
Python object performance
Python's sqlite3 module has the ability to convert values to and from Python objects as they are read and written to the database. By default it ships with adapters for converting timestamps to the Python datetime class and converting arbitrary precision decimal types to Decimal objects. Both reading and writing adapters for both types benefitted from memoization decorators that cached the instantiation of those types. The increased memory usage was negligible for our data set because it had lots of identical values. Memoization successfully moved the bottleneck in some parts of the report such as loading data out of an Oracle database into SQLite from being limited on CPU usage in Python to being limited by the Oracle database's I/O speed.
One consequence of memoization and Python's reference types is that a single instance of a value will be used for every instance of the value. You must take care not to mutate that object in Python code or you will break every object and every future object that tries to reference that value. It's best to keep as much work in SQLite as possible as both a performance optimization and to minimize errors like this one.
The default adapter for the Python datetime class stores an ISO 8601 formatted string in the database. This string's lexicographical order gives sane results with sorts and inequalities but takes up lots of space on disk and prevents you from doing math on dates in queries. I replaced the default datetime adapter/converters with ones that used time.mktime to get UNIX timestamps out of a datetime and stored the resulting integer (I didn't have any fractional seconds in any timestamps). This let me replace the report's frequent usage of the DATEADD function with arithmetic on integer seconds in all but one case. For the one place where date math was required I implemented the function in Python, registered it as a function in SQLite and even got to take advantage of memoization on the Python side.
General performance issues
SQLite's query planner is not nearly as sophisticated as SQL Server's planner and most queries with large, multi-table joins had to be rewritten. The naive SQLite port of these queries would spin for hours and never complete. I had success with decomposing joins into a chain of smaller queries that would do a single join each and write their results to a temporary table, eventually ending with one last join to update the main table. The main data populating query in SQL Server joined five subquery tables onto a single large table in a ~100 line query. It was decomposed into four main temporary tables that held intermediate join results and several smaller temporary tables for the results of subquery processing which broke the giant query into manageable bits. You must take care to make sure that functions whose results depend on the order of arguments (such as coalesce) are kept in order across the smaller queries.
SQLite puts out a new release every few months. There's usually a few new features, minor performance optimizations and bug fixes in each release. Our report spends at least an hour and a half of its time in parts where SQLite is the bottleneck which is enough time to warrant keeping our own SQLite built locally and up to date. If you're on a Linux distribution that ships a few years old version of SQLite I would highly recommend you use something modern. Updating to a new version has always been painless, SQLite's code is well-tested and release management is conservative.
At some point I switched the project over to WAL logging. Although I no longer have the numbers I believe this was a win for us as it speeds up the initial bulk insert of data at the start of the report. Up until version 3.11 (released February 2016) WAL mode performed very poorly with large log and transaction sizes. Our logs for the bulk import are gigabytes in size requiring us to use the newer releases of SQLite and gave us another win from keeping SQLite up to date. The original design of the report carved its queries up into a series of about 40 high-level steps and I kept this design with the SQLite port. These steps gave us clean locations to commit the database which keeps the log size down and lets SQLite run its WAL checkpointing at regular intervals.
Database functions and features
SQL Server ships with tons of built-in functions and the original authors of the report made good use of them. I wrote a case-insensitive regular expression to match the functions and syntax that had to be changed and that helped a lot with fixing everything while porting the queries over. Here are some of the things that I had to change:
It's called substr in SQLite and there's no change in behavior.
It's called length in SQLite and there's no change in behavior. I'm kind of sad that SQLite was inconsistent with substr and length.
It's called instr in SQLite and the arguments are flipped. If you forget to flip the needle and haystack the query will still work but will return incorrect results. This was a huge pain.
RIGHT and LEFT
In SQL Server RIGHT(string, N) returns the rightmost N characters of a string. I replaced this with substr(string, -N) in SQLite. For LEFT, just use substr and keep the same N argument.
This syntax doesn't exist in SQLite but it's not too hard to change it to INSERT INTO…SELECT.
SQL Server's CONVERT(type, value) is cast(value AS type) in SQLite.
This is just COALESCE with two values. Replace it with COALESCE in SQLite.
SQL Server doesn't have the normal LIMIT functionality yet (or at least it didn't in SQL Server 2005). Convert these to a LIMIT, it's easy to match in your regex.
The INTO after INSERT is optional in SQL Server but required in SQLite. Doesn't hurt to throw insert\s+(?!into) into your regex to catch this.
+ (string concatenation)
Strings are concatenated with the SQL standard || in SQLite. There were enough string concatenations in the SQL Server code that it made sense to put + in my regex.
SQL Server has a neat syntax for creating temporary tables: prepend the table name with a #. You can create temporary tables explicitly with it (CREATE TABLE #asdf) or implicitly (INSERT INTO #asdf SELECT …). SQLite requires all temporary tables to be explicitly declared with CREATE TEMPORARY TABLE asdf
. If you have any text fields don't forget to declare them as COLLATE NOCASE in the temporary definitions as well! I had # in my regex and that worked well to find all the temporary table declarations.
SQL Server lets you declare columns as a numeric type and choose a precision and scale. See the docs for a thorough explanation. The database will enforce the precision by rounding numbers if you insert them into a column with a smaller precision. SQLite supports a NUMERIC type that preserves precision but does not allow fine-grained control over rounding like SQL Server does. The rules for the NUMERIC types are documented here. If you are not familiar with SQLite's type affinities I strongly urge you to read that documentation as SQLite's type system is unlike the ones in any other relational database.
The finished product is an Excel spreadsheet. It has enough formatting to look nice but nothing insanely complex. I used the openpyxl library to write the spreadsheet and it got the job done. The library supports lots of XLSX features but is not totally feature complete: development is ongoing and there are frequent new releases. There has been lots of minor API changes as new features are added so be prepared to read source code and the documentation to figure out how to use the library. StackOverflow is often out of date. The openpyxl code is not fast, taking over 45 minutes to create a ~20 megabyte spreadsheet. In all fairness to the openpyxl authors I did not attempt to optimize this code, once my naive attempt was working correctly it was good enough for my needs. If you have the lxml library installed it'll be autodetected by openpyxl and give you a measurable but minor speedup.
Conclusion and recommendations
At the end of the day the project worked: SQLite was a capable replacement for the much more sophisticated SQL Server and the aging Windows server hardware could be decommissioned. The Python code was vastly more reliable and debuggable than the mess of SQL Server jobs and Excel macros. I was happy with SQLite's performance and flexibility in this exotic situation and I'd use it again. If I had to recommend anything to make it easier I'd say that SQLite should support setting collation across an entire database. It is very tedious to catch every single text column to make sure it's functioning correctly. I agree that the SQL Server default is braindead but needing a custom collation is not so exotic that it can't be supported. I also think it would be neat if SQLite (which supports loading libraries at runtime to extend the database) shipped with an optional library with compatible implementations of functions available in 3rd party databases.
1. SQLite does have the ability to use a SELECT in an UPDATE/DELETE although not with the exact UPDATE…FROM schema that SQL Server has. I don’t remember exactly what issues I ran into with it, some may have been performance/functionality and lack of full JOIN support. I also believe I had to rewrite queries to get an accurate count of how many rows we were mutating which was an important feature for us. At any rate I did make use of rowid in many places to optimize deletes and updates. And I swear that the SQLite documentation had a whole bunch of stuff on semi-joins (the EXISTS operator) but it looks they've ripped that out of the documentation sometime in the past few months and you can just use regular joins everywhere.
Bitcoin engineers face a future where rising transaction fees price users out of using bitcoin for everyday commerce. There are a few solutions for lowering those fees and the latest, most hyped solution is the Lightning Network, a system built on a web of payment channels. I'll explain how payment channels work, how they can be put together to make the Lightning Network and show what this new economy looks like.
The payment channels that the Lightning Network is built on are a construction unique to Bitcoin. They're a way to coalesce a bunch of smaller transactions into a single large transfer and a solution to the prohibitive fees that each individual transaction might have if made independently. To open a payment channel you take cash on hand and spend it in a special kind of deposit transaction which opens the payment channel to the other party. It's useful to think of it as a deposit as it takes this cash and locks it up for a certain period of time where neither the sending party or receiving party can spend it. The payment channel also has an expiration date where the deposit is returned. When you want to pay the other party your software creates an updated version of the payment channel where instead of returning the entire deposit back to you at the end some of the balance is sent on to the other party and the remainder of that is sent back to you. You repeat this as many times as you want and when you get close to the expiration date the very last version of the transaction gets put into the Bitcoin network. You could have made a dozen transactions with that payment channel but at the end of the day you only had to put two transactions into the Bitcoin system and paid only two transaction fees.
I think Netflix is a great example to show how payment channels might be used in practice. With Netflix, you pay your $8 a month and you get full access to the entire Netflix library for the entire month. But if Netflix was built on payment channels instead of paying them $8 you'd take that money and lock it up in the deposit transaction that opens the payment channel. Later, when you want to watch a movie Netflix's software requests your Bitcoin software to create a new, updated payment channel transaction that sends maybe 50 cents on to Netflix for the movie and returns the remaining $7.50 back to you. Once Netflix has a copy of this updated transaction they can show you the movie. As the month goes on you repeat the same process and create a new, updated transaction for Netflix every time you want to watch a movie. At the end of the month the very last transaction is put into the Bitcoin system, the payment channel is closed and Netflix gets their money and you get anything left over from the deposit.
These payment channels reduce the number of Bitcoin transactions made reducing the total number of fees paid but they came at the cost of your liquidity and the time value of money. When you open the channel you have to have all of that money on hand in order to lock it up in the deposit. For an $8/month Netflix subscription this isn't a challenge for many but what if you were paying for groceries or gas with payment channels? You're locking up an entire month's worth of grocery money at the start of the month in order to save on transaction fees. You can't do that at all if you're living paycheck to paycheck and for everyone else you're quickly locking up all of your cash on hand in these payment channels to various vendors in hopes of saving money on fees. You can close a payment channel at any time to reclaim your money but you'll run up extra Bitcoin transaction fees when you do this and kill the efficiency of your payment channels. And finally these payment channels are supposed to enable micropayments, transactions from the furthest reaches of the demand curve. It does not make sense to open a payment channel to a merchant you might do business with once every one or even ten years for a few cents worth of business. Payment channels by themselves aren't an improvement over the status quo.
The Lightning Network works around these limitations. Instead of every participant opening channels to everyone else individuals open payment channels to central hubs. When a payment needs to make it across the graph a route is found and each payment channel along the way is updated. Here's an example network:
Bob wants to pay $20 to Amazon. There is a $100 payment channel open between himself and hub 1, a $1,000,000 payment channel open between hub 1 and hub 2 and a $100 payment channel open between hub 2 and Amazon. Bob updates his payment channel to send $100 to hub 1, leave $22 with hub 1 at the closing of the channel and returns $78 back to himself. Hub 1 updates its payment channel to send $1,000,000 to hub 2, leave $21 with hub 2 at the closing of the payment channel and $999,979 back to itself. It keeps that $1 of Bob's original transfer as a fee. Hub 2 then updates its payment channel to Amazon to send $100 to Amazon, leave $20 with Amazon and return $80 back to itself, once again keeping $1 as a fee. The payment channels used by the Lightning Network are a bit more sophisticated then what I've explained here, they are actually bi-directional and have protections built into them to guarantee that everyone helps move the money along and can't steal any bitcoins. But this is the high level view of how the Lightning Network works: users only have to open a single payment channel to their hub and the rest of the payments get aggregated in giant payment channels between hubs that handle huge transaction volumes.
That's how the Lightning Network is supposed to work, at least on paper. Nobody's written any of this code yet, so who really knows how it'll play out. The system may look different or not work at all when they come around to building it. Execution is everything and the authors here haven't done it yet. I can't vouch for the security of this system. Auditing for security is something I have no practice with and I am just not qualified to review something as complex as this. And don't expect competent peer review for a while: the prose in the Lightning Network paper is awful (although much improved from the original version) and the only people with the knowledge and stomach to get through it right now are the Kool-Aid drinkers. And finally, even if it does work the community has to start using it: everyone in the Bitcoin economy has to switch to software that uses the Lightning Network which is a big change from how Bitcoin is used today both in user and software interfaces. Rising transaction fees could push people onto this network but it'll be a painful switch.
But should it become widely used what does this network look like? At one point Lightning Network engineers were using the name "hub" in the same way I did above but lately they're promoting a vision of a wider P2P network with more intermediaries and smaller payment channels between them. Unfortunately market pressures work against decentralization here and the model of a few, large hubs holding open massive payment channels is the future. Automated routing that finds the cheapest path across the market will reward large hubs that can create short paths across the network and service huge volumes and leave the smaller, lengthier chains without customers.
It's also slightly dishonest to call these hubs instead of banks. In the same way that the individuals using payment channels by themselves needed to lock up cash and manage their liquidity in order to create payment channels these "hubs" need to deal with getting access to huge amounts of cash that they can tie up to open their own massive payment channels. The name for an organization that makes its profit from carefully managing its investments and liquidity is a bank and that's exactly what these hubs are doing. And the only organizations that have the kind of cash (or access to the cash) to invest into a payment channel is a bank. Hopefully these hubs look like investment banks and get their cash from the same customers that investment banks have but there's not a whole lot stopping them from being retail banks capitalized by the deposits of ordinary people. Investing in a Lightning Network hub is low risk and low return. Your losses are limited to the cost of bitcoin transactions but your profit is the little bit of fees you make off of being a hop in the chain. Competitive pressure to keep prices down is high because anyone can spin up a node and nodes with smaller capitalization can still compete on lower fees and better connectivity. If you really need the cash you can close out the payment channel at any time so it's at least a pretty liquid investment. If the Lightning Network finds common use these will be attractive investments for retail banks. Now we've come full circle to something similar to our current system only with Bitcoin and the Lightning Network.
If the Lightning Network catches on the underlying Bitcoin network needs to be able to handle its traffic demands. The biggest debate right now in the Bitcoin world is over the future of the block size, the maximum amount of data that can be used to update the Bitcoin blockchain. On one side of the debate is the sensible, pragmatic engineers who want to keep the block size increasing with an eye towards future growth of the Bitconomy and future increases in computing power. On the other side is a bunch that wants to keep the network running at its current transaction rate for at least a few years. The constant size group (called "small blockists" by the larger Bitcoin community) have a bunch of motivations behind their decision I won't get into here. This is relevant to the Lightning Network as that same small blockist group heavily pushes Lightning Network as a workaround for small block sizes. Because the Lightning Network coalesces the transactions that need to be put on to the blockchain it lets you get more use out of that small block size.
Even though the small blockists are some of the biggest proponents of the Lightning Network because at its face it makes small blocks look like they can handle the network's traffic demands the Lightning Network itself requires large blocks! I didn't go into too much technical detail in this blog post but the Lightning Network has several systems that keep everyone participating honest. One of them is a special Breach Remedy transaction that keeps either party in a payment channel from reverting to older channel balances. If you try and revert back to the older version of the channel the opposite party can spend this Breach Remedy transaction which closes out the channel and gives all of the money in the channel to the party that didn't cheat. This is meant to be a deterrent to keep both parties honest but it only works if you can reliably use the Breach Remedy transaction. Blockchain congestion is a failure mode for the Lightning Network because they assume you can always get a transaction included should you need to do so. There are a few hacky workarounds in the Lightning Network paper for this but for successful implementation of the Lightning Network you'll need large blocks or possibly even unlimited blocks. Miners can still be profitable in a large block world - they can price discriminate against those 'million dollar' inter-hub payment channels and charge them larger fees. So even though the current LN boosters might be pushing small blocks they will eventually need to be large or even unlimited.
If this network works it's also a pretty good realization of the dream of Austrian economists. Bitcoin is a hard currency and with the Lightning Network it really beats out the incumbent hard currencies on ease of use and features. It may even be possible to capitalize Lightning Network payment channels with smart contracts that guarantee the funds from closed channels are returned to their investors. So not only do you have a hard currency that can be spent easily, securely and quickly but you've created 100% reserve banks in the Lightning Network hubs that can still earn a tiny bit of profit and can't be plundered by management. Austrians will love it! But what happens when the business cycle contracts? Austrians solve this problem by pretending that never happens. For the rest of us, though, it's an inevitable part of business and we need to know how the system works when that happens. The revenue from payment channels is proportional to the volume of transactions they handle. If that goes down the payment channel's yield goes down and they lose their attractiveness as an investment. If capital starts fleeing elsewhere and the number and size of the payment channels goes down the entire economy's velocity is brought down by higher Lightning Network fees competing over fewer payment channel space. And if it gets really bad it turns into another Great Depression as there is no central bank to add liquidity to the payment channel market. Everyone has to sit on their hands until everything else in the world tanks to the point where investing in payment channels becomes attractive again. You may not be able to spend your money on anything because the Lightning Network is non-functional or too pricey but you'll still have full access to your bitcoins whether they are in your 100% reserve bank or your personal bitcoin wallet. Is that a better system than what we've got now? It's interesting to think about and it rounds out a bunch of hard corners in the usual Austrian proposals.