Notes on Marc Wortman's “Admiral Hyman Rickover”

Based on what I’ve read and watched, Hyman G. Rickover, lauded as the “father of the nuclear Navy”, does not seem to have been what I’d call a good person. Nor does he appear to have been a happy person. But wow, did he ever get it done. So while I don’t include him in my list of people I respect and aim to emulate, I can’t help but admire his achievements.

I recently read Marc Wortman’s solid new biography Admiral Hyman Rickover, Engineer of Power and was struck by its many resonances with Caro’s bio of LBJ. Both men were possessed by a single-minded obsession with achievement regardless of cost to themselves or anybody else (and suffered multiple heart attacks, possibly as a result). However, LBJ makes Rickover seem angelic by comparison, both in his general personality and in his motivation. Rickover wrote love letters to his wife and was apparently faithful to her; he loved his son and spent time with him, though his mad devotion to his work made that time rather limited. LBJ seems clearly unencumbered by that sense of morality or interest in his family.

Rickover was uncompromising in his principles, which were: first, safe operation of nuclear power reactors, both at sea and on shore, for both their operators and the public; second, the careful use of public funds; and third, the highest standard of hard work from himself and his staff. LBJ appears to have simply lacked principles from birth and was a paragon of cheating, corruption, and dirty tricks, though he also worked extremely hard. Of course, the deep mystery at the heart of Caro’s work is why such an unprincipled person would fight to get the Civil Rights Act passed, and whether there actually was, in the end, a principle of fairness and social justice operating way down deep in LBJ.

Both men grew up in poverty but it had a different effect in each case. Rickover always lived simply and humbly, the Platonic ideal of a civil servant, and transferred his sense of the value of a dollar to the taxpayer, fighting tooth and nail with defense contractors. LBJ, by contrast, was avaricious to a stunning degree, using his position of power to accumulate vast wealth for himself and his backers.

But what stands out to me most from both stories is the unusual relationships both men had to their “host” institutions. Normally an individual wields power within an institution by obtaining a position of power within the institution’s hierarchy. But in the case of LBJ and the Senate, he subsumed the senate; he brought it under his control in a way never seen before or since (see Master of the Senate). By contrast, Rickover didn’t subsume the Navy—it being much too vast—but somehow obtained and maintained his power despite the wishes of his host organization by building his own, effectively independent institution within the Navy, like a growth, which the Navy tried to excise for decades—without success due to Rickover cultivating powerful patrons in Congress. Rickover was basically an entrepreneur in his approach, ignoring all the rules and outrunning and outmaneuvering those who were trying to shut him down. But he was an entrepreneur within a colossal organization and operating with his own agenda that almost always ran counter to the Navy’s. I can’t think of another example like this, though I’m sure they exist.

Both men seem to have had abusive relationships with underlings, possibly enjoying humiliating and degrading people for its own sake, though this is to be somewhat expected within the military. But in their relationships upward in the chain of command, LBJ was obsequious and Rickover was pugilistic.

I think the deepest commonality was that both men had superhuman motivation to achieve their ends. But Rickover’s ends were generally the public good, and LBJ’s were generally his own good. And despite being generally despised, and usually opposed, both men managed to bring about enormously impactful and lasting achievements.

Leadership is Really Just Three Things

I’m often surprised by how unclear people are on what leadership is and what leaders do. This is partly due to the barrage of confusing nonsense out there like “leadership is about making sure every voice is heard” (sorry, no). But while it may often be difficult to do, the good news is that leadership is very easy to understand. Fundamentally, leadership is about getting multiple people to accomplish one thing. Everything else is just implementation details.

In many areas of human endeavor, starting with ancient ones like hunting and warfare, a team will usually outperform an individual (due to greater numbers) or an uncoordinated group of individuals (due to greater effectiveness). And leadership is usually required to form, direct, and motivate a team. In other words, leadership exists because some tasks require more than one person to pull them off, more coordination than just showing up, and more motivation than just hanging out.

For instance, let’s say a band of hunter-gatherers spies some bison grazing near a cliff. One person alone can’t do much to move the herd. But many people acting together can surround the animals, get them into a panic, and drive them over the cliff to their demise.¹ For this to happen, someone needs to have the basic idea and sell everyone else on it—“Hey, what if we worked together?”; someone needs to form a plan on how to get it done—“Ok, you’ll go here, I’ll go there, she’ll go that way and start shouting” (and direct the action in real time as the situation evolves); and someone may need to supply motivation if the band is more fearful of getting trampled than hungry—“C’mon, this will provide meat for the whole winter!”

So in my analysis, leadership involves three distinct functions: unification, direction, and motivation. These functions can all be performed by the same person, or each function can be performed by different (or multiple) people. It doesn’t matter; it’s all leadership. For instance, a great orator can persuade several clans to join forces to defend their land from invaders (unification). Then a trio of generals can decide the strategy of the now-unified army (direction). And then a dozen courageous captains can lead their men to victory by remaining confident about their chances, choosing tactics, and fighting courageously (direction and motivation). Every one of these individuals can be fairly said to have been a leader in the victory, though they did so in different ways and at different scales. (In some cases a situation may be so threatening that it spontaneously creates unity without anyone needing to instigate it, and supplies plenty of motivation besides, leaving the leaders to focus on directing the response to the threat.)

To beat this horse to death, for a group of humans to successfully accomplish a shared mission, three things must happen. First, they need to become a team, which usually entails defining the mission, and may also require persuading each member that they’re better off as part of the team than on their own, or even negotiating the terms of a union. Second, they usually need to undertake the mission as a directed force rather than chaotically, lest a key advantage of being unified (coordination) be lost. And third, because humans have their own interests and will often cheat, desert, or slack off if they can get away with it, they need to be encouraged to try hard and/or punished for trying to dodge their responsibility. A leader is simply anyone who supplies one or more of these aspects of leadership. Great leaders are able to do this in very difficult situations.

To sum up: operating as a team is massively advantageous in many high-stakes situations, and leadership is the art of creating and operating teams to maximum effectiveness. It’s the key to how a group of human individuals can transform into a distributed superorganism for the benefit of all its members. Ants don’t need leaders to achieve this! But humans do.

Amplifying expertise — and accelerating learning — through apprenticeship

“Organization is a means of multiplying the strength of an individual.”

—Peter Drucker

In software engineering, conventional wisdom holds that the time a senior engineer spends not being hands-on is time wasted, and this includes time spent discussing and reviewing the work of junior engineers. As one writer put it: “[Having] junior developers report to senior ones…reduces the time spent on development by the best technicians.”

I contend the opposite is true: any time a lead engineer spends doing low-value work like writing accessors is an expensive waste, like using an atomic clock to time your eggs. I’d say the ideal use of a senior engineer’s time is directing junior engineers in the execution of designs, investigations, and experiments. At architecture firms, partners don’t make the blueprints.

In my ideal dev team, a master engineer designs the solution, then communicates it to midlevel engineers, who create skeleton classes and tests and then hand these off to junior engineers, who fill in the required functionality. Defects flow in the opposite direction - junior engineers do the initial investigation, asking for help if they need to. Once the bug has been found, the junior engineer can fix it or escalate it; either way, his solution is reviewed by a more experienced engineer.

In this way, the junior members of the team learn from the senior members, just as they do in law firms and hospitals - under close supervision. This is very different from just turning inexperienced engineers loose on the codebase, as is commonly done. I know I would have benefited enormously from closer supervision and assistance early in my career, in both far greater productivity, which would also have benefited my team, and accelerated professional development.

What does it mean to be a dev team lead?

In journalism, a reporter writes a story and an editor edits it - reading through the whole piece, catching errors or omissions in writing or reporting, asking questions, and making suggestions at all levels - from how to catch and hold the reader’s interest to how to structure the story to how to smooth out an awkward paragraph or sentence. This is A Good Thing, because a) it’s always good to have someone else go over your work and b) the editor is usually the more experienced writer. In fact, it’s such a good thing that on the occasions when an editor writes a piece, it undergoes something called a “top-edit”, where it’s edited by another editor. This is because it would obviously be crazy to publish something without someone else having read and reviewed it. But in software engineering we do this all the time, and consider it a cornerstone of our hacker-ethos, meritocratic culture.

I think this is bonkers. I’ve noticed that the code of junior engineers is often flawed, sometimes at multiple levels, in ways that are easy to avoid and easy to teach them to avoid. And I’ve noticed that if I delegate a modest-sized project and then walk away until a code review a few weeks later, it will often be the case that the architecture was ill-conceived and the process won’t scale, or has lots of nasty failure modes, and we’ll have to either go with what’s been built or go back and do it properly.

I’ve also noticed that junior engineers will silently bang their heads against things that I can often help them resolve if I just occasionally stop by and ask them how things are going. But instead of just giving them the solution I’ll try to teach them how to ask the right questions and design and run experiments to figure out what’s going on for themselves.

Eventually, I came to think of these editorial and mentoring activities as not ancillary to or a distraction from my job, but the core of my job, and my reasoning comes down to amplification. In journalism, you typically have several reporters funneling their work through one editor, and it seems obvious to me that the aim is to achieve the volume of output of many writers at the quality level of a very experienced writer, while maintaining a consistent voice across the entire organization. The editor isn’t sidelined by not doing her own writing, she’s amplified. This pattern is everywhere - from the fine arts to the sciences to law to most trades I can think of - but I’ve never seen it in software engineering, perhaps because senior engineers tend to enjoy doing their own coding and resist being pulled away from it.

So I now view my role as something like an editor, coordinating and overseeing a team of engineers to amplify my own skill and experience and help them develop theirs. Any time I’m off by myself churning out test cases is time I’m not solving hard problems or helping someone else learn how to solve a hard problem. But for any code I write, I implement top-edits by submitting pull requests for my own contributions - and sure enough, junior engineers will catch me making stupid mistakes. Also, having them examine my code critically keeps them abreast of changes to the codebase and gives them a chance to learn new techniques and ask questions. It’s similar to pair programming, but one of the pair is always me, and I’m always rotating among the team, offering help and feedback wherever I can. The result? The productivity of many engineers, at the quality level of a lead engineer, and a happy team, rapidly growing their skills.

How to add NAT to an existing EC2 instance in an AWS VPC

I’m in the process of moving infrastructure from the AWS public cloud into a VPC, and since my VPC has private subnets, I need NAT to allow machines in these subnets to open connections to S3, Ubuntu repos, etc. The VPC wizard can create a NAT EC2 instance for you, but I’m a penny pincher and didn’t want to dedicate a whole instance just to NAT - e.g. I also wanted to use it as my VPN server.

But I couldn’t find specific instructions on exactly how to add NAT to an existing instance with a single NIC (most solutions require two NICs in a standard gateway configuration, one for the LAN IP and one for the WAN IP, which you can actually do with VPC, but why do it if you don’t have to?). After a bit of research and some reverse engineering of the AWS NAT solution, here’s what I came up with:

1. In the EC2 console, disable source/dest checking by right clicking on the instance you want to use for NAT and choosing “Change Source / Dest Check”.
2. Create a security group having an inbound rule allowing ALL from 10.0.0.0/16 and associate it with your NAT instance.
3. On the NAT instance, create /etc/network/if-pre-up.d/nat-setup as:

   #!/bin/sh
   echo 1 > /proc/sys/net/ipv4/ip_forward
   iptables -t nat -A POSTROUTING -s 10.0.0.0/16 -j MASQUERADE

1. chmod +x the script, then run it. This script will automatically be run when the machine reboots, so your NAT will survive a restart.
2. Make sure all your private subnets have a default route to use the NAT instance as a gateway (create a route for 0.0.0.0/0 and associate it with your NAT instance in the route table(s) associated with your private subnets).
3. test your NAT by pinging something from an EC2 instance in a private subnet

Note: the NAT instance needs to have an Elastic IP, of course.

How the relay got its name

I’ve always wondered how the relay got its name; I thought it was a strange moniker for an electromechanical switch. Reading The Information the other day, I finally found out. I had thought relays were originally invented to enable automatic circuit switching in telephone networks, but it turns out the telegraph was the necessity of their invention.

Telegraph signals weaken as they travel down a wire; beyond a certain distance the original signal can no longer power a receiver. So what do you do? You say, well, let’s have an intermediate telegraph operator: he can be stationed where the signal is still intelligible, and simply re-transmit what he receives, using his own power supply to generate a full-strength signal.

And then someone clever realizes you can use an electromagnet to operate the re-transmitting key and the incoming line to power it. Yes, the first relay was a telegraph key with an electromagnet stuck on it.