(via milkysa)

1. What song or album are you currently addicted to?


2. What game are you currently playing?

The Internet Meme game.

3. What was the last movie you saw?

Word Wars, a 2004 Scrabble documentary. Solid!

4. Asian or Mexican?

There is this Indian burrito truck called Curry Up Now. The food is pure genius. The people seem very nice. The overall experience is a bit frustrating.

5. My feet are cold. Are yours?


6. What’s your current fandom/obsession/addiction?

Computational geometry.

7. Are you a cat person or a dog person?


8. What’s the last thing you broke?

REDACTED. Don’t worry, it’s fixed now.

9. What was the last thing you bought?

Blueberry toaster waffles from Trader Joe’s. Not my favorite flavor, but there’s apparently been some sort of nationwide toaster waffle shortage in recent weeks, and these are the only ones to have reappeared thus far.

10. Do you have more friends online or off?

There have been times in my life when that question would have made a lot more sense than it does now.

11. Something that made you laugh today?

Waving at Jessica as I pulled the car out of the driveway this morning. It got so wavy in there!

12. Favorite non-alcoholic beverage?


13. What was the last meal you ate?

Chicken salad sandwich on organic baguette with inorganic yellow mustard, sliced industrial tomatoes, baby spinach leaves, and brie; and quinoa salad with tomatoes and red onion and red cabbage and lots of freshly squozen lime juice. Both delicious! I made the sandwich, and Jessica made the salad.

14. What kind of car do you drive?

2010 Toyota Prius. I keep wondering how hard it would be to reflash the navigation computer with some more tractable software. Am I really the only one who wonders that? Come on, Internet!

15. Any plans for today?


16. Favorite TV show that you don’t normally tell people you watch? Favorite TV show that you think more people should watch?

30 Rock. Caprica.

17. What kind of music is contained in your hard drive?

The complete works of Adam Benjamin. Ha, I only wish that were true. I don’t even have his latest CD yet.

18. Are you attending conventions of any kind this year?

Not insofar as I am aware at this time.

19. Something you’re scared of?

Skeeter eaters! They’re called that because they’re supposed to eat mosquitoes, even though they don’t eat anything. They just flutter around terrifyingly.

20. Right or left handed?

Tag, you’re it! Yes, you! Right.


Long Live toastball!

My computer died again.

The computer in question is toastball. That’s the one that currently lives at the Shiny Devices workshop, over on Coliseum Way, and serves the web site, and runs the robots and so on. Why yes, the Shiny Devices web site is served by the very same computer that controls the machines that makes the devices that are available for sale on the Shiny Devices web site.

Or it was, at least. Those of you who have been paying attention will already have noticed that is up right now, and has been up this whole time, with nary so much as a momentary glitch. That’s because this started out as scheduled downtime. My landlord (Rod) left a note in my mailbox [which is not illegal, by the way — these are different mailboxes from the one the USPS delivers to], from PG&E, about how they were going to be cutting the power sometime between 9 p.m. on Friday and 6 a.m. Saturday, for several entire blocks around Coliseum Way and High St, in order to “install new equipment.” So I switched the DNS over to OFB earlier in the week (I’ve been keeping automatic nightly backups there since forever), and shut everything down on Friday in an orderly way. But when I came back on Saturday, toastball refused to power on. Sad!

I’ve pretty much torn the machine apart now, and my guess is that it needs a new power supply. Either that or new RAM (yeah right) or a new CPU (seems unlikely) or a new motherboard (depressingly probable). But oftentimes, when a toastball dies, it seems to be the power supply.

I have a wild theory now, actually, about why that might be. I’ve been using an uninterruptible power supply — a battery backup power supply for my computer, basically — ever since we had those rolling blackouts back in 2000 or whenever that was. Possibly even before then. Anyway, I can’t help but wonder whether the UPS might be cranking power in the form of a nasty sharp square wave, rather than the nice rounded sine wave PG&E delivers. Could be it only does it when it’s running on battery power. But my theory is that the power supply is desperately trying to smooth out the high frequency nastiness and round off the corners of the squares, and that the resulting stress is reducing its life expectancy.

I’m kind of tempted to investigate this theory by hooking a scope up to the battery-backed outlet in the UPS and see what the waveform looks like. But that wouldn’t be very safe — I don’t have the right kind of equipment in the workshop for dealing with high voltages. It isn’t that the measuring devices won’t go that high; in fact the garden-variety multimeter I’ve been using will measure 600 volts or more, no problem. But you’re not supposed to just stick the little probes in the wall socket and measure the voltage that comes out, because that would be a dangerous and stupid thing to do.

You know, I have a funny story about that, actually.

Some of you will have heard this story before. This is the one about the beautiful apartment that Jessica and I are renting, with its expansive windows overlooking Lake Merritt and its gorgeous non-conductive hardwood floors.

At some point Jessica went down the Oakland public library to find out more about the building we lived in. (Thanks Jessica!) They ushered her into this special room where they keep all of these old maps and Thomas Guides and property records and original plans filed with the fire marshall — all this great stuff! And she discovered that our building was built around 1920, as a 2-unit dwelling, by an unmarried woman named Louise Graff, who was sufficiently well-to-do to build a building and own a car. Louise herself lived in one of the units. She must have kept on building, because by 1925, there were 3 units, and as I write this in 2010, there are 5. Our back bedroom is a few inches higher and has different floors and was clearly added on later. My point is that it’s a beautiful apartment, but it’s had time and opportunity to accumulate a few quirks.

Now, as a computer person who had lived in a couple of older apartment before, usually with a mixture of 2-prong outlets and 3-prong outlets, some of which were not actually hooked up to power, I was glad when I saw that this place had 3-prong outlets everywhere, and that the correct lights lit up on my little 3-light outlet tester when I tried them. Not that it would have influenced our decision much — we’d fallen in love with the view and the floors — but it’s always good to know about these things before moving in.

About a month after we moved in, I flew down to Burbank and drove back in a rented SUV with a brand new fully automated pick-and-place machine in the back, which I temporarily set up in the back bedroom (thanks Marc and Blake!), along with the toastball of that era (the Shuttle SN41G2), so as to reverse-engineer the serial protocol that the horrific Windows control software that came with it was using. [This was also perfectly legal, as I understand it, even though I don’t think our apartment is zoned for light manufacturing; Shiny Devices was not yet open for business, and I wasn’t manufacturing anything back there.]

Most people have never seen or heard of a pick-and-place machine. It’s a type of industrial robot that picks up unbelievably tiny surface-mounted electronic parts [think sand grains] with a tiny little robotic vacuum nozzle and places them on circuit boards that have been pre-smeared with solder paste, usually with a laser-cut stainless steel stencil. Then you run the boards through an oven (usually the kind with a conveyor belt, so you don’t have to open and close it) that melts the paste — a mixture of powdered tin, a bit of copper, a smidge of silver if you’re not too cheap, plus the nasty liquid chemical solvent equivalents of whatever it is that gives Velveeta that smooth creamy consistency in the microwave — and turns it into solid metal connections between the chips and the board, which you then shove into an injection-molded plastic case and wrap up in child-suffocation-warning-adorned plastic film and a glossy pre-printed cardstock box and ship off to Best Buy by the full truckload. That’s tradition, anyway. Shiny Devices is a little bit different.

In order to place the parts accurately, most pick-and-place machines have one or more video cameras connected to powerful computers that use clever machine-vision pattern-matching algorithms to place each part in exactly the correct position and orientation. Mine came with a fancy PCI card with six video inputs (this particular robot only has 4 cameras) and a 3.5″ disk (remember those?) with a license key on it for the terrible proprietary Windows-only device driver that knows how to talk to the PCI card. (I had already explained to the nice folks who made me the robot that I would be providing my own computer.) There was no way that card was going to fit inside toastball’s stylish but cramped Shuttle case, so I borrowed an old hulking Dell from my brother Marc (thanks Marc!) and installed the PCI card and the factory-provided Windows-based robot control software on it for long enough to make sure that everything worked, with toastball as man-in-the-middle monitoring communications between the software and the robot.

Remember how I said the robot has 4 cameras? Three of those are miniature black-and-white video cameras that put out a regular TV signal (NTSC). The fourth, amusingly, is an ordinary Sony digital camcorder, complete with the original glossy pre-printed cardstock box, that attaches to a standard camera mount on the robot frame. I even bought it a little Sony digital video cassette as a present, just so that it would quit whining about not having one. It has a regular TV out, which I’d connected to the 6-input PCI card, but (being a digital camcorder and all) it also has a digital output, which uses a connector that Sony calls iLink and IEEE calls IEEE 1394 and Apple calls FireWire, or DV when it’s being used for video. As it happened, toastball had a FireWire port already, and when I downloaded dvgrab and got the appropriate cable and plugged it in, everything worked great — so well, in fact, that I ended up purchasing an NTSC-to-DV converter box and a computer-controlled video switcher box to hook up the other three cameras. Which meant that toastball would only be able to see one NTSC camera at a time, but that was OK.

Now, a very interesting and unexpected thing happened the first time I connected the video cable between the video switcher (which was plugged in to the three cameras on the robot) and the video converter (which was connected to toastball’s FireWire port). There was a loud noise! And sparks! And smoke! And toastball’s screen went blank! Yikes!

Well of course I unplugged the cable as fast as I could, and when I’d recovered my composure a bit I assessed the damage. The video converter appeared to be dead. toastball refused to power up. The robot, video switcher, cameras, and everything else seemed to be just fine, thank goodness.

So I dialed the customer service number for the company that made the video converter, and I explained to them that I’d just bought their product, and I sort of outlined to them what had happened the first time I’d tried to used it. The fellow who took my call asked me a couple of basic questions, then explained what had undoubtedly happened. The video converter was indeed fried, as was my computer, but it wasn’t strictly the video converter’s fault. Possibly I had inserted the FireWire connector backwards. (It’s an idiot-proof connector — one side is rounded, the other side has square corners — but I can imagine someone managing to force it in backwards if they were sufficiently motivated.) More likely, my computer’s FireWire card (the motherboard, in this case) was shoddy in some way. The fellow went on to explain that their policy was to allow their customers to make this mistake exactly once, but that I should try to be more careful next time. I decided to bite my tongue. He was offering to replace the dead video converter, after all, which was what I wanted.

The new converter arrived soon after, postage paid and exactly as promised, as did the new Shuttle XPC power supply I’d bought online to see if I could resurrect toastball. In the meantime I’d transplanted its hard drive soul into Marc’s old computer, where it was a bit disoriented but managing well enough.

Replacing the power supply wasn’t enough this time, as it turned out, which meant I’d probably need a new motherboard, which meant I’d probably need a new computer, as SN41G2 motherboards were already hard to come by and you’d have to completely disassemble everything and it was so cramped in there that it hardly seemed worth it to go to all that trouble to resurrect what was really quite an obsolete computer. So I decided to shop for a new toastball.

In the meantime, I figured I’d try the new video converter with Marc’s computer, to make sure it worked. I admit to feeling a bit of trepidation when I connected the video cable, which turned out to be justified when the sparks and smoke came out and the screen went blank again.

Now I had two dead computers! (Sorry Marc!)

Well, of course I felt pretty silly! So I resolved to figure out exactly what the heck was going on here before connecting any more video cables to anything.

The whole situation was extremely puzzling! The thing about video cables is that, in principle, they carry only relatively tiny electrical signals back and forth. Certainly not enough to make fireworks! Not normally, anyway.

What about the FireWire cable? Well, according to Wikipedia, the fellow on the phone had a good point there: FireWire cables are actually set up to deliver quite a bit of power, in addition to information. In fact this video converter was being powered directly from the FireWire cable.

But when I plugged the FireWire cable between the video converter and my freshly-assembled computer (toastball), everthing seemed to be fine: the little power light lit up, and there were no sparks or smoke or explosions or anything. It was only when I had connected the video cable that something had gone wrong. Clearly, there was a lot of power flowing through that cable for some reason.

I’d been reading a lot about electronics recently, perhaps because I was going into the electronics business, and one of the things I’d been reading was Tim William’s excellent book, The Circuit Designer’s Companion, which I highly recommend. The very first thing that he talks about in this book is grounding — how important it is, and how easy it is to get wrong, even though it seems like such a simple, basic thing.

You can think of electricity as being a bit like water flowing through a pipe. Electricity is to water as voltage is to water pressure: just as differences in pressure between point A and point B will cause water to try to flow through a pipe between them, differences in voltage are what cause electricity to flow. If I pick up a piece of pipe that’s open on both ends and drop it in a bucket of water, the water won’t spontaneously start flowing through the pipe, because there’s no difference in pressure. This is true whether the bucket is sitting on the surface of the moon or whether it’s at the bottom of the ocean. The absolute pressure doesn’t matter; only the differences. It’s the same way with voltage. When people talk about “zero volts” (or “ground”), what they’re really talking about is the arbitrary reference voltage that they’re measuring every other voltage in the system against.

The point that Tim Williams makes in the very first chapter of his book is that, if you’re going to assume that ground is ground and has the same voltage everywhere in your circuit, you’d better take steps to make sure that stays true. He gives a few examples, one of which involves a workbench in an electronics lab he used to work in where the ground prong in the electrical outlet unexpectedly turned out to have a voltage that was something like 50 volts off from ground in the rest of the lab. As long as you were making all of your measurements relative to that particular ground, everything looked fine — and in fact it was fine. But if you were to plug something in to that lab bench and try to make a measurement with a scope that was plugged in elsewhere…. Well, 50 volts is starting to be a lot of volts, you know? If part of your body were the only thing standing between two things that were 50 volts apart — I mean, it wouldn’t kill you or anything, probably, but you would definitely notice.

I guess that anecdote must have stuck in my mind, because I started thinking: Gosh, you know, the robot is plugged into that outlet over on the far side of the room, and toastball is plugged into this outlet over on this side of the room. What if one of these outlets were like the one in the story? All of these cables have a bunch of different wires in them, and one of the wires is always ground — it’s the reference voltage that everything else pushes against. Usually everything that’s ground is connected together. So if ground were a bit different on each side of this room, you’d get current flowing through that video cable. Maybe even enough to make sparks. How much would you need for that? A few volts? More? I guess I have no idea.

Anyway, it seemed like an easy enough theory to test. I got a voltmeter, and I got the two things I’d plugged together, and I carefully stuck the little probes onto the little conductors on the connectors to try to measure how much voltage would be trying to leap across them if I were to connect them. (It wouldn’t be able to actually leap across; in order to measure the pressure difference between two points, you have to stop the water from flowing between them. Otherwise the pressure will end up being the same!)

Unfortunately the readings I was getting on the voltmeter were not very helpful. They were usally pretty stable, which was good — if you try to measure the voltage between two things that aren’t electrically connected at all, it tends to sort of drift around randomly within a certain range. And they weren’t quite zero. But they were awfully close. The voltage difference seemed to be hovering right around 0.3 volts. Could you get sparks and smoke from 0.3 volts? I was kinda pretty sure you couldn’t, actually. That’s just not a lot of volts. In fact, 0.3 volts is about exactly the minimum amount you need in order for your average piece of sensitive digital electronics to be able to start to notice that the voltage is different at all.

By now I’d been banging my head against this problem for a couple of days, and was starting to get desperate enough to wonder if maybe I should resort to trying to scrounge up another old computer or two just to be able to do more experiments in order to have some hope of making progress. And then late one night I had a thought.

I always set my voltmeter to measure DC volts, as a matter of habit. What would happen if I turned it to AC instead?

DC is electricity flowing steadily in one direction. AC is electricity flowing back and forth and back and forth, rhythmically, alternating directions — there might be lots of electricity flowing back and forth, but if you total up the net flow, you get zero, or something pretty close. Which turns out to be a lot more efficient, in a lot of cases, for reasons I won’t go into today, even though AC seems a bit more complicated to think about. That’s part of the reason why household current is AC, not DC — unlike the teeny digital circuits I’m used to dealing with, PG&E ships a pretty sizeable amount of power around the state, and the efficiency gains end up being totally worth it.

Here’s a better analogy. Instead of water, say the electricity is people who want to visit a museum. The museum lobby is like a wire, and the museum itself is, I dunno, a rechargeable battery or something. The multimeter is the guard who stands in the lobby with the little clicker to count how many people go past. (Actually that guard is measuring current, not voltage, but in this case they’re proportional. Current is how much electricity or how many people are actually flowing through the doors. There are basically two things that determine how many people come in: the voltage, which is how motivated the people are to go in or out [“OMG, is that the Mona Lisa?”], and the resistance, which is how hard it is to get in or out [“What do you mean, tickets are $400?”]. The factors that get lumped together under “resistance” can be broken down further into things like the cross-section of the wire [number of doors or turnstiles or ticket lines, or the cross-sectional area of the pipe in the water analogy] and what the wire is made of — some metals have higher ticket prices than others, or else they enforce their no-glass-bottles policy by X-raying purses, or they have elevators, or they make everybody wear blindfolds and go through a maze. Or something. I don’t know. What am I, a materials scientist now?)

Now let’s say you’re the museum guard with the little clicker, and your job is to make sure that there are no people left in the museum when it’s closing time. You’re going to want to add 1 to your count when somebody goes in, and subtract 1 when somebody leaves. At the end of the day, the count should be zero. You are measuring DC (net) current (people) averaged out over a certain time scale (one business day). The direction the people/electrons are going (in or out) matters a lot to you in that case.

Next let’s suppose you wanted to measure the people/current by sticking part of your body — your finger, say — into the stream of people, such that they have to go through you to get in or out, and measuring the pain you feel whenever somebody steps on your finger. It turns out that when measuring the flow of people in this way, your finger feels pretty much exactly the same amount of pain whether the people are entering the museum or leaving the museum. Your finger is measuring AC people. This might be a good method to use if you are a masochist who wants a very crude estimate of total ticket sales. Similarly, you could try to make a very rough measurement of the the AC voltage of your electrical outlet by sticking your fingers in there and seeing how much it hurts. Strictly speaking, to estimate voltage, you have to factor in the combined resistance of your finger and of the rest of the lobby in order to deduce how desperate the people must be to either visit or escape from the museum, but that’s probably just some fixed scaling factor that you could lump in to the whole “volts per OW!” conversion.

So the next day I went back to my sketchy setup with the connectors and the voltmeter and I turned the little knob from DC to AC and the number on the meter went from 0.3 to 110. Just like that.


Which brings me back to how I got started on this topic: This particular multimeter is designed in such a way that it can accurately measure voltages of at least 600 volts, and sometimes even 1000 volts, without smoke or sparks or whatever. What it isn’t designed to be able to do is to protect the human being presumed to be holding the meter from those kind of voltages. There are some fairly well-thought-out rules about these things, you know? It sounds a bit confusing at first, but it’s a perfectly sensible distinction to make. What I was doing right then with these fiddly little metal probes was safe for the meter, but it was not especially safe for me. Or for toastball, for that matter.


But wait! It gets sketchier.

Remember that 3-light outlet tester I was talking about? You can find them at almost any hardware store. They’re cheap. There are three lights because there are three prongs on an electrical plug, and if you want to look at the voltage across every possible pair of prongs, there are three pairs. So each light tells you whether there’s a voltage difference between a particular pair of prongs. There are two yellow lights for voltage between hot (110 volts AC) and neutral (0 volts), and between hot (still 110 VAC) and ground (also 0 volts). There’s a red light for voltage between neutral and ground.

Now, you might wonder why an eletrical plug would have two different prongs that are both supposed to have exactly the same voltage, namely 0 volts. Why not just have one prong? And, of course, in older apartments with 2-prong outlets, that’s exactly what you find. The reasoning back then was that it doesn’t really matter which prong is hot (110 V) and which is neutral (0 V); all that matters is that the difference is always 110 V. (The direction in which the electricity is going doesn’t matter either, because this is 110 volts AC, so the direction changes back and forth constantly — 60 times per second, in this case.)

That’s fine reasoning if all you care about is your stuff getting the power it needs, but if you also care about safety, then you also might decide that you want to pay attention to the voltage difference between (say) the metal casing of my toaster or computer or what have you, and the voltage of the person who might be touching that case. If the difference is 0 volts, nothing happens and everyone is happy. But if the difference is large — 110 volts, say — then you will notice, and you will be unhappy. This can happen if, for example, a live wire comes loose inside your appliance and comes into contact with the metal case. It’s pretty rare, but it can happen.

The idea behind the third prong is that that prong is electrically connected directly to ground. There’s a big metal spike or pipe or something sticking deep into planet Earth at a single point underneath the house, and there are all of these wires radiating out from that point to all of the ground prongs in the outlets, keeping them at the same voltage as that patch of the actual literal ground. Which is why it’s called that. There is also an entirely separate network of wires radiating out from that single point to all of the neutral prongs on all of the outlets, and yet another wire connecting that point to the ground wire coming from PG&E.

When you plug in your toaster or computer or other 3-pronged appliance, the metal case of the computer is electrically connected to the 3rd prong, to make sure it has the same voltage you do, assuming that you are standing on the ground, to the extent that it’s possible to ensure that. If a live wire were to come loose inside the toaster, instead of silently electrifying the outside of your toaster, it will get into a fight with the ground wire over what the voltage should be. There will be a huge difference of opinion! Lots of electricity will flow! So much, in fact, that the circuit breaker will trip or the fuse will blow, and you’ll be safe.

Why not just have 2 prongs and connect the outside of the toaster to the neutral wire, instead of having all these redundant 0 volt wires? Well, first there’s the problem that the video converter tech support guy was talking about — polarized plugs are designed to fit only one way, but if you’re motivated enough, you can plug them in backwards. Now the outside of the toaster is connected to
110 V, not 0 V, and you’re in trouble.

The other problem is that, unlike the ground wire, the neutral wire is being used to carry significant amounts of eletricity. When it does that, the voltage rises above 0 — not all the way to 110, usually, but if you’ve ever noticed the lights dimming when the fridge comes on or when you turn on the vacuum, you’ve seen that the voltages of the hot and neutral wires are drifting noticeably closer together. You do not want the AC voltage of the outside of your fridge to rise every time the compressor kicks in! That’s why it needs separate prongs for ground and neutral. This is exactly the sort of problem that Tim Williams talks about in the first chapter of his book, by the way, though the context is a bit different. The point is that it’s always important to ground things correctly! And these things aren’t always immediately obvious. That’s why, when high voltages start to become involved, we have safety rules about these things.

Anyway, whoever did the wiring in this apartment either didn’t know about those rules or didn’t figure they were very important. All those 3-prong outlets? They don’t have separate ground and neutral wires. The wiring in the walls is the older kind, with just 2 conductors: hot and neutral. Each outlet has its own little wire tail that connects the ground prong directly to the neutral prong. Presumably the tail is there solely to fool the 3-light tester, which turns out to be exactly the tool the eletrical inspector from the city usually uses to make sure everything is hooked up correctly. It’s pretty hard to tell the difference unless you set up an elaborate experiment where you draw lots of power and measure the voltage more accurately, or open up the wall and look inside.

Which is fine, mostly. Usually wires don’t spontaneously come loose inside of electrical appliances, and usually appliances don’t draw enough power to bring the neutral wire far enough away from 0 volts to be really dangerous, and if they did, the fuse would still blow. The difference between 2 prongs and 3 is usually a subtle one, and one that people didn’t worry about at all until a few decades ago.

No, the real problem here was that, on one of the outlets in this back bedroom where I had the robot set up, the hot and neutral wires were reversed.

Normally, the 3-light tester would catch this right away. But if you’re tying neutral and ground together right at the outlet, so they both have the same voltage (110 V AC, in this case), the 3-light tester can’t tell anything is wrong. If you look at the instructions that come for the tester, you’ll find some vague disclaimer about how this this is possible, even though it’s usually pretty unlikely to happen by accident.

The other way you’d normally prevent this situation is by the color-coding of the wires inside the wall. The hot wire is black, the neutral wire is white, and the ground wire is usually just a bare metal wire, because it’s always at 0 volts and is just a wire stuck into the ground and it’s about as safe to touch as anything can be.

Back to the story. When I saw the number 110 on the multimeter, I set it down carefully, backed away slowly, thought about it for a long time, and went to fetch my 1-light tester. This is like the 3-light tester, but it just has two prongs and one light. The prongs are on flexible wires, so you can use them on either a 2-prong outlet (which was why I’d bought mine — to check to see if a 2-prong outlet was hooked up to power) or on different pairs holes in a 3-prong outlet. Unlike the multimeter probes, the prongs on the 1-light tester are designed to be relatively safe to stick into an electrical outlet; once you stick it in far enough to make eletrical contact, there’s little or no metal sticking out where you could touch it accidentally, just like with a regular eletrical plug.

I got an extension cord, and I connected my 1-light tester between the ground prong at the bottom of one outlet (the one toastball was plugged into) and the ground prong of the other one (that the robot was plugged into). And it lit up. Exciting!

After doing a bunch of similar experiments, and a few quite different ones, I was starting to get a pretty good picture of what was going on here, and what it might take to fix it. Probably I could just turn off the power, unscrew that outlet, swap the hot and neutral wires, close it back up, and everything would be fine, more or less. It might even be legal for me to do that, for all I know. But there are lots of rules, and I don’t know them, and by now I’m starting to get tired of this thing where I end up doing stupid dangerous things because somebody wasn’t paying attention to the rules. The rules are there for a reason.

So I called up our landlord, and I tried to explain. Our landlord’s name is Joe, and he’s a really nice guy. He works hard to keep the property in good shape. He didn’t really understand what I was going on about. But he agreed to bring in an electrician.

The electrician shows up and asks to see what I think is wrong. I wordlessly repeat my demo with the extension cord and the 1-light tester. The light comes on. “This,” I maintain, “is the problem. The entire metal case of whatever I plug in here gets electrified. If it weren’t for these fine non-conductive wood floors, I might already be dead.” I may have been exaggerating a little bit for dramatic effect. 110 volts won’t usually kill you directly. I bet it could start a pretty good fire, though.

Joe is looking on. He seems pretty nonplussed. The electrician too. I mean, come on! If this stuff were really as dangerous as I seem to be incoherently claiming, wouldn’t it be more obvious than that?

“Hold on,” says the electrician. And he goes back to his truck and grabs a 3-light tester and comes back upstairs and plugs it in to each outlet, and the 2 yellow lights come on, and the red light stays off. “See?” he says. “It’s fine.”

“Yes, actually, I have one of those too,” I begin.

And then we start to have a discussion. And then he stops me to tell me a story. I don’t remember the details anymore, but the story involves a situation in which this same fellow is working on something, in a dark, wet, cramped, unpleasant crawl space, with some exposed pipes, and he isn’t wearing boots, and the ground is wet, and his feet are wet, and at some point he happens to touch one of the pipes, and he gets a nasty shock.

And as he’s telling me this story, I start to realize that the punch line of the story is not, as one might hope, that he stopped what he’d been doing and figured out why the pipes were all eletrified and fixed the problem and made it safe. No! The moral of the story turns out to be that sometimes these things happen, and 110 volts usually won’t kill you, and that’s just the way the world is, and I should really just learn to suck it up.

“I can’t help you,” he concludes.

“Great!” I say. “You say you can’t help me, and I believe you 100%. Now get out of my house!” I think that, anyway. I must have used different words, at the time. Joe is looking a little bit embarassed. Later I wonder if I should have asked that dude to show me his license to practice electricity. I still wonder about that.

Once they’re gone and I’ve calmed down a bit, I go on the Berkeley Parents Network and troll around for recommendations for competent electricians, with a license from the state, and I come up with a list, and I make a few calls. Competent electricians, it turns out, are in demand. They tend to have busy schedules. But I do manage to get in touch with a couple of guys who call themselves Fighting Amish Electrical Services. I explain to them that I’m pretty sure I know what’s wrong, but I want to hire them to come in and check out the wiring at my house, and see what they think. I explain that I’ll be paying them for their time, and maybe Joe will reimburse me and maybe not — I’ve decided that it’s better to ask forgiveness than permission — but either way I want to do whatever I need to do to make sure the house is safe, dammit.

So they come in, and I congratulate them for having an awesome business name, and I show them my little demo, and I try to explain what I think is going on. And they look pretty skeptical, especially when the 3-light tester shows it’s OK, but I’ve agreed to pay them a considerable amount to check things out for an hour or two and they’ve agreed to do it. So they turn off the power and unscrew the outlets — sure enough, they’re not properly grounded, and that’s not right, but it’s nothing to get excited about, either especially in an old house — and they yank and scrape and test with the meter, and pretty soon they’re like, huh, you’re right. This one is backwards. These wires are the wrong color! Yikes!

Some time later, they ask if I know how to get into the attic. They say there’s probably a junction box up there, with the other ends of the wrong-colored wires, and it would be good to fix those before somebody else comes along and gets into trouble. I tell them that I have no idea how to get up there, but I’ll ask Joe. It later turns out that Joe doesn’t know either. I don’t think he wants to know. I think he’s afraid there might be asbestos or something. He told us, when we moved in, that there might be, somewhere, for all he knew, so we should probably assume there was, and not go stirring things up. Sounds like good advice to me.

So the Fighting Amish guys swap the wires and wrap the in appropriately-colored tape as best they can, in all the spots that they can reach without demolishing the house. Then they ask what I want to do about the 3-prong outlets that aren’t really. They explain that they can’t legally leave them like that. (Not the ones in the bedroom, anyway. They quite rightly refuse to speculate about the outlets elsewhere in the apartment, since they haven’t looked at those.) They can either replace them with 2-prong outlets, maybe, or they can disconnect the ground prong and put a little blue sticker on each one that says “NO EQUIPMENT GROUND.” (In theory, they could also come back and demolish the house and put in actual proper wiring in the walls. But not today.) 2-prong outlets are hard to come by these days, but they have lots of stickers. Sure, I say, put stickers. So they do that. By now they’ve been at it for least 2 hours, and they have other appointments and a non-trivial hourly rate, so I let them go.

Once they’re gone, I plug everything back in and, with some trepidation, I connect the video cable for the third time. Nothing explodes. All 4 cameras now work perfectly.

I gladly pay the electricians’ bill when it arrives. They even forward a copy to Joe, along with a more comprehensible and credible explanation than I was able to provide. He apologizes to me by phone that his electrician couldn’t fix the problem, and he reimburses me. Good for him! I offhandedly mention the idea of tearing the entire house apart to redo all the wiring with 3 conductors instead of 2, and we share a nervous little laugh, because the truth is that neither of us especially wants to turn the place into a war zone. I once witnessed a previous landlord make a much less ambitious improvement to some dodgy grounding. Not fun! I sure got some good use out of my UPS at that place, let me tell you.

Meanwhile, back at our Lake Merritt apartment, I got out my extension cord and my 1-light tester, and I checked every single outlet in the apartment to make sure none of the others were wired backwards. Turns out they’re all good. Though there are two in the living room that stay on even when I pull all of the fuses and turn off all of the breakers. (Why yes, this apartment has both.) The oddness with those outlets didn’t really have any practical impact on our lives until quite a bit later. But I’ll have to tell you that story another time. Right now I’ve got to run over to Fry’s to see if a new power supply will resurrect my dead computer….

The Making Of

If I had any sense at all, I would probably have used some kind of screen recording software to make yesterday’s video.  I doubt it would have taken more than an hour or two.  But where would be the fun in that?

Here’s what I did instead:

  • Had a Saturday morning conversation with Jessica about a fancy Internet valentine one of her friends had just received, and about the Internet in general.

  • Idea!

  • Outlined a script in Google Docs.  Turns out you can copy and paste formatted text from Web pages, including images, directly into your document.  At least, you can do it with the kind of markup that was popular in early 1997.  So my script turned into a mock-up.

  • Captured some raw footage.  On the Mac, you can press Cmd-Shift-4 to turn the cursor into a crosshair that you can drag to save a rectangular area of the screen into a PNG file on the desktop.  Who knew?  Bonus: the crosshair shows the cursor’s coordinates!  This came in handy later, I’m sorry to say.  I grabbed a chunk straight off of primordial Yahoo, seven or so pasted-and-edited groups of links right out of my script/mock-up, a big chunk of Firefox dolled up in a fanciful outfit, and a smaller chunk of Firefox with the Back button held down.  (If you hold it for a bit it shows a menu and stays pressed.  I chopped off the menu underneath.)

  • Still missing a very important piece of raw footage: the mouse cursor!  Cmd-Shift-4 won’t buy me that.  Grovelled through Google Images until I found something pretty close, then basically traced over it using the pencil tool in a drawing program.  The real OS X arrow cursor has fancy antialiased edges and a subtle drop shadow with blended transparency.  I didn’t need or even want subtle for this, so my low-tech version only has 3 kinds of pixels: 100% black interior, 100% white border, and 100% transparent background.

  • The original draft of the shooting script called for a ridiculously zoomed-in look at the web page, with big fat square pixels and no extraneous browser decorations in the frame, then a jump cut over to the Back button and back to the text.  Two problems: First, my 2010 web browser was drawing nice smooth blended antialiased text, which would have drawn undue attention to itself when zoomed in so the pixels were visible.  Probably there’s a way to turn that off.  But, also, the groups of 5 bullet points I’d ended up with were much wider than they were tall — too wide even for a 16:9 widescreen version of the movie.  Rather than go back and wrack my brain for more bullet points or edit down to get the good parts to fit, I settled on the 1:1 scale 320×240 version you see now, where the browser chrome takes up half the frame and the Back button is visible the whole time.  (I’d heard that YouTube scales uploaded videos to 320×240 anyway, so I figured I’d try for that.)

  • Futzed around with ancient Unix command-line image manipulation tools to crop the unevenly-sized margins out of my bulleted lists (pnmcrop), add uniform margins (pnmpad), and paste the new text into the content area of the web browser shot (pnmcomp).  Did the same with a cropped rectangle of the Yahoo page (pnmcut, pnmcomp).  Needed the height to match exactly for that one, which took me a few tries.

    [Side rant: Last time I tried to install pbmplus from source and typed pnmcut --help — which is of course the first thing you do when you want to use pnmcut, because who can remember all those crazy options? — instead of giving me a cheat sheat, it told me to run “man pnmcut.”  Fine, I said, and typed man pnmcut.  But then, instead of revealing how to use the program, it gave a link to the HTML documentation on the web site.  Which, okay, maybe that’s not so terrible.

    Now, the manual page with the link is auto-generated, and the HTML doc is maintained in sync with the program itself, so presumably it would have been possible (though perhaps not convenient) for the pbmplus people to rejigger it so that it auto-generates actual useful man pages from the HTML, instead of autogenerating crappy placeholders.  Maybe it could even have taken the good stuff from the top of the HTML and put it in the –help output.  But, you know, it’s a pain to do stuff like that, and I can totally understand why the pbmplus people might have felt that their time was better spent working on the actual software to do image manipulation and what have you.

    However, this placeholder man page took a novel approach that I haven’t seen elsewhere: after telling me ha ha, just kidding, no useful information here, please go to such-and-such URL instead, it had a new boilerplate paragraph with words to the effect of: Please note that, if your system administrator truly loved you, he or she could have installed this suite of programs in such a way as to make it so that right now, after typing man pnmcut, you would be reading the very documentation you are looking for, instead of this stupid autogenerated boilerplate with just a link in it.

    Now, if you yourself are the poor chump who originally installed this thing on the computer, and you read that message, you’re probably going to be all like, “Okay, I’ll bite — how do I install this thing such that I will get man pages that aren’t completely useless?”  And you go look it up… and, after much research, it turns out that, what the pbmplus people apparently expect you to do (or expected, at the time — no idea if this is still true) is to take your perfectly good “man” command and replace it with a different man command, written by the pbmplus people, that checks to see if you’re trying to view the man page for one of these pbmplus programs and, if so, pulls up the corresponding web page in, like, lynx or something.  Because that’s totally reasonable, right?  It just starts up a completely different program!  It has different keyboard navigation keys and everything!  But, you know, probably no one will notice.  That is, until they finish reading and press q to quit and it’s all like Are you sure you want to quit lynx? and they’re like “Yes!!!  WTF am I doing in lynx in the first place?  What is this, 1997?  This is terrible!  I never asked for this!”

    And oh, man, I can just imagine the conversations that must have ensued between innocent users typing man pnmcut and getting that passive-aggressive message and going straight to the IT manager or whoever and saying, “Dude!  Where’s my man page!  I just want to know how to run pnmcut, and this thing says that you could have done it right but you didn’t because you hate freedom!”  And then this poor system administrator has to sit there and somehow try to explain to this irate user that, no, actually, the cake is a lie.  And it’s sad to think about, because you just know that conversation is not going to end well for anyone.  Because, I mean, best case: the sysadmin is like, fine, no problem, let’s see… replace man, ok, sounds perfectly reasonable, I’ll get right on that… and the user ends up in Lynx, right?  And nothing works like it should!  And the user comes back and says, “Dude, you broke the man command!”  And the sysadmin says, “Only because you insisted I do it!”  And then they both go find the pbmplus people, and the pbmplus people are all like, “Geez guys, what do you expect?  It’s free software!

    And then, even after everyone has jumped through all those hoops, the pnmcut web page that you end up reading still says infuriating things like, oh, so sorry, pnmcut is deprecated now, we decided last Wednesday afternoon that we don’t like it anymore, so now we use pamcut, which is highly superior in every way but works differently.  You’ll love it, we promise.  Don’t worry, pnmcut is still there, but we’re certainly not going to tell you how to use it.  Now run along. So you dutifully click the link for “pamcut” and figure out how to rewrite all your stuff to use it and then finally go back and type pamcut -foo=bla frobozz.ppm and it’s all like pamcut: command not found.  Because you still have the version from Tuesday morning.  Oops!  Sucks to be you, I guess.

    Thankfully I don’t have to think about any of this anymore because somebody (either the pbmplus folks, or some other fork, or Ubuntu, or Debian, or someone else — my money’s on Debian, actually, but I’m too lazy to try to find out) stepped up and installed some sane man pages on Ubuntu, so if you type man pnmcut it very calmly explains how to use pnmcut with no passive aggression whatsoever.  And absolutely no lynx.  I’d managed to forget the whole thing until I clicked through that pnmcut link.  Wouldn’t that have been a shame?

    Here endeth the rant.  Back to making a YouTube!]

  • Turns out I forgot about a few small details!  If you click on a link in a web browser, even in early 1997, a number of things happen:

    1. As soon as you move the cursor over the link, it changes from an arrow into a little pointing hand.
    2. The link itself turns red during the click.
    3. A little dotted rectangle appears around the link as soon as you click it, to show that it has the keyboard focus.
    4. When you come back to it later, the link is purple.

    I decided to forego 1, 3 and 4 — again, not going for realism here! — but 2 is kind of an important cue, especially if you’re not going to bother trying to do foley.  Rather than go back to my script and make versions of the bulleted lists where some of the links were red instead of blue (which would have produced identical results and been much easier to do — Google Docs has a little “font color” doodad right there, for Pete’s sake), I ended up finding the coordinates of the rectangles corresponding to my links (which involved scribbling down numbers from the crosshair cursor onto a bunch of sticky notes and subtracting them with bc -l), cropping them out with pnmcut, swapping the red and blue channels using a 17-line Python program whipped up for the purpose, then splicing the channel-swapped rectangles back in with pnmcomp.  Now I had clicked and unclicked versions of each page.  Used a similar but less complicated trick to comp the depressed Back button into copies of the applicable frames.

  • Another missing ingredient: Title cards!  My script called for a black background with centered white text in some suitably florid font.  Fired up the ol’ drawing program again.  Zapfino was the closest font on the list to what I had in mind, so I ran with that.  I have no idea how to get this GUI to center text, and I’m far too lazy to try to center it by hand, so I ended up doing the pnmcrop/pnmpad trick again.  Except that if you do it that way, it centers the bounding box for the visible white pixels, not the baseline of the text!  I probably wouldn’t have noticed or cared, except that “Love” has no descenders and “Jacques” has this giant swoosh under the Q, which makes the whole line jump upwards by a very noticeable amount when pnmpad tries to center it vertically.  So I had to manually crank it back down by 16 pixels or something with an extra trip through pnmpad.  After that, everything looked sufficiently plausible to my eye, so I moved on.

  • Almost there!  By this time I have a big ol’ shell script that’s assembling all of these frames in order and putting them in numbered files and stitching them together into a video file, so I can view my work in progress.  Originally the video conversion happened in two steps: mencoder mf://*.png -o out.avi to make an AVI file, and then ffmpeg -i out.avi -o out.flv to convert to Flash Video, since YouTube’s going to do that if I don’t and I’d rather find out sooner what it’s going to eventually look like.  But then it turned out that some of the title cards at the end had only 16 shades of gray (you’d think only 2, black and white, but no: antialiased text strikes again!), so pnmtopng was using a 4-bit encoding for those frames when the rest of the frames used 8 bits per pixel (I guess — maybe this is why it so proudly announces how many colors it found?), which in theory you’d think wouldn’t matter at all but in practice mencoder was apparently failing to do the right thing with its crazy MPNG encoding, so the frames at the end were getting squished horizontally and were pretty much illegible.  I’m sort of surprised it didn’t just decide to douse my computer with gasoline and set it on fire at that point, actually.

    Anyway, it turns out that, unlike mencoder, ffmpeg can read PPM files directly, so long as you number them in strict sequence, starting from 1.  You just do ffmpeg -r 6 -i %d.ppm -o out.flv.  No pnmtopng, no mencoder, no stupid issues with the PNG encoding scheme changing mid-stream.  Terrific.  The -r 6 tells it to play 6 frames per second, for that extra lo-fi feel, just like the terrible Saturday morning cartoons of my youth.  (It also means I need to make fewer frames.)

  • You know, I still have to somehow splice a non-rectangular mouse cursor into all of these images and animate it moving around and all that.  Apparently you can use pnmtopng -alpha to extract a transparency mask from the PNG and get pnmcomp to use that when blending in your image, and then all you need is the X and Y coordinates of where you want your mouse cursor to go.  Wrapped that logic up in a shell function, cut and pasted a bunch of lines (one for each frame, basically), and started plugging in numbers for the mouse coordinates.  At first I was measuring things with the Cmd-Shift-4 crosshair cursor and the Post-it notes and the subtraction, but after a while I was like, well, gee, all I really want for this next shot is for the cursor to move up by two bullet points, and they seem to be about 16 or 19 pixels tall I think, so let’s just try this (type type type)… oops, didn’t quite get there, let’s try that again (type type type)… ok, that looks better.  But it’s going too fast!  Need to duplicate some frames, to make it pause a bit between steps, to slow down the pacing.  It’s funnier if you don’t rush, you know?  Probably could have stretched it out to run twice as long and ended up with something twice as funny.  Ah well, maybe next time.

  • Woo, done!  How’d that happen?  Wound up with 370 frames, which, at 6 frames per second, comes to 61 and 2/3rds seconds, so I cut a total of 10 frames from the various title cards to bring it down to 60 seconds exactly.

And there you have it!  Only took two days.  I even had enough time left over to have dinner with my favorite Valentine!

Jacques Visits the Internet

I made a YouTube!

Web Dream

I’m standing next to Keith‘s desk, asking him a question.

“I’m just trying to find a site where the hostname –”

“Actually,” he says, “we call that the domain.” He swivels his chair and glances up at me.

“Right, okay. But all I need is to be able to find a site where the name in the url bar,” I say while trying to point at at Keith’s browser with my chin cuz my hands are busy making air quotes, “matches the name in the SSL certificate.”

Job Search 2009

I recently started a new full-time job writing software.  Here is how that came about:

  • Thu Oct 1: Returned from trip. (I’d previously decided to look for work upon returning.)
  • Thu Oct 8: Sent Peter a blurb to go out with Friday’s SF Bay Area Caltech Monthly Luncheon invite: Grad seeks software engineering job to supplement income.  10+ years experience.  (I mistakenly thought this might motivate me to update my resume before the luncheon.)
  • Thu Oct 15: Attended luncheon.  Discovered that one of my fellow alumni and former co-workers is now a hiring manager with a possible opening.  (Nobody else ever contacted me about my blurb.)
  • Mon Oct 19: Beat resume into submission.  Emailed link to former co-worker and a few other close friends (less than 10, certainly).  Linked to resume from Facebook status.
  • Tue Oct 20: Visited 5 friends in 3 Bay Area cities.  Exchanged more email.
  • Wed Oct 21: Posted final draft of resume, incorporating various suggestions.  Took first call from a prospective employer (specifically, from a recruiter in a Human Resources department).
  • Thu Oct 22: First technical phone screen.  Thus far, I’d heard from 6 potential employers in total.
  • Wed Oct 28: Heard from 10th potential employer.  (I’d elected to pursue 7 of those 10.)
  • Thu Oct 29: First on-site interview.
  • Thu Nov 12: 9th and last on-site interview.  (3 of the 9 were second visits.)
  • Fri Nov 13: Reviewed notes; deliberated; chose.  Notified all 7 contenders.  Drove down to sign employment agreement.
  • Sat/Sun Nov 14/15: Wrote to those who had requested a more detailed explanation.
  • Mon Nov 16: First day at the new job.  (I took Tuesday off so we could celebrate our monthaversary as planned.)

Further factoids

  • I was referred to all 7 companies by current employees who know me.  5 were former co-workers.  2 were hiring managers; the rest were engineers.
  • All interviews were for a full-time position as some variety of software engineer, programmer, MTS, code monkey, or other equivalent term.
  • At least 3 companies asked me to sign a non-disclosure agreement prior to my first on-site interview.
  • I would rather not identify all 7 companies by name or group too much information by company.  Sorry.
  • All 7 companies did some kind of work involving distributed systems running on clusters of Internet-connected computers running Unix-like operating systems.  [So did various leads I didn’t pursue, for that matter.]
  • 2 of the 7 companies had between 6 and 50 employees; 1 had 51 to 500; 2 had 501 to 5,000; 2 had 5,001 to 50000.
  • Locations ranged from San Francisco to Mountain View.  [I live in Oakland.]
  • 5 companies did phone screens, 4 of which were divided into a call from someone in HR and a call from an engineer with technical questions; the 5th was from a hiring manager who did some of both.
  • 6 of the 7 companies invited me for on-site interviews.  I visited 3 of those companies twice, for a total of 9 on-site interviews.  I did complete a technical phone screen with the 7th company, but I don’t know whether they would have invited me on-site or not.
  • Technical phone screen start times ranged from 9 a.m. to 3 p.m. or later; scheduled duration was between 30 and 60 minutes.
  • Scheduled on-site start times ranged from 10 a.m. to 1 p.m.; scheduled duration ranged from 2 to 5 hours.
  • 5 companies scheduled on-site interviews overlapping with the noon hour.  Of those, 3 took me out to lunch; the other 2 had no lunch break.
  • 2 different companies invited me to talk to an engineer based in Europe.  One called me at home; I dialed the other from a conference room during an on-site interview.  Both calls were in the morning, at 9 or 10 a.m. Pacific time.
  • My new job is at Blekko.  No, I’d never heard of them either.  No, I can’t tell you much about it.
  • Yes, I am still working at Shiny Devices.  No, this is not a secret.  Yes, I’d be happy to tell you all about it.

I like to think that I’m at least somewhat aware of how lucky and how privileged I am, and I might editorialize about that or some other aspect of this experience another time.  For now, I thought it might be interesting just to list some of the questions I was asked.  As far as I know, nothing written here is a direct quote from anybody.  Any lack of specificity means that my notes or recollections are vague, or that I’m being lazy, or that I think the specifics are none of your business.

Preliminary questions

Here are some of the non-technical initial questions the HR people asked me.

  • Could you send me your resume?  [This only happened once.]
  • If employed, can you submit verification of your legal right to work in the U.S.?
  • Of the specific job openings listed on our web site, which ones are you primarily interested in applying for?  (You’d think I’d learn my lesson after being asked this once, but no.)
  • Are you looking for a full-time position?  [I’d initially made noises about being open to part-time work.]
  • Have you worked here before?
  • Have you or applied for a position at this company before?
  • I understand that you were referred by X; is that correct?  [A: Yes.]
  • Who else at this company do you know?  [I was asked this at least 2 times, and listed several other people in each case.]
  • Where are you in your interview process?  [I tended to interpret this as: “How many companies have you contacted, how many have contacted you, when did you last hear from each, and what stage have you reached in each case?  And for that matter, which companies are these, and (for the more obscure ones) how big are they and what do they do?”  But nobody ever pressed me for anywhere near that level of detail, so I never ended up providing it.]
  • What is your timeline?  [A favorite!  I tended to interpret this as being equivalent to the previous question or to “When could you hypothetically start work?”]
  • What salary do you require?  How much did your previous employers pay you?  [Only 1 company asked this during the initial HR call, but see below.]

I spent most of my time on this type of call repeating and expanding on the contents of my resume.  For example:

  • What programming languages are you the most comfortable using?  [I was repeatedly told that they were asking me this specifically in order to select an interviewer for the technical phone screen.]
  • I see on your resume that you worked for Pagebites as a System Architect.  Could you talk a bit more about what that actually entailed?  [Mostly for recent and/or relevant experience, but not exclusively.  Sometimes I’d have guesses afterwards about which word in that job description had caught their eye.  Several times people told me they were just curious about some not-so-relevant project, such as Shiny Devices or my Ditch Day stack.]
  • How are things at Shiny Devices?  What was your reason for leaving Pagebites?  [Mostly about current/recent jobs, but a few questions about older jobs too.]
  • How many employees work(ed) for Shiny Devices?  Pagebites?  SafeWeb?  iSpheres?  [Nobody asked this about Microsoft or Symantec.  Strange!]
  • Didn’t our employee X (the one who referred you to us), also work at company Y (listed on resume)?  Were you both there at the same time?  [A: Yes indeed we were.]
  • Have you ever worked with MySQL?  How comfortable are you with C?  C++?  Perl?  Writing kernel drivers?  [Some language/tool/technology that they used, or that was somehow related to something they did, often in a way that wasn’t obvious to me at the time.]
  • Which of the following 20 or so options sounds most interesting to you?  Choose at most 3 or 4.  [One place did this with groups I might be interested in interviewing with; another did it with areas of computer science, again in order to decide who might interview me.]
  • What interests you?  [Open-ended version of the above multiple-choice question.  I can’t think of any company who didn’t ask me something very close to this at some point.]

Technical phone screens

4 out of 4 HR people mentioned my interviewer’s name when scheduling the technical phone screen.  In 3 of the 4 cases, I elected to webstalk my interviewer beforehand.  I was moderately entertained in 2 cases, regretted it in 0 cases, and decided it had been useful in 0 cases.  In the 4th case, I got a first name only (or forgot the last name, perhaps), did no “research,” and ended up getting a call from someone with a different first name.  Go figure.

One engineer did ask me what language I wanted to use for the coding question (2 companies used web-based shared whiteboards to ask coding questions during the phone screen), and several spent a few minutes up front asking me to discuss specific projects I’d listed on my resume.  All four spent the majority of the call asking me technical questions.  At least one launched into a somewhat involved technical question before I’d uttered more than a few words; I had to ask for some details to be repeated so I could write them down.  In 1 case I had no time to ask any questions whatsoever; the other 3 interviewers gave me a few minutes at the very end of the call.  Several people implied that they thought I would be invited for an on-site interview, but 0 promised this, and some didn’t mention the topic at all.

Telephone interviewers from 2 different companies asked superficially similar questions about what happens when you invoke wget or curl on a URL.  I believe both used as their example.  One interviewer seemed to be more interested in network and system level stuff, such as the sequence of calls to routines in the BSD socket API, e.g. gethostbyname(), socket(), connect(), and a few high-level details of DNS, HTTP, and TCP.  The follow-up question in that case was a seemingly open-ended question about making a large download go faster; the specific answer desired apparently involved fetching ranges using multiple HTTP connections.  The other cast a wider net, reflected in the original question’s phrasing: I was to describe what happens upon entering the command into bash running in an xterm on a Linux machine, which (once I’d summarized my previous spiel and realized I wasn’t done) gave me an excuse to talk about USB HID, Unix domain sockets, ptys, fork and exec… we ran out of time before I got started about, which was probably just as well.

A few more discussion questions I was asked at this stage:

  • What do malloc() and free() do?  Describe how you might implement them.  Can you think of any ways to improve on that implementation?  Follow-up: Where does the memory used by malloc() and free() come from?  List a few advantages of virtual memory.  [And so on.  Memorable near-quote: “No, actually, I think a more pressing issue is that your malloc() sucks, if you’ll excuse my French.”]
  • Thread 1 requires simultaneous access to resources A and B; thread 2 needs A and C; thread 3 needs A, B, and C.  How might one mediate access to those resources?  [A: Guard all 3 resources with 1 mutex.]
  • Describe how to find the 10 most frequently-used words in a large corpus of text.  Follow-up: What if your histogram is too large to fit in RAM?  Follow-up: List some of the worst deficiencies of that approach.  [A: Well, it needs to make at least two passes over the data, which is kind of irksome… oh, and it’s not guaranteed to terminate… yeah, that last is probably the worst, actually.]

2 different companies used a shared web-based whiteboard [my term] to ask me a coding question over the phone.  I was warned about this ahead of time in both cases, so as to ensure that I’d be near an Internet-connected computer.  Each web-based whiteboard was developed in house; neither was entirely glitch-free, but both were more than adequate.  The 2 questions were as follows:

  • Write a function that sorts an array of integers in place.  Is your function correct?  Are you sure?  What’s it’s asymptotic efficiency?  Really?  What’s the asymptotic efficiency of an optimal solution?  [And so on.]
  • Write a function that finds the longest non-decreasing subsequence of an array of integers, e.g. 1 10 2 8 3 5 6 4 7 12 11 becomes 1 2 3 5 6 7 12.

[I managed to flummox both interviewers by stumbling upon novel and unexpectedly inefficient initial solutions, thereby cutting off the usual avenues for incremental improvement.]

On-site interviews

I was given a start and end time for all 9 on-site interviews (as well as a location, for the 6 first-time visits at least).  I was never given a more detailed schedule ahead of time, though several of the interviews started with a meeting with a hiring manager or someone in HR who gave me the day’s schedule.  Each company divided the interview into uniform 30, 45, or 60-minute slots.  The only non-uniformities I can recall were either for lunch or (in 2 cases) to wait for an opening in a busy executive’s schedule.  There was one interviewer per slot in all but 2 or 3 cases with 2 or 3 simultaneous interviewers, one of whom invariably did most of the talking.

The on-site questions didn’t really differ much from the telephone questions.  There were fewer of the easy preliminary questions, but still quite a lot of questions about things I’d done in the past or might be interested in doing in the future.  In fact, I’d guess that I spent more time overall talking about past projects than answering technical questions, though the proportion varied wildly between companies.  Many of my on-site interviewers, even if they were engineers, asked only non-technical questions.  Some, especially higher-ups, spent the whole time selling me on the company.  Many, if not most, started by describing the company and their particular product or role.  These descriptions didn’t usually overlap as much as they seemed to fear.  Some of my interviewers were in a different department or group than what I was interviewing for; these tended to pre-emptively justify their presence.  Almost all sacrificed at least a few minutes of their precious time slots for me to ask further questions.  I mostly stuck with variations on “What do you, personally, do here?” and “What’s your typical day like?”  In a couple of cases I asked somebody to sketch a partial org chart.  Once I asked somebody to show me some code.  [They did.]

On-site non-technical questions

  • Where do you see yourself in 5 or 10 years?  Follow-up: Do you aspire to become a manager?  [Fairly common.]
  • What would you do with your life, given unlimited resources (such as money and employees)?  [A memorable one-time variation.]
  • Describe a past situation where you responded to a customer request / encountered an external obstacle / disagreed with someone about how something should be done / were introduced to a large pre-existing codebase.  [And maybe a few more I’m forgetting.  Once each, I think.]
  • What are your hobbies outside of work?  [A: “Well… uh… um… I’ve been learning how to do a pull-up!”  I was only asked this once during the entire process.  Some might consider this a dodgy question, legally speaking.  More on that below.]

On-site technical discussions

  • Have you used or do you know anything about mysql/Agile/perl/hadoop/xen/Linux drivers/OUTER JOIN/Erlang?  [I’m not making any of those up.  Yes, I was asked about some of those during phone screens as well.]
  • Which databases have you worked with in the past?  When would you use a sub-select vs an inner join?  How does a WHERE clause differ from the HAVING clause of a GROUP BY?  [This interviewer mentioned having been tasked with asking me database-related questions.]
  • Define: object, inheritance.  List some types of inheritance.  [A: private/protected/public/default — not my first guess.]
  • How would you choose between Java and C++?  [I managed to turn this one into an excuse to discuss the design of the Shiny Devices firmware configuration and upgrade utility for the 999th time.]
  • How might one design a distributed, transactional database?  [2 times!  Took up most of the interview slot in each case.  Lots of whiteboard diagrams.]
  • Name some of Python’s built-in types.  Which of those types are immutable?  [Python became my stock answer to “What language shall we use for the following question?”]
  • Tell me about the database schema at Pagebites.  How many tables were there?
  • What’s the most technically challenging thing you’ve done?  [I only recall being asked this 1 time.]
  • What technologies do you like, e.g. Rails?  What’s your favorite web framework for Python?  [Interviewer liked Rails.]
  • What’s a TLB?  [3 or 4 times at least!]
  • What kind of software load might run especially slowly under a hypervisor?  [Once.]
  • When and how do you hand off code to QA?  What test frameworks have you used?  [Interviewer worked in QA department.]
  • What’s the return type of a constructor in C++?  [Asked during a coding question.]
  • How might you go about finding the 1 million most common bigrams [pairs of consecutive words] in a 100 million document corpus?  [I mentioned that another company had asked me something similar during a phone screen, and wished aloud that I’d researched it afterward.]
  • I know this isn’t especially relevant, but I’m curious: What processor did you use for your embedded Linux prototype?  Was that an ARM7TDMI?  Any on-chip Flash?  How much RAM?  How’d you get it to boot?  [A rapid-fire barrage of questions which I cannot possibly hope to replicate here.]

On-site coding questions

Some of these were prefaced by a question about what programming language I’d prefer to use.  Some were not.

  • Interviewers at 2 different companies asked similar questions about analyzing a hypothetical web server access log.  Both interviewers invited me to write code on a whiteboard in any language I liked.  Each line in the log file contains a fixed number of space-separated fields, always in the same order, e.g. URL, browser name, client IP address, seconds required to serve the request.  In one case, the log was compressed with gzip (which made it easier to guess at the preferred form of my answer, I felt), and I was to count unique client IP addresses.  [A: zcat access.log.gz | cut -d’ ‘ -f3 | sort -u | wc -l]  In the other case, I was to generate a deduplicated list of IP addresses that made at least one request requiring at least 2 seconds to serve.

  • Describe how you would test whether a number is a power of 2 in C?  Follow-up: What does the following C function do?

    int something(int N)
     int i = 0;
       N &= N-1;
     return i;

    Follow-up: How might you modify the above function to compute parity?

  • This next question was framed in terms of a book recommendation algorithm, but was phrased in terms of a bipartite graph.  [I was amused to hear those words come out of my interviewer’s mouth before we’d even gotten to the question.]  You’ve got a list of people and a list of books, and you know which people have read which books.  Given a person P, write a function in the language of your choice that returns the list of all books B such that P has not read B and B has been read by at least one person Q such that Q has read every book P has read.  [My interviewer asked the question in a less confusing but equally exact way, and may even have sketched out a diagram.]  Start by making and stating some reasonable assumptions about how the information is stored and how your function can retrieve it.  You may assume the existence of lower-level data retrieval functions.

  • Write a function in the language of your choice that checks to see whether a string is a syntactically valid IP address.

  • Write a C++ prototype for a function that accepts an HTML document as input (as a NUL-terminated const char *) and returns a deduplicated set of link URLs as output.  [A: void findLinks(const char* htmlIn, std::set<std::string>& linksOut);]

  • List some inputs for a test plan for a function that checks email addresses for syntactic validity.  Follow-up: Write the function, in Perl.

  • Verbally sketch out the design for a class representing an email message, in Perl.  Follow-up: Write code for the constructor.

  • Here is a printout of a Unix man page for strcmp() [and strncmp()].  Here is a ballpoint pen.  Write code for strcmp() in the bottom margin.  Follow-up: Does your implementation work correctly for UTF-8 strings? [A: Yes.]

  • I forget the setup for the next question, but I think it involved some sort of unnecessary apology/excuse for asking me to reinvent the wheel.  My interviewer wrote the following on a piece of paper (with a large blank space between the curly braces, and not much space between the class declaration and the following statements):

    class String
    String s1("abc");
    String s2;
    s2 = s1;
    cout << s2;

    Describe what class members String would need in order for the four lines at the bottom to work.  [Phrasing was a bit misleading, perhaps intentionally, as operator << needs to be declared outside the class.]  Follow-up: Write a prototype for the assignment operator.  Follow-up: Describe some potential pitfalls in the implementation of the assignment operator.  [A: self-assignment]

  • You have a SQL table called “employees” with three rows and three columns:

    id  name   salary
    1   Alice       9
    2   Bob        14
    3   Carol      12

    Write a SQL UPDATE statement that changes Alice’s salary to 15.  [I used the id column in the WHERE clause rather than the name, eliciting no comment before we moved on to another question.]

  • Write Python code to flatten a nested list, e.g. [1, 5, [8, 12, [17]], 3, [27]] becomes [1, 5, 8, 12, 17, 3, 27].  Follow-up: Does your code work for all inputs?  [I thought mine would, but it turns out Python limits the call stack to 1000 levels or so, preventing my recursive solution from flattening deeply nested lists.]  Follow-up: Write one that does.

  • For this next question, my interviewer first wrote and described a big nasty Perl data structure similar to the following:

    my %people = (
      123 /*UID*/ => ["Alice", ["project 1", "project 2", "project 3"], 1234567890 /* birthdate as Unix timestamp */],
      456 => ["Bob", [], 1234565432],
      /* ... */

    Write Perl code to print name, number of projects, and UID as three space-separated fields, one hash entry per line, sorted by number of projects (most to fewest) and then by name (in case of a tie).  Follow-up: convert those dates of birth to human-readable form.  [We didn’t actually get to that.]

  • Write a non-trivial function in the language of your choice to convert a string into a floating point number [or flag an error]. Follow-up: How might you go about testing the function you just wrote?

  • Define: hash table.  Describe how you might implement one, assuming string keys and values.  Now draw a diagram of the data structure you just described.  Next, assume the following C++ declarations:

    int hash(char* c) { /* */ }
    class hash_table {
        void insert(char* key, char* value);
        char* find(char* key);

    Fill in the private data members.  Now write the constructor.  Now write the insert method. [A: See photo below.]


One company contacted me after an on-site interview to schedule a time to email me a coding question.  I was given 2.5 hours to complete the task, or as much of it as I could.  The task involved implementing a data structure (in C, using only the standard library, or in Java, using the standard library minus java.util).  The detailed instructions included a list of operations, arranged in order of priority, that the data structure should support.  [I didn’t get to them all, or even close.  My submission was written in C and contained 4471 bytes in total.  It was quite a stressful experience, in part due to my own foolish overscheduling, and I had mixed feelings about it.  I later heard that it had passed “with flying colors,” which was nice to hear.]

Taboo questions

In about 2 instances, I elected not to answer a specific question about some aspect of something I did for a previous employer that I felt was proprietary.  In those cases, I said that I’d rather not go into so much detail about X, offered an extremely vague answer, and tried to suggest a related area where I felt that I could go into more detail.  So that’s been pretty rare for me.  It may help that many of the projects I’ve worked on no longer exist, although I’ve been known to refuse to comment even in that case.

In California (and in the USA generally), certain interview questions are considered illegal.  I got a few questions that brushed on these topics.  A recruiter asked about my work authorization; I stumbled a bit and admitted that I was a U.S. citizen.  Over lunch, somebody asked me whether I’d grown up in California; I mentioned Ottawa.  As I understand it, employers are forbidden from using a few specific types of information to make discriminatory hiring decisions, even though I’m still technically allowed to provide such information.  Even so, I wish I’d answered those questions in the spirit in which they were asked (“Yes, I am indeed prepared provide evidence of my U.S. work authorization if and when you employ me” and “Gosh, I don’t know if that’s really such a good topic for us to chat about in the context of a job interview here in these United States”), rather than accidentally putting a prospective employer in a situation that might cause unnecessary worry about legal issues.

Just as in every other job search I’ve done, all 7 companies asked at some point about my compensation requirements, my salary history, or both.  In fact, at least 3 of the companies I interviewed with had a formal job application form (paper or electronic) including both questions, to be completed before receiving an offer, with some redundant but intimidating boilerplate at the bottom about how failure to answer any question on the form is grounds for rejection and/or future dismissal (usually right next to the “at-will” bit about how either party will be able to terminate the employment agreement without notice or justification).  Totally normal stuff.  However, as it happens, I can’t recall ever having disclosed my salary history or specific requirements to any potential employer, during this or any other job search.  As far as I am aware, this has never been a significant problem for anyone, though folks do often seem to push back quite hard, for perfectly understandable reasons.  While some companies played it cool this time around, I feel like at least 3 of the 7 acted sort of surprised about it, which was too bad.

Some of the same companies (and more besides) seemed to react somewhat negatively when I accepted Blekko’s offer without soliciting a formal offer from everyone else (or, rather, from the negatively-reacting company in particular), which also seems like kind of a shame.  I suppose I could have gone to the other extreme and tried to spark some sort of bidding war.  Imagine the flak I could have drawn by blogging about that!

Monthaversary Dream

Went to bed early last night, and woke up this morning from a dream:

Jessica had written an excited blog post about a place she’d been, posted mere moments ago.  I followed the link and found myself in standing front of an exhibit in an enormous, sun-dappled, high-ceilinged room, all exposed wooden beams and skylights and dark wood paneling.  The exhibit was on a small wooden table, about the size of a standard-issue grade-school desk, and not much taller.  There were papers on the table–printed, not hand-written.  The pages were either laminated or enclosed in protective transparent plastic pouches, like the ones they sell at the giant drugstore.  There were quite a few loose pages, but most were bound together into something like a book, open to what looked at first like prose, but read more like poetry.  The author was well-known; it might have been Imogen Heap or Tori Amos, though I don’t remember exactly.

One of the paragraphs on the right-hand page was almost entirely obscured by a printed and laminated note, from Jessica, affixed Post-It style by a big loop of Scotch tape on the back so that it popped out and quivered a bit when tapped.  It was rectangular, about the same shape as the short wide paragraph it was stuck to, printed in the same black-on-white serif typeface.  There was a little 2D barcode in the upper-left corner, which I assumed was some sort of backlink or anchor, and there were also some kanji mixed in, Firefly style: “I am so 峠榊 excited!  I came back to this page because I liked it so much and there was a new 畑辻 paragraph!”  The note went on to explain that the paragraph in question was some sort of behind-the-scenes digression on the making of, which was extra-cool.

I glanced up and saw that someone was looking at me, a young woman.  I could tell from her shy smile that she’d just arrived through Jessica’s link.  In fact, I surmised, this whole place was likely to be slashdotted by a flash mob any minute now.  I nodded and strode away so that she could have her turn before the deluge.

The place turned out to be more of a children’s library than a museum.  There were kids everywhere, a few teachers, and row upon row of low wooden bookshelves.  I hooked a right down an empty row, crouched below shelf level, closed my eyes and snapped my fingers to discreetly teleport back to where I’d come from.  I wondered if any of the kids would notice or wonder about my disappearance.  Ah well.  They’d figure it out soon enough….


Sometimes I get headaches.  Doesn’t everyone?  I always get weirded out by the checkbox on those medical history forms they always want you to get there early to fill out.  Shouldn’t they be asking about frequency of headaches?  Maybe some people have never experienced a headache.  “What, you mean like when I grab a towel from the cupboard over the sink and forget to close the door and bang my head on it and it hurts?”  Yeah.  Like that.  I guess.

I think frequency is correlated with stress.  Stress leads to procrastination, procrastination leads to blogging, blogging leads to… where was I going with this?  Oh, right!  An email newsletter on treating headaches with acupuncture, which I will now brazenly quote without context or permission:

Two separate systematic reviews by Cochrane Researchers show that acupuncture is an effective treatment for prevention of headaches and migraines.

In each study, the researchers tried to establish whether acupuncture could reduce the occurrence of headaches. One study focused on mild to moderate but frequent “tension-type” headaches, whilst the other focused on more severe but less frequent headaches usually termed migraines. Together the two studies included 33 trials, involving a total of 6,736 patients.

Overall, following a course of at least eight weeks, patients treated with acupuncture suffered fewer headaches compared to those who were given only pain killers. In the migraine study, acupuncture was superior to proven prophylactic drug treatments.

In one specific study involving 270 patients, acupuncture cut tension headache rates almost in half. Researchers divided the patients who reported similarly severe tension headaches into three groups for the study. Over the project’s eight-week period, one group received traditional acupuncture, one received only minimal acupuncture (needles inserted at non-acupuncture points, and at only shallow levels), and the third group received no treatment.

Those receiving the traditional acupuncture reported headache rates of nearly half that of those who received no treatments, suffering 7 fewer days of headaches. The minimal acupuncture group suffered 6.6 fewer days, and the non-acupuncture group suffered 1.5 fewer days. When they received acupuncture after the main study period, the “no treatment” group also reported significantly fewer headache days.

The improvements continued for months after the treatments were concluded, rising slightly as time went on.

British Medical Journal, July 2005
Acupuncture for tension-type headache. Cochrane Database of Systematic Reviews, Issue 1. Art. No.: CD007587 DOI: 10.1002/14651858.CD007587
Acupuncture for migraine prophylaxis. Cochrane Database of Systematic Reviews, 2009, Issue 1. Art.No.: CD001218 DOI: 10.1002/14651858.CD001218.pub2

If you get regular headaches, please call for a consultation today.

[Boldface and lack of hyperlinks in the original.]

Now, I’m supposed to be doing something productive right now, so my priority is to act on this immediately.  Sure enough, here’s the British Medical Journal article about the “one specific study” from which those factoids were extracted (published in July 2005, as promised, though BMJ turns out to be weekly — who knew?), and here are the two Cochrane meta-analyses on tension headaches and migraines, respectively.  And when I say “here’s the article” and “here’s the meta-analysis,” what I really mean is “here’s the abstract.”  You don’t expect me to actually read the papers, do you?  What, just because I’m avoiding work, I should try to evaluate their methodology?  No thanks!

[Fun fact: Although the not-for-profit Cochrane Collaboration’s papers are only available to paid subscribers — simply complete the free registration process and accept the terms of use to see prices! — it is perfectly legal for a member of the general public to BART over to Doe Library with a Flash drive and take full advantage of the University of California’s extensive institutional subscriptions.  Provided, of course, that you don’t let just anyone get their hands on the perfectly legal contents of your Flash drive!  That would almost certainly be a crime.]

Lucky for me, I don’t even need to click through to the free PDF of the BMJ article on tension headaches to get this juicy sound bite:

Results The number of days with headache decreased by 7.2 (SD 6.5) days in the acupuncture group compared with 6.6 (SD 6.0) days in the minimal acupuncture group and 1.5 (SD 3.7) days in the waiting list group (difference: acupuncture v minimal acupuncture, 0.6 days, 95% confidence interval -1.5 to 2.6 days, P=0.58; acupuncture v waiting list, 5.7 days, 3.9 to 7.5 days, P<0.001).

In other words, according to this particular cherry-picked study, acupuncture is almost certainly quite a bit better than nothing.  (The meta-analyses agree.)  But it doesn’t seem to work significantly better than anything else, including placebo (or “sham” treatment, as one of the meta-analyses aptly puts it — this is where they pretend to give you acupuncture, taking care to do it in all the wrong places, pressing hard enough that you notice the needles but not hard enough to go all the way through your skin).

[This is just one study, mind you.  The tension headache meta-analysis found “[s]mall but statistically significant benefits of acupuncture over sham,” contradicting my conclusion.  The migraine meta-analysis mostly agrees with me but then goes on to wonder whether the pretend acupuncture was too realistic.  What if it turns out that real acupuncture and pretend acupuncture are both better than some other, more placebo-ish placebo?  An excellent question for advocates of both real and fake acupuncture to consider!]

In fact, for some of these headaches (certain migraines, say), there might well be no known treatment that works better than placebo!  And, hey, acupuncture has fewer side effects than Tylenol or caffeine or other similarly-questionable cures.

[Again, this is what’s known as an “uninformed opinion,” meaning “I don’t really know what I’m talking about.”  But I really do believe what I’m saying!  It feels right to me, you know?  Sure, I could be all wrong, and I accept that, but I do have an opinion.]

Now, imagine you’re a researcher, and you’ve somehow arrived at similar conclusions — privately, at least — which, unlike me, you’re in a position to defend with well-reasoned arguments and hard evidence, if need be.  Acupuncture: better than nothing, just as as drugs (with fewer side effects!), but no better than placebo.  For the sake of argument, if nothing else.

It’s sort of a tricky situation, isn’t it?  I mean, what would you recommend that headache sufferers do?

Well… acupuncture, right?  It’s the best available treatment!  Duh!

But it’s kind of weird to just come out and say, “Yeah, turns out acupuncture totally works,” or even to let the press to quote you saying that (and you know that’s going to be the sound bite, even if you spend the rest of the day hemming and hawing), because that statement kind of misses the point, doesn’t it?  In fact, it looks a lot like there is nothing special about acupuncture at all!  So coming out and endorsing it specifically feels sort of like endorsing snake oil (or JC Tonic, or bottled water) on the strength of the fact that many of the people who buy into it are seeing real, quantifiable benefits.  Which is presumably the case, right?  But, I mean, hey, I don’t know for sure.  I haven’t heard of any studies.  Which I guess means I can’t actually recommend bottled water as a headache remedy, even if it might be equally effective and, you know, cheaper than fake acupuncture.

Or can I?  For, like, the benefit of humanity?  What if I did a study, and the results turned out to support my agenda, and I somehow convinced all the bottled-water companies to plaster them all over all water bottles everywhere?  Even with all that, would anyone even listen to my message?

Now, acupuncturists are not scammers — they’re not malicious, and they don’t seem to be doing any unintentional harm either (another point of general agreement).  Well, I mean, sure, there must be malicious and/or harmful acupuncture out there somewhere, genuine or otherwise.  You can say the same of almost anything, right?  So does it really make sense to accord much weight to that fear?

It’s important to be objective about this.  We’re scientists.  We can’t let ourselves be influenced by our “emotions” and our “gut feelings” and our “values,” just because we “know them in our hearts to be true!”  Can you imagine trying to get such a flimsy justification published in a reputable journal?  You’d be the laughingstock of your peer review group!  All that touchy-feely stuff is seriously unseemly and uncool and unmanly — er — unquantifiable, is what I mean!  Can’t be measured, you know.  Unless you’re a “social scientist,” of course… but those are “soft” sciences, and we’d much rather think of ourselves as “hard” scientists, wouldn’t you say?  Gotta stay reductionist, here.  Reality is complicated.  It isn’t our place to try to consider all possible variables at once.  That way lies madness and despair.  Focus!  Be systematic.  Divide and conquer.  It’s our only hope for progress and salvation!

Turns out that, as a scientist, you’re not supposed to make a recommendation.  You’re supposed to state the facts in mind-numbing detail, toss it over the wall, and let whoever happens to find it (if anyone) draw his or her own conclusions about the larger implications (if any).  There is no “tricky situation” here.  These are not the droids you’re looking for.  This is science.

Now, the same goes for journalism, right?  Be objective, state the facts, give both sides of the argument, and let people form their own opinions instead of stating yours as if they were, you know, true or something.  You want the truth?  The truth is that your readers don’t want your opinion.  They don’t care!  Who do you think you are, some kind of domain expert rocket scientist?  You don’t know what you’re talking about.  Even if you did, there’s no need to spoon-feed them predigested conclusions!  Don’t be so cynical.  People are smart.  Trust your audience!  Just give them all the facts.  They’ll figure it out somehow.

(And no, for the last time, you may not print the DOI number of the original study in your article!  This is a newspaper!  We have standards to uphold!)

So, um, anyway.

Do you suppose it’s realistic of me to expect significant improvement in my headaches if I were to spring for eight weeks of acupuncture, or whatever it was?  Best available treatment, right?  Or weasel words to that effect?

Does it matter if I’m initially skeptical about the efficacy of the treatment?  Did any of those studies control for that?

Don’t even make me ride BART to find out.  I have important work I need to be doing right now, I’m stressing out about it, and for some reason I keep falling further and further behind schedule….

A Brief History of toastball

The very first toastball was a Gateway desktop that sat on my dorm room desk, directly under my bunk.  It had a 200 MHz Pentium inside, and 64 megabytes of this so-called “SDRAM.”  Or was it 16?  Lots, anyway.  It ran Slackware Linux, mostly, because of how it was 1995.

The second, purchased some time after graduation, was one of those cute little Shuttle XPCs, an SN41G2 if I remember correctly.  Served me well for many years.  It ran toast linux, about which more in a moment.

The current incarnation lives in a somewhat bulkier SilverStone case.  I was shocked at what a few hundred dollars will buy at Fry’s nowadays.  It runs Ubuntu.  Works okay, except when it occasionally decides for no obvious reason that the CPU has overheated (lies, I tell you!) and turns itself off in an orderly but inconvenient manner.

I’ve used the same keyboard with all three toastballs.  It’s an IBM Model M that came with the PS/2 model 80 I used in high school.  That computer didn’t really have a proper name of its own, but it did host “Not Just Another BBS,” so I suppose that’s something.

Why “toastball”?

People often used to ask me why I would call my computer “toastball,” to which I would truthfully reply that “” was already taken.  (It was a CNAME for ““.  Still is, as I write this.)

By now most people I know have learned to stop asking me that.

So what was this “toast linux”?

I used to enjoy compiling software from source, where by “enjoy” I mean “hate with a fiery passion.”  But the other option, using only pre-compiled software, was clearly worse!  You always had to wait forever for the Powers That Be in charge of your Distribution of Choice to deign to package up the latest greatest whatever-it-was and pronounce it fit for use, so that you could for instance convince your shiny new video card to, like, display images on the screen.

In the meantime, to get working video, you basically had to build the latest X server from source, then rebuild all the programs that used X in order to get them to behave correctly in the New World Order, and so on until you’d manually rebuilt everything.  Everything!  (I exaggerate only slightly.)

So why not just drop the pretense and start from scratch?

Of course back in those days we didn’t have Linux From Scratch (or running water, or working C++, or Gentoo).  If you wanted to compile something you’d download it and unpack it somewhere and tell it to “./configure && make && su make install” (sorry, no sudo back then either!), and then you’d go get something to eat and come back to enter the root password, and then what would happen is that it would strew a bunch of files willy-nilly all over your filesystem, leaving orphaned bits of previous versions bleeding on the floor.  There was no “make uninstall”, and even when there was, it only made things worse.

The above incantation worked only for well-behaved packages, such as most medium-sized GNU software.  To build something big and important, like gcc or glibc or XFree86, you’d invariably have to read the instructions and type something ever-so-slightly different in order to make it go.  (“make World“, anyone?)  I always assumed these packages were so big and important that their maintainers knew they could get away with making the rest of us jump through as many hoops as they liked, because, really, what choice did we have?  What, you’re seriously going to try to use some other compiler (or C library, or X Window System) and expect anything at all to work even slightly?  Good luck with that!

Of course the small obscure packages all had more and bigger hoops, because their maintainers didn’t know better or didn’t care or both.  If you just crossed your fingers and typed “make”, it wasn’t unusual for the Makefile to scold you. (Flaming hoops — I like it!)  Typically you were expected to hand-edit the Makefile, but only after carefully reading it and understanding it.  If you were really lucky, there’d be some likely-looking stuff not too far down that you could uncomment without having to think very much at all.  If you weren’t so lucky, you’d have to spend half an hour rooting through the source tree trying to find the Makefile (often hidden among a flock of platform-specific decoys) or figure out what crazy tool you were expected to download and install in order to generate one, or give up and write one by hand.  Which, admittedly, was (and to some extent remains) a totally reasonable state of affairs.  I mean, it’s free software, right?

So I started to write this thing called toast.  It was a program whose purpose was to install software on toastball.  Hence the name.  Ha, another FAQ answered!

The idea was that you would feed toast a source tarball (as root, so it didn’t have to stop and beg for a password), and it would unpack it, examine it, figure out how to jump through all the hoops, compile it (as a non-root user, of course!), and install it in a clean and sane and self-contained way (inspired by GNU stow and such — not that I’d heard of stow back then, though apparently it’s been around a while) so that you could undo it all a year later if you needed to.

Most of the “real” root filesystem would just be a big old symlink farm, pointing into all these private microcosms where each software package lived.  Which was great, because you could tell exactly what version of which package a given file originally came from just by looking at it.  And if you needed to comb through the symlinks to perform some tedious task, like listing installed packages or undoing a change, toast did that too.  Later on it even learned to scour the net for the latest version of a package, given its name.

The original incarnation of toast (version 0.x, as I retroactively called it) worked by finding, examining, and strategically altering the Makefile.  Plus all the symlink stuff, but that was small potatoes.

I always considered it “cheating” to have toast decide how to treat a package based on its name, so it always made its decisions based solely on the contents.  Instead of writing a set of rules for compiling each package, toast used a single set of rules, some of which worked on all sorts of software, and some of which ended up only being useful for a single major release of a single obscure package, until the maintainer would spontaneously wise up and get with the conformist program.

At some depressingly late juncture, I caught wind of the DESTDIR trick, which was (and is) very conformist indeed, coming straight out of GNU’s official coding standard.  You’ve read that, right?  Yeah, me either.  But, listen: Those GNU folks are smart.  Rather: They’re smart and they’ve been around long enough to learn how to write software that people (and other software!) can figure out how to compile and install properly.  Decades hence, brilliant people will still be making bad assumptions that GNU discarded decades ago.  (Me: “Don’t you think?”  Victor: “I think.  Do you think?”  We were roommates at the time.)

The upshot of DESTDIR is that, if a software package knows about it, you can tell it to install itself into a private, self-contained directory tree, and you can do so in a nice standardized conformist way.  You don’t even have to look at the Makefile.  Of course most packages didn’t support DESTDIR, or did it wrong, because nothing works unless you test it, and most people had never even heard of DESTDIR and wouldn’t have cared if they had.  I could hardly blame them then, and I don’t now.  So the Makefile-patching continued.

Although toast still occasionally falls back on its ancient habits, the biggest improvement in version 1.x was the LD_PRELOAD trick, which turns out to be the software equivalent of an orbital mind control laser.  toast 1.x would (and still does) use LD_PRELOAD to fool an installer into believing that it was scattering files hither and yon throughout the filesystem, when in fact its flailings were being intercepted and redirected into its own private sandbox.  Sometimes a particularly intrepid package will try to read back something it just installed, so toast also intercepts reads in order to maintain some pretense of a self-consistent universe.

The illusion is imperfect — it won’t fool a human for long — not because perfection would be impossible, but because it turns out to be the wrong thing to do in practice.  For instance, toast does blatantly unrealistic things like spontaneously creating parent directories whenever an installer assumes some other package must surely have created it already.  If it’s everyone’s job, it ends up being nobody’s job, which makes it toast’s job as far as I’m concerned.

Even back in the days of toast 0.x (never released), the original toastball eventually got to the point where there wasn’t really all that much left of the original Slackware.  (Hence the stray comma after “mostly.”)  Instead there were lots of symlinks pointing to /toast/gcc/2.95/bin/gcc or whatever.  (For those keeping score, toast’s directory structure changed a bit in 1.x too.)

Occasionally I’d meet new people, and we’d get to talking about computers (as people always do), and they would ask what OS I used, and I’d say Linux, and they’d invariably say, oh, really, what distribution? And for a while I’d say that I didn’t use any of them, really, because I used this toast thing, and I’d explain, or not, but anyway the point was that I didn’t actually distribute this purported operating system to anyone else, so it wouldn’t make sense to call it a “distribution.”

So that was “toast linux,” huh?

No!  That’s just the backstory!  We’re still talking about the first toastball here.  The second toastball was the one that ran toast linux.  Pay attention!


No, no, I’m the one who should apologize.  Sometimes I get excited, and I… overitalicize.

Where was I?

Uh… the second toastball…?


Of course I would have popped a toast linux install CD into the second toastball, if only such a thing had existed.  I did have toast, though.  So what I did was I installed some Linux distro or other… gosh, I think it might actually have been SuSE, you know?  But it hardly matters, because it was only there as a sort of incubator out of which to boostrap a system built entirely on toast.  Which wasn’t quite as easy as you’d think, because… well, for one thing, it turns out that different major versions of glibc were incompatible.  Still are, I imagine.

It’s like… think of your filesytem as a planet, and the C library (glibc, in this case) as the atmosphere, and all the life forms that evolve on that planet are all the programs that are compiled against that version of glibc.  So here you have this alien planet, with the new version of glibc, and if you try to run a program on the wrong filesystem it can’t breathe the air, see?

I mean, sure, in theory you could have a planet with two different glibcs, only in practice it’s a bit of a pain because they both want to be /lib/ but only one of them gets to.  (Which seems silly! If the .2 is a version number, why don’t they change it when needed?  Does anyone know?  Why can’t we all just get along?!)  So somebody gets stuck being the sad nonconformist.  But it can’t be SuSE, because that’s already been chiseled onto stone tablets, and I don’t want it to be my toasty new system either because then I’d have to live on a sad nonconformist planet until the end of time, see?

Not true, obviously, but what it amounts to is that at some point you’ve got to play any of a number of different games with static linking, cross-compiling and/or chroot.  Which are like, uh, gosh, I don’t know, robots and space suits and trans-dimensional portals, I suppose.  So it got a bit science fictional in there for a while, you know?

Still, when all the moon dust had cleared, I had this beautiful blue-green planet made of pure toast, unpolluted by any traces of SuSE.

Sure, go ahead and laugh at my sad, hypocritical, heterodox ways.  It’s not about the ideology, I swear.

So that was toast linux?

For all intents and purposes, yes — it was a complete, self-contained operating system, built entirely using toast.  But in another sense, no, not at all!  Because it still wasn’t a distribution, you see.  It wasn’t even automated, let alone released.

Luckily for me, Linux From Scratch had become a thing by then, so I was able to crib off of their notes on some of the trickier parts.  Because, hey, you’ve got all these interlocking pieces written by all these different people, none of whom feel in any way responsible for anything beyond their little fiefdom.  Because, well, they really aren’t, you see.  So you’ve got to work hard to assemble the puzzle, you know? Everybody gets to assume that you’re doing their piece last, and that all the other pieces are good to go and working perfectly, because that’s by far the most common case.

Think about it: Approximately how many Linux distributions are there?  Of any consequence, I mean.  Zero, right?  So basically, if you want to roll your own, you get to violate everybody’s perfectly justified assumptions.  Fun, I tell you.

The other thing about distributing an operating system is that you can’t just give someone the source code, because you can’t run source code, and without an operating system you’ve got no way to build it.  They need a binary.  I mean, you could ask them to get a binary somewhere else to bootstrap from, like Linux From Scratch does, or like I did with SuSE, but at that point you may as well hand them a free SuSE CD, which is, as I was saying, a binary.  If you actually want them to use your thing, I mean, as opposed to making them jump through hoops because you’ve got better things to do than pre-jump a bunch of hoops, flaming or otherwise.

I wanted the smallest binary possible — just enough to download everything it needs, preferably from the original publisher rather than from me.  In particular, I wanted a binary that would fit on a 1.44 meg floppy disk.  Install CDs were the norm by then, true, but floppy drives hadn’t completely dissapeared yet, and it sure sounded like a neat idea.  But you can’t fit enough binaries on a floppy to build everything else from source.  I mean, in theory, sure, but in practice, for GNU/Linux, you needed (and almost certainly still need) 10 or 20 MB minimum.

So my arbitrary mission was to find a third-party site from which my floppy-sized binary could download that 10 or 20 MB of extra binaries it needed to build Rome.  The answer was pretty obvious, actually, since I already wanted to use busybox and uClibc to make the floppy: buildroot.  Which was only just starting to be a thing, but it did exist, and it was called that, and there was a place on where you could download a tiny but complete Linux-based build environment in binary form.

So I told the floppy to grab buildroot first thing when it woke up, right after it finished doing all the other first things, like finding the network and resizing the Windows partition.  Once it had buildroot it would grab the gcc and binutils sources and build a uClibc-to-glibc cross-compiler, and use that to build a native glibc compiler, and so on.

The whole thing was just one long shell script, actually, written in simple enough dialect that busybox could understand it.  It had a GUI and everything — well, a text-based GUI, with a master progress bar and a list of packages showing what’s been downloaded and built and what’s still to come, and you could switch over to look at the downloader downloading or the build thread spitting out the usual reams of impressive-looking text.  And, yes, at some point very early on it would grab and build some minimal subset of Perl 5 so it could run toast, which would build everything else.

Automatically locating and downloading the most recent version of every package on its list was a doomed approach.  Every time anything new came out, the odds of everything breaking were high.  So I spoon-fed it a list of URLs of specific versions of specific packages, usually not the latest and greatest, but something known to work with everything else.  Plus, you know, some relative download and build time estimates, for the progress bar.

I actually got it to the point where it worked reasonably well, running in a chroot jail or VMware, with a caching proxy keeping it from repeatedly downloading massive amounts of source code.

I actually installed it on a real computer once, too.  Not on toastball, as it turns out, but on melba, a cute little Flybook laptop, then shiny and new.  The resulting OS served me well for years.

So, uh… would that have been the real toast linux, then?

Yes!  It totally was!  Or, I mean, it definitely would have been if I’d, you know, released it.  But I never got it to the point where it seemed likely enough to work on any given day to run the risk of someone else trying it.

OMG!  Why not?!  What happened?!?!!!

Well, for one thing, the half-life of the wad of URLs was too short.

So you just… gave up?


Well, no.  I mean, that wasn’t the real reason for abandoning the project.  Nothing like that.

Updating URLs was work, yes, and switching strategies also sounded like work, and I was busy doing other things.

I’d released toast, which now seems to have reached maturity, if only in the sense that I don’t have to do much to keep myself or others from complaining about it.  Even now that I’ve switched to Ubuntu at work (the current toastball being a work computer) and a MacBook Air called toastbook running OS X at home, I still use toast regularly on both machines, and on shared machines like ofb, just to quickly grab the odd piece of software.

Whereas toast linux was in essence an OS installer, needed only when setting up a new computer, which hardly ever happens.  Assuming I even wanted toast linux the operating system, which it turns out I don’t.  At least, not if it means worrying about keeping everything current without destroying the delicate balance of nature.

Honestly, I don’t think there’s any real need for toast linux, you know?  I mean, now that 2010 is almost upon us, toastball can display video with barely half an hour of futzing per biannual upgrade.  (I always remember too late to be careful when buying video hardware.)  That thermal alarm thing?  Hardly ever happens, and so far never when I’m away.

The Mac works perfectly, of course.

So what if Snow Leopard doesn’t ship with mtr?  Who cares if Jaunty and MacPorts have two different versions of Cython, neither one current?  That’s what toast is for!

Beeping: Fixed!

It’s been hot lately, so last weekend, on Blake’s advice, I stopped by the giant Longs and picked up a Lasko Cyclone Model 3540 20″ Power Air Circulator.

Fan box

I’ve never owned a fan that came with its own remote control before.  It has a built-in timer, too.  It certainly seems to move a lot of air, even on the lowest setting.

It’s quite wide, which makes it a bit awkward to find room for it in cramped quarters.  I was going to put it on top of a bookcase, but it turns out not to want to tilt down.  I ended up stacking some books to make a little shelf for it in front of the window.  It’s working well so far.

Fan on makeshift shelf

The most serious problem with this fan is the beeping noise it makes whenever you press any of the buttons, either on the fan itself or on the remote control.  It even makes an extra-long beeping noise when the timer shuts the fan off after you’ve fallen asleep.

This is such a notable deficiency that Blake and I had actually discussed it at some length prior to my trip to Longs.  I told him I thought it should be possible to correct the problem by cutting a trace on the circuit board, using a hobby knife or a razor blade.

In fact, it turns out to be easier than that.  All you really need is a screwdriver.

Step 1: Unplug the fan.

Always a good starting point.  Not that the plastic blades look all that dangerous to me, but there are some exposed high-voltage wires on the circuit board.

Step 2: Remove the front cover.

Step 2

There are two long narrow Phillips-head screws on the back of the fan, a few inches above each hinge.  Your screwdriver will need to be narrow enough to reach them.  Mine wasn’t, but then I found another one that was.

Once the screws are off, the front cover comes out of its clips much more easily than before.

Step 3: Remove the circuit board cover.

Step 3

If you look inside, underneath the pod with all the buttons and lights, you’ll see four screws.  Remove the two larger screws, one from each side, and pull off the small plastic cover to reveal a mysterious green circuit board.

Step 4: Unscrew the circuit board.

Step 4

Step 4

There are four screws holding the circuit board in place, one near each corner.  (The lower right screw is a bit farther away from the corner than the other three.)  Unscrew those, and carefully remove the board from its housing, taking note of how the pieces fit together.  You can also unclip the top plastic piece with the buttons at this stage, if you like.  In fact, it might be necessary to do so.  I can’t remember anymore.

Step 5: Disable the beeper.

PCB top

The beeper is the round black thing with the little hole in the middle, over on the right side, labeled “BUZZER.”  It looks like it would be a pain to remove it using average household tools.  You could mute it somewhat by covering the hole with electrical tape, I imagine.

Chip U1, the small black rectangle just right of center, is an EM78P153S microcontroller.  I had hoped it might be reprogrammable, but it isn’t.

If you follow the traces on the green side of the board, you’ll soon discover that capacitor C4, the skinnier of two cylinders immediately above and to the left of the microcontroller, serves no purpose other than connecting the microcontroller to the beeper, through the magic of AC coupling.

I was able to cut one of capacitor C4’s little wire legs using a wire cutter.

Step 5

A good pair of scissors might have worked almost as well.  In fact, if you were to bend the capacitor back and forth a few times with your finger, the two legs would break right off with no need for any fancy “tools.”

Keep in mind that once you’ve disconnected this capacitor, it will be quite difficult to reconnect it properly, should you ever decide that you rather enjoyed having the beep after all.

Make sure that any remaining metal bits don’t touch each other.  Otherwise you might sometimes get beeping and other times not.  It is difficult to imagine anything worse!

Step 6: Put everything back together.

Reverse steps 4, 3, and 2 to reassemble the fan.  The screws from each step are different, so this would be a good time to retroactively keep track of which is which.  The two long narrow screws belong to step 2, the two wide screws are from step 3, and the four remaining medium screws came from step 4.  Phew!

If all went well, you should now find yourself with a fully assembled fan that has working buttons, lights, and remote control, but absolutely no beeping!

Step 7: Blog about it.

This step is optional.  In fact, I highly recommend skipping it.

Steps 1 through 6 took me 10 or 15 minutes at most.  Discovering those first two screws, taking a few pictures, and looking up the data sheet for the microcontroller took another half-hour or so.  Sitting down and writing about it has easily taken four hours.  Crazy!

But, hey, I at least I have a blog now.  In which I can, you know, blog about stuff.  Bound to come in handy, right?  Like when I someday decide to rejigger those hinges so the fan can point downwards.  Won’t that be something?