Sometimes I get headaches.  Doesn’t everyone?  I always get weirded out by the checkbox on those medical history forms they always want you to get there early to fill out.  Shouldn’t they be asking about frequency of headaches?  Maybe some people have never experienced a headache.  “What, you mean like when I grab a towel from the cupboard over the sink and forget to close the door and bang my head on it and it hurts?”  Yeah.  Like that.  I guess.

I think frequency is correlated with stress.  Stress leads to procrastination, procrastination leads to blogging, blogging leads to… where was I going with this?  Oh, right!  An email newsletter on treating headaches with acupuncture, which I will now brazenly quote without context or permission:

Two separate systematic reviews by Cochrane Researchers show that acupuncture is an effective treatment for prevention of headaches and migraines.

In each study, the researchers tried to establish whether acupuncture could reduce the occurrence of headaches. One study focused on mild to moderate but frequent “tension-type” headaches, whilst the other focused on more severe but less frequent headaches usually termed migraines. Together the two studies included 33 trials, involving a total of 6,736 patients.

Overall, following a course of at least eight weeks, patients treated with acupuncture suffered fewer headaches compared to those who were given only pain killers. In the migraine study, acupuncture was superior to proven prophylactic drug treatments.

In one specific study involving 270 patients, acupuncture cut tension headache rates almost in half. Researchers divided the patients who reported similarly severe tension headaches into three groups for the study. Over the project’s eight-week period, one group received traditional acupuncture, one received only minimal acupuncture (needles inserted at non-acupuncture points, and at only shallow levels), and the third group received no treatment.

Those receiving the traditional acupuncture reported headache rates of nearly half that of those who received no treatments, suffering 7 fewer days of headaches. The minimal acupuncture group suffered 6.6 fewer days, and the non-acupuncture group suffered 1.5 fewer days. When they received acupuncture after the main study period, the “no treatment” group also reported significantly fewer headache days.

The improvements continued for months after the treatments were concluded, rising slightly as time went on.

British Medical Journal, July 2005
Acupuncture for tension-type headache. Cochrane Database of Systematic Reviews, Issue 1. Art. No.: CD007587 DOI: 10.1002/14651858.CD007587
Acupuncture for migraine prophylaxis. Cochrane Database of Systematic Reviews, 2009, Issue 1. Art.No.: CD001218 DOI: 10.1002/14651858.CD001218.pub2

If you get regular headaches, please call for a consultation today.

[Boldface and lack of hyperlinks in the original.]

Now, I’m supposed to be doing something productive right now, so my priority is to act on this immediately.  Sure enough, here’s the British Medical Journal article about the “one specific study” from which those factoids were extracted (published in July 2005, as promised, though BMJ turns out to be weekly — who knew?), and here are the two Cochrane meta-analyses on tension headaches and migraines, respectively.  And when I say “here’s the article” and “here’s the meta-analysis,” what I really mean is “here’s the abstract.”  You don’t expect me to actually read the papers, do you?  What, just because I’m avoiding work, I should try to evaluate their methodology?  No thanks!

[Fun fact: Although the not-for-profit Cochrane Collaboration’s papers are only available to paid subscribers — simply complete the free registration process and accept the terms of use to see prices! — it is perfectly legal for a member of the general public to BART over to Doe Library with a Flash drive and take full advantage of the University of California’s extensive institutional subscriptions.  Provided, of course, that you don’t let just anyone get their hands on the perfectly legal contents of your Flash drive!  That would almost certainly be a crime.]

Lucky for me, I don’t even need to click through to the free PDF of the BMJ article on tension headaches to get this juicy sound bite:

Results The number of days with headache decreased by 7.2 (SD 6.5) days in the acupuncture group compared with 6.6 (SD 6.0) days in the minimal acupuncture group and 1.5 (SD 3.7) days in the waiting list group (difference: acupuncture v minimal acupuncture, 0.6 days, 95% confidence interval -1.5 to 2.6 days, P=0.58; acupuncture v waiting list, 5.7 days, 3.9 to 7.5 days, P<0.001).

In other words, according to this particular cherry-picked study, acupuncture is almost certainly quite a bit better than nothing.  (The meta-analyses agree.)  But it doesn’t seem to work significantly better than anything else, including placebo (or “sham” treatment, as one of the meta-analyses aptly puts it — this is where they pretend to give you acupuncture, taking care to do it in all the wrong places, pressing hard enough that you notice the needles but not hard enough to go all the way through your skin).

[This is just one study, mind you.  The tension headache meta-analysis found “[s]mall but statistically significant benefits of acupuncture over sham,” contradicting my conclusion.  The migraine meta-analysis mostly agrees with me but then goes on to wonder whether the pretend acupuncture was too realistic.  What if it turns out that real acupuncture and pretend acupuncture are both better than some other, more placebo-ish placebo?  An excellent question for advocates of both real and fake acupuncture to consider!]

In fact, for some of these headaches (certain migraines, say), there might well be no known treatment that works better than placebo!  And, hey, acupuncture has fewer side effects than Tylenol or caffeine or other similarly-questionable cures.

[Again, this is what’s known as an “uninformed opinion,” meaning “I don’t really know what I’m talking about.”  But I really do believe what I’m saying!  It feels right to me, you know?  Sure, I could be all wrong, and I accept that, but I do have an opinion.]

Now, imagine you’re a researcher, and you’ve somehow arrived at similar conclusions — privately, at least — which, unlike me, you’re in a position to defend with well-reasoned arguments and hard evidence, if need be.  Acupuncture: better than nothing, just as as drugs (with fewer side effects!), but no better than placebo.  For the sake of argument, if nothing else.

It’s sort of a tricky situation, isn’t it?  I mean, what would you recommend that headache sufferers do?

Well… acupuncture, right?  It’s the best available treatment!  Duh!

But it’s kind of weird to just come out and say, “Yeah, turns out acupuncture totally works,” or even to let the press to quote you saying that (and you know that’s going to be the sound bite, even if you spend the rest of the day hemming and hawing), because that statement kind of misses the point, doesn’t it?  In fact, it looks a lot like there is nothing special about acupuncture at all!  So coming out and endorsing it specifically feels sort of like endorsing snake oil (or JC Tonic, or bottled water) on the strength of the fact that many of the people who buy into it are seeing real, quantifiable benefits.  Which is presumably the case, right?  But, I mean, hey, I don’t know for sure.  I haven’t heard of any studies.  Which I guess means I can’t actually recommend bottled water as a headache remedy, even if it might be equally effective and, you know, cheaper than fake acupuncture.

Or can I?  For, like, the benefit of humanity?  What if I did a study, and the results turned out to support my agenda, and I somehow convinced all the bottled-water companies to plaster them all over all water bottles everywhere?  Even with all that, would anyone even listen to my message?

Now, acupuncturists are not scammers — they’re not malicious, and they don’t seem to be doing any unintentional harm either (another point of general agreement).  Well, I mean, sure, there must be malicious and/or harmful acupuncture out there somewhere, genuine or otherwise.  You can say the same of almost anything, right?  So does it really make sense to accord much weight to that fear?

It’s important to be objective about this.  We’re scientists.  We can’t let ourselves be influenced by our “emotions” and our “gut feelings” and our “values,” just because we “know them in our hearts to be true!”  Can you imagine trying to get such a flimsy justification published in a reputable journal?  You’d be the laughingstock of your peer review group!  All that touchy-feely stuff is seriously unseemly and uncool and unmanly — er — unquantifiable, is what I mean!  Can’t be measured, you know.  Unless you’re a “social scientist,” of course… but those are “soft” sciences, and we’d much rather think of ourselves as “hard” scientists, wouldn’t you say?  Gotta stay reductionist, here.  Reality is complicated.  It isn’t our place to try to consider all possible variables at once.  That way lies madness and despair.  Focus!  Be systematic.  Divide and conquer.  It’s our only hope for progress and salvation!

Turns out that, as a scientist, you’re not supposed to make a recommendation.  You’re supposed to state the facts in mind-numbing detail, toss it over the wall, and let whoever happens to find it (if anyone) draw his or her own conclusions about the larger implications (if any).  There is no “tricky situation” here.  These are not the droids you’re looking for.  This is science.

Now, the same goes for journalism, right?  Be objective, state the facts, give both sides of the argument, and let people form their own opinions instead of stating yours as if they were, you know, true or something.  You want the truth?  The truth is that your readers don’t want your opinion.  They don’t care!  Who do you think you are, some kind of domain expert rocket scientist?  You don’t know what you’re talking about.  Even if you did, there’s no need to spoon-feed them predigested conclusions!  Don’t be so cynical.  People are smart.  Trust your audience!  Just give them all the facts.  They’ll figure it out somehow.

(And no, for the last time, you may not print the DOI number of the original study in your article!  This is a newspaper!  We have standards to uphold!)

So, um, anyway.

Do you suppose it’s realistic of me to expect significant improvement in my headaches if I were to spring for eight weeks of acupuncture, or whatever it was?  Best available treatment, right?  Or weasel words to that effect?

Does it matter if I’m initially skeptical about the efficacy of the treatment?  Did any of those studies control for that?

Don’t even make me ride BART to find out.  I have important work I need to be doing right now, I’m stressing out about it, and for some reason I keep falling further and further behind schedule….

A Brief History of toastball

The very first toastball was a Gateway desktop that sat on my dorm room desk, directly under my bunk.  It had a 200 MHz Pentium inside, and 64 megabytes of this so-called “SDRAM.”  Or was it 16?  Lots, anyway.  It ran Slackware Linux, mostly, because of how it was 1995.

The second, purchased some time after graduation, was one of those cute little Shuttle XPCs, an SN41G2 if I remember correctly.  Served me well for many years.  It ran toast linux, about which more in a moment.

The current incarnation lives in a somewhat bulkier SilverStone case.  I was shocked at what a few hundred dollars will buy at Fry’s nowadays.  It runs Ubuntu.  Works okay, except when it occasionally decides for no obvious reason that the CPU has overheated (lies, I tell you!) and turns itself off in an orderly but inconvenient manner.

I’ve used the same keyboard with all three toastballs.  It’s an IBM Model M that came with the PS/2 model 80 I used in high school.  That computer didn’t really have a proper name of its own, but it did host “Not Just Another BBS,” so I suppose that’s something.

Why “toastball”?

People often used to ask me why I would call my computer “toastball,” to which I would truthfully reply that “toast.caltech.edu” was already taken.  (It was a CNAME for “toast.srl.caltech.edu“.  Still is, as I write this.)

By now most people I know have learned to stop asking me that.

So what was this “toast linux”?

I used to enjoy compiling software from source, where by “enjoy” I mean “hate with a fiery passion.”  But the other option, using only pre-compiled software, was clearly worse!  You always had to wait forever for the Powers That Be in charge of your Distribution of Choice to deign to package up the latest greatest whatever-it-was and pronounce it fit for use, so that you could for instance convince your shiny new video card to, like, display images on the screen.

In the meantime, to get working video, you basically had to build the latest X server from source, then rebuild all the programs that used X in order to get them to behave correctly in the New World Order, and so on until you’d manually rebuilt everything.  Everything!  (I exaggerate only slightly.)

So why not just drop the pretense and start from scratch?

Of course back in those days we didn’t have Linux From Scratch (or running water, or working C++, or Gentoo).  If you wanted to compile something you’d download it and unpack it somewhere and tell it to “./configure && make && su make install” (sorry, no sudo back then either!), and then you’d go get something to eat and come back to enter the root password, and then what would happen is that it would strew a bunch of files willy-nilly all over your filesystem, leaving orphaned bits of previous versions bleeding on the floor.  There was no “make uninstall”, and even when there was, it only made things worse.

The above incantation worked only for well-behaved packages, such as most medium-sized GNU software.  To build something big and important, like gcc or glibc or XFree86, you’d invariably have to read the instructions and type something ever-so-slightly different in order to make it go.  (“make World“, anyone?)  I always assumed these packages were so big and important that their maintainers knew they could get away with making the rest of us jump through as many hoops as they liked, because, really, what choice did we have?  What, you’re seriously going to try to use some other compiler (or C library, or X Window System) and expect anything at all to work even slightly?  Good luck with that!

Of course the small obscure packages all had more and bigger hoops, because their maintainers didn’t know better or didn’t care or both.  If you just crossed your fingers and typed “make”, it wasn’t unusual for the Makefile to scold you. (Flaming hoops — I like it!)  Typically you were expected to hand-edit the Makefile, but only after carefully reading it and understanding it.  If you were really lucky, there’d be some likely-looking stuff not too far down that you could uncomment without having to think very much at all.  If you weren’t so lucky, you’d have to spend half an hour rooting through the source tree trying to find the Makefile (often hidden among a flock of platform-specific decoys) or figure out what crazy tool you were expected to download and install in order to generate one, or give up and write one by hand.  Which, admittedly, was (and to some extent remains) a totally reasonable state of affairs.  I mean, it’s free software, right?

So I started to write this thing called toast.  It was a program whose purpose was to install software on toastball.  Hence the name.  Ha, another FAQ answered!

The idea was that you would feed toast a source tarball (as root, so it didn’t have to stop and beg for a password), and it would unpack it, examine it, figure out how to jump through all the hoops, compile it (as a non-root user, of course!), and install it in a clean and sane and self-contained way (inspired by GNU stow and such — not that I’d heard of stow back then, though apparently it’s been around a while) so that you could undo it all a year later if you needed to.

Most of the “real” root filesystem would just be a big old symlink farm, pointing into all these private microcosms where each software package lived.  Which was great, because you could tell exactly what version of which package a given file originally came from just by looking at it.  And if you needed to comb through the symlinks to perform some tedious task, like listing installed packages or undoing a change, toast did that too.  Later on it even learned to scour the net for the latest version of a package, given its name.

The original incarnation of toast (version 0.x, as I retroactively called it) worked by finding, examining, and strategically altering the Makefile.  Plus all the symlink stuff, but that was small potatoes.

I always considered it “cheating” to have toast decide how to treat a package based on its name, so it always made its decisions based solely on the contents.  Instead of writing a set of rules for compiling each package, toast used a single set of rules, some of which worked on all sorts of software, and some of which ended up only being useful for a single major release of a single obscure package, until the maintainer would spontaneously wise up and get with the conformist program.

At some depressingly late juncture, I caught wind of the DESTDIR trick, which was (and is) very conformist indeed, coming straight out of GNU’s official coding standard.  You’ve read that, right?  Yeah, me either.  But, listen: Those GNU folks are smart.  Rather: They’re smart and they’ve been around long enough to learn how to write software that people (and other software!) can figure out how to compile and install properly.  Decades hence, brilliant people will still be making bad assumptions that GNU discarded decades ago.  (Me: “Don’t you think?”  Victor: “I think.  Do you think?”  We were roommates at the time.)

The upshot of DESTDIR is that, if a software package knows about it, you can tell it to install itself into a private, self-contained directory tree, and you can do so in a nice standardized conformist way.  You don’t even have to look at the Makefile.  Of course most packages didn’t support DESTDIR, or did it wrong, because nothing works unless you test it, and most people had never even heard of DESTDIR and wouldn’t have cared if they had.  I could hardly blame them then, and I don’t now.  So the Makefile-patching continued.

Although toast still occasionally falls back on its ancient habits, the biggest improvement in version 1.x was the LD_PRELOAD trick, which turns out to be the software equivalent of an orbital mind control laser.  toast 1.x would (and still does) use LD_PRELOAD to fool an installer into believing that it was scattering files hither and yon throughout the filesystem, when in fact its flailings were being intercepted and redirected into its own private sandbox.  Sometimes a particularly intrepid package will try to read back something it just installed, so toast also intercepts reads in order to maintain some pretense of a self-consistent universe.

The illusion is imperfect — it won’t fool a human for long — not because perfection would be impossible, but because it turns out to be the wrong thing to do in practice.  For instance, toast does blatantly unrealistic things like spontaneously creating parent directories whenever an installer assumes some other package must surely have created it already.  If it’s everyone’s job, it ends up being nobody’s job, which makes it toast’s job as far as I’m concerned.

Even back in the days of toast 0.x (never released), the original toastball eventually got to the point where there wasn’t really all that much left of the original Slackware.  (Hence the stray comma after “mostly.”)  Instead there were lots of symlinks pointing to /toast/gcc/2.95/bin/gcc or whatever.  (For those keeping score, toast’s directory structure changed a bit in 1.x too.)

Occasionally I’d meet new people, and we’d get to talking about computers (as people always do), and they would ask what OS I used, and I’d say Linux, and they’d invariably say, oh, really, what distribution? And for a while I’d say that I didn’t use any of them, really, because I used this toast thing, and I’d explain, or not, but anyway the point was that I didn’t actually distribute this purported operating system to anyone else, so it wouldn’t make sense to call it a “distribution.”

So that was “toast linux,” huh?

No!  That’s just the backstory!  We’re still talking about the first toastball here.  The second toastball was the one that ran toast linux.  Pay attention!


No, no, I’m the one who should apologize.  Sometimes I get excited, and I… overitalicize.

Where was I?

Uh… the second toastball…?


Of course I would have popped a toast linux install CD into the second toastball, if only such a thing had existed.  I did have toast, though.  So what I did was I installed some Linux distro or other… gosh, I think it might actually have been SuSE, you know?  But it hardly matters, because it was only there as a sort of incubator out of which to boostrap a system built entirely on toast.  Which wasn’t quite as easy as you’d think, because… well, for one thing, it turns out that different major versions of glibc were incompatible.  Still are, I imagine.

It’s like… think of your filesytem as a planet, and the C library (glibc, in this case) as the atmosphere, and all the life forms that evolve on that planet are all the programs that are compiled against that version of glibc.  So here you have this alien planet, with the new version of glibc, and if you try to run a program on the wrong filesystem it can’t breathe the air, see?

I mean, sure, in theory you could have a planet with two different glibcs, only in practice it’s a bit of a pain because they both want to be /lib/ld-linux.so.2 but only one of them gets to.  (Which seems silly! If the .2 is a version number, why don’t they change it when needed?  Does anyone know?  Why can’t we all just get along?!)  So somebody gets stuck being the sad nonconformist.  But it can’t be SuSE, because that’s already been chiseled onto stone tablets, and I don’t want it to be my toasty new system either because then I’d have to live on a sad nonconformist planet until the end of time, see?

Not true, obviously, but what it amounts to is that at some point you’ve got to play any of a number of different games with static linking, cross-compiling and/or chroot.  Which are like, uh, gosh, I don’t know, robots and space suits and trans-dimensional portals, I suppose.  So it got a bit science fictional in there for a while, you know?

Still, when all the moon dust had cleared, I had this beautiful blue-green planet made of pure toast, unpolluted by any traces of SuSE.

Sure, go ahead and laugh at my sad, hypocritical, heterodox ways.  It’s not about the ideology, I swear.

So that was toast linux?

For all intents and purposes, yes — it was a complete, self-contained operating system, built entirely using toast.  But in another sense, no, not at all!  Because it still wasn’t a distribution, you see.  It wasn’t even automated, let alone released.

Luckily for me, Linux From Scratch had become a thing by then, so I was able to crib off of their notes on some of the trickier parts.  Because, hey, you’ve got all these interlocking pieces written by all these different people, none of whom feel in any way responsible for anything beyond their little fiefdom.  Because, well, they really aren’t, you see.  So you’ve got to work hard to assemble the puzzle, you know? Everybody gets to assume that you’re doing their piece last, and that all the other pieces are good to go and working perfectly, because that’s by far the most common case.

Think about it: Approximately how many Linux distributions are there?  Of any consequence, I mean.  Zero, right?  So basically, if you want to roll your own, you get to violate everybody’s perfectly justified assumptions.  Fun, I tell you.

The other thing about distributing an operating system is that you can’t just give someone the source code, because you can’t run source code, and without an operating system you’ve got no way to build it.  They need a binary.  I mean, you could ask them to get a binary somewhere else to bootstrap from, like Linux From Scratch does, or like I did with SuSE, but at that point you may as well hand them a free SuSE CD, which is, as I was saying, a binary.  If you actually want them to use your thing, I mean, as opposed to making them jump through hoops because you’ve got better things to do than pre-jump a bunch of hoops, flaming or otherwise.

I wanted the smallest binary possible — just enough to download everything it needs, preferably from the original publisher rather than from me.  In particular, I wanted a binary that would fit on a 1.44 meg floppy disk.  Install CDs were the norm by then, true, but floppy drives hadn’t completely dissapeared yet, and it sure sounded like a neat idea.  But you can’t fit enough binaries on a floppy to build everything else from source.  I mean, in theory, sure, but in practice, for GNU/Linux, you needed (and almost certainly still need) 10 or 20 MB minimum.

So my arbitrary mission was to find a third-party site from which my floppy-sized binary could download that 10 or 20 MB of extra binaries it needed to build Rome.  The answer was pretty obvious, actually, since I already wanted to use busybox and uClibc to make the floppy: buildroot.  Which was only just starting to be a thing, but it did exist, and it was called that, and there was a place on uclibc.org where you could download a tiny but complete Linux-based build environment in binary form.

So I told the floppy to grab buildroot first thing when it woke up, right after it finished doing all the other first things, like finding the network and resizing the Windows partition.  Once it had buildroot it would grab the gcc and binutils sources and build a uClibc-to-glibc cross-compiler, and use that to build a native glibc compiler, and so on.

The whole thing was just one long shell script, actually, written in simple enough dialect that busybox could understand it.  It had a GUI and everything — well, a text-based GUI, with a master progress bar and a list of packages showing what’s been downloaded and built and what’s still to come, and you could switch over to look at the downloader downloading or the build thread spitting out the usual reams of impressive-looking text.  And, yes, at some point very early on it would grab and build some minimal subset of Perl 5 so it could run toast, which would build everything else.

Automatically locating and downloading the most recent version of every package on its list was a doomed approach.  Every time anything new came out, the odds of everything breaking were high.  So I spoon-fed it a list of URLs of specific versions of specific packages, usually not the latest and greatest, but something known to work with everything else.  Plus, you know, some relative download and build time estimates, for the progress bar.

I actually got it to the point where it worked reasonably well, running in a chroot jail or VMware, with a caching proxy keeping it from repeatedly downloading massive amounts of source code.

I actually installed it on a real computer once, too.  Not on toastball, as it turns out, but on melba, a cute little Flybook laptop, then shiny and new.  The resulting OS served me well for years.

So, uh… would that have been the real toast linux, then?

Yes!  It totally was!  Or, I mean, it definitely would have been if I’d, you know, released it.  But I never got it to the point where it seemed likely enough to work on any given day to run the risk of someone else trying it.

OMG!  Why not?!  What happened?!?!!!

Well, for one thing, the half-life of the wad of URLs was too short.

So you just… gave up?


Well, no.  I mean, that wasn’t the real reason for abandoning the project.  Nothing like that.

Updating URLs was work, yes, and switching strategies also sounded like work, and I was busy doing other things.

I’d released toast, which now seems to have reached maturity, if only in the sense that I don’t have to do much to keep myself or others from complaining about it.  Even now that I’ve switched to Ubuntu at work (the current toastball being a work computer) and a MacBook Air called toastbook running OS X at home, I still use toast regularly on both machines, and on shared machines like ofb, just to quickly grab the odd piece of software.

Whereas toast linux was in essence an OS installer, needed only when setting up a new computer, which hardly ever happens.  Assuming I even wanted toast linux the operating system, which it turns out I don’t.  At least, not if it means worrying about keeping everything current without destroying the delicate balance of nature.

Honestly, I don’t think there’s any real need for toast linux, you know?  I mean, now that 2010 is almost upon us, toastball can display video with barely half an hour of futzing per biannual upgrade.  (I always remember too late to be careful when buying video hardware.)  That thermal alarm thing?  Hardly ever happens, and so far never when I’m away.

The Mac works perfectly, of course.

So what if Snow Leopard doesn’t ship with mtr?  Who cares if Jaunty and MacPorts have two different versions of Cython, neither one current?  That’s what toast is for!