A Brief History of toastball

The very first toastball was a Gateway desktop that sat on my dorm room desk, directly under my bunk.  It had a 200 MHz Pentium inside, and 64 megabytes of this so-called “SDRAM.”  Or was it 16?  Lots, anyway.  It ran Slackware Linux, mostly, because of how it was 1995.

The second, purchased some time after graduation, was one of those cute little Shuttle XPCs, an SN41G2 if I remember correctly.  Served me well for many years.  It ran toast linux, about which more in a moment.

The current incarnation lives in a somewhat bulkier SilverStone case.  I was shocked at what a few hundred dollars will buy at Fry’s nowadays.  It runs Ubuntu.  Works okay, except when it occasionally decides for no obvious reason that the CPU has overheated (lies, I tell you!) and turns itself off in an orderly but inconvenient manner.

I’ve used the same keyboard with all three toastballs.  It’s an IBM Model M that came with the PS/2 model 80 I used in high school.  That computer didn’t really have a proper name of its own, but it did host “Not Just Another BBS,” so I suppose that’s something.

Why “toastball”?

People often used to ask me why I would call my computer “toastball,” to which I would truthfully reply that “toast.caltech.edu” was already taken.  (It was a CNAME for “toast.srl.caltech.edu“.  Still is, as I write this.)

By now most people I know have learned to stop asking me that.

So what was this “toast linux”?

I used to enjoy compiling software from source, where by “enjoy” I mean “hate with a fiery passion.”  But the other option, using only pre-compiled software, was clearly worse!  You always had to wait forever for the Powers That Be in charge of your Distribution of Choice to deign to package up the latest greatest whatever-it-was and pronounce it fit for use, so that you could for instance convince your shiny new video card to, like, display images on the screen.

In the meantime, to get working video, you basically had to build the latest X server from source, then rebuild all the programs that used X in order to get them to behave correctly in the New World Order, and so on until you’d manually rebuilt everything.  Everything!  (I exaggerate only slightly.)

So why not just drop the pretense and start from scratch?

Of course back in those days we didn’t have Linux From Scratch (or running water, or working C++, or Gentoo).  If you wanted to compile something you’d download it and unpack it somewhere and tell it to “./configure && make && su make install” (sorry, no sudo back then either!), and then you’d go get something to eat and come back to enter the root password, and then what would happen is that it would strew a bunch of files willy-nilly all over your filesystem, leaving orphaned bits of previous versions bleeding on the floor.  There was no “make uninstall”, and even when there was, it only made things worse.

The above incantation worked only for well-behaved packages, such as most medium-sized GNU software.  To build something big and important, like gcc or glibc or XFree86, you’d invariably have to read the instructions and type something ever-so-slightly different in order to make it go.  (“make World“, anyone?)  I always assumed these packages were so big and important that their maintainers knew they could get away with making the rest of us jump through as many hoops as they liked, because, really, what choice did we have?  What, you’re seriously going to try to use some other compiler (or C library, or X Window System) and expect anything at all to work even slightly?  Good luck with that!

Of course the small obscure packages all had more and bigger hoops, because their maintainers didn’t know better or didn’t care or both.  If you just crossed your fingers and typed “make”, it wasn’t unusual for the Makefile to scold you. (Flaming hoops — I like it!)  Typically you were expected to hand-edit the Makefile, but only after carefully reading it and understanding it.  If you were really lucky, there’d be some likely-looking stuff not too far down that you could uncomment without having to think very much at all.  If you weren’t so lucky, you’d have to spend half an hour rooting through the source tree trying to find the Makefile (often hidden among a flock of platform-specific decoys) or figure out what crazy tool you were expected to download and install in order to generate one, or give up and write one by hand.  Which, admittedly, was (and to some extent remains) a totally reasonable state of affairs.  I mean, it’s free software, right?

So I started to write this thing called toast.  It was a program whose purpose was to install software on toastball.  Hence the name.  Ha, another FAQ answered!

The idea was that you would feed toast a source tarball (as root, so it didn’t have to stop and beg for a password), and it would unpack it, examine it, figure out how to jump through all the hoops, compile it (as a non-root user, of course!), and install it in a clean and sane and self-contained way (inspired by GNU stow and such — not that I’d heard of stow back then, though apparently it’s been around a while) so that you could undo it all a year later if you needed to.

Most of the “real” root filesystem would just be a big old symlink farm, pointing into all these private microcosms where each software package lived.  Which was great, because you could tell exactly what version of which package a given file originally came from just by looking at it.  And if you needed to comb through the symlinks to perform some tedious task, like listing installed packages or undoing a change, toast did that too.  Later on it even learned to scour the net for the latest version of a package, given its name.

The original incarnation of toast (version 0.x, as I retroactively called it) worked by finding, examining, and strategically altering the Makefile.  Plus all the symlink stuff, but that was small potatoes.

I always considered it “cheating” to have toast decide how to treat a package based on its name, so it always made its decisions based solely on the contents.  Instead of writing a set of rules for compiling each package, toast used a single set of rules, some of which worked on all sorts of software, and some of which ended up only being useful for a single major release of a single obscure package, until the maintainer would spontaneously wise up and get with the conformist program.

At some depressingly late juncture, I caught wind of the DESTDIR trick, which was (and is) very conformist indeed, coming straight out of GNU’s official coding standard.  You’ve read that, right?  Yeah, me either.  But, listen: Those GNU folks are smart.  Rather: They’re smart and they’ve been around long enough to learn how to write software that people (and other software!) can figure out how to compile and install properly.  Decades hence, brilliant people will still be making bad assumptions that GNU discarded decades ago.  (Me: “Don’t you think?”  Victor: “I think.  Do you think?”  We were roommates at the time.)

The upshot of DESTDIR is that, if a software package knows about it, you can tell it to install itself into a private, self-contained directory tree, and you can do so in a nice standardized conformist way.  You don’t even have to look at the Makefile.  Of course most packages didn’t support DESTDIR, or did it wrong, because nothing works unless you test it, and most people had never even heard of DESTDIR and wouldn’t have cared if they had.  I could hardly blame them then, and I don’t now.  So the Makefile-patching continued.

Although toast still occasionally falls back on its ancient habits, the biggest improvement in version 1.x was the LD_PRELOAD trick, which turns out to be the software equivalent of an orbital mind control laser.  toast 1.x would (and still does) use LD_PRELOAD to fool an installer into believing that it was scattering files hither and yon throughout the filesystem, when in fact its flailings were being intercepted and redirected into its own private sandbox.  Sometimes a particularly intrepid package will try to read back something it just installed, so toast also intercepts reads in order to maintain some pretense of a self-consistent universe.

The illusion is imperfect — it won’t fool a human for long — not because perfection would be impossible, but because it turns out to be the wrong thing to do in practice.  For instance, toast does blatantly unrealistic things like spontaneously creating parent directories whenever an installer assumes some other package must surely have created it already.  If it’s everyone’s job, it ends up being nobody’s job, which makes it toast’s job as far as I’m concerned.

Even back in the days of toast 0.x (never released), the original toastball eventually got to the point where there wasn’t really all that much left of the original Slackware.  (Hence the stray comma after “mostly.”)  Instead there were lots of symlinks pointing to /toast/gcc/2.95/bin/gcc or whatever.  (For those keeping score, toast’s directory structure changed a bit in 1.x too.)

Occasionally I’d meet new people, and we’d get to talking about computers (as people always do), and they would ask what OS I used, and I’d say Linux, and they’d invariably say, oh, really, what distribution? And for a while I’d say that I didn’t use any of them, really, because I used this toast thing, and I’d explain, or not, but anyway the point was that I didn’t actually distribute this purported operating system to anyone else, so it wouldn’t make sense to call it a “distribution.”

So that was “toast linux,” huh?

No!  That’s just the backstory!  We’re still talking about the first toastball here.  The second toastball was the one that ran toast linux.  Pay attention!


No, no, I’m the one who should apologize.  Sometimes I get excited, and I… overitalicize.

Where was I?

Uh… the second toastball…?


Of course I would have popped a toast linux install CD into the second toastball, if only such a thing had existed.  I did have toast, though.  So what I did was I installed some Linux distro or other… gosh, I think it might actually have been SuSE, you know?  But it hardly matters, because it was only there as a sort of incubator out of which to boostrap a system built entirely on toast.  Which wasn’t quite as easy as you’d think, because… well, for one thing, it turns out that different major versions of glibc were incompatible.  Still are, I imagine.

It’s like… think of your filesytem as a planet, and the C library (glibc, in this case) as the atmosphere, and all the life forms that evolve on that planet are all the programs that are compiled against that version of glibc.  So here you have this alien planet, with the new version of glibc, and if you try to run a program on the wrong filesystem it can’t breathe the air, see?

I mean, sure, in theory you could have a planet with two different glibcs, only in practice it’s a bit of a pain because they both want to be /lib/ld-linux.so.2 but only one of them gets to.  (Which seems silly! If the .2 is a version number, why don’t they change it when needed?  Does anyone know?  Why can’t we all just get along?!)  So somebody gets stuck being the sad nonconformist.  But it can’t be SuSE, because that’s already been chiseled onto stone tablets, and I don’t want it to be my toasty new system either because then I’d have to live on a sad nonconformist planet until the end of time, see?

Not true, obviously, but what it amounts to is that at some point you’ve got to play any of a number of different games with static linking, cross-compiling and/or chroot.  Which are like, uh, gosh, I don’t know, robots and space suits and trans-dimensional portals, I suppose.  So it got a bit science fictional in there for a while, you know?

Still, when all the moon dust had cleared, I had this beautiful blue-green planet made of pure toast, unpolluted by any traces of SuSE.

Sure, go ahead and laugh at my sad, hypocritical, heterodox ways.  It’s not about the ideology, I swear.

So that was toast linux?

For all intents and purposes, yes — it was a complete, self-contained operating system, built entirely using toast.  But in another sense, no, not at all!  Because it still wasn’t a distribution, you see.  It wasn’t even automated, let alone released.

Luckily for me, Linux From Scratch had become a thing by then, so I was able to crib off of their notes on some of the trickier parts.  Because, hey, you’ve got all these interlocking pieces written by all these different people, none of whom feel in any way responsible for anything beyond their little fiefdom.  Because, well, they really aren’t, you see.  So you’ve got to work hard to assemble the puzzle, you know? Everybody gets to assume that you’re doing their piece last, and that all the other pieces are good to go and working perfectly, because that’s by far the most common case.

Think about it: Approximately how many Linux distributions are there?  Of any consequence, I mean.  Zero, right?  So basically, if you want to roll your own, you get to violate everybody’s perfectly justified assumptions.  Fun, I tell you.

The other thing about distributing an operating system is that you can’t just give someone the source code, because you can’t run source code, and without an operating system you’ve got no way to build it.  They need a binary.  I mean, you could ask them to get a binary somewhere else to bootstrap from, like Linux From Scratch does, or like I did with SuSE, but at that point you may as well hand them a free SuSE CD, which is, as I was saying, a binary.  If you actually want them to use your thing, I mean, as opposed to making them jump through hoops because you’ve got better things to do than pre-jump a bunch of hoops, flaming or otherwise.

I wanted the smallest binary possible — just enough to download everything it needs, preferably from the original publisher rather than from me.  In particular, I wanted a binary that would fit on a 1.44 meg floppy disk.  Install CDs were the norm by then, true, but floppy drives hadn’t completely dissapeared yet, and it sure sounded like a neat idea.  But you can’t fit enough binaries on a floppy to build everything else from source.  I mean, in theory, sure, but in practice, for GNU/Linux, you needed (and almost certainly still need) 10 or 20 MB minimum.

So my arbitrary mission was to find a third-party site from which my floppy-sized binary could download that 10 or 20 MB of extra binaries it needed to build Rome.  The answer was pretty obvious, actually, since I already wanted to use busybox and uClibc to make the floppy: buildroot.  Which was only just starting to be a thing, but it did exist, and it was called that, and there was a place on uclibc.org where you could download a tiny but complete Linux-based build environment in binary form.

So I told the floppy to grab buildroot first thing when it woke up, right after it finished doing all the other first things, like finding the network and resizing the Windows partition.  Once it had buildroot it would grab the gcc and binutils sources and build a uClibc-to-glibc cross-compiler, and use that to build a native glibc compiler, and so on.

The whole thing was just one long shell script, actually, written in simple enough dialect that busybox could understand it.  It had a GUI and everything — well, a text-based GUI, with a master progress bar and a list of packages showing what’s been downloaded and built and what’s still to come, and you could switch over to look at the downloader downloading or the build thread spitting out the usual reams of impressive-looking text.  And, yes, at some point very early on it would grab and build some minimal subset of Perl 5 so it could run toast, which would build everything else.

Automatically locating and downloading the most recent version of every package on its list was a doomed approach.  Every time anything new came out, the odds of everything breaking were high.  So I spoon-fed it a list of URLs of specific versions of specific packages, usually not the latest and greatest, but something known to work with everything else.  Plus, you know, some relative download and build time estimates, for the progress bar.

I actually got it to the point where it worked reasonably well, running in a chroot jail or VMware, with a caching proxy keeping it from repeatedly downloading massive amounts of source code.

I actually installed it on a real computer once, too.  Not on toastball, as it turns out, but on melba, a cute little Flybook laptop, then shiny and new.  The resulting OS served me well for years.

So, uh… would that have been the real toast linux, then?

Yes!  It totally was!  Or, I mean, it definitely would have been if I’d, you know, released it.  But I never got it to the point where it seemed likely enough to work on any given day to run the risk of someone else trying it.

OMG!  Why not?!  What happened?!?!!!

Well, for one thing, the half-life of the wad of URLs was too short.

So you just… gave up?


Well, no.  I mean, that wasn’t the real reason for abandoning the project.  Nothing like that.

Updating URLs was work, yes, and switching strategies also sounded like work, and I was busy doing other things.

I’d released toast, which now seems to have reached maturity, if only in the sense that I don’t have to do much to keep myself or others from complaining about it.  Even now that I’ve switched to Ubuntu at work (the current toastball being a work computer) and a MacBook Air called toastbook running OS X at home, I still use toast regularly on both machines, and on shared machines like ofb, just to quickly grab the odd piece of software.

Whereas toast linux was in essence an OS installer, needed only when setting up a new computer, which hardly ever happens.  Assuming I even wanted toast linux the operating system, which it turns out I don’t.  At least, not if it means worrying about keeping everything current without destroying the delicate balance of nature.

Honestly, I don’t think there’s any real need for toast linux, you know?  I mean, now that 2010 is almost upon us, toastball can display video with barely half an hour of futzing per biannual upgrade.  (I always remember too late to be careful when buying video hardware.)  That thermal alarm thing?  Hardly ever happens, and so far never when I’m away.

The Mac works perfectly, of course.

So what if Snow Leopard doesn’t ship with mtr?  Who cares if Jaunty and MacPorts have two different versions of Cython, neither one current?  That’s what toast is for!


4 Responses to “A Brief History of toastball”

  1. Jacques Says:

    Well, now it’s happened when I was away. I’m going to try this foolhardy thermal.nocrt thing and see if the CPU melts or what.

  2. Dan Says:

    Yay for toast!

  3. The Making Of « toastblog Says:

    […] video.  I doubt it would have taken more than an hour or two.  But where would be the fun in […]

  4. Long Live toastball! « toastblog Says:

    […] computer in question is toastball. That’s the one that currently lives at the Shiny Devices workshop, over on Coliseum Way, and […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: