How to Build a Supercomputer from Scratch, in 10000 Easy Steps

When Sparky got to his last USAF job before retirement, he found that his new duty title "Assistant Branch Chief" really meant "Glorified Gofer Boy" for some bored PhD-type USAF civilians. These people really didn't want to fool around with all this military-type stuff, they just wanted to play with their pet projects.

Anyway, due to the heavy workload (sarcasm evident?) Sparky flailed about for awhile (which is the same in every tech job Sparky's ever been in), until he found a vacuum to fill.

A foreign researcher at Sparky's new place of employment was working with Finite Difference Time Domain (FDTD) modeling of radiofrequency (RF) absorption in living tissue. Say that five times fast. The model of most research interest is the Visible Human Project (VHP), since, with a sufficiently powerful computational machine, one can predict RF effects on humans.

The default resolution of the VHP is 1mm in three dimensions (colloquially known as the "1mm Man"). The resultant data set is about 18 GB, which must reside in the memory of the machine processing the data, or disk-swapping (much slower than RAM access) will kill productivity on the process.

So this researcher was trying to run the model at a resolution of 5mm in three dimensions (the "5mm Man", with a dataset of 720MB) and trying to get meaningful data out of a two-processor Pentium 500 MHz PC using Windows 98 for the Operating System (OS). This machine was no match for the job, and the output only allowed for generalized predictions of RF tissue absorption/heating. In fact, just getting this machine to output data was something of a task. The average run time for a simulation was on the order of a month, if it didn't crash first, or output a corrupted data file. Tortuous workarounds were required - this machine was once moved to a different building, still running, powered by two UPSes, one plugged into the other, so that a month of runtime would not be lost.

Through discussions around the office, various serfs, Sparky included, decided we needed a Beowulf Parallel Computing Cluster to run the model for the VHP. And we only had a few problems to overcome; very little experience with Linux, zero experience with parallel computing, and zero (none, nada) hardware.

One of us;

Mr Aldon, shown here putting together a Beowulf node, had the Great Idea of latching onto some of the used desktop computers various units in the USAF routinely turn in to Dr Mo. (Actually it's now Dr Ms, which must be in keeping with the New! Improved! Kinder! Gentler! Politically Correct Military - which makes an Old Fart like Sparky wanna puke.) Due to Mr Aldon's ingenuity, we ended up with a pile of Pentium 166 and 200 MHz Desktop Boxes, which we cobbled together using Redhat 7.0 Linux for the OS, and came up with a

Beige Box Breadrack Beowulf, a 12-node machine which actually produced meaningful data (at the 3mm man resolution), albeit after much wailing and gnashing of teeth. The runtime for a simulation on this machine was approximately 4 days, which was a vast improvement over a one month runtime, and more importantly; It Didn't Crash.

Another problem immediately surfaced; with output files of 1 to 2GB, we had no way to get the data into the Windows desktop machines the researchers used to smursh data and write reports. At this time, DvD recorders (at least those which we, with no budget, could afford) were not on the market yet, and CD recording (only 650MB at this time) was not a mature art, at least under Linux (or under Windows, for that matter). We tried splitting the datasets up into separate files and burning them onto multiple CDs, but the nature of the data required even more wailing and gnashing of teeth from the researchers as they tried to put the files back together again.

What we needed was a way to get the data from the Beowulf machine, across the base network, right to the Windows desktop machines on the researchers' desks. At this point in time (and most likely, still today), running a Linux machine on a USAF Base network was(/is) a Big No-No. This is due to the ignorance of Microsoft-Centric System Administrators who know absolutely nothing about any OS but Windows. Microsoft drills into their heads that Linux is not secure, and Windows is. What a crock!!! As an example, Sparky has been running nothing but Linux since 2001, and has no (zero, nada) anti-virus software on any machine. Never been hacked or had a virus/trojan/bot/malware/etc either. Try THAT with Windows! This Windows monopoly of government IT systems represents a humongous waste of taxpayer money. Sparky could climb up on a large soapbox at this point and continue ad nauseam, but you get the idea.

Suffice it to say that this was a hostile environment in which to run a Linux server. An html or ftp server was out of the question, and secure shell (SSH) software was command-line only during this time frame. The only protocol which we had at our disposal was SMB/CFIS (Network Neighborhood on your Windows box) using a Linux box as a Samba server. Remember the dual Pentium 500 MHz machine the model was originally running on? Well Sparky turned it into a dandy Samba server and the data semi-magically appeared in Network Neighborhood on the researchers' desktop machines.

The box next to the printer is the samba server which farms the data out to the researchers. An interesting tidbit is the box at the upper right. That is a Pentium 166 MHz machine with two NIC cards running a floppy (yes, floppy) disc-based Coyote Linux firewall. This was mandatory in order to keep the Powers-That-Be on the base network from screwing up our server in the event they were able to break through the firewall. And Oh Yes, they tried. Although with their limited tools (i.e; Windows-Based) they were never successful.

Along the way, we kept getting more powerful desktop machines (although not as powerful as we really needed) from Dr Mo, and we tried the

Naked Breadrack Beowulf, a 24-node machine which produced quite a few datasets over the course of several months, but was even uglier than the Beige Box Breadrack Beowulf. In fact, its ugliness was its undoing, which we found out The Hard Way. It seems that certain PhDs, especially those types who tend to be in charge, like bright, shiny things, and this machine was Sofa King Ugly, that they couldn't bear to show it to any of the MFWICs who held the purse strings.

In an attempt to hide the offending contraption out of sight, such as in a closet (although one with plenty of ventilation) the Naked Breadrack Beowulf was morphed into the

Sanford and Son Beowulf. Talk about ugly! But since the purpose was to make it compact and hideable, it would have been successful, except for a bastardization of one of Einsteins maxims; "Things should be made as simple as possible,but not simpler". As can be seen here;

this thing was just Toofa King Compact! Many simulation runs were aborted because someone bumped the machine and one of the nodes would reboot. And yes, with all the components of 24 desktop machines in a space that size, maintenance was a nightmare. So we had a new rule to live by; Design for Maintainability! Also, the heat coming off of this rack was amazing, and these were only Pentium 233 and 266MHz nodes!

It was a useful machine though, and enabled us to gain much-needed experience with building, operating and maintaining a production Beowulf machine.

Sparky had a rude awakening about this time; he had naively assumed that the research grant process depends solely on scientific merit. He found out that just ain't so - politics plays much too large a role in determining which research projects get funded.

Somewhere amongst all this brew-ha-ha, and the datasets of meaningful RF absorption values rolling out in production-line fashion, someone who held purse strings began to notice, and a contract (with actual funds) was granted to cousin Tony's company (Sparky and Tony didn't know they were cousins at the time, so don't blame us) to actually build a machine from the ground up to run the 1mm Man dataset. As noted previously, this required the entire 18 GB model to reside in memory on the machine.

Cousin Tony's company eventually delivered the three bay machine shown here;

Although when it showed up, it looked like a bunch of boxes and equipment - and no rack. Prior to delivery, Mr Aldon had been in lengthy discussions with the base metal shop, wheedling and cajoling them into fabricating metal plates to mount the individual nodes in the rack which we purchased.

A closeup of one of the nodes;

Showing the fabricated plate and method of mounting the node hardware. Mr Aldon obviously remembered The Good Old Days, when Quasar TV Sets came with the "Works in a Drawer" for ease of maintenance. Remember what we learned on the Sanford and Son Beowulf?!? Design for Maintainability!!!
How's this for ease of maintenance???

Some detail of the interior of the rack, showing the Mambo HP switch to connect the whole thing together;

As originally delivered, the machine had 72 nodes, made up of AMD Athlon 750MHz and 1GHz CPUs with 500MHz of RAM each. This gave a total system memory of 36GB.

After several complaints from certain HMFICs that the machines might be producing data, but they weren't bright and shiny enough, having no banks of blinkee lights nor Star-Trek-esque visual displays, Sparky scrounged up a

Red Light Special which signified that the Beowulf room was indeed a Happening Place. This conferred a warm, fuzzy feeling unto the Questers of Bright, Shiny Objects.

As some of you may know, if you can cheaply narrow down what you are trying to find out through an experiment, then you have a savings multiplier which allows you to refine the design of your test. This has several effects - one is to make some tests cheaper, and another is to allow for more complicated experiments, with greater probability of success.

A short while after the 72-node system went online, the MFWICs realized that all this screwing around we had been doing (with essentially zero funding - except for the contract to Tony's company) was actually enabling the researchers to predict heating effects on living tissue, as verified by experiments that were ongoing. And by some sort of funding miracle they gave us more money to upgrade the system to 96 nodes.

One of the cool things about a job like this is you get to play with lots of neat equipment that you don't have to buy!

Stuff like a box full of CPUs;

Or (at the time) about $5K worth of memory;

Or maybe a pile of disk drives;

Here's Mr Aldon loading the OS on (yet another) node;

And the finished add-on rack, with the additional 24 nodes installed (not too sure of they were all running at this point);

It's tough to see in this picture, but the flourescent light grating (with no light) above the rack is actually a vent into the air-conditioned space above. Our facilities guy, Mr Frank, was constantly engaged in heated negotiations with the Base Civil Engineers. Apparently the Beowulf room required more AC capacity than any other office space on the base. Keep that in mind, kiddies, when you're planning your own parallel cluster to rip Dvds!!!

So what's the outcome of all this? The answer is - a bunch of numbers! However, using graphics software, pretty pictures can be made from All This Data. Here's an example;

What you see is the 1mm Man, with an RF source, about 915MHz, illuminated from the front, not sure about the polarization. The red colors represent more intense heating of the tissue - just like looking at the WX radar.

The Researchers made much foofaraw from the results of this work, but Sparky's just an engineer;

From: The 2002 Bioelectromagnetics Society Symposium

FINITE DIFFERENCE TIME DOMAIN (FDTD) MODELS PREDICT ORGAN RESONANCES IN THE 1-MM MAN MODEL. J.M. Ziriax, J.A. D' Andrea, W.D. Hurt, D. Verrett, P.A. Mason, D. Hatcher, and D. Cox. Naval Health Research Center Detachment, Brooks Air Force Base, Texas 78235, USA; Air Force Research Laboratory, Directed Energy Bioeffects Division, Brooks Air Force Base, Texas 78235, USA.

INTRODUCTION: Empirical dosimetry of even a single frequency is time consuming and labor intensive, and as such, is typically limited to critical dosimetric data. Computer models can be used to explore a wide range of frequencies and exposure conditions in anatomically realistic models without the expense and effort of empirical measurements.

OBJECTIVES: Localized SARs vary dramatically with frequency and exposure orientation. Here we report on an ongoing exploration of localized and whole body SARs in our 1-mm human model across a range of exposure frequencies and orientations.

METHODS: NHRC-DET and AFRL/HEDR have jointly developed models of mouse, rat, goat, and rhesus monkey in addition to the human model (Mason et al., 1995, 1999). The 1-mm man model was developed from images provided by the National Library of Medicine as part of the Visible Human Project (http://www.nlm.nih.gov/research/visible/visible_human.html). Our FDTD program, based on code originally developed by Kunz and Luebbers (1993), was used to calculate SARs in the 1-mm man model. The electrical properties of each of tissue were set according to data and fits published by Gabriel (1996). The code has been parallelized using the MPI message-passing library. The advantage of using the MPI is that the code can run on parallel computer systems composed of networks of computers. These may be networked workstations, or massively parallel systems such as Linux-based Beowulf systems. These systems are easily constructed of relatively inexpensive PC-hardware. The figure shows sample results for whole body, skin and eye structures for 1700 to 4000 MHz in the PEKH orientation (RF source is to right of then man with a vertical E-field).

Note: Graph not available yet.

CONCLUSIONS: In this frequency range, SARs for whole body and skin change monotonically, while SARs for the eye structures change non-monotonically. We will expand the graph to lower frequencies where we have seen eye (900 MHz) and whole body (70 MHz) resonance in coarser or head-only models. As the FDTD predicts SARs with the same resolution as the anatomical model, in this case in 1-mm volumes, then this resolution easily exceeds that of typical empirical methods. The thermal and biological
significance of these highly localized SARs will have to be determined experimentally.

References;
Gabriel C. Compilation of the Dielectric Properties of Body Tissue at RF and Microwave Frequencies,
Brooks AFB, TX: Armstrong Laboratory Report, AL/OE-TR-1996-0037, 1996.
Kunz, KS and Luebbers RJ. The Finite Difference Time Domain Method for Electromagnetics, CRC
Press, Inc., Boca Raton, FL, 1993. P. A. Mason, et. al., Database created from magnetic resonance images
of a Sprague-Dawley rat, rhesus monkey, and pigmy goats, FASEB J., 9: 434-440 1995
Mason PM, Ziriax JM, Hurt WD, Andrea JA. 3-Dimensional models for EMF dosimetry. In Electricity
and Magnetism in Biology and Medicine edited by Bersani, Kluwer Academic/Plenum Publishers, 1999.

The views expressed in this article are those of the authors and do not necessarily reflect the official policy
or position of the Department of the Navy, Department of the Air Force, Department of Defense, or the U.S.
Government.