With my supercomputer-as-art background from Thinking Machines, I’m perhaps pre-disposed to appreciate nicely done computers and computer installations. I was blown away when I saw the photos below of MareNostrum, currently the 9th largest supercomputer in the world and the largest in Europe, which is installed in a former chapel at the Barcelona Supercomputing Center (BSC). Wow.
As beautiful as this is, the site is a bit more typical when you look deeper. Here are a few shots:
MareNostrum’s Myrinet interconnect fabric requires four cabinets. Myricom did a nice job of reducing the cable count as much as they could by using quad-link ribbon cables between their switch elements. But with 12 separate switch elements in the fabric that means they still have a lot of cables and more cables means more connectors, more points of potential failure. They also use one cable per compute node, as is typically done in cluster configurations. Lots more cables, lots more connectors.
Contrast this with Sun’s Constellation System approach, which uses one large, ultra-dense InfiniBand switch (therefore no inter-switch cabling) and node connections that are bundled three per-cable and connected directly to Constellation’s ultra-dense blade chassis. This takes complexity management to the next level and makes petascale computing systems an achievable goal.
For a cogent description of Constellation and its value proposition, including some nice diagrams, check out Jonathan’s blog.