The Most Beautiful Supercomputer in the World

Sorry everyone, but I’m not talking about Ranger, Sun’s 500 TFLOP system which is still a-birthing at TACC in Texas. More on Ranger and Sun’s Constellation System in the second half of this entry…

With my supercomputer-as-art background from Thinking Machines, I’m perhaps pre-disposed to appreciate nicely done computers and computer installations. I was blown away when I saw the photos below of MareNostrum, currently the 9th largest supercomputer in the world and the largest in Europe, which is installed in a former chapel at the Barcelona Supercomputing Center (BSC). Wow.

[marenostrum at the barcelona supercomputing center, view1]

[marenostrum at the barcelona supercomputing center, view2]

As beautiful as this is, the site is a bit more typical when you look deeper. Here are a few shots:

[marenostrum at the barcelona supercomputing center, under floor]

[marenostrum at the barcelona supercomputing center, rear view]

MareNostrum’s Myrinet interconnect fabric requires four cabinets. Myricom did a nice job of reducing the cable count as much as they could by using quad-link ribbon cables between their switch elements. But with 12 separate switch elements in the fabric that means they still have a lot of cables and more cables means more connectors, more points of potential failure. They also use one cable per compute node, as is typically done in cluster configurations. Lots more cables, lots more connectors.

Contrast this with Sun’s Constellation System approach, which uses one large, ultra-dense InfiniBand switch (therefore no inter-switch cabling) and node connections that are bundled three per-cable and connected directly to Constellation’s ultra-dense blade chassis. This takes complexity management to the next level and makes petascale computing systems an achievable goal.

For a cogent description of Constellation and its value proposition, including some nice diagrams, check out Jonathan’s blog.


18 Responses to “The Most Beautiful Supercomputer in the World”

  1. Anonymous Says:

    The Myrinet crossbars in this fabric have 32 ports. So, it would be possible to build a 8192 ports switch with a diameter 5 like TACC. However, it’s not cost effective, because the vast majority of clusters out there have less than 256 nodes, which is the port count on a single Myrinet 2000 switch.
    Sun’s primary motivation to build Magnum was definitively not to make a profitable business.

  2. Josh Simons Says:

    Thanks for your comment.
    I did not mean to imply that Myrinet could not be used to build large interconnect fabrics. My primary point was that using increasing numbers of small switch elements to achieve that size introduces significant cabling and reliability issues. This is not a Myrinet-specific observation–it is true iof InfiniBand fabrics as well when built from smaller components.
    A few observations relative to your comment about Magnum (Sun’s switch.) First, I generally agree with your implied point that business at the top of the HPC market does not generally make a lot of money for a vendor. Those high-end customers will argue that "solving their problem solves the problem for all customers with less stringent requirements." While, strictly speaking, this is a true statement, it is also generally true to say (going back to your point about the vast majority of customer
    requirements) that if we solve problems at perhaps two orders of magnitude BELOW their requirements, we have still solved the problem for the majority of the market (I am especially thinking of software when I say this.)
    Having said that, there ARE ways for a full system vendor like Sun to find a reasonable business case for these systems when you look beyond just the compute nodes and the interconnect fabric. (And even the interconnect in this case offers something valued by customers.)
    Last point. HPC is very often the incubator space for new technologies that eventually become important for the much larger commercial, non-technical datacenter. Developing expertise in very large, scalable machine room interconnect fabrics seems a good thing from that perspective, at least that’s my view.
    Josh Simons

  3. Max Says:

    That is so sweet. I wish my chapel had one of these …

    Max … Out! – free online operating system


    wooow :O

  5. ebrahim Says:


  6. John Sargis Says:

    this supercomputer uses the IBM PPC970 microprocessor and I was on that design team. Nice to see people appreciate the machine!

  7. trey Says:

    I studied at UPC, the university that hosts the supercomputer in Barcelona and got to visit it as part of the program I was with. They had monastic chanting in the background as we walked in, and some of the guys I was with got down and prayed at the thing. They had some of there old nodes on display as well, some of them looked straight out of Kubrick. If your ever in Barcelona and you’ve got some connections, this is a really cool place to visit.

  8. Travis Says:

    Why is it in a chapel again? No one questions that at all?

  9. Matthias Says:

    It’s funny that despite all these awesome technological advances that we are enjoying today, when you want to build a super-computer you still need to use miles of cable to hook the damn thing up!

  10. myxomop Says:

    Looks realy fantastic!

  11. JMR Says:

    Putting a supercomputer in a former chapel – this reminds me of something James Watson wrote in "The Double Helix":
    The book I poked open the most was Francis’ copy of The Nature of the Chemical Bond. Increasingly often, when Francis needed it to look up a crucial bond length, it would turn up on the quarter bench of lab space that John had given me for experimental work. Somewhere in Pauling’s masterpiece I hoped the real secret would lie. Thus Francis’ gift to me of a second copy was a good omen. On the flyleaf was the inscription, "To Jim from Francis – Christmas ’51." The remnants of Christianity were indeed useful.

  12. lawstud Says:

    I’m pretty sure you misused the word "cogent" in the last sentence.

  13. eSpy Says:


  14. fugazzi Says:

    Awesome, I can really play Crysis on high now.

  15. Singe Says:

    I’m sorry but this compy looks absolutely nothing like GlaDOS.

  16. Snale Says:

    It’s really impressive. I study at the UPC, very close to it. It has a surreal feeling… very nice.
    /trey: no one really cares, it wasn’t used anyway.

  17. mirc Says:

  18. Portal Lifestyle On The Net Says:

    as beautiful as it’s performance…combining technology with art… Let’s Think Out of The Box Portal Lifestyle On The Net Tricks and Tips Big Conspiracies

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: