In a post from late June, Jeff Atwood at Coding Horror discusses the horrible cost of a large HP server (scaling up to 32 processor cores in eight AMD x86 sockets), compared to a bunch of simple single-socket basic servers. There are some interesting notes on relative costs of small-and-simple servers, including things like administration and power. There is an undercurrent to the post and the comments that the big HP machine is “overpriced”. I don’t think it is. If you have ever had Erik Hagersten as a teacher in computer architecture, you will know why.
Essentially, the cost of connecting a bunch of processors goes up exponentially as the number of processors increase. I think this is just as true for Hypertransport-connected AMD 4-way chips as it was for Sun 10000 servers ten years ago. The backplane takes over as the cost driver, from the processors and memories and other obviously useful stuff. Scaling up beyond the commodity space (which is a moving target over time, certainly) requires a lot of engineering and custom hardware design. This makes the cost exponentially higher, but for a good reason.
Note that this is one of the reasons that the Sun Niagara/UltraSparc T-line machines are compelling: with 32 or 64 threads per socket, getting to 100+ hardware threads is way cheaper using that architecture than anything else in the server space (in deep embedded, 100+ cores is a yawn).
Just a small rant, while on vacation.