Bluecrab is the main cluster at MARCC with close to 20,000 cores (November 2015) and a combined theoretical performance of over 930 TFLOPs. The compute nodes are a combination of Intel Haswell and Ivy Bridge processors and several Nvidia K80 GPUS linked via FDR-14 Infiniband interconnects. It also features two types of storage: 2 PB Lustre (IEEL) and 15 PB ZFS/Linux.
The standard compute nodes contain 2 Intel Xeon E5-2680v3 (Haswell) processors, with 12 cores and 128 GB DDR4, 2.5 GHz (Marked TDP frequency) or 2.1GHz AVX base frequency. Each node has a theoretical speed of 960 GFlop/s.
The large memory nodes are Dell PowerEdge R920 servers with quad Intel “Ivy Bridge” Xeon E7-8857v2, (3.0GHz, 12 core, 30MB, 130W). Each node has 1024 GB RAM
The GPU nodes are Dell PowerEdge R730 servers with dual Intel Haswell Xeon E5-2680v3 (12 core, 2.5 GHz, 120W), 128 GB of 2133 MHz DDR4 RAM. (AVX frequency: 2.1GHz) and two Nvidia K80s per node.
The FDR-14 Infiniband topology is 2:1 with 56 Gbps bandwidth. The lustre file system provides an aggregate bandwidth of 25 GBps (read) and 20 GBps (write).
|Nodes||Type||Description||Total cores||Theoretical Performance/TFLOPs|
|676||Regular compute nodes||Intel Haswell dual socket, 12-core processors, 2.5GHz, 30MB cache, 128GB RAM||16,224||648.9|
|50||Large memory nodes||Intel Ivy Bridge quad socket, 12-core processors, 3 GHz, 1024GB RAM||2,400||57.6|
|48||GPU nodes||similar to compute nodes, 2 Nvidia Tesla K80 GPUs per node||1,152||46 (cpu) + 179.5 (gpu)|
|–||Lustre||2 PB usable high performance file system|
|10||ZFS||18 PB raw|
|TOTALS||19,776||Total combined: 932.5 TFLOPs|
|48||Regular compute nodes||Intel Broadwell dual socket, 14-core processors, 2.6GHz, 30MB cache, 128GB RAM||1,344||55.9|
|24||GPU compute nodes||Intel Broadwell dual socket, 14-core processors, 2.6GHz, 30MB cache, 128GB RAM plus 2 Nvidia K80 GPUS||672||28 (cpus) + 89.75 (gpu)|
|8||Regular compute nodes||Intel Broadwell dual socket, 14-core processors, 2.6GHz, 30MB cache, 128GB RAM (condo)||224||9.3|
|New TOTALS with Additional hardware||21,792||Total combined: 1.12 PFLOPs|