Bluecrab is the main cluster at MARCC with close to 20,000 cores (November 2015) and a combined theoretical performance of over 930 TFLOPs. The compute nodes are a combination of Intel Haswell and Ivy Bridge processors and several Nvidia K80 GPUS linked via FDR-14 Infiniband interconnects. It also features two types of storage: 2 PB Lustre (IEEL) and 15 PB ZFS/Linux.

The standard compute nodes contain 2 Intel Xeon E5-2680v3 (Haswell) processors, with 12 cores and 128 GB DDR4, 2.5 GHz (Marked TDP frequency) or 2.1GHz AVX base frequency. Each node has a theoretical speed of 960 GFlop/s.

The large memory nodes are Dell PowerEdge R920 servers with quad Intel “Ivy Bridge” Xeon E7-8857v2, (3.0GHz, 12 core, 30MB, 130W). Each node has 1024 GB RAM

The GPU nodes are Dell PowerEdge R730 servers with dual Intel Haswell Xeon E5-2680v3 (12 core, 2.5 GHz, 120W), 128 GB of 2133 MHz DDR4 RAM. (AVX frequency: 2.1GHz) and two Nvidia K80s per node.

The FDR-14 Infiniband topology is 2:1 with 56 Gbps bandwidth. The lustre file system provides an aggregate bandwidth of 25 GBps (read) and 20 GBps (write).

 

 

Nodes Type   Description Total cores Theoretical Performance/TFLOPs
676 Regular compute nodes Intel Haswell dual socket,  12-core processors, 2.5GHz, 30MB cache, 128GB RAM 16,224 648.9
50 Large memory nodes Intel Ivy Bridge quad socket, 12-core processors,  3 GHz, 1024GB RAM 2,400 57.6
48 GPU nodes similar to compute nodes, 2 Nvidia Tesla K80 GPUs per node 1,152 46 (cpu) + 179.5 (gpu)
Lustre 2 PB usable high performance file system
10 ZFS 18 PB raw
TOTALS 19,776 Total combined: 932.5 TFLOPs
48 Regular compute nodes Intel Broadwell dual socket, 14-core processors, 2.6GHz, 30MB cache, 128GB RAM 1,344 55.9
24 GPU compute nodes Intel Broadwell dual socket, 14-core processors, 2.6GHz, 30MB cache, 128GB RAM plus 2 Nvidia K80 GPUS 672 28 (cpus) + 89.75 (gpu)
8 Regular compute nodes Intel Broadwell dual socket, 14-core processors, 2.6GHz, 30MB cache, 128GB RAM (condo) 224 9.3
New TOTALS with Additional hardware 21,792 Total combined: 1.12 PFLOPs