Blue Crab is the main cluster at MARCC with over 23,000 cores (June 2018) and a combined theoretical performance of over 1.4 PFLOPs. The compute nodes are a combination of Intel Ivy Bridge (large memory nodes), Haswell, Broadwell and Skylake processors and several Nvidia K80/P100 GPUS linked via FDR-14 InfiniBand interconnects. It also features two types of storage: 2 PB Lustre (IEEL) and 14 PB ZFS on Linux.

The standard compute nodes are  Intel Xeon E5-2680v3 (Haswell, 12 cores per CPU), E5-2690v4 (Broadwell, 14 cores per CPU) and Gold 6126 (Skylake, 12 cores per CPU) and 128/96 GB DDR4, 2.5/2.6/2.6 GHz (Marked TDP frequency) or 2.1/2.6/2.3 GHz AVX2(AVX512) base frequency.

The large memory nodes are Dell PowerEdge R920 servers with quad Intel “Ivy Bridge” Xeon E7-8857v2, (3.0GHz, 12 core, 30MB, 130W). Each node has 1024 GB RAM.

The GPU nodes are Dell PowerEdge R730 servers with dual Intel Haswell Xeon E5-2680v3 (12 core, 2.5 GHz, 120W), 128 GB of 2133 MHz DDR4 RAM. (AVX frequency: 2.1GHz) and two Nvidia K80s per node.

The FDR-14 Infiniband topology is 2:1 with 56 Gbps bandwidth. The Lustre file system provides an aggregate bandwidth of 25 GBps (read) and 20 GBps (write).

 

 

Nodes Type   Description Total cores Theoretical Performance/TFLOPs
676 Regular compute nodes Intel Haswell dual socket,  12-core processors, 2.5GHz, 30MB cache, 128GB RAM 16,224 648.9
50 Large memory nodes Intel Ivy Bridge quad socket, 12-core processors,  3 GHz, 1024GB RAM 2,400 57.6
48 GPU nodes Intel Haswell dual socket,  12-core processors, 2.5GHz, 30MB cache, 128GB RAM, plus
2 Nvidia K80 GPUs per node
1,152 46 (cpu) + 279.36 (gpu)
Lustre 2 PB usable high performance file system
10 ZFS 18 PB raw
TOTALS 19,776 Total combined: 1,032TFLOPs
48 Regular compute nodes Intel Broadwell dual socket, 14-core processors, 2.6GHz, 30MB cache, 128GB RAM 1,344 55.9
24 GPU compute nodes Intel Broadwell dual socket, 14-core processors, 2.6GHz, 30MB cache, 128GB RAM plus,
2 Nvidia K80 GPUS per node  (2.91TF per gpu)
672 28 (cpus) + 139.68 (gpu)
8 Regular compute nodes Intel Broadwell dual socket, 14-core processors, 2.6GHz, 30MB cache, 128GB RAM (condo) 224 9.3
4 GPU-P100 compute nodes Intel Broadwell dual socket, 14-core processors, 2.6GHz, 30MB cache, 128GB RAM plus 2 Nvidia P100 12Gb GPUS (condo) 112 4.65 (cpu) + 4.73TF/ (gpu)
48 Regular compute nodes Intel Skylake Gold 6126 dual socket, 12-core processors, 2.6GHz, 30MB cache, 96GB RAM (condo) 1152 99.8 (cpu) + 37.6 (gpu)
New TOTALS with Additional hardware 23,280 Total combined: 1.4PFLOPs