Before I go into the specifics of my record attempt, it is important for you to have some background information. Over the past few years I have been steadily acquiring server hardware to increase my knowledge of system administration and cyber security. This is not only for personal growth: it has also helped me immensely in my professional career. I have served in various roles including cyber security (Cyber Defense Analyst, Chief Information Security Officer) and system administration (Senior System Administrator). Historically, I have also donated my spare system computing resources to various distributed computing projects through the BOINC1 platform. In late 2018 I decided to shift the primary focus of my lab to distributed computing. I still maintain a small 1U server for personal use, but most of the other servers run BOINC exclusively. UPDATE: Partway through the world record attempt (July, 2019), I founded a nonprofit organization, North Alabama Charitable Computing, which repurposes enterprise-grade compute and storage equipment for STEM research. After I break the Pi world record, most of my servers will be donated where they will provide scientists and various research projects with computing power. Anyways, back to the reason you are here: My primary reason for attempting to break the record set by Emma Haruka Iwao/Google2 (March, 2019 – 31.4 trillion digits) is to test the limits of my hardware. Once I am finished breaking the Pi world record, the server and hard drives will be repurposed for STEM research (e.g., BOINC, Open Science Grid3, university research projects). One of the first things I did was reach out to Alexander Yee, who wrote and maintains a program called y-cruncher4. y-cruncher is a program that computes various mathematical constants based on user input. y-cruncher is able to take advantage of the newest processor features and features many optimizations, which in part explains why it has been used to set several of the past Pi records (and other mathematical constants). After emailing back and forth with Alex, I started to gain an understanding of what would be required for such a feat: 50 trillion digits of Pi. In order to calculate this many digits, y-cruncher determined I would need 256TiB of ephemeral storage, in addition to 38TiB for the final output. One of the first things that you may notice is that y-cruncher calculates everything in tebibytes (base-2), rather than terabytes (base-10). It can be confusing since these units of measure are sometimes used interchangeably (I’m looking at you, Windows5, and Linux6 df “-h” vs “-H”), but the difference can be significant: for example, 256TiB is equivalent to 281TB. You don’t want to get 3/4 of the way through a record attempt and realize you don’t have enough storage. Once I had a rough estimate of the amount of equipment and disk required for the Pi computation, I started looking for used hardware, primarily on eBay. From January to March of 2019, I started purchasing equipment that would be required for the attempt: a new server, several hard drives, various PCI-E cards, and a tape libary and drive, to name just a few. Below is a picture of my home lab in it’s entirety (except for an Asus ESC8000 G4 in my office). Everything inside the red box is used for the Pi computation. HP PROLIANT DL580 Gen8 This server (Ubuntu 18.10) performs the Pi computation. The final compressed Pi digits will be transferred to the Dell R720xd via a cifs-mounted directory (/drives/main_output). (4) Intel Xeon E7-4880V2 2.5GHz 15C/30T CPU 320GB DDR3 PC3-8500R ECC RAM (5) 2.5” 100GB SATA MLC SSD (RAID 6) (4) 6Gb LSI 9207-8e Dual Port SAS HBA Connected to each HPE D3600 disk shelf via single SFF-8088 to SFF-8644 cable 2m Intel 10-Gigabit X540-AT2 Dual Port Ethernet Adapter Dedicated link to Dell R720xd for Veeam backups Intel 1-Gigabit I340-T4 Quad Port Ethernet Adapter Connection to the Internet (4) HP StorageWorks D3600 These drives are used to perform computations in lieu of server memory (known as y-cruncher Swap Mode7) to provide the resources required to calculate Pi (mounted on the DL580 as /drives/1 – /drives/48). I originally used HP StorageWorks D2600 disk shelves, but I didn’t realize the SATA backplane was limited to 3Gb/s. I upgraded to the D3600 disk shelf which supports 6Gb/s SATA in June of 2019. (12) 3.5” 6TB SATA 7.2K HGST Ultrastar He8 HDD (JBOD) Dell PowerEdge R720xd This server (Windows Server 2016) serves two primary purposes. First, it is connected to the HP tape library. A Veeam job is manually run each month, which backs up data from the 48 drives running the Pi computation directly to tape. Second, it has an additional RAID6 storage volume (see Dell PowerVault MD1200) which will contain the 50 trillion compressed Pi digits. (2) Intel Xeon E5-2670 8C/16T CPU 128GB DDR3 ECC RAM (2) 2.5” 600GB SAS 10K Seagate HDD (RAID1) Emulex LPE 12002, Dual Port 8Gb Fiber Channel HBA Connected to tape library using (2) LC-LC OM3 2m fib
Source: Calculating Pi: My attempt at breaking the Pi World Record | Bits and Bytes