This article is an informal, detailed description of our multi-organization collaboration to get protein folding Computational Biology applications running on Arm platforms, to assist with coronavirus research. The intended audience includes the participants in the project, their managers and executives who would like visibility into the project, and external observers, including journalists, who may be interested in the project.
The first confirmed case of coronavirus in the United States was reported on January 20, 2020. On February 21, 2020, the Rosetta@home lab announced their work in fighting coronavirus. On February 27, 2020, Dr. Greg Bowman announced that the Folding@home project was calling for more GPU and CPU devices to assist in their coronavirus research.
On March 14, 2020, Lloyd Watts and Eric Skup at Neocortix independently saw news coverage of the call for devices, here and here, and we recognized that we might be able to run Folding@home on our Scalable Compute platform, which consists of secure Debian Linux containers running on Android/Arm mobile devices.
Neocortix software architects Michael Coffey and Dmitry Moskalchuk quickly found that neither Folding@home or Rosetta@home could run on Arm devices, because no Linux-on-Arm build was provided by those projects. At about the same time, David Tischler at MiniNodes/Balena was independently investigating, and came to the same conclusion. So, to make progress, we would need a Linux-on-Arm build of Folding@home and/or Rosetta@home.
On March 15, 2020, Lloyd posted a message on foldingforum.org, asking for a Linux-on-Arm build of Folding@home, and volunteering Neocortix help to build one. And simultaneously, we tried reaching out to Dr. Vijay Pande, who was listed on the Folding@home website as the leader of the project. But that information was out-of-date; we did not realize at that time that Dr. Pande had retired from the project and handed it off to Dr. Greg Bowman.
The next day, March 16, 2020, Rex St. John at Arm responded to our posted message, offering help from the Arm Developer Community. On March 17, 2020, Lloyd, Rex, and David Tischler had a phone meeting and agreed to work together to reach the Folding@home and Rosetta@home projects, and try to get access to their source code so we could help them with a Linux-on-Arm build of their projects.
Only a few hours later, Rex had made positive contact with the Rosetta team. Brian Koepnick and David Kim at Rosetta responded with a link to the Rosetta source code repository, and guidance about how we could contribute changes to it without being official Rosetta team members. Dennis Hanson at UW CoMotion provided definitive guidance about the licensing terms.
Building Rosetta for Arm
The next step was to build Rosetta for Arm. Dmitry Moskalchuk at Neocortix took the approach of a cross-build, i.e. building an aarch64 executable on an Intel x86_64 machine. Sahaj Sarup at Linaro.org and David Tischler at MiniNodes/Balena took the approach of a native build, i.e. building an aarch64 executable on an Arm64 machine. Ed Vielmetti at Packet.com gave input on available server-class Arm64 machines to use for native build and testing.
On March 20, 2020, after several days of intense work, Dmitry completed the aarch64 build of the minirosetta executable, as would be needed to run Rosetta@home in the BOINC framework. He tested it using an Arm64 emulator running on his x86_64 machine. Dmitry provided his source code patches and instructions on how to build minirosetta and run the unit tests to the Rosetta team and the extended developer group. David Kim immediately began verifying the new patches, cross-building the aarch64 application, and preparing for testing. He advised the group that verification, testing, and integration with other parts of the broader Rosetta project would take 3-7 days, before a new Linux-on-Arm build could be published.
On March 22, 2020, Sahaj Sarup and David Tischler completed a native build of minirosetta on their Arm64 machines, based on Dmitry's cross-build.
Testing Rosetta on Arm
David Kim sent out test scripts and instructions for a functional test of minirosetta which could be run on an Arm64 machine. Sahaj ran it quickly and sent the results back to David Kim, who confirmed successful operation of aarch64 minirosetta on an Arm64 machine (AMD Seattle Arm-based SoC). Soon after, Dmitry ran the tests on his Raspberry Pi 4 and Galaxy Note 9 phone, running inside a Neocortix Scalable Compute instance secure Debian Linux container. Here is a screenshot of the output of the ab initio model produced on aarch64 from Sahaj's first run:
On March 23, 2020, David Kim verified all of the test results, and reported performance of the different devices on the tests (average runtimes in seconds per model):
Dmitry’s Raspberry Pi 4 with SD storage: 272s
Dmitry’s Galaxy Note 9: 118s
Sahaj’s AMD Seattle Arm-based SoC: 166s
David Kim's Intel i7 desktop at work: 49s
On March 26, 2020, the Rosetta team has done further integration work, and provided a new build for testing. Linaro, MiniNodes and Neocortix have all verified the new build on their platforms. The Rosetta team is continuing their work on integration and testing, toward the publication of the new Linux-on-Arm version of Rosetta@home.
Deployment of Rosetta@home on Arm
On March 31, the Rosetta team published the new Linux-on-Arm build.
Since then, the group effort has shifted to deployment onto Arm devices. As of April 16, the group has over 793 Arm devices contributing to the Rosetta project, with over 6.6M credits earned. Please see Sahaj Sarup's blog which includes a description of Arm devices in his lab that are being used for Rosetta@home. David Tischler's blog gives a thorough description of how to run Rosetta@home on Arm devices. And David has also created FoldForCovid.io which allows people to volunteer their laptop and Raspberry Pi devices for Rosetta/Covid-19 research (but not mobile devices).
On April 8, Neocortix completed its upgrades for temperature and battery monitoring, and is now ramping up deployment onto its existing network of 3,000 mobile devices, soon to be followed by up to 42,000 volunteer devices from our PhonePaycheck waiting list.
As of April 16, the broader group has over 793 Arm devices contributing to the Rosetta project, with over 6.6M total credits earned. By May 6, Neocortix had ramped up to 613 simultaneous devices, and reached the Top 1% of Rosetta contributors with 2,303,633 total credits earned and a Recent Average Credit score of 102,963. By May 28, Neocortix has earned a total of 4,315,994 credits. By June 8, the broader group has 5628 devices contributing to Rosetta, with over 102.7M total credits earned.
Connecting with Folding@home
On March 17, 2020, Michael Coffey at Neocortix figured out that Dr. Greg Bowman is now leading the Folding@home project, and Lloyd Watts reached out to Dr. Bowman via email. Rex St. John at Arm independently figured this out at about the same time and left a message for Dr. Bowman.
On March 18, 2020, Dr. Bowman replied, saying "Thanks, appreciate the offer! I'm going to have to put this on hold for now though, as our focus for now is getting our server-side infrastructure scaled up to meet all the new interest in participating. I'd love to explore mobile devices in the future though, especially as they get more powerful!"
On April 1, 2020, Dr. Bowman joined the group videoconference review meeting, and indicated that the Folding@home project had made good progress with expanding their server-side capacity, and he was interested in supporting our group effort to provide Arm support for Folding@home. As a starting point, he directed us to Gromacs.org, which is an open-source library for molecular dynamics. Dmitry quickly built it for aarch64 (which was already supported, including Arm SIMD support). On April 6, the Gromacs-core team was introduced to the project with the creation of a Slack channel #gromacs-core. The group is now eagerly awaiting access to the Folding@home sources to begin the work of a Linux-on-Arm build.
On April 16, with help from Joseph Coffland (Folding@home) and Artem Zhmurov (Gromacs), Dmitry completed a build of FaHCore with the new Gromacs 2020.1 engine, for both x86_64 and aarch64 targets, with SIMD optimization (AVX2_256 and ARM_NEON_ASIMD, respectively). On April 20, Dmitry completed a build of FaHClient. On April 24, Dmitry provided a performance-improved version of FaHCore using MPI, OpenMP, and Arm Performance Libraries. On May 5, the Neocortix, Gromacs and Folding@home teams completed a number of outstanding tasks to allow Folding@home to fully use the Gromacs 2020.1 engine. On May 20, the changes were accepted and merged into the Folding@home master branch.
The next step was to make Folding@home Core builds for all supported platforms: Windows, Linux, macOS, with CUDA. On May 26, Joseph asked for help to complete those builds, and Dmitry agreed to do it, with some help from Terry Mahaffey at Microsoft on the CUDA build. On May 30, Dmitry completed the builds, and the Folding@home team accepted and internally published the cores. On May 31, the Folding@home team established an internal project for testing the new A8 version of the Core with real Work Units. Testing is currently under way. On June 7, the Folding@home team requested more variations of CUDA/Intel builds, and Dmitry provided them. Through the month of June, the extended teams identified and solved several problems in the F@h Core. On July 2, Dmitry solved the last remaining problem with resuming a terminated job from a checkpoint, and submitted the F@h Core version A8 0.0.4 to the Folding@home team.
By July 7, the extended team had reached a consensus that F@h Core version A8 0.0.4 was ready to ship. Dr. Bowman called a meeting for July 13 to discuss release planning. On July 8, Dmitry and Joseph got the Folding@home servers reconfigured to give real work units to Arm devices, and shortly thereafter, Dmitry got Folding@home running on a real work unit in a Neocortix Scalable Compute instance on his Samsung Galaxy S20 phone in his lab.
On July 10, Michael got Folding@home running on real work units on Neocortix Scalable Compute instances on 10 phones on our worldwide network of PhonePaycheck devices. On July 12, the first work units completed. By August 6, we had completed 3,000 work units.
On July 13, the teams from Neocortix, Arm, Folding@home and Gromacs met to plan for publication and announcements for the Linux-on-Arm version of Folding@home. We agreed that we would aim for July 28, to allow two weeks for Folding@home to do more beta testing and develop device-specific job sizing. On July 28, Neocortix launched its press release, as planned. But unfortunately, Folding@home did not publish their Linux-on-Arm version at that time. On August 10, Folding@home published a beta version of the Linux-on-Arm release.
We are continuing to support Folding@home as they prepare their public release of the Arm version, and we are working hard, preparing for very broad deployment onto Android mobile devices.
We are grateful for support and collaboration from the following companies and organizations: