This article is an informal, detailed description of our multi-organization collaboration to get protein folding Computational Biology applications running on Arm platforms, to assist with coronavirus research. The intended audience includes the participants in the project, their managers and executives who would like visibility into the project, and external observers, including journalists, who may be interested in the project.
Background
The first confirmed case of coronavirus in the United States was reported on January 20, 2020. On February 21, 2020, the Rosetta@home lab announced their work in fighting coronavirus. On February 27, 2020, Dr. Greg Bowman announced that the Folding@home project was calling for more GPU and CPU devices to assist in their coronavirus research.
On March 14, 2020, Lloyd Watts at Neocortix saw news coverage of the call for devices, here and here, and we recognized that we might be able to run Folding@home on our Scalable Compute platform, which consists of secure Debian Linux containers running on Android/Arm mobile devices.
Neocortix software architects Michael Coffey and Dmitry Moskalchuk quickly found that neither Folding@home or Rosetta@home could run on Arm devices, because no Linux-on-Arm build was provided by those projects. At about the same time, David Tischler at MiniNodes/Balena was independently investigating, and came to the same conclusion. So, to make progress, we would need a Linux-on-Arm build of Folding@home and/or Rosetta@home.
Initial Outreach
On March 15, 2020, Lloyd posted a message on foldingforum.org, asking for a Linux-on-Arm build of Folding@home, and volunteering Neocortix help to build one. And simultaneously, we tried reaching out to Dr. Vijay Pande, who was listed on the Folding@home website as the leader of the project. But that information was out-of-date; we did not realize at that time that Dr. Pande had retired from the project and handed it off to Dr. Greg Bowman.
The next day, March 16, 2020, Rex St. John at Arm responded to our posted message, offering help from the Arm Developer Community. On March 17, 2020, Lloyd, Rex, and David Tischler had a phone meeting and agreed to work together to reach the Folding@home and Rosetta@home projects, and try to get access to their source code so we could help them with a Linux-on-Arm build of their projects.
Only a few hours later, Rex had made positive contact with the Rosetta team. Brian Koepnick and David Kim at Rosetta responded with a link to the Rosetta source code repository, and guidance about how we could contribute changes to it without being official Rosetta team members. Dennis Hanson at UW CoMotion provided definitive guidance about the licensing terms.
Building Rosetta for Arm
The next step was to build Rosetta for Arm. Dmitry Moskalchuk at Neocortix took the approach of a cross-build, i.e. building an aarch64 executable on an Intel x86_64 machine. Sahaj Sarup at Linaro.org and David Tischler at MiniNodes/Balena took the approach of a native build, i.e. building an aarch64 executable on an Arm64 machine. Ed Vielmetti at Packet.com gave input on available server-class Arm64 machines to use for native build and testing.
On March 20, 2020, after several days of intense work, Dmitry completed the aarch64 build of the minirosetta executable, as would be needed to run Rosetta@home in the BOINC framework. He tested it using an Arm64 emulator running on his x86_64 machine. Dmitry provided his source code patches and instructions on how to build minirosetta and run the unit tests to the Rosetta team and the extended developer group. David Kim immediately began verifying the new patches, cross-building the aarch64 application, and preparing for testing. He advised the group that verification, testing, and integration with other parts of the broader Rosetta project would take 3-7 days, before a new Linux-on-Arm build could be published.
On March 22, 2020, Sahaj Sarup and David Tischler completed a native build of minirosetta on their Arm64 machines, based on Dmitry's cross-build.
Testing Rosetta on Arm
David Kim sent out test scripts and instructions for a functional test of minirosetta which could be run on an Arm64 machine. Sahaj ran it quickly and sent the results back to David Kim, who confirmed successful operation of aarch64 minirosetta on an Arm64 machine (AMD Seattle Arm-based SoC). Soon after, Dmitry ran the tests on his Raspberry Pi 4 and Galaxy Note 9 phone, running inside a Neocortix Scalable Compute instance secure Debian Linux container. Here is a screenshot of the output of the ab initio model produced on aarch64 from Sahaj's first run:
On March 23, 2020, David Kim verified all of the test results, and reported performance of the different devices on the tests (average runtimes in seconds per model):
Dmitry’s Raspberry Pi 4 with SD storage: 272s
Dmitry’s Galaxy Note 9: 118s
Sahaj’s AMD Seattle Arm-based SoC: 166s
David Kim's Intel i7 desktop at work: 49s
On March 26, 2020, the Rosetta team has done further integration work, and provided a new build for testing. Linaro, MiniNodes and Neocortix have all verified the new build on their platforms. The Rosetta team is continuing their work on integration and testing, toward the publication of the new Linux-on-Arm version of Rosetta@home.
Deployment of Rosetta@home on Arm
On March 31, the Rosetta team published the new Linux-on-Arm build.
Since then, the group effort has shifted to deployment onto Arm devices. As of April 16, the group has over 793 Arm devices contributing to the Rosetta project, with over 6.6M credits earned. Please see Sahaj Sarup's blog which includes a description of Arm devices in his lab that are being used for Rosetta@home. David Tischler's blog gives a thorough description of how to run Rosetta@home on Arm devices. And David has also created FoldForCovid.io which allows people to volunteer their laptop and Raspberry Pi devices for Rosetta/Covid-19 research (but not mobile devices).
On April 8, Neocortix completed its upgrades for temperature and battery monitoring, and is now ramping up deployment onto its existing network of 3,000 mobile devices, soon to be followed by up to 42,000 volunteer devices from our PhonePaycheck waiting list.
As of April 16, the broader group has over 793 Arm devices contributing to the Rosetta project, with over 6.6M total credits earned. By May 6, Neocortix had ramped up to 613 simultaneous devices, and reached the Top 1% of Rosetta contributors with 2,303,633 total credits earned and a Recent Average Credit score of 102,963. By May 28, Neocortix has earned a total of 4,315,994 credits. By June 8, the broader group has 5628 devices contributing to Rosetta, with over 102.7M total credits earned.
