A community roadmap for scientific workflows research and development

Rafael Ferreira da Silva; Henri Casanova; Kyle Chard; Ilkay Altintas; Rosa M. Badia; Bartosz Balis; Tainã Coleman; Frederik Coppens; Frank Di Natale; Bjoern Enders; Thomas Fahringer; Rosa Filgueira; Grigori Fursin; Daniel Garijo; Carole A. Goble; Dorran Howell; Shantenu Jha; Daniel S. Katz; Daniel Laney; Ulf Leser; Maciej Malawski; Kshitij Mehta; Loïc Pottier; Jonathan Ozik; J. Luc Peterson; Lavanya Ramakrishnan; Stian Soiland-Reyes; Douglas Thain; Matthew Wolf

doi:10.1109/WORKS54523.2021.00016

A community roadmap for scientific workflows research and development

Rafael Ferreira da Silva, Henri Casanova, Kyle Chard, Ilkay Altintas, Rosa M. Badia, Bartosz Balis, Tainã Coleman, Frederik Coppens, Frank Di Natale, Bjoern Enders, Thomas Fahringer, Rosa Filgueira, Grigori Fursin, Daniel Garijo, Carole A. Goble, Dorran Howell, Shantenu Jha, Daniel S. Katz, Daniel Laney, Ulf LeserMaciej Malawski, Kshitij Mehta, Loïc Pottier, Jonathan Ozik, J. Luc Peterson, Lavanya Ramakrishnan, Stian Soiland-Reyes, Douglas Thain, Matthew Wolf

School of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

The landscape of workflow systems for scientific applications is notoriously convoluted with hundreds of seemingly equivalent workflow systems, many isolated research claims, and a steep learning curve. To address some of these challenges and lay the groundwork for transforming workflows research and development, the WorkflowsRI and ExaWorks projects partnered to bring the international workflows community together. This paper reports on discussions and findings from two virtual “Workflows Community Summits” (January and April, 2021). The overarching goals of these workshops were to develop a view of the state of the art, identify crucial research challenges in the workflows community, articulate a vision for potential community efforts, and discuss technical approaches for realizing this vision. To this end, participants identified six broad themes: FAIR computational workflows; AI workflows; exascale challenges; APIs, interoperability, reuse, and standards; training and education; and building a workflows community. We summarize discussions and recommendations for each of these themes.

Original language	English
Title of host publication	Proceedings of WORKS21
Subtitle of host publication	16th Workshop on Workflows in Support of Large-scale Science
Editors	Rafael Ferreira da Silva, Rosa Filgueira
Place of Publication	Piscataway, NJ
Publisher	IEEE
Pages	81-90
Number of pages	10
ISBN (Electronic)	9781665411363
ISBN (Print)	9781665411370
DOIs	https://doi.org/10.1109/WORKS54523.2021.00016
Publication status	Published - 28 Dec 2021
Event	The International Conference for High Performance Computing, Networking, Storage and Analysis - America's Center, St. Louis, United States Duration: 14 Oct 2021 → 19 Oct 2021 http://sc21.supercomputing.org

Conference

Conference	The International Conference for High Performance Computing, Networking, Storage and Analysis
Abbreviated title	SC21
Country/Territory	United States
City	St. Louis
Period	14/10/21 → 19/10/21
Internet address	http://sc21.supercomputing.org

Keywords

Interoperability
Scientific workflows
Community roadmap
Data management
AI workflows
Exascale computing

Access to Document

10.1109/WORKS54523.2021.00016

https://arxiv.org/abs/2110.02168Licence: CC BY

Cite this

Silva, R. F. D., Casanova, H., Chard, K., Altintas, I., Badia, R. M., Balis, B., Coleman, T., Coppens, F., Natale, F. D., Enders, B., Fahringer, T., Filgueira, R., Fursin, G., Garijo, D., Goble, C. A., Howell, D., Jha, S., Katz, D. S., Laney, D., ... Wolf, M. (2021). A community roadmap for scientific workflows research and development. In R. F. da Silva, & R. Filgueira (Eds.), Proceedings of WORKS21: 16th Workshop on Workflows in Support of Large-scale Science (pp. 81-90). IEEE. https://doi.org/10.1109/WORKS54523.2021.00016

@inproceedings{b8bec28694f1421ebb789ea2a38f52ea,

title = "A community roadmap for scientific workflows research and development",

abstract = "The landscape of workflow systems for scientific applications is notoriously convoluted with hundreds of seemingly equivalent workflow systems, many isolated research claims, and a steep learning curve. To address some of these challenges and lay the groundwork for transforming workflows research and development, the WorkflowsRI and ExaWorks projects partnered to bring the international workflows community together. This paper reports on discussions and findings from two virtual “Workflows Community Summits” (January and April, 2021). The overarching goals of these workshops were to develop a view of the state of the art, identify crucial research challenges in the workflows community, articulate a vision for potential community efforts, and discuss technical approaches for realizing this vision. To this end, participants identified six broad themes: FAIR computational workflows; AI workflows; exascale challenges; APIs, interoperability, reuse, and standards; training and education; and building a workflows community. We summarize discussions and recommendations for each of these themes.",

keywords = "Interoperability, Scientific workflows, Community roadmap, Data management, AI workflows, Exascale computing",

author = "Silva, {Rafael Ferreira da} and Henri Casanova and Kyle Chard and Ilkay Altintas and Badia, {Rosa M.} and Bartosz Balis and Tain{\~a} Coleman and Frederik Coppens and Natale, {Frank Di} and Bjoern Enders and Thomas Fahringer and Rosa Filgueira and Grigori Fursin and Daniel Garijo and Goble, {Carole A.} and Dorran Howell and Shantenu Jha and Katz, {Daniel S.} and Daniel Laney and Ulf Leser and Maciej Malawski and Kshitij Mehta and Lo{\"i}c Pottier and Jonathan Ozik and Peterson, {J. Luc} and Lavanya Ramakrishnan and Stian Soiland-Reyes and Douglas Thain and Matthew Wolf",

note = "Funding: This work was funded by NSF awards #2016610, #2016682, and #2016619, and by the Exascale Computing Project (17-SC- 20-SC), a collaborative effort of the U.S. Department of Energy (DOE) Office of Science and the National Nuclear Security Administration (NNSA). This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. DOE under Contract #DE-AC05-00OR22725. CG and SSR acknowledge funding from European Commission{\textquoteright}s contracts BioExcel- 2 (H2020-INFRAEDI-2018-1 823830), EOSC-Life (H2020-INFRAEOSC- 2018-2 824087), IBISBA 1.0 (H2020-INFRAIA-2017-1-two-stage 730976), PREP-IBISBA (H2020-INFRADEV-2019-2 871118), SyntheSys+ (H2020- INFRAIA-2018-1 823827). UL acknowledges funding from the German Research Council for CRC 1404 FONDA. FC is supported by the Research Foundation-Flanders (FWO, I002819N) and by the European Union{\textquoteright}s Horizon 2020 research and innovation programme under grant #824087 (EOSC-Life). BSC authors acknowledge EuroHPC JU under contract 955558 (eFlows4HPCproject). We thank all participants of the Workflows Community Summits, held in January 2021 and April 2021.; The International Conference for High Performance Computing, Networking, Storage and Analysis, SC21 ; Conference date: 14-10-2021 Through 19-10-2021",

year = "2021",

month = dec,

day = "28",

doi = "10.1109/WORKS54523.2021.00016",

language = "English",

isbn = "9781665411370",

pages = "81--90",

editor = "{da Silva}, {Rafael Ferreira} and Rosa Filgueira",

booktitle = "Proceedings of WORKS21",

publisher = "IEEE",

url = "http://sc21.supercomputing.org",

}

Silva, RFD, Casanova, H, Chard, K, Altintas, I, Badia, RM, Balis, B, Coleman, T, Coppens, F, Natale, FD, Enders, B, Fahringer, T, Filgueira, R, Fursin, G, Garijo, D, Goble, CA, Howell, D, Jha, S, Katz, DS, Laney, D, Leser, U, Malawski, M, Mehta, K, Pottier, L, Ozik, J, Peterson, JL, Ramakrishnan, L, Soiland-Reyes, S, Thain, D & Wolf, M 2021, A community roadmap for scientific workflows research and development. in RF da Silva & R Filgueira (eds), Proceedings of WORKS21: 16th Workshop on Workflows in Support of Large-scale Science. IEEE, Piscataway, NJ, pp. 81-90, The International Conference for High Performance Computing, Networking, Storage and Analysis, St. Louis, Missouri, United States, 14/10/21. https://doi.org/10.1109/WORKS54523.2021.00016

A community roadmap for scientific workflows research and development. / Silva, Rafael Ferreira da; Casanova, Henri; Chard, Kyle et al.
Proceedings of WORKS21: 16th Workshop on Workflows in Support of Large-scale Science. ed. / Rafael Ferreira da Silva; Rosa Filgueira. Piscataway, NJ: IEEE, 2021. p. 81-90.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A community roadmap for scientific workflows research and development

AU - Silva, Rafael Ferreira da

AU - Casanova, Henri

AU - Chard, Kyle

AU - Altintas, Ilkay

AU - Badia, Rosa M.

AU - Balis, Bartosz

AU - Coleman, Tainã

AU - Coppens, Frederik

AU - Natale, Frank Di

AU - Enders, Bjoern

AU - Fahringer, Thomas

AU - Filgueira, Rosa

AU - Fursin, Grigori

AU - Garijo, Daniel

AU - Goble, Carole A.

AU - Howell, Dorran

AU - Jha, Shantenu

AU - Katz, Daniel S.

AU - Laney, Daniel

AU - Leser, Ulf

AU - Malawski, Maciej

AU - Mehta, Kshitij

AU - Pottier, Loïc

AU - Ozik, Jonathan

AU - Peterson, J. Luc

AU - Ramakrishnan, Lavanya

AU - Soiland-Reyes, Stian

AU - Thain, Douglas

AU - Wolf, Matthew

N1 - Funding: This work was funded by NSF awards #2016610, #2016682, and #2016619, and by the Exascale Computing Project (17-SC- 20-SC), a collaborative effort of the U.S. Department of Energy (DOE) Office of Science and the National Nuclear Security Administration (NNSA). This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. DOE under Contract #DE-AC05-00OR22725. CG and SSR acknowledge funding from European Commission’s contracts BioExcel- 2 (H2020-INFRAEDI-2018-1 823830), EOSC-Life (H2020-INFRAEOSC- 2018-2 824087), IBISBA 1.0 (H2020-INFRAIA-2017-1-two-stage 730976), PREP-IBISBA (H2020-INFRADEV-2019-2 871118), SyntheSys+ (H2020- INFRAIA-2018-1 823827). UL acknowledges funding from the German Research Council for CRC 1404 FONDA. FC is supported by the Research Foundation-Flanders (FWO, I002819N) and by the European Union’s Horizon 2020 research and innovation programme under grant #824087 (EOSC-Life). BSC authors acknowledge EuroHPC JU under contract 955558 (eFlows4HPCproject). We thank all participants of the Workflows Community Summits, held in January 2021 and April 2021.

PY - 2021/12/28

Y1 - 2021/12/28

N2 - The landscape of workflow systems for scientific applications is notoriously convoluted with hundreds of seemingly equivalent workflow systems, many isolated research claims, and a steep learning curve. To address some of these challenges and lay the groundwork for transforming workflows research and development, the WorkflowsRI and ExaWorks projects partnered to bring the international workflows community together. This paper reports on discussions and findings from two virtual “Workflows Community Summits” (January and April, 2021). The overarching goals of these workshops were to develop a view of the state of the art, identify crucial research challenges in the workflows community, articulate a vision for potential community efforts, and discuss technical approaches for realizing this vision. To this end, participants identified six broad themes: FAIR computational workflows; AI workflows; exascale challenges; APIs, interoperability, reuse, and standards; training and education; and building a workflows community. We summarize discussions and recommendations for each of these themes.

AB - The landscape of workflow systems for scientific applications is notoriously convoluted with hundreds of seemingly equivalent workflow systems, many isolated research claims, and a steep learning curve. To address some of these challenges and lay the groundwork for transforming workflows research and development, the WorkflowsRI and ExaWorks projects partnered to bring the international workflows community together. This paper reports on discussions and findings from two virtual “Workflows Community Summits” (January and April, 2021). The overarching goals of these workshops were to develop a view of the state of the art, identify crucial research challenges in the workflows community, articulate a vision for potential community efforts, and discuss technical approaches for realizing this vision. To this end, participants identified six broad themes: FAIR computational workflows; AI workflows; exascale challenges; APIs, interoperability, reuse, and standards; training and education; and building a workflows community. We summarize discussions and recommendations for each of these themes.

KW - Interoperability

KW - Scientific workflows

KW - Community roadmap

KW - Data management

KW - AI workflows

KW - Exascale computing

UR - https://doi.org/10.1109/WORKS54523.2021

U2 - 10.1109/WORKS54523.2021.00016

DO - 10.1109/WORKS54523.2021.00016

M3 - Conference contribution

SN - 9781665411370

SP - 81

EP - 90

BT - Proceedings of WORKS21

A2 - da Silva, Rafael Ferreira

A2 - Filgueira, Rosa

PB - IEEE

CY - Piscataway, NJ

T2 - The International Conference for High Performance Computing, Networking, Storage and Analysis

Y2 - 14 October 2021 through 19 October 2021

ER -

A community roadmap for scientific workflows research and development

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this