Dispel4py: An agile framework for data-intensive eScience

Rosa Filgueira, Amrey Krause, Malcolm Atkinson, Iraklis Klampanos, Alessandro Spinuso, Susana Sanchez-Exposito

Research output: Chapter in Book/Report/Conference proceedingConference contribution

17 Citations (Scopus)


We present dispel4py a versatile data-intensive kit presented as a standard Python library. It empowers scientists to experiment and test ideas using their familiar rapid-prototyping environment. It delivers mappings to diverse computing infrastructures, including cloud technologies, HPC architectures and specialised data-intensive machines, to move seamlessly into production with large-scale data loads. The mappings are fully automated, so that the encoded data analyses and data handling are completely unchanged. The underpinning model is lightweight composition of fine-grained operations on data, coupled together by data streams that use the lowest cost technology available. These fine-grained workflows are locally interpreted during development and mapped to multiple nodes and systems such as MPI and Storm for production. We explain why such an approach is becoming more essential in order that data-driven research can innovate rapidly and exploit the growing wealth of data while adapting to current technical trends. We show how provenance management is provided to improve understanding and reproducibility, and how a registry supports consistency and sharing. Three application domains are reported and measurements on multiple infrastructures show the optimisations achieved. Finally we present the next steps to achieve scalability and performance.

Original languageEnglish
Title of host publicationProceedings - 11th IEEE International Conference on eScience, eScience 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages11
ISBN (Electronic)9781467393256
Publication statusPublished - 22 Oct 2015
Event11th IEEE International Conference on eScience, eScience 2015 - Munich, Germany
Duration: 31 Aug 20154 Sept 2015

Publication series

NameProceedings - 11th IEEE International Conference on eScience, eScience 2015


Conference11th IEEE International Conference on eScience, eScience 2015


  • Data intensive application
  • Distributed systems
  • ESciences workflows
  • Python frameworks
  • Run time analysis


Dive into the research topics of 'Dispel4py: An agile framework for data-intensive eScience'. Together they form a unique fingerprint.

Cite this