Web-scale distributed eScience AI search across disconnected and heterogeneous infrastructures

T. Kelsey, M. McCaffery, L. Kotthoff

Research output: Chapter in Book/Report/Conference proceedingChapter

1 Downloads (Pure)

Abstract

We present a robust and generic framework for web-scale distributed e-Science Artificial Intelligence search. Our validation approach is to distribute constraint satisfaction problems that require perfect accuracy to 10, 12 and 15 digits. By checking solutions obtained using the framework against known results, we can ensure that no errors, duplications nor omissions are introduced. Unlike other approaches, we do not require dedicated machines, homogeneous infrastructure or the ability to communicate between nodes. We give special consideration to the robustness of the framework, minimising the loss of effort even after a total loss of infrastructure, and allowing easy verification of every step of the distribution process. The unique challenges our framework tackles are related to the combinatorial explosion of the space that contains the possible solutions, and the robustness of long-running computations. Not only is the time required to finish the computations unknown, but also the resource requirements may change during the course of the computation. We demonstrate the applicability of our framework by using it to solve challenging problems using two separate large-scale distribution paradigms. The results show that our approach scales to e-Science computations of a size that would have been impossible to tackle just a decade ago.
Original languageEnglish
Title of host publicationProceedings - 2014 IEEE 10th International Conference on eScience, eScience 2014
PublisherIEEE
Pages39-46
Number of pages8
Volume1
ISBN (Print)9781479942886
DOIs
Publication statusPublished - 2 Dec 2014

Fingerprint

Dive into the research topics of 'Web-scale distributed eScience AI search across disconnected and heterogeneous infrastructures'. Together they form a unique fingerprint.

Cite this