Asynchronous modeling workflows in CyberWater with on-demand HPC/Cloud access

• 全文链接
• 请求原文

作   者：

Ranran Chen;  Feng Li;  Daniel Luna;  Isuru Ranawaka;  Fengguang Song;  Sudhakar Pamidighantam;  Xu Liang;  Yao Liang; 

作者机构：

2709 E. 10th Street;  728 Benedum Hall;  Pittsburgh;  Department of Civil and Environmental Engineering;  PA;  Department of Computer and Information Science;  Georgia Institute of Technology;  Atlanta;  SL 280;  United States||Center for AI in Science and Engineering (ARTISAN);  47408;  University of Pittsburgh;  1283B CODA Building;  GA 30308;  756 W Peachtree St NW;  Indianapolis;  46202;  IN;  Cyberinfrastructure Integration Research Center;  15261;  Bloomington;  Indiana University-Purdue University Indianapolis;  3700 O'Hara Street;  Pervasive Technology Institute Indiana University;  United Stales;  United States;  723 W. Michigan Street; 

关键词：

Hydrologic modeling;  Site recommendation;  On-demand HPC access;  Asynchronous workflow control;  Scientific workflow; 

期刊名称：

Future generations computer systems: FGCS

i s s n：

0167-739X

年卷期：

2024 年 159 卷 Oct. 期

页   码：

307-322

页   码：

摘   要：

Workflow management systems (WfMSs) are commonly used to organize/automate sequences of tasks as workflows to accelerate scientific discoveries. During complex workflow modeling, a local interactive workflow environment is desirable, as users usually rely on their rich, local environments for fast prototyping and refinements before they consider using more powerful computing resources. However, existing WfMSs do not simultaneously support local interactive workflow environments and HPC resources. In this paper, we present a mechanism for on-demand access to remote HPC resources from desktop/laptop-based workflow management software to compose, monitor, and analyze scientific workflows in the CyberWater project. CyberWater is an open-data and open-modeling software framework for environmental and water communities. In this work, we extend the open-model, open-data design of CyberWater with on-demand HPC accessing capacity. In particular, we design and implement the LaunchAgent and JobManager, which can be integrated into a local desktop/Laptop environment to allow on-demand usage of remote HPC resources for computational modeling workflows effectively and efficiently. LaunchAgent manages user authentication to remote resources, prepares computation-intensive or data-intensive tasks as batch jobs, submits jobs to remote resources, and monitors quality of services for users. LaunchAgent interacts seamlessly with other components in CyberWater, providing advantages of user-friendly feature-rich desktop software experience and increased computing power through on-demand HPC/Cloud usage. In our evaluations, we demonstrate how a hydrological modeling workflow that consists of both local and remote tasks can be constructed and show that our new on-demand HPC/Cloud usage helps speed up hydrology workflows while allowing asynchronous HPC/Cloud access from workflows using a desktop graphical user interface.