Container orchestration for scientific workflows

W Gerlach, W Tang, A Wilke, D Olson… - … conference on cloud …, 2015 - ieeexplore.ieee.org
2015 IEEE International conference on cloud engineering, 2015ieeexplore.ieee.org
Recently, Linux container technology has been gaining attention as it promises to transform
the way software is developed and deployed. The portability and ease of deployment makes
Linux containers an ideal technology to be used in scientific workflow platforms. AWE/Shock
is a scalable data analysis platform designed to execute data intensive scientific workflows.
Recently we introduced Skyport, an extension to AWE/Shock, that uses Docker container
technology to orchestrate and automate the deployment of individual workflow tasks onto the …
Recently, Linux container technology has been gaining attention as it promises to transform the way software is developed and deployed. The portability and ease of deployment makes Linux containers an ideal technology to be used in scientific workflow platforms. AWE/Shock is a scalable data analysis platform designed to execute data intensive scientific workflows. Recently we introduced Skyport, an extension to AWE/Shock, that uses Docker container technology to orchestrate and automate the deployment of individual workflow tasks onto the worker machines. The installation of software in independent execution environments for each task reduces complexity and offers an elegant solution to installation problems such as library version conflicts. The systematic use of isolated execution environments for workflow tasks also offers a convenient and simple mechanism to reproduce scientific results.
ieeexplore.ieee.org