{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# What is CyVerse?\n", "\n", "CyVerse started out in 2008 as the NSF-funded 'iPlant Collaborative' project to help the changing needs of the life sciences, which have increasingly heavy computational demands (think protein folding, genetic phenotyping, etc. etc.), but have historically not had high-performance computing facilities. The project was rechristened 'CyVerse' in 2015.\n", "\n", "The two co-I's of the grant, Nirav Merchant and Eric Lyons, are life scientists here at UA and offer the course \"Applied Concepts in Cyberinfrastructure\". They had found that researchers were \"actively avoiding advanced computing for their research projects because they viewed it as inflexible, complicated, frustrating, and time-consuming.\"\n", "\n", "CyVerse offers:\n", "1. Cloud computing\n", "2. Data storage\n", "3. Support" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Why should an astronomer care about it?\n", "\n", "CyVerse cyberinfrastructure has the serendipitous effect of providing computational resources to basically anyone here at UA and >8,000 participating institutions. \n", "\n", "Possible reasons to care: \n", "1. If you have computational needs that are greater than what your office machines can provide, but less than what would usually pass for HPC jobs\n", "2. If you want to calculate something now, and don't want to put your job in a queue\n", "3. Very flexible for access remotely or from low-performance machines (basically all you need to do is 'ssh')\n", "4. This infrastructure serves as a lead-in for addressing needs that span the community, like interoperability and reproducibility.\n", "\n", "I (Eckhart) first heard about it at a Software Carpentry workshop. CyVerse made a brief appearance at a talk at the Internal Symposium in fall 2016. \n", "\n", "Using CyVerse computational resources, I performed MCMC and image-processing-related computations for three publications either published or in prep.). Using resources for which I did not require any extra human approval, it has shortened high-contrast imaging dataset reductions from >10 hours of wall time to ~30-40 minutes by distributing the jobs across several cores.\n", "\n", "Ya-Lin Wu (grad student close to graduating) and Jared Males (new research professor here) used CyVerse to do image processing involving frame-by-frame parameterization and subtraction of host star PSFs, and fake planet injection for determination of sensitivity or brightnesses. In Males' case, this shortened reduction times from four months to two days. (This was actually a project of the \"Applied Concepts in Cyberinfrastructure\" class.)\n", "\n", "See, for example,\n", "\n", "### Cloud computing:\n", "\n", " - [This article](http://www.cyverse.org/single-post/2016/03/17/In-Search-of-a-Planet) on Males et al.'s work with CyVerse. There's also a video featuring the work done by \"Jared Malice\" et al. [here](https://www.youtube.com/watch?v=8tILpvMINYU&feature=youtu.be).\n", " - 'High-Contrast Imaging in the Cloud with klipReduce and Findr' (Haug-Baltzell+ 2016)\n", " - 'An ALMA and MagAO Study of the Substellar Companion GQ Lup B' (Wu+ 2017)\n", "\n", "### Data curation:\n", "\n", "[This arXiv post](https://arxiv.org/pdf/1802.03629.pdf) from two days ago is about [Astrolabe](http://astrolabe.arizona.edu), an outgrowth of CyVerse to curate astronomical data and prevent it from \"going dark\". (Ben Weiner is involved in this.)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Storage space with CyVerse Data Store\n", "\n", "[CyVerse Data Store](http://www.cyverse.org/data-store) gives you \"100 GB of data storage, with the ability to request more\". Large file transfers can be facilitated with multiple threads (iRODS iCommands).\n", "\n", "# Data curation with Astrolabe\n", "\n", "Follow the instructions [here](http://astrolabe.arizona.edu/index.php/contribute/). (One option is to upload data to Astrolabe from Data Store.)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Virtual machines with CyVerse Atmosphere\n", "\n", "[CyVerse Atmosphere](http://www.cyverse.org/atmosphere) is CyVerse's cloud-computing arm. With it, you can slap together a remote machine with a wide choice of operating system and hardware. \n", "\n", "Computational work is measured in 'AU' (allocation units?) which you use up and request for more. In my experience, when I make a new request, it is granted somewhere between a couple hours later up to the next weekday morning." ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "# Put together an instance\n", "\n", "1. First, obviously, open a CyVerse account.\n", "2. Log in to CyVerse Atmosphere, and go to Projects -> Create New Project. A 'Project' can encompass multiple instances and volumes.\n", "3. Once you're in the new Project, go to New -> Instance. An 'Instance' is a fusion of the hardware, operating system, and software of your virtual machine.\n", "4. Choose the operating system, number of cores, hard drive size, and RAM that you want.\n", "5. Deploy it! (This takes some minutes to build and network.)\n", "6. Use the IP address to ssh into the instance, or use the 'Web Shell' option. (The IP address changes each time you suspend and resume an instance.)\n", "\n", "For lots of details, do not fail to consult the CyVerse Wiki." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Configure\n", "\n", "Once you ssh in for the first time, run the command\n", "\n", "```\n", "ezj\n", "```\n", "\n", "The CyVerse wizards added this to install a lot of the popular packages you might need. (I recommend also installing Anaconda as an all-in-one installer and an enironment manager. The latter will enable you to switch on-the-fly between versions of Python. For installation instructions, go [here](https://conda.io/docs/user-guide/install/index.html), and see info on managing environments [here](https://conda.io/docs/user-guide/tasks/manage-environments.html#activating-an-environment).)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Day-to-day use\n", "\n", "There are a list of 'Action' buttons on the right-hand side of the instance page. \n", "\n", "'Report' -> Report a problem\n", "\n", "'Image' -> Take a snapshot of the instance with the things you've installed on it, for storing and re-use later.\n", "\n", "'Suspend' -> Pause an instance if you're not using it (If you forget to do this, your AU units will slowly be eaten away!)\n", "\n", "'Shelve' -> Longer-term storage, without hogging resources (I am told it also avoids the risk of the instance being deleted due to disuse)\n", "\n", "'Stop' -> Everything powered off, but uses up AU units \n", "\n", "'Reboot' -> sometimes useful if it gets into a weird state\n", "\n", ". . . and to request more resources, go to the project page and click the button with the circled up arrow." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Add an instance or a volume\n", "\n", "On the project page, click on 'New' and select what you want. If you have created a new volume, you need to 'attach' it (button in the upper right of the project page) to the instance, using the same provider location." ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "# Let's compare run speeds of a for-loop: local and remote dewarping of 8.4 Mb FITS frames" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# import and define stuff\n", "\n", "import numpy as np\n", "from astrom_lmircam_soln import *\n", "from astrom_lmircam_soln import polywarp\n", "from astrom_lmircam_soln import dewarp\n", "from astropy.io import fits\n", "import matplotlib.pyplot as plt\n", "import os\n", "import time\n", "from multiprocessing import Pool\n", "\n", "#####################################################################\n", "# SET THE DEWARP COEFFICIENTS\n", "\n", "# the below coefficients are relevant for LMIRCam 2048x2048 readouts after modifications in summer 2016\n", "# the below were taken in 2017B (DX ONLY)\n", "Kx = [[ -1.34669677e+01, 2.25398365e-02, -7.39846082e-06, -8.00559920e-11],\n", " [ 1.03267422e+00, -1.10283816e-05, 5.30280579e-09, -1.18715846e-12],\n", " [ -2.60199694e-05, -3.04570646e-09, 1.12558669e-12, 1.40993647e-15],\n", " [ 8.14712290e-09, 9.36542070e-13, -4.20847687e-16, -3.46570596e-19]]\n", "Ky = [[ 1.43440109e+01, 9.90752231e-01, -3.52171557e-06, 7.17391873e-09],\n", " [ -2.43926351e-02, -1.76691374e-05, 5.69247088e-09, -2.86064608e-12],\n", " [ 1.06635297e-05, 8.63408955e-09, -2.66504801e-12, 1.47775242e-15],\n", " [ -1.10183664e-10, -1.67574602e-13, 2.66154718e-16, -1.13635710e-19]]\n", "\n", "\n", "#####################################################################\n", "# set file paths\n", "\n", "dirTreeStem = ('')\n", "\n", "retrievalPiece = ('retrieve/')\n", "depositPiece = ('deposit/')\n", "fileNameStem = ('lm_171002_')\n", "\n", "#####################################################################\n", "\n", "# DEWARP \n", "\n", "# map the coordinates that define the entire image plane (2048x2048)\n", "dewarp_coords = dewarp.make_dewarp_coordinates((2048,2048), np.array(Kx).T, np.array(Ky).T) # transposed due to a coefficient definition change btwn Python and IDL\n", "\n", "def dewarp_frame_multiproc(frameNum, extra_dim=True):\n", "\n", " start_time = time.time()\n", "\n", " print('Dewarping frame '+str(frameNum)+'...')\n", "\n", " # grab the pre-dewarp image and header\n", " image, header = fits.getdata(dirTreeStem+\n", " retrievalPiece+\n", " fileNameStem+\n", " str(\"{:0>5d}\".format(frameNum))+\n", " '.fits',\n", " 0,\n", " header=True)\n", "\n", " # dewarp the image\n", " dewarped = dewarp.dewarp_with_precomputed_coords(image, dewarp_coords, order=3)\n", "\n", " # write out\n", " dewarped = np.squeeze(dewarped) # remove dimensions of size 1\n", " if extra_dim: # the LEECH pipeline still requires a singleton dimension\n", " dewarped = dewarped[None,:,:]\n", "\n", " hdu = fits.PrimaryHDU(dewarped, header=header)\n", " hdulist = fits.HDUList([hdu])\n", " hdulist.writeto(dirTreeStem+depositPiece+fileNameStem+str(\"{:0>5d}\".format(frameNum))+'.fits',\n", " overwrite=True)\n", "\n", " elapsed_time = time.time() - start_time\n", " print(elapsed_time)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## Run on 1 local core:\n", "\n", "# I have to run the function inside a for-loop, because with foresight I designed the function to be mapped to N CPUs\n", "\n", "for frame in range(1,100):\n", " dewarp_frame_multiproc(frame)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## Run on all (2) local cores:\n", "\n", "pool = Pool(2) # initialize Pool object for 2 streams of worker processes\n", "channel_eq = pool.map(dewarp_frame_multiproc,range(1,100)) # map the processes and run" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## Run on 16 remote cores of CyVerse instance:\n", "\n", "## pool = Pool(16)\n", "## channel_eq = pool.map(dewarp_frame_multiproc,range(1,100))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# In-person support\n", "\n", "PhTea: Tues. 8-10 a.m. at the Nucleus Café in the Keating building\n", "\n", "[Hacky Hour](https://www.meetup.com/ResBazAZ/events/247026059/): Thurs. 4-7 PM, currently at Gentle Ben's" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Scaling up: what other resources are there?\n", "\n", "Open Science Grid with Pegasus\n", "\n", "Google Cloud Platform\n", "\n", "Exsede\n", "\n", "Jetstream\n", "\n", "Docker and DockerHub --> stay tuned for C.K.'s spiel\n", "\n", "Singularity and SingularityHub (note OSG and Exsede use Singularity)\n", "\n", "Learn about cloud computing with containers at Container Camp" ] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.6" } }, "nbformat": 4, "nbformat_minor": 1 }