Using jupyter notebooks on Blue Waters …

Using jupyter notebooks on Blue Waters

Roland Haas (NCSA / University of Illinois) Email: rhaas@ncsa.illinois.edu

Jupyter notebooks

2/18

interactive, browser based interface to Python

freely mix

? python commands ? shell commands ? graphics ? markdown

explore data and debug scripts interactively

? more memory (128GB) than your laptop ? no need to copy data off Blue Waters

export result as Python script share notebook with collaborators

Three ways to use jupyter on Blue Waters

Jupyter starts a notebook server that one connects to using a browser

on login node

? good for quick, inexpensive tasks ? simple to set up

on a single compute node

? for tasks longer than what the process killer allows

? for tasks using large amounts of resources (IO, memory, CPU)

? somewhat harder to set up

on multiple compute nodes

? large scale python applications using MPI, e.g. yt

? task farms with many jobs ? workflow systems like parsl ? complex to set up

laptop

ssh

Notebook server

TCP

Python kernel

Blue Waters

notebook file

3/18

Common setup

jupyter notebook server is provided by bwpy module module load bwpy

notebook server listens for connections using TCP

? anyone on Blue Waters can connect ? provides full access to your account ? protect access using a password

password is set in

jupyter_notebook_config.py

file in ~/.jupyter directory

4/18

from socket import * ips = gethostbyname_ex(gethostname())[2] for ip in ips:

if(ip.startswith("10.")): internal_ip = ip break

from IPython.lib import passwd c.NotebookApp.password = passwd("$passwd$") c.NotebookApp.open_browser = False c.NotebookApp.ip = internal_ip

config file available on portal

thonnotebooks notebook server listens on internal network interface only default password is $passwd$

Jupyter on login nodes

bw$ module load bwpy bw$ jupyter notebook The Jupyter Notebook is running at: laptop% ssh -L 8888:10.0.0.147:8981 bw.ncsa.illinois.edu laptop% open

Notebook server accessible Blue Waters wide but not from the public internet

? jupyter outputs connection information to stdout on startup ? use second ssh connection to any login node to forward ports ? on Windows ssh -L 127.0.0.1:8888:10.0.0.147:8981 bw may be required

5/18

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download