Neuron Jupyter User Guideline

A. Overview of Jupyter

1. JupyterHub

JupyterHub is an open-source software that enables users to use JupyterLab and Jupyter Notebook in a multi-user environment.
In addition to supporting various environments (JupyterLab, Notebook, RStudio, Nteract, etc.), JupyterHub can be flexibly integrated with authentication servers (OAuth, LDAP, GitHub, etc.) and batch schedulers (PBSPro, Slurm, LSF, etc.).
In addition, JupyterHub can be easily integrated with Kubernetes, a container management platform; hence, it can also be easily linked with a container-based cloud environment.

※ The Neuron-based JupyterHub has added Unit #5 Bright LDAP and OTP authentication functions and integrates with a Slurm batch scheduler to allocate resources that run Jupyter. Currently, it provides Jupyter Notebook by default, including JupyterLab.

2. Jupyter Notebook

Jupyter Notebook is a web-based open-source application that programmers use for document creation, code creation and execution, data visualization using mathematical libraries, statistical modeling, and machine learning/deep learning programming.
Jupyter Notebook supports approximately 40 programming languages, including Python, R, Julia, and Scala.
Codes written in programming languages can be converted into various file types, such as HTML, image, video, and LaTeX.
Jupyter Notebook can be integrated with various tools and libraries, such as Apache Spark, Pandas, Scikit-learn, ggplot2, and TensorFlow.

3. JupyterLab

JupyterLab comprises extensible modules by introducing functions to the Jupyter Notebook interface for user convenience.
Unlike Jupyter Notebook, JupyterLab provides several documents or different functions on a single work screen using Tabs and Splitters.

B. Run Jupyter through a Script

1. Overview

$ ssh [userID]@neuron.ksc.re.kr

(required) Run the Jupyter-specific package installation script

$ sh /apps/jupyter/kisti_conda_jupyter.sh

(optional) Run other package auto-install scripts as needed, or run them yourself

* notebook conda env activate : $ conda activate notebook
(notebook) $ sh  /apps/jupyter/kisti_conda_tensorflow.sh  
(notebook) $ sh  /apps/jupyter/kisti_conda_pytorch.sh
(notebook) $ sh /apps/jupyter/kisti_conda_etc.sh

※You can use JupyterLab/Notebook by accessing the web page (C. How to use JupyterLab) immediately after the configuration is completed.

2. Run the script

Connect to the login node (neuron.ksc.re.kr) with a terminal and execute the following script /apps/jupyter/kisti_conda_jupyter.sh.
The script creates a notebook Conda environment in the /scratch/[userID]/.conda/envs directory and automatically installs the jupyterhub, jupyterLab, and notebook packages, as well as cudatoolkit=11.6 and cudnn for multi-GPU environments.

※ This file needs to be run just once. You can then immediately connect to the webpage and use JupyterLab and Jupyter Notebook. ※ Executable files can be run from the shared directory using the /apps/jupyter/kisti_conda_jupyter.sh script. ※ User ID a1113a01 was adopted in the following test.

[a1113a01@glogin02 ~]$ sh /apps/jupyter/kisti_conda_jupyter.sh
 ... ...
 modified /home01/a1113a01/.bashrc
 ...prepare conda environment for jupyter user.
 Exporting CONDA ENVS and PKGS PATH to bash File.
 Downloading and Extracting Packages
 #################################################################### | 100%
 #################################################################### | 100%
 #################################################################### | 100%
 #################################################################### | 100%
 Preparing transaction: done
 Verifying transaction: done
 Executing transaction: / WARNING conda.core.prefix_data:_load_single_record(167):
 Ignoring malformed prefix record at:
 /scratch/a1113a01/.conda/envs/notebook/conda-meta/001.pycurl-7.43.0.5-py37h16ce93b_0.json
 done

Restart the shell and turn off the function that automatically activates the base environment. (Run only once)

[a1113a01@glogin01 ~]$ source ~/.bashrc
 (base) [a1113a01@glogin01 ~]$ conda config --set auto_activate_base false
 (base) [a1113a01@glogin01 ~]$ conda deactivate

※ Set the function that automatically activates the base environment to false prevent the automatic activation of the base environment next time. (If the base environment is not activated, run the conda activate notebook command right after source ~/.bashrc.)

Activate the conda notebook using the following command.

[a1113a01@glogin01 ~]$ conda activate notebook
 (notebook) [a1113a01@glogin01 ~]$

Users who want to employ Tensorboard and IPyParallel must install the tensorboard and ipyparallel extension packages for Jupyter Notebook/JupyterLab before connecting to the webpage. These packages can be installed by running the automated installation script /apps/jupyter/kisti_conda_plugins.sh provided by KISTI (this takes approximately 5 min).

※ Caution: The action must be executed from a notebook user environment.

(notebook) 757% [a1113a01@glogin01 ~]$ sh /apps/jupyter/kisti_conda_plugins.sh
 Installing /scratch/a1113a01/.conda/envs/notebook/lib/python3.7/site-packages/ipyparallel/nbextension/static -> ipyparallel
 Up to date: /scratch/a1113a01/.conda/envs/notebook/share/jupyter/nbextensions/ipyparallel/clusterlist.css
 Up to date: /scratch/a1113a01/.conda/envs/notebook/share/jupyter/nbextensions/ipyparallel/clusterlist.js
 Up to date: /scratch/a1113a01/.conda/envs/notebook/share/jupyter/nbextensions/ipyparallel/main.js
 - Validating: OK
To initialize this nbextension in the browser every time the notebook (or other app) loads:
 
 jupyter nbextension enable ipyparallel --py --sys-prefix
 
 Enabling tree extension ipyparallel/main...
 - Validating: OK
 Enabling: ipyparallel.nbextension
 - Writing config: /scratch/a1113a01/.conda/envs/notebook/etc/jupyter
 - Validating...
 ipyparallel.nbextension OK
 [ProfileCreate] Generating default config file: '/home01/a1113a01/.ipython/profile_slurm/ipython_config.py'
 [ProfileCreate] Generating default config file: '/home01/a1113a01/.ipython/profile_slurm/ipython_kernel_config.py'
 [ProfileCreate] Generating default config file: '/home01/a1113a01/.ipython/profile_slurm/ipcontroller_config.py'
 [ProfileCreate] Generating default config file: '/home01/a1113a01/.ipython/profile_slurm/ipengine_config.py'
 [ProfileCreate] Generating default config file: '/home01/a1113a01/.ipython/profile_slurm/ipcluster_config.py'

※ Henceforth, users can directly access the webpage and use Jupyter Notebook. (The tasks up to this point only need to be run once.)

3. Connect to the JupyterHub webpage

Connect to https://jupyter.ksc.re.kr and enter your Neuron account username, OTP, and password.

You can check the resource usage status on the main page and click the Refresh button to update the resource usage status.

4. Select a Queue and run Jupyter

Click the Refresh button to check the resource status before running Jupyter
- Total CPUs: Total number of CPU cores
- Alloc CPUs: Number of CPU cores in use
- Total GPUs: Total number of GPUs
- Alloc GPUs: Number of GPUs in use
- Node Usage: Node utilisation
View Queue information
- jupyter queue (free): For environment installation, preprocessing, and debugging
- other queues (paid): For running and visualising models for deep learning/machine learning, etc.

※ jupyter queue can run up to 20 Jupyter Lab/Notebooks (10 per node) with 2 nodes (multiple users share the node's CPU+GPU [v100]).

※ GPUs in jupyter queue are shared resources, so the allocation information (Alloc GPUs) is displayed as 0.

※ When selecting a queue, please check whether there are any waiting tasks and run your notebook.

※ When selecting a queue other than jupyter queue, please select *:gpu=1 to run jupyter as much as possible.

※ The billing policy follows the existing Neuron system billing policy, and the information can be found on the National Supercomputing website's billing page (https://www.ksc.re.kr/jwjg/gjbg/ygan).

※ Select the appropriate queue from the Job queue and click the Submit button to run Jupyter Notebook (other queues can also be run, but they will be charged, see Neuron Charging Information on the KSC website).

The following screen is displayed for a few seconds, and resources are allocated.

The https://jupyter.ksc.re.kr/user/a1113a01/tree Jupyter Notebook screen is displayed by default.

C. How to Use Jupyter Notebook

1. Jupyter work environment

Jupyter environment directory: /scratch/[user ID]/.conda/envs/notebook
Directory where logs are saved: /scratch/[user ID]/log/job ID.log
Directory where job files are saved: /scratch/[user ID]/workspace/

※ Users install machine learning/deep learning libraries they need in the .../notebook conda environment. Therefore, these libraries are installed in the /scratch/user ID/ directory, which has a large default quota. (Log files that are generated after running Jupyter are also saved in the /scratch/user ID directory.) ※ Codes written by the user are saved in the /scratch/user ID/ directory. ※ Information on exporting and importing a conda environment for backing up a conda environment can be found in the software guidelines on the KISTI homepage.

Launch Terminal, and click the Terminal icon in the Launcher tab.

※ If you don't see the Launcher tab, click the + icon in the Menu Bars.

Launch Tensorboard, click Menu Bars->+icon->Launcher->Tensorboard.

2. End a running session

Click the Session tab in the Left Side Bar and click the Shut Down button to close the running Terminal Sessions or Kernel Sessions.

※ If you exit the JupyterHub webpage without closing the session, it will remain on the next Jupyter run. (You will not be charged)

3. Close Jupyter

File -> Hub Control Panel -> Stop My Server

※ All running Jupyter and sessions are automatically closed when you log out of the homepage.

D. Execute the Machine Learning/Deep Learning Example Code

1. Install libraries that are necessary to execute the example code

Click Terminal in Launcher to install the libraries required for machine learning/deep learning.

Run the conda activate notebook command from the terminal environment to activate the notebook environment and install libraries that are required for the notebook environment.

※ The libraries must be installed in the notebook conda environment to apply them to the Jupyter web screen.

Install the machine learning/deep learning-related libraries requested by the user in the notebook environment (This example shows the process of installing commonly used libraries, such as machine learning framework Keras, pandas, NumPy, and SciPy, using conda. It takes approximately 7 min to install the necessary libraries for this example.)

772% [a1113a01@gpu06 workspace]$ conda activate notebook
(notebook) 773% [a1113a01@gpu06 workspace]$ conda install -c conda-forge theano keras graphviz numpy scipy scikit-learn matplotlib pandas pydot -y
or
## You can replace it with the installation script below and run it
(notebook) 773% [a1113a01@gpu06 workspace]$ /apps/jupyter/kisti_conda_etc.sh

2. Create and execute an example code

Click the example file iris_ex.ipynb in your working directory.
In the Edit/Run Programme window, press Shift+Enter to execute the example code.

※ You can ignore the warnings during the execution process, and the warning message will not be displayed when you run the same code again. (The warnings are grammatical suggestions based on the version of the code)

Graph output using the matplotlib library

3. Run Tensorboard

Click tfboard_ex.ipynb in Menu Bar -> Files.
Run the code with Shifter+Enter (takes about 1 minute)

Run Tensorboard.

※ Log data is stored in the logs folder.

TensorBoard -> Scalars

TensorBoard -> Graphs

TensorBoard -> Distributions

TensorBoard -> Histograms

4. Create a new Launcher and write the Python code

You can write new Python code by clicking the New -> Python 3 menu as shown below.

This will launch the new Jupyter Notebook Launcher, which can use the Python 3 kernel.

E. How to Stop Jupyter

1. Terminate running sessions

As illustrated in the following figure, click the Shutdown button in the Running tab to terminate the running Terminal, Kernel, or Tensorboard sessions.

※ If you close the JupyterHub webpage without terminating the sessions, these sessions will still remain the next time you run the Jupyter. (Fees are not charged in the meantime)

2. Close Jupyter

(JupyterLab) When you are finished using Jupyter, you must shut down Jupyter to return the resources.
You can do this by clicking Hub Control Panel from the File menu, coming to the Home page, and clicking Stop My Server.

※ When you log out of the homepage, all running Jupyter and sessions are automatically closed.

F. Jupyter Environment Initialization Method

※ If you install packages with pip in the conda virtual environment notebook, and a version conflict occurs with the packages that were previously installed using conda install, such that the Jupyter notebook fails to execute, use the following command to initialize the environment.

Run the /apps/jupyter/reset_env.sh script from the login node terminal.
After running this script, all packages installed in the notebook virtual environment, which were created in the /scratch/[user ID]/.conda/envs directory, are deleted, and the basic packages for running Jupyter for the first time are reinstalled.
The data in the /scratch/[user ID]/workspace/ directory are preserved.

[a1113a01@glogin02 ~]$ sh /apps/jupyter/reset_env.sh
Remove all packages in environment /scratch/a1113a01/.conda/envs/notebook:
Preparing transaction: done
Verifying transaction: done
Executing transaction: / WARNING conda.core.prefix_data:_load_single_record(167): Ignoring malformed prefix record at: /scratch/acnta03/.conda/envs/notebook/conda-meta/001.pysocks-1.7.1-py37hc8dfbb8_1.json
done

One more step left, please execute these commands !!!!!!!
1. conda activate notebook
2. sh /apps/jupyter/kisti_conda_plugins.sh

[a1113a01@glogin02 ~]$ conda activate notebook
(notebook) [a1113a01@glogin02 ~]$ sh /apps/jupyter/kisti_conda_plugins.sh