# Neuron Jupyter

## A. Overview of Jupyter

### **1. JupyterHub**

JupyterHub is open-source software that allows the use of JupyterLab/Notebook in a multi-user environment.

JupyterHub supports various environments (JupyterLab, Notebook, RStudio, Nteract, etc.) and can be seamlessly integrated with authentication servers (OAuth, LDAP, GitHub, etc.) and batch schedulers (PBSPro, Slurm, LSF, etc.).

JupyterHub easily integrates with Kubernetes, a container management platform, making it easy to connect to container-based cloud environments.

※ The Neuron-based JupyterHub has added Bright LDAP and OTP authentication features for the 5th system, integrates with the Slurm batch scheduler to allocate resources for running Jupyter, and currently provides Jupyter Notebook by default, with JupyterLab also available.



### **2. Jupyter Notebook**

Jupyter Notebook is a web-based open-source application used by programmers for document creation, code generation and execution, data visualization using mathematical libraries, statistical modeling, and machine learning/deep learning programming.

It supports over 40 programming languages, including Python, R, Julia, and Scala.

Code written in a programming language can be converted into various types, such as HTML, images, video files, and LaTeX.

It can be integrated with various tools/libraries such as Apache Spark, Pandas, Scikit-learn, ggplot2, and TensorFlow.

![](https://2000842020-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtvPPtXiAYArocipMRdp9%2Fuploads%2Fgit-blob-ddde89892fe05cbdf032ec9b18e1242ce33408e7%2FJupyter%20Notebook.png?alt=media)

### **3. JupyterLab**

JupyterLab is an extensible module that adds user-friendly features to the Jupyter Notebook interface.

Unlike Jupyter Notebook, it provides multiple documents or other features in a single workspace using tabs and splitters.

![](https://2000842020-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtvPPtXiAYArocipMRdp9%2Fuploads%2Fgit-blob-075282db770e07662d9a747378a38dd208003494%2FJupyterLab.png?alt=media)



## B. Running Jupyter through script

### 1. Overview

* Access the login node

```shell-session
$ ssh [userID]@neuron.ksc.re.kr
```

* ❍ (Required) Run the script to install Jupyter-related packages

```shell-session
$ sh /apps/jupyter/kisti_conda_jupyter.sh 
```

※ Create a notebook conda environment and install the packages jupyter notebook, jupyterlab, cudatoolkit 11.6, and cudnn.

* (Optional) Run a script to automatically install additional packages as needed, or install them manually.

※ Activate the notebook conda environment: $ conda activate notebook

#### 1) TensorFlow work environment

```
(notebook) $ sh  /apps/jupyter/kisti_conda_tensorflow.sh  
```

#### 2) PyTorch work environment

```
(notebook) $ sh  /apps/jupyter/kisti_conda_pytorch.sh
```

※ Run this only once; once the environment is set up, you can immediately access the web page (C. How to use JupyterLab) You can use JupyterLab/Notebook.

### **2. Run the script**

Access the login node (neuron.ksc.re.kr) via terminal and run the following script:\
Execute /apps/jupyter/kisti\_conda\_jupyter.sh.

Running this script will create a notebook Conda environment in the /scratch/\[user ID]/.conda/envs directory, automatically install the JupyterHub, JupyterLab, and Notebook packages, and install cudatoolkit=11.6 and cudnn necessary for a multi-GPU environment.

※ You only need to run this file once, and afterward, you can directly access and use it via the web page.

※ The executable file can be run directly from the shared directory at\
/apps/jupyter/kisti\_conda\_jupyter.sh.

※ The following test was conducted using the user ID a1113a01.

```shell-session
[a1113a01@glogin02 ~]$ sh /apps/jupyter/kisti_conda_jupyter.sh
... ...
modified /home01/a1113a01/.bashrc
...prepare conda environment for jupyter user.
Exporting CONDA ENVS and PKGS PATH to bash File.
Downloading and Extracting Packages
#################################################################### | 100%
#################################################################### | 100%
#################################################################### | 100%
#################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: / WARNING conda.core.prefix_data:_load_single_record(167):
done
```

* Restart the shell and disable automatic activation of the base environment. (Run only once)

```shell-session
[a1113a01@glogin01 ~]$ source ~/.bashrc
(base) [a1113a01@glogin01 ~]$ conda config --set auto_activate_base false
(base) [a1113a01@glogin01 ~]$ conda deactivate
```

※ Set the automatic activation of the base environment to false to prevent it from automatically activating the next time. (If the base environment is not activated, execute source \~/.bashrc followed by conda activate notebook immediately.)

* Activate the conda notebook environment using the following command.

```shell-session
[a1113a01@glogin01 ~]$ conda activate notebook
(notebook) [a1113a01@glogin01 ~]$
```

Users who wish to use TensorFlow (including TensorBoard) or PyTorch can install them by running the automated installation script provided by KISTI.

※ Caution: Make sure to run the script in the notebook user environment.

```shell-session
## Install the tensorflow-gpu, tensorboard packages, and the TensorBoard extension for jupyter_notebook and jupyter_lab
(notebook) 757% [a1113a01@glogin01 ~]$ sh /apps/jupyter/kisti_conda_tensorflow.sh (takes approximately 10 minutes)

## Install the pytorch, torchvision, and torchaudio packages
(notebook) 757% [a1113a01@glogin01 ~]$ sh /apps/jupyter/kisti_conda_pytorch.sh (takes approximately 5 minutes)
```

※ From now on, you can directly access the web and use Jupyter Notebook. (These tasks only need to be performed once.)

### **3.** Accessing the JupyterHub Web Page

Go to https://jupyter.ksc.re.kr and enter your registered Neuron account, OTP, and password.

![](https://2000842020-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtvPPtXiAYArocipMRdp9%2Fuploads%2Fgit-blob-b926069f5b63aef0a44d6c595c93dd8fbf922c3f%2FJupyterHub%20%EC%9B%B9%20%ED%8E%98%EC%9D%B4%EC%A7%80%20%EC%A0%91%EC%86%8D.png?alt=media)

On the main screen, you can check resource usage and click the Refresh button to update the resource usage status.

<figure><img src="https://2000842020-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtvPPtXiAYArocipMRdp9%2Fuploads%2FVcAV2FIkGxsriurGqc6p%2Fnoname01.png?alt=media&#x26;token=0511a65b-3824-4cf1-8e93-753d44734367" alt=""><figcaption></figcaption></figure>

### 4. Select a Queue and Run Jupyter

Before running Jupyter, click the Refresh button to check the resource status.

* Total CPUs: Total number of CPU cores
* Alloc CPUs: Number of CPU cores in use
* Total GPUs: Total number of GPUs
* Alloc GPUs: Number of GPUs in use
* Node Usage: Node utilization rate

Check the queue information

* jupyter queue (free): For environment installation, preprocessing, and debugging
* other queues (paid): For running and visualizing deep learning/machine learning models

&#x20;※ The jupyter queue currently allows up to 20 Jupyter Lab/Notebook sessions (10 per node) across 2 nodes (multiple users share the CPU+GPU \[v100] of the nodes).

※ The GPU in the jupyter queue is a shared resource, so the allocation information (Alloc GPUs) is displayed as 0.

※ When selecting a queue, please check for any pending jobs before launching the notebook.

※ When selecting queues other than the jupyter queue, it is recommended to choose \*:gpu=1 to run Jupyter.

※ The paid billing policy follows the existing Neuron system billing policy, and details can be found on the National Supercomputing Center's fee information page (https://www.ksc.re.kr/jwjg/gjbg/ygan).

Select the desired queue from the job queue, click the Submit button to run Jupyter Notebook (you can also run it on other queues, but charges will apply; for billing information, refer to the KSC website's Neuron billing information).

<figure><img src="https://2000842020-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtvPPtXiAYArocipMRdp9%2Fuploads%2FahESWUzg1WaGDyGB8kS1%2Fnoname01.png?alt=media&#x26;token=55616409-f34f-4cc1-8233-62b8f6b90a5d" alt=""><figcaption></figcaption></figure>

The following screen will appear for a few seconds as resources are allocated:

![](https://2000842020-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtvPPtXiAYArocipMRdp9%2Fuploads%2Fgit-blob-b98388e3bd930810a79ea03f9c2bce5fe628e6de%2F%ED%81%90%20\(Queue\)%20%EC%84%A0%ED%83%9D%20%EB%B0%8F%20Jupyter%20%EC%8B%A4%ED%96%89\(1\).png?alt=media)

By default, the JupyterLab screen will be launched at https://jupyter.ksc.re.kr/user/a1113a01/lab.

<figure><img src="https://2000842020-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtvPPtXiAYArocipMRdp9%2Fuploads%2Fi20Iw63Kye4StnnTuvAm%2Fnoname01.png?alt=media&#x26;token=8f85d3f6-7733-4fa0-9cf4-d95340d3895a" alt=""><figcaption></figcaption></figure>

## C. How to Use Jupyter Notebook

### **1. Jupyter work environment**

* <mark style="color:red;">**Jupyter environment directory: /scratch/\[user ID]/.conda/envs/notebook**</mark>&#x20;
* <mark style="color:red;">**Directory where logs are saved: /scratch/\[user ID]/log/job ID.log**</mark>
* <mark style="color:red;">**Directory where job files are saved: /scratch/\[user ID]/workspace/**</mark>

※ Users install the machine learning/deep learning libraries they need in the .../notebook conda environment, so by default, these installations are made in /scratch/\[user ID], where there is a larger quota. (Log files generated after running Jupyter are also saved in /scratch/\[user ID].)

※ Code written by the user is stored in /scratch/\[user ID].

※ Information on exporting and importing conda environments for backup can be found in the software guide on the KISTI website.

&#x20;

To open a terminal, click the Terminal icon in the Launcher tab.

* If the Launcher tab is not visible, click the + icon in the Menu Bars.

<figure><img src="https://2000842020-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtvPPtXiAYArocipMRdp9%2Fuploads%2Ffe9umXyGr2RWRNNxSzS9%2Fnoname01.png?alt=media&#x26;token=9075b3fa-61fe-40b7-8f4e-aa89841b84fe" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2000842020-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtvPPtXiAYArocipMRdp9%2Fuploads%2FzEp9ERz0YhIEMvFoTmGb%2Fnoname02.png?alt=media&#x26;token=2bf57e28-4883-4f95-a908-f3dbcba9fcd1" alt=""><figcaption></figcaption></figure>

To run TensorBoard, go to Menu Bars → + icon → Launcher → Tensorboard.

<figure><img src="https://2000842020-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtvPPtXiAYArocipMRdp9%2Fuploads%2FuA6CmmoSxMx5iPA9K1Fd%2Fnoname01.png?alt=media&#x26;token=2e63ed68-77fb-48f3-a71f-e337747c8b63" alt=""><figcaption></figcaption></figure>

<figure><img src="https://2000842020-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtvPPtXiAYArocipMRdp9%2Fuploads%2FqwjeOjjHA3jmom37MKWl%2Fnoname02.png?alt=media&#x26;token=065617c4-0e85-4814-bdf0-2b91c28438a1" alt=""><figcaption></figcaption></figure>

#### 2) End a running session

To end running sessions, click the Session tab in the Left Side Bar and shut down any running Terminal Sessions or Kernel Sessions by clicking the Shut Down button.

※ If you close the JupyterHub web page without ending the sessions, they will remain active the next time you start Jupyter. (No billing occurs during this time)

<figure><img src="https://1364771727-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfBmICgXvD3ihLC5OZla4%2Fuploads%2FnmjOo0RTyUvlg13FtoOu%2Fshutdown.png?alt=media&#x26;token=0df6b7b3-6ee1-4f22-98c3-8993815fa1fb" alt=""><figcaption></figcaption></figure>

#### 3) Shutting Down Jupyter

File -> Hub Control Panel -> Stop My Server

<figure><img src="https://1364771727-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfBmICgXvD3ihLC5OZla4%2Fuploads%2FJ8EtVvtOvCA8yYJXgBwu%2Fstopserver.png?alt=media&#x26;token=3bc8f7bb-e148-49c0-b630-44e61370057f" alt=""><figcaption></figcaption></figure>

<mark style="color:red;">**※ All running Jupyter and sessions are automatically closed when you log out of the homepage.**</mark>

<figure><img src="https://2000842020-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtvPPtXiAYArocipMRdp9%2Fuploads%2Fevgu4xCTIcTEecRwhtJp%2Fnoname01.png?alt=media&#x26;token=5df02cf2-8f09-4e45-a58c-de5f49474505" alt=""><figcaption></figcaption></figure>



## D. Running Machine Learning/Deep Learning Example Code

### **1.** Install the Libraries Necessary for Running Example Code

In the Launcher, click Terminal to install the libraries needed for machine learning/deep learning

<figure><img src="https://1364771727-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfBmICgXvD3ihLC5OZla4%2Fuploads%2FywAn0bxY7tj3noweAptx%2F1.png?alt=media&#x26;token=305e2693-8595-4b7a-afcc-95eb02b66c88" alt=""><figcaption></figcaption></figure>

In the terminal, activate the notebook environment with the command conda activate notebook and install the necessary libraries in the notebook environment.

※ The libraries must be installed in the notebook conda environment to be applied to the Jupyter web interface.

<figure><img src="https://1364771727-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfBmICgXvD3ihLC5OZla4%2Fuploads%2FkVK7Vs1FXlFykCw41ciM%2F2.png?alt=media&#x26;token=dfed52e8-4331-4457-8acc-5a1d8efb56eb" alt=""><figcaption></figcaption></figure>

Example of installing machine learning/deep learning-related libraries in the notebook environment.

```shell-session
772% [a1113a01@gpu06 workspace]$ conda activate notebook
(notebook) 773% [a1113a01@gpu06 workspace]$ conda install -c conda-forge [conda package name]-y
```

### **2.** Writing and Running Example Code

In your job directory, click the example file iris\_ex.ipynb.

In the program editing/execution window, run the example code by pressing Shift+Enter.

※ You can ignore warnings that appear during execution, and they will not be displayed if you run the same code again. (The warnings provide guidance on syntax differences due to version changes.)

<figure><img src="https://2000842020-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtvPPtXiAYArocipMRdp9%2Fuploads%2FBMCjA84DiZwLUPN2TONc%2Fnoname01.png?alt=media&#x26;token=a187ef3e-e99a-4ac8-9329-f509d98d5062" alt=""><figcaption></figcaption></figure>

Output a graph using the matplotlib library

<figure><img src="https://1364771727-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfBmICgXvD3ihLC5OZla4%2Fuploads%2Fgit-blob-b9e83a5859c0bb4a60aec44cbaf180005cf5ea87%2FSX6ZxYNxMoZSub3.png?alt=media" alt=""><figcaption></figcaption></figure>

### **3.** Running Tensorboard

In the Menu Bar -> Files, click tfboard\_ex.ipynb.

Run the code with Shift+Enter (this takes about 1 minute).

<figure><img src="https://1364771727-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfBmICgXvD3ihLC5OZla4%2Fuploads%2F0jJ0Cvtjll6PjjzTt6xh%2F4.png?alt=media&#x26;token=cd959d48-c484-4447-bdfc-c95b9eb2d201" alt=""><figcaption></figcaption></figure>

Run Tensorboard.&#x20;

※ Log data is saved in the logs folder.

<figure><img src="https://1364771727-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfBmICgXvD3ihLC5OZla4%2Fuploads%2FFNZnPDvJryZdwG5V9Vud%2F5.png?alt=media&#x26;token=d496eae5-bd86-4bbd-b706-25bfd412b054" alt=""><figcaption></figcaption></figure>

TensorBoard -> Scalars

<figure><img src="https://1364771727-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfBmICgXvD3ihLC5OZla4%2Fuploads%2Fgit-blob-90d35012082dbce38ea745f7355823a93d3b872b%2FnNt6sLANd0iPvq2.png?alt=media" alt=""><figcaption></figcaption></figure>

TensorBoard -> Graphs

<figure><img src="https://1364771727-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfBmICgXvD3ihLC5OZla4%2Fuploads%2Fgit-blob-43987b52bc0faf02c49f21dc1b4f566207a669e2%2FE7Q3W0udQjkzOXy.png?alt=media" alt=""><figcaption></figcaption></figure>

TensorBoard -> Distributions

<figure><img src="https://1364771727-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfBmICgXvD3ihLC5OZla4%2Fuploads%2Fgit-blob-f4c4c75cd3548e45a15c8b7858b8f81ddfcf8ad4%2FYhCiSqGRPGn79il.png?alt=media" alt=""><figcaption></figcaption></figure>

TensorBoard -> Histograms

<figure><img src="https://1364771727-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfBmICgXvD3ihLC5OZla4%2Fuploads%2Fgit-blob-cf5dd7f814163e3a7f2083e835e4b5186446c2de%2FidcwepwUTPBQ8v8.png?alt=media" alt=""><figcaption></figcaption></figure>

### **4.** Creating a New Launcher and Writing Python Code

Click New -> Python 3 as shown below to start writing new Python code.

<figure><img src="https://1364771727-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfBmICgXvD3ihLC5OZla4%2Fuploads%2FV1QCpyXlv7tJC7KeuFo2%2F6.png?alt=media&#x26;token=cc2a918c-0b03-4cae-83a8-ad4664692b77" alt=""><figcaption></figcaption></figure>

❍ A new Jupyter Notebook Launcher with the Python 3 kernel will be launched.

<figure><img src="https://1364771727-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfBmICgXvD3ihLC5OZla4%2Fuploads%2FKNUxt1H58RrSyYKbJWGS%2F7.png?alt=media&#x26;token=ab98d19b-42f7-43a1-88dd-004930290413" alt=""><figcaption></figcaption></figure>



## E. How to Shut Down Jupyter

### **1.** Ending Running Sessions

To end running sessions, click the Session tab in the Left Side Bar and shut down any running Terminal Sessions or Kernel Sessions by clicking the Shut Down button.

※ If you close the JupyterHub web page without ending the sessions, they will remain active the next time you start Jupyter. (No billing occurs during this time)

<figure><img src="https://1364771727-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfBmICgXvD3ihLC5OZla4%2Fuploads%2FzEdAGaqZGQsMh8W09dCi%2F8.png?alt=media&#x26;token=bb567800-6de2-4912-8689-b2214e7ab7fa" alt=""><figcaption></figcaption></figure>

### **2.** Shutting Down Jupyter

(JupyterLab) When finished using Jupyter, be sure to shut it down to release the resources.

You can release the resources by clicking Hub Control Panel in the File menu to return to the Home page, then clicking Stop My Server.

<figure><img src="https://2000842020-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtvPPtXiAYArocipMRdp9%2Fuploads%2FCkUbCeMa7RTFoIoxxMVC%2Fnoname01.png?alt=media&#x26;token=e652dd1b-2c67-47e8-aedd-2a208626e7cf" alt=""><figcaption></figcaption></figure>



## F. Resetting the Jupyter Environment

If installing packages via pip in the notebook conda environment causes version conflicts with packages installed via conda, preventing Jupyter Notebook from launching, you can reset the environment using the following command.

Run /apps/jupyter/reset\_env.sh on the login node as terminal.

Running this script will delete all packages installed in the notebook virtual environment located in /scratch/\[user ID]/.conda/envs and reinstall the basic packages needed to run Jupyter.

Data in /scratch/\[user ID]/workspace/ will be preserved.

```shell-session
[a1113a01@glogin02 ~]$ sh /apps/jupyter/reset_env.sh

Remove all packages in environment /scratch/a1113a01/.conda/envs/notebook:

Preparing transaction: done

Verifying transaction: done

If you need another packages, you can run the installation scripts for some packages such as below ! 
1.(mandatary) conda activate notebook
2.(option)
  a. [tensorfow] sh /apps/jupyter/kisti_conda_tensorflow.sh
  b. [pytorch] sh /apps/jupyter/kisti_conda_pytorch.sh
  c. [etc] sh /apps/jupyter/kisti_conda_etc.sh

[a1113a01@glogin02 ~]$ conda activate notebook

## Run the automated installation script for additional packages as needed

(notebook)[a1113a01@glogin02 ~]$ /apps/jupyter/kisti_conda_tensorflow.sh  
(notebook)[a1113a01@glogin02 ~]$ /apps/jupyter/kisti_conda_pytorch.sh
```

{% hint style="info" %}
Last updated on November 11, 2024.
{% endhint %}
