Neuron Jupyter
Last updated
Last updated
JupyterHub is open-source software that allows the use of JupyterLab/Notebook in a multi-user environment.
JupyterHub supports various environments (JupyterLab, Notebook, RStudio, Nteract, etc.) and can be seamlessly integrated with authentication servers (OAuth, LDAP, GitHub, etc.) and batch schedulers (PBSPro, Slurm, LSF, etc.).
JupyterHub easily integrates with Kubernetes, a container management platform, making it easy to connect to container-based cloud environments.
※ The Neuron-based JupyterHub has added Bright LDAP and OTP authentication features for the 5th system, integrates with the Slurm batch scheduler to allocate resources for running Jupyter, and currently provides Jupyter Notebook by default, with JupyterLab also available.
Jupyter Notebook is a web-based open-source application used by programmers for document creation, code generation and execution, data visualization using mathematical libraries, statistical modeling, and machine learning/deep learning programming.
It supports over 40 programming languages, including Python, R, Julia, and Scala.
Code written in a programming language can be converted into various types, such as HTML, images, video files, and LaTeX.
It can be integrated with various tools/libraries such as Apache Spark, Pandas, Scikit-learn, ggplot2, and TensorFlow.
JupyterLab is an extensible module that adds user-friendly features to the Jupyter Notebook interface.
Unlike Jupyter Notebook, it provides multiple documents or other features in a single workspace using tabs and splitters.
Access the login node
❍ (Required) Run the script to install Jupyter-related packages
※ Create a notebook conda environment and install the packages jupyter notebook, jupyterlab, cudatoolkit 11.6, and cudnn.
(Optional) Run a script to automatically install additional packages as needed, or install them manually.
※ Activate the notebook conda environment: $ conda activate notebook
※ Run this only once; once the environment is set up, you can immediately access the web page (C. How to use JupyterLab) You can use JupyterLab/Notebook.
Access the login node (neuron.ksc.re.kr) via terminal and run the following script: Execute /apps/jupyter/kisti_conda_jupyter.sh.
Running this script will create a notebook Conda environment in the /scratch/[user ID]/.conda/envs directory, automatically install the JupyterHub, JupyterLab, and Notebook packages, and install cudatoolkit=11.6 and cudnn necessary for a multi-GPU environment.
※ You only need to run this file once, and afterward, you can directly access and use it via the web page.
※ The executable file can be run directly from the shared directory at /apps/jupyter/kisti_conda_jupyter.sh.
※ The following test was conducted using the user ID a1113a01.
Restart the shell and disable automatic activation of the base environment. (Run only once)
※ Set the automatic activation of the base environment to false to prevent it from automatically activating the next time. (If the base environment is not activated, execute source ~/.bashrc followed by conda activate notebook immediately.)
Activate the conda notebook environment using the following command.
Users who wish to use TensorFlow (including TensorBoard) or PyTorch can install them by running the automated installation script provided by KISTI.
※ Caution: Make sure to run the script in the notebook user environment.
※ From now on, you can directly access the web and use Jupyter Notebook. (These tasks only need to be performed once.)
Go to https://jupyter.ksc.re.kr and enter your registered Neuron account, OTP, and password.
On the main screen, you can check resource usage and click the Refresh button to update the resource usage status.
Before running Jupyter, click the Refresh button to check the resource status.
Total CPUs: Total number of CPU cores
Alloc CPUs: Number of CPU cores in use
Total GPUs: Total number of GPUs
Alloc GPUs: Number of GPUs in use
Node Usage: Node utilization rate
Check the queue information
jupyter queue (free): For environment installation, preprocessing, and debugging
other queues (paid): For running and visualizing deep learning/machine learning models
※ The jupyter queue currently allows up to 20 Jupyter Lab/Notebook sessions (10 per node) across 2 nodes (multiple users share the CPU+GPU [v100] of the nodes).
※ The GPU in the jupyter queue is a shared resource, so the allocation information (Alloc GPUs) is displayed as 0.
※ When selecting a queue, please check for any pending jobs before launching the notebook.
※ When selecting queues other than the jupyter queue, it is recommended to choose *:gpu=1 to run Jupyter.
※ The paid billing policy follows the existing Neuron system billing policy, and details can be found on the National Supercomputing Center's fee information page (https://www.ksc.re.kr/jwjg/gjbg/ygan).
Select the desired queue from the job queue, click the Submit button to run Jupyter Notebook (you can also run it on other queues, but charges will apply; for billing information, refer to the KSC website's Neuron billing information).
The following screen will appear for a few seconds as resources are allocated:
By default, the JupyterLab screen will be launched at https://jupyter.ksc.re.kr/user/a1113a01/lab.
Jupyter environment directory: /scratch/[user ID]/.conda/envs/notebook
Directory where logs are saved: /scratch/[user ID]/log/job ID.log
Directory where job files are saved: /scratch/[user ID]/workspace/
※ Users install the machine learning/deep learning libraries they need in the .../notebook conda environment, so by default, these installations are made in /scratch/[user ID], where there is a larger quota. (Log files generated after running Jupyter are also saved in /scratch/[user ID].)
※ Code written by the user is stored in /scratch/[user ID].
※ Information on exporting and importing conda environments for backup can be found in the software guide on the KISTI website.
To open a terminal, click the Terminal icon in the Launcher tab.
If the Launcher tab is not visible, click the + icon in the Menu Bars.
To run TensorBoard, go to Menu Bars → + icon → Launcher → Tensorboard.
To end running sessions, click the Session tab in the Left Side Bar and shut down any running Terminal Sessions or Kernel Sessions by clicking the Shut Down button.
※ If you close the JupyterHub web page without ending the sessions, they will remain active the next time you start Jupyter. (No billing occurs during this time)
File -> Hub Control Panel -> Stop My Server
※ All running Jupyter and sessions are automatically closed when you log out of the homepage.
In the Launcher, click Terminal to install the libraries needed for machine learning/deep learning
In the terminal, activate the notebook environment with the command conda activate notebook and install the necessary libraries in the notebook environment.
※ The libraries must be installed in the notebook conda environment to be applied to the Jupyter web interface.
Example of installing machine learning/deep learning-related libraries in the notebook environment.
In your job directory, click the example file iris_ex.ipynb.
In the program editing/execution window, run the example code by pressing Shift+Enter.
※ You can ignore warnings that appear during execution, and they will not be displayed if you run the same code again. (The warnings provide guidance on syntax differences due to version changes.)
Output a graph using the matplotlib library
In the Menu Bar -> Files, click tfboard_ex.ipynb.
Run the code with Shift+Enter (this takes about 1 minute).
Run Tensorboard.
※ Log data is saved in the logs folder.
TensorBoard -> Scalars
TensorBoard -> Graphs
TensorBoard -> Distributions
TensorBoard -> Histograms
Click New -> Python 3 as shown below to start writing new Python code.
❍ A new Jupyter Notebook Launcher with the Python 3 kernel will be launched.
To end running sessions, click the Session tab in the Left Side Bar and shut down any running Terminal Sessions or Kernel Sessions by clicking the Shut Down button.
※ If you close the JupyterHub web page without ending the sessions, they will remain active the next time you start Jupyter. (No billing occurs during this time)
(JupyterLab) When finished using Jupyter, be sure to shut it down to release the resources.
You can release the resources by clicking Hub Control Panel in the File menu to return to the Home page, then clicking Stop My Server.
If installing packages via pip in the notebook conda environment causes version conflicts with packages installed via conda, preventing Jupyter Notebook from launching, you can reset the environment using the following command.
Run /apps/jupyter/reset_env.sh on the login node as terminal.
Running this script will delete all packages installed in the notebook virtual environment located in /scratch/[user ID]/.conda/envs and reinstall the basic packages needed to run Jupyter.
Data in /scratch/[user ID]/workspace/ will be preserved.
