Jupyter Blog - Matt McCormick

Embed interactive itkwidgets 3D renderings into JupyterLite deployments

2023-03-10T14:04:00+00:00

A tutorial that demonstrates a zero-server, interactive 3D rendering notebook and walks through the quick and easy configuration that can be customized to your needs.

Zero-install web applications have transformed the way we consume and deliver software. Browser-based interfaces facilitate rapid discovery, exploration, and universal access.

However, for research software engineers (RSEs), developing traditional software stacks for web applications is not only onerous, but those stacks may limit essential future scalability and may be even more onerous to sustain. A RSE must face difficult questions:

Who is going to pay to keep the servers online?
Who is going to pay to scale the servers for many user or datasets?
When are you going to find the time to learn and keep up-to-date with all the devops knowledge and skills required?
Who is going to maintain the system and address security vulnerabilities as they arise?
How is private data on the server managed and kept secure?

As one of my favorite professors used to say, in cases like this we can look to the advice offered by a wise doctor:

Patient: Oh, Doctor, it hurts badly when I move my knee like this.

Doctor: Stop moving your knee like that!

In some cases, components of the traditional web application software stack are necessary, and some of those components are easier, more scalable, and more sustainable than others. However, for many RSE use cases, we now can create useful web applications while avoiding traditional server-related hardships altogether.

In this tutorial, we will demonstrate how to create a zero-server JupyterLite deployment that embeds interactive 3D renderings into advanced scientific applications, such as for deep learning medical image analysis applications using MONAI.

JupyterLite is a JupyterLab distribution that runs entirely in the browser built from the ground-up using JupyterLab components and extensions. JupyterLite uses a WebAssembly-based distribution of scientific Python called Pyodide.

ITKWidgets provides interactive widgets to visualize images, point sets, and 3D geometry on the web. ITKWidgets is powered by the same WebAssembly technology. It is built on ITK-Wasm and ImJoy, a hybrid computing platform that communicates via symmetrical transparent remote procedure calls. ImJoy and ITKWidgets support browser-based Pyodide communication along with a number of additional server-client communication transport mechanisms.

In this tutorial, we will first demonstrate a zero-server, interactive 3D rendering notebook. Then, we walk through the quick and easy configuration that can be customized to your needs. Let’s get started! 🚀

0. Preliminaries

Reproduce the figure below, a rendering of medical imaging volume of an abdominal aortic stent, by running the notebook in your web browser! After the page has loaded, use the standard Shift+Enter keys to execute the Jupyter notebook cells.

a medical imaging volume of an abdominal aortic stent rendered in JupyterLite

Note that unlike other Jupyter deployments, the python code runs on your system instead of a server.

1. Create the JupyterLite environment

To build our sustainable JupyterLite deployment, we will use a Python environment that contains Python packages for:

jupyterlite
imjoy_jupyterlab_extension
Any other JupyterLab federated extensions (JupyterLab 3 extensions) that you want in your JupyterLab deployment.

Create a requirements.txt file with:

jupyterlite[all]==0.1.0b17
imjoy_jupyterlab_extension

And install the packages:

python -m pip install -r ./requirements.txt

See also the related JupyterLite extension addition documentation.

2. Add itkwidgets and other Python packages

Next, we will add itkwidgets, its dependencies, and other Python packages and their dependencies, that we wish to include into the JupyterLite configuration for deployment. These packages, along with the packages available in the Pyodide distribution, will be available in the deployed site.

Create a jupyterlite_config.json file, which specifies the locations of the itkwidgets wheel Python packages. Add other desired packages and their dependencies as follows.

{
    "PipliteAddon": {
        "piplite_urls": [
            "https://files.pythonhosted.org/packages/4c/ee/56f970ca26375176d3e4885f58471a12d5a6794bcefe8ad0ccb8d7158ca3/itkwasm-1.0b82-py3-none-any.whl",
            "https://files.pythonhosted.org/packages/6c/55/c3fc7e2b9671d15f0c0becdcb9fad6c330172988744ad6eaa17b71bace88/imjoy_rpc-0.5.16-py3-none-any.whl",
            "https://files.pythonhosted.org/packages/69/d9/5a6c8af2f4b4f49a809ae316ae4c12937d7dfda4e5b2f9e4167df5f15c0e/imjoy_utils-0.1.2-py3-none-any.whl",
            "https://files.pythonhosted.org/packages/c9/dc/3504845528418aff0b71f4b622bb0e8e12adec2d8f2c1ba21d695b9ac6e6/itkwidgets-1.0a24-py3-none-any.whl",
            "https://files.pythonhosted.org/packages/bb/3e/3667ac685ae83887b874896bcb55584797ba6b52a292df3e4b37736a9610/ngff_zarr-0.1.6-py3-none-any.whl"
        ]
    }
}

You can find links to these URLs by browsing the package on PyPI and copying the link from the Download files page for a package.

For packages that do not have a wheel on PyPI, you can provide one locally by placing them in the pypi/ directory of your JupyterLite configuration. For example, if you want to use a local version of itkwidgets instead of the version on PyPi, you could directly add this dask-image wheel to your pypi/ directory.

$ ls pypi/
pypi/dask_image-2022.9.0-py2.py3-none-any.whl

3. Add notebooks and data

Add notebooks and data you would like available in the deployment in the files/ directory. In this tutorial, we will add a Hello3DWorld.ipynb notebook.

$ ls files/
files/Hello3DWorld.ipynb

In your notebook, install additional packages in the first cell with piplite.

import piplite
await piplite.install(["itkwidgets==1.0a24"])

4. Build and deploy

Build your site with the command:

jupyter lite build

Serve the site locally with python -m http.server --directory ./_output or:

jupyter lite serve

The site can be deployed and shared with free static file hosting services such as GitHub Pages, Netlify, or Fleek.

What’s Next

In this post, we learned how to create a scalable, sustainable, zero-server Jupyter deployment that uses ITKWidgets for 3D rendering. In subsequent posts, we will discuss how to create simple, zero-server, custom web applications written in Python with PyScript.

Enjoy ITK!

SlicerJupyter: a 3D Slicer kernel for interactive publications

2020-07-08T16:29:00+00:00

Use Jupyter and 3D Slicer kernel to implement biomedical data processing workflows in a notebook.

The Jupyter ecosystem is a powerful platform for exploratory computational science, and now it can connect with some of the deep and rich domain-specific desktop applications that have decades of feature development already invested in them. With the integration of 3D Slicer with Jupyter through the xeus-python’s interpreter, we demonstrate how a Qt-based graphical desktop application with 3D visualization provided by Visualization Toolkit (VTK), image processing provided by the Insight Toolkit (ITK), can be used through a Jupyter notebook. This approach is available on the SlicerJupyter GitHub repository and could be extended to other applications that embed Python, such as Blender, FreeCAD, or ParaView.

This xeus-python integration is beneficial both for the Jupyter ecosystem and desktop applications. Features that have been developed for decades for desktop applications become readily available for Jupyter users without learning a new working environment or redeveloping features. For a desktop application, the Jupyter notebook can serve as a way to create reproducible data processing workflows, scientific publications, and maintainable tutorials without requiring local software installation. From Jupyter, you can now quickly create simple medical imaging applications with extremely rich interactivity from 3D Slicer. Read below for a discussion of the features available and the history of the project.

Powerful Medical Imaging Capabilities Available Through Jupyter

3D Slicer (or Slicer for short) is a C++ desktop application that uses Qt, ITK, and VTK libraries for visualization and medical image analysis. Slicer’s embedded Python interpreter makes all its features accessible with the Python programming language. Slicer has a simple built-in console to run Python commands interactively and can run Python scripts from files, but these are not as convenient as cell-base interactive notebooks, which have become popular among data scientists and researchers in recent years.

By integrating the xeus-python kernel, we can use a Slicer process as a Jupyter kernel. xeus-python leverages the xeus C++ implementation of the Jupyter kernel protocol. xeus-python is an alternative to ipykernel, which can be used with a vanilla CPython interpreter, interfacing to standard CPython Jupyter widgets like itkwidgets, but it can also be coupled with custom interpreters and GUI event loops, like the Slicer interpreter and its Qt event loop. This allows you to represent a complete scene in Medical Reality Markup Language, MRML, Slicer’s internal data structure. The kernel exposes the full medical imaging API and representation of your data in a meaningful way for Python developers, allowing access through standard Python ecosystem formats such as pandas dataframes and NumPy arrays in the Notebook.

Interactivity Levels

You can also use Jupyter interactive widgets (sliders, buttons, etc.) to control Slicer, modify data, or adjust processing and visualization parameters. Interactivity can be implemented at different levels.

Level 1: Standard Jupyter widgets display application specific objects by automatic conversion of application-specific data objects to standard Python objects. For example, Slicer markup fiducial lists are displayed as a nicely formatted table and model nodes are rendered as 3D objects.
Level 2: Static image widgets display content that the desktop application renders. These widgets can be made interactive by modifying data and rendering parameters using additional standard widgets. This makes rich visualization capabilities — sophisticated rendering various data types, rendering of very large data sets, etc. directly available in Jupyter.
Level 3: Dynamic viewer widgets display 2D and 3D views rendered by the desktop application. Mouse and keyboard events are forwarded to the desktop application that allows zooming/rotating views, and to utilize all 3D interactions implemented in the desktop such as placing annotations, making measurements, or segmenting images the same way as if it was done on the desktop appication’s screen. This is implemented in the Slicer Jupyter kernel using ipycanvas and ipyevents packages.
Level 4: Full desktop graphical user interface integration. Users can see parts of the application window rendered in notebook cells, including standard desktop widgets (sliders, menus, etc.). It is implemented using noVNC and TigerVNC in Slicer Jupyter. This is particularly useful when the application runs on a remote server.

Video demonstrating how to run 3D Slicer using Binder

These interactive tools allow developers to implement complete data processing workflows in a notebook, even if certain steps require manual user inputs such as segmenting 3D regions or setting seed points.

Try it online!

Since Jupyter notebooks can be used from any web browser, it can essentially turn any desktop application to a web application. By setting up a remote Jupyter server, users do not have to install anything on their computers. We have set up a demonstration of this using Binder (www.mybinder.org) that anybody can try at https://mybinder.org/v2/gh/Slicer/SlicerNotebooks/master.

Click on the binder image to launch the demo

The docker image that installs and configures Slicer and all dependencies (ipycanvas, ipyevents, VNC, etc.) is available at https://github.com/Slicer/SlicerDocker/tree/master/slicer-notebook. For deployment to non-technical users, applications can be deployed using Voilà (https://voila.readthedocs.io/), which only shows relevant content and interactive widgets, so the notebook looks like a simple dynamic web page.

The current implementation is already stable and offers a wide range of features, but there is still room for design and performance improvements. For example, we could not implement fully automatic conversion of application-specific data objects to displayable Python objects (due to complex implementation of display hooks); xeus-python debugger’s threading model needs to be improved to allow using it without locking the application’s main thread; and dynamic viewer widget’s performance (level 3 interaction) could be optimized to achieve higher refresh rates.

History of Slicer and this integration

Built over two decades with support from the NIH and a worldwide open source developer community, 3D Slicer is a unique, multi-platform desktop application for analysis, integration, and visualization of medical images that is heavily used by researchers globally for basic and applied research in a wide range of topics. The 3D Slicer Community includes physician-scientists with disease-specific knowledge of clinical challenges, computer scientists and physicists who develop novel algorithms, imaging informatics researchers, software engineers with the ability to understand clinical problems and create reliable tools, and application engineers with the multidisciplinary skills to deploy these tools in a range of cancer research settings. Slicer is maintained by Kitware, Inc., and the NAMIC consortium, and has an active open source software community. Slicer is used in hospitals and by researchers, with more than 10k academic citations and more than 150k downloads in the last year alone.

Click on the image to download 3D Slicer

It can be used for image processing workflows on 2D, 3D, and 4D images. What makes Slicer so useful is its huge set of community contributed modules that extend its functionality, as well as its ability to read DICOM and a wide array of exotic file formats.

Slicer has been scriptable in Python for well over a decade, but a robust SlicerJupyter extension is born from significant development effort by the community. The idea for this integration dates back to SciPy in 2014, when Jean-Christophe Fillion-Robin of Kitware organized a sprint intended to integrate Slicer and an IPython notebook, motivated by the dream of creating interactive and fun tutorials for Slicer. Mike Sarahan and Jean-Christophe created this proof of concept https://github.com/commontk/QEmbedIPython#qt-embed-ipython, but it was far from usable. In 2015 Matt McCormick of Kitware, created the SlicerDocker repository to support headless builds and rendering in a Docker image. Then in June 2018, while attending the Slicer Project Week, Andras Lasso (Queen’s University) and Jean-Christophe learned about Xeus, a C++ implementation of the Jupyter kernel protocol developed by QuantStack that would help streamline the integration of Slicer with Jupyter. To support this effort, Andras and Jean-Christophe created the Slicer/SlicerJupyter GitHub repository. They also contributed changes to Xeus to support this new integration paradigm where the event loop of the kernel is driven by a Qt-based Desktop application (see here). Building on this foundation, Isaiah Norton, then working at Brigham and Women’s Hospital, contributed additional improvements like a better auto-completion using jedi (see https://github.com/Slicer/SlicerJupyter/pull/12) as well as integration with Binder. More recently, Jean-Christophe and Sylvain Corlay (QuantStack) met after the Slicer project week while attending SciPy in Austin. Following the creation of a new project called xeus-python (started by Martin Renou at QuantStack), we took the integration of Jupyter and Slicer to the next level by adding support for improved interactive use and MRML data node visualization directly in the notebook.

About the authors

Alphabetically ordered

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored xeus and xeus-python.

Sylvain Corlay

Jean-Christophe Fillion-Robin is an open-source enthusiast, original author of the SlicerJupyter extension and a principal engineer at Kitware Inc where he leads the development of “3D Slicer” based commercial applications. J-Christophe also maintains scikit-build, an improved build system generator for CPython C/C++/Fortran/Cython extensions.

Jean-Christophe Fillion-Robin

Mike Grauer is a Technical Leader on the data and analytics team at Kitware. He is specialized in building scalable server-side processing frameworks that enable scientific workflows over web platforms.

Mike Grauer

Andras Lasso is an original author of the SlicerJupyter extension, Senior Research Engineer and Associate Director of the Laboratory for Percutaneous Surgery at Queen’s University.

Andras Lasso

Matt McCormick is an open source, medical imaging researcher working at Kitware Inc. Matt is an active, contributing member of scientific open source software efforts such as the Insight Toolkit (ITK) and scientific Python (SciPy) communities, and maintains the Jupyter 3D widget, itkwidgets.

Matt McCormick

Isaiah Norton is a Senior Software Developer at TileDB, Inc. with experience in digital pathology and image-guided surgical navigation.

Isaiah Norton

Steve Pieper is a Chief Architect and active developer of the 3D Slicer application for well over a decade. He is CEO of Isomics, Inc, where he uses a range of software technologies to perform medical imaging research with leading universities and companies.

Steve Pieper

Martin Renou is a scientific software developer at QuantStack. He is the original author of xeus-python, the xeus-based Python kernel, and contributed to the new concurrency model used for the debugger.

Martin Renou

Mike Sarahan is a software engineer at RStudio, PBC working on bringing language ecosystems together. He is passionate about making software work for data scientists in ways that are easy to maintain and improve.

Mike Sarahan