Jupyter Blog - Johan Mabille

Congratulations, Distinguished Contributors!

2026-06-24T15:08:00+00:00

We are proud to announce the recipients of the Jupyter Distinguished Contributor (JDC) award for the 2025 cohort of contributors.

Project Jupyter Distinguished Contributors are recognized for their substantial contributions to Jupyter itself in both quality and quantity over at least two years. Contributions may include code, code review, infrastructure work, mailing list and chat participation, community help/building, education and outreach, fundraising, branding, marketing, inclusion and diversity, UX design and research, etc.

Please congratulate the winners of the 2025 cohort of Jupyter Distinguished Contributors!

James Colliander

Florence Haudin

Greg Mooney

Kirstie Whitaker

Congratulations, Distinguished Contributors!

2025-05-15T15:15:00+00:00

We are proud to announce the recipients of the Jupyter Distinguished Contributor (JDC) award for the 2024 cohort of contributors.

Please congratulate the winners of the 2024 cohort of Jupyter Distinguished Contributors!

Denisa Checiu

Rowan Cockett

Mahendra Paipuri

Isabel Paredes

David L. Qiu

Rick Wagner

Announcing the 2023 Jupyter Distinguished Contributor Award Recipients

2024-09-04T16:46:00+00:00

We are delighted to announce the recipients of the Jupyter Distinguished Contributor (JDC) award for the 2023 cohort.

Project Jupyter Distinguished Contributors are recognized for their significant and sustained contributions to Jupyter over a period of at least two years. These contributions encompass a wide range of areas, including code development, code review, infrastructure work, participation in mailing lists and chats, community building and support, education and outreach, fundraising, branding and marketing, efforts in inclusion and diversity, UX design, and research.

Please join us in congratulating the winners of the 2023 cohort of Jupyter Distinguished Contributors:

Nicolas Brichet
Piyush Jain
Duc Trung Le
Jason Weill

Their dedication and hard work have been instrumental in advancing the Jupyter project and community.

Johan Mabille — on behalf of the Jupyter Distinguished Contributors

Congratulations, Distinguished Contributors!

2023-06-02T12:41:00+00:00

We are proud to announce the recipients of the Jupyter Distinguished Contributor (JDC) award for the 2022 cohort of contributors.

Please congratulate the winners of the 2022 cohort of Jupyter Distinguished Contributors!

Tania Allard

Carlos Herrero

Itay Dafna

Eric Charles

Max Klein

Ryan Lovett

Sharan Foga

Rollin Thomas

Ayaz Salikhov

Improving the accessibility of Jupyter

2023-02-24T11:11:00+00:00

The adoption of Jupyter has grown significantly in the past few years — especially in education contexts. The project has become a foundational component of our digital experience, from the first lines of code written by high-school students to the most advanced research and engineering use cases.

In this context of global adoption, it is even more important to ensure that as many people as possible can use the project. It is estimated that 15% of the population has a disability that may impair their ability to use online services. If we don’t want them to be excluded from learning sciences, technology, and engineering, we must improve the tools to make them usable by everyone…

To address this issue, a Jupyter accessibility working group formed and has now become an official Jupyter Subproject and received a grant from the Chan Zuckerberg Initiative.

Creating accessible user interfaces is a challenging task as it requires accommodating a broad range of disabilities, including vision, motor, and cognitive impairments. This article presents some of the recent accessibility improvements in the Jupyter Notebook codebase.

CodeMirror 6 and Notebook 7

Jupyter Notebook 7, which is the next major release of the Jupyter Notebook frontend, has been rebuilt with modern JupyterLab components and extensions while maintaining the classic Jupyter Notebook user experience. Notebook 7 will be released shortly after JupyterLab 4 and will take advantage of JupyterLab improvements.

Improving the accessibility of Jupyter had long been impeded by significant obstacles. The primary obstacle was that the text editor underlying the Jupyter Notebook (CodeMirror 5) had major accessibility issues.

Fortunately, this accessibility bottleneck has been unblocked as JupyterLab has been upgraded to use CodeMirror 6, a complete rewrite of the text editor with a strong focus on accessibility. Although this upgrade required extensive codebase modifications, the changes landed a few months ago and will be available with JupyterLab 4. Jupyter Notebook 7 will benefit from the CodeMirror 6 upgrade.

Initial accessibility audits

Shortly after the CodeMirror 6 migration landed, we made an automated audit of the accessibility of Notebook 7 with these changes, and found that the number of warnings and errors reported by Axe Auditor went down from several hundreds to a few dozen, most of which seemed fixable. Encouraged by these results, we decided to work on bringing that count to zero!

With a series of fixes ranging from simple changes to the DOM structure of components to fixing up base Lumino components (Lumino is a JavaScript framework that underlies a lot of the JupyterLab frontend architecture), we were able to make the Notebook 7 codebase pass the Axe Auditor tests with zero error or warning. While we have not reached that yet in JupyterLab, both JupyterLab and Jupyter Notebook benefit from these improvements made for Notebook 7 since they are built from the same components.

Axe Auditor output with the “classic” Notebook UI (v6) with a test notebook.

Axe Auditor output with the “next” Notebook UI (v7 alpha 13) with the same test notebook.

Similar results were achieved with the file explorer and text editor tabs of the application.

Beyond accessibility audits

Improving the accessibility of Jupyter requires more than just addressing issues flagged by automated audit tools and the Jupyter Accessibility Subproject is working on improving accessibility across the board in Project Jupyter. These results with the Notebook 7 codebase are very encouraging, but working with end-users and getting their feedback will also be necessary to make the user interface truly accessible.

To address the diversity of accessibility requirements, we will approach specific use cases separately (for example, users with screen readers, or users who can operate a keyboard but not a pointing device). While the resulting changes may improve usability for everyone, we need to learn from users who have specific needs to make meaningful improvements.

Finally, many of the accessibility challenges in Jupyter stem from the content of the notebooks, as notebook authors may not follow the best practices to make their content usable by everyone (such as adding alt text to images and figures, or properly using headings to communicate the organization of a notebook). To promote these best practices, Jupyter could provide linting tools that produce inline warnings and hints for notebooks that do not follow these guidelines.

Acknowledgements

The work by Johan Mabille at QuantStack on migrating JupyterLab to use CodeMirror 6 was funded by Two Sigma. This upgrade also enabled significant performance improvements in the rendering of Jupyter notebooks, as detailed in this earlier post by Frédéric Collonval.

We are grateful to members of the Jupyter Accessibility Subproject who produced a thorough review of the Notebook 7 UI with a focus on keyboard navigation. This is an example of an issue that cannot be easily detected by auditing tools.

About the Authors

Nicolas Brichet is a scientific software developer at QuantStack active in the Jupyter ecosystem. Among other things, Nicolas made significant contributions to the JupyterLab and Voilà projects, and worked on porting the nbgrader package to JupyterLab. Nicolas worked on accessibility improvements in the JupyterLab and Lumino packages.

Johan Mabille is a technical director at QuantStack. He was honored with the Distinguished Contributors award in 2020 for his contributions to the Jupyter project. Among other things, he is one of the main authors of the JupyterLab visual debugger, and the creator of Xeus, a C++ implementation of the Jupyter kernel protocol at the basis of many Jupyter kernels. Johan was responsible for the migration of JupyterLab to CodeMirror 6.

Jeremy Tuloup is a technical director at QuantStack, and a Jupyter Distinguished Contributor. He is a core maintainer of JupyterLab and Voilà, and the creator of the JupyterLite project. Jeremy is the main initiator of the Notebook 7 project.

Frédéric Collonval is a technical director at QuantStack, and a Jupyter Distinguished Contributor. He is a core maintainer of the JupyterLab project. Frédéric contributed to the CodeMirror 6 migration and helped numerous new contributors to get their enhancements accepted.

Sylvain Corlay is the founder and CEO of QuantStack, and a Jupyter Distinguished Contributor. He has worked on many areas of the Jupyter project, from interactive widgets to language kernels and other core components.

Accelerating JupyterLab

2022-10-10T17:38:00+00:00

The next major release of JupyterLab will be significantly faster than previous versions. This was achieved both through systematic tracking of performance bugs and through significant upgrades to the Jupyter communication protocol and rendering mechanism for documents.

1. Setting up rigorous performance measurements

The first step to any measurable improvement in performance is to set up systematic measurement of performance.

The JupyterLab project now includes a UI performance benchmarking tool, in the form of a GitHub action that can be triggered on any pull request to check how performance is impacted by the change. The implementation of this new GitHub action is available in this repository: https://github.com/jupyterlab/benchmarks.

This tool measures the time required for performing the following actions: opening a test notebook, switching from the test notebook to a copy of it opened in another tab, switching from the test notebook to a text editor, switching back, searching for a word in the test notebook and closing the test notebook. There are multiple example notebooks in the test suites. Benchmark results are posted as comments on the pull request. You can see such a benchmark report here: #11494#issuecomment-976393815

Example report from the new benchmarking tool — each execution time distribution is represented by a box-plot graph (the box spans from the 1st to the 3rd quartiles with the white line positioned at the median value).

The addition of this benchmarking tool immediately allowed for optimization on how notebooks are hidden when switching tabs. Hiding can be done by adding a CSS class that enables some CSS rule, or forcibly setting display to “none”. Depending on the browser, picking one way or another of hiding content may trigger a reflow of the entire page, so we made this a settable with an option in the JupyterLab config.

The benchmark GitHub action was developed by Frédéric Collonval.

2. Upgrading to CodeMirror 6

The rendering of the text editor used in notebooks can be very expensive, especially in the case of large notebooks with many cells. Jupyter has historically relied on CodeMirror as its based text editor.

JupyterLab 4 includes an upgrade from CodeMirror 5 to CodeMirror 6, which is a complete rewrite of the text editor. This work can be found in pull requests #11638, #12877, and #12861 — modifying over 150 files of the JupyterLab codebase. Benchmarks indicate a rendering speedup factor between 2 and 3 on the large notebooks used in the benchmarking suite.

Benchmark report on the CodeMirror 6 migration PR

Note: CodeMirror 6 is also an important stepping stone towards making Jupyter notebooks accessible to people who need screen readers and other devices.

The migration of JupyterLab to CodeMirror 6 was performed by Johan Mabille.

3. Virtual rendering of notebooks

In JupyterLab 4, the notebook will only render the parts of the documents that are visible in the viewport. It significantly improves the rendering speed of large notebooks. The main pull request implementing this feature is available here: #12554.

Significant preparation work was required for this Pull Request, especially regarding the “search feature” and the “table of content” components that both made use of the notebook view instead of the document model. This was done in PRs #11689 and #12374 respectively.

The end results showed significant improvement in the rendering speed of large notebook files, with a speedup of 3 to 4, which come on top of the already improved performance from the CodeMirror 6 migration.

Benchmark report on the Virtual Rendering of notebooks

The virtual rendering of notebooks was developed by Frédéric Collonval.

4. Jupyter protocol alignment

The Jupyter server serves as a relay between the frontends such as JupyterLab or the notebook and kernels. The server ⇄ kernel communication is done over ZeroMQ sockets, with the well-specified Jupyter kernel protocol. The server ⇄ client communication is done over WebSockets.

Unfortunately, up until recently, the server ⇄ kernel (ZMQ), and the server ⇄ client (WebSocket) protocols differed slightly so that the server had to parse each message and re-serialise it in both directions. This processing cost is small for short messages such as execution requests and replies which are typically very short, however, it can become very costly when dealing with larger datasets being sent or retrieved from the front-end, such as large tables, complex mime type rendering. This misalignment of the ZMQ and WebSocket protocol can then become a real bottleneck.

In Jupyter Server 2, the WebSocket connection supports a new “aligned” protocol, in which messages can simply be copied over to and from ZeroMQ messages, which is supported by JupyterLab 4. (This work was done in PRs #657 (jupyter-server), #154 (jupyverse), and #11841 (JupyterLab)). This new aligned protocol is an opt-in, so that legacy Jupyter front-end are still expected to function with Jupyter Server 2.

Benchmarks indicate a large speedup factor (at least one order of magnitude, and more for larger messages) in the performance of the Jupyter server when displaying large data sets in Jupyter widgets. However, this is not captured by the JupyterLab benchmark tests which focus on the rendering performances.

The procol alignment work was done by David Brochart.

5. Lumino 2

The JupyterLab frontend is built upon the Lumino framework, which provides utilities for building in-browser desktop-like applications. It provides the foundations for such applications, including a uniform component wrapper that handles lifecycle management and efficient propagation of front-end events to an entire application, (e.g., resize events, drag-and-drop, layout calculation). Lumino also provides several high-performance components such as a drag-and-drop dock panel (used as the application shell for JupyterLab) and a best-in-class data grid component.

JupyterLab 4 includes a major upgrade of the Lumino. The main changes in Lumino 2 include the migration to ES2018, which allowed for the removal of large parts of the codebase prodiving features that are now natively available in JavaScript, such as native iterators, removing polyfills for promises, and special-case logic for idiosyncrasies of legacy browsers like IE. This upgrade is also leading to across-the-board performance improvements in the front-end, although not for the rendering of large documents. Lumino 2 supports background processing of UI components when the application resides in a background browser tab (a feature that may be back-ported to Lumino 1.x as well).

The Lumino 2 upgrade and integration in JupyterLab was done by Afshin Darian.

6. A faster Lumino data grid

Optimizations to the Lumino data grid widget were also implemented, speeding up the rendering in the case of merged cells (cf. PR #394). The Lumino datagrid is used in various parts of the JupyterLab UI, such as the table view for CSV files. It is also used extensively in third-party extensions such as the ipydatagrid Jupyter widget and the BeakerX table display.

The Lumino data grid optimization was done by Martin Renou.

Acknowledgement

The work by the QuantStack team on JupyterLab performance improvements was done in collaboration with Two Sigma. Several of these pull requests required major changes across the JupyterLab codebase. We are very grateful to Two Sigma for supporting the development of the Jupyter project at such a deep level.

We are grateful to Juliette Taka for the illustrations.

About the authors

Frédéric Collonval, who led the charge on JupyterLab performance improvements, is a technical director at QuantStack. He is a member of the core JupyterLab core team and authored several JupyterLab extensions.

Johan Mabille is a technical director at QuantStack, very active in the Jupyter ecosystem. He regularly contributes to JupyterLab, and developed the Xeus framework for creating Jupyter kernels.

David Brochart is a scientific software developer at QuantStack, very active in the Jupyter ecosystem. He is a maintainer of the Jupyter-server project, and the main author of Jupyverse. David also contributes to the geo-science open-source stack built atop Jupyter.

Afshin Darian is a technical director at QuantStack. He is the co-creator of the JupyterLab project and continues working on the project to this day.

Congratulations, Distinguished Contributors!

2022-03-16T16:23:00+00:00

We are proud to announce the recipients of the Jupyter Distinguished Contributor (JDC) award for the 2021 cohort of contributors.

Please congratulate the winners of the 2021 cohort of Jupyter Distinguished Contributors!

Mehmet Bektas

David Brochart

S. Chris Colbert

Frédéric Collonval

Martha Cryan

Wayne Decatur

Sarah Gibson

Mariana Meireles

Isabela Presedo-Floyd

Nicolas Thiéry

Xeus 2.0

2021-09-28T09:41:00+00:00

We have just released Xeus 2. This is a major release of the library. While it includes backward-incompatible changes, they are very limited and upgrading your kernels should be relatively easy.

How to upgrade Xeus-based kernels

Let’s illustrate that with the code of a simple kernel before and after upgrading to Version 2. Most typically, only the kernel’s main file will have to be updated:

diff --git a/example/src/main.cpp b/example/src/main.cpp
index 29de437..eb8fdba 100644
--- a/example/src/main.cpp
+++ b/example/src/main.cpp
@@ -21,12 +21,14 @@ int main(int argc, char* argv[])
     std::string file_name = (argc == 1) ? "connection.json" : argv[2];
     xeus::xconfiguration config = xeus::load_configuration(file_name);

+    auto context = xeus::make_context<zmq::context_t>();
+
     // Create interpreter instance
     using interpreter_ptr = std::unique_ptr<custom::custom_interpreter>;
     interpreter_ptr interpreter = interpreter_ptr(new custom::custom_interpreter());

     // Create kernel instance and start it
-    xeus::xkernel kernel(config, xeus::get_user_name(), std::move(interpreter), xeus::make_xserver_zmq);
+    xeus::xkernel kernel(config, xeus::get_user_name(), std::move(context), std::move(interpreter), xeus::make_xserver_zmq);
     kernel.start();

     return 0;

The first change is that you now have to instantiate and pass an additional argument to the kernel constructor: the context. Fortunately, Xeus provides a function to build such a context object, making the update straightforward. We will detail the reason for this change in the next section.

The second change is that the kernel constructor now requires a function to build the server while a default value was provided for it in Xeus 1.x. Again, the motivations for this change will be detailed in the next section. Finally the functions for building the servers have moved to different headers.

To summarize, here is the list of the breaking changes:

The kernel’s constructor now requires an additional argument, the context. Xeus provides a function to easily instantiate it.
The order of kernel’s constructor arguments has changed, and the function for building the server is now required.
The functions for building the servers have moved to the headers where the corresponding servers are defined.

Motivation for these changes

When we created Xeus, kernels were expected to run in separate processes. The communication between the kernels and the client was achieved with the ZeroMQ middleware framework, and it was natural to heavily rely on it in Xeus.

Recent developments have pushed in a different direction, where the kernel can run directly in the browser. In that case, a middleware library is not required anymore to communicate with the frontend. The architecture of Xeus was not suitable for this new kind of kernels. An update was required so that it is possible to build Xeus without the dependency on ZeroMQ.

About the developers

Johan Mabille is a scientific software developer at QuantStack. Johan is a co-author of Xeus, and developed the debugger extension to xeus-python.

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored Xeus and xeus-python.

Enabling the JupyterLab debugger with ipykernel

2021-05-13T11:45:00+00:00

Support for the Jupyter Debugger Protocol just landed in ipykernel

JupyterLab 3.0 includes a visual debugger that allows to interactively set breakpoints, step into functions, and inspect variables with any Jupyter kernel that implements the Jupyter debugger protocol.

The first two language kernels to implement the new protocol were xeus-python (a Python kernel) and xeus-robot (a kernel for Robot Framework). Unfortunately, the reference Python kernel, ipykernel, did not support debugging yet, until now!

Today, we are pleased to announce that debugging support landed in ipykernel, and will be available in the next major release, ipykernel 6.0. Pre-releases including the ipykernel debugger are available.

Debugging with ipykernel

What will change with ipykernel 6.0?

Enabling support for debugging in ipykernel required important changes in the code base regarding the concurrency model of the kernel. The main change is that the processing of messages on the “control channel” now happens in a different thread, allowing for the processing to happen while user code is running.

Ipykernel 6.0 includes several other updates. Tornado coroutines were dropped in favor of native coroutines. The Matplotlib inline backend was split into a separate package, and ipykernel depends on debugpy, an implementation of the Debug Adapter Protocol for Python.

If you are interested in testing out the new features, check out the beta release!

pip install ipykernel --pre

Try it now!

Thanks to MyBinder, you can also try it out without the need of installing anything on your computer. Just follow this link:

What about the future?

A lot of new features are in the works with the JupyterLab debugger.

JupyterLab 3.1 will include several usability improvements to the debugger.
It will also add the ability to submit code for execution when stopped at a breakpoint.
We are working on a richer variable explorer, using Jupyter’s rich display system to enable the rich-rendering of variables in the explorer, to e.g. render dataframes as tables.

Finally, we also plan on adding debugging support to other language kernels.

Acknowledgements

The work of Johan and Sylvain at QuantStack on the debugger support in ipykernel was funded by Two Sigma. We are grateful to Min Ragan Kelley and Matthias Bussonnier, who reviewed the pull requests on debugger support.

About the authors

Johan Mabille is a scientific software developer at QuantStack.

Johan is very active in the Jupyter ecosystem, as the creator of xeus, a C++ implementation of the Jupyter protocol, and several language kernels, such as xeus-python, xeus-cling, and xeus-robot. Johan also made contributions to the Jupyter widgets ecosystem and to JupyterLab.

Sylvain Corlay is the founder and CEO of QuantStack.

As an open-source developer, Sylvain is very active in the Jupyter project with contributions in several components of the stack, including widgets, kernels, nbconvert, and others. He is also a steering committee member of the project.

Sylvain also does volunteer work for the community, as member of board of directors of NumFOCUS, co-organizer of the PyData Paris Meetup, and vice-chair of JupyterCon 2020.

Robotic Process Automation with JupyterLab

2021-01-18T10:36:00+00:00

Robotic Process Automation (RPA) differs from classical automation tools in that the actions to automate can be developed by observing a user perform a task in a graphical user interface, across multiple applications.

It is a means to lower the entry barrier of process automation and enable the use of tools that don’t provide a programmatic interface.

Most typically, RPA developers will use a mixed approach between textual programming and performing actions manually. The resulting programs are typically called software robots. Therefore, interactive computing tools like Jupyter are a natural environment for RPA, as the interactive nature of Jupyter allows for quick iterations and trial-and-errors when developing such robots.

While many RPA tools are commercial software, Robot Framework and the tooling developed by Robocorp provide an open-source RPA programming language, with a high-level syntax, extensible with Python plugins. It has a rich ecosystem of libraries and tools that are developed as separate projects.

Robot Framework and Project Jupyter

Today, we are happy to announce the first release of xeus-robot, a Jupyter kernel for Robot Framework based on xeus, supporting the new JupyterLab 3.0 visual debugger, auto-completion, and much more!

But before we dive into xeus-robot, we should recognize the earlier work by Nick Bollweg and Asko Soukka, who developed two Jupyter kernels for Robot Framework, irobotframework, and robotkernel, and gave an amazing talk together at RoboCon in 2019!

The motivation for us to reboot this effort with a kernel based on xeus was to enable the JupyterLab Visual Debugger for this kernel. This requires a different concurrency model than that of ipykernel, which underlies both irobotframework and robotkernel. In the end, we were able to provide the same features and more, including e.g. code completion in Python cells, debugging etc.

Diving into Xeus-robot

xeus-robot is a reboot of the already existing robotkernel, based on xeus.

Like most language kernels, xeus-robot supports code completion, inspection, error handling, etc. It also allows using Python cells to define custom robot keywords in Python, those Python cells support code completion as well.

Code Completion in Robot Framework

By using the libraries robotframework-seleniumlibrary and robotframework-seleniumscreenshots, you can even complete the selection of elements on the page you are currently testing!

Code completion with an action of the user in the UI.

The %%python cell magic makes it possible to extend Robotframework with Python modules in the notebook.

Extending RobotFramework with Python, and code completion in Python cells

Widget-based UIs are provided to test Robotframework “keywords”.

Testing Robot Framework “keywords” with Jupyter widgets

Last but not least, xeus-robot comes with full support for the JupyterLab Visual Debugger! After the xeus-python kernel, it is the second Jupyter kernel to support the Jupyter Debugger Protocol! We hope that many more will come.

You can set breakpoints, step in defined keywords (the equivalent of functions), inspect variables, and see the callstack, as shown below.

The JupyterLab Visual Debugger in action with Robot Framework

Installing xeus-robot

Xeus-robot is available for all platforms on conda-forge, and can be installed with conda or mamba.

mamba install xeus-robot -c conda-forge

The jupyterlab-robotmode package, which provides JupyterLab syntax highlighting for Robot Framework will also be installed as a dependency.

For your conda installation, we recommend starting from mambaforge or miniforge which are available for download here, and default to the conda-forge channel (making the -c conda-forge argument unnecessary).

Try it online

Thanks to MyBinder, you can try it out without the need of installing anything on your computer. Just follow this link:

Acknowledgments

The work on xeus-robot by QuantStack was funded by Robocorp, and is part of their initiative to create an open-source RPA ecosystem. Beyond the xeus-robot kernel, Robocorp also funds the integration of Robot Framework with Visual Studio Code, RPA Framework, and other development to push the Robot Framework project forward.

The implementation of the debugger in xeus-robot relies on robotframework-lsp by Fabio Zadrozny and we want to thank him for his help in integrating it in xeus-robot.

About the authors

Martin Renou is a scientific software developer at QuantStack. He is the creator of the xeus-python and xeus-robot kernels.

Johan Mabille is a scientific software developer at QuantStack. He is a co-author of the xeus project, and developed the debugger extensions to xeus-python and xeus-robot. He also co-authored the front-end for the JupyterLab visual debugger.

Xeus is now a Jupyter subproject

2020-02-04T12:08:00+00:00

It is a great pleasure to announce that the Xeus project has been incorporated as a Jupyter subproject. Xeus will now be subject to the Jupyter governance and code of conduct.

For reference, the Jupyter Enhancement Proposal (JEP) for the Xeus incorporation is available here.

What is Xeus?

The Xeus project is a C++ implementation of the Jupyter kernel protocol. Xeus is not a kernel, but a library meant to facilitate the authoring of kernels.
Several Jupyter kernels have been created with Xeus:

xeus-cling, a kernel for the C++ programming language, based on the Cling C++ interpreter. The cling project comes from CERN and is at the foundation of the ROOT project.

The xeus-cling Jupyter kernel for the C++ programming language.
xeus-python, a kernel for the Python programming language, embedding the Python interpreter.

The xeus-python Jupyter kernel for the Python programming language
xeus-calc, a calculator kernel, meant as an educational example on how to make Jupyter kernels with Xeus.

Beyond these three kernels built on top of Xeus by the Xeus maintainers, third-parties have developed other Jupyter kernels with Xeus:

JuniperKernel, a kernel for the R programming language by Spencer Aiello.
xeus-fift, a kernel for the fift programming language by Michael Zaikin. The fift programming language was developed by Telegram to create TON blockchain contracts.
SlicerJupyter, a kernel for the Python programming language by Kitware which integrates into the Qt event loop of the Kitware “Slicer” project.

Finally, the xeus-python kernel includes a first implementation of the Jupyter debugger protocol used by the Jupyter debugger project. xeus-python enables the Debug Adapter Protocol over the Control channel through new debug request/reply and debug event messages.

Why moving Xeus under the Jupyter governance?

While Xeus started as a side project for QuantStack engineers, the project now has several stakeholders who depend on it. We think that moving the project to an open governance organization may be a better way to reflect this situation.

Acknowledgements

Xeus was started by the team of open-source developers at QuantStack as a separate project, but with the full intent to incorporate it into Jupyter. The initial project development at QuantStack was funded by Bloomberg.
Now, Xeus contributors work in many institutions, including Université Paris-Saclay and École Polytechnique.

About the Author

Johan Mabille is a Scientific Software Developer at QuantStack, specializing in high-performance computing in C++. He holds master’s degree in computer science from Centrale-Supelec.

As an open source developer, Johan coauthored xtensor, xeus, and xsimd. He also made major contributions to the JupyterLab debugger project and bqplot.

Interactive Workflows for C++ with Jupyter

2017-11-29T16:33:00+00:00

Scientists, educators and engineers not only use programming languages to build software systems, but also in interactive workflows, using the tools available to explore a problem and reason about it.

Running some code, looking at a visualization, loading data, and running more code. Quick iteration is especially important during the exploratory phase of a project.

For this kind of workflow, users of the C++ programming language currently have no choice but to use a heterogeneous set of tools that don’t play well with each other, making the whole process cumbersome, and difficult to reproduce.

We currently lack a good story for interactive computing in C++.

In our opinion, this hurts the productivity of C++ developers:

Most of the progress made in software projects comes from incrementalism. Obstacles to fast iteration hinder progress.
This also makes C++ more difficult to teach. The first hours of a C++ class are rarely rewarding as the students must learn how to set up a small project before writing any code. And then, a lot more time is required before their work can result in any visual outcome.

Project Jupyter and Interactive Computing

The goal of Project Jupyter is to provide a consistent set of tools for scientific computing and data science workflows, from the exploratory phase of the analysis to the presentation and the sharing of the results. The Jupyter stack was designed to be agnostic of the programming language, and also to allow alternative implementations of any component of the layered architecture (back-ends for programming languages, custom renderers for file types associated with Jupyter). The stack consists of

a low-level specification for messaging protocols, standardized file formats,
a reference implementation of these standards,
applications built on top of these libraries: the Notebook, JupyterLab, Binder, JupyterHub
and visualization libraries integrated into the Notebook and JupyterLab.

Adoption of the Jupyter ecosystem has skyrocketed in the past years, with millions of users worldwide, over a million Jupyter notebooks shared on GitHub and large-scale deployments of Jupyter in universities, companies and high-performance computing centers.

Jupyter and C++

One of the main extension points of the Jupyter stack is the kernel, the part of the infrastructure responsible for executing the user’s code. Jupyter kernels exist for numerous programming languages.

Most Jupyter kernels are implemented in the target programming language: the reference implementation ipykernel in Python, IJulia in Julia, leading to a duplication of effort for the implementation of the protocol. A common denominator to a lot of these interpreted languages is that the interpreter generally exposes a C API, allowing the embedding into a native application. In an effort to consolidate these commonalities and save work for future kernel builders, we developed xeus.

Xeus is a C++ implementation of the Jupyter kernel protocol. It is not a kernel itself but a library that facilitates the authoring of kernels, and other applications making use of the Jupyter kernel protocol.

A typical kernel implementation using xeus would in fact make use of the target interpreter as a library.

There are a number of benefits of using xeus over implementing your kernel in the target language:

Xeus provides a complete implementation of the protocol, enabling a lot of features from the start for kernel authors, who only need to deal with the language bindings.
Xeus-based kernels can very easily provide a back-end for Jupyter interactive widgets.
Finally, xeus can be used to implement kernels for domain-specific languages such as SQL flavors. Existing approaches use a Python wrapper. With xeus, the resulting kernel won’t require Python at run-time, leading to large performance benefits.

Interpreted C++ is already a reality at CERN with the Cling C++ interpreter in the context of the ROOT data analysis environment.

As a first example for a kernel based on xeus, we have implemented xeus-cling, a pure C++ kernel.

Redirection of outputs to the Jupyter front-end, with different styling in the front-end.

Complex features of the C++ programming language such as, polymorphism, templates, lambdas, are supported by the cling interpreter, making the C++ Jupyter notebook a great prototyping and learning platform for the C++ users. See the image below for a demonstration:

Features of the C++ programming language supported by the cling interpreter

Finally, xeus-cling supports live quick-help, fetching the content on cppreference in the case of the standard library.

Live help for the C++standard library in the Jupyter notebook

We realized that we started using the C++ kernel ourselves very early in the development of the project. For quick experimentation, or reproducing bugs. No need to set up a project with a cpp file and complicated project settings for finding the dependencies… Just write some code and hit Shift+Enter.

Visual output can also be displayed using the rich display mechanism of the Jupyter protocol.

Using Jupyter’s rich display mechanism to display an image inline in the notebook

Another important feature of the Jupyter ecosystem are the Jupyter Interactive Widgets. They allow the user to build graphical interfaces and interactive data visualization inline in the Jupyter notebook. Moreover it is not just a collection of widgets, but a framework that can be built upon, to create arbitrary visual components. Popular interactive widget libraries include

bqplot (2-D plotting with d3.js)
pythreejs (3-D scene visualization with three.js)
ipyleaflet (maps visualization with leaflet.js)
ipyvolume (3-D plotting and volume rendering with three.js)
nglview (molecular visualization)

Just like the rest of the Jupyter ecosystem, Jupyter interactive widgets were designed as a language-agnostic framework. Other language back-ends can be created reusing the front-end component, which can be installed separately.

xwidgets, which is still at an early stage of development, is a native C++ implementation of the Jupyter widgets protocol. It already provides an implementation for most of the widget types available in the core Jupyter widgets package.

C++ back-end to the Jupyter interactive widgets

Just like with ipywidgets, one can build upon xwidgets and implement C++ back-ends for the Jupyter widget libraries listed earlier, effectively enabling them for the C++ programming language and other xeus-based kernels: xplot, xvolume, xthreejs…

xplot is an experimental C++ back-end for the bqplot 2-D plotting library. It enables an API following the constructs of the Grammar of Graphics in C++.

In xplot, every item in a chart is a separate object that can be modified from the back-end, dynamically.

Changing a property of a plot item, a scale, an axis or the figure canvas itself results in the communication of an update message to the front-end, which reflects the new state of the widget visually.

Changing the data of a scatter plot dynamically to update the chart

Warning: the xplot and xwidgets projects are still at an early stage of development and are changing drastically at each release.

Interactive computing environments like Jupyter are not the only missing tool in the C++ world. Two key ingredients to the success of Python as the lingua franca of data science is the existence of libraries like NumPy and Pandas at the foundation of the ecosystem.

xtensor is a C++ library meant for numerical analysis with multi-dimensional array expressions.

xtensor provides

an extensible expression system enabling lazy NumPy-style broadcasting.
an API following the idioms of the C++ standard library.
tools to manipulate array expressions and build upon xtensor.

xtensor exposes an API similar to that of NumPy covering a growing portion of the functionalities. A cheat sheet can be found in the documentation:

Scrolling the NumPy to xtensor cheat sheet

However, xtensor internals are very different from NumPy. Using modern C++ techniques (template expressions, closure semantics) xtensor is a lazily evaluated library, avoiding the creation of temporary variables and unnecessary memory allocations, even in the case complex expressions involving broadcasting and language bindings.

Still, from a user perspective, the combination of xtensor with the C++ notebook provides an experience very similar to that of NumPy in a Python notebook.

Using the xtensor array expression library in a C++ notebook

In addition to the core library, the xtensor ecosystem has a number of other components

xtensor-blas: the counterpart to the numpy.linalg module.
xtensor-fftw: bindings to the fftw library.
xtensor-io: APIs to read and write various file formats (images, audio, NumPy’s NPZ format).
xtensor-ros: bindings for ROS, the robot operating system.
xtensor-python: bindings for the Python programming language, allowing the use of NumPy arrays in-place, using the NumPy C API and the pybind11 library.
xtensor-julia: bindings for the Julia programming language, allowing the use of Julia arrays in-place, using the C API of the Julia interpreter, and the CxxWrap library.
xtensor-r: bindings for the R programming language, allowing the use of R arrays in-place.

Detailing further the features of the xtensor framework would be beyond the scope of this post.

If you are interested in trying the various notebooks presented in this post, there is no need to install anything. You can just use binder:

The Binder project, which is part of Project Jupyter, enables the deployment of containerized Jupyter notebooks, from a GitHub repository together with a manifest listing the dependencies (as conda packages).

All the notebooks in the screenshots above can be run online, by just clicking on one of the following links:

xeus-cling: the main xeus-cling example notebook,
xtensor: the C++ N-D array expression library in a C++ notebook,
xwidgets: the C++ back-end for Jupyter interactive widgets,
xplot: the C++ back-end to the bqplot 2-D plotting library for Jupyter.

JupyterHub is the multi-user infrastructure underlying open wide deployments of Jupyter like Binder but also smaller deployments for authenticated users.

The modular architecture of JupyterHub enables a great variety of scenarios on how users are authenticated, and what service is made available to them. JupyterHub deployment for several hundreds of users have been done in various universities and institutions, including the Paris-Sud University, where the C++ kernel was also installed for the students to use.

In September 2017, the 350 first-year students at Paris-Sud University who took the “Info 111: Introduction to Computer Science” class wrote their first lines of C++ in a Jupyter notebook.

The use of Jupyter notebooks in the context of teaching C++ proved especially useful for the first classes, where students can focus on the syntax of the language without distractions such as compiling and linking.

Acknowledgements

The software presented in this post was built upon the work of a large number of people including the Jupyter team and the Cling developers.

We are especially grateful to Patrick Bos (who authored xtensor-fftw), Nicolas Thiéry, Min Ragan Kelley, Thomas Kluyver, Yuvi Panda, Kyle Cranmer, Axel Naumann and Vassil Vassilev.

We thank the DIANA/HEP organization for supporting travel to CERN and encouraging the collaboration between Project Jupyter and the ROOT team.

We are also grateful to the team at Paris-Sud University who worked on the JupyterHub deployment and the class materials, notably Viviane Pons.

The development of xeus, xtensor, xwidgets and related packages at QuantStack is sponsored by Bloomberg.

About the Authors (alphabetical order)

Sylvain Corlay, Scientific Software Developer at QuantStack

Loic Gouarin, Research Engineer at Laboratoire de Mathématiques at Orsay

Johan Mabille, Scientific Software Developer at QuantStack

Wolf Vollprecht, Scientific Software Developer at QuantStack