Jupyter Blog - Sylvain Corlay

JupyterLite Officially Joins Project Jupyter!

2026-02-12T16:14:00+00:00

We are thrilled to announce that JupyterLite is now an official part of Project Jupyter. This milestone marks a significant step forward for interactive computing in the browser and strengthens JupyterLite’s role within the Jupyter ecosystem.

What is JupyterLite?

JupyterLite is a JupyterLab distribution that runs entirely in your web browser. Kernels execute directly in the browser using WebAssembly, eliminating the need for an application server. This means:

Instant access: Start computing with a single click: no prior Python setup, environment configuration, or server management required.
Scalability: Host thousands of concurrent users from a static website (e.g., GitHub Pages) with zero per-user server costs.
Privacy and portability: Code and data remain in the user’s browser, making it ideal for embedding in documentation, tutorials, and interactive demos.

JupyterLite greatly expands the options available in the Jupyter ecosystem for education, documentation, and demos, where reducing friction is critical.

As Brian Granger reminded us during his JupyterCon 2025 keynote, Jupyter’s mission is “to empower people of all backgrounds to think, collaborate, and share knowledge using computational storytelling.” From this perspective, JupyterLite is a logical and key next step to take, letting Jupyter take advantage of the impressive strides made in recent years by the WebAssembly/JavaScript ecosystem (in reach and capability) to advance this mission. JupyterLite is an excellent complement to other forms of accessing Jupyter (whether through a local Python installation or a hosted infrastructure service), that facilitates new use cases and lowers barriers in many others.

Interactive computing in the browser with JupyterLite

The Journey of JupyterLite

JupyterLite’s development began in 2021, led by Jeremy Tuloup at QuantStack. Over the past four years, it has benefited from the dedication of several other team members, Martin Renou, Trung Le, and Ian Thomas — as well as invaluable contributions from community members like Nick Bollweg and many others.

Beyond the JupyterLite repository, the project includes comprehensive tooling for creating in-browser language kernels based on xeus. These kernels support languages like Python, R, C++, GNU Octave, Lua, and SQLite, sharing the same codebase as their backend counterparts, developed by Thorsten Beier, Isabel Paredes, Johan Mabille, and Antoine Prouvost. These kernels are built upon the emscripten-forge software distribution for WebAssembly. The project also includes a Python kernel based on Pyodide, and kernels for JavaScript and p5.js. This architecture means that the same language-agnostic model for kernels that Jupyter pioneered over a decade ago, carries over to the WebAssembly world.

The JupyterLite GitHub organization also features the JupyterLite Terminal, a terminal and shell emulator that runs entirely in the browser and was developed by Ian Thomas. It enables the use of basic Unix commands like grep, sed, cat, touch, and even vim or nano, all built to WebAssembly.

The JupyterLite terminal emulator, with basic Unix commands

JupyterLite now powers the official Jupyter website and is used by projects like numpy.org, sympy.org. It also underlies services such as Jupyter Everywhere and notebook.link.

A Natural Fit with JupyterLab

JupyterLite has always been closely tied to JupyterLab. It is increasingly becoming a set of JupyterLab extensions that replace core plugins to manage kernels, settings, and content in the browser. With contributors overlapping significantly with the Jupyter Frontends group, this integration formalizes what many in the community already recognized: JupyterLite is a core part of the Jupyter ecosystem.

The proposal to transfer JupyterLite to the Jupyter governance received strong support from the Jupyter community and by the Jupyter Frontends council.

Better integration with the rest of Jupyter

With JupyterLite now being an official part of Jupyter, it will be easier to find areas for better integration with other aspects of our ecosystem: we can reduce duplication, smooth out the documentation and model for creating and using both “traditional” (server-hosted) and WebAssembly kernels, and make JupyterLite a natural instant-access component of MyST/JupyterBook-based sites, and more.

Mission-wise, JupyterLite is a natural next step for Jupyter, and having it be an official part of the project makes it much easier for the community to integrate its benefits throughout. We hope you’ll try it, use it and contribute to its growth!

Try it in your browser

If you would like to try JupyterLite in your browser, click on the following link:

Acknowledgments and team credits

JupyterLite’s success is thanks to the support of individuals and organizations who believed in its vision. This includes:

QuantStack, for their continued investment in the project since 2021 and the broader Jupyter ecosystem,
Bloomberg, for funding improvements to JupyterLite by QuantStack since 2023,
The Gates Foundation, for funding the development of the xeus-R kernel and its port to WebAssembly,
The Pyodide project, whose pioneering work on Python in the browser via WebAssembly made JupyterLite possible.

Key developers to this stack include:

Jeremy Tuloup (Director at QuantStack, creator of JupyterLite, and JupyterLab maintainer),
Nicholas Bollweg (JupyterLite maintainer and #2 all-time committer),
Martin Renou (Director at QuantStack, JupyterLite maintainer, responsible for integrating xeus with emscripten-forge, and creator of JupyterLite-Sphinx),
Thorsten Beier (Software developer at QuantStack, lead developer of emscripten-forge, and xeus-lite),
Isabel Paredes (Software developer at QuantStack, led the packaging of R and GNU Octave in Emscripten-forge),
Anutosh Bhat (Software developer at QuantStack, C++ kernel in the browser),
Ian Thomas (Software developer at QuantStack, creator of the JupyterLite terminal, and the cockle shell emulator),
Johan Mabille (Director at QuantStack, creator of xeus),
Agriya Khetarpal (Pyodide contributor, JupyterLite-Sphinx maintainer),
Albert Steppi (JupyterLite-Sphinx maintainer).

Thank you for being part of this journey, we cannot wait to see what you build with JupyterLite!

ipydatagrid is now part of Project Jupyter

2024-08-22T15:01:00+00:00

Today, we are proud to announce that the ipydatagrid open source project has been incorporated into Project Jupyter as part of the Jupyter Widgets subproject.

What is ipydatagrid?

ipydatagrid is a fast data grid widget for Jupyter Notebooks and JupyterLab. Since its inception in 2019, it has been developed as an open source project in Bloomberg’s GitHub organization.

It offers a high-performance fully-featured DataGrid interface, fully integrated with ipywidgets. Built upon the Lumino datagrid, which also powers the JupyterLab CSV viewer, ipydatagrid provides users with a robust and versatile tool for data visualization and manipulation.

Key Features

Users can customize the way data is represented in their grid using a variety of renderers.

ipydatagrid includes a sophisticated selection model with two-way data binding.

It enables conditional formatting powered by Vega Expressions.

The transfer to Project Jupyter

We would like to extend our gratitude to the developers and contributors who have significantly improved the ipydatagrid project over the years: Itay Dafna, Martin Renou, Mehmet Bektas, Kaia Young, Bernát Gábor, Vasilis Themelis, Supriya K., Greg Mooney, Ian Thomas, Olly Hensby, and John Gunstone. Special thanks to Chris Colbert, the creator of the Lumino Datagrid, which forms the foundation of ipydatagrid’s front-end.

Bloomberg’s contribution to Project Jupyter

The transfer of ipydatagrid to Project Jupyter is just one of many contributions that Bloomberg has made to Project Jupyter. As a long-standing sponsor, Bloomberg has been instrumental in the development and success of Jupyter. Home to core maintainers and a major funder of JupyterLab since its inception, Bloomberg has also sponsored all editions of JupyterCon. The sustained support by Bloomberg has been crucial to Jupyter’s growth and impact.

We are excited about the future of ipydatagrid within Project Jupyter, and look forward to continued innovation and collaboration within the broader community.

— on behalf of the Jupyter Widgets Council, Sylvain Corlay

JupyterGIS

2024-06-12T17:09:00+00:00

We are thrilled to announce that the European Space Agency (ESA) is funding our proposal “Real-time collaboration and collaborative editing for GIS workflows with Jupyter and QGIS.”

The consortium spearheading this project comprises QuantStack and Simula Research Lab (Simula), two organizations with a long history of contributions to the Jupyter project and the broader open-source scientific computing ecosystem.

The goal of the project is to build solid foundations for a versatile web-based user interface for Geographic Information Systems (GIS) workflows. It will comprise several components, including a JupyterLab extension for collaboratively editing QGIS project files and the integration of these APIs in the Jupyter Notebook. We will then explore integration with platforms like the Copernicus Data Space Ecosystem (CDSE) and the European Open Science Cloud (EOSC).

Collaborative workflows in geosciences

Collaborative editing of documents has become an integral part of our digital lives and has made us collectively more productive. Gone are the days of cumbersome email exchanges with documents shuttling back and forth.

Looking ahead, the potential of co-editing extends far beyond text documents and will apply to all authoring UIs, from CAD to image processing. We think that the shift to collaborative editing will be even more transformative for larger and more complex projects, which are inherently social and require the concerted effort of large teams. Whether you are designing a stadium, a plane, or an ocean liner, you need to coordinate a diverse expertise to build a unified model. In geosciences, it may range from climate modeling to agriculture, ecology, urban planning, and many more areas of expertise. For such endeavors, we must embrace tools favoring collaboration, and this applies to the future web-based user interfaces for geoscience research.

We have been working on collaborative editing in the core of JupyterLab for the past three years. Our approach is based on the Yjs framework, an implementation of CRDT data structures (Conflict-free Replicated Data Type). We have learned that since it is so tied to the data model, retrofitting these features into an existing application is considerably more arduous than building the initial data model on the appropriate paradigm from inception. This is why JupyterGIS will be built from the ground up with collaborative editing in mind.

JupyterGIS will be the first open-source GIS tool to provide collaborative editing features.

Importantly, JupyterGIS is part of a broader mission to enable collaborative workflows in open-source technical computing. The JupyterCAD project, a collaborative CAD modeler, is another example of this endeavor. By sharing knowledge and resources, both JupyterGIS and JupyterCAD will benefit from and contribute to each other’s progress, ultimately driving innovation and productivity in their respective fields.

From the desktop to the web

Advanced authoring tools, including IDEs, CAD modelers, image processing software, and GIS applications, are essential for professionals who rely on them for extended periods. These users have high expectations and demand key features to optimize their productivity and workflow. Among these features are extensibility with plugins, configurable keyboard shortcuts, themability, internationalization, scriptability, a unified settings system, and the ability to operate across multiple browser windows and devices.

Developing a new application from the ground up that meets all these requirements is a formidable challenge. This is where the JupyterLab application framework proves invaluable. By leveraging this framework, developers can build custom, feature-rich authoring tools that incorporate these essential features from the outset, making it an ideal solution for creating modern, user-centric applications.

Beyond the pure authoring user interface, for which JupyterLab is a great tool, it is also the perfect tool for integration with notebook-based workflows. As part of this project, we will develop an advanced Python API to manage JupyterGIS sessions, leveraging Jupyter’s robust display system to incorporate sophisticated GIS features inline in Jupyter notebooks and consoles. This integration will further enhance the capabilities and versatility of our application. This feature will follow the same architecture as that of JupyterCAD for the integration with the notebook user interface.

Real-world applications

JupyterGIS is meant to become a general-purpose tool. However, we will work with teams of practitioners on specific real-world applications to ensure that it addresses their needs.

One such project concerns the use of digital and GIS solutions for emergency management. While practitioners involved in emergency management already have experience with such tools, we are convinced that web-based applications built with collaboration in mind from the start can foster improved coordination and collaboration, and therefore the effectiveness of the response.

Response teams comprise different organizations, including the affected municipalities, police, emergency response organizations like the Red Cross, and advisers such as the U.S. Army Corps of Engineers (USACE) in the United States or the Norwegian Water Resources and Energy Directorate (NVE) in Norway. Even when using the same incident management system, miscommunications and misunderstandings can occur between these stakeholders, resulting in delays or improper use of resources. An improved integration of collaborative GIS software can improve upon the existing tooling.

Building collaborations in Europe and beyond

We will partner with key practitioners and make sure JupyterGIS addresses their use cases. Integration in the Copernicus Data Space Ecosystem (CDSE) and the European Open Science Cloud (EOSC) will be key to the adoption of JupyterGIS and demonstrate its deployment in such environments.

Beyond Europe, we are currently working on including this project into a broader scope. We are excited to be partnering with the “LIVE-Env” project, an open-source initiative spearheaded by Alyssa Goodman at Harvard University. Furthermore, we are establishing a key collaboration with the Berkeley Institute for Data Sciences (BIDS) and the Schmidt Center for Data Science and Environment (DSE) at UC Berkeley and 2i2c, with a focus on Jupyter-based geosciences.

Affiliations

Sylvain Corlay: QuantStack
Anne Fouilloux: Simula
Monika Weissschnur: Simula

🎉 JupyterCon 2023 recordings now live on YouTube! 🎉

2023-07-22T21:14:00+00:00

Get ready to relive the magic of JupyterCon 2023, because the long-awaited moment is finally here! The JupyterCon YouTube channel has just dropped a treasure trove of content — all the talk and keynote recordings from the most epic conference of the year.

Held at the Cité des Sciences in the city of Paris, from May 10 to 12, JupyterCon 2023 was a celebration of all things Jupyter and beyond.

You can now access all the knowledge-packed sessions online. Don’t miss this opportunity to be inspired by visionaries and experts from different fields who shared their insights on the future of Jupyter.

We extend our heartfelt gratitude to the sponsors, speakers, attendees, and volunteers who contributed to the vibrant atmosphere and made JupyterCon 2023 a memorable experience. Together, we’ve created a platform for knowledge sharing, innovation, and collaboration, empowering data scientists, researchers, educators, and developers worldwide. Thank you for making this conference possible!

On behalf of the JupyterCon 2023 organization committee,

Gayle Ollington — Jupyter Community Events Manager
Sylvain Corlay — JupyterCon 2023 General Chair

Improving the accessibility of Jupyter

2023-02-24T11:11:00+00:00

The adoption of Jupyter has grown significantly in the past few years — especially in education contexts. The project has become a foundational component of our digital experience, from the first lines of code written by high-school students to the most advanced research and engineering use cases.

In this context of global adoption, it is even more important to ensure that as many people as possible can use the project. It is estimated that 15% of the population has a disability that may impair their ability to use online services. If we don’t want them to be excluded from learning sciences, technology, and engineering, we must improve the tools to make them usable by everyone…

To address this issue, a Jupyter accessibility working group formed and has now become an official Jupyter Subproject and received a grant from the Chan Zuckerberg Initiative.

Creating accessible user interfaces is a challenging task as it requires accommodating a broad range of disabilities, including vision, motor, and cognitive impairments. This article presents some of the recent accessibility improvements in the Jupyter Notebook codebase.

CodeMirror 6 and Notebook 7

Jupyter Notebook 7, which is the next major release of the Jupyter Notebook frontend, has been rebuilt with modern JupyterLab components and extensions while maintaining the classic Jupyter Notebook user experience. Notebook 7 will be released shortly after JupyterLab 4 and will take advantage of JupyterLab improvements.

Improving the accessibility of Jupyter had long been impeded by significant obstacles. The primary obstacle was that the text editor underlying the Jupyter Notebook (CodeMirror 5) had major accessibility issues.

Fortunately, this accessibility bottleneck has been unblocked as JupyterLab has been upgraded to use CodeMirror 6, a complete rewrite of the text editor with a strong focus on accessibility. Although this upgrade required extensive codebase modifications, the changes landed a few months ago and will be available with JupyterLab 4. Jupyter Notebook 7 will benefit from the CodeMirror 6 upgrade.

Initial accessibility audits

Shortly after the CodeMirror 6 migration landed, we made an automated audit of the accessibility of Notebook 7 with these changes, and found that the number of warnings and errors reported by Axe Auditor went down from several hundreds to a few dozen, most of which seemed fixable. Encouraged by these results, we decided to work on bringing that count to zero!

With a series of fixes ranging from simple changes to the DOM structure of components to fixing up base Lumino components (Lumino is a JavaScript framework that underlies a lot of the JupyterLab frontend architecture), we were able to make the Notebook 7 codebase pass the Axe Auditor tests with zero error or warning. While we have not reached that yet in JupyterLab, both JupyterLab and Jupyter Notebook benefit from these improvements made for Notebook 7 since they are built from the same components.

Axe Auditor output with the “classic” Notebook UI (v6) with a test notebook.

Axe Auditor output with the “next” Notebook UI (v7 alpha 13) with the same test notebook.

Similar results were achieved with the file explorer and text editor tabs of the application.

Beyond accessibility audits

Improving the accessibility of Jupyter requires more than just addressing issues flagged by automated audit tools and the Jupyter Accessibility Subproject is working on improving accessibility across the board in Project Jupyter. These results with the Notebook 7 codebase are very encouraging, but working with end-users and getting their feedback will also be necessary to make the user interface truly accessible.

To address the diversity of accessibility requirements, we will approach specific use cases separately (for example, users with screen readers, or users who can operate a keyboard but not a pointing device). While the resulting changes may improve usability for everyone, we need to learn from users who have specific needs to make meaningful improvements.

Finally, many of the accessibility challenges in Jupyter stem from the content of the notebooks, as notebook authors may not follow the best practices to make their content usable by everyone (such as adding alt text to images and figures, or properly using headings to communicate the organization of a notebook). To promote these best practices, Jupyter could provide linting tools that produce inline warnings and hints for notebooks that do not follow these guidelines.

Acknowledgements

The work by Johan Mabille at QuantStack on migrating JupyterLab to use CodeMirror 6 was funded by Two Sigma. This upgrade also enabled significant performance improvements in the rendering of Jupyter notebooks, as detailed in this earlier post by Frédéric Collonval.

We are grateful to members of the Jupyter Accessibility Subproject who produced a thorough review of the Notebook 7 UI with a focus on keyboard navigation. This is an example of an issue that cannot be easily detected by auditing tools.

About the Authors

Nicolas Brichet is a scientific software developer at QuantStack active in the Jupyter ecosystem. Among other things, Nicolas made significant contributions to the JupyterLab and Voilà projects, and worked on porting the nbgrader package to JupyterLab. Nicolas worked on accessibility improvements in the JupyterLab and Lumino packages.

Johan Mabille is a technical director at QuantStack. He was honored with the Distinguished Contributors award in 2020 for his contributions to the Jupyter project. Among other things, he is one of the main authors of the JupyterLab visual debugger, and the creator of Xeus, a C++ implementation of the Jupyter kernel protocol at the basis of many Jupyter kernels. Johan was responsible for the migration of JupyterLab to CodeMirror 6.

Jeremy Tuloup is a technical director at QuantStack, and a Jupyter Distinguished Contributor. He is a core maintainer of JupyterLab and Voilà, and the creator of the JupyterLite project. Jeremy is the main initiator of the Notebook 7 project.

Frédéric Collonval is a technical director at QuantStack, and a Jupyter Distinguished Contributor. He is a core maintainer of the JupyterLab project. Frédéric contributed to the CodeMirror 6 migration and helped numerous new contributors to get their enhancements accepted.

Sylvain Corlay is the founder and CEO of QuantStack, and a Jupyter Distinguished Contributor. He has worked on many areas of the Jupyter project, from interactive widgets to language kernels and other core components.

JupyterCon 2023 keynotes

2023-01-18T20:39:00+00:00

Alyssa Goodman, Paul Romer, Cory Gwin, Craig Peters

We are excited to announce the keynote speakers for JupyterCon 2023!

First up, we have Alyssa Goodman from Harvard University. Alyssa is a professor of astronomy at Harvard University, and was the founding director of the Harvard Initiative in Innovative Computing. She will be sharing her insights on how Jupyter, together with the glue project, opens up new opportunities for exploratory data exploration and visualization in the field of astronomy and beyond.

Next, we have Paul Romer from NYU. Paul is a Nobel laureate economist and former Chief Economist of the World Bank. He will describe the potential for Jupyter to revolutionize not just how scholars do their research but also how they communicate their results.

Finally, we have a joint talk from Craig Peters and Cory Gwin from GitHub. They will be discussing the development and capabilities of GitHub Codespaces, and how Jupyter is integrated into this powerful tool for code collaboration and development.

We are thrilled to have such an inspiring group of keynote speakers sharing their expertise at JupyterCon 2023. JupyterCon will be held at the Cité des Sciences in Paris on May 10–12, 2023.

Don’t miss out on this opportunity to hear from these inspiring speakers and get your tickets now at https://www.jupytercon.com/. We look forward to seeing you all there!

— Sylvain Corlay, JupyterCon 2023 General Chair

Jupyter Community Workshop: Jupyter for Education

2022-12-05T23:37:00+00:00

We are excited to announce the next in-person Jupyter Community Workshop! It will focus on the use of Jupyter for Education.

The event will be held at the Conservatoire National des Arts et Métiers (CNAM) headquarters in Paris, France, from January 24th to January 26th, 2023. Funding for travel expenses is available for attendees from academia and those from groups which are not well-represented in the Jupyter and wider tech community!

Jupyter Community Workshop are a series of community-organized events to tackle challenging development and design projects, growing the community of contributors, and strengthening collaborations.

This workshop will focus on the use of Jupyter for education. The goal is to bring together contributors, and community members to further the development of Jupyter-based tools and practices for education. Given the broad variety of topics relevant to this workshop (pedagogical practices, automatic grading, deployment challenges), we will compose the program based on the attendance. We are also planning an installment of the PyData Paris meetup at CNAM on the 26th of January, with a focus on open-source tools for education.

The workshop will last three days, with hands-on discussions, hacking sessions, and technical presentations. The goal of this event is to foster collaboration and the sharing of knowledge between Jupyter maintainers and downstream library authors and power users.

Should you be interested in joining us for this workshop, please fill this form. A limited number of spots are available for this event.

We are grateful to CNAM for hosting us and to INRIA for supporting this event. We are also grateful to the sponsors of the Jupyter Community Workshop series, Bloomberg and Amazon AWS.

Avion III de Clément Ader, at the CNAM — photo by Roi Boshi for Wikimedia Commons

MyBinder @ OVHcloud

2022-11-23T11:28:00+00:00

OVHcloud is a long-time supporter of the Jupyter project. In the past few years, they have provided computing resources to several Jupyter-related endeavors to support its sustainability.

In 2019, Project Jupyter and OVHcloud started collaborating on the hosting of the MyBinder service. With tens of thousands of sessions daily, MyBinder was increasingly dependent on a single instance on a single cloud provider. The offer of OVHcloud triggered a community effort to make MyBinder truly multi-cloud. This is how the Binder Federation was created, with OVHcloud as its first member. Since then, the Binder Federation has grown with new members gracefully hosting a portion of the traffic. OVHcloud also hosts the nbviewer.org service.
When the global pandemic hit in 2020, the JupyterCon organizers decided to pivot from an in-person event set to happen in Berlin to a fully online experience. Lorena Barba, the general chair of the committee, envisioned an experience for talks and tutorials in which attendees could interact with the technology. OVHcloud stepped up as the Platinum sponsor, gracefully providing the infrastructure for the various deployments of Jupyter required for the conference, including GPUs for the online tutorials requiring them.

A new partnership with NumFOCUS

Today, we are thrilled to announce that OVHcloud is partnering with NumFOCUS to provide infrastructure to affiliated projects.

Two pilot projects will benefit from this new agreement: Pandas and Jupyter.

On the Jupyter side, these new resources will be used to increase the OVHcloud hosting of MyBinder, thereby helping the project’s sustainability in the long term, continuing to avoid dependency on a single cloud provider. A significant proportion of the traffic of the Binder federation will now be handled by the OVHcloud deployment.

Acknowledgement

We are grateful to Kevin Amil, Marie-Christine Ribeiro, Marie Hering, and Mael Le Gal from OVHcloud for facilitating this new partnership and helping with the migration to the new infrastructure.

We are grateful to OVHcloud for their continued support to the Jupyter project and the NumFOCUS foundation.

Jupyter Community Workshop: JupyterLite

2022-11-10T15:45:00+00:00

We are thrilled to announce the next in-person Jupyter Community Workshop, which will focus on the JupyterLite project!

The event will be held at the OVHCloud headquarters in Paris, France, from December 7th to December 9th, 2022. Funding for travel expenses is available for attendees from academia and those from groups which are not well-represented in the Jupyter and wider tech community!

Jupyter Community Workshop are a series of community-organized events to tackle challenging development and design projects, growing the community of contributors, and strengthening collaborations.

This specific workshop will focus on the JupyterLite project, a JupyterLab distribution that runs entirely in the browser built from the ground-up using JupyterLab components and extensions. JupyterLite allows for very scalable deployments, and already powers inline consoles and notebooks on the websites of major projects of our ecosystem, such as NumPy, SymPy, Pandas, and many more.

Should you be interested in joining us for this workshop, please fill this form. A limited number of spots are available for this event.

We are grateful to OVHCloud for hosting this event. We are also grateful to the sponsors of the Jupyter Community Workshop series, Bloomberg and Amazon AWS.

Jupyter Community 2021 Update

2021-12-20T17:53:00+00:00

Attendees to the June 2019 Community Workshop on Dashboarding in Paris (Photo credit to Lindsey Heagy)

New committee announcement

Project Jupyter is happy to share some exciting news in key efforts to connect the global Jupyter community. First, we’d like to introduce a new committee that was created to act on behalf of the Project Jupyter Steering Council with the objective of growing, building, and connecting the global Jupyter community of users and contributors. You can learn more about the Jupyter Community Building committee here.

We’re hiring!

We are hiring a Jupyter Community Events Manager to help manage Jupyter Community Workshops and the JupyterCon conference. Check out the blog post about the position at NumFOCUS blog.

Jupyter Community Workshops

The Jupyter Community Workshop program has been an outstanding success in bringing together small groups of Jupyter community members and core contributors for high-impact strategic work and community engagement on focused topics. Community members have hosted 11 community workshops since 2018 exploring using Jupyter in specific disciplines (education, scientific facilities with supercomputers), promoting Jupyter around the world (Hawaii, South America, D.R. Congo), and developing core Jupyter projects and attracting new contributors (Jupyter server, Jupyter widgets, Jupyter kernels, nbgrader, Voilà).

Many more workshops were in planning stages when the COVID-19 pandemic abruptly paused in-person gatherings globally in March, 2020. All workshops were put on hold until 2021, and organizers are currently working through how to best proceed in the still-uncertain environment around global in-person gatherings (for example, scheduling far into the future, or moving to a virtual gathering). Two upcoming workshops have been scheduled for 2022 (on accessibility in Jupyter and Jupyter in musculoskeletal research), and more workshops are in the planning stages. Stay tuned for more details.

JupyterCon

In 2017 and 2018, our first two global user conferences were made possible through a grant from the Leona M and Harry B Helmsley Charitable Trust and a partnership with O’Reilly Media and NumFOCUS.

Unlike in 2017 and 2018, JupyterCon 2020 was entirely led by the community. Together with NumFOCUS, a team of community volunteers made the event possible, despite the enormous challenges from the COVID-19 pandemic. With over 170 tutorials, talks, and posters, and over 700 attendees, JupyterCon 2020 was a huge success. We are incredibly grateful to all the volunteers who worked on the conference and to all the participants for their contributions.

Acknowledgements

Many thanks to our generous sponsors! These community building efforts, including the hiring of an Event Manager, is made possible through significant donations from our partners at Bloomberg and AWS.

About the Authors

This post was written by the Jupyter Community Building Committee, currently composed of Ana Ruvalcaba, Jason Grout, and Sylvain Corlay.

Enabling the JupyterLab debugger with ipykernel

2021-05-13T11:45:00+00:00

Support for the Jupyter Debugger Protocol just landed in ipykernel

JupyterLab 3.0 includes a visual debugger that allows to interactively set breakpoints, step into functions, and inspect variables with any Jupyter kernel that implements the Jupyter debugger protocol.

The first two language kernels to implement the new protocol were xeus-python (a Python kernel) and xeus-robot (a kernel for Robot Framework). Unfortunately, the reference Python kernel, ipykernel, did not support debugging yet, until now!

Today, we are pleased to announce that debugging support landed in ipykernel, and will be available in the next major release, ipykernel 6.0. Pre-releases including the ipykernel debugger are available.

Debugging with ipykernel

What will change with ipykernel 6.0?

Enabling support for debugging in ipykernel required important changes in the code base regarding the concurrency model of the kernel. The main change is that the processing of messages on the “control channel” now happens in a different thread, allowing for the processing to happen while user code is running.

Ipykernel 6.0 includes several other updates. Tornado coroutines were dropped in favor of native coroutines. The Matplotlib inline backend was split into a separate package, and ipykernel depends on debugpy, an implementation of the Debug Adapter Protocol for Python.

If you are interested in testing out the new features, check out the beta release!

pip install ipykernel --pre

Try it now!

Thanks to MyBinder, you can also try it out without the need of installing anything on your computer. Just follow this link:

What about the future?

A lot of new features are in the works with the JupyterLab debugger.

JupyterLab 3.1 will include several usability improvements to the debugger.
It will also add the ability to submit code for execution when stopped at a breakpoint.
We are working on a richer variable explorer, using Jupyter’s rich display system to enable the rich-rendering of variables in the explorer, to e.g. render dataframes as tables.

Finally, we also plan on adding debugging support to other language kernels.

Acknowledgements

The work of Johan and Sylvain at QuantStack on the debugger support in ipykernel was funded by Two Sigma. We are grateful to Min Ragan Kelley and Matthias Bussonnier, who reviewed the pull requests on debugger support.

About the authors

Johan Mabille is a scientific software developer at QuantStack.

Johan is very active in the Jupyter ecosystem, as the creator of xeus, a C++ implementation of the Jupyter protocol, and several language kernels, such as xeus-python, xeus-cling, and xeus-robot. Johan also made contributions to the Jupyter widgets ecosystem and to JupyterLab.

Sylvain Corlay is the founder and CEO of QuantStack.

As an open-source developer, Sylvain is very active in the Jupyter project with contributions in several components of the stack, including widgets, kernels, nbconvert, and others. He is also a steering committee member of the project.

Sylvain also does volunteer work for the community, as member of board of directors of NumFOCUS, co-organizer of the PyData Paris Meetup, and vice-chair of JupyterCon 2020.

A Curiously Recurring Widget Library

2021-01-27T14:22:00+00:00

Interactive widgets allow Jupyter users to create user interfaces inline in their notebooks, and to turn them into standalone applications with tools such as Voilà.

Language backends for Jupyter interactive widgets exist in Python (with ipywidgets), and C++ (with xwidgets, and the xeus-cling C++ Jupyter kernel), reusing the same frontend implementation of the widgets in JavaScript.

Simple interactive widgets at play in JupyterLab with the xeus-cling C++ kernel

In this article, we dive into some of the C++ techniques used in the implementation of the xwidgets library, including

discussions on value semantics and RAII (Resource Acquisition Is Initialization),
an original application of CRTP (Curiously Recurring Template Pattern),
an original implementation of the observer pattern, xproperty,
a plea for allowing the overloading of the dot operator in C++, to enable better proxy types.

For readers interested in knowing more about interpreted C++, we recommend the following posts:

Interactive workflows for C++ with Jupyter (Jupyter blog), by Sylvain Corlay, Loic Gouarin, Johan Mabille, and Wolf Vollprecht

Interactive C++ for Data Science (LLVM blog), by Vassil Vassilev, David Lange, Simeon Ehrig, and Sylvain Corlay

Interpreted C++ for GIS with Jupyter (Jupyter blog), by Martin Renou

Jupyter widgets are special objects that trigger the creation of a counterpart JavaScript model object in the Jupyter frontend upon creation. The state of the object in the backend is synchronized with the state of the JavaScript frontend object. Views of that widget model are instantiated upon display.

The MVC (Model View Controller) architecture of Jupyter widgets, and synchronization with the backend

This MVC (Model-View-Controller) architecture for Jupyter interactive widgets allowed us to reuse all of the frontend implementation, by simply providing an alternative backend in C++, implementing the same messaging protocol.

In order to tie the lifetime of the kernel and frontend objects, we decided to use the RAII (Resource Acquisition Is Initialization) pattern. RAII is a common programming idiom that consists of tying the lifetime of an object with the holding of a resource. Most typically, the resource is

acquired in the constructor of the object,
released in the destructor of the object.

The RAII pattern is not common in garbage-collected languages because unlike in C++, the time when objects are destroyed is not deterministic. The Python programming language mitigates that issue by introducing context managers, often used to e.g. open and close files.

A consequence of relying on object lifetime for resource management in C++ is to adopt the “value semantics” for widget instances, instead of “reference semantics” which is more typical for widget frameworks like Qt.

Note: value semantics vs reference semantics

If you are not familiar with the concepts of value and reference semantics in C++, I recommend reading the excellent FAQ of isocpp.org.

To summarize:

with value semantics, objects hold actual values, and copying an object copies their attributes. With value semantics, one should provide implementations of copy, and move constructors, as well as copy and move assignment operators. Besides, values should not have virtual methods.

with reference semantics, objects are manipulated through references (or pointers) and never by value. Reference semantics is typically used for polymorphic programming with virtual methods. With reference semantics, it is recommended to delete copy and move constructors, as well as copy and move assignment operators to avoid accidental copies when passing objects to functions taking arguments by values, causing object slicing. (They can also be made private). Explicit cloning of a reference semantics object is generally allowed via a call to a clone virtual method.

In a C++ codebase, the existence of public copy or move constructors alongside virtual methods in the same class is generally a sign of a bad design.

The use of value semantics for xwidgets provides clear lifetime management for associated resources. Careful use of the move semantics provides fine-grained control.

Illustration of the move semantics for xwidgets

Another advantage of using value semantics in xwidgets is that C++ beginners who are typical users of the Jupyter notebook (often used by instructors) can easily manipulate “widgets as values” without having to deal with manual memory allocation etc.

With value semantics, addressing the lifetime of objects becomes the responsibility of the framework author, in a carefull implementation of (copy and move) constructors, destructors, and (copy and move) assignment operators.

Static polymorphism and the CRTP pattern

While the use of value semantics provides fine-grained control over the lifetime of widgets, it prevents the use of virtual methods in their implementation. Code reuse is achieved with static polymorphism techniques and specifically the CRTP pattern.

The Curiously Recurring Template Pattern (CRTP) is the practice of making a class X derive from a class template instantiation using X itself as a template argument.

class X : public base<X>

CRTP is commonly used by matrix or tensor algebra libraries making use of expression templates (such as xtensor, eigen, or blitz) to prevent the overhead of virtual function dispatch for operations likely to be performed in a loop, such as element access.

Placing common code in a CRTP base allows code reuse without the overhead of virtual dispatch. However, the template base class is specialized for each final type increasing the resulting binary size. This can be mitigated by factoring as much of the code in a base class that would not be templated by the derived type.

CRTP in xwidgets — closing the recursion

In order to allow for code reuse without virtual inheritance, xwidgets’ class hierarchy is entirely based on CRTP. Our naming scheme is that CRTP bases, which should not be instantiated directly are prefixed with the letter x and final concrete widget types are not.

Upon construction of e.g. the button widget, constructors of base types are called in the order of inheritance, which is why we need to establish the connection with the frontend in the constructor of the most derived type, after all attributes have been initialized, and send a message to the frontend with all the values.

The pattern for creating the most derived type being always the same, we defined the xmaterialize template class closing the CRTP hierarchy with

using button = xmaterialize<xbutton>;

The xmaterialize template class is defined as final, to prevent further inheritance. The constructors and assignment operators forward to those of the CRTP bases and implement the RAII pattern.

template <template <class> class B, class... P>
class xmaterialize final : public B<xmaterialize<B, P...>>
{
public:

    using self_type = xmaterialize<B, P...>;
    using base_type = B<self_type>;

    template <class... A>
    xmaterialize(A&&...)
        : base_type(std::forward<A>(args)...)
    {
        this->open();      //  RAII: create the frontend model.
    }

    template <template <class> class B, class... P>
    inline xmaterialize<B, P...>::~xmaterialize()
    {
        if (!this->moved_from())
        {
            this->close();  // RAII: delete the frontend model.
        }
    }

    ...

};

The full implementation of xmaterialize (as of xwidgets 0.25) is available here. Beyond the logic described in this section, it also includes the handling of the move semantics and the method chaining API which is the subject of a later section.

Precompilation and binary size optimization

Even though xwidgets is fully based on template types, we decided to precompile all final widget types for faster interactive use with the xeus-cling kernel. This is achieved with

an extern declaration in the header (here in xbutton.hpp)
```
extern template class xmaterialize<xbutton>;
```
and in the source file (here in xbutton.cpp), an instruction for the precompilation
```
template class XWIDGETS_API xmaterialize<xbutton>;
```

Doing this for all widget types of the library initially resulted in a large compiled binary size. Using the button widget as an example, we see that base types are templated by the final type in the class hierarchy button -> xbutton<button> -> xwidgets<button> -> xobject<button>, and therefore, their binary representation is duplicated for each final type.

A strategy for reducing the binary size has been to factor out as much of the logic of xobject<D> in a non-template base xcommon improving compilation speed and preventing binary code duplication.

Xproperty: an implementation of the observer pattern

In order to update the frontend upon changes of widget properties, xwidgets relies on an implementation of the observer pattern called xproperty. xproperty is to xwidgets what traitlets are to ipywidgets.

In order to trigger observers and validators in the owner object upon assignment of new values, xproperty relies on the overload of the assignment operator =.

template <class T, class O>
template <class V>
inline auto xproperty<T, O>::operator=(V&& value) -> reference
{
    // Before assigning the new value, invoke validators
    // which may also mutate the value.
    m_value = owner()->template invoke_validators<T>(m_name, std::forward<V>(value));

    // Call class-level observer.
    owner()->notify(m_name, m_value);

    // Call registered observers for that attribute
    owner()->invoke_observers(m_name);

    // Return the new value 
    return m_value;
}

Jupyter interactive widgets have many attributes that may be specified at construction time.

In the Python implementation, this is handled with keyword arguments, but the C++ programming language does not support keyword arguments. There exist various approaches to enable this feature with advanced metaprogramming techniques. In the case of xwidgets, this need is limited to the initialization of xproperty attributes, which allowed us to adopt a more scoped approach: a method chaining API for property initialization:

auto slider = slider<double>::initialize()
  .min(-1.0)
  .max(1.0)
  .description("Another slider")
.finalize();

this was enabled by overriding the function call operator () on xproperties to pass an intial value (only when the said property is an rvalue). A static initialize method is used instead of the default constructor to prevent the initialization of the JavaScript counterpart while all attributes may not have been set yet. The frontend counterpart is only acquired with the finalize() call.

Building upon xwidgets

Jupyter interactive widgets are not limited to the controls available in the core package. In fact, there is a rich ecosystem of widget libraries built upon the core framework: ipyvolume (3-D plotting), ipyleaflet (maps visualization), bqplot (2-D plotting), ipygany (3-D mesh visualization), ipycanvas (generic drawing), ipywebrtc (streaming video and audio), and many many more.

For the C++ programming language, we have already provided:

xleaflet (the C++ equivalent to ipyleaflet, reusing the same frontend),
xwebrtc (the C++ equivalent to ipywebrtc, reusing the same frontend).

Screencast of xleaflet in JupyterLab, loading and visualizing a GeoJSON dataset in a C++ notebook.

Potentially, C++ backend to all Jupyter interactive widget packages could be provided, creating a huge opportunity for interactive data visualization in C++.

C++ should allow overloading the dot operator

xwidgets and xproperty makes heavy use of value semantics, and proxy objects.

At the moment, to access an attribute or method of a value held in an xproperty object, we must first call the function call operator () to access the undelying object first, which is cumbersome.
This is also an issue in other places in the xwidgets stack, when making use of e.g. reference proxies.

Being able to automatically map all methods of the underlying type to be accessible in the xproperty would be incredibly powerful, and remove the need for explicitly accessing the underlying.

Interestingly, the C++ standard does have the equivalent operator overload when it comes to pointer semantics, with the arrow operator ->. If overloading the dot operator . was allowed, this could enable this kind of usecase:

struct V
{
    void f();
};

struct X
{
    V& operator.() { return m_value; }
    V m_value;
};

in which case, X::f would call V::f .

Allowing the overloading of the dot operator . would also enable usecases such as smart references (similar to smart pointers, but with value semantics) and fully-fledged reference proxies (such as the return type of operator[] for std::vector<bool>).

Try it online!

Thanks to MyBinder, you can try xwidgets without the need of installing anything on your computer. Just follow this link:

About the author

Sylvain Corlay is the founder and CEO of QuantStack, an open-source software development studio comprising maintainers of key projects of the scientific computing ecosystem.

As an open-source developer, Sylvain is very active in the Jupyter project, contributing to the Xeus stack, Jupyter interactive widgets, Voilà dashboards. He is a member of the Jupyter steering committee and was the vice chair of JupyterCon 2020. Sylvain also contributes to the conda-forge project and he is the co-creator of the Xtensor C++ tensor algebra library.

Benjamin Ragan-Kelley

2020-10-03T21:02:00+00:00

Benjamin Ragan Kelley

It all started in April 2005 when freshman Min RK reached out to his physics professor, Brian Granger,

Dr. Granger,
Talking to you this afternoon prompted me to put the rather vague idea I had into a less vague writing, so I thought I would send you a better explanation of the application I had in mind…

laying out his vision for an interactive development environment for computational physics. Reading this today, we can see how prescient that vision was, and included key elements of what Jupyter has become.

Min then joined Brian and Fernando in contributing to IPython and became one of the main driving forces behind a project that has grown to global prominence, with a lasting influence on the entire field of scientific computing.

Some of Min’s early experiments for a web-based notebook.

The rest is history, Min has spent the following fifteen years contributing to IPython and Jupyter. We owe him some of the key components of the project such as JupyterHub. He was honored along with the rest of the Jupyter steering council with the 2017 ACM Software System Award.

Today, Min is a senior research engineer and the head of the Department of Numerical Analysis and Scientific Computing at the Simula Research Laboratory in Oslo, Norway.

Beyond Jupyter, he has contributed widely to open source software, especially in the scientific Python community. He helps maintain numerous scientific packages in the conda-forge package management system.

It is an honor for me to introduce Min’s keynote! We hope you will join us at the conference and enjoy his talk. There are only a few days left and it is still time to sign up.

— Sylvain Corlay, JupyterCon 2020 Vice Chair

The JupyterCon schedule of tutorials and talks is published. Join us!

The templating system of nbconvert 6

2020-09-26T07:31:00+00:00

One of the main changes in nbconvert 6 is the refactor of the template system, which should be easier to extend and build upon.

In this article, we dive into the template system, and provide a tutorial on how to build a custom template for nbconvert or Voilà.

From Classic to Lab

My notebooks look different!

If you are accustomed to convert notebook files to HTML by typing

jupyter nbconvert notebook.ipynb --to html

you may have noticed differences in the generated HTML when switching to the latest release of nbconvert. In fact, nbconvert now produces the same DOM structure as JupyterLab’s notebook implementation, which is styled with JupyterLab’s CSS.

One can even apply the dark theme:

jupyter nbconvert xleaflet.ipynb --to html --HTMLExporter.theme=dark

The HTML output of nbconvert with the lab template and the dark theme

While JupyterLab uses CodeMirror to render code cells, nbconvert makes use of the Pygments library to produce syntax-colored static HTML. To mimick the JupyterLab CodeMirror styling, we created a Pygments theme called jupyterlab-pygments. JupyterLab Pygments uses JupyterLab’s CSS variables for coloring and will therefore reflect the theme that is applied to the notebook.

Note: at the moment, only the default light and dark themes are supported, but we plan on adding support for third-party JupyterLab themes after the release of JupyterLab 3, which introduces a new packaging system for extensions.

But I wanted my notebooks to look the same!

Well, if you want to retain the classic notebook styling that was used in earlier versions of CSS, it is still possible using the classic template.

jupyter nbconvert xleaflet.ipynb --to html --template classic

The HTML output of nbconvert with the classic template

With this template, you retrieve the original style of nbconvert outputs and of the classic notebook.

Another perk of the new nbconvert release is the WebPDF exporter. The WebPDF exporter supports the same templates and themes as the HTML exporter, and produces a PDF output that renders the same rich content as the HTML exporter, such as rich HTML tables, widgets etc.

jupyter nbconvert xleaflet.ipynb --to webpdf --HTMLExporter.theme=dark

The WebPDF output of nbconvert with the lab template and the dark theme

Executing the notebook before rendering

Nbconvert’s two main categories of transformations are preprocessors and exporters. Preprocessors take a notebook as an input, and return a transformed notebook, while exporters return other types of content, such as HTML or PDF. An important preprocessor is the ExecutePreprocessor, which spawns a kernel for the notebook, execute all cells, and populate outputs.

It can be invoked before the export by passing --execute. For example, the xleaflet.ipynb notebook uses the xeus-cling C++ kernel and makes use of the xleaflet interactive widget, which can be displayed when converting to HTML or with the WebPDF exporter.

jupyter nbconvert xleaflet.ipynb --to html --execute

The HTML output of nbconvert with the execute preprocessor

Similarly, the WebPDF exporter will also display interactive widgets!

The PDF output of nbconvert with the execute preprocessor

Note: Future plans for the WebPDF exporter include offering more options to users with respect to page breaks, and providing bookmarks for the main sections of the document.

Nbconvert templates

Unlike with earlier versions of nbconvert, templates are now directories, which may contain a jinja template but also other assets, such as macros, CSS files etc. The nbconvert template system also provides an inherittance mechanism which makes it simple to tweak existing templates in a derived one, by overriding bits of it.

Selecting a template

Most exporters in nbconvert are subclasses of TemplateExporter, and make use of jinja to render notebooks into the destination format. Nbconvert templates can be selected by name with the --template command line option.

For example, the reveal template, shipped with nbconvert, turns Jupyter notebooks into HTML slideshows using the RevealJS library. Which cells should be skipped, or where breaks betweens slides should be, are specified in the notebook cell metadata. The classic notebook and JupyterLab both provide means to set the appropriate values.

To select the reveal template, simply type:

jupyter nbconvert <path-to-notebook> --to html --template reveal

In the case of the xleaflet.ipynb notebook showed earlier, we get:

Creating a reveal slideshow with the dark theme

This demonstrates that the nbconvert templating system can be used to completely override how we look at notebook documents. Using metadata, we can even create arbitrary layouts and rich views of the same content.

Where are nbconvert templates installed?

Nbconvert 6 templates are directories containing resources such as jinja templates and other assets. They are installed in the data directory of nbconvert, namely <installation prefix>/share/jupyter/nbconvert.

Running jupyter --paths shows all Jupyter directories and search paths. For example, on Linux, jupyter --paths returns:

$ jupyter --paths
config:
    /home/<username>/.jupyter
    /<sys-prefix>/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
data:
    /home/<username>/.local/share/jupyter
    /<sys-prefix>/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter
runtime:
    /home/<username>/.local/share/jupyter/runtime

In our case, only the data section is relevant. Listing the content of <sys-prefix>/share/jupyter/nbconvert/templates in a raw installation of nbconvert will show

$ ls <sys-prefix>/share/jupyter/nbconvert/templates
asciidoc base classic compatibility html lab latex markdown python reveal rst script

The base template should not be used directly, but is typically inherited from. The compatibility directory provides some content for backward compatibility with earlier verions of nbconvert. Three templates are available for the HTML exporter: lab, classic, and reveal.

The content of nbconvert templates

The `conf.json` file

Nbconvert templates all include a conf.json file used to indicate the base template that it is inheriting from, the mimetype corresponding to that template (which determines which exporters are compatible with it, and which file is the entry point), and preprocessors to run when using that template before running the exporter. For example, inspecting the configuration of the reveal template we see that

it inherits from the lab template,
exports text/html, and therefore will only work with the HTML and WebPDF exporters.

and runs two preprocessors called 100-pygments and 500-reveal:

{
  "base_template": "lab",
  "mimetypes": {
    "text/html": true
  },
  "preprocessors": {
    "100-pygments": {
      "type": "nbconvert.preprocessors.CSSHTMLHeaderPreprocessor",
      "enabled": true
    },
    "500-reveal": {
      "type": "nbconvert.exporters.slides._RevealMetadataPreprocessor",
      "enabled": true
    }
  }
}

The CSSHTMLHeaderPreprocessor inlines the CSS required for the syntax highlighting of input cells.
The RevealMetadataPreprocessor massages the notebook metadata and consumes the information required to set up the layout of the slideshow.

Nbconvert walks up the inheritance structure determined by conf.json and produces an agregated configuration, merging the dictionaries of registered preprocessors. The ordering of the preprocessor names determines the order in which they will be run.

Jinja templates

Besides the conf.json file, nbconvert templates most typically include jinja templates files. They may also override files from the base templates, or provide extra content.

For example, inspecting the content of the classic template located in share/jupyter/nbconvert/templates/classic, we find the following content:

share/jupyter/nbconvert/templates/classic
├── static
│   └── styles.css
├── conf.json
├── index.html.j2
└── base.html.j2

we see that it includes the index.html.j2 jinja template (which is the main entry point for HTML exporters) as well as CSS and a base template file in base.html.j2. The only preprocessor listed in conf.json is the pygments syntax highlighting…

Inheritance in Jinja

In nbconvert, jinja templates can inherit from any other jinja template available in its current directory or base template directory by name. Jinja templates of other directories can be addressed by their path from the Jupyter data directory. Using the path is also useful when using a jinja template that may be overriden locally.

For example, in the reveal template, index.html.j2 extends base.html.j2 which is in the same directory, and base.html.j2 extends lab/base.html.j2. This approach allows using content that is available in other templates or may be overriden in the current template.

Building a custom template

Now, let’s create a custom template! If you work at ACME Corporation, you may want to create a template that follows the graphical charter of ACME Corp, and includes the logo in a banner.

Besides the logo, titles should also use the “ACME Regular” font, which has a cartoon-style look. The other parts of the template are inherited from the regular lab template.

Setting up a logo banner, and the font change, the acme nbconvert template produces the following result with a very simple notebook:

jupyter nbconvert acme.ipynb --to html --template acme

The HTML output of nbconvert with the acme template

Now you can create sophisticated templates making use of sophisticated front-end framework and processing the notebook metadata in creative ways!

The source for the ACME nbconvert template is available here.
Beyond the template files in share/jupyter/nbconvert/acme, the repo provides the logic for packaging this template into a PyPI wheel with data files. You will also find content related to the use of that template with Voilà, which is the subject of the next section.

From nbconvert templates to Voilà templates

Voilà turns Jupyter notebooks into standalone web applications and dashboards.

It differs from nbconvert in that the output web application is connected to a Jupyter kernel, allowing it to respond to user input through widget controls and other UI components, while with nbconvert, any action that requires a roundtrip to the kernel will not work.

However, Voilà leverages the nbconvert template system to benefit from their flexibility in overriding the front-end looks and behavior. From a user standpoint, the system is made so that the same templates will be usable for both systems.

However, template authors interested in advanced features of Voilà may be interested in the following information.

Where are Voilà templates installed?

Voilà templates are installed in the <installation prefix>/share/jupyter/voila directory (while nbconvert templates are in <installation prefix>/share/jupyter/nbconvert).
Just like nbconvert templates, Voilà templates are directories, they use the same conf.json configuration mechanism.
Voilà can use nbconvert HTML templates without modification.
When there exists an nbconvert and a Voilà template of the same name, the conf.json files are recursively merged, as well as the content of the directory, with a higher precedence for the Voilà template.

Overriding nbconvert templates with Voilà

When specifying the acme template that we developed earlier, with command

voila xleaflet.ipynb --template acme

Voilà will look pick up the acme nbconvert template. Since this template inherits from lab and Voilà has an overridden lab template (with e.g. the logic for rendering widgets), it will pick up the Voilà flavor of the lab template. Most typically, the Voilà flavor of a template does not add much on top of nbconvert besides boilerplate such as

a call to the macro that includes the JavaScript assets for the Voilà front-end logic.
calls to macro related to error logging when using the Voilà preview.
calls to macros related to the progressive rendering of notebooks as it is being executed, and the display of an “in-progress” spinner.

These macros are provided in the base Voilà template, and can also be overridden in derived templates.

Future developments

In the coming weeks and months, we plan on polishing the experience of nbconvert users.

JupyterLab 3, which should be released shortly, includes a more dynamic extension system which does not require the main JupyterLab application to be rebuilt. We plan on adding support for a category of lab extensions called “mime renderers“ which are used for rich rendering of data in cell outputs. This should enable the use of complex mime types such as GeoJSON or Vega visualizations in nbconvert and Voilà.
We are working on improving the reveal template, to include a custom reveal theme making use of JupyterLab CSS variables, so that it can be easily combined with JupyterLab themes.
The JupyterLab 3 extension system may also enable us to enable third-party JupyterLab themes in nbconvert.
The WebPDF exporter should expose options on output format and where page breaks should be. At the moment, we prevent page breaks until the maximum dimensions of PDF documents are reached.

Acknowledgements

Many people were involved in the nbconvert 6 release! The full list of contributors is available here.

Among them, we are especially indebted to Maarten Breddels, who was the main architect of the new template system.
We owe the split of the execute preprocessor and the new nbclient package to Matthew Seal and David Brochart. Matthew took on a large amount of maintenance work on the project over the past year.

The work on Voilà and nbconvert by the QuantStack team was funded by Bloomberg.

About the Author

Sylvain Corlay is the CEO of QuantStack, an open-source software development team specialized scientific computing comprising maintainers of major projects of the ecosystem.

As an open-source developer, Sylvain is mostly active in the Jupyter ecosystem, and the general PyData stack. He is currently a steering committee member for Project Jupyter, and a member of the board of directors of NumFOCUS.

SlicerJupyter: a 3D Slicer kernel for interactive publications

2020-07-08T16:29:00+00:00

Use Jupyter and 3D Slicer kernel to implement biomedical data processing workflows in a notebook.

The Jupyter ecosystem is a powerful platform for exploratory computational science, and now it can connect with some of the deep and rich domain-specific desktop applications that have decades of feature development already invested in them. With the integration of 3D Slicer with Jupyter through the xeus-python’s interpreter, we demonstrate how a Qt-based graphical desktop application with 3D visualization provided by Visualization Toolkit (VTK), image processing provided by the Insight Toolkit (ITK), can be used through a Jupyter notebook. This approach is available on the SlicerJupyter GitHub repository and could be extended to other applications that embed Python, such as Blender, FreeCAD, or ParaView.

This xeus-python integration is beneficial both for the Jupyter ecosystem and desktop applications. Features that have been developed for decades for desktop applications become readily available for Jupyter users without learning a new working environment or redeveloping features. For a desktop application, the Jupyter notebook can serve as a way to create reproducible data processing workflows, scientific publications, and maintainable tutorials without requiring local software installation. From Jupyter, you can now quickly create simple medical imaging applications with extremely rich interactivity from 3D Slicer. Read below for a discussion of the features available and the history of the project.

Powerful Medical Imaging Capabilities Available Through Jupyter

3D Slicer (or Slicer for short) is a C++ desktop application that uses Qt, ITK, and VTK libraries for visualization and medical image analysis. Slicer’s embedded Python interpreter makes all its features accessible with the Python programming language. Slicer has a simple built-in console to run Python commands interactively and can run Python scripts from files, but these are not as convenient as cell-base interactive notebooks, which have become popular among data scientists and researchers in recent years.

By integrating the xeus-python kernel, we can use a Slicer process as a Jupyter kernel. xeus-python leverages the xeus C++ implementation of the Jupyter kernel protocol. xeus-python is an alternative to ipykernel, which can be used with a vanilla CPython interpreter, interfacing to standard CPython Jupyter widgets like itkwidgets, but it can also be coupled with custom interpreters and GUI event loops, like the Slicer interpreter and its Qt event loop. This allows you to represent a complete scene in Medical Reality Markup Language, MRML, Slicer’s internal data structure. The kernel exposes the full medical imaging API and representation of your data in a meaningful way for Python developers, allowing access through standard Python ecosystem formats such as pandas dataframes and NumPy arrays in the Notebook.

Interactivity Levels

You can also use Jupyter interactive widgets (sliders, buttons, etc.) to control Slicer, modify data, or adjust processing and visualization parameters. Interactivity can be implemented at different levels.

Level 1: Standard Jupyter widgets display application specific objects by automatic conversion of application-specific data objects to standard Python objects. For example, Slicer markup fiducial lists are displayed as a nicely formatted table and model nodes are rendered as 3D objects.
Level 2: Static image widgets display content that the desktop application renders. These widgets can be made interactive by modifying data and rendering parameters using additional standard widgets. This makes rich visualization capabilities — sophisticated rendering various data types, rendering of very large data sets, etc. directly available in Jupyter.
Level 3: Dynamic viewer widgets display 2D and 3D views rendered by the desktop application. Mouse and keyboard events are forwarded to the desktop application that allows zooming/rotating views, and to utilize all 3D interactions implemented in the desktop such as placing annotations, making measurements, or segmenting images the same way as if it was done on the desktop appication’s screen. This is implemented in the Slicer Jupyter kernel using ipycanvas and ipyevents packages.
Level 4: Full desktop graphical user interface integration. Users can see parts of the application window rendered in notebook cells, including standard desktop widgets (sliders, menus, etc.). It is implemented using noVNC and TigerVNC in Slicer Jupyter. This is particularly useful when the application runs on a remote server.

Video demonstrating how to run 3D Slicer using Binder

These interactive tools allow developers to implement complete data processing workflows in a notebook, even if certain steps require manual user inputs such as segmenting 3D regions or setting seed points.

Try it online!

Since Jupyter notebooks can be used from any web browser, it can essentially turn any desktop application to a web application. By setting up a remote Jupyter server, users do not have to install anything on their computers. We have set up a demonstration of this using Binder (www.mybinder.org) that anybody can try at https://mybinder.org/v2/gh/Slicer/SlicerNotebooks/master.

Click on the binder image to launch the demo

The docker image that installs and configures Slicer and all dependencies (ipycanvas, ipyevents, VNC, etc.) is available at https://github.com/Slicer/SlicerDocker/tree/master/slicer-notebook. For deployment to non-technical users, applications can be deployed using Voilà (https://voila.readthedocs.io/), which only shows relevant content and interactive widgets, so the notebook looks like a simple dynamic web page.

The current implementation is already stable and offers a wide range of features, but there is still room for design and performance improvements. For example, we could not implement fully automatic conversion of application-specific data objects to displayable Python objects (due to complex implementation of display hooks); xeus-python debugger’s threading model needs to be improved to allow using it without locking the application’s main thread; and dynamic viewer widget’s performance (level 3 interaction) could be optimized to achieve higher refresh rates.

History of Slicer and this integration

Built over two decades with support from the NIH and a worldwide open source developer community, 3D Slicer is a unique, multi-platform desktop application for analysis, integration, and visualization of medical images that is heavily used by researchers globally for basic and applied research in a wide range of topics. The 3D Slicer Community includes physician-scientists with disease-specific knowledge of clinical challenges, computer scientists and physicists who develop novel algorithms, imaging informatics researchers, software engineers with the ability to understand clinical problems and create reliable tools, and application engineers with the multidisciplinary skills to deploy these tools in a range of cancer research settings. Slicer is maintained by Kitware, Inc., and the NAMIC consortium, and has an active open source software community. Slicer is used in hospitals and by researchers, with more than 10k academic citations and more than 150k downloads in the last year alone.

Click on the image to download 3D Slicer

It can be used for image processing workflows on 2D, 3D, and 4D images. What makes Slicer so useful is its huge set of community contributed modules that extend its functionality, as well as its ability to read DICOM and a wide array of exotic file formats.

Slicer has been scriptable in Python for well over a decade, but a robust SlicerJupyter extension is born from significant development effort by the community. The idea for this integration dates back to SciPy in 2014, when Jean-Christophe Fillion-Robin of Kitware organized a sprint intended to integrate Slicer and an IPython notebook, motivated by the dream of creating interactive and fun tutorials for Slicer. Mike Sarahan and Jean-Christophe created this proof of concept https://github.com/commontk/QEmbedIPython#qt-embed-ipython, but it was far from usable. In 2015 Matt McCormick of Kitware, created the SlicerDocker repository to support headless builds and rendering in a Docker image. Then in June 2018, while attending the Slicer Project Week, Andras Lasso (Queen’s University) and Jean-Christophe learned about Xeus, a C++ implementation of the Jupyter kernel protocol developed by QuantStack that would help streamline the integration of Slicer with Jupyter. To support this effort, Andras and Jean-Christophe created the Slicer/SlicerJupyter GitHub repository. They also contributed changes to Xeus to support this new integration paradigm where the event loop of the kernel is driven by a Qt-based Desktop application (see here). Building on this foundation, Isaiah Norton, then working at Brigham and Women’s Hospital, contributed additional improvements like a better auto-completion using jedi (see https://github.com/Slicer/SlicerJupyter/pull/12) as well as integration with Binder. More recently, Jean-Christophe and Sylvain Corlay (QuantStack) met after the Slicer project week while attending SciPy in Austin. Following the creation of a new project called xeus-python (started by Martin Renou at QuantStack), we took the integration of Jupyter and Slicer to the next level by adding support for improved interactive use and MRML data node visualization directly in the notebook.

About the authors

Alphabetically ordered

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored xeus and xeus-python.

Sylvain Corlay

Jean-Christophe Fillion-Robin is an open-source enthusiast, original author of the SlicerJupyter extension and a principal engineer at Kitware Inc where he leads the development of “3D Slicer” based commercial applications. J-Christophe also maintains scikit-build, an improved build system generator for CPython C/C++/Fortran/Cython extensions.

Jean-Christophe Fillion-Robin

Mike Grauer is a Technical Leader on the data and analytics team at Kitware. He is specialized in building scalable server-side processing frameworks that enable scientific workflows over web platforms.

Mike Grauer

Andras Lasso is an original author of the SlicerJupyter extension, Senior Research Engineer and Associate Director of the Laboratory for Percutaneous Surgery at Queen’s University.

Andras Lasso

Matt McCormick is an open source, medical imaging researcher working at Kitware Inc. Matt is an active, contributing member of scientific open source software efforts such as the Insight Toolkit (ITK) and scientific Python (SciPy) communities, and maintains the Jupyter 3D widget, itkwidgets.

Matt McCormick

Isaiah Norton is a Senior Software Developer at TileDB, Inc. with experience in digital pathology and image-guided surgical navigation.

Isaiah Norton

Steve Pieper is a Chief Architect and active developer of the 3D Slicer application for well over a decade. He is CEO of Isomics, Inc, where he uses a range of software technologies to perform medical imaging research with leading universities and companies.

Steve Pieper

Martin Renou is a scientific software developer at QuantStack. He is the original author of xeus-python, the xeus-based Python kernel, and contributed to the new concurrency model used for the debugger.

Martin Renou

Mike Sarahan is a software engineer at RStudio, PBC working on bringing language ecosystems together. He is passionate about making software work for data scientists in ways that are easy to maintain and improve.

Mike Sarahan

Report on the Jupyter Community Workshop on Dashboarding

2020-02-14T21:06:00+00:00

This report is long overdue! From June 3rd to June 6th 2019, thirty-five developers from the Jupyter community met in Paris for a four-day workshop on dashboarding with Project Jupyter.

Attendees to the Jupyter Community Workshop on Dashboarding (Photo credit to Lindsey Heagy)

For four days, attendees worked full time on the Jupyter project, including hacking sessions and discussions on improvements to Jupyter components and new development. We were lucky to count a large number of core developers to the project in the group.

Beyond the hacking sessions, each day was concluded with a series of presentations and demos of the progress made during the workshop. In partnership with the PyData Paris team, we had a special installment of the PyData Paris Meetup with

an invited presentation by Emmanuelle Gouillart on Plotly Dash,
a series of lightning talks by attendees of the workshop on their achievements, including a talk by Philip Rudiger on the first release of Panel, and an announcement of the first releases of Voilà and the Voilà Gallery.

We ended the week with a social evening at the QuantStack offices in Paris.

Why a workshop on Jupyter Dashboarding with Jupyter?

The Jupyter ecosystem is used extensively in scientific computing both in academia and industry, and a rich ecosystem of data visualization tools has been developed around the Jupyter widgets frameworks, from geographical data visualization to protein folding simulation.

However, the Jupyter ecosystem still did not provide a means for developers to transition from notebooks to stand-alone web applications that can be accessed by multiple users.

This has been a longstanding request from the community: provide better tools built upon the Jupyter stack to share results with students, peers, or the general public.

These are the challenges that we decided to tackle during that week. The workshop was attended by many Jupyter core developers.

Highlights of the week

Many of the developers spent the week working on the Voilà and Panel projects. Both projects had their first public releases during that week (see the first public announcement of Panel and Voilà).

During this week, a team of participants including Yuvi Panda, Pascal Bugnion, and Jeremy Tuloup iterated on the first version of the Voilà gallery. Several first-time contributors to the widget framework authored example dashboards for the gallery, showcasing their existing work. Yuvi also produced the first deployment scenarii for Voilà on Heruku.
Cheryl Quah, from Bloomberg MC-ed a panel on dashboarding in the Jupyter ecosystem, including lots of questions and comparisons with Dash.
Philip Rudigger started working on a Bokeh/ipywidgets integration for better interoperability between the two frameworks.
Other contributors iterated on creating new Voilà templates, such as voila-vuetify, adding the ability to position Jupyter widgets and outputs in arbitrary location in the dashboard template. Grant Nestor created visual mockups for a UI for creating dashboard layouts in JupyterLab. Grant also helped iterating on logos for the project, and gave a presentation on dynamically loading JavaScript modules in the browser.

Acknowledgments

This event would not have been possible without the generous support provided by Bloomberg, who made this workshop series possible

We are grateful to Société Générale for funding the catering for the workshop.

The hosting of the workshop at CRI was paid for by QuantStack.

The public meetup was organized in partnership with the PyData Paris team.

Finally, we especially thank Ana Ruvalcaba from Project Jupyter for her incredible work on the logistics and finances of the Jupyter Community Workshop series.

Voilà is now a Jupyter subproject

2019-12-29T12:11:00+00:00

It is a great pleasure to announce that the Voilà project has been incorporated as a Jupyter subproject. Voilà will now be subject to the Jupyter governance and code of conduct.

For reference, the Jupyter Enhancement Proposal (JEP) for the Voilà incorporation is available here.

What is Voilà?

Voilà helps you communicate insights, by transforming a Jupyter Notebook into a stand-alone web application you can share. It gives you control over what your readers experience in a secure and customizable interactive dashboard.

The easiest way to get started with Voilà is to install it via pip or conda and type voila some_notebook.ipynb to turn the said notebook into a dashboard.

Besides, Voilà includes a templating system that allows to overload the behavior of the front-end. Using this templating system, Voilà can be used to create

slideshows (with voila-reveal)

A Voilà slideshow created with the voila-reveal template.
dashboards (with voila-gridstack)

A Voilà Dashboard based on the voila-gridstack template.

Why moving Voilà under the Jupyter governance?

While the project was initially started by QuantStack, the team now comprises developers from Bloomberg, UC Berkeley, JP Morgan, and Cal Poly San Luis Obispo. OVH has been supportive of the project by kindly providing the free hosting of the gallery on their infrastructure.

We believe that the multi-stakeholder nature of the Voilà project is well-suited for the Jupyter organization.

The Voilà project is largely built upon Jupyter subprojects and standards.

The standard notebook file format is the main entry point to Voilà.
nbconvert is used for the conversion to progressively-rendered HTML.
naturally, we use jupyter_client for handling the execution of notebook cells
jupyter_server is the default back-end.
JupyterHub is at the foundation of the voila-gallery project.
JupyterLab components (mime renderers, input and output areas) are used in the front-end implementation. Voilà also includes a preview JupyterLab extension.
ipywidgets and custom jupyter widget libraries such as bqplot, ipyvolume, ipyleaflets provide the bulk of the interactivity of Voilà applications.

Voilà is more a remix of existing Jupyter components (with changes to enable that use case) than a completely new application.

Resources

Should you be interested in Voilà, feel free to try it on Binder or locally! You can also engage with the developer community during our public team meetings and the various GitHub repositories of the project:

the Voilà GitHub repository is available here: https://github.com/voila-dashboards/voila.
the Documentation is hosted on Read the Docs: https://voila.readthedocs.io.
the Team Compass holds the calendar for the public developer meetings, as well as the meeting minutes: https://voila-dashboards.github.io.
the Announcement of the first Voilà release was published on this blog: And voilà!

Acknowledgements

Voilà was started by the team of open-source developers at QuantStack as a separate project, but with the full intent to incorporate it into Jupyter. The initial project development at QuantStack was funded by Bloomberg.
Now, Voilà contributors work in many institutions, including UC Berkeley, Cal Poly San Luis Obispo, JP Morgan, and Faculty (formerly ASI Data Science).
The Voilà Gallery is kindly hosted by OVH.

Field Report on the Kernel Community Workshop

2019-10-07T13:24:00+00:00

From May 27th to May 29th 2019, thirty developers from the Jupyter community met in Paris for a three-days workshop on the Jupyter kernel protocol.

Attendees to the Jupyter Community Workshop on Kernels

For three days, attendees worked full time on the Jupyter project, including hacking sessions and discussions on improvements to Jupyter protocols and standards.

We were lucky to count five core developers to the project and several more regular contributors to the larger ecosystem in the group!

Beyond the hacking sessions, each day was concluded with a series of presentations and demos of the progress made during the workshop.

Why a workshop on Jupyter kernels?

The Jupyter Kernel protocol is one of the main extension points of the Jupyter ecosystem. Dozen of language kernels have been developed by the community, and these languages can then leverage the other components of the stack, such as the notebook, the console, and interactive widgets.

One of the objectives of this event was to foster collaboration between kernel developers and core contributors and develop common tools used across all language kernels. Another goal was to work improving the protocol, to support visual debugging, parameterized kernels, etc.

Technical Achievements

A lot of work was done during the workshop. Several people were working on new Jupyter kernels for programming languages.

We were amazed to see how much was achieved in just three days. Today, we are still working on projects that stemmed from this community workshop.

JupyterLab Debugger

One area in which the Jupyter team is actively working is support for visual debugging in JupyterLab. This is a major endeavor spanning from the front-end to the back-end, including changes to the Jupyter protocol.

The kernel workshop was the occasion for several developers (Wolf Vollprecht, Maarten Breddels, Johan Mabille, Loic Gouarin) to get together and hack on the implementation of the frontend. A lot of work had been done ahead of the workshop with a new Python kernel based on the Xeus library. This new kernel includes a backend to the Debug Adapter Protocol over kernel messages.

The JupyterLab Visual Debugger

There are other key contributors to the debugger project who did not attend the community workshop. We should mention Jeremy Tuloup and Afshin Darian who are spearheading the frontend development.

A Calculator Jupyter Kernel based on xeus

Vasavan Thirusittampalam and Thibault Lacharme were in the middle of their summer internships as scientific software developers at QuantStack when they attended the workshop. During the event, they set themselves up to implement a new Jupyter kernel in C++!

At the end of the workshop, they were able to demonstrate a functional calculator kernel based on xeus, a C++ implementation of the Jupyter protocol! This was later polished and resulted in the publication of a blog post on the Jupyter blog: “Building a Calculator Jupyter Kernel”. This example may serve as an example for people interested in creating new language kernels with xeus.

A prototype Julia backend to Jupyter interactive Widgets.

During the workshop Sebastian Gutsche (who is the author of the Singular Jupyter kernel and a co-author of the GAP kernel), set himself to develop a backend to Jupyter interactive widgets for the Julia progamming language. The current state of the code can be found here.

The Julia backend to jupyter widgets is still early-stage but should this project be completed, it would enable other Jupyter widget libraries for the users of the Julia Jupyter kernel, such as ipyvolume, ipyleaflet, bqplot, etc.

The Julia backend to Jupyter widgets

Parameterized Kernelspecs

Paul Ivanov and Kyle Kelley worked on ironing out the proposal for the support of parameterized kernelspecs. The objective is to make it possible to pass user-specified arguments to kernels upon launch.

In the proposal, the parameters expected by the kernel executable and the possible values for these parameters may be specified in the kernelspec file or a companion file to the kernelspec in the form of a JSON schema.

Paul and Kyle produced extensive notes on various aspects on the subjects such as

the storing of previously used values of the parameters (in notebooks)
the web UI design on how to specify parameters
the definition of default values

More details will be shared at a later stage on the subject of parameterized kernelspecs.

Note: the subject of parameterized kernelspecs was also central in the earlier community workshop about the Jupyter server organized at IBM by Luciano Resende.

Forking of Jupyter Kernels

Starting of kernels is an expensive operation, and executing a whole notebook even more. For certain use cases (such as dashboards based on Voilà), it is useful to have a pre-executed notebook ready, so that dashboards can be presented instantly to a user. To enable this, Maarten Breddels came up with the idea of forking kernels and put together a proof-of-concept implementation. The implementation can be found in pull requests to jupyter_client (PR #441) and ipykernel (PR #410).

Forking kernels will also allow interesting features such as an undo/rollback of cell execution in the notebook. While it is still a proof of concept, this subject has already sparked interest in the community.

Acknowledgments

This event would not have been possible without the generous support provided by Bloomberg, who made this workshop series possible

We are grateful to CFM for gracefully hosting the event.

We should also thank Ana Ruvalcaba from Project Jupyter for her incredible work on the logistics and finances of the Jupyter Community Workshop series.

Interactive GIS in Jupyter with ipyleaflet

2019-09-24T22:54:00+00:00

As Jupyter grew in popularity, a broad ecosystem of visualization packages based on Jupyter widgets has been developed, bringing even more interactivity to the Jupyter world.

In this article, we dive into Jupyter Interactive Widgets and the ipyleaflet package, an interactive maps visualization system for Jupyter.

A non-interactive map.

Jupyter Widgets

Jupyter Interactive Widgets are “special objects” that can be instantiated by the user in their code and result in a counterpart component being created in the front-end.

The core ipywidgets package provides a collection of controls that Jupyter users can use to build simple UIs as part of their notebooks (sliders, buttons, dropdowns, layout components).

Basic ipywidgets controls.

More than a collection of controls, it provides a framework upon which a large ecosystem of components has been built, allowing notebook authors to capture user inputs in very diverse ways.

Popular libraries built upon interactive widgets include

bqplot, a 2-D plotting system for Jupyter,
ipyvolume, a 3-D plotting package based on WebGL and ThreeJS,
PythreeJS, a 3-D scene description package exposing a large part of the ThreeJS API to Jupyter,
ipyvuetify, a large collection of VuetifyJS components exposed to Jupyter,
ipywebrtc, a library exposing the features of the WebRTC protocol to Jupyter kernels,
beakerx, a collection of widgets, extensions, and kernels for Jupyter,

and many more… This list is not comprehensive and dozens of other widget packages have been developed.

Key aspects of Jupyter widgets include:

Bidirectionality: Widgets are not just meant for display but can also be used to capture user inputs, which can then trigger new computation. Notebook authors can compose sophisticated applications including a variety of components from different packages.
Language Agnosticism: Built upon the Jupyter ecosystem, the interactive widget protocol used for the synchronization between the kernel and the front-end is well-specified and can be implemented for any kernel.
Back-ends for other languages than Python already exist, such as for C++ (with the xeus-cling Jupyter kernel), and languages of the JVM such as Clojure or Groovy (with the beakerx kernels).
Extensibility: Jupyter widgets are not meant as a monolithic system with one and only one way to achieve a specific task. We strive to provide a foundational layer allowing third-party widget authors to be as inventive as possible.

A common pattern for Jupyter widget packages has been to bring the capabilities of popular JavaScript visualization frameworks to Jupyter with a bridge based on ipywidgets. This is the case for the ipyleaflet package, as well as pythreejs and ipyvuetify.

Another use case is the development of ad hoc controls that are not necessarily relevant for a mainstream visualization package, but may be specific to a scientific field. An example is the ipymesh project by Chris Kees which can be used to draw PSLG (planar straight-line graphs) in the Jupyter notebook.

ipyleaflet

ipyleaflet is a Jupyter - LeafletJS bridge, bringing mapping capabilities to the notebook and JupyterLab.

Built as a bridge between the LeafletJS package and Jupyter, the ipyleaflet API maps to that of LeafletJS, bringing most of the core features of the package to Jupyter, and enabling a few popular LeafletJS extensions. A small difference is that following the Python coding style, ipyleaflet makes use of snake_case instead of CamelCase for attribute names.

ipyleaflet features

The main components to the library are layers and controls, respectively items to be displayed on the map, and interactive widgets overlayed on the map area for greater interactivity.

The first thing for which you may want to change the default value are the zoom level, the position, or the base layer for the map.

Changing the basemap layer in an ipyleaflet map.

Maps can be interactively edited in the Jupyter notebook, by dynamically changing or adding layers. In this screenshot, we add a custom layer including a GeoJSON dataset.

Note: An alternative layer to GeoJSON is GeoData, which lets the user load the data in the form of a GeoPandas dataframe instead of raw GeoJSON.

Adding a GeoJSON dataset to an interactive map.

A number of simple primitives are available such as markers and heatmaps. In the following screencast, we show how primitive properties can be linked with other widgets, and used as means to take user input on a map:

Linking properties of ipyleaflet primitives to other Jupyter widgets.

The splitmap control can be used to compare to different set of ipyleaflet layers at the same location.

The splitmap control.

Another interesting layer is the velocity layer which can be used to display wind velocity data. This control can take data in the form of an xarray dataset.

The velocity layer.

More classical visualization tools are also available, such as choropleths.

A choropleth layer.

JupyterLab Integration with the Sidecar and Theming Support

ipyleaflet is well integrated with JupyterLab

ipyleaflet controls make use of the JupyterLab themes for coloring so that they don’t stand out when using e.g. a dark theme.

The support for JupyterLab themes in ipyleaflet.
ipyleaflet can be used in combination with the JupyterLab sidecar widget.

Using the sidecar widget in JupyterLab to display a map aside of the notebook.

Together with the sidecar, a map can be programmatically added to the right-side toolbar of the JupyterLab application, and interactively edited in the notebook. This prevents the back-and-forth scrolling often required to see how changes are reflected visually.

The workflow enabled by ipyleaflet in combination with the sidecar is similar to that of the geonotebook project by Christopher Kotfila, Jonathan Beezley, and Dan LaManna from Kitware).

The geonotebook (which was based on the classic notebook) provided a similar workflow to the lab sidecar.

A more advanced example

Combined with other widget libraries such as bqplot or the core ipywidget package, ipyleaflet users can easily compose more complex applications and dashboards. In the following screencast, we explore the wealth-of-nations dataset with a leafletmap, a bqplot line chart and a dropdown widget:

Exploring the wealth-of-nations dataset with ipyleaflet, bqplot and core ipywidgets.

Trying ipyleaflet online with mybinder

If you would like to try out ipyleaflet now, it is possible thanks to the binder project. Just click on the image below!

Interactive GIS in C++: xleaflet

As mentioned earlier, interactive widgets back-ends for other programming languages have be implemented. There is a C++ back-end for ipywidgets: xwidgets which is the building block for creating other C++ widgets back-ends.

xleaflet is the C++ - LeafletJS bridge that exposes almost the same API as ipyleaflet, only that you use it from a C++ interpreter! It is based upon xwidgets which brings the bidirectional communication with the front-end.

You can learn more about the xeus-cling Jupyter kernel, xwidgets and xleaflet by reading the following blogpost.

xleaflet is the C++ backend to the jupyter-leaflet integration.

Deploying ipyleaflet-based dashboards with Voilà

Voilà is a tool that turns Jupyter notebooks into standalone dashboards. Built upon the Jupyter stack, it inherits the language agnosticism of the ecosystem, and can be used to produce standalone applications based on ipyleaflet.

A companion project to Voilà is the Voilà gallery project, a public facing set up of JupyterHub serving Voilà dashboard. It is kindly hosted by OVH. You can check out the Voilà gallery at URL https://voila-gallery.org.

For more resources about voilà, check out

the original announcement of the project: And Voilà!
and for the gallery: A Gallery of Voilà Examples.

If you check out the Voilà gallery, don’t miss Jeremy Tuloup’s GPX loader demo!

Jupyter for Geo Sciences

Jupyter’s adoption is exploding in the GeoScience space. Notable projects building upon Jupyter include

JEODPP (JRC Earth Observation Data and Processing Platform) is a EU project providing petabyte scale storage and high-throughput computing capacities to facilitate large scale analysis of Earth Observation data. The main front-end to the platform is base on Jupyter. The mapping capability is based on ipyleaflet. End users can request custom visualization that are returned to them in the form of lazily computed tile layers.
Pangeo is a community platform providing open, reproducible, and scalable sciences. The Pangeo software ecosystem involves open source tools such as xarray, Iris, Dask, Jupyter, and many other packages.
It was recently announced that the NSF would be funding the UC Berkeley + NCAR “EarthCube” proposal “Jupyter meets the Earth: Enabling discovery in geoscience through interactive computing at scale”. The plan involves the development of interactive dashboards with Voilà and contributions to the Voilà codebase.

Jupyter-gmaps

We should mention the jupyter-gmaps project by Pascal Bugnion. Jupyter-gmaps is a bridge between Google maps and Jupyter. Just like ipyleaflet, jupyter-gmaps is built upon the jupyter interactive widgets framework but relies on Google maps for the display instead of LeafletJS library.

This is a high-quality widget by another core developer of ipywidgets. Pascal is also one of the people behind Voilà and the Voilà gallery.

Folium

The Folium project enables maps visualization in the Jupyter notebook. Just like ipyleaflet, it is based on LeafletJS. Folium was created by Rob Story and is now maintained by Frank Conengmo and Filipe Fernandes.

A key difference between Folium and ipyleaflet is that ipyleaflet is built upon ipywidgets and allows bidirectional communication between the front-end and the backend enabling the use of the map to capture user input, while Folium is meant for displaying static data only. Folium enables many LeafletJS extensions, some of which may not be available in ipyleaflet at the moment.

Acknowledgements

The ipyleaflet project was started in 2015 by Brian Granger, and funded by the ERDC. The further development by Sylvain Corlay and Martin Renou was supported by QuantStack.

We are grateful to OVH for kindly hosting the Voilà gallery.

Other key contributors to Voilà and ipyleaflet that should be thanked here are Maarten Breddels, Pascal Bugnion, and Yuvi Panda. We should also mention the recent contributions by Vasavan Thirusittampalam who worked on the full-screen control and better interoperability with geopandas.

About the Authors

Sylvain Corlay, and Martin Renou are Scientific Software Developers at QuantStack.

Sylvain Corlay

Sylvain Corlay is the founder and CEO of QuantStack.

As an Open Source Developer, Sylvain contributes to Project Jupyter in the areas of interactive widgets and language kernels and is a steering committee member of the Project. Beyond QuantStack, Sylvain serves as a member of the board of directors of the NumFOCUS foundation. He also co-organizes the PyData Paris Meetup.

Martin Renou

Martin Renou is a Scientific Software Developer at QuantStack. Prior to joining QuantStack, Martin also worked as a Software developer at Enthought. He studied at the French Aerospace Engineering School ISAE-Supaero, with major in autonomous systems and programming.

As an open source developer, Martin has worked on a variety of projects, such as SciviJS (a JavaScript 3-D mesh visualization library), Xtensor, and Xeus.

And voilà!

2019-06-11T17:42:00+00:00

The goal of Project Jupyter is to improve the workflows of researchers, educators, scientists, and other practitioners of scientific computing, from the exploratory phase of their work to the communication of the results.

But interactive notebooks are not the best communication tool for all audiences. While they have proven invaluable to provide a narrative alongside the source, they are not ideal to address non-technical readers, who may be put off by the presence of code cells, or the need to run the notebook to see the results. Finally, following the order as the code often results in the most interesting content to be at the end of the document.

Another challenge with sharing notebooks is the security model. How can we offer the interactivity of a notebook making use of e.g. Jupyter widgets without allowing arbitrary code execution by the end user?

We set ourselves to solve these challenges, and we are happy to announce the first release of Voilà.

Voilà turns Jupyter notebooks into standalone web applications.

Voilà supports Jupyter interactive widgets, including the roundtrips to the kernel.
Voilà does not permit arbitrary code execution by consumers of dashboards.
Built upon Jupyter standard protocols and file formats, Voilà works with any Jupyter kernel (C++, Python, Julia), making it a language-agnostic dashboarding system.
Voilà is extensible. It includes a flexible template system to produce rich application layouts.

Installation and first-time use

Voilà can be installed from pypi:

pip install voila

or conda-forge:

conda install voila -c conda-forge

Upon installation, several components are installed, one of which is the voila command-line utility. You can try it by typing voila notebook.ipynb. It results in the browser opening to a new tornado application showing markdown cells, rich outputs, and interactive widgets.

From a notebook to a standalone web application

As you can see in the screencast, Jupyter interactive widgets remain fully functional even when they require computation by the kernel.

You can immediately try out some of the command-line options to Voilà

with --strip_sources=False, input cells will be included in the resulting web application (as read-only pygment snippets).
with --theme=dark, Voilà will make use of the dark JupyterLab theme, which will apply to code cells, widgets and all other visible components.

Making use of the dark theme and including input cells

Note that code is only shown, voilà does not allow users to edit or execute arbitrary code.

Voilà’s execution model

The execution model of Voilà is the following: upon connection to a notebook URL, Voilà launches the kernel for that notebook, and runs all the cells as it populates the notebook model with the outputs.

The execution model of Voilà

After the execution, the associated kernel is not shut down. The notebook is converted to HTML and served to the user. The rendered HTML includes JavaScript that establishes a connection to the kernel. Jupyter interactive widgets referred in cell outputs are rendered and connected to their counterpart in the kernel. The kernel is only shut down when the user closes their browser tab.

The current version of Voilà only responds to the initial GET request when all the cells have finished running, which may take a long time, but there is ongoing work on enabling progressive rendering, which should make it into a release soon.

An important aspect of this execution model is that the front-end does not determine what code is run by the backend. In fact, unless specified otherwise (with option --strip-sources=False), the source of the rendered notebook does not even make it to the front-end. The instance of the jupyter_server instantiated by Voilà actually disallows execute requests by default.

Support for custom interactive widgets

Voilà can render custom Jupyter widget libraries, including (but not limited to) bqplot, ipyleafet, ipyvolume, ipympl, ipysheet, plotly, ipywebrtc, etc.

Including bqplot figures with Voilà

Together with ipympl, Voilà is actually a simple means to render interactive matplotlib figures in a standalone web application:

Rendering interactive matplotlib figures in a web application with voilà

Voilà is language-agnostic

Voilà can be used to produce applications with any Jupyter kernel. The following screencast shows how Voilà can be used to produce a simple dashboard in C++ making use of leaflet.js maps, with the xeus-cling C++ kernel and the xleaflet package.

A standalone Voilà page making use of the C++ Jupyter kernel, xeus-cling (input cells display enabled).

We hope that Voilà will be a stimulant to other languages (R, Julia, JVM/Java) to provide stronger widgets support.

Richer layouts with Voilà templates

The main extension point to Voilà is the custom template system. The HTML served to the end-user is produced from the notebook model by applying a Jinja template, which can be defined by the user.

An example template for voilà is the voila-gridstack template, which can be installed from pypi with

pip install voila-gridstack

You can try it by typing voila notebook.ipynb --template=gridstack.

Making use of the Gridstack template to produce a dashboard with bqplot charts

The gridstack Voilà template makes use of the cell metadata to lay out the application.

A roadmap item for the gridstack Voilà template is to support the entire spec for the deprecated jupyter dashboards and to create a WYSIWYG editor for these templates in the form of a JupyterLab extension.

Note that voila-gridstack template is still at an early stage of development.

How to make custom Voilà templates?

A Voilà template is actually a folder placed in the standard directory PREFIX/share/jupyter/voila/templates and which may include

nbconvert templates (the jinja templates used to transform the notebook into HTML)
static resources
custom tornado templates such as 404.html etc.

All of these are optional. It may also contain a conf.json file to set up which template to use as a base. The directory structure for a Voilà template is the following:

PREFIX/share/jupyter/voila/templates/template_name/
|
├── conf.json                # Template configuration file
├── nbconvert_templates/     # Custom nbconvert templates
├── static/                  # Static directory
└── templates/               # Custom tornado templates

The Voilà template system can be used to completely override the behavior of the front-end. One can make use of modern JavaScript frameworks such as React or Vue.js to produce modern UI including Jupyter widgets and outputs.

Another example template for Voilà is voila-vuetify, which is built upon vue.js:

The voila-vuetify template makes use of Vue.js

The voila-gridstack and voila-vuetify templates are still at an early stage of development, but will be iterated upon quickly in the next weeks as we are exploring templates.

A Jupyter server extension

Beyond the voila command-line utility, the Voilà package also include a Jupyter server extension, so that Voilà dashboards can be served alongside the Jupyter notebook application.

When Voilà is installed, a running Jupyter server will serve the Voilà web application under BASE_URL/voila.

The Jupyter Community Workshop on Dashboarding

From June 3rd to June 6th 2019, a community workshop on dashboarding with Project Jupyter took place in Paris. Over thirty Jupyter contributors and community members gathered to discuss dashboarding technologies and hack together.

The participants to the Paris Jupyter community workshop on dashboarding

Several dashboarding solutions such as Dash and Panel were presented during the workshop and featured at the PyData Paris Meetup which was organized on the same week.

The workshop was also the occasion for several contributors to start working on Voilà. Custom templates, a dashboard gallery, logos and UX mockups for JupyterLab extensions have been developed.

We will soon publish a more detailed post on the workshop, detailing the many tracks of development that have been explored!

What is coming?

There is a lot of planned work around Voilà in the next weeks and months. Current work streams include better integration with JupyterHub for publicly sharing dashboard between users, as well as JupyterLab extensions (a Voilà “preview” extension for notebooks, and a WYSIWYG editor for dashboard layouts). There are also ongoing discussions with the OVH cloud provider (which already supports binder by handling some of its traffic) on hosting a binder-like service dedicated to Voilà dashboards. So stay tuned for more exciting developments!

Last but not least, we are especially excited about what you will be building upon Voilà!

Acknowledgments

The development of Voilà and related packages at QuantStack is sponsored by Bloomberg.

We are also grateful to the attendees of the Jupyter Community Workshop on Dashboarding for their numerous contributions to Voilà!

We would like to thank Chris Holdgraf for his work on improving documentation, and integration with JupyterHub.

We should mention Yuvi Panda and Pascal Bugnion for getting the voila-gallery project off the ground during the workshop. We are grateful to Zach Sailer for his continued work on improving jupyter_server. We should finally not forget to mention the prior art by Pascal Bugnion with the Jupyter widgets server which was also an inspiration for Voilà.

About the Authors

Sylvain Corlay is the founder and CEO of QuantStack and a core team member for Project Jupyter.

Maarten Breddels is an independent scientific software developer partnering with QuantStack on numerous projects, and a core developer of Project Jupyter.

Jupyter Blog - Sylvain Corlay

JupyterLite Officially Joins Project Jupyter!

What is JupyterLite?

The Journey of JupyterLite

A Natural Fit with JupyterLab

Better integration with the rest of Jupyter

Try it in your browser

Acknowledgments and team credits

ipydatagrid is now part of Project Jupyter

What is ipydatagrid?

Key Features

The transfer to Project Jupyter

Bloomberg’s contribution to Project Jupyter

JupyterGIS

Collaborative workflows in geosciences

From the desktop to the web

Real-world applications

Building collaborations in Europe and beyond

Affiliations

🎉 JupyterCon 2023 recordings now live on YouTube! 🎉

Improving the accessibility of Jupyter

CodeMirror 6 and Notebook 7

Initial accessibility audits

Beyond accessibility audits

Acknowledgements

About the Authors

JupyterCon 2023 keynotes

Jupyter Community Workshop: Jupyter for Education

MyBinder @ OVHcloud

A new partnership with NumFOCUS

Acknowledgement

Jupyter Community Workshop: JupyterLite

Jupyter Community 2021 Update

New committee announcement

We’re hiring!

Jupyter Community Workshops

JupyterCon

Acknowledgements

About the Authors

Enabling the JupyterLab debugger with ipykernel

What will change with ipykernel 6.0?

Try it now!

What about the future?

Acknowledgements

About the authors

A Curiously Recurring Widget Library

Using the RAII pattern for a widget library

Static polymorphism and the CRTP pattern

CRTP in xwidgets — closing the recursion

Precompilation and binary size optimization

Xproperty: an implementation of the observer pattern

Method chaining for widget initialization

Building upon xwidgets

C++ should allow overloading the dot operator

Try it online!

About the author

Benjamin Ragan-Kelley

The templating system of nbconvert 6

From Classic to Lab

My notebooks look different!

But I wanted my notebooks to look the same!

Executing the notebook before rendering

Nbconvert templates

Selecting a template

Where are nbconvert templates installed?

The content of nbconvert templates

The conf.json file

Jinja templates

Inheritance in Jinja

Building a custom template

From nbconvert templates to Voilà templates

Where are Voilà templates installed?

Overriding nbconvert templates with Voilà

Future developments

Acknowledgements

About the Author

SlicerJupyter: a 3D Slicer kernel for interactive publications

Powerful Medical Imaging Capabilities Available Through Jupyter

Interactivity Levels

Try it online!

The `conf.json` file