Jupyter Blog - Frédéric Collonval

A theme editor for JupyterLab

2023-06-19T08:11:00+00:00

JupyterLab is a comprehensive web user interface for scientific and technical computing providing tools such as notebooks, text editors, consoles, and terminals. Like many similar tools, it allows users to choose from light or dark color themes. However, users may want to further adjust the looks of the interface, just for fun or for specific personal preferences or needs.

The ability to fine-tune contrast, color palettes, and fonts can be very useful for accessibility, an essential requirement for software to be usable by the whole community. There is an ongoing project-wide effort to improve accessibility in Jupyter. Choices of color palettes impact color-blind users while font choices can have a significant impact on people affected by dyslexia, or who suffer from migraines.

This diversity of requirements and preferences shows that end users should have the means to adjust parameters and tweak existing themes. Subtle differences can improve comfort significantly.

JupyterLab themes

Theming in JupyterLab is enabled by the fact that most of the layout and colors of the UI are defined by a set of CSS variables. There is no need for a complete stylesheet: one can set values for the base parameters from which the appearance of the user interface is derived. This system ensures a consistent look and feel throughout the application.

Custom themes provide a set of values for the base CSS variables and package the resulting CSS file in JupyterLab extensions. A search on GitHub returns 29 repositories defining such custom themes for JupyterLab.

Creating a theme requires implementing values for dozens of base CSS variables. This is often achieved by theme authors tweaking the values given in the default dark or light themes.

Generating a consistent palette, testing, and packaging the outcome in the form of a JupyterLab extension requires development skills and is neither direct nor trivial.

A new tool for authoring JupyterLab themes

To lower the bar for customizing JupyterLab we created a new tool providing a simple interface for tuning the JupyterLab appearance interactively, allowing theme authors to:

use the selected colors in their application and “pin down” the result in the configuration,
export the outcome in a form amenable to packaging into a new JupyterLab theme.

The theme editor extension is a theme prototyping tool, a sandbox to test changes in colors, font family and font size, and a configuration tool for end users. It displays a reduced set of parameters one can play with, allowing users to select:

colors through color pickers,
numerical values from sliders (for font size, border radius and width),
font families from a predefined dropdown list.

A screenshot of JupyterLab with the theme editor panel is displayed in Figure 1, with a notebook opened in light theme. Figure 2 shows different screenshots with different custom themes.

Figure 1: The theme editor extension in JupyterLab.

Figure 2: Examples of dynamically editing the color and fonts scheme of JupyterLab with the theme editor.

Color palettes

For all the colors (accent, brand, border, error, info, layout, success, warn), the same logic is applied. The user picks a base color and a palette is automatically calculated using Microsoft’s fast-colors library.

Let’s take the specific example of the layout colors impacting the background of most elements of the interface. They are defined using --jp-layout-color[i] CSS variables from white to light grays (for the light theme).

--jp-layout-color0: white;
--jp-layout-color1: white;
--jp-layout-color2: var(--md-grey-200);
--jp-layout-color3: var(--md-grey-400);
--jp-layout-color4: var(--md-grey-600);

The number of colors in the palette fits the number of colors in the CSS set of variables (here the palette contains 5 colors). There are different ways to define colors (hexadecimal, RGBA, HSLA). The latter format has the advantage of giving direct access to luminance L, a measure of the lightness of a given color.

An inverse layout palette is computed for use with most text elements. The luminance of the inverse layout color L’ is calculated as 1-L plus a correction depending on how 1-L is close to 0.5. There isn’t a general CSS rule in JupyterLab concerning elements on top of a background but some PRs were proposed to reinforce the coupling between inverse layout elements and layout backgrounds with the same color index. The contrast still needs to be improved though, by using other palettes or better corrections when defining the inverse layout colors.

The full process just described from picking a layout color to resulting palettes and the corresponding interface is illustrated in Figure 3.

Figure 3: From a color to the layout, and inverse layout palettes with the resulting interface.

Font size and font family

As mentioned previously, font size and font family can be tuned too. Let’s take the example of the --jp-ui-font-size[i] CSS variables defining the text size of most of the text elements in the interface:

--jp-ui-font-scale-factor: 1.2;
--jp-ui-font-size0: 0.8333em;
--jp-ui-font-size1: 13px;  /* Base font size */
--jp-ui-font-size2: 1.2em;
--jp-ui-font-size3: 1.44em;

A slider lets the user control the base font size associated with --jp-ui-font-size1. The other font sizes are then calculated by applying a scale factor to the base font.

Concerning font families, a non-exhaustive list is proposed (i.e., default browser fonts or Google fonts). It includes both standard accessible fonts (Arial, Courier, Helvetica, Sans Serif, system-ui, Times New Roman, Verdana) and less usual ones to create artistic or special interfaces with cursive fonts ( for instance Dancing Script or Single Day). Only 2 fonts are proposed for code: Space Mono and monospace.

Storing the new theme

Once satisfied with a new theme, users can export it using a button at the top of the theme editor panel. This creates a variable.css file that can be packaged in a theme extension (see the extension template). The new parameters are also synchronously saved in the settings editor and can be restored for the next JupyterLab opening if the boolean useSettings is set to true. If not, the interface will look like in Figure 1: with light theme default CSS values. Resetting useSettings to true will restore the formerly tuned parameters.

Future development

The look and feel of the theme editor UI still needs to be improved. We will work on unifying the styling of all interfaces making use of react-jsonschema-form. It is already used for the JupyterLab settings editor, the notebook metadata editor of JupyterLab 4.0, and several extensions.

Moreover, the current implementation is bound to fast-colors palettes and we may want to use a different approach, like decoupling the layout and inverse layout palettes to reach better contrasts or give more freedom in the color choices.

Any help (e.g., filling issues for bugs or enhancement requests, opening pull requests) to improve the extension is welcome.

Try it out

You can install the PyPI package by running:

pip install jupyter_theme_editor

You can also try the theme editor on Binder.

Aknowledgement

Many thanks to Isabel Paredes for drawing the palette icon!

About the authors

Florence Haudin is a scientific software developer at QuantStack. She contributes to JupyterLab and to ipyleaflet.

Frédéric Collonval supervised this work as a technical director at QuantStack. He is a member of the JupyterLab core team and authored several JupyterLab extensions.

Jupyter Notebook format workshop outcomes

2023-04-12T17:40:00+00:00

Workshop Social Event challenging our senses

The Jupyter Community Workshop on the notebook file format took place at the Safran Campus near Paris from February 28th to March 2nd. It was a great opportunity to gather various Jupyter stakeholders from private and public affiliations to bootstrap new features for the Notebook file format.

Jupyter Community Workshops are a series of events designed to bring together small groups of Jupyter community members and core contributors for high-impact strategic work and community engagement on focused topics.

Community discussions

The community has lots of ideas to improve the Notebook format. So we split in three smaller groups with the aim of drafting Jupyter Enhancement Proposal (JEP): the markdown group, the text-format group and the cell types group.

The markdown group focused on backward compatible enhancement for the Markdown cells. The discussions focused on two subjects:

The specification of the Markdown flavor (pre-proposal): the goals are to specify which Markdown flavor (e.g. GitHub, CommonMark, MyST,…) is used for the cell source, how to store rendered output for easier cross-compatibility and what is the default Markdown flavor.
The persistence of user expression (pre-proposal): in order to display inline expressions within Markdown cells, the results obtained from the kernel should be stored in the notebook. This proposal aims to define the schema modification to store such information.

The text-format group lays out a specification for an official Jupyter notebook textual format. The discussion went on after the meeting to prepare that proposal.

Finally the cell-types group took the hypothesis of starting from the blank page to create the best Jupyter notebook format building on top of 10-years of experience. The discussion will take time to settle down on a new specification. So if you are interested, join the weekly discussion (see the meeting notes for all the details). In addition to the fully new specification, two backward compatible enhancements have been proposed:

Adding $schema to the notebook format and deprecate the nbformat version keys (see proposal).
Adding extraSchema to the notebook format to optionally extend the schema to specify in particular metadata (see pre-proposal).

Follow-up

Six JEP’s are foreseen from the workshop discussions. But as mentioned earlier, the community has lots of great other ideas (like SQL cells, low-/no-code cells for inputs or visualization). So we would like to encourage anyone interested by any Jupyter enhancement to open issue on the Jupyter Enhancement Proposals repository (see the guidelines for more information).

With the new Jupyter governance in place, the new Software Steering Council is responsible for ensuring those proposals get reviewed and go through the approval process.

Acknowledgements

I deeply want to thank all participants to the workshop that took the time (some of them despite time zone difference) to bring very constructive and thoughtful discussion.

We are really grateful to Bloomberg and Amazon Web Services for their donations to the Jupyter Community Workshops program. This event would not have been possible without their generous support.

We are also grateful to Safran Group for hosting this workshop and the NumFOCUS foundation for helping and mentoring this workshop organization.

Improving the accessibility of Jupyter

2023-02-24T11:11:00+00:00

The adoption of Jupyter has grown significantly in the past few years — especially in education contexts. The project has become a foundational component of our digital experience, from the first lines of code written by high-school students to the most advanced research and engineering use cases.

In this context of global adoption, it is even more important to ensure that as many people as possible can use the project. It is estimated that 15% of the population has a disability that may impair their ability to use online services. If we don’t want them to be excluded from learning sciences, technology, and engineering, we must improve the tools to make them usable by everyone…

To address this issue, a Jupyter accessibility working group formed and has now become an official Jupyter Subproject and received a grant from the Chan Zuckerberg Initiative.

Creating accessible user interfaces is a challenging task as it requires accommodating a broad range of disabilities, including vision, motor, and cognitive impairments. This article presents some of the recent accessibility improvements in the Jupyter Notebook codebase.

CodeMirror 6 and Notebook 7

Jupyter Notebook 7, which is the next major release of the Jupyter Notebook frontend, has been rebuilt with modern JupyterLab components and extensions while maintaining the classic Jupyter Notebook user experience. Notebook 7 will be released shortly after JupyterLab 4 and will take advantage of JupyterLab improvements.

Improving the accessibility of Jupyter had long been impeded by significant obstacles. The primary obstacle was that the text editor underlying the Jupyter Notebook (CodeMirror 5) had major accessibility issues.

Fortunately, this accessibility bottleneck has been unblocked as JupyterLab has been upgraded to use CodeMirror 6, a complete rewrite of the text editor with a strong focus on accessibility. Although this upgrade required extensive codebase modifications, the changes landed a few months ago and will be available with JupyterLab 4. Jupyter Notebook 7 will benefit from the CodeMirror 6 upgrade.

Initial accessibility audits

Shortly after the CodeMirror 6 migration landed, we made an automated audit of the accessibility of Notebook 7 with these changes, and found that the number of warnings and errors reported by Axe Auditor went down from several hundreds to a few dozen, most of which seemed fixable. Encouraged by these results, we decided to work on bringing that count to zero!

With a series of fixes ranging from simple changes to the DOM structure of components to fixing up base Lumino components (Lumino is a JavaScript framework that underlies a lot of the JupyterLab frontend architecture), we were able to make the Notebook 7 codebase pass the Axe Auditor tests with zero error or warning. While we have not reached that yet in JupyterLab, both JupyterLab and Jupyter Notebook benefit from these improvements made for Notebook 7 since they are built from the same components.

Axe Auditor output with the “classic” Notebook UI (v6) with a test notebook.

Axe Auditor output with the “next” Notebook UI (v7 alpha 13) with the same test notebook.

Similar results were achieved with the file explorer and text editor tabs of the application.

Beyond accessibility audits

Improving the accessibility of Jupyter requires more than just addressing issues flagged by automated audit tools and the Jupyter Accessibility Subproject is working on improving accessibility across the board in Project Jupyter. These results with the Notebook 7 codebase are very encouraging, but working with end-users and getting their feedback will also be necessary to make the user interface truly accessible.

To address the diversity of accessibility requirements, we will approach specific use cases separately (for example, users with screen readers, or users who can operate a keyboard but not a pointing device). While the resulting changes may improve usability for everyone, we need to learn from users who have specific needs to make meaningful improvements.

Finally, many of the accessibility challenges in Jupyter stem from the content of the notebooks, as notebook authors may not follow the best practices to make their content usable by everyone (such as adding alt text to images and figures, or properly using headings to communicate the organization of a notebook). To promote these best practices, Jupyter could provide linting tools that produce inline warnings and hints for notebooks that do not follow these guidelines.

Acknowledgements

The work by Johan Mabille at QuantStack on migrating JupyterLab to use CodeMirror 6 was funded by Two Sigma. This upgrade also enabled significant performance improvements in the rendering of Jupyter notebooks, as detailed in this earlier post by Frédéric Collonval.

We are grateful to members of the Jupyter Accessibility Subproject who produced a thorough review of the Notebook 7 UI with a focus on keyboard navigation. This is an example of an issue that cannot be easily detected by auditing tools.

About the Authors

Nicolas Brichet is a scientific software developer at QuantStack active in the Jupyter ecosystem. Among other things, Nicolas made significant contributions to the JupyterLab and Voilà projects, and worked on porting the nbgrader package to JupyterLab. Nicolas worked on accessibility improvements in the JupyterLab and Lumino packages.

Johan Mabille is a technical director at QuantStack. He was honored with the Distinguished Contributors award in 2020 for his contributions to the Jupyter project. Among other things, he is one of the main authors of the JupyterLab visual debugger, and the creator of Xeus, a C++ implementation of the Jupyter kernel protocol at the basis of many Jupyter kernels. Johan was responsible for the migration of JupyterLab to CodeMirror 6.

Jeremy Tuloup is a technical director at QuantStack, and a Jupyter Distinguished Contributor. He is a core maintainer of JupyterLab and Voilà, and the creator of the JupyterLite project. Jeremy is the main initiator of the Notebook 7 project.

Frédéric Collonval is a technical director at QuantStack, and a Jupyter Distinguished Contributor. He is a core maintainer of the JupyterLab project. Frédéric contributed to the CodeMirror 6 migration and helped numerous new contributors to get their enhancements accepted.

Sylvain Corlay is the founder and CEO of QuantStack, and a Jupyter Distinguished Contributor. He has worked on many areas of the Jupyter project, from interactive widgets to language kernels and other core components.

Jupyter Community Workshop: The notebook file format

2022-12-08T16:09:00+00:00

We are excited to announce the next in-person Jupyter Community Workshop! It will focus on the Notebook file format.

Jupyter Community Workshops are a series of events designed to bring together small groups of Jupyter community members and core contributors for high-impact strategic work and community engagement on focused topics.

The Jupyter notebook file format has been around for 10 years. Its usage has grown in countless fields from teaching to data analysis in production pipelines. A great number of software applications and online services have added support for it.

We want this workshop to be an opportunity for various stakeholders to push forward the format while preserving its reusability in as many applications as possible. We could for example prototype a syntax for injecting variable values in Markdown cells, specify an alternative more textual format like RMarkdown, define the Markdown variant we support,… the boundary is our imagination. By the end of the workshop, we will submit those new specifications as Jupyter Enhancement Proposals to kick start the validation process for enhancing the official notebook format.

The workshop will last three days, with hands-on discussions, hacking sessions, and technical presentations. The goal of this event is to foster collaboration and the sharing of knowledge between maintainers of various platforms supporting the file format, downstream library authors and power users.

The workshop will be held at the Safran Campus in Paris suburb, France from February 28th to March 2nd, 2023. Travel funding assistance is available for attendees from academia and those from groups which are not well-represented within the Jupyter and wider tech community!

Application and all other details can be found in this form.

We are grateful to Safran Group for sponsoring this event. We are also grateful to the sponsors of the Jupyter Community Workshop series, Bloomberg and Amazon Web Services.

Accelerating JupyterLab

2022-10-10T17:38:00+00:00

The next major release of JupyterLab will be significantly faster than previous versions. This was achieved both through systematic tracking of performance bugs and through significant upgrades to the Jupyter communication protocol and rendering mechanism for documents.

1. Setting up rigorous performance measurements

The first step to any measurable improvement in performance is to set up systematic measurement of performance.

The JupyterLab project now includes a UI performance benchmarking tool, in the form of a GitHub action that can be triggered on any pull request to check how performance is impacted by the change. The implementation of this new GitHub action is available in this repository: https://github.com/jupyterlab/benchmarks.

This tool measures the time required for performing the following actions: opening a test notebook, switching from the test notebook to a copy of it opened in another tab, switching from the test notebook to a text editor, switching back, searching for a word in the test notebook and closing the test notebook. There are multiple example notebooks in the test suites. Benchmark results are posted as comments on the pull request. You can see such a benchmark report here: #11494#issuecomment-976393815

Example report from the new benchmarking tool — each execution time distribution is represented by a box-plot graph (the box spans from the 1st to the 3rd quartiles with the white line positioned at the median value).

The addition of this benchmarking tool immediately allowed for optimization on how notebooks are hidden when switching tabs. Hiding can be done by adding a CSS class that enables some CSS rule, or forcibly setting display to “none”. Depending on the browser, picking one way or another of hiding content may trigger a reflow of the entire page, so we made this a settable with an option in the JupyterLab config.

The benchmark GitHub action was developed by Frédéric Collonval.

2. Upgrading to CodeMirror 6

The rendering of the text editor used in notebooks can be very expensive, especially in the case of large notebooks with many cells. Jupyter has historically relied on CodeMirror as its based text editor.

JupyterLab 4 includes an upgrade from CodeMirror 5 to CodeMirror 6, which is a complete rewrite of the text editor. This work can be found in pull requests #11638, #12877, and #12861 — modifying over 150 files of the JupyterLab codebase. Benchmarks indicate a rendering speedup factor between 2 and 3 on the large notebooks used in the benchmarking suite.

Benchmark report on the CodeMirror 6 migration PR

Note: CodeMirror 6 is also an important stepping stone towards making Jupyter notebooks accessible to people who need screen readers and other devices.

The migration of JupyterLab to CodeMirror 6 was performed by Johan Mabille.

3. Virtual rendering of notebooks

In JupyterLab 4, the notebook will only render the parts of the documents that are visible in the viewport. It significantly improves the rendering speed of large notebooks. The main pull request implementing this feature is available here: #12554.

Significant preparation work was required for this Pull Request, especially regarding the “search feature” and the “table of content” components that both made use of the notebook view instead of the document model. This was done in PRs #11689 and #12374 respectively.

The end results showed significant improvement in the rendering speed of large notebook files, with a speedup of 3 to 4, which come on top of the already improved performance from the CodeMirror 6 migration.

Benchmark report on the Virtual Rendering of notebooks

The virtual rendering of notebooks was developed by Frédéric Collonval.

4. Jupyter protocol alignment

The Jupyter server serves as a relay between the frontends such as JupyterLab or the notebook and kernels. The server ⇄ kernel communication is done over ZeroMQ sockets, with the well-specified Jupyter kernel protocol. The server ⇄ client communication is done over WebSockets.

Unfortunately, up until recently, the server ⇄ kernel (ZMQ), and the server ⇄ client (WebSocket) protocols differed slightly so that the server had to parse each message and re-serialise it in both directions. This processing cost is small for short messages such as execution requests and replies which are typically very short, however, it can become very costly when dealing with larger datasets being sent or retrieved from the front-end, such as large tables, complex mime type rendering. This misalignment of the ZMQ and WebSocket protocol can then become a real bottleneck.

In Jupyter Server 2, the WebSocket connection supports a new “aligned” protocol, in which messages can simply be copied over to and from ZeroMQ messages, which is supported by JupyterLab 4. (This work was done in PRs #657 (jupyter-server), #154 (jupyverse), and #11841 (JupyterLab)). This new aligned protocol is an opt-in, so that legacy Jupyter front-end are still expected to function with Jupyter Server 2.

Benchmarks indicate a large speedup factor (at least one order of magnitude, and more for larger messages) in the performance of the Jupyter server when displaying large data sets in Jupyter widgets. However, this is not captured by the JupyterLab benchmark tests which focus on the rendering performances.

The procol alignment work was done by David Brochart.

5. Lumino 2

The JupyterLab frontend is built upon the Lumino framework, which provides utilities for building in-browser desktop-like applications. It provides the foundations for such applications, including a uniform component wrapper that handles lifecycle management and efficient propagation of front-end events to an entire application, (e.g., resize events, drag-and-drop, layout calculation). Lumino also provides several high-performance components such as a drag-and-drop dock panel (used as the application shell for JupyterLab) and a best-in-class data grid component.

JupyterLab 4 includes a major upgrade of the Lumino. The main changes in Lumino 2 include the migration to ES2018, which allowed for the removal of large parts of the codebase prodiving features that are now natively available in JavaScript, such as native iterators, removing polyfills for promises, and special-case logic for idiosyncrasies of legacy browsers like IE. This upgrade is also leading to across-the-board performance improvements in the front-end, although not for the rendering of large documents. Lumino 2 supports background processing of UI components when the application resides in a background browser tab (a feature that may be back-ported to Lumino 1.x as well).

The Lumino 2 upgrade and integration in JupyterLab was done by Afshin Darian.

6. A faster Lumino data grid

Optimizations to the Lumino data grid widget were also implemented, speeding up the rendering in the case of merged cells (cf. PR #394). The Lumino datagrid is used in various parts of the JupyterLab UI, such as the table view for CSV files. It is also used extensively in third-party extensions such as the ipydatagrid Jupyter widget and the BeakerX table display.

The Lumino data grid optimization was done by Martin Renou.

Acknowledgement

The work by the QuantStack team on JupyterLab performance improvements was done in collaboration with Two Sigma. Several of these pull requests required major changes across the JupyterLab codebase. We are very grateful to Two Sigma for supporting the development of the Jupyter project at such a deep level.

We are grateful to Juliette Taka for the illustrations.

About the authors

Frédéric Collonval, who led the charge on JupyterLab performance improvements, is a technical director at QuantStack. He is a member of the core JupyterLab core team and authored several JupyterLab extensions.

Johan Mabille is a technical director at QuantStack, very active in the Jupyter ecosystem. He regularly contributes to JupyterLab, and developed the Xeus framework for creating Jupyter kernels.

David Brochart is a scientific software developer at QuantStack, very active in the Jupyter ecosystem. He is a maintainer of the Jupyter-server project, and the main author of Jupyverse. David also contributes to the geo-science open-source stack built atop Jupyter.

Afshin Darian is a technical director at QuantStack. He is the co-creator of the JupyterLab project and continues working on the project to this day.