Authoring Custom Jupyter Widgets

2018-03-05T12:12:00+00:00

Jupyter interactive widgets enhance the notebook experience by allowing users to create graphical user interfaces. They enable richer interaction with the data and computing resources.

While the base ipywidgets library comes with a number of controls such as sliders, buttons, and dropdowns, it is in fact much more than a collection of basic controls: it is the foundation of a framework upon which one can build arbitrarily complex interactions.

Examples of custom widget libraries built upon the foundational package are

bqplot, a d3-Jupyter bridge, and a 2-D plotting library following the constructs of the Grammar of Graphics,
ipyleaflet, a leaflet-Jupyter bridge enabling maps visualization in the Jupyter notebook,
pythreejs, a 3-D visualization library bringing the functionalities of Three.js into the Jupyter notebook,
ipyvolume, a 3-D plotting library also based on Three.js enabling volume rendering, quiver plots and much more.

Jupyter Widgets, a Bridge Between Two Continents

Jupyter widgets provide a means to bridge the kernel and the rich ecosystem of JavaScript visualization libraries for the web browser. It is an amazing opportunity for scientific developers to use all these resources in their language of choice.

This article is meant to serve as a guide for developers interested in authoring a custom widget library and bridge the gap between the basic examples of the official documentation and fully-fledged visualization libraries like the ones listed above.

A number of resources are provided alongside the article, including the complete source code of the examples, notebooks, and Binder links.

In this post, we focus on the Python back-end, even though dozens of Jupyter kernels exist. The Python kernel is the reference implementation of the Jupyter protocol and remains the most featureful. We should also mention QuantStack’s Xeus, a native C++ implementation of Jupyter kernel protocol, which supports interactive widgets. Xeus is used as the foundation for the support of Jupyter widgets for the R and C++ kernels.

Xeus: C++ implementation of Jupyter kernel protocol

A Hands-On Guide

While the Jupyter community is thriving, and we start seeing a growth in the number of custom widget libraries as well. Although the learning curve from the examples of the official Jupyter documentation to authoring state-of-the-art libraries like the ones listed above is steep and may be intimidating.

We started on this path a few months ago and benefited from guidance from core Jupyter developers along the way. Now, we would like to share the lessons learned. We hope this will help turn this mountain trail into a new silk road!

Let us start with the ‘Hello World’ example widget from the ipywidgets documentation, before moving on to more advanced use cases.

1 — Improving on the Hello-World Example from the Documentation

This section is derived from the ipywidgets documentation. The code snippets are CC-0 licensed.

The idea behind Jupyter widgets is to enable a bi-directional communication channel between the kernel (back-end) and the JavaScript front-end. In Python, widgets are special objects which are automatically synchronized with a counterpart object in the JavaScript front-end. In the back-end, the change events are handled by the traitlets package, which implements the observer pattern, while on the JavaScript side, this is done with the Backbone.js library.

The front-end implementation follows the MVC (Model View Controller) pattern. This allows the rendering of the same widget in multiple cell outputs, where all views share the same model, analogous to printing out a string variable multiple times.

The official documentation includes a Hello-World example widget, which offers an example of synchronization from Python to JavaScript, but not the other way around. Our jupyter-widget-hello-world-binder example provides a slightly more advanced version of it which demonstrates the bi-directional communication between the Python kernel and the JavaScript front-end. You can experiment with this widget on Binder or simply check out the example notebook with nbviewer.

Let us briefly present the main aspects of the implementation.

The JavaScript front-end defines a custom view by extending the base DOMWidgetView class from the base package:

var HelloView = widgets.DOMWidgetView.extend({
        render: function() {
            this.value_changed();
            this.model.on('change:value', this.value_changed, this);
        },

        value_changed: function() {
            this.el.textContent = this.model.get('value');
        }
    });

The Python back-end extends the corresponding base class from the ipywidgets package:

class HelloWidget(widgets.DOMWidget):
    _view_name = Unicode('HelloView').tag(sync=True)
    _view_module = Unicode('hello').tag(sync=True)
    _view_module_version = Unicode('0.1.0').tag(sync=True)

    value = Unicode('Hello World!').tag(sync=True)

The value attribute get synchronized between the back-end and the front-end.

The value_changed callback, which changes the HTML representation of the widget is attached to changes of the value property with the line:

this.model.on('change:value', this.value_changed, this);

In the Hello-World example from the documentation, there is no way to change the JavaScript value from the notebook front-end. We have added this feature in order to illustrate the bi-directional synchronization between JavaScript and Python. The idea is to trigger an event in the browser which will update the JavaScript model and then automatically - that’s the magic of ipywidgets - the Python back-end. These additional lines trigger the model update:

// update the JavasScript model
this.model.set('value', formElement[0].value);

// sync with Python
this.touch();

Conversely, when changing the value from the Python kernel the corresponding JavaScript value gets updated, as explained above.

Check out the jupyter-widget-hello-world-binder repo for more information. The notebook includes additional details and comments.

2 — First Example Involving Bi-Directional Communication

In the previous section, widgets were entirely defined in the notebook. We now show how to move the implementation outside of the notebook document and produce a proper installable package.

To help you create your own custom widget, the Jupyter team provides a cookiecutter template, producing a custom Jupyter widget library containing all the boilerplate for packaging. The cookiecutter is initialized with the hello-world widget from the documentation.

We used the widget cookiecutter to put the hello-world widget presented in the previous section into a well-organized GitHub repository. It contains

the Python part in the first_widget folder
the JavaScript part in the js folder.

The README provides detailed instructions to install the package, together with information about how to enable the Jupyter extension, and tips for custom widget authors. It delves into the technical details a bit more than this overview blog post.

Now that the basics of two-way synchronization are covered, you can create arbitrarily complex widgets! The key is to identify the data you would like to synchronize between the front-end and the back-end. Then you can set up the events that will update this data when a change is detected using the building blocks above.

Now you can ‘widget-ify’ any JavaScript library!

2.2 — Barebones `setup.py`

The setup.py described in the official documentation tries to automate many of the build steps, at the cost of readability. Fortunately, packaging Jupyter widgets will work with the bare-bones setup.py we provide.

The gain is a clearer, lighter (~100 less lines) setup.py, giving you a better understanding of what is happening.
The drawback is that you need one extra step to install the widget from source.

# Example: d3-slider
$ git clone https://gitlab.com/oscar6echo/jupyter-widget-d3-slider.git
$ cd js
$ npm install
$ cd ..
$ pip install -e .

# Extra line `jupyter nbextension install`
$ jupyter nbextension install --py --symlink --sys-prefix jupyter_widget_d3_slider
$ jupyter nbextension enable --py --sys-prefix jupyter_widget_d3_slider

More importantly we thought that it was clearer to keep the build steps of the Python and JavaScript packages separated. Both build processes are well documented, making the steps easier to follow. Finally, the inclusion of compiled JavaScript bundles in the Python package, is more explicit.

2.3 — Side Note: Naming Conventions

Naming conventions for Python packages are covered by PEP8. They can be a bit tricky in the case of 2-words names like “first-widget”. Where to use underscore (_) or hyphen (-) ? Typically “-” is used in GitHub repositories, and URLs and JavaScript while any folder or file in a Python module can only contain “_”. In the case of a Jupyter widget there is an extra attention point: in the setup.py file, the data_files argument in the setup function is a list of tuples. For each, the first element (representing a path in the filesystem) contains “-” as it relates to JavaScript code while the paths in the second contain “_” as they represent paths in the Python package. For a full example see the first-widget repo and the detailed README.

3 — Increasingly Complex Widgets

We made the following three widgets, gradually adding complexity. The first two examples are meant as educational examples:

d3-slider, a d3.js-based slider,
drawing-pad, a 2-D drawing pad,

We hope that the last example will become more than a demonstration:

ipypivot, a visual Pivot Table UI

In each case, the GitHub repository includes a demo notebook and the required boilerplate for Binder.

3.1 — d3-slider

This custom d3-slider widget wraps a simple custom slider based on the fantastic d3.js library. You can run and try it on the Binder repo or watch it on nbviewer.

d3-slider widget

To install:

$ pip install jupyter_widget_d3_slider

3.2 — drawing-pad

This small drawing pad app, is inspired from this codepen. You can run and try it on the Binder repo or watch it on nbviewer.

drawing pad widget

To install:

$ pip install jupyter-drawing-pad

3.3 — ipypivot

The ipypivot widget, wraps the convenient PivotTable.js library. You can run and try it on the binder repo or watch it on nbviewer.

ipypivot widget

To install:

$ pip install ipypivot

$ conda install -c conda-forge ipypivot

NOTE: The PivotUI widget is a combo of a custom widget and core widgets. This modular approach is more flexible (and looks nicer) but the same features can be made in a single custom widget containing extra buttons and display fields. The alt branch of the repo contains this version.

3.4 — Including JavaScript Callbacks

The ipypivot widget is an example of a widget that transparently wraps a JavaScript library. Transparent in the sense that all parameters of the JavaScript API are exposed, including functions, which are exposed in the form of strings on the Python side. In Python, JavaScript functions can only be strings, so there is an eval() to convert them to actual JavaScript functions.

The benefit is that all the functionalities of the JS libraries are exposed to the Python users with a very thin API. Developers can ‘widget-ify’ a large array of interesting libraries, thereby boosting the productivity of a Jupyter notebook user.

The downside is naturally the security concerns of enabling arbitrary JavaScript code to be injected by the notebook users. It is less a concern in the context of notebooks being shared within a small team of coworkers.

We are currently exploring means to execute the user-provided arbitrary JavaScript function in a sandboxed fashion, for example using the iframe srcdoc field (Cf. this repo for an example, though not in a Jupyter widget context) and messages can be sent back and forth between the main page and an iframe with the Window.postMessage function (see this gist for a bare bones example).

3.5 — Enabling Jupyter Widgets By Default

The jupyter nbextension enable command, arguably cumbersome, has become unnecessary as Jupyter widgets can be enabled by default from notebook version 5.3 (included). See this PR.

In order to be future-proof, all the widgets in this article include the file which triggers this ‘automatic enable’, and require notebook >= 5.3. Thus the pip-installation of our widgets is a one-line command. However, in dev mode, you still need to install and enable the notebook example (see section 3.1 for an example).

If you are working with an older version of notebook (run jupyter notebook --version to check), you will have to run the following command after pip-installing a widget:

$ jupyter nbextension enable --py --sys-prefix name_of_the_widget

4 — Packaging and Publishing

4.1 — PyPI and npm

If you have followed the “best practices” so far, packaging shouldn’t be an issue. Indeed, the cookiecutter provides a template of an easily-packageable widget.

Once your package is ready, you can publish it on:

PyPI for the package to be pip-installable
npmjs for the JavaScript extension, which is necessary to use the widget as a standalone application (outside of the notebook), render it with nbviewer, and also in the JupyterLab context.

To do so, you can follow these instructions in the documentation of first-widget.

4.2 — conda-forge

Conda has several advantages over pip:

It is a general-purpose package manager, which allows for non-python dependencies.
Unlike pip, it also has a real dependency solver, which prevents breaking your environments when updating a single package.
It allows creating virtual environments to isolate your projects.
It allows for one-line installation of Jupyter extension, including the enabling of the extension as a “post-link” script (temporary advantage: see the previous section).

Conda packages are available on different channels. The default channel is administrated by Anaconda Inc. The usually recommended channel to upload open source projects is conda-forge, as this article from Anaconda announces. To add this channel to your conda configuration, run the following command:

conda config --add channels conda-forge

Several steps are needed to publish a package on conda forge, using a so-called ‘recipe’:

writing the recipe, which describes how to build the package along with the dependencies required for building and running it
testing the recipe
publishing the package on the conda-forge GitHub by forking their staged-recipes repo
maintaining the package

The conda doc and the conda-forge doc are very clear and give you much more detailed information about this subject. If you do not want to read the full doc, and jump straight to the necessary information for publishing a new package, you may want to have a look at our first-widget repo. The README contains a section describing the publishing process for conda-forge.

4.3 — Automatic Push Script

The sequence of steps to update the version of a Jupyter widget and do all the pushing to the various repositories is quite long.

run npm prepare to build the js in folder static/
Update version in __meta__.py
push package to pypi
compute package sha256
tag repo with version
push to GitHub / GitLab / Bitbucket including tag
push js to npmjs
push to conda-forge (which includes sha256 hash)

If you want to automate the process we advise to have a look at Maarten Breddels’ releash package (release with relish :-) ), and how it is used in the context of ipyvolume and ipysheet.

4.4 — Binder and nbviewer

nbviewer and Binder are two fantastic tools to share both static and live notebooks.

nbviewer requires a URL to a valid notebook JSON file — typically hosted on GitHub/GitLab. To make widgets render in nbviewer, you need (1) to make the JavaScript package for your widget available on npm, (2) to run the notebook with the corresponding JavaScript extension (with the same version), and (3) to save the notebook widget state (in the ‘Widgets’ tab) before pushing it to GitHub / GitLab.
Binder requires a URL to a GitHub repository containing notebooks and a manifest of the dependencies required to run this notebook, which is used to produce a Docker image including all the resources to run the notebook. Check out the mybinder.org documentation to know how exactly to make use of it. Another resource is the BinderHub documentation if you want to host your own deployment of Binder. Either way we also highly recommend the article Binder 2.0, a Tech Guide.

5 — Conclusion

Hopefully you will have learned something reading this article. We believe in the potential of Jupyter widgets and hope that this intermediate-level article will help getting more people involved in the development of the ecosystem.

Note: This article only covers the case of the classic Jupyter notebook. The integration with JupyterLab will be covered in a future article!

If you find bugs or are interested in improving the example widgets presented here, please do not hesitate contact the authors or open a pull request!

About the Authors

Alphabetical order:

Olivier Borderies, Société Générale
Olivier Coudray, Student at École Polytechnique
Pierre Marion, Student at École Polytechnique

The software presented in this post was built upon the work of a large number of people including the Jupyter team. We are especially grateful to Sylvain Corlay, Jason Grout, Paul Ivanov, and Steven Silvester from the Jupyter Steering Council, as well as Maarten Breddels and Pascal Bugnion.

Jupyter Blog - Olivier Coudray

Authoring Custom Jupyter Widgets

Jupyter Widgets, a Bridge Between Two Continents

A Hands-On Guide

1 — Improving on the Hello-World Example from the Documentation

2 — First Example Involving Bi-Directional Communication

2.1 — First Widget

2.2 — Barebones `setup.py`

2.3 — Side Note: Naming Conventions

3 — Increasingly Complex Widgets

3.1 — d3-slider

3.2 — drawing-pad

3.3 — ipypivot

3.4 — Including JavaScript Callbacks

3.5 — Enabling Jupyter Widgets By Default

4 — Packaging and Publishing

4.1 — PyPI and npm

4.2 — conda-forge

4.3 — Automatic Push Script

4.4 — Binder and nbviewer

5 — Conclusion

About the Authors

Jupyter Blog - Olivier Coudray

Authoring Custom Jupyter Widgets

Jupyter Widgets, a Bridge Between Two Continents

A Hands-On Guide

1 — Improving on the Hello-World Example from the Documentation

2 — First Example Involving Bi-Directional Communication

2.1 — First Widget

2.2 — Barebones setup.py

2.3 — Side Note: Naming Conventions

3 — Increasingly Complex Widgets

3.1 — d3-slider

3.2 — drawing-pad

3.3 — ipypivot

3.4 — Including JavaScript Callbacks

3.5 — Enabling Jupyter Widgets By Default

4 — Packaging and Publishing

4.1 — PyPI and npm

4.2 — conda-forge

4.3 — Automatic Push Script

4.4 — Binder and nbviewer

5 — Conclusion

About the Authors

2.2 — Barebones `setup.py`