Jupyter Blog - Wolf Vollprecht

JupyterLab, the Cloud Robotics Command Station

2020-08-11T13:19:00+00:00

JupyterLab-ROS with Voilà to quickly make a standalone web app

The ROS open-source project (Robot Operating System) has become a de-facto industry standard in the robotics community. Very much like with Project Jupyter, a large collection of software has been built upon well-specified communication protocols. ROS has become a rich ecosystem of standardized tools for building and distributing ROS-based software, exploratory robotics development, and much more.

With Industry 4.0, the number of robots and smart devices has been increasing exponentially. We are talking about fully automated factories with hundreds of robots and thousands of connected sensors, generating large amounts of raw data.

There is an opportunity in bridging the ROS and the open-source data science ecosystems and tools.

For this reason, we set ourselves to bring together these ecosystems,

by producing a collection of JupyterLab extensions for ROS, turning JupyterLab into a cloud robotics command station,
by integrating ROS packages with conda / mamba package management system, to enable their installation alongside other open-source data-science packages.

JupyterLab, the cloud robotics command station

While the traditional developer tools from the ROS stack such as RViz are desktop-based, we think that a robotics development environment running in a web browser would provide a lot of flexibility. Also, the ideal robotics developer platform should be plugin-based, and easily extensible to account for the diversity of use cases to enable, from simple interactive scripting to visualization or large datasets. Finally, we should build this environment upon broadly adopted foundations, with open governance, and not let a single entity hold the keys to the ROS ecosystem.

Jupyter is a natural candidate to serve as the foundation for a web-based robotics development environment.

JupyterLab is a web-based interactive development environment that has many of the much-needed features

In JupyterLab, everything is a plugin, including the core components to the application. Anyone can produce a “remix” of core and third-party JupyterLab extensions to their needs.
Very much in the spirit of ROS, one of the keys to Jupyter’s success is that the project was built upon well-documented and specified protocols and file formats that anyone could implement.
Jupyter is a multi-stakeholder project, not backed by a single corporation, but by a community of developers at a variety of companies, universities, as well as individual contributors.

Today, we are announcing the first beta of JupyterLab-ROS, a collection of JupyterLab plugins to integrate ROS with the JupyterLab platform. We combine the best of the data-science and robotics worlds to help developers build custom solutions for the industry. ROS can run in a high-performance server while JupyterLab is accessed in a web-browser (on Windows, OS X or Linux).

JupyterLab-ROS

JupyterLab-ROS automatically starts a rosmaster and rosbridge_server making all the painful setup of WebSockets connections completely transparent to users. A status bar widget allows you to start and stop the rosmaster server with a simple click and displays its state in real-time. Internally this widget runs a launch file which can be changed from the JupyterLab settings UI to launch additional nodes.

Master status widget

Another important component of the ROS stack is RViz, a desktop-based 3-D visualization tool. It’s web-based counterpart in JupyterLab is Zethus, started by Rapyuta Robotics. This plugin supports most of the display types supported by RViz. It also provides an info panel that displays the raw messages in realtime, and a web-based version of rqt_graph for visualizing the ROS node graph.

Zethus

A robotics station would be incomplete without a tool for debugging. For that purpose, ROS provides rosconsole, a package which allows developers to send messages to rosout and make them available on every node. The console gives access to these debugging messages in JupyterLab and provides additional features such as filtering by level and by node.

Log Console

Robotics Notebooks

A key component to the Jupyter stack is the Notebook, an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. It’s frequently used to do data cleaning and transformation, numerical simulation, statistical modelling, data visualization, machine learning etc.

The notebook could be the perfect tool for robotics researchers and educators to make (and share) experiments in control algorithms, working in a dynamic and interactive development environment that allows quick prototyping and exploratory analysis while having access to the sophisticated mathematical libraries already well integrated with Jupyter. It can be tedious to have a different platform for interacting with robots and processing the data obtained from these, JupyterLab-ROS combines both toolsets and enables an integrated workflow.

At the same time, many projects are becoming more and more interdisciplinary. Currently, it is common to see projects mixing topics like machine learning and robotics. JupyterLab-ROS opens a new world of possibilities for machine learning researchers that are interested in applying their models to robots: now they have the possibility of connecting a robot to a development environment that they are very familiar with.

JupyROS is another powerful library that leverages ipywidgets and bqplot to integrate ROS in Jupyter Notebooks, being able to show real time plots from ROS messages and create intuitive forms to control robots in minutes.

Interactive Robotics in the Jupyter Notebook, with live plotting

ROS Package management

Modern scientific computing applications typically depend on a large number of libraries. Conda-forge is a community initiative to package scientific libraries for the conda package manager, and has become a de-facto standard source of packages for the scientific computing community, with tens of thousands of available packages, and thousands of contributors.

During the last year, we have worked hard to package more and more of ROS as conda packages. Conda-packages for ROS Melodic, including RViz are now available on all three operating systems! You can find the packages on the RoboStack channel (a more detailed blog post about this is going to follow soon).

Conda can be used to create virtual environments for ROS projects, which makes it possible to install Melodic, Kinect and Dashing side-by-side on the same Linux system, and have exact control over package versions.

We are also working on Mamba, a conda-compatible package manager implemented in C++. Together with Boa this effort will help to continuously build and release ROS packages on conda.

Try it online!

You can try the JupyterLab-ROS extension online without installing anything by just clicking on the following binder link:

JupyterLab-ROS is under active development. We are working on new features such as a rosbag widget to play and record bag files and improving existing ones like the roslaunch widget. If you are interested in contributing to JupyterLab-ROS you can find the code at this repository in the RoboStack organization. Contributions are always welcome!

Installation

You can install JupyterLab-ROS using Mamba, the first step to create a new environment with JupyterLab and the ROS packages necessary, then you can install the extension.

mamba create -n test -c conda-forge -c robostack python=3.6 nodejs=12 jupyterlab ros-melodic-ros-core ros-melodic-rosauth ros-melodic-rospy ros-melodic-rosbridge-suite ros-melodic-rosbag ros-melodic-tf2-web-republisher

conda activate test

pip install jupyter-ros-server

If you have any questions don’t hesitate to contact us on the RoboStack or QuantStack chats.

This work has many collaborators around the world, and we are especially grateful to Deep Chaitanya & Ashish Chaudhary from Rapyuta Robotics, Sean Yen from Microsoft for his amazing work on vinca that automagically generates the conda-recipes, Tobias Fischer for many patches and pushing us to bring OS X support to the conda packages as well. Many thanks to Jeremy Tuloup for his help with the JupyterLab extension system.

About the authors

Wolf Vollprecht works as a scientific and robotics software developer for QuantStack in Paris and Berlin.

Carlos Herrero is a Computer Engineer passionate about AI and his applications on robotics. Currently working at QuantStack helping to develop Open Source projects.

ROS @ Jupyter

2019-04-07T21:15:00+00:00

Project Jupyter is a huge hit in data science, but it has not yet found widespread adoption in robotics. Today, we are releasing the first version of jupyter-ros, a collection of Jupyter interactive widgets inspired by Qt and RViz, to bring their features to the Jupyter ecosystem. This may be the right time for Jupyter-based developer tools, as cloud robotics is taking off.

A PR2 robot in the browser, and making him dance through the traditional Qt interface

Historically, the ROS (Robot Operating System) community has relied on Qt for building complex user interfaces. Nowadays, the Jupyter notebook and the ipywidgets framework offer a compelling alternative for several reasons:

Code and interface are not separated — both are in the same notebook.
Complex widgets using browser technology are possible: from JavaScript sliders to 3D with WebGL, real time video streaming with WebRTC…
Works with any web browser — not bound to Linux, and no Qt applications need to be compiled.
Doesn’t need to run locally! Applications can run on a server far away, without any manual setup or installation procedure.

However, ROS does not play nicely with Jupyter. For example, rospy is inherently multi-threaded (every new ROS topic subscriber spawns a new thread). Debug printouts in callbacks result in Jupyter printing that content into the output area of the active notebook cell, quickly flooding the user interface — and there is no easy way to stop logging threads once started.

ipywidgets to the rescue: jupyter-ros

That is why jupyter-ros was created. It is a suite of plugins to the Jupyter ecosystem to make working with ROS inside Jupyter a breeze.

Subscribing with jupyter-ros produces an interactive widget

First, when you subscribe using jupyter-ros, the function returns an ipywidget with a start/stop button and a dedicated output area for debug prints. Internally this re-routes all print outs from your subscriber thread to this Jupyter cell, and gives full control over the thread (by being able to stop and restart it at any time).

The jupyros publishing sugar

If you call the publish function in jupyter-ros, a widget is automatically generated from the message specification. For example, a ROS message String field automatically becomes a text input widget, a Float32 becomes a FloatSlider…

Live plotting IMU data from a BBC micro:bit

Another interesting functionality is the live plotting, similar to rqt_plot. For this we use bqplot, a “jupyter-native” solution to plotting. With jupyter-ros, you can select a couple of fields from your message, and they will be automatically plotted to a bqplot.

Going 3-D

Most robots are three dimensional, and so should be the visualizations! The go-to tool in the ROS ecosystem at the moment is RViz, a very powerful 3-D visualization tool.

Some of RViz’s functionality has already been ported over to the web browser as part of the RobotWebTools effort. We’ve been able to piggy-back on their impressive work, and we’re releasing the first iteration of ROS3D Jupyter widgets. Currently, they allow you to programatically plug together RViz like visualizations for different data types, such as laser scans, robot trajectories, and 3D (URDF) models of the robot!

Thanks to the ipywidgetification, you can now bring complex visualizations to the web, without writing any JavaScript, and arrange those visualizations freely in JupyterLab.

Interactive widgets showing ROS data inside JupyterLab

Jupyter, Widgets, and Cloud Robotics

We hope that Jupyter, and the jupyter-ros widgets will play a big role in cloud robotics. In cloud robotics, some of the software powering one or multiple robots runs on powerful computers in data centers. For monitoring purposes, or development, Jupyter and JupyterLab are perfect candidates. Robot customers will be able to login to a single user-friendly interface, without having to install any custom software on their machine, or run a specialized operating system (ROS usually runs on Ubuntu). Exciting possibilities arise: it is already possible to run a Docker container running JupyterLab and jupyter-ros on the leading cloud robotics platforms, Rapyuta Robotics and their freshly launched platform rapyuta.io.

Installation & source

The jupyter-ros widgets can be installed from PyPI using

$ pip install jupyros
$ jupyter nbextension enable --py --sys-prefix jupyros

The examples and the source code can be found on github: https://github.com/robostack/jupyter-ros

This work is still early stages and you might find some rough edges. We look forward to collaborating with the community to polish these widgets to the highest standards!

About QuantStack

QuantStack is located in the center of Europe (Paris). We are core Jupyter developers, and love ROS, and the cloud. If you are interested in working with us on professional user interfaces for developers or clients in the cloud, do not hesitate to send us an email: wolf.vollprecht@quantstack.net

A Diagram Editor for JupyterLab

2018-02-26T11:33:00+00:00

With the success of the notebook file format as a medium for communicating scientific results, more than an interactive development environment, Jupyter is turning into an interactive scientific authoring environment.

JupyterLab viewing LaTeX source code and a PDF document

The new JupyterLab interface is much more than a replacement for the classic notebook. It aims to bring together all the pieces required for a complete scientific workflow. The extension-based architecture of JupyterLab comes with a number of components already enabled:

a Jupyter notebook,
a text editor,
a file browser in the sidebar,
a number of editors and viewers for various file formats,

and much more. However, some pieces are still missing to complete the picture for a scientific authoring environment. One would be a featureful LaTeX editor. The first LaTeX editor for JupyterLab is a step in the right direction and offers an easy way to live-compile tex documents. Another piece is — of course — a means to produce diagrams, flow charts and draw figures!

Drawing charts and diagrams

On the occasion of the Paris Jupyter Widgets workshop, I started working on a feature to fill that gap and built a JupyterLab extension for the Draw.io diagram editor.

Draw.io is a diagram editor that runs in the web browser and is Apache 2.0 licensed. It’s got a really mature code base, which has been around for many years. However, unlike the other components used by JupyterLab, Draw.io has not yet embraced the new JavaScript packaging tooling such as NPM, which complicated the integration with JupyterLab a little bit, but it all paid off eventually!

Now, I am really pleased to announce the first release of the draw.io extension, a fully fledged integration for JupyterLab of the fully-fledged diagram editor!

The Draw.io JupyterLab extension takes advantages of the JupyterLab architecture: i.e. registering a new mime type (.dio) with the file explorer to open files, and adding a launcher button and menu items. Besides that, multiple synchronized views of the same diagrams can be displayed at the same time, allowing a user to visualize the same content with different zoom levels, or with a bare text editor.

Installation

You can install the jupyterlab-drawio extension with the following command:

jupyter labextension install jupyterlab-drawio

This should set up the extension inside your JupyterLab environment. I hope this will be a useful extension for the larger community. All the code is available on GitHub: https://github.com/QuantStack/jupyterlab-drawio. Don’t hesitate to open issues and come contribute to jupyterlab-drawio.

The future

There are other projects just waiting to be packaged for use inside of JupyterLab: one great webapp for JupyterLab would probably be the ShareLaTeX application, which is Open Source as well and provides a very nicely integrated editing experience for LaTeX documents, with autocomplete of LaTeX commands and reference search. Eventually, we might be able to integrate with the official ShareLaTeX server for a collaborative, hosted, editing experience for LaTeX documents from inside JupyterLab.

Maybe we as a community can come together and start building integrations for these amazing free tools into JupyterLab!

To conclude, thanks to all who’ve organized and participated in the workshop (especially Sylvain for the organization). I’ve used the opportunity to chat with the core developers and get their helpful input: Steven, Afshin, and Jason, thanks for helping me out in getting this off the ground and making JupyterLab! And honestly, the biggest shoutout has to go to the people who’ve worked on improving draw.io and thankfully open sourced this amazing code base: the entire draw.io team.

About the Author

Wolf Vollprecht is a scientific software developer at QuantStack, passionate about High-Performance Computing and Robotics. He is one of the core developers of xtensor.

Interactive Workflows for C++ with Jupyter

2017-11-29T16:33:00+00:00

Scientists, educators and engineers not only use programming languages to build software systems, but also in interactive workflows, using the tools available to explore a problem and reason about it.

Running some code, looking at a visualization, loading data, and running more code. Quick iteration is especially important during the exploratory phase of a project.

For this kind of workflow, users of the C++ programming language currently have no choice but to use a heterogeneous set of tools that don’t play well with each other, making the whole process cumbersome, and difficult to reproduce.

We currently lack a good story for interactive computing in C++.

In our opinion, this hurts the productivity of C++ developers:

Most of the progress made in software projects comes from incrementalism. Obstacles to fast iteration hinder progress.
This also makes C++ more difficult to teach. The first hours of a C++ class are rarely rewarding as the students must learn how to set up a small project before writing any code. And then, a lot more time is required before their work can result in any visual outcome.

Project Jupyter and Interactive Computing

The goal of Project Jupyter is to provide a consistent set of tools for scientific computing and data science workflows, from the exploratory phase of the analysis to the presentation and the sharing of the results. The Jupyter stack was designed to be agnostic of the programming language, and also to allow alternative implementations of any component of the layered architecture (back-ends for programming languages, custom renderers for file types associated with Jupyter). The stack consists of

a low-level specification for messaging protocols, standardized file formats,
a reference implementation of these standards,
applications built on top of these libraries: the Notebook, JupyterLab, Binder, JupyterHub
and visualization libraries integrated into the Notebook and JupyterLab.

Adoption of the Jupyter ecosystem has skyrocketed in the past years, with millions of users worldwide, over a million Jupyter notebooks shared on GitHub and large-scale deployments of Jupyter in universities, companies and high-performance computing centers.

Jupyter and C++

One of the main extension points of the Jupyter stack is the kernel, the part of the infrastructure responsible for executing the user’s code. Jupyter kernels exist for numerous programming languages.

Most Jupyter kernels are implemented in the target programming language: the reference implementation ipykernel in Python, IJulia in Julia, leading to a duplication of effort for the implementation of the protocol. A common denominator to a lot of these interpreted languages is that the interpreter generally exposes a C API, allowing the embedding into a native application. In an effort to consolidate these commonalities and save work for future kernel builders, we developed xeus.

Xeus is a C++ implementation of the Jupyter kernel protocol. It is not a kernel itself but a library that facilitates the authoring of kernels, and other applications making use of the Jupyter kernel protocol.

A typical kernel implementation using xeus would in fact make use of the target interpreter as a library.

There are a number of benefits of using xeus over implementing your kernel in the target language:

Xeus provides a complete implementation of the protocol, enabling a lot of features from the start for kernel authors, who only need to deal with the language bindings.
Xeus-based kernels can very easily provide a back-end for Jupyter interactive widgets.
Finally, xeus can be used to implement kernels for domain-specific languages such as SQL flavors. Existing approaches use a Python wrapper. With xeus, the resulting kernel won’t require Python at run-time, leading to large performance benefits.

Interpreted C++ is already a reality at CERN with the Cling C++ interpreter in the context of the ROOT data analysis environment.

As a first example for a kernel based on xeus, we have implemented xeus-cling, a pure C++ kernel.

Redirection of outputs to the Jupyter front-end, with different styling in the front-end.

Complex features of the C++ programming language such as, polymorphism, templates, lambdas, are supported by the cling interpreter, making the C++ Jupyter notebook a great prototyping and learning platform for the C++ users. See the image below for a demonstration:

Features of the C++ programming language supported by the cling interpreter

Finally, xeus-cling supports live quick-help, fetching the content on cppreference in the case of the standard library.

Live help for the C++standard library in the Jupyter notebook

We realized that we started using the C++ kernel ourselves very early in the development of the project. For quick experimentation, or reproducing bugs. No need to set up a project with a cpp file and complicated project settings for finding the dependencies… Just write some code and hit Shift+Enter.

Visual output can also be displayed using the rich display mechanism of the Jupyter protocol.

Using Jupyter’s rich display mechanism to display an image inline in the notebook

Another important feature of the Jupyter ecosystem are the Jupyter Interactive Widgets. They allow the user to build graphical interfaces and interactive data visualization inline in the Jupyter notebook. Moreover it is not just a collection of widgets, but a framework that can be built upon, to create arbitrary visual components. Popular interactive widget libraries include

bqplot (2-D plotting with d3.js)
pythreejs (3-D scene visualization with three.js)
ipyleaflet (maps visualization with leaflet.js)
ipyvolume (3-D plotting and volume rendering with three.js)
nglview (molecular visualization)

Just like the rest of the Jupyter ecosystem, Jupyter interactive widgets were designed as a language-agnostic framework. Other language back-ends can be created reusing the front-end component, which can be installed separately.

xwidgets, which is still at an early stage of development, is a native C++ implementation of the Jupyter widgets protocol. It already provides an implementation for most of the widget types available in the core Jupyter widgets package.

C++ back-end to the Jupyter interactive widgets

Just like with ipywidgets, one can build upon xwidgets and implement C++ back-ends for the Jupyter widget libraries listed earlier, effectively enabling them for the C++ programming language and other xeus-based kernels: xplot, xvolume, xthreejs…

xplot is an experimental C++ back-end for the bqplot 2-D plotting library. It enables an API following the constructs of the Grammar of Graphics in C++.

In xplot, every item in a chart is a separate object that can be modified from the back-end, dynamically.

Changing a property of a plot item, a scale, an axis or the figure canvas itself results in the communication of an update message to the front-end, which reflects the new state of the widget visually.

Changing the data of a scatter plot dynamically to update the chart

Warning: the xplot and xwidgets projects are still at an early stage of development and are changing drastically at each release.

Interactive computing environments like Jupyter are not the only missing tool in the C++ world. Two key ingredients to the success of Python as the lingua franca of data science is the existence of libraries like NumPy and Pandas at the foundation of the ecosystem.

xtensor is a C++ library meant for numerical analysis with multi-dimensional array expressions.

xtensor provides

an extensible expression system enabling lazy NumPy-style broadcasting.
an API following the idioms of the C++ standard library.
tools to manipulate array expressions and build upon xtensor.

xtensor exposes an API similar to that of NumPy covering a growing portion of the functionalities. A cheat sheet can be found in the documentation:

Scrolling the NumPy to xtensor cheat sheet

However, xtensor internals are very different from NumPy. Using modern C++ techniques (template expressions, closure semantics) xtensor is a lazily evaluated library, avoiding the creation of temporary variables and unnecessary memory allocations, even in the case complex expressions involving broadcasting and language bindings.

Still, from a user perspective, the combination of xtensor with the C++ notebook provides an experience very similar to that of NumPy in a Python notebook.

Using the xtensor array expression library in a C++ notebook

In addition to the core library, the xtensor ecosystem has a number of other components

xtensor-blas: the counterpart to the numpy.linalg module.
xtensor-fftw: bindings to the fftw library.
xtensor-io: APIs to read and write various file formats (images, audio, NumPy’s NPZ format).
xtensor-ros: bindings for ROS, the robot operating system.
xtensor-python: bindings for the Python programming language, allowing the use of NumPy arrays in-place, using the NumPy C API and the pybind11 library.
xtensor-julia: bindings for the Julia programming language, allowing the use of Julia arrays in-place, using the C API of the Julia interpreter, and the CxxWrap library.
xtensor-r: bindings for the R programming language, allowing the use of R arrays in-place.

Detailing further the features of the xtensor framework would be beyond the scope of this post.

If you are interested in trying the various notebooks presented in this post, there is no need to install anything. You can just use binder:

The Binder project, which is part of Project Jupyter, enables the deployment of containerized Jupyter notebooks, from a GitHub repository together with a manifest listing the dependencies (as conda packages).

All the notebooks in the screenshots above can be run online, by just clicking on one of the following links:

xeus-cling: the main xeus-cling example notebook,
xtensor: the C++ N-D array expression library in a C++ notebook,
xwidgets: the C++ back-end for Jupyter interactive widgets,
xplot: the C++ back-end to the bqplot 2-D plotting library for Jupyter.

JupyterHub is the multi-user infrastructure underlying open wide deployments of Jupyter like Binder but also smaller deployments for authenticated users.

The modular architecture of JupyterHub enables a great variety of scenarios on how users are authenticated, and what service is made available to them. JupyterHub deployment for several hundreds of users have been done in various universities and institutions, including the Paris-Sud University, where the C++ kernel was also installed for the students to use.

In September 2017, the 350 first-year students at Paris-Sud University who took the “Info 111: Introduction to Computer Science” class wrote their first lines of C++ in a Jupyter notebook.

The use of Jupyter notebooks in the context of teaching C++ proved especially useful for the first classes, where students can focus on the syntax of the language without distractions such as compiling and linking.

Acknowledgements

The software presented in this post was built upon the work of a large number of people including the Jupyter team and the Cling developers.

We are especially grateful to Patrick Bos (who authored xtensor-fftw), Nicolas Thiéry, Min Ragan Kelley, Thomas Kluyver, Yuvi Panda, Kyle Cranmer, Axel Naumann and Vassil Vassilev.

We thank the DIANA/HEP organization for supporting travel to CERN and encouraging the collaboration between Project Jupyter and the ROOT team.

We are also grateful to the team at Paris-Sud University who worked on the JupyterHub deployment and the class materials, notably Viviane Pons.

The development of xeus, xtensor, xwidgets and related packages at QuantStack is sponsored by Bloomberg.

About the Authors (alphabetical order)

Sylvain Corlay, Scientific Software Developer at QuantStack

Loic Gouarin, Research Engineer at Laboratoire de Mathématiques at Orsay

Johan Mabille, Scientific Software Developer at QuantStack

Wolf Vollprecht, Scientific Software Developer at QuantStack