Jupyter Blog - Loïc Gouarin

Deploying JupyterHub with Kubernetes on OpenStack

2018-10-15T13:11:00+00:00

Jupyter is now widely used for teaching and research. The use of Kubernetes for deploying a JupyterHub has enabled reliable setups scaling to thousands of users.

There are many cloud computing vendors (Google, Amazon, …) and the first attempts to use JupyterHub with Kubernetes is based on them. But relying on vendor clouds increases the risk of vendor lock-in.

In addition, there are many pre-existing academic clouds managed by people with a high level of expertise and a thorough knowledge of their infrastructure and associated tools. These are often more cost-effective for research and education. Could we build upon these academic cloud computing to provide scalable and high-quality infrastructure for education and research?

In this post, we will focus on how to deploy JupyterHub with Kubernetes on OpenStack. A first attempt to create academic cloud computing in France.

This post is split into two parts (see links below).

Why to deploy JupyterHub on OpenStack is a high-level description of our problem, and our steps to solve it. It explains why we want to deploy a JupyterHub on OpenStack, what difficulties we have encountered, and what we want to do in a near future.

A technical guide to deploying JupyterHub on OpenStack is an in-depth guide that you may follow in order to deploy your own JupyterHub on Kubernetes on OpenStack. It’s designed for any person interested in how to replicate our deployment on their own infrastructure.

Our story, our difficulties and our plans

Why deploy JupyterHub on OpenStack?

JupyterHub, the multi-user Jupyter server, has been actively developed since 2014 and has seen a rapidly growing adoption in the past year.

You may know about Zero to JupyterHub, which provides step-by-step instructions for installing JupyterHub using a vendor-managed Kubernetes cluster. In the guide, you can also find how to set up a Kubernetes cluster on many vendor clouds such as AWS, Azure, and more recently on OpenShift. But what about other cloud infrastructures based on open-source infrastructure, such as OpenStack? While cloud vendors often provide you with many tools that make your life easier, OpenStack requires more explicit configuration and setup.

Earlier this year, we set up a working group across several France universities to explore how to easily set up JupyterHub for teaching and research in our academic cloud infrastructures. It turns out that the technology used across these academic clouds is OpenStack. One of the objectives of this working group is to make just as easy to deploy JupyterHub on OpenStack as compared to following Zero to JupyterHub and using vendor infrastructure.

Note: We are not the first to work on this problem, and we should also mention the work done in Canada through Syzygy.ca which is a project of PIMS, Compute Canada, and Cybera. They have developed their own deployment tools using terraform and ansible scripts. Our approach differs in that while we use the same technological stack, we prefer not to build a custom deployment tool that we would need to maintain over time.

Issues we encountered

We started a deployment using Kubespray in January of this year and have had a bumpy path since then. To begin, we looked at what existed already in the OpenStack world. We came across Kubespray, which offers a great facility and a lot of flexibility when you want to deploy a Kubernetes cluster. An interesting fact about Kubespray is that it’s not dedicated to OpenStack infrastructures, so you should be able to follow the same procedure for other deployments such as a baremetal cluster.

Using Kubespray, we very quickly had a Kubernetes cluster on OpenStack. However, we ran into network problems, and would lose network packets that made the JupyterHub completely unusable. It took us a long time to realize that we had MTU issues and even longer to solve it. To make things harder, we used a production platform which made it very difficult to update. We finally solved the problem by using a test platform where we could have more freedom.

In Kubespray, there are various CNIs (Container Network Interface) and one of them (weave) allows to modify the MTU. We tried to configure it carefully on the production platform, but we continued facing the same problem. Trying a new version of OpenStack on the test platform, we were able to solve the problem. That means that something bad had also happened with the LoadBalancer. For more explanation, see the MTU section in the technical description below.

We thought we could deploy JupyterHub with Kubernetes on OpenStack in a few weeks but as you can see, that’s not what happened at all. That’s why it was important for us to share our experience in the hopes that it makes the process easier for others. In the last section, we’ll cover more of the technical details for our deployment

What’s next ?

For us, the installation of JupyterHub on OpenStack was just the first step of a long journey. We are able now to offer to our researchers and our students a JupyterHub but we want more. Here’s a short wish-list our deployments.

On-demand environments. Imagine offering researchers and teachers an even more flexible platform where they can create their work environment and distribute them without needing to use central IT for the installation of their packages. As you may have guessed, we are more interested in what BinderHub has to offer.

The steps described above also work for the installation of BinderHub. We deployed a BinderHub on OpenStack alongside a DockerHub registry. Kubespray also offers the possibility to deploy a private registry and we would like to test it with BinderHub.

Cluster monitoring. It would also be great to have monitoring of the Kubernetes cluster. This would allow us to inspect the usage rates and resources available on the deployment. In the roadmap of Kubespray, it is planned to add Grafana and Prometheus installations.

Authentication for BinderHub. Currently BinderHub does not support authentication for users. However, note that a recent pull request on this subject was merged in BinderHub (see https://github.com/jupyterhub/binderhub/pull/666).

Persistent storage in BinderHub. It is also currently not possible to persist storage across BinderHub sessions. Once authentication is possible in BinderHub, we’d also like to connect user accounts to their storage so that they can keep their work over time. This will also require being able to mount the home directory of each user.

We will work on all these items in the next months.

The Technical Details

This part details the set up of a Kubernetes cluster and JupyterHub using a bare OpenStack infrastructure. To make it as reproducible as possible, we will start by listing the versions we have used.

OpenStack: Pike
Kubespray: commit 3632290
Kubernetes: 1.11.3
Helm: 2.9.1
JupyterHub: 0.7.0

Now that we’ve described the components and the versions used, let’s start to deploy our JupyterHub on OpenStack !!

The deployment steps are the following

Connect to our OpenStack infrastructure
Download Kubespray
Create your infrastructure using terraform
Deploy your Kubernetes cluster using ansible
Deploy your JupyterHub using Helm chart
Enjoy!

Connect to OpenStack

Kubespray needs a access to your OpenStack infrastructure in order to create all the instances needed for your Kubernetes cluster using the OpenStack CLI (Command-Line Interface). When you log in to your OpenStack dashboard, you can download all the environment variables to use the CLI.

We chose to download the OpenStack RC File V3. You should obtain something like this:

#!/usr/bin/env bash

export OS_AUTH_URL=https://keystone.xxxxxx:5000/v3
export OS_PROJECT_ID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
export OS_PROJECT_NAME="xxxxxxxx"
export OS_USER_DOMAIN_NAME="xxxxxxxx"
if [ -z "$OS_USER_DOMAIN_NAME" ]; then unset OS_USER_DOMAIN_NAME; fi

# unset v2.0 items in case set
unset OS_TENANT_ID
unset OS_TENANT_NAME

export OS_USERNAME="xxxxxxxx"
export OS_REGION_NAME="xxxxxxxx"
# Don't leave a blank variable, unset it if it was empty
if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi

export OS_INTERFACE=public
export OS_IDENTITY_API_VERSION=3

# to be added
export OS_PASSWORD="xxxxxxxx"
export OS_CLOUD=xxxxxxxx
export OS_CACERT=/home/loic/.certs/openstack_cacert.pem

Example of rc_file

Note that we’ve removed the lines which ask for your password when you use the CLI and add it to “never ask again”. You also have to provide OS_CLOUD and OS_CACERT (even if you don’t have a certificate to access to your OpenStack infrastructure, you must provide one but you can keep it blank).

Now, you can install the OpenStack CLI with the command line. We’ll show two ways to do this below:

with virtualenv

virtualenv ~/openstack
source ~/openstack/bin/activate
pip install python-openstackclient

with conda

conda create -n openstack python=3.6
source activate openstack
pip install python-openstackclient

Next, source your rc file to activate it

source rc_file

and test your connection

openstack project list

You should be able to see your projects listed.

Once you have access, you will need some information in order to use terraform with Kubespray. You should find the following things (we have highlighted them in the images below):

The name of the image you want to deploy. To list it, run the following command:
```
openstack image list
```
List of OpenStack images
The id of the flavor describing the type of machine you want to deploy (the flavor must be aUUID and not an integer ID). To find it, run this command:
```
openstack flavor list
```

List of OpenStack flavors

Once you’ve got this information, it’s time to install Kubespray.

Install Kubespray

Because Kubespray is simply a GitHub repository, we don’t “install” it in a traditional sense, we only clone the repository to our machine. Since Kubespray is a project that evolves quickly, we’ll list the commit that we used for this post. Run the following command to get Kubespray:

git clone https://github.com/kubernetes-incubator/kubespray.git
cd kubespray
git checkout 3632290

Next, prepare all the files describing your Kubernetes cluster. We’ll follow the documentation given by Kubespray and will just change some flags. We encourage you to follow the procedure described below as the documentation seems to have some errors.

Kubespray uses terraform and ansible to deploy your Kubernetes cluster. ansible needs an inventory file which describes your cluster in order to execute the playbook roles on it. Kubespray provides a skeleton dedicated to OpenStack platform to provision your cluster using terraform and create the inventory file for ansible accordingly. To use the skeleton provided by Kubespray, the steps are the following

cp -LRp contrib/terraform/openstack/sample-inventory inventory/jhub
cd inventory/jhub
ln -s ../../contrib/terraform/openstack/hosts

jhub is the name directory we choose to store our inventory but you can choose what you want.

If you look at the inventory/jhub directory you will see

cluster.tf: the terraform file describing your inventory.
group_vars: the directory where we set all the variables used by ansible scripts provided by Kubespray.

Let’s describe our inventory.

Initialize Terraform

In the cluster.tf file, you can specify different kinds of Kubernetes clusters with floating IP for each VM. floating ip means that you ask to OpenStack to give you a public IP address in order to connect to the VM from an external network. You can also have a bastion where you have to log before reaching your Kubernetes cluster.

In the following, we only choose to have a master VM and two nodes for our Kubernetes cluster. Another important part is to specify a GlusterFS to have some storage resources for JupyterHub (database and home directories). Our inventory file cluster.tf looks like this

# your Kubernetes cluster name here
cluster_name = "jhub"

# SSH key to use for access to nodes
public_key_path = "~/.ssh/id_rsa.pub"

# image to use for bastion, masters, standalone etcd instances, and nodes
image = "CentOS-7-x86_64-GenericCloud-20180108.qcow2"
# user on the node (ex. core on Container Linux, ubuntu on Ubuntu, etc.)
ssh_user = "centos"

# 0|1 bastion nodes
number_of_bastions = 0
# standalone etcds
number_of_etcd = 0

# masters
number_of_k8s_masters = 1
number_of_k8s_masters_no_etcd = 0
number_of_k8s_masters_no_floating_ip = 0
number_of_k8s_masters_no_floating_ip_no_etcd = 0
flavor_k8s_master = "5e879c45-e709-4671-a38b-45ed0573dc38"

# nodes
number_of_k8s_nodes = 0
number_of_k8s_nodes_no_floating_ip = 2
flavor_k8s_node = "5e879c45-e709-4671-a38b-45ed0573dc38"

# GlusterFS
# either 0 or more than one
number_of_gfs_nodes_no_floating_ip = 2
gfs_volume_size_in_gb = 100
# Container Linux does not support GlusterFS
image_gfs = "CentOS-7-x86_64-GenericCloud-20180108.qcow2"
# May be different from other nodes
ssh_user_gfs = "centos"
flavor_gfs_node = "5e879c45-e709-4671-a38b-45ed0573dc38"

# networking
#network_name = "<network>"
external_net = "6cd08271-..."
#subnet_cidr = "<cidr>"
floatingip_pool = "public"

dns_nameservers = ["8.8.8.8", "8.8.4.4"]

The flavors are the same for each master, node, and GlusterFS but you can do what you want. We also added dns_nameservers to be sure that we have a correct DNS on each nodes. We will check in future experiments if it’s really necessary.

The ID of the external network and the name of the floatingip_pool can be obtained with the following command:

openstack network list

List of OpenStack networks

You need ssh keys to be able to connect to the nodes. From the documentation of Kubespray:

Ensure your local ssh-agent is running and your ssh key has been added. This step is required by the terraform provisioner:
eval $(ssh-agent -s)
ssh-add ~/.ssh/id_rsa

Now, it’s time to initialize terraform. It’s important for the next steps to be run from the root directory of Kubespray.

terraform init contrib/terraform/openstack

Now, create your VMs!

terraform apply -state=inventory/jhub/terraform.tfstate -var-file=inventory/jhub/cluster.tf contrib/terraform/openstack

At the end of this process, you can see your instances in the dashboard of OpenStack. It’s important to keep the information given at the end of the output.

At this stage, you’ve just created several VMs with the images given in the cluster.tf file. You don’t have a Kubernetes cluster up and running yet. It’s the next step!

Note: If you want to destroy all that you’ve done, run this command:

terraform destroy -state=inventory/jhub/terraform.tfstate -var-file=inventory/jhub/cluster.tf contrib/terraform/openstack

Configure your Kubernetes cluster

Again, Kubespray lets you configure your Kubernetes cluster with a lot of possibilities. We will show you one setup, but once you understand the procedure, you should be able to make your own choices.

Let’s start to see if we can ping our VMs. You have to add the following script that we called ssh-nodes.conf in your inventory/jhub directory

Host 10.0.0.*
    User centos
    UserKnownHostsFile=/dev/null
    StrictHostKeyChecking=no
    ProxyCommand ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -W %h:%p centos@134.xx.xx.xx

You also have to modify the ssh_args variable in ansible.cfg script in the root directory of Kubespray accordingly

ssh_args = -F inventory/jhub/ssh-nodes.conf -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no

Just pay attention that you give the right external address and that your internal network is 10.0.0.*.

To check that everything is configured correctly, this command:

ansible -i inventory/jhub/hosts -m ping all

should have an output like this

You can now install your Kubernetes cluster with the ansible scripts provided by Kubespray. To do that, you will edit the files found in the group_vars directory in inventory/jhub.

You need several things to have a JupyterHub up and running

A CNI (Container Network Interface). Kubespray offers different CNI for your Kubernetes cluster: cilium, calico, contiv, weave or flannel. We will choose calico.

Storage for the data. We’ll deploy a GlusterFS and add storage on the Kubernetes cluster to have access to it.

A LoadBalancer to have access to the service from the external network. You have two kinds of LoadBalancer on Openstack: Neutron or Octavia. You can use both with Kubespray. We will choose Neutron but it will be preferable to use Octavia in the future.

So how do we configure all these items?

First, open the file inventory/jhub/group_vars/all/all.yml and modify the following entries

bootstrap_os: centos

upstream_dns_servers:
    - 8.8.8.8
    - 8.8.4.4

cloud_provider: openstack

Note that the dns address is specific to our infrastructure.

Now, open the file inventory/jhub/group_vars/all/openstack.yml and configure the LoadBalancer

openstack_lbaas_enabled: True
openstack_lbaas_subnet_id: "48ec8433-..."
openstack_lbaas_floating_network_id: "6cd08271-..."

The two IDs are those given at the end of the terraform apply step.

Open the file inventory/jhub/group_vars/k8s-cluster/k8s-cluster.yml and set persistent_volumes_enabled to true and resolvconf_mode to host_resolvconf.

Our OpenStack cloud infrastructure is configured with a VXLAN tunnel where the header size is 50 bytes. We use Calico with Kubernetes which also uses a VXLAN tunnel with a header of 50 bytes. Then, for a default MTU of 1500 bytes, we already have 100 bytes taken by the headers. So, we need to configure carefully the MTU of calico in order to be sure that the packet size (headers included) doesn’t exceed the 1500 bytes.

To configure the MTU of calico, we have to edit the file inventory/jhub/group_vars/k8s-cluster/k8s-net-calico.yml and set the calico_mtu flag to 1400.

It’s important to notice that setting MTU had no effect for OpenStack versions earlier than Pike. The LoadBalancer didn’t work correctly.

The last file to modify is inventory/jhub/group_vars/k8s-cluster/addons.yml. JupyterHub uses Helm charts to deploy all that you need on the Kubernetes cluster and Kubespray can install Helm for you. So just set the helm_enabled flag to true.

Now we can run ansible playbook

ansible-playbook --become -i inventory/jhub/hosts cluster.yml

You can take a coffee break because it takes time to install all the stuff. At the end of this process, you have a Kubernetes cluster up and running.

To be sure, log in on the master nodes (the external address given by terraform) and enter the command

kubectl -n kube-system get pods

You should be able to see all pods of the kube-system namespace running.

The last step is to install the persistent volume from our GlusterFS.

ansible-playbook --become -i inventory/jhub/hosts ./contrib/network-storage/glusterfs/glusterfs.yml

If you log in again to the master of your Kubernetes cluster and enter the following command

kubectl get pv

you will see your GlusterFS storage connected to your Kubernetes cluster.

Install JupyterHub

Now that you have a Kubernetes cluster running, the procedure to install JupyterHub is exactly the same as the one described in Zero to JupyterHub. The only difference is that you don’t have to install Helm, since Kubespray did it for you. We’ll post the commands below, and you can go to the Zero to JupyterHub website for more information.

The first step is to log in to the master node of your Kubernetes cluster. Then, initialize Helm.

helm init --service-account tiller

kubectl patch deployment tiller-deploy --namespace=kube-system --type=json --patch='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["/tiller", "--listen=localhost:44134"]}]'

Next, follow the procedure described here:

Setting up JupyterHub - Zero to JupyterHub with Kubernetes 0.7.0 documentation

If the LoadBalancer did its job, you should be able to see the external IP to connect to your JupyterHub

kubectl -n jhub  get svc

If you enter this address in your web browser. If everything worked, you will see the JupyterHub login page:

JupyterHub login page

Wrapping up and feedback

The steps above described our attempts at running a JupyterHub on Kubernetes using OpenStack. There are likely many other ways to accomplish the same thing, and we’d love to hear feedback on the best procedure to install JupyterHub or BinderHub on OpenStack infrastructure. If you have encountered any issues, please ping us on the gitter channel of JupyterHub or Binder.

Thanks to the Project Jupyter team for their review and helpful comments and especially to Sylvain Corlay and Chris Holdgraf.

About the Authors (alphabetical order)

David Delavennat, Research Engineer in Scientific Infrastructures at CMLS (Polytechnique/CNRS) and INSMI (CNRS)
Loïc Gouarin, Research Engineer in Scientific Computing at CMAP (Polytechnique/CNRS)
Guillaume Philippon, Research Engineer in Scientific Infrastructures at LAL (IN2P3/CNRS)

Interactive Workflows for C++ with Jupyter

2017-11-29T16:33:00+00:00

Scientists, educators and engineers not only use programming languages to build software systems, but also in interactive workflows, using the tools available to explore a problem and reason about it.

Running some code, looking at a visualization, loading data, and running more code. Quick iteration is especially important during the exploratory phase of a project.

For this kind of workflow, users of the C++ programming language currently have no choice but to use a heterogeneous set of tools that don’t play well with each other, making the whole process cumbersome, and difficult to reproduce.

We currently lack a good story for interactive computing in C++.

In our opinion, this hurts the productivity of C++ developers:

Most of the progress made in software projects comes from incrementalism. Obstacles to fast iteration hinder progress.
This also makes C++ more difficult to teach. The first hours of a C++ class are rarely rewarding as the students must learn how to set up a small project before writing any code. And then, a lot more time is required before their work can result in any visual outcome.

Project Jupyter and Interactive Computing

The goal of Project Jupyter is to provide a consistent set of tools for scientific computing and data science workflows, from the exploratory phase of the analysis to the presentation and the sharing of the results. The Jupyter stack was designed to be agnostic of the programming language, and also to allow alternative implementations of any component of the layered architecture (back-ends for programming languages, custom renderers for file types associated with Jupyter). The stack consists of

a low-level specification for messaging protocols, standardized file formats,
a reference implementation of these standards,
applications built on top of these libraries: the Notebook, JupyterLab, Binder, JupyterHub
and visualization libraries integrated into the Notebook and JupyterLab.

Adoption of the Jupyter ecosystem has skyrocketed in the past years, with millions of users worldwide, over a million Jupyter notebooks shared on GitHub and large-scale deployments of Jupyter in universities, companies and high-performance computing centers.

Jupyter and C++

One of the main extension points of the Jupyter stack is the kernel, the part of the infrastructure responsible for executing the user’s code. Jupyter kernels exist for numerous programming languages.

Most Jupyter kernels are implemented in the target programming language: the reference implementation ipykernel in Python, IJulia in Julia, leading to a duplication of effort for the implementation of the protocol. A common denominator to a lot of these interpreted languages is that the interpreter generally exposes a C API, allowing the embedding into a native application. In an effort to consolidate these commonalities and save work for future kernel builders, we developed xeus.

Xeus is a C++ implementation of the Jupyter kernel protocol. It is not a kernel itself but a library that facilitates the authoring of kernels, and other applications making use of the Jupyter kernel protocol.

A typical kernel implementation using xeus would in fact make use of the target interpreter as a library.

There are a number of benefits of using xeus over implementing your kernel in the target language:

Xeus provides a complete implementation of the protocol, enabling a lot of features from the start for kernel authors, who only need to deal with the language bindings.
Xeus-based kernels can very easily provide a back-end for Jupyter interactive widgets.
Finally, xeus can be used to implement kernels for domain-specific languages such as SQL flavors. Existing approaches use a Python wrapper. With xeus, the resulting kernel won’t require Python at run-time, leading to large performance benefits.

Interpreted C++ is already a reality at CERN with the Cling C++ interpreter in the context of the ROOT data analysis environment.

As a first example for a kernel based on xeus, we have implemented xeus-cling, a pure C++ kernel.

Redirection of outputs to the Jupyter front-end, with different styling in the front-end.

Complex features of the C++ programming language such as, polymorphism, templates, lambdas, are supported by the cling interpreter, making the C++ Jupyter notebook a great prototyping and learning platform for the C++ users. See the image below for a demonstration:

Features of the C++ programming language supported by the cling interpreter

Finally, xeus-cling supports live quick-help, fetching the content on cppreference in the case of the standard library.

Live help for the C++standard library in the Jupyter notebook

We realized that we started using the C++ kernel ourselves very early in the development of the project. For quick experimentation, or reproducing bugs. No need to set up a project with a cpp file and complicated project settings for finding the dependencies… Just write some code and hit Shift+Enter.

Visual output can also be displayed using the rich display mechanism of the Jupyter protocol.

Using Jupyter’s rich display mechanism to display an image inline in the notebook

Another important feature of the Jupyter ecosystem are the Jupyter Interactive Widgets. They allow the user to build graphical interfaces and interactive data visualization inline in the Jupyter notebook. Moreover it is not just a collection of widgets, but a framework that can be built upon, to create arbitrary visual components. Popular interactive widget libraries include

bqplot (2-D plotting with d3.js)
pythreejs (3-D scene visualization with three.js)
ipyleaflet (maps visualization with leaflet.js)
ipyvolume (3-D plotting and volume rendering with three.js)
nglview (molecular visualization)

Just like the rest of the Jupyter ecosystem, Jupyter interactive widgets were designed as a language-agnostic framework. Other language back-ends can be created reusing the front-end component, which can be installed separately.

xwidgets, which is still at an early stage of development, is a native C++ implementation of the Jupyter widgets protocol. It already provides an implementation for most of the widget types available in the core Jupyter widgets package.

C++ back-end to the Jupyter interactive widgets

Just like with ipywidgets, one can build upon xwidgets and implement C++ back-ends for the Jupyter widget libraries listed earlier, effectively enabling them for the C++ programming language and other xeus-based kernels: xplot, xvolume, xthreejs…

xplot is an experimental C++ back-end for the bqplot 2-D plotting library. It enables an API following the constructs of the Grammar of Graphics in C++.

In xplot, every item in a chart is a separate object that can be modified from the back-end, dynamically.

Changing a property of a plot item, a scale, an axis or the figure canvas itself results in the communication of an update message to the front-end, which reflects the new state of the widget visually.

Changing the data of a scatter plot dynamically to update the chart

Warning: the xplot and xwidgets projects are still at an early stage of development and are changing drastically at each release.

Interactive computing environments like Jupyter are not the only missing tool in the C++ world. Two key ingredients to the success of Python as the lingua franca of data science is the existence of libraries like NumPy and Pandas at the foundation of the ecosystem.

xtensor is a C++ library meant for numerical analysis with multi-dimensional array expressions.

xtensor provides

an extensible expression system enabling lazy NumPy-style broadcasting.
an API following the idioms of the C++ standard library.
tools to manipulate array expressions and build upon xtensor.

xtensor exposes an API similar to that of NumPy covering a growing portion of the functionalities. A cheat sheet can be found in the documentation:

Scrolling the NumPy to xtensor cheat sheet

However, xtensor internals are very different from NumPy. Using modern C++ techniques (template expressions, closure semantics) xtensor is a lazily evaluated library, avoiding the creation of temporary variables and unnecessary memory allocations, even in the case complex expressions involving broadcasting and language bindings.

Still, from a user perspective, the combination of xtensor with the C++ notebook provides an experience very similar to that of NumPy in a Python notebook.

Using the xtensor array expression library in a C++ notebook

In addition to the core library, the xtensor ecosystem has a number of other components

xtensor-blas: the counterpart to the numpy.linalg module.
xtensor-fftw: bindings to the fftw library.
xtensor-io: APIs to read and write various file formats (images, audio, NumPy’s NPZ format).
xtensor-ros: bindings for ROS, the robot operating system.
xtensor-python: bindings for the Python programming language, allowing the use of NumPy arrays in-place, using the NumPy C API and the pybind11 library.
xtensor-julia: bindings for the Julia programming language, allowing the use of Julia arrays in-place, using the C API of the Julia interpreter, and the CxxWrap library.
xtensor-r: bindings for the R programming language, allowing the use of R arrays in-place.

Detailing further the features of the xtensor framework would be beyond the scope of this post.

If you are interested in trying the various notebooks presented in this post, there is no need to install anything. You can just use binder:

The Binder project, which is part of Project Jupyter, enables the deployment of containerized Jupyter notebooks, from a GitHub repository together with a manifest listing the dependencies (as conda packages).

All the notebooks in the screenshots above can be run online, by just clicking on one of the following links:

xeus-cling: the main xeus-cling example notebook,
xtensor: the C++ N-D array expression library in a C++ notebook,
xwidgets: the C++ back-end for Jupyter interactive widgets,
xplot: the C++ back-end to the bqplot 2-D plotting library for Jupyter.

JupyterHub is the multi-user infrastructure underlying open wide deployments of Jupyter like Binder but also smaller deployments for authenticated users.

The modular architecture of JupyterHub enables a great variety of scenarios on how users are authenticated, and what service is made available to them. JupyterHub deployment for several hundreds of users have been done in various universities and institutions, including the Paris-Sud University, where the C++ kernel was also installed for the students to use.

In September 2017, the 350 first-year students at Paris-Sud University who took the “Info 111: Introduction to Computer Science” class wrote their first lines of C++ in a Jupyter notebook.

The use of Jupyter notebooks in the context of teaching C++ proved especially useful for the first classes, where students can focus on the syntax of the language without distractions such as compiling and linking.

Acknowledgements

The software presented in this post was built upon the work of a large number of people including the Jupyter team and the Cling developers.

We are especially grateful to Patrick Bos (who authored xtensor-fftw), Nicolas Thiéry, Min Ragan Kelley, Thomas Kluyver, Yuvi Panda, Kyle Cranmer, Axel Naumann and Vassil Vassilev.

We thank the DIANA/HEP organization for supporting travel to CERN and encouraging the collaboration between Project Jupyter and the ROOT team.

We are also grateful to the team at Paris-Sud University who worked on the JupyterHub deployment and the class materials, notably Viviane Pons.

The development of xeus, xtensor, xwidgets and related packages at QuantStack is sponsored by Bloomberg.

About the Authors (alphabetical order)

Sylvain Corlay, Scientific Software Developer at QuantStack

Loic Gouarin, Research Engineer at Laboratoire de Mathématiques at Orsay

Johan Mabille, Scientific Software Developer at QuantStack

Wolf Vollprecht, Scientific Software Developer at QuantStack