Jupyter Blog - Maarten Breddels

Build your Jupyter dashboard using Solara

2024-12-17T10:26:00+00:00

Welcome to the first part of a series of articles showing you how to create a dashboard in Jupyter and deploy it as a standalone web app. Importantly, you won’t need to rewrite your app in a different framework for deployment. We will use a pure Python solution with no JavaScript or CSS required.

Jupyter notebooks are an incredible data analysis tool since they blend code, visualization, and narrative into a single document. However, we do not want to show the code if the insights must be presented to a non-technical audience.

Built on top of ipywidgets, the Solara framework integrates into the Jupyter Notebook, Jupyter Lab, and other Jupyter environments and is Open Source. As we will see in a later article, Solara apps can also be deployed efficiently using the Solara server. This, by itself, makes Solara a perfect solution for creating dashboards or data apps.

In this article, we will create a simple dashboard using Solara’s UI components in Jupyter. The final product will allow an end-user to filter, visualize, and explore a dataset on a map.

The end result will be a modern looking deploy app

Pre-requisites

You need to install pandas, matplotlib, folium and solara. Assuming you are using pip, you can execute on your shell:

$ pip install pandas matplotlib folium solara

Or in your notebook

%pip install pandas matplotlib folium solara

The start

We will use a subsample of the San Fransisco crime dataset which contains information on types of crimes and where they were committed.

Download the CSV file to run this locally, or let the code below sort it out.

The first thing we do when we read the data is to print it out to see what the dataset contains.

import pandas as pd
from pathlib import Path
import solara

ROOT = Path(solara.__file__).parent / "website" / "pages" / "docs" / "content" / "04-tutorial"
path = ROOT / Path("SF_crime_sample.csv.gz")
url = "https://raw.githubusercontent.com/widgetti/solara/master/solara/website/pages/docs/content/04-tutorial/SF_crime_sample.csv"

if path.exists():
    df_crime = pd.read_csv(path)
else:
    df_crime = pd.read_csv(url)

df_crime

Pandas printout of the Crime dataset Dataframe

The data looks clean, but since we will work with the Category and PdDistrict column data, let us convert those columns to title case.

df_crime["Category"] = df_crime["Category"].str.title()
df_crime["PdDistrict"] = df_crime["PdDistrict"].str.title()
df_crime

Category and PdDistrict columns nicely formatted with title casing

Using proper software engineering practices, we write a function that filters a dataframe to contain only the rows that match our chosen districts and categories.

def crime_filter(df, district_values, category_values):
    df_dist = df.loc[df["PdDistrict"].isin(district_values)]
    df_category = df_dist.loc[df_dist["Category"].isin(category_values)]
    return df_category

dff_crime = crime_filter(df_crime, ["Bayview", "Northern"], ["Vandalism", "Assault", "Robbery"])

Now, with our filtered dataset, we create two bar charts. We use regular Pandas and Matplotlib, but Seaborn or Plotly are also appropriate choices.

import matplotlib.pyplot as plt

def crime_charts(df):
    cat_unique = df["Category"].value_counts()
    cat_unique = cat_unique.reset_index()

    dist_unique = df["PdDistrict"].value_counts()
    dist_unique = dist_unique.reset_index()

    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20, 10))

    ax1.bar(cat_unique["Category"], cat_unique["count"])
    ax1.set_title("Amount of Criminal Case Based on Category")
    ax2.bar(dist_unique["PdDistrict"], dist_unique["count"])
    ax2.set_title("Amount of Criminal Case in Selected District")

    display(fig)
    plt.close(fig)

crime_charts(dff_crime)

Since we do not need bidirectional communication (e.g., we do not need to receive events or data from our map), we use Folium to display the locations of the committed crimes on a map. If we do need bidirectional communication, we can use ipyleaflet.

Since we cannot display all the data on the map without crashing your browser, we limit it to a maximum of 50 points.

import folium
import folium.plugins

def crime_map(df):
    latitude = 37.77
    longitude = -122.42

    sanfran_map = folium.Map(location=[latitude, longitude], zoom_start=12)

    incidents = folium.plugins.MarkerCluster().add_to(sanfran_map)

    # loop through the dataframe and add each data point to the mark cluster
    for (
        lat,
        lng,
        label,
    ) in zip(df.Y, df.X, df.Category):
        folium.Marker(
            location=[lat, lng],
            icon=None,
            popup=label,
        ).add_to(incidents)

    # show map
    display(sanfran_map)

crime_map(dff_crime.iloc[0:50, :])

Making our first reactive visualization

The above code works nicely, but if we want to explore different types of crimes, we need to modify and run all cells that determine our output manually. Would it not be much better to have a UI with controls determining the filtering and a view displaying the filtered data interactively?

Let’s start by importing the solara package and creating three reactive variables.

import solara

districts = solara.reactive(["Bayview", "Northern"])
categories = solara.reactive(["Vandalism", "Assault", "Robbery"])
limit = solara.reactive(100)

A reactive variable is a container around a value (like an int, string, or list) that allows the UI to listen to changes automatically. Any change to your_reactive_variable.value will be picked up by Solara components that use them so that they can automatically redraw or update themselves.

Let us now create our first component (View), which filters the data based on the reactive variables and shows the map and the charts. Solara supports the display mechanism of Jupyter so that we can use our previously defined functions.

@solara.component
def View():
    dff = crime_filter(df_crime, districts.value, categories.value)
    row_count = len(dff)
    if row_count > limit.value:
        solara.Warning(f"Only showing the first {limit.value} of {row_count:,} crimes on map")
    crime_map(dff.iloc[: limit.value])
    if row_count > 0:
        crime_charts(dff)
    else:
        solara.Warning("You filtered out all the data, no charts shown")

View()

Our reactive visualization auto-updates when any reactive variable changes

Note that some UI parts (like the warning and the charts) are conditional. Solara will automatically find out what to add, remove, or update without you having to do this manually. Solara is declarative (similar to ReactJS) but also reactive. If we change the reactive variables, Solara will see those changes and notify the component instances that use its value.

If we run the next lines of code in our notebook, our View will automatically update.

limit.value = 70
districts.value = ["Soutern", "Northern"]

We can now explore our data much faster since we don’t need to re-run the cells that depend on it.

Solara’s reactive and declarative nature makes it scalable to much larger applications than regular ipywidgets, where keeping the UI in sync and adding, removing, and updating widgets is a manual and bug-prone process.

Adding controls

We created a declarative and reactive mini app in our notebook, but we still need to manually modify the values by executing a code cell in our Notebook. Now, let us create a UI to control it. All Solara input components support reactive variables. This means that controlling a reactive variable using a UI element is often a one-liner.

solara.SelectMultiple("District",
    all_values=[str(k) for k in df_crime["PdDistrict"].unique().tolist()],
    values=districts)

In one line of code we made our app filtering interactive by adding a dropdown

Whow, that was simple! We can now easily change the filter and see the results update. Lets do this for all our reactive variables, and put them into a single component.

@solara.component
def Controls():
    solara.SelectMultiple("District", all_values=[str(k) for k in df_crime["PdDistrict"].unique().tolist()], values=districts)
    solara.SelectMultiple("Category", all_values=[str(k) for k in df_crime["Category"].unique().tolist()], values=categories)
    solara.Text("Maximum number of rows to show on map")
    solara.SliderInt("", value=limit, min=1, max=1000)

Controls()

All controls for the filtering of our dataframe and visualization

Note that the reactive variables are bi-directional, meaning that if you change it in the UI elements, it gets reflected on the Python code!

# Note that we can read AND write reactive variables
categories.value = [*categories.value, "Warrants"]

The final dashboard

We now have two parts of our UI in separate cells. This can be an amazing experience when developing in a notebook, as it flows naturally in the data exploration process while writing your notebook.

However, your end user will probably want something more coherent. The components we created are perfectly reusable, so we put them together in a single UI.

@solara.component
def Page():
    with solara.Sidebar():
        Controls()
    View()

Page()

Our end result

Finally, our dashboard is finished!

Conclusions

Using Solara, you created an interactive dashboard within a Jupyter Notebook. Your Solara components are declarative, and when using reactive variables, they are also reactive. Whether you change reactive variables via code or the UI elements, your visualizations and maps update automatically.

Your dashboard prototype now runs in your Jupyter Notebook environment, but we can only deploy it as an app. In our next article, we will focus on deploying our notebook without making any code changes. In our third tutorial, we will expand our dashboard with a few more components and create a more advanced layout.

All documentation for Solara can be found at https://solara.dev, the GitHub repo can be found at https://github.com/widgetti/solara/ and feel free to join our Discord for a chat.

And voilà!

2019-06-11T17:42:00+00:00

The goal of Project Jupyter is to improve the workflows of researchers, educators, scientists, and other practitioners of scientific computing, from the exploratory phase of their work to the communication of the results.

But interactive notebooks are not the best communication tool for all audiences. While they have proven invaluable to provide a narrative alongside the source, they are not ideal to address non-technical readers, who may be put off by the presence of code cells, or the need to run the notebook to see the results. Finally, following the order as the code often results in the most interesting content to be at the end of the document.

Another challenge with sharing notebooks is the security model. How can we offer the interactivity of a notebook making use of e.g. Jupyter widgets without allowing arbitrary code execution by the end user?

We set ourselves to solve these challenges, and we are happy to announce the first release of Voilà.

Voilà turns Jupyter notebooks into standalone web applications.

Voilà supports Jupyter interactive widgets, including the roundtrips to the kernel.
Voilà does not permit arbitrary code execution by consumers of dashboards.
Built upon Jupyter standard protocols and file formats, Voilà works with any Jupyter kernel (C++, Python, Julia), making it a language-agnostic dashboarding system.
Voilà is extensible. It includes a flexible template system to produce rich application layouts.

Installation and first-time use

Voilà can be installed from pypi:

pip install voila

or conda-forge:

conda install voila -c conda-forge

Upon installation, several components are installed, one of which is the voila command-line utility. You can try it by typing voila notebook.ipynb. It results in the browser opening to a new tornado application showing markdown cells, rich outputs, and interactive widgets.

From a notebook to a standalone web application

As you can see in the screencast, Jupyter interactive widgets remain fully functional even when they require computation by the kernel.

You can immediately try out some of the command-line options to Voilà

with --strip_sources=False, input cells will be included in the resulting web application (as read-only pygment snippets).
with --theme=dark, Voilà will make use of the dark JupyterLab theme, which will apply to code cells, widgets and all other visible components.

Making use of the dark theme and including input cells

Note that code is only shown, voilà does not allow users to edit or execute arbitrary code.

Voilà’s execution model

The execution model of Voilà is the following: upon connection to a notebook URL, Voilà launches the kernel for that notebook, and runs all the cells as it populates the notebook model with the outputs.

The execution model of Voilà

After the execution, the associated kernel is not shut down. The notebook is converted to HTML and served to the user. The rendered HTML includes JavaScript that establishes a connection to the kernel. Jupyter interactive widgets referred in cell outputs are rendered and connected to their counterpart in the kernel. The kernel is only shut down when the user closes their browser tab.

The current version of Voilà only responds to the initial GET request when all the cells have finished running, which may take a long time, but there is ongoing work on enabling progressive rendering, which should make it into a release soon.

An important aspect of this execution model is that the front-end does not determine what code is run by the backend. In fact, unless specified otherwise (with option --strip-sources=False), the source of the rendered notebook does not even make it to the front-end. The instance of the jupyter_server instantiated by Voilà actually disallows execute requests by default.

Support for custom interactive widgets

Voilà can render custom Jupyter widget libraries, including (but not limited to) bqplot, ipyleafet, ipyvolume, ipympl, ipysheet, plotly, ipywebrtc, etc.

Including bqplot figures with Voilà

Together with ipympl, Voilà is actually a simple means to render interactive matplotlib figures in a standalone web application:

Rendering interactive matplotlib figures in a web application with voilà

Voilà is language-agnostic

Voilà can be used to produce applications with any Jupyter kernel. The following screencast shows how Voilà can be used to produce a simple dashboard in C++ making use of leaflet.js maps, with the xeus-cling C++ kernel and the xleaflet package.

A standalone Voilà page making use of the C++ Jupyter kernel, xeus-cling (input cells display enabled).

We hope that Voilà will be a stimulant to other languages (R, Julia, JVM/Java) to provide stronger widgets support.

Richer layouts with Voilà templates

The main extension point to Voilà is the custom template system. The HTML served to the end-user is produced from the notebook model by applying a Jinja template, which can be defined by the user.

An example template for voilà is the voila-gridstack template, which can be installed from pypi with

pip install voila-gridstack

You can try it by typing voila notebook.ipynb --template=gridstack.

Making use of the Gridstack template to produce a dashboard with bqplot charts

The gridstack Voilà template makes use of the cell metadata to lay out the application.

A roadmap item for the gridstack Voilà template is to support the entire spec for the deprecated jupyter dashboards and to create a WYSIWYG editor for these templates in the form of a JupyterLab extension.

Note that voila-gridstack template is still at an early stage of development.

How to make custom Voilà templates?

A Voilà template is actually a folder placed in the standard directory PREFIX/share/jupyter/voila/templates and which may include

nbconvert templates (the jinja templates used to transform the notebook into HTML)
static resources
custom tornado templates such as 404.html etc.

All of these are optional. It may also contain a conf.json file to set up which template to use as a base. The directory structure for a Voilà template is the following:

PREFIX/share/jupyter/voila/templates/template_name/
|
├── conf.json                # Template configuration file
├── nbconvert_templates/     # Custom nbconvert templates
├── static/                  # Static directory
└── templates/               # Custom tornado templates

The Voilà template system can be used to completely override the behavior of the front-end. One can make use of modern JavaScript frameworks such as React or Vue.js to produce modern UI including Jupyter widgets and outputs.

Another example template for Voilà is voila-vuetify, which is built upon vue.js:

The voila-vuetify template makes use of Vue.js

The voila-gridstack and voila-vuetify templates are still at an early stage of development, but will be iterated upon quickly in the next weeks as we are exploring templates.

A Jupyter server extension

Beyond the voila command-line utility, the Voilà package also include a Jupyter server extension, so that Voilà dashboards can be served alongside the Jupyter notebook application.

When Voilà is installed, a running Jupyter server will serve the Voilà web application under BASE_URL/voila.

The Jupyter Community Workshop on Dashboarding

From June 3rd to June 6th 2019, a community workshop on dashboarding with Project Jupyter took place in Paris. Over thirty Jupyter contributors and community members gathered to discuss dashboarding technologies and hack together.

The participants to the Paris Jupyter community workshop on dashboarding

Several dashboarding solutions such as Dash and Panel were presented during the workshop and featured at the PyData Paris Meetup which was organized on the same week.

The workshop was also the occasion for several contributors to start working on Voilà. Custom templates, a dashboard gallery, logos and UX mockups for JupyterLab extensions have been developed.

We will soon publish a more detailed post on the workshop, detailing the many tracks of development that have been explored!

What is coming?

There is a lot of planned work around Voilà in the next weeks and months. Current work streams include better integration with JupyterHub for publicly sharing dashboard between users, as well as JupyterLab extensions (a Voilà “preview” extension for notebooks, and a WYSIWYG editor for dashboard layouts). There are also ongoing discussions with the OVH cloud provider (which already supports binder by handling some of its traffic) on hosting a binder-like service dedicated to Voilà dashboards. So stay tuned for more exciting developments!

Last but not least, we are especially excited about what you will be building upon Voilà!

Acknowledgments

The development of Voilà and related packages at QuantStack is sponsored by Bloomberg.

We are also grateful to the attendees of the Jupyter Community Workshop on Dashboarding for their numerous contributions to Voilà!

We would like to thank Chris Holdgraf for his work on improving documentation, and integration with JupyterHub.

We should mention Yuvi Panda and Pascal Bugnion for getting the voila-gallery project off the ground during the workshop. We are grateful to Zach Sailer for his continued work on improving jupyter_server. We should finally not forget to mention the prior art by Pascal Bugnion with the Jupyter widgets server which was also an inspiration for Voilà.

About the Authors

Sylvain Corlay is the founder and CEO of QuantStack and a core team member for Project Jupyter.

Maarten Breddels is an independent scientific software developer partnering with QuantStack on numerous projects, and a core developer of Project Jupyter.