Jupyter Blog - Min RK

JupyterHub 6.0

2026-09-11T08:54:00+00:00

JupyterHub is a collection of tools that provides standardized, user-friendly computing environments for education and research. Today, we’re pleased to announce the release of JupyterHub 6.0. This is a substantial release with several new features. The breaking changes ought to be minimal, and should affect very few deployments. There is a small database schema upgrade, so make sure to backup your database prior to upgrading, as usual.

Some changes to highlight:

Stricter named server rules: Named servers now have stricter rules on their names, and a new less-restricted “display name” field. This might be the most significant change affecting you! See the documentation for more details.
Python 3.10 and database upgrade: JupyterHub 6.0 now requires Python 3.10 and includes a database schema upgrade. Make sure to back up your database before upgrading.
Updates to metrics: The old StatsD metrics have been removed, in favor of the Prometheus metrics that most people are already using. For Spawner (and spawn hook) authors, JupyterHub 6.0 also introduces SpawnException, which provides more granular information for classifying spawn failures in metrics, to help deployments separate errors from rejections.
Internal communication changes: JupyterHub now uses aiohttp for internal HTTP requests, which may require tuning for large deployments since internal HTTP requests. This update has facilitated using unix sockets for internal communication on single-node deployments.
Improved granularity in API access: JupyterHub provides new API endpoints for working with single servers, along with more flexible ways to define permissions, such as start:servers scope and extra_user_scopes.
See the documentation for more information.

Beyond these main changes, there have been lots of other improvements and bugfixes. You can find more details in the upgrading documentation and changelog.

Thank you to our contributors

The following people contributed discussions, new ideas, code and documentation contributions, and review: agoose77, anny0811, ateucher, betatim, bl-aire, clhedrick, consideRatio, dometto, dragz, erawn, ericvd-ucb, jezwilkinson, jnywong, jrdnbradford, kellyrowland, kreuzert, LUC4R4T0R, manics, mayli, milliams, minrk, mynkyu, nsychev, Paul2708, Rameshkumar31595, rgaiacs, shaneknapp, sixonenines, tbizouerne, Tech-Abhang, wilkinvr, willingc, yuvipanda

mybinder.org adding request Origin to events archive

2026-02-02T16:27:00+00:00

Binder logo

In February, the mybinder.org service will start tracking the “origin” of requests that launch a Binder session. The website that created a request to mybinder.org will be visible in the Binder analytics archive. This will help us monitor which websites are embedding Binder sessions for impact or abuse reports. It shouldn’t have any implications for private information but we are sharing here for transparency. The only cases where new information will be recorded is websites which embed mybinder.org kernels via tools like Thebe.

When folks use mybinder.org to run sessions, each launch is added to a public events archive. That means that every time anyone launches a session on mybinder.org, the repo and version used and the time of the launch is public information. There is no such thing as a private launch on mybinder.org (this is part of why we don’t support private repos). No information about the user requesting the build is recorded (ip address, user agent, browser info, etc.), though some of that is sent to Plausible and aggregated and anonymized for analytics. Notably, Plausible only covers regular browser visits, not launches via the API.

There are 3 main ways that people launch sessions on mybinder.org:

(the vast majority) — visiting mybinder.org in a browser
embedding kernels from mybinder.org on another website to make e.g. interactive documentation with a tool like Thebe
‘headless’ launches via scripts

Up to now, these events are indistinguishable in the events archive. We’d like to start distinguishing these cases in the events archive to better classify how mybinder.org is used, but that means recording new information we didn’t track before, so we want to communicate clearly about it ahead of time.

We are planning to add the Origin of the launch request to the public events archive at the end of February. This still doesn’t track anything about the user (no ip address, user agent, etc.), but rather the host of the page where the launch request originated. For the vast majority of cases, this is not new information, as the origin is e.g. gesis.mybinder.org (which we already track confusingly as ‘origin’, though this is not the request Origin, it is the federation member where the launch event takes place). Notably, case 1 still includes any regular browser links to mybinder.org/v1/gh/repo/name/HEAD. We don’t record where those come from (that would be Referer).

Where this changes is cross-origin requests (case 2), that is, pages which embed kernels from mybinder.org, such as documentation websites that use mybinder.org to embed kernels from mybinder.org via tools like Thebe to make executable documentation (which is rad!). These requests will now include the Origin of the request, which would be the hostname (e.g. https://course.spacy.io ), not the full page.

Script launches (case 3) typically lack this information, so will usually have null data in the new field. But of course, scripts can do whatever they want, so this isn’t guaranteed to be accurate for malicious or deliberately spoofing scripts.

If you want to have a private, not-recorded use of Binder, you must run your own BinderHub instance. If you want to suggest that we not do this, or do it differently, please let us know in the proposal discussion. You can also check the privacy policy for more information. We plan to apply the change at the end of February, 2026.

Mybinder.org reducing capacity

2023-04-25T07:28:00+00:00

Funding is running out for the largest member of mybinder.org, which has been sponsored by the Google Open Source Program Office. gke.mybinder.org will shut down and there will be a reduction in capacity at the end of April, 2023 (end of this week).

Key points:

Our funds on gke.mybinder.org, the biggest part of the federation, will run out in April. Google has decided not to provide more credits.
Traffic previously sent to GKE will mostly be delivered to OVHcloud, which has less capacity. We are also working to deploy new federation members to spread the load.

Service reductions to expect:

This may come with an increase in errors and failures, as GKE has been the largest and most stable federation member
We are currently looking at reducing the total capacity of Binder by 60%, so you may get rate limit or capacity errors
We will start more aggressively deleting older cached builds, as storage costs have increased.

We are still working out the details of the shutdown and deploying new federation members, but mybinder.org users should expect disruption to the mybinder.org service and to see an increase in messages that the service is at capacity in the short term.

The plan:

Thursday: gke.mybinder.org will be removed from the federation (user sessions not disrupted, but no new sessions will start there)
Friday: gke.mybinder.org will be shutdown, and mybinder.org migrated to a smaller cluster

You can help us keep mybinder.org running by donating to Binder or encouraging your organization to join the BinderHub Federation.

We are grateful to the Google Open Source Program for their funding thus far, and appreciate the continued support of continuing federation members, OVHCloud and GESIS, which enable us to keep mybinder.org going without shutting down completely when one funding source goes away.

Updating defaults on mybinder.org

2023-03-09T08:31:00+00:00

This week, we are deploying some updates to repo2docker, the image-building tool that underpins mybinder.org. This may affect your repos. Please get in touch if you have problems! Hopefully this will make things better, more stable, and less surprising overall, but change always breaks something, however hard we work to keep things working.

To highlight the changes:

Python 3.10 is now the default Python version
Python 3.11 is now supported, and new versions of Python should be supported more quickly
If a repo requests an old version of Python (≤ 3.6), it will be installed in a separate environment, as we’ve been doing with Python 2.7 for a long time.
The server implementation is switched from jupyter-notebook to jupyter-server

More detail on each change, the reasons behind them, and their consequences below.

Python 3.11 and separate envs for legacy Python support

When building an image, repo2docker installs packages in one or two environments — the kernel environment where your code runs, and the server environment, which runs the Jupyter server. If you’re using a reasonably supported version of Python, these are actually the same environment. This is the vast majority of repositories on mybinder.org. In order to function properly, we have certain requirements of what goes in the ‘server’ environment. Eventually, those requirements start making it difficult to setup the server environment with an outdated version of Python, as packages march ahead with their minimum required Python (The widely adopted NEP 29 suggested dropping Python 3.7 support in December, 2021). Our solution for this has been to separate the kernel environment from the server environment, so they don’t create a conflict. Until recently, only Python 2 was handled this way (most packages dropped Python 2 support years ago). As part of adding support for Python 3.11, we have expanded this behavior to include Python ≤3.6, so if you are using a ‘legacy Python’, currently defined to be Python ≤3.6, your requested packages will be installed in the kernel environment instead of the server environment. This should only have a noticeable effect if you are installing something with server extension components , which includes anything modifying the JupyterLab or notebook interfaces— those will not be available in the server environment. To address this, you will have to upgrade Python or handle the server extension installation in postBuild. PR adding Python 3.11 and adding Python ≤3.6 to legacy Python versions.

Default Python is now 3.10

The default Python for repo2docker and thereby mybinder.org is now Python 3.10, upgraded from 3.7. 3.10 will be used when a repository does not specify which version of Python to use. Following Python’s own annual release cadence, we plan to update this more regularly in the future, lagging behind Python itself by a version or two. This has no effect on your repos if they specify a Python version, which is why we encourage you to specify your Python version! If you don’t, then your Python version will change over time, probably about once a year. PR switching the default Python.

Now launching Jupyter Server

The default UI for mybinder.org has been JupyterLab for some time, with the classic notebook still available if you specify a url of /tree/ or /notebooks/mynb.ipynb. When we rolled out that change, we kept the default server as the jupyter-notebook server, for improved compatibility with existing server extensions. As the newer Jupyter Server has taken over, the situation is becoming reversed — using the older notebook server is now the cause of compatibility issues with server extensions, so we are switching the server to launch jupyter-server instead of jupyter-notebook. The classic UI is still available at the same URLs, thanks to nbclassic. This should have no visible effects, as jupyter-server and nbclassic make a great deal of effort to keep classic notebook extensions working, but some extensions may need an update for jupyter-server compatibility if they haven’t been updated already. Because of where the server is specified, there is no supported way for images to express a preference for the deprecated notebook server (PR switching the default server command.

Please don’t disable authentication in Jupyter servers

2022-04-05T09:50:00+00:00

We are aware of ransomware attacks specifically targeting Jupyter servers.

First off: we have no evidence of a vulnerability in Jupyter associated with the attack. The attacks appear to focus on Jupyter servers with authentication disabled. That’s what this post is about: please don’t do that!

Jupyter servers are authenticated with a random token by default, so you should be fine with the defaults. That default random token is regenerated on each server launch. If that random token is inconvenient for you (there are plenty of situations where this is the case), the answer is not to disable auth! More on that below.

We wrote about this before, but it bears repeating:

Jupyter exposes arbitrary code execution and a shell. If folks gain access, they can do anything you can do. You aren’t just protecting your notebooks, you are protecting your whole computer.
People can find your notebook server if it’s accessible on the Internet. If they locate your server and it has auth disabled, that’s game over, and appears to be what happened in the above ransomware event. Jupyter’s popular enough (yay) that folks are always looking (boo).
If the generated token pattern is inconvenient for how you work, you can set a persistent token or password in configuration instead of retrieving the generated token on each server launch. Save this token or password in your password manager for safe keeping, and accessing your server will be just as convenient as auth being disabled.

If you have an interactive prompt, you can run jupyter server password

$ jupyter server password
Enter password:🔐
Verify password:🔐
[JupyterPasswordApp] Wrote hashed password to ~/.jupyter/jupyter_server_config.json

which generates and stores config that looks like:

{
  "ServerApp": {
    "password": "argon2:$argon2id$v=19$m=10240,t=10,p=8$aFE/DuLj//6oGF2PHWy2DQ$eXHn6AbJe8Lryl4z9oGvMtZhX7iEdt41m+mhvyDvw88"
  }
}

This is a hashed, salted form of your password. We can’t get the password out of this, but we can use it to check if you typed it correctly on the login page. If an interactive prompt isn’t available to you (e.g. a cloud vm that starts the notebook server automatically), you can run this command on any machine, and copy the resulting config file to the destination machine as part of setup.

If it works better for you, you can also generate just the hashed password:

from jupyter_server.auth import passwd
hashed_password = passwd("my-great-passphrase") # just the "argon2:..."

and get that into your config files, as best suits your needs. For example, in ~/.jupyter/jupyter_server_config.py:

c.ServerApp.password = "argon2:..."

or the same config in ~/.jupyter/jupyter_server_config.json

{
  "ServerApp": {
    "password": "argon2:..."
  }
}

Read more details in our earlier post, and security documentation.

Please contact security@ipython.org if you have a vulnerability to report.

JupyterHub 2.0

2021-12-01T14:15:00+00:00

We’ve just published JupyterHub 2.0, our biggest release in a while.

Roles and scopes (RBAC)

Thanks to the work of Omar Richardson and Ivana Huskova and more contributors along the way, JupyterHub now has fine-grained control of permissions, via scopes (a role is a named collection of scopes).

This makes JupyterHub more secure because you no longer need to give every user or service full admin permissions if they need to take one elevated action (such as a cull-idle service). Now, you can give users and services exactly the permissions they actually need. admin still exists, but nothing should need to be an admin anymore. Fine-grained scopes also allow new configurations that weren’t previously available, such as granting instructors permission to access their students’ servers, which is useful with JupyterLab’s new real-time collaboration features, without needing to grant them full admin permissions, or even access to servers of users not in their specific class.

The significant change in how permissions work means that upgrading to 2.0 requires some downtime. You will need to shut down all services and single-user servers across the upgrade, and make sure you upgrade JupyterHub to 2.0 in both the user environment and the Hub environment at the same time.

All tokens issued prior to the 2.0 upgrade will be revoked.

As always, back up your database before upgrading!

Read more about roles and scopes in the docs.

JupyterLab by default

2.0 also marks JupyterHub’s transition to using JupyterLab as the default UI. You can stick with the classic notebook server with:

c.Spawner.environment = {
    "JUPYTERHUB_SINGLEUSER_APP": "notebook.notebookapp.NotebookApp",
}

or try RetroLab, a classic-style UI built on the newer Jupyter Server:

c.Spawner.default_url = "/retro/"

(assuming you’ve installed it in the user environment, with e.g. pip install retrolab).

And more

There are various other changes and improvements in 2.0, including a new admin UI, API pagination, and no longer relying on any command-line arguments for single-user servers. Check out the changelog for more details.

We encourage you to take 2.0 for a spin, try out the roles and scopes, and let us know how it goes.

Thanks to all the contributors who helped us with 2.0, especially prerelease testers!

IPython Parallel in 2021

2021-11-30T07:55:00+00:00

Updates on IPython Parallel; new features, future direction

This post describes work funded by Bodo, Inc.

IPython Parallel’s a bit of an odd duck in the parallel computing space. In a world with dask, ray, bodo, pyspark, and other parallel computing tools, what is IPython Parallel’s role in 2021?

Context

IPython Parallel began (in 2006!) as a natural extension of the Jupyter messaging protocol(though it predates the Jupyter name by a few years): when you have a protocol for REPL-style remote code execution, what can you do with multiple remote execution environments? This question led to the development two basic models for parallel execution in IPython Parallel:

“multiplexed execution” — where you send code explicitly to different workers, and
“load-balanced execution” — where you tell IPython Parallel what to run, and a scheduler takes care of assigning each task to an available worker.

What IPython Parallel doesn’t do

To IPython, your tasks are black boxes (either Python functions or blocks of Python code as text) and you are in complete control of where and when your tasks run, as well as any dependencies or side effects they may have.

IPython Parallel specifically doesn’t and won’t do several things that other tools might:

infer dependencies or relationships between tasks (execution graph)
manage data distribution or locality
construct parallel algorithms or represent ‘natively’ parallel distributed data structures, such as distributed arrays or data frames

Everything is very explicit and implemented at the client-level in IPython Parallel, meaning that while it doesn’t do these things, you can. It also means that many of the things that have overhead costs associated with parallel computing (communication, data movement) have particularly high costs in IPython Parallel.

IPython Parallel doesn’t hide anything from you, for better and worse.

What IPython Parallel does

IPython Parallel makes explicit parallel computations interactive.

IPython Parallel is not just parallel Python, it’s parallel IPython. That means you have all the power of IPython’s inspection, interactivity, magics, and debugging available on all of your distributed workers. This makes it especially well-suited to prototyping and experimentation.

IPython Parallel also presents standard APIs such as Python Executors, compatible with many other implementations, to make it easy to migrate to and from IPython Parallel, enabling developers to write code that uses a single multicore laptop or a thousand cores on an HPC cluster or cloud.

IPython Parallel in 2021

Interactive progress across parallel engines. Those progress bars are interactive widgets running locally, and on each remote engine!

For a lot of today’s workloads, my default recommendation is: use dask or bodo or another modern tool. IPython even makes this easier if you already happen to have an IPython Parallel cluster, you can tell it to “become dask,” and off you go:

dask_client = rc.become_dask(ncores=1)

However, where IPython Parallel can shine, is making traditional SPMD (e.g., MPI) workloads interactive, especially for prototyping and debugging.

If you have an MPI simulation and you wish you could pause it in the middle, poke around and make plots and interact with just one node or all of them with all the interactive tools available to you in Jupyter, IPython Parallel may be the tool for you.

Recent developments

Because I see the main problem IPython Parallel solves well is adding interactivity to direct parallel execution, the focus of recent developments has been on improving that story, based on feedback from users who are often in traditional HPC environments like SLURM or PBS, or using MPI in the cloud.

Much of the feedback on challenges over the years have been around:

Scaling to larger numbers of engines (1,–10,000)
Better feedback and recovery when things go wrong
Security requirements

Thanks to Bodo, I’ve been able to spend a lot of time this year addressing several of those issues, now available as IPython Parallel 8.0.

Last year, Tom-Olav Bøyum developed a broadcast scheduler as part of his Master’s thesis, which vastly improves the efficiency of sending the same task to all engines (the main thing we do in IPP+MPI). This work is now mostly done and landed in IPython Parallel 7. We’ve also addressed longstanding issues with registering large numbers of engines (5–10,000) at once, a common source of failure on HPC clusters.

One of the most frustrating / challenging things when working with MPI or other parallel computations is dealing with hangs and restarting tasks. Through the new Cluster API, IPython Parallel now has one of its most requested features: the ability to send signals to one or all engines, and forcefully restart the cluster if it’s stuck.

We’ve also improved interactive feedback with progress bars, live streaming output, a new JupyterLab plugin based on dask-labextension, and support for the Jupyter widget protocol, so you can instantiate widgets on engines and interact with them directly from a notebook.

Security questions have also been raised, because IPython (and Jupyter’s) use of ZeroMQ assumes a ‘trusted network’, typically localhost or BSD sockets for notebooks. Since IPython Parallel usually runs across a network, security can be more of a concern (whereas notebooks typically only use standard HTTPS connections over a network), and the level of security in the Jupyter protocol alone may not be sufficient. In the past, we’ve resorted to tunneling TCP over SSH for more secure network traffic, but still always implicitly trusting localhost.

IPython Parallel 7.1 enables CurveZMQ for full authentication, encryption, and forward-secrecy at the transport level, solving a longstanding security shortcoming of IPython Parallel.

IPython Parallel in the future

Scaling is the biggest challenge for IPython Parallel due to its communication model. However, that scaling is on the order of IPython engines, which is not the same as the number of cores. You can benefit from this today if your tasks are already multi-threaded, e.g., through OpenMP threads in numpy, or your own Python threads or multiprocess breakdown of tasks.

Better support of multi-level parallelism, where each “engine” may represent a multicore node, as individual nodes get bigger and bigger would mean moving the bar where IPython Parallel communication is the bottleneck back two orders of magnitude on large machines, because one ‘engine’ could represent 128 cores or more. Good support for 1000 128-core nodes would give IPython Parallel some pretty comfortable headroom for our target use cases, scale-wise.

Also related to scaling, bringing the BroadcastView to maturity should allow us to completely replace the DirectView scheduler, as it should be able to match or beat it in almost every scenario with some further development. There are some features lacking, especially when it comes to error handling, but those can certainly be addressed in time.

Finally, I’ll invite you to get involved. Especially if you are interested in prototyping parallel code, or making traditional MPI-style code interactive, check out IPython Parallel 8 and let us know how it goes, or contribute your use case as an example.

pip install --upgrade ipyparallel

nbviewer has a new host: OVHcloud

2020-03-06T19:46:00+00:00

nbviewer has moved from Rackspace to OVHcloud. That means moving from Docker to Kubernetes+Helm.

For several years, nbviewer has been generously hosted by Rackspace. That sponsorship program appears to be ending, so nbviewer needed a new home; it has found one in OVHcloud. We are extremely grateful to OVHcloud for their support in keeping nbviewer running, building on their existing participation in the Binder Federation (literally—nbviewer is now running on the same kubernetes cluster as ovh.mybinder.org).

How we moved

Where were we before?

nbviewer was previously deployed using a private repo (because it contained credentials) and various commands using invoke. It was a mixture of custom steps, using the openstack Python API and docker machine to:

allocate two VMs
build docker images
deploy two nbviewer instances per node
deploy memcached via nbcache on each node
deploy statuspage publisher as a separate step
update fastly to point to the running nbviewer instances

The upside was that we had a repo that could immediately deploy nbviewer to anywhere with docker. With this repo, we moved our deployment strategy from a CoreOS cluster to Rackspace’s short-lived Carina service to deploying VMs ourselves with docker-machine. Migrating to a new source of VMs would not have been hard, but it wouldn’t have solved any of our challenges.

Known downsides of this deployment that we’ve experienced over the years:

only a few folks ever knew how to use it and could thus deploy updates to nbviewer
it was private, contributing to above. It’s hard to onboard folks in an open community to a private repo!
independent machines meant cache was not shared (minor, but contributes to our consumption of the GitHub API rate limit)
no automatic recovery based on health monitoring, so when a container had issues, some humans got automated emails but no action was automatically taken. Extra frustrating because the fix was ~always to restart the container.

The credentials used by this repo have been revoked and an archived version of the repo is available (with credentials redacted from history) in the new, public nbviewer.org-deploy repo.

What have we learned?

We’ve learned a lot about open, automatic, and sustainable deployments, and can now comfortably address all of the downsides above. Most of this has been learned from the communities participating in the JupyterHub and Binder projects, as seen in the mybinder.org-deploy repo. mybinder.org-deploy is a public repo that automatically deploys and tests updates to at least four different Kubernetes clusters at the push of a button (the Big Green Merge Button, to be precise). Some things we have learned in the years since we set up our nbviewer deployment:

kubernetes and helm are great :)
git-crypt allows us to have public repos with some secret contents, so deployment repos don’t need to be fully private just to protect a couple api keys.
Continuous Deployment via services like Travis or Circle lowers the bar to adding maintainers on a given deployment since all that’s needed is to press the Big Green Button.

The OVH sponsorship came in the form of a slice of a kubernetes cluster; that meant we had to migrate the nbviewer deployment tools from using docker-machine to kubernetes, which for us means helm.

Step 1: helm chart for nbviewer

The first step was to create a helm chart for nbviewer, which is done here. Before, nbviewer was two docker containers running nbviewer and one running memcached per server. To turn this into a helm chart we need:

dependency on memcached helm chart for easy deployment of the cache. All the nbviewer instances will talk to this memcache, which should improve our GitHub rate limit consumption, since the cache will be shared. This means the deprecation of our own nbcache repo.
Deployment for nbviewer. This is (one of) the kubernetes wrappers around containers. It has a nice ‘replicas’ field to easily scale nbviewer up and down. We are currently running with 3 replicas.
Service to expose nbviewer to the Internet (likely also need an ingress in the future)
Deployment for the statuspage publisher, which updates https://status.jupyter.org with the remaining github rate limit available

We also needed helm charts and docker images for some other components, such as cdn.jupyter.org and the nbviewer statuspage data source.

Step 1b: helm chart for cdn.jupyter.org

nbviewer and some other services interact with cdn.jupyter.org, which is a lightweight nginx configuration to serve the classic notebook’s javascript and css as static files(this is no longer needed for npm-based jupyterlab, which can use unpkg as a CDN). This used to run on a multipurpose server VM, but that is also being retired for the same reason. The result was creating a docker image and helm chart for serving the contents of cdn.jupyter.org. The scripts used to run the existing CDN were already in a repo, so it was a small amount of work to adapt this to run in a container instead of on a server. We may retire cdn.jupyter.org in the future, so please don’t rely on it :)

This is where we are right now — nbviewer.jupyter.org and cdn.jupyter.org are being served by OVH and the Rackspace machines are being retired.

Step 2: automatic deployment

It would have been ideal for this to be step 1, but that’s now how it happened. Sometimes you need to get it done quickly before you get it done right. The task here will be to make a new, public nbviewer-deploy repo following the patterns we have learned in mybinder.org-deploy. That will mean:

create public nbviewer.org-deploy repo with configuration and git-crypt encrypted secrets for deploying nbviewer on the ovh cluster (done)
adopting chartpress to publish our helm charts for nbviewer and version-tagged images from the nbviewer repo
configure Travis-CI or other CI service to automatically deploy updates with helm, so merging a PR is all we need to do to deploy updates to nbviewer

Thanks to OVHCloud for their continued support of Jupyter and Binder, and the JupyterHub and Binder communities for teaching me how to operate services in the open with kubernetes and helm.

Announcing JupyterHub 1.0

2019-05-04T03:58:00+00:00

Today, we are pleased to announce the 1.0 release of JupyterHub. We’ve come a long way since our first release in March, 2015. There are loads of new features and improvements covered in the changelog, but we’ll cover a few of the highlights here.

You can upgrade jupyterhub with conda or pip:

(Before upgrading, always make sure to backup your database!)

conda install -c conda-forge jupyterhub==1.0.*
pip install --upgrade jupyterhub==1.0.*

UI for named servers

Named servers are a concept in JupyterHub that allows each user to have more than one server (e.g. a ‘gpu’ server with access to gpus, or a ‘cs284’ server with the necessary resources for a given class). JupyterHub 1.0 introduces UI for managing these servers, so users can create/start/stop/delete their servers from the hub home page:

UI for managing named servers

Improved pending spawn handling and less implicit spawn behavior

When users launch their server, they will be faced with a progress bar showing the progress. Spawners can emit custom messages to indicate the stages of launch, which are especially useful when it can take a while.

a spawn taking a bit

Support for total internal encryption with SSL

If your JupyterHub is publicly accessible, we hope you are using HTTPS (it’s never been easier, thanks to letsencrypt)! However, JupyterHub exclusively used HTTP for internal communication. This is usually fine, especially for single-machines, but for deployments on distributed or shared infrastructure it’s a good idea to encrypt communication between your components. With 1.0, you can enable SSL encryption and authentication of all internal communication as well. Spawners must support this feature by defining Spawner.move_certs. Currently, local spawners and DockerSpawner support internal ssl.

Support for checking and refreshing authentication

JupyterHub authentication is most often managed by an external authority, e.g. GitHub OAuth. Auth state can be used to persist credentials, such as enabling push access to repositories or access to resources. These credentials can sometimes expire or need refreshing. Until now, expiring or refreshing authentication was not well supported by JupyterHub. 1.0 introduces new configuration to refresh or expire authentication info:

c.Authenticator.auth_refresh_age allows authentication to expire after a number of seconds.

c.Authenticator.refresh_pre_spawn forces a refresh of authentication prior to spawning a server, effectively requiring a user to have up-to-date authentication when they start their server, which can be important when the user environment should have credentials with access to external resources.

Authenticator.refresh_auth defines what it means to refresh authentication (default is nothing, but can check with external authorities to re-load info such as tokens or group membership) and can be customized by Authenticator implementations.

Thanks!

Huge thanks to the many people who have contributed to this release, whether it was through discussion, testing, documentation, or development.

Open Redirect Vulnerability in Jupyter notebook, JupyterHub

2019-03-28T14:03:00+00:00

Update: notebook 5.7.7 and JupyterHub 0.9.5 contained incomplete fixes for this issue. 5.7.8 and 0.9.6 are released with more complete fixes.

JupyterHub ≤ 0.9.5 and Notebook ≤ 5.7.7 are affected. Versions 0.9.6 and 5.7.8 are released with fixes. Upgrade with pip or conda:

pip install --upgrade 'notebook>=5.7.8'
pip install --upgrade 'jupyterhub>=0.9.6'

conda install -c conda-forge 'notebook>=5.7.8'
conda install -c conda-forge 'jupyterhub>=0.9.6'

notebook 5.7.8 also fixes some compatibility issues and regressions introduced in 5.7.5 and 5.7.6, especially on some Windows systems. If you use the jupyterhub helm chart, version 0.8.2 upgrades JupyterHub to 0.9.6.

What is an Open Redirect Vulnerability?

Login pages tend to take a parameter for redirecting back to a page after successful login, e.g. /login?next=/notebooks/mynotebook.ipynb, so that you aren’t disrupted too much if you try to visit a page, but have to authenticate first. An Open Redirect Vulnerability is when a malicious person crafts a link pointing to the login page of a trusted site, but setting the “redirect after successful login” parameter to send the user to their own site, instead of a page on the authenticated site (the notebook or JupyterHub server), e.g. /login?next=http://badwebsite.biz. This doesn’t necessarily compromise anything immediately, but it enables phishing if users don’t notice that the domain has changed, e.g. by showing a fake “re-enter your password” page. Servers generally have to validate the redirect URL to avoid this. Both JupyterHub and Notebook already do this, but the validation didn’t take into account all possible ways to redirect to other sites, so some malicious URLs could still be crafted to redirect away from the server (the above example does not work in any recent version of either package). Only certain browsers (Chrome and Firefox, not Safari) could be redirected from the JupyterHub login page, but all browsers could be redirected away from a standalone notebook server.

This vulnerability has been assigned CVE-2019-10255. This post will be updated when it has been assigned. Thanks to Ronak Odhaviya for reporting the issue. You can report security issues by emailing ipython-security@googlegroups.com.

Jupyter notebook XSSI security fix

2019-03-09T20:46:00+00:00

We have just released Jupyter notebook 5.7.6 with a security fix for a cross-site inclusion (XSSI) vulnerability, where content from a Jupyter server could be included in another page if the visitor is logged in to the Jupyter server and the author of the page knows the URL of the server and the path within the server’s notebook directory that they would like to include. Further, it has been demonstrated with the Internet Explorer browser that some content from the accessed file can be retrieved by the attacking page. This has not yet been demonstrated with other browsers, however.

To upgrade:

pip install --upgrade 'notebook>=5.7.6'

or conda:

conda install 'notebook>=5.7.6'

This vulnerability was reported by the HackerOne hacker Abhishek Bundela, via Jonathan Kamens at Quantopian. The patch was developed by Min Ragan-Kelley with help from Devdatta Akhawe. Jupyter is grateful to Quantopian’s bug bounty program, which has found and reported this and other security flaws in Jupyter.

This vulnerability has been assigned CVE-2019-9644.

Zero to JupyterHub helm chart 0.8

2019-02-21T18:34:00+00:00

We’ve just released version 0.8 of the jupyterhub helm chart.

For those who may not know, Zero to JupyterHub is a guide and helm chart for deploying JupyterHub on Kubernetes. Helm is like a package manager for Kubernetes, which aims to make it easy to install and manage applications on your cluster.

There are loads of bugfixes and improvements to this release, and we encourage you to check out the change log. Here, we’ll discuss two highlights from the new features: profiles and autoscaling.

Profiles

By adopting the profiles pattern established in WrapSpawner, Kubernetes users can now pick from a selection of “profiles” or pre-set configurations, defined by the operator of a cluster. For example, defining small, medium, and large resource requests, or special profiles for GPUs.

The above sample is created from the following snippet in your values.yaml:

singleuser:
  profileList:
    - display_name: "Small: default"
      description: |
        A small job. 1 CPU, no GPU. This is the default.
      default: True
      kubespawner_override:
        cpu_limit: 1
        cpu_guarantee: 1
        mem_limit: "1G"
        mem_guarantee: "512M"
        extra_resource_limits: {}
    - display_name: "Big: 8 CPUs"
      description: |
        A big job. 8 CPUs, no GPU, and 64GB of RAM
      kubespawner_override:
        cpu_limit: 8
        cpu_guarantee: 8
        mem_limit: "64G"
        mem_guaranttee: "64G"
        extra_resource_limits: {}
    - display_name: "GPU job"
      description: |
        This configuration gives you 4 CPUs, 16GB of RAM, and a GPU
      kubespawner_override:
        image: consideratio/singleuser-gpu:v0.3.0
        cpu_limit: 4
        cpu_guarantee: 4
        mem_limit: "16G"
        mem_guarantee: "16G"
        extra_resource_limits:
          nvidia.com/gpu: "1"

Autoscaling

Autoscaling is a big focus of this release, and many of the new features in 0.8 of the chart relate to scaling in some way. At the center is a new ‘user scheduler’ a custom kubernetes scheduler responsible for assigning user pods to nodes. To enable the user scheduler:

scheduling:
  userScheduler:
    enabled: true

There are two major pain-points for autoscaling with JupyterHub. First, is scaling up. Kubernetes scale-up is very basic: If scheduling a pod fails due insufficient resources, a new node is requested. This isn’t a great fit for JupyterHub. If you happen to be the unlucky user who is the first to need a new node, launching your server can take several minutes as the new node is requested and warming up. To prevent this, the chart adds the notion of “placeholder pods,” which are pods that request the same resources as user pods, but have lower priority and will thus be evicted immediately if a user pod needs their resources. This allows a cluster to always have a certain amount of “headroom” of vacant slots available for users so that requesting a new node occurs not when there are 0 slots available for users, but instead when there are 5 or 10 (you choose) slots available. See this exploration of how mybinder.org arrived at 25 placeholders for its deployment. To enable 10 user placeholders:

scheduling:
  podPriority:
    enabled: true
  userPlaceholder:
    enabled: true
    replicas: 10

The second major improvement to autoscaling that comes with the user scheduler is in scaling down. If your cluster is automatically scaling up, you probably want it to scale down as well when you no longer need the added capacity. Unfortunately, because user pods cannot be kicked out, Kubernetes’ default scheduler doesn’t manage to drain nodes that are no longer needed without some manual intervention. The user scheduler adds this feature, ensuring that, over time, your newest nodes eventually drain when they are no longer needed. It accomplishes this by prioritizing the node with the most other user pods when picking a node to start new users. If there is an extra node, over time eventually all the users on that node will stop and the node will become idle and culled by the cluster autoscaler. This is the default behavior . There’s loads more scheduling optimizations available in the updated documentation, including dedicating certain nodes to users so that user pods never run on the same node as others, etc.

And finally, a huge thanks to the numerous contributors who helped make this release, be it via documentation contributions, testing, discussion, or code.

Check it out

Jupyter Notebook security fixes

2018-11-19T22:20:00+00:00

Two security issues have been found and fixed this week, where untrusted javascript could be executed if malicious files could be delivered to the users system and the user takes specific actions with those malicious files.

The first allowed nbconvert endpoints (such as Print Preview) to render untrusted HTML and javascript with access to the notebook server. This is fixed in notebook 5.7.1. All notebook versions prior to 5.7.1 are affected. Thanks to Jonathan Kamens of Quantopian for reporting. This issue has been assigned CVE-2018-19351.

The second issue allowed maliciously crafted directory names to execute javascript when opened in the tree view. This is fixed in notebook 5.7.2. All versions of notebook from 5.3.0 to 5.7.1 are affected. Thanks to Marvin Solano Quesada for reporting. This issue has been assigned CVE-2018-19352.

You can check your version of the notebook package by issuing the following command:

jupyter notebook --version

Whether you are using classic notebook, JupyterLab or any other notebook server extensions, we recommend that you update the notebook package with :

pip install --upgrade notebook

or if you are using conda-forge

conda upgrade notebook

Thanks especially to Jonathan and Marvin for reporting these issues! If you find a security issue in a Jupyter project, please report it to security@ipython.org.

Security fix for Jupyter Notebook

2018-07-18T04:31:00+00:00

We have just released Jupyter Notebook 5.6.0. This release fixes a vulnerability that could allow a maliciously crafted notebook to execute JavaScript when it is opened, bypassing the trusted-notebook mechanism.

We recommend updating the notebook immediately, via pip:

pip install notebook>=5.6.0

or conda:

conda install notebook>=5.6.0

Affected versions: all releases prior to 5.6.0

JupyterLab users are affected, independent of the version of JupyterLab itself. Upgrading the notebook package to 5.6.0 resolves the issue for users of both JupyterLab and the classic notebook.

A CVE has been requested for the vulnerability. Release notes for 5.6.0 and this post will be updated as the CVE is assigned. More details of the vulnerability will be released in 30 days, on August 16, 2018.

Security reports for Jupyter are greatly appreciated. You can report security issues to security@ipython.org.

Thanks to Jonathan Kamens for reporting this issue to the security list.

[Update July 28] The vulnerability have been assigned number CVE-2018-1999024.

[Update August 16] MathJax versions prior to version 2.7.4 contains a Cross Site Scripting (XSS) vulnerability in the \unicode{} macro that can result in potentially untrusted Javascript running within a web browser, for example at notebook load. Notebook 5.6.0 ships with an updated mathjax version 2.7.4 which fixes this vulnerability.

JupyterHub 0.9

2018-06-18T13:18:00+00:00

We are pleased to announce the latest release of JupyterHub. JupyterHub is the multi-user server for Jupyter notebooks, allowing students or researchers to have their own workspace. This release has lots of improvements, especially for stability and performance with large numbers of users.

The biggest change is probably the adoption of asyncio coroutines throughout instead of tornado coroutines and improved support for asyncio coroutines in general, adopting async def coroutine syntax. This means that JupyterHub 0.9 requires Python ≥ 3.5 and tornado ≥ 5.0.

There are lots of improvements to the REST API, including more detailed time information and token management.

There are also several improvements to the ability to customize JupyterHub’s HTML templates.

See the changelog for a more detailed list of changes.

Make sure to backup your jupyterhub database prior to upgrade, and then upgrade with pip:

python3 -m pip install --upgrade 'jupyterhub==0.9.*'

or conda:

conda install -c conda-forge jupyterhub=0.9

For Kubernetes users, we will have an 0.7 release of the helm chart before too long.

Thanks to everyone who has contributed to JupyterHub!

Security fix for JupyterHub GitLab OAuthenticator Group Whitelists

2018-02-16T22:12:00+00:00

This vulnerability has been assigned CVE-2018-7206

If you are using JupyterHub with the GitLab OAuthenticator and its gitlab_group_whitelist support, there is a security issue where the authenticator will allow users outside your intended group whitelist to create accounts. A fix has been released as OAuthenticator 0.6.2 and 0.7.3. No other authentication mechanism, including GitLabOAuthenticator without using the group whitelist feature, is affected. If you are using GitLab authentication with group whitelist support, upgrade oauthenticator immediately:

python3 -m pip install --upgrade oauthenticator

Thanks to Joseph Weston for reporting the issue and providing the fix.

Timeline (all times UTC):

2018-02-16 09:51 Joseph Weston reports security issue to the Jupyter security list
2018-02-16 16:08 Fix is verified and applied to oauthenticator master
2018-02-16 21:52 oauthenticator 0.7.3 and 0.6.2 are released with the fix
2018-02-18 03:02 CVE-2018-7206 assigned

Public Notebooks and Security

2017-11-28T19:33:00+00:00

tl;dr: don’t disable notebook authentication!

Let’s chat a little bit about public Jupyter notebook servers and security. A lot of folks run notebook servers that are publicly accessible, whether they are the access point to a cloud server or remote access to their home or office computer. This is great! It is important to keep in mind that, since arbitrary code execution is the main point of IPython and Jupyter, running a publicly accessible server without authentication or encryption is a very bad idea indeed. Prior to notebook 4.3, this was the default behavior of Jupyter, which has led to a lot of insecure notebook servers running on cloud services. One aspect that may not be apparent is that people can find your public notebook server, especially if it is on one of the popular cloud providers. Systematic port scans will turn up notebook servers, especially for those running on the default port of 8888. And Jupyter has become popular enough (🎉) that we know people are doing this (😠). Running a publicly accessible notebook server without authentication is making your computing resources free for the world to use.

The highlights to keep in mind:

Always enable a password or token for publicly accessible notebook servers (this includes shared computers)
Use HTTPS for authenticated notebooks over the Internet
Unsecured public notebook servers are discoverable
Running on a cloud service should be treated as if it’s publicly accessible

The first two points shouldn’t be new information, and anyone following the documentation should be adding basic authentication when exposing a server to the public. If you are already accessing a notebook server over https with a password or token, you have nothing to worry about. However, if you are running unsecured notebook servers on the Internet, please stop and enable a password. The latter points are important and may not be obvious, but the gist is: don’t disable notebook authentication, even if you think your bases are covered via other means.

In notebook 4.3, a token is enabled by default, making it a little bit more inconvenient to run a totally insecure notebook server. We can’t feasibly require HTTPS by default, but free, trusted HTTPS has never been easier, thanks to LetsEncrypt. If you can’t assign a domain name to your server, self-signed certificates can be used, with some caveats (notably: Safari requires self-signed certificates be trusted at the system level before it will allow websocket connections). For shared deployments, you can also use JupyterHub, which allows you to authenticate notebooks via the system (PAM) or OAuth providers such as GitHub, Google, etc.

Be safe out there.

JupyterHub 0.8

2017-10-03T19:42:00+00:00

We are pleased to announce the release of JupyterHub 0.8. This is a big release with many fixes and improvements, and some major changes.

To upgrade from jupyterhub 0.7:

Stop jupyterhub
pip3 install --upgrade jupyterhub==0.8
Backup your database: e.g.cp -v jupyterhub.sqlite jupyterhub-backup-$(date +%Y-%m-%d).sqlite
Upgrade your database: jupyterhub upgrade-db

See the changelog for more details. The “Notes on Upgrading” below also provides more specifics.

OAuth

Perhaps the biggest change is the switch to using OAuth 2.0 for the internal authentication mechanism in JupyterHub. This shouldn’t have much of a visible effect on your deployments other than hopefully a reduction in redirect-loops when users login.

If you are writing a JupyterHub Service, then you can switch to OAuth by using the HubOAuthenticated or HubOAuth classes if you were previously using HubAuthenticated or HubAuth. All of these classes now support token-based access via the Authorization header, just like the Hub itself.

Stability, Scalability, and Performance

As part of supporting larger deployments, we have done some scalability stress testing of the Hub, finding and fixing several race conditions that could manifest as redirect loops, bugs, and performance improvements when the Hub is under load. This has been led largely by the Berkeley Data Science Education Program team, which has over one thousand students using a JupyterHub instance this Fall. We now know that one JupyterHub 0.8 instance can support a few thousand users logging in and attempting to spawn their servers at the same time. Once servers are active, the Hub is minimally involved, and the proxy becomes the next bottleneck.

Custom Proxy Implementations

Another major new feature targeted at scalability is custom proxy implementations. JupyterHub has always used Configurable-HTTP-Proxy (CHP), a single-process Node.js HTTP proxy. This is a single process bottleneck and potential single point of failure for Hub deployments. CHP can handle a lot of concurrent active users (at least several thousand), but for larger scale applications, Hub deployments may want to use a more scalable and/or robust proxy implementation.

To achieve this, JupyterHub 0.8 introduces a Python API abstracting the proxy needs of JupyterHub and developers can provide their own proxy implementations as alternatives to CHP. The first such implementation uses a Kubernetes Traefik-based Ingress for the proxy, as part of KubeSpawner.

Persisting Authentication State

Another new feature available to Authenticators is the encrypted persistence of authentication state. This is aimed at preserving and passing authenticator-related state, such as client certificates or GitHub API tokens. Authenticators provided by oauthenticator 0.7 support persisting auth state from upstream authentication services.

Other Highlights

Preliminary (API-only, no GUI) support for multiple named servers per user
Token-based access to the single-user server API
A page for users to request new API tokens from the Hub

Notes on upgrading

As with every update to JupyterHub, make sure to back-up your database and run upgrade-db. For example, if using JupyterHub with SQLite:

Backup:cp -v jupyterhub.sqlite jupyterhub-backup-$(date +%Y-%m-%d).sqlite
Upgrade: jupyterhub upgrade-db

Due to the change in internal authentication, it is important that you make sure that single-user servers and the Hub itself are upgraded at the same time. If you are running everything in one env (e.g. the default Spawner, SudoSpawner, or similar), there is nothing to do. However, if you are using a container-based setup to launch single-user server (e.g. DockerSpawner or KubeSpawner), make sure that your user image and your Hub both get the upgrade at the same time. In most cases, this means adding to your Dockerfile:

RUN pip3 install jupyterhub==0.8.0

which ensures the correct version of JupyterHub is installed.

Thanks to everyone who has contributed to this release, especially users who have helped out with testing JupyterHub during the beta process, which helps make this a great release!

nbviewer and GitHub’s rate limit

2017-03-01T07:45:00+00:00

If you’ve been using nbviewer recently, you may have seen this message more often than you would like:

It turns out that some overzealous SEO bots were eating up all of our resources. Last week, we discovered this cause of the API rate limit exhaustion and took a few actions to resolve it.

tl;dr; nbviewer shouldn’t hit the rate limit so often anymore. This is what our API usage availability has looked like since we deployed the changes I’ll describe here:

nbviewer’s architecture

I’d like to take this opportunity to give an overview of how we deploy nbviewer, so we can see how each piece comes into play.

To start off, nbviewer is two things:

the web application that finds notebooks on the internet and renders them, and
the specific deployment of nbviewer serving https://nbviewer.jupyter.org

You can even roll your own nbviewer by grabbing the public docker image and running:

docker pull jupyter/nbviewer
docker run -p 5000:5000 -it jupyter/nbviewer

You can then start rendering notebooks left and right. However, there’s a limit to how scalable such a basic deployment can be.

Rendering a notebook

Rendering a notebook involves two major steps:

download the notebook from somewhere on the web
render it to HTML with nbconvert

Each step can take a bit of time. Fetching and rendering a notebook start to finish can take as much few seconds if the notebook is big and/or the upstream server is busy or slow. We use tornado’s async APIs to handle concurrent requests. Fetching notebooks from the web is handled via a subclass of AsyncHTTPClient and rendering is done in the background via concurrent.futures. A single nbviewer instance can handle quite a few concurrent visitors, depending on how many rendering threads are available.

Caching

Most notebooks on the web don’t change very often, so going through the whole process when you just downloaded and rendered the same notebook a minute ago is a bit of a waste. This is where caching comes in.

nbviewer supports caching via a simple in-memory cache or memcache at two levels:

Every time nbviewer completes a render, it caches the result for a period of time (usually 10 minutes to an hour, depending on render time), so that a very popular notebook doesn’t consume too many resources. This is unconditional, and doesn’t take into account whether the original notebook has been updated. If it’s in the cache, it gets reused.
nbviewer separately caches upstream requests. When nbviewer asks an external service for anything, whether it’s a regular web server or GitHub’s API, the response is saved for a time. The next time the same resource is requested, nbviewer checks for a cached response. Unlike the full render cache, this cache doesn’t prevent a new upstream request. Instead, we check the headers of the cached reply (ETag and Last-Modified). nbviewer then takes the content of those headers to populate If-None-Match and If-Modified-Since headers before sending a new request. The server can then check these headers and return an empty reply with status 304, telling nbviewer that it should reuse the cached response instead of running the full process of handling the request again. Critically for nbviewer, when the GitHub API responds with 304, the rate limit is not consumed.

Our nbviewer deployment

To deal with the traffic our public nbviewer instance gets, we have deployed nbviewer with a few extra elements. The moving parts:

we have multiple nbviewer instances on multiple servers deployed. Right now, there are two nbviewer instances (running in docker) on each of two servers running on rackspace, making four total instances.
memcache, also running in docker, with separate instances on each server
fastly sits in front of our nbviewer instances and load-balances requests across them, performing health checks, etc. It also has its own cache layer.
Finally, cloudflare is out front, handles SSL termination, and a small amount of caching

Four instances is quite a bit more than we need for the load nbviewer gets, but having it spread out lets us do things like reboot or upgrade servers without any downtime.

What happened?

So, why were we seeing the GitHub rate limit hit so often? Did nbviewer just get too popular? Sadly, no. How did we figure out what happened? To the logs!

Fun fact: until now, we were using docker’s default json logging, and nbviewer was producing 50GB of logs each month. docker logs --tail does not behave super well with a 30GB log file. I started digging through the logs to see what was going on, but quickly realized that this wasn’t working very well. To try to get a better sense of things, I started by hooking up our logs to Loggly via syslog. Loggly lets me start to extract some metrics, such as status, URL, IP address, and user-agent of failing requests.

What did we learn?

One of the first things I did was group incoming requests by IP address, to see if anything stood out. This is what I saw:

So about two thirds of requests were coming from just two ip addresses! That’s not right. Checking for user-agent showed something else:

Two bots, Ahrefs and DotBot were accounting for almost three quarters of requests. That’s too much.

What we did about it

The very first thing we did was to block the two IP addresses that were driving way too much traffic. This was done in the fastly layer, sending a custom 429 error for any request from these IPs:

This improved availability a bunch, and would probably have been plenty on its own to resolve the current situation. Of course, it wouldn’t be enough to prevent similar situations in the future. Since the problem was caused by robots, the next place to look was robots.txt. Our robots.txt had been the most permissive possible:

User-agent: *
Disallow:

explicitly allowing any bot. Turns out we were being too nice. We’ve now updated our robots.txt to tell all robots to slow way down, and instructed the two overzealous bots that they are not welcome:

User-agent: *
Crawl-delay: 10

User-agent: dotbot
Disallow: /
User-agent: AhrefsBot
Disallow: /

Again, this should be enough to resolve the issue if these and all other bots are honorable and respect the robots.txt.

However, we would prefer to not rely on the kindness of strange robots.

Better caching and better use of the cache

As mentioned earlier, we cache the responses we get from upstream sources and use those to reduce the load on the GitHub API and other sources of notebooks. However, we were not making the best use of that cache. First of all, the responses were only being cached for two hours, which means this caching only help us for relatively short bursts of activity (e.g. a popular notebook being shared on social media). This certainly happens, but does not represent a large fraction of typical traffic. Since the way we check for updates with If-Not-Modified has no risk of stale results, we changed this to cache upstream responses forever, until memcache pushes them out when it gets full. Since most public notebooks don’t change often, we should get a cache hit almost every time.

The next change we made was an improvement to behavior when the rate limit is hit. Previously, if we had a cached upstream response and GitHub told us there was an update that we couldn’t fetch due to the rate limit, we would serve the 503 error page. But we had valid data right there! Now, if we hit the rate limit but happen to have an old version of the data, we work with the stale version rather than serving an error page. Something’s better than nothing, right? This applies to any kind of error fetching from an upstream server, not just failures due to hitting the GitHub rate limit.

So we are now much less likely to hit the rate limit in the first place, and much more likely to be able to serve something if and when we hit it.

Our own rate limit

The last thing we did to help limit the consumption of resources by a small number of actors was to implement our own rate limit in nbviewer. This is a pretty simple rate limit, where we identify a source by the combination of ip address and user-agent. Currently, this limit is set at 60 requests in ten minutes. We only count non-cached requests against the limit, so refreshing the same few pages a bunch won’t consume the limit.

Like the cache, the rate limit is actually implemented with memcache. When we see a request from a given source, we initialize a counter to memcache with an expiry of our rate limit window:

key = cache_key(request)  # combines ip and user-agent
new_visitor = memcache.add(key, 1, rate_limit_window)

If it’s not a new visitor, we increment the counter with memcache’s atomic increment operation:

if not new_visitor:
    count = memcache.incr(key)

The last thing to do is to check if the rate limit is exceeded, and return an HTTP 429 error instead of proceeding with the render:

if count > rate_limit:
    raise HTTPError(429, "Rate limit exceeded...")

If you hit the rate limit you will see:

From looking at our logs, we can see that this rate limit has blocked ~800 requests since deploying it a few days ago. More than 600 of these were from a bot that apparently doesn’t respect robots.txt. A few dozen were made from phantomjs, presumably some manual scraping that made a few too many fetches. That leaves a few probably-real humans who hit the rate limit, but no IP has hit it often or for a sustained period of time.

Let us know if the rate limiting is causing an issue for you. We will keep an eye on the limits, and continue to tune both the limits and limiting conditions to find the best balance.

Monitoring

We run our instances with the New Relic Python agent, which gets us some basic health monitoring. We also have some monitoring and alerts in our Rackspace account to notify us when things go down.

We’ve added the error rate tracked by New Relic and a new metric that tracks how much of our GitHub API limit is available to our public status page, so you can see how healthy nbviewer is at any given time.

One last thing

After confirming that everything was working well, I did a final deploy on a Friday evening, and moved on. Then this came in:

https://twitter.com/williamrthomas/status/835255588116447232

Turns out there had been an update to the New Relic Python agent since the last deploy, which introduced this tiny off-by-one error:

if len(args) > 0:
    args = list(args)
    args[1] = wrapped_callback
    # what if len(args) == 1 ?!

(that > 0 should be > 1)

This little bug just so happens to prevent nbviewer from making any outgoing requests :shrug:. This served as a friendly reminder that it is a good idea to pin your dependencies when deploying applications so that updates only happen when you decide they should.

Security release: Jupyter Notebook 4.3.1

2016-12-22T02:25:00+00:00

We have just released Jupyter Notebook 4.3.0 and 4.3.1 with some important security fixes.

You can check out the changelog for more details on the many fixes and improvements. I’m going to focus on the security changes in this post.

4.3.0: Token authentication

The biggest change in 4.3 is the addition of token-based authentication, which is enabled by default. Several of the security issues the notebook has had over the years would have been avoided if the notebook ran with authentication by default. We did not do this for a long time, because the process of enabling and entering a password on a localhost-only web application was deemed too cumbersome, and the risks too small, given that it listened only on localhost. With automatic token authentication, I think we have a solution that should be convenient enough for most users that we can now switch to enabling authentication by default.

If you use Jupyter in its default configuration, which is to open your browser automatically when you type jupyter notebook, this shouldn’t have much impact on your daily work. In this case, a one-time token is generated, and your browser opens with this in the URL. That token is immediately consumed, and your browser will be authenticated, storing the information in a cookie for future requests.

If you access the notebook without authentication, you will see this page:

This may happen if you switch browsers, or use the notebook with the --no-browser option, in which you will need to enter the token manually the first time you connect to a server with a given browser. This can be done at the login page above, or via a URL containing the token. The notebook server logs this URL when it starts, so you can paste the URL into your browser:

[C 16:11:17.581 NotebookApp]
    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8888/?token=20abd368...

After you have logged in with this token just once, a cookie is set and used thereafter, even across restarts of the notebook server. So you shouldn’t need to deal with the token after the first time you have used the notebook on a given machine, until/unless you switch browsers or launch new notebook servers on different ports. This does mean that cookies are required for authenticated access to notebooks, and clearing cookies means needing to login again.

At any time, you can see all of your current notebook servers with their token-authenticated URLs by running the jupyter notebook list command:

$ jupyter notebook list
Currently running servers:
http://localhost:8888/?token=1234... :: /Users/you/notebooks

We didn’t communicate the information about token authentication very well in 4.3.0, and I apologize for that. 4.3.1 should make the token behavior much clearer, with more information in both the login form and the notebook log output about how to use the tokens and when you need them. You can see the docs for more details. We’re always open to improvements, so please open an Issue if you are having trouble.

4.3.1

4.3.1 follows shortly after 4.3.0, with a fix for a CSRF vulnerability (CVE-2016-9971), affecting users of the Firefox or Microsoft (IE, Edge) browsers, and any other browsers that do not set the Origin header on cross-site forms. WebKit and Blink based browsers like Safari and Chrome are not affected. The effect of this vulnerability is the ability of forms on malicious websites visited by the user to spawn new kernels and create empty, untitled files on the user’s notebook server. CORS protections already in place prevent further access to these kernels and editing the content of any files.

Summary:

CVE-2016-9971
Affected notebook versions: all notebook releases < 4.3.1
Affected users: Users of Firefox or Microsoft browsers, or other browsers that do not set the Origin header on forms.
Results: Visiting malicious pages with these browsers could result in unauthorized launch of new kernels or creation of empty, untitled files on the user’s notebook server
Fix: Upgrade to notebook-4.3.1, which uses an xsrf token via cookies to validate the origin of forms. See tornado docs for details.

Update: practical effects of xsrf for extensions

This note was omitted when first posted.

The use of the xsrf token affects existing extensions / etc. that work directly with the notebook server via REST API. To make requests to the rest API, the _xsrf cookie must be sent back to the server in the X-XSRF-Token header of API requests, for example:

function _get_cookie(name) {
  // from tornado docs: http://www.tornadoweb.org/en/stable/guide/security.html
  var r = document.cookie.match("\\b" + name + "=([^;]*)\\b");
  return r ? r[1] : undefined;
}

function _add_xsrf_token(settings) {
  // Adds xsrf token header to jquery ajax settings
  settings = settings || {};
  var xsrf_token = _get_cookie('_xsrf');
  if (xsrf_token) {
    if (!settings.headers) {
      settings.headers = {};
    }
    settings.headers['X-XSRFToken'] = xsrf_token;
  }
  return settings;
}
function ajax(url, settings) {
  // like $.ajax, but ensure xsrf token is added to headers
  settings = _add_xsrf_token(settings);
  return $.ajax(url, settings);
};

This code will also work for any previous version of the notebook, where the xsrf cookie will be undefined and ignored.

If you’re using fetch, you’ll need to set credentials: true as well so that the Cookie header is set.

var xsrfToken = _get_cookie('_xsrf')
fetch("http://127.0.0.1:8888/api/contents/",
  {
    method: "POST",
    credentials: "include",
    headers: {
      "X-XSRFToken": xsrfToken,
      "Content-Type": "application/json",
    },
    body: { name: "Test.py" }
  }
)