Jupyter Blog - Matthias Bussonnier

Release of IPython 8.0

2022-01-12T13:37:00+00:00

IPython is a powerful Python REPL that gives you tab completion, better tracebacks, multiline editing, and several useful features on top of pure Python Scripts. It is also the library that powers the Jupyter Kernel via the IPykernel.

Today I am pleased to announce the release of IPython 8.0, which has been long in the making and arrived a bit over three years after the 7.0 release.

I also suggest to read the companion blog post on Quansight-Labs site that goes into some technical details on how we removed some old code.

EDIT: You can also ask questions/comments on the jupyter discourse, or read the discussion on HN

Many new features

This major release comes with many improvements to the existing codebase and several new features. These new features are code reformatting with Black in the CLI, ghost suggestions, and better tracebacks which highlight the error node, thus making complex expressions easier to debug (see below).

Tracebacks now highlight in which AST node the error occurs. In complex code, this helps to quickly narrow down what causes an error. Here we also turn on xmode verbose to see values of local variables (xmode verbose is disabled by default for security reasons as it could leak secrets, but is highly recommended)

But really, if you want to read about all the new features in-depth, we recommend you spend some time reading the what’s new and the list of features in 7.x.

Moving forward by shedding weight

While IPython 8.0 has many new features described in the what’s new section, the majority of changes that demanded a bump in major version numbers are removals.

IPython was created more than two decades ago by Fernando Pérez while procrastinating on his graduation. Even if a few lines from this period remain, it was time to remove a large number of deprecated and unused code in the IPython code base and drop old dependencies (like nose) in favor of more recent ones (pytest).

Thanks to NumFOCUS Small Developer Grant, we hired Nikita Kniazev (@Kojoley) to help us with some of the heavy lifting. Nikita did a fantastic job updating our codebase, migrating to pytest, enhancing our coverage, and fixing numerous bugs. I highly recommend contacting them if you need help with C++ and Python projects.

Some of Nikita’s most notable contributions are:

IPython is no longer reliant on Nose, which has been unmaintained for many years.
Significantly increased IPython coverage.
Considerable refactor of areas that were calling into deprecated features.

On top of this, IPython 8.0 also comes with the following capabilities:

Remove most of the deprecated functions and parameters that were marked as such between IPython 1.0 and 5.0
Bumped the minimal required Python version to 3.8 (following NEP 29). This means that we now use the native Python top-level async instead of crazy hacks. And we can start to use def fun(a,/,b=None): syntax for positional only arguments.

Nikita also went well beyond the scope of work we gave them by suggesting more robust and often simpler code patterns.

In total, this allowed us, despite all the added features, added type annotations and added tests, to decrease the size of the codebase from 37 500 LOC across 348 files to 36 100 across 294 files. We hope that this reduction in the codebase size combined with work on speeding the startup of the CLI will make IPython easier to use and contribute to.

Some advice on how to do that same are on the companion blog post and the Quansight-Labs site.

Moving toward pyproject.toml

Another place in IPython that has seen many changes in the build process. We now use pyproject.toml to have a declarative build. We still require setuptools and have setup.py files right now, but we are working towards removing/simplifying them soon.

IPython’s wheels and sdist can now be built with python -m build instead of invoking setup.py directly.

Our process still needs manual setup of SOURCE_DATE_EPOCH and repacking of the sdist wheel to obtain reproducible builds. We encourage you to get the git repository, build it yourself, and get a byte-for-byte identical artifacts.

Monthly Release and following NEP 29

We strongly believe that predictability is key to adoption and trust in IPython. In particular, in business settings, it is critical to plan ahead while rapidly receiving bug fixes. This is why for IPython 8.0 we’ll continue to do minor monthly releases on the last Friday of each month as long at it’s reasonable and does not impact maintainers life too much.

We’ll still publish a few 7.x releases with critical bug fixes, but starting last Friday of this month, you will start to see stable releases of 8.x.
All releases are announced in this subcategory in discourse, to which you can get subscribed to get notifications.

We also now follow NEP 29, meaning that IPython 8.0 is not only compatible with NumPy 1.19+ and Python 3.8+. But our future Python support schedule is predictably aligned with all other libraries following NEP 29.
We believe the new NEP 29 is critical. It gives the core Scientific Python Ecosystem a clear schedule of available features and the removal of deprecated features.

Looking forward to your contributions

Don’t think IPython is too hard to contribute to or has no missing features. We try to keep a list of beginner-friendly issues, but we really want your feedback and your ideas. In the meantime, have fun with this new release that you can install with :

pip install ipython ipykernel --upgrade

conda update ipython ipykernel

(updating ipykernel may not be necessary for you, but we recommend it).

Thanks

Thanks to Tania Allard and Eric Charles for review , Quansight Labs and Quansight (my employer) for all the open-source work I can do. Of course, thanks to all the contributors to the IPython and Jupyter community, and in particular MrMino who recently joined as an IPython core dev…

Jupyter’s role in #ChaosDB

2021-09-01T21:43:00+00:00

On August 26 it was revealed that a misconfiguration in Microsoft’s internal deployment of CosmosDB using Jupyter would allow attackers to access all customer data. Fortunately, they report no evidence that customer data was compromised.

Nonetheless, articles onlines, like Ars Technica’s and Reuter’s have strong headlines associated Jupyter, for example “Worst cloud vulnerability you can imagine.”

This can be especially alarming for our community as no details on the vulnerability have been released yet, and members of our community wonder about Jupyter’s possible role in this vulnerability.

What the Jupyter team knows

We learned about the CosmoDB vulnerability at the same time as everyone else; we had no prior notice, and received no privileged communication about this issue. We have not seen any evidence suggesting this relates to a vulnerability in Jupyter itself, as opposed to a misconfiguration of Microsoft’s internal services.

We also had no prior interaction with the Microsoft team about their internal Jupyter deployment in CosmoDB.

From the descriptions posted by Wiz and Microsoft, there is no suggestion of any vulnerability in Jupyter itself, and rather expect that Jupyter was used as convenient shell to exploit a vulnerability in the configuration of Microsoft’s internal services, but we have no information beyond what is publicly available to support that claim.

What are we doing internally

Even if Jupyter does not have a vulnerability to fix, it is often possible for us to warn end users when risky configurations options are set. For example, if you try to login to JupyterHub over a non https connections, you will a see a warning.

JupyterHub warning about login over unsecured HTTP

We are preparing for the full information release of #ChaosDB details, to see if there are any relevant safeguards and warnings to implement on the Jupyter side. We are also trying to reach the involved Microsoft Security Team personally to know whether there are steps we can take before public disclosure.

In the meantime you can contribute and get involved:

The security working group meets every other Friday. You can follow and open issues the repository for public questions and meeting minutes.
You can send your questions and concerns about security to security@ipython.org to reach out to our security team.

We always welcome feedback, questions, and help regarding security in Jupyter.

CVE-2021-32797 and CVE-2021-32798 Remote Code execution in JupyterLab and Jupyter Notebook

2021-08-09T20:05:00+00:00

TL:DR; All recent JupyterLab and Notebook versions are susceptible to a attack where a maliciously crafted notebook can trigger arbitrary code execution when a user views these malicious files.

We strongly advise all users to deploy the new version of JupyterLab and Jupyter Notebook.

Jupyter Notebook 6.4.1 or above, 5.7.11 or above.

Jupyter Lab 3.1.4 or above, 3.0.17 or above, 2.3.2 or above, 2.2.10 or above , 1.2.21 or above

This blog post will be updated with links to the various patches, exploit and disclosure later once the final links are available.

Jupyter Security Model

Jupyter Notebook act as a REPL (Read Eval Print Loop) in a browser, our main goal is to expose as many functionalities to our users, with the least restrictions. We also want users to be able to share their results with other, and let everyone be capable of reproducing the result.

When receiving an untrusted notebook from a potentially malicious source we still want users to be able to inspect a notebook without risks, our approach is that an untrusted notebook has restrictive capabilities until all cells have been manually inspected and explicitly run by a user, or the notebook is explicitly marked as trusted. If one finds a way to bypass this trust mechanism, a notebook might be able to execute code in the browser at at a time where a user is not expecting execution to occur.

This is what happen in these particular CVEs, where some content of a notebook were improperly handled.

[This part of the blog post will be updated with links to actual reports, and proof of concept once the patched version have reached enough package repositories]

CVE Timeline

We want to thanks Guillaume Jeanne (Google), and Timo Schmid (Google) for the vulnerability report and helping us through the fixing process.

Thursday, 15 Jul 2021 07:11:13 -0700 (PDT): Initial question about security issue disclosure on security@ipython.org (no specific on the vulnerability)
Tuesday, 20 Jul 2021 05:52:09 -0700 (PDT): Actual Vulnerability Report.
Tuesday, 20 Jul 2021 : Open relevant GitHub security advisory on Notebook and JupyterLab Repositories
Thursday, 5 August 2021 : First releases with patched version on PyPI.
Monday, 9 August : publication of this blog post and publish security advisory on GitHub.
[Further item may be added to list publication by downstream repositories, like conda, conda-forge, debian…, contact us to add an item]

What we learned

Dealing with Security vulnerability in Jupyter and more generally in Open-Source is far from easy. We initially thought about treating the Notebook and Lab CVE separately but they are/were too similar and the cross discussion too revealing to treat both independently.

As Jupyter is mostly a volunteer based organisation, there is often no contributor responsible to replying to security issues, and when someone steps up they might not be a expert, nor available all the time. I want to give special thanks to Steve Silvester (Apple), Afshin Darian (Two Sigma), and Zach Sailer (Apple), and Matthias Bussonnier (Quansight) for writing the fixes, planning through the release and notifying stakeholders.

Advance notice to stakeholder is complicated, we know of a few large deployment and have personal connections to a couple organisation, and were able to reached out to let them know critical release would be published. There is an inherent tension between publicly warning our user base that security release would be published, which might push malicious actor to closely survey the codebase changes and re-derive the attack vectors, and publishing the release first, with the details later. Especially since we are trying to be transparent in our communication, security fixes are the opposite of our normal communication workflow.

Our communication process is imperfect. We have 2 mailing list for security-related discussion. The first one – security@ipython.org – has only a couple of members all core contributors and can receive email from the outside, it is used for triage. It receive a high number of spam as this is public email. As it has only a few members and we are all busy, mail can slip through. The second mailing list is slightly larger, and used for internal announcement for stakeholder. It has a fairly open membership model, (ask a Jupyter developer if you can be on it, and the reason why and we’ll likely add you), though it’s content seem to be ignored (it even lands on my spam folder, not sure why).

What we’ll do better

In order to attempt to better react and be proactive with respect to security we are attempting to form and new security-focused subproject/workgroup to educate and have procedure for everything security related. We welcome your involvement and feedback in how to improve Jupyter security and how to better involve the community.

IPython 7.0, Async REPL

2018-09-27T17:41:00+00:00

Today we are pleased to announce the release of IPython 7.0, the powerful Python interactive shell that goes above and beyond the default Python REPL with advanced tab completion, syntactic coloration, and more. It’s the jupyter kernel for python used by millions of users, hopefully including you. This is the second major release of IPython since we stopped support for Python 2.

Not having to support Python 2 allowed us to make full use of new Python 3 features and bring never before seen capability in a Python Console. We are still encouraging library authors and users to look at the Python 3 Statement to learn about the end of life of Python 2 and how to stop support for Python 2 without breaking installation for Python 2 end users.

As developers and maintainers of IPython, it was a large gain of time to be able to only develop for a single version of python. Avoiding the use of conditional imports, being able to rely on type annotations, and make use of the newly available Python APIs were some of the advantages that made us more productive. Especially as most of the work on IPython is done by volunteers who work on nights and weekends, with only a couple of minutes here and there, this often made the difference between a patch reaching completion, or the contributor moving on to other pastures.

One of the core features we focused on for this release is the ability to (ab)use the async and await syntax available in Python 3.5+. There are of course many other improvements in this release you can read about in the what’s new.

Demo of awaiting coroutine in IPython 7.0

TL;DR: You can now use async/await at the top level in the IPython terminal and in the notebook, it should — in most of the cases — “just work”. Update IPython to version 7+, IPykernel to version 5+, and you’re off to the races.

See how to install IPython by reading the “what’s new”.

The recipes are currently building on conda-forge and should be available soon. For the time being you can install it via pip:

$ pip install ipython ipykernel --upgrade

A Primer on concurrency

You may have heard about async/await, threads, concurrency, preemptive scheduling and cooperative scheduling without really understanding what all this is about. If you are not familiar will all the above terms, all the hype may be confusing so let’s talk about concurrency in a really high level way.

Typically when your computer needs to execute many tasks, it will switch between them really fast, so from the human point of view it looks like everything is being processed at the same time. There are two main ways of doing so under the hood: Preemptive Scheduling, and Cooperative Scheduling.

With preemptive scheduling changing tasks can happen at any time. For example, while writing this blog post, I could stop in the middle of a word to start writing an email, which will itself be interrupted to check Gitter/Slack, before coming back, writing 5 words and stopping to get dinner.

TL;DR: Concurrency (from Geek And Poke, 2009)

With cooperative scheduling, the task switches can happen only at agreed spots. The term co-operative comes from the fact that tasks need to co-operate for the whole process to function. If a task decides to never take a break to let you to do something else, the illusion of many tasks being completed at once disappears.

Each approach has its own advantages and drawbacks, and we will not focus on these. Let’s just say that with co-operative scheduling async/await let you mark the areas where interruption is allowed to occur.

Moreover, async/await syntax allows cooperative scheduling in Python in a way that lets you write code that looks synchronous (without task switches), while actually being able to be interrupted, from the point of view of the computer. It also keeps the programmers from having to worry about global state changing under their feet, as this can occur only at the proximity of await keywords.

When going to a restaurant, social conventions (and common sense) tell us when and how these interactions can or cannot be interrupted, but programming languages need markers when using cooperative scheduling. These are async and await keywords in Python. Async marks a function that may be interrupted, await is required to call async-functions (aka coroutine) and marks a point were task can be switched.

If you want to learn more we strongly recommend reading the Trio Tutorial Primer on async programming.

Async in the Python world

In the current Python ecosystem, packages tend to standardize around AsyncIO, provided in the Python standard library. AsyncIO can sometimes be judged as complex even by well known developers; this is in part due to the necessity of supporting other older asynchronous projects like twisted or tornado, but it’s also what makes a lots of its power: One event loop to rule them all.

Running a single async task requires you to learn about AsyncIO, write a non negligible amount of boilerplate code in order to fetch a single result. This can be especially cumbersome when doing interactive exploration, and likely will keep users from experimenting with AsyncIO code.

How to run a single async task in Python repl without async integration.

As Raymond Hettinger would says (slamming hand on podium): “There must be a better way”.

IPython AsyncIO Integration

Thanks to a multiple month effort (actually this work started close to 2 years ago), and the work of many talented people, you can now directly await code in the REPL and IPython will do “the right thing”.

Awaiting AsyncIO code should now automagically work.

With the new integration, you don’t have to import or learn about asyncio, deal with the loop yourself, or wrap your task in its own function. You are now able to just focus on the business logic and move along.

The only thing you need to remember is: If it is an async function you need to await it.

We hope that this will free users to experiment and play with asynchronous programming. Of course this will not magically make your code faster, or run in parallel, simply easier to write and reason about.

Other Async Libraries (aka: curio and trio integration)

The addition of async and await keyword in Python did not only simplify the use of asynchronous programing and the standardization around asyncio; it also allowed experimentation with new paradigms for asynchronous libraries. David Beazley created Curio, and Nathaniel Smith Trio, which both explore new ways to write asynchronous programs and explore how async, await and coroutines could be used when starting from a blank slate. The Trio documentation introduction and which problems it attempt to solve [1, 2, 3, 4, 5, 6] are highly recommended reading with varying level of technicality.

Interactive uses of libraries is key to getting insight and intuition on how a system works, intuition is critical to rapid prototyping, development and creation of higher levels of abstraction. It was natural for us to build support for Curio, Trio, (and potentially new other async libraries) into IPython.

You can setup IPython to run async code via Curio, or Trio and experiment or write production code using these libraries. To do so use the %autoawait magic, and tell it which library to use.

Defining an asynchronous function and spawning multiple concurrent task in IPython using Trio.

As you can see code looks really natural, and it is easy to forget that the above snippet is usually a syntax error in Python or older version of IPython. The astute reader and IPython expert will have suggested to use the %%time cell magic instead of doing it manually, though a couple of magics still need updates to properly handle async code. We look forward to your contribution on this front, and are excited to see what you can come up with.

Async in Notebooks (and other Jupyter Clients)

If you are a Jupyter user, you most likely use a Notebook interface, and interact with IPython via the ipykernel package.

Using AsyncIO in nteract desktop works out of the box with newer IPython and IPykernel

We’ve been working hard on making async code work in a notebook when using ipykernel. While most of the heavy lifting was done in IPython, the work in IPykernel was non-negligible, and required the accommodation of a number of use cases, which are not working. You now have to update both IPython to 7.0+ and ipykernel to version 5.0+ for async to be available. If you are using pip: $ pip install IPython ipykernel --update. As for conda, the packages should be available on conda-forge soon. With these new releases, async will work with all the frontends that support the Jupyter Protocol, including the classic Notebook, JupyterLab, Hydrogen, nteract desktop, and nteract web. The default code will run in the existing asyncio/tornado loop that runs the kernel. Integration with Trio and Curio is still available, but tasks will not be interleaved with the asyncio one — at least not yet. We welcome work on this front.

Submitting background tasks still requires you to access the asyncio event loop, and we are still be looking for contributions on this front as well, to make it even easier to run async code.

There are still some question on how to handle nested asyncio eventloop. It is indeed usually impossible to run nested eventloop, in the case of asyncio, trying to do so raises a RuntimeError the kernel already ran in and asyncio eventloop, calling directly or indirectly loop.run_until_complete and alike is not possible. There are discussions to use libraries like nest_asyncio as pointed out on this comment, but until those are more battle tested we do not want to commit a default solution in the core of IPython and let the ecosystem develop.

Future improvements

As far as we know, this is the first Async-aware Python REPL, and libraries like Trio/Curio are still young, thus there are still a number of use-cases we have not yet even thought about! We are encouraging you to come forward to talk about your use cases, what you tried and what did not work. There is also a number of new features to implement (making magics work with async, tab completion, background tasks) on which we would welcome new contributors.

I Python, You R, We Julia

2018-05-29T16:11:00+00:00

When we decided to rename part of the IPython project to Jupyter in 2014, we had many good reasons. Our goal was to make (Data)Science and Education better, by providing Free and Open-Source tools that can be used by everyone. The name “Jupyter” is a strong reference to Galileo, who detailed his discovery of the Moons of Jupiter in his astronomical notebooks. The name is also a play on the languages Julia, Python, and R, which are pillars of the modern scientific world. While we ❤️🐍(Love Python), and use it for much of the architecture in Jupyter, we believe that all open-source languages have an important role in scientific and data analysis workflows. We have strived to make Jupyter a platform that treats all open-source languages as first-class citizens.

You may know that Jupyter has several dozen kernels in as many languages, and that you can choose any of them to power the code execution in a single notebook. However, the possibilities for cross-language integration go way beyond this, which I’ll attempt to demonstrate here.

What I’ll describe below has been possible for now several years – from even before the name Jupyter was first mentioned. It relies on the work of many Open Source libraries, too many to cite all the authors. It is not the only solution — neither the first, not the last. RStudio recently blogged about reticulate, which allows you to intertwine Python and R code. BeakerX is also another solution that appears to to support many languages.

We hope that showing how multiple languages can be use together will help make you more efficient in your work, and that it promotes cooperation across our communities to use the strengths of each language. This article only scratches the surface, you can read more in depth what you can do and how this works in a notebook I wrote some time ago.

Follow along on Binder

We created Jupyter and Binder to make science more trustworthy and allow results to be replicate. If you doubt what I have written below, or just want to follow along feel free to try on your own using Binder — the docker image is quite big so can take a while to launch. In the linked notebook we show a couple of extra languages.

The Tail of Fibonacci

A famous example of recursion in Computer Science is the Fibonacci series, its ubiquity allows the reader not to focus on the sequence itself but on the environment around it. As a reminder, the Fib sequence is defined with its first two terms being one, then each subsequent term as the sum of the two preceding terms; i.e F(1)= 1, F(2)=1, F(n) = F(n-1)+F(n-2)

We can calculate the first few terms: 1, 1, 2, 3, 5, 8 … note that F(5) is a fixed point F(5) = 5, and trust that asymptotically the sequence behaves like exp(n).

Let’s see how one can use many languages to play with fibonacci.

I, Python

For this exploration we’ll start with Python. It is my language of choice, the one I’m the most familiar with:

def fib(n):
    """
    A simple definition of fibonacci manually unrolled
    """
    if n<2:
        return 1
    x,y = 1,1
    for i in range(n-2):
        x,y = y,x+y
    return y

We can check that the fib function works correctly.

>>> [fib(i) for i in range(1,10)]
[1, 1, 2, 3, 5, 8, 13, 21, 34]

And plot it:

%matplotlib inline
import numpy as np
X = np.arange(1,30)
Y = np.array([fib(x) for x in X])
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.scatter(X, Y)
ax.set(xlabel='n', ylabel='fib(n)',
       title='The Fibonacci sequence grows fast !')

As you can see it grows quite quickly, actually it’s exponential. Now let’s see how we can check this exponential behavior using multi-language integration.

You R

With the fantastic RPy2 package, we can integrate code seamlessly between Python and R, allowing you to send data back and forth between the two languages. RPy2 will translate R data structures to Python and NumPy, and vice versa.

In addition, RPy2 has extra integration with IPython and provides “Magics” to write inline or multiline R code. Loading the RPy2 extension exposes the %R, %%R, %Rpush and %%Rpull commands for writing R.

%load_ext rpy2.ipython

We can use %RPush to send data to a stateful R process.

%Rpush Y X

and use %%R in order to instruct the R process to run an R cell.

%%R
my_summary = summary(lm(log(Y)~X))
val <- my_summary$coefficients

plot(X, log(Y))
abline(my_summary)

Here we make a linear regression model on log(Y) vs X. As Y is (hopefully) exponential, we should get a nice line. RPy2 provides rich display integration which will nicely display outputs and plots inline in a notebook:

We can of course ask for the linear regression summary:

%%R
my_summary

Which outputs:

Call:
lm(formula = log(Y) ~ X)

Residuals:
      Min        1Q    Median        3Q       Max 
-0.183663 -0.013497 -0.004137  0.006046  0.296094 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.775851   0.026173  -29.64   <2e-16 ***
X            0.479757   0.001524  314.84   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.06866 on 27 degrees of freedom
Multiple R-squared:  0.9997,	Adjusted R-squared:  0.9997 
F-statistic: 9.912e+04 on 1 and 27 DF,  p-value: < 2.2e-16

We can also lift the results from R to Python using %Rget:

coefs = %Rget val
y0,k = coefs.T[0:2]
y0,k

Which yields

(-0.77585097534858738, 0.4797570904348315)

Here we saw that RPy2 allows us to pass data back and forth between Python and R; This is incredibly useful to leverage the strengths of each language. This is a toy example, but you could imagine using various python libraries to get data from servers, and move to R for the statistical analysis.

However, sometime moving data between languages may be too limiting. Let’s see how we can leverage the same mechanism to gain some performance, by integrating with a lower level language.

Let’s C

Python and R are not the most performant languages for pure numerical speed. When performance improvement is necessary, developers tend to utilize compiled language like C/C++/Fortran.

Unfortunately, compiled languages generally have a poor interactive experience, and where CPU cycles are gained, human developer time may be lost.

Using magics, we can, as we did for R, include snippets of C, Cython, Fortran, Rust … and many other languages.

import cffi_magic
%%cffi int cfib(int);

int cfib(int n)
{
    int res=0;
    if (n <= 2){  
        res = 1;
    } else {
        res = cfib(n-1)+cfib(n-2);
    }
    return res;
}

We can interactively redefine this function, and it will magically appear on the Python namespace. It works identically to the fib we defined earlier, but is much faster. Note that the Python Fib, and C fib time here are difficult to compare as the C one is recursive (behave in O(exp(n))) and the Python one is hand unrolled, so behave in O(n).

More technical details can be found in a notebook I wrote earlier, but the same can be done with other languages that call one another, and lines like the following work perfectly:

assert py_fib(cython_fib(c_fib(fortran_fib(rust_fib(5)))) == 5

Julia to bind them all

The last example is a technical marvel that was first developed by Steven Johnson and Fernando Pérez, it relies on starting a Julia and Python interpreter together, allowing them to share memory. This allow both languages not only to exchange data and functions, but to manipulate live object references from the other interpreter. Extra integration with IPython via magics allows us to run inline Julia in Python (%julia, %%julia), while Julia Macros (@pyimport) allows python code to be run from within Julia.

Below we’ll show integration with Graphing libraries (matplotlib), so let’s set up our environment.

%matplotlib inline
%load_ext julia.magic

We’ll start from within Julia, and import a few python packages:

%julia @pyimport matplotlib.pyplot as plt
%julia @pyimport numpy as np

We now have access – from within Julia – to matplotlib and numpy. We can now seamlessly integrate Julia native numerical capabilities and functions with our Python kernel.

%%julia                                        
t = linspace(0, 2*pi,1000);             
s = sin(3*t + 4*np.cos(2*t));           
fig = plt.gcf()                         
plt.plot(t, s, color="red", linewidth=2.0, linestyle="--", label="sin(3t+4.cos(2t))")

Note that in above block, t, pi are native julia; s is computed via sin (julia), t (julia), cos (numpy); fig is a Python object. As the Julia Magic provides IPython display integration, the code above displays this nice graph.

We now want to annotate this graph from Python, as the API is more convenient:

import numpy as np
fig = %julia fig
fig.axes[0].plot(X[:6], np.log(Y[:6]), '--', label='fib')
fig.axes[0].set_title('A weird Julia function and Fib')
fig.axes[0].legend()
fig

After passing a reference to fig from Julia to Python, we can annotate it (and plot one of the fib functions we defined earlier in C, Fortran, Rust, etc…)

Here we can see that unlike BeakerX, R-Reticular or RPy2, we are actually sharing live objects, and can manipulate them from both languages. But let’s push things a bit further.

The fib function can be defined recursively; let’s have some fun and define a pyfib function in Python that recurses via the a jlfib function in Julia. Meanwhile, the jlfib function in Julia recurses using the python function. We’ll print (J, or (P when switching language:

jlfib = %julia _fib(n, pyfib) = n <= 2 ? 1 : pyfib(n-1, _fib) + pyfib(n-2, _fib)

def pyfib(n, _fib):
    print('(P', end='')
    if n <= 2:
         r = 1
    else:
        print('(J', end='')
        # here we tell julia (_fib) to recurse using Python
        r =  _fib(n-1, pyfib) + _fib(n-2, pyfib)
        print(')',end='')
    print(')',end='')
    return r

fibonacci = lambda x: pyfib(x, jlfib)
fibonacci(10)

We can now transparently call the function :

(P(J(P(J(P(J(P(J(P)(P)))(P(J))(P(J))(P)))(P(J(P(J))(P)(P)(P)))(P(J(P(J))(P)(P)(P)))(P(J(P)(P)))))(P(J(P(J(P(J))(P)(P)(P)))(P(J(P)(P)))(P(J(P)(P)))(P(J))))(P(J(P(J(P(J))(P)(P)(P)))(P(J(P)(P)))(P(J(P)(P)))(P(J))))(P(J(P(J(P)(P)))(P(J))(P(J))(P)))))
55

If you are interested in diving more into details see this post from a couple of years ago with all the actual code.

I hope that this post has convinced you that Jupyter – via the IPython kernel – has deep cross-language integration (and has had this for many years). I also hope it lifted the misconception in that in Jupyter “1 kernel == 1 language” or even that “1 notebook == 1 language”. Each of the approaches shown here (as well as Reticulate, BeakerX, etc) have their pros and cons. Use the approach that fits your needs and makes your workflow efficient, regardless of the tool, language, or libraries you use.

JupyterCon 2018: Registration Open

2018-04-11T15:01:00+00:00

Dear Jupyter Community,

For the past six months, the Project Jupyter team in collaboration with O’Reilly Media and NumFOCUS have been planning JupyterCon 2018. In January, we opened the Call For Proposal, during which we received numerous high-quality proposals. The total submissions exceed our expectations: more than 3 times the number of available slots! With the help of the Program Committee Reviewers and co-chairs Fernando Pérez, Brian Granger and Paco Nathan, we had the hard job of selecting among the fantastic submissions we received. Today we are happy to announce that most of the JupyterCon 2018 Program is ready and registration is open! We are exited to bring you sessions about Scaling JupyterHub, leveraging GPUs for Jupyter, Running C++, in Jupyter, and many more.

Fernando Pérez and Andrew Odewahn during JupyterCon 2017 Opening Keynote

As Paco Nathan previously announced, this year will have a dedicated Education Track, and Business Summit to supplement the Main Tracks, Trainings, and Tutorials that we already brought to you last year. Many of the highlights of last year such as “Meet the Experts” office hours, a Poster Session for extended discussions with presenters, and the Vendor Expo Hall will return this year.

As with last year, JupyterCon will be held at the New York Hilton Midtown, NYC, August 21-24 and Saturday 25th. You can register today for the main conference. Early Bird pricing ends on May 18th. You can also use the discount code PJ20. We also have a limited amount of financial support for JupyterCon for attendees thanks to our partners. Last year, we provided scholarships to 13 students from diverse backgrounds to attend JupyterCon 2017.

Community Sprint Day, August 25th.

Thanks to Bloomberg, Saturday, August 25th will be reserved for a separate Community Sprint day, free of charge. All community members, whether or not you plan to attend the main conference, are invited. This day will be focused on community, contributing to Jupyter, and Open Source in a “Open Studio” form. Whether you are new to Jupyter or a power user, we invite you to come and mingle with the rest of the attendees to lean about any Jupyter-related project.

Several activities will be available. Whether you have coding, design, or writing skills, we encourage you to contribute, pitch your ideas, and get started on something brand new.

Are you new to open source, Git, and GitHub? We’ll offer a introduction to Open-Source 101 and how to get development versions on your machine.

You’ve never tried Jupyter or you are an advanced user of Jupyter with specific needs? We’ll be hosting a JupyterLab user-testing session where you will have the chance to try upcoming features and give us critical insight on how to improve usability.

Interested in contributing back to Jupyter or related projects? Many experts will be here to help you move your project forward. If you are coming to JupyterCon and would like to help with Community Sprint Day, or have a project you’d like attendees to work on let us know !

The Community Sprint Day is free to attend, but registration is required.

Jupyter Pop-Up, DC, May 15th

You can’t wait to attend JupyterCon 2018 ? You can attend Jupyter Pop-Up, DC, May 15th for a Day long event, and a taste of what is to come !

Do not forget to follow this blog, @projectJupyter or @jupytercon, for further updates on JupyterCon.

Thanks to The O’Reilly Media Team; The Conference Chairs (Brian Granger, Paco Nathan and Fernando Pérez); The Program Committee (Dan Allan, Ian Allison, Paige Bailey, Lorena Barba, Tom Caswell, Afshin Darian, John Detlefs, Chris Erdmann, Jessica Forde, Stuart Geiger, Tim George, Michelle Gill, Tim Head, Jennifer Klay, Cierra Martinez, Emiliy Jane McTavish, Omoju Miller, M Pacer, Peter Parente, Eszti Schoell, Steve Silvester, Robert Talbert, Dwight Townsend, Wolf Vollprecht, Jamie Whitacre, Kevin Zielnicki) and all the people making JupyterCon 2018 possible, and Jupyter a reality.

Jupyter Pop-Up, March 21, Boston

2018-02-09T22:20:00+00:00

Jupyter Pop-Up, brought to you by NumFOCUS and O’Reilly Media, March 21, Boston, MA.

Many of you are looking forward to JupyterCon 2018 and have submitted a talk. Alongside these large, multi-day events, we are seeing demand for smaller, local events as well. Since 2015 we have co-organized several Jupyter Days events (Paris, Hawaii, Atlanta, Boston, Philadelphia, NYC) with local community organizers. In 2018, we are preparing Jupyter Day Atlanta (March 31st), and hoping to offer other community organized events as well.

In addition to Jupyter Days, we are pleased to announce the first Jupyter Pop-Up, which is brought to you by the NumFOCUS Foundation and O’Reilly Media.

This single-day conference will take place in the District Hall of Boston, MA on March 21, Early Bird ticket prices end on February 23rd, and the Call For Proposals is still open until the end of the week (Sunday, Feb. 11th).

You can learn more on the Jupyter Pop-Up conference website.

Looking forward to seeing you there !

JupyterCon 2018: Call For Proposal

2018-01-18T22:53:00+00:00

Dear fellow Jovyans,

It is with great pleasure that we are opening the Call For Proposals (CFP) for JupyterCon 2018!

Last August, Project Jupyter, the NumFOCUS Foundation, and O’Reilly Media came together to host our first JupyterCon. We attracted over 700 attendees and 23 scholarship recipients for 4 days of talks and tutorials. There were 5 parallel session tracks featuring 55 talks, 11 keynotes, 55 talks, 8 tutorials, and 2 training courses. In addition, the conference poster session featured 33 posters and fostered great discussions within the community. Our Community Day, held at at the end of the conference featured free registration was open to the general public. Videos of the event have been made available on Safari Online and YouTube.

JupyterCon 2018, CFP Open

JupyterCon 2017 was a huge success and we’ve been working hard since then to make JupyterCon 2018 even better. It will be held in New York City in August from Tuesday the 21st to Friday the 24th. We’ll also host an open Community Day on August 25th, which will be open to everyone.

Today we are happy to open the conference website and open the Call For Proposal with submissions due by early March. A couple of changes have been made to the CFP since last year. In particular if your talk is not accepted, you can ask us to automatically consider the proposal for the poster session.

We encourage you to submit a proposal, and reach out to us if you have any questions. We’ll do our best to help you and and give you feedback on your proposal.

Like last year, we will have diversity and student scholarships available; further information will be provided on the website. We also encourage you to follow the JupyterCon Twitter account for announcements or corrections.

Community Day

The final day of JupyterCon 2017 was a blast with a large number of people making their first contribution to the Jupyter codebase, to the documentation, editing the wiki, or deploying it in the cloud. During the conference days, a separate room was also reserved for user testing of different Jupyter software, which proved to be fantastic source of feedback for User Experience (UX) and driving various Jupyter Tools forward.

We are happy to offer this “Community Day” experience again. At JupyterCon 2017, the Saturday was branded “Sprints” with the connotation of a code-centric experience. While we’re happy to see users coming to “Sprint” on code, we want to let you know that the Community Day will be open to anyone. Whether you are a teacher, coder, researcher, or user of Jupyter, the Community Day will have something for you. The Community Day is not limited to attendees of the main JupyterCon event, and it’s intended to be a “grass-roots” celebration of Jupyter and its community. We hope to see you at JupyterCon 2018.

Thanks

JupyterCon 2018 would not be possible without O’Reilly Media, NumFocus, as well as our sponsors.

Incident Report: Jupyter services down

2017-12-14T19:52:00+00:00

update: December 14, 20:45 UTC, all services should be restored and back up.

On December 13, at 22:10 UTC (4:10pm EST), a large number of Jupyter-provided services stopped responding. This included, but was not limited to https://nbviewer.jupyter.org, https://try.jupyter.org (powered by tmpnb) and https://cdn.jupyter.org. We quickly narrowed this down to an issue with our hosting provider and have been working with them to resolve the issue as fast as possible.

When outages happen, the Jupyter Status page should show which services are affected and we publish updates there.

How are Jupyter services hosted?

To understand the cause of the outage, we need to understand how the Jupyter services are hosted and maintained. As Jupyter is an open organization which is mostly maintained by volunteers, we do not have a dev-ops team assigned to maintaining our infrastructure. Even with full-time developers hired through universities or companies, the time spent fixing infrastructure is taken on nights and weekends. These developers are often stretched thin and cannot be available 24/7.

Most of our cloud infrastructure is donated to us by companies like CloudFlare, Rackspace, Fastly, Google, and Microsoft. Donating resources can be challenging, both technically and legally. In this particular case, Rackspace graciously created a special account for Jupyter that handles invoices on our behalf, thereby making resources free to the project. Following a hiccup, this Jupyter account was suspended and all services are unavailable as a result.

Temporary resolution

As nbviewer is one of the most used services provided by Jupyter, we’ve moved it to one of our personal account at another cloud-provider. Fastly was set up to load-balance on the yet-to-come-back-up instances as well as this newly created instance, so all should be fine now.

The other services (tmpnb, mails@jupyter.org, cdn.jupyter.org, …) will still unavailable or highly degraded until a permanent solution is found, or the services are restarted. try.jupyter.org will likely redirect to a repo on https://mybinder.org in the meantime so people can still try out Jupyter.

Low bus factor

The outage of all these services lasted for a significant time (more than 18 hours). Which perturbed many of you relying on these services. We understand that this is hardly acceptable and we hope you’ll indulge us as these services are provided for free and without ads. One of the factors leading to the slow reestablishment of service was a relatively low bus factor, with only one and a half of our developers knowing how to deploy and maintain these services. Documentation and access to credentials was also limited.

This is one of the challenges in a distributed team like Jupyter where contributors self-organize. It is easy to forget that new code is not the only way to contribute and that infrastructure and maintenance are crucial.

We also overly rely on a single vendor (in this case Rackspace), and while we are happy with Rackspace and have no reason to move to another provider, we should have a plan to restore critical services even temporarily in case of failure.

A couple of months ago, the subject was brought to our attention, and we developed a plan to move many of our deployment to Kubernetes (which is provider agnostic). We underestimated the probability to need an emergency plan this early.

How can you help

Jupyter is mainly governed by the community all around the world. Contributing is not limited to writing code! We need members with knowledge in multiple languages, in design, dev-ops, etc. Whether you are an expert, or still learning, we would like you to get involved.

Thanks everyone for your patience and the kind words when you reached to us when discovering the services were down.

Enjoy JupyterCon 2017 on YouTube

2017-11-24T10:09:00+00:00

About a year ago the Jupyter team started a partnership with O’Reilly Media and NumFOCUS to organize JupyterCon 2017, the first in a series of Jupyter-related conferences. JupyterCon 2017 drew over 700 attendees and offered a fantastic set of tutorials, talks and keynotes from all corners of the Jupyter ecosystem. We could not have offered you such a beautiful conference without the O’Reilly Media and NumFOCUS partnership and teams.

Did you miss JupyterCon 2017?

We have a strong commitment to open-source and we believe that everyone should have access to high-quality tools and content to push science, education and technology forward. O’Reilly Media is one of the top publishers in this space, and they already provide a variety of material and training related to Jupyter and the open-source data ecosystem of which it is a part (see O’Reilly’s Jupyter related content here). It is no surprise then that the full content of JupyterCon 2017, including keynotes, tutorials, talks and exclusive interviews, is available as videos on Safari Online.

We are also pleased to announce that videos of all keynotes and talks (69 videos in total) are now available on YouTube on the O’Reilly Media channel. If you missed JupyterCon, or want to re-watch a talk, you can now add these to your watch-list! As an example here is the opening keynote given by Fernando Pérez:

See you next August for JupyterCon 2018

We are thrilled to be able to provide you with all this content from JupyterCon 2017 to enjoy on YouTube. JupyterCon 2018 will be in NYC next August 22–24. We hope to see you there next year as an attendee, speaker or sponsor. You can sign up for email updates about JupyterCon 2018 on this page.

[Update Jan 2018] Missing Talks

The Talk “Jupyter and the changing rituals around computation” is missing from the channel due to technical issue during the recording.

Release of IPython 5.5 and 6.2

2017-09-15T18:42:00+00:00

Four month after releasing IPython 6.1 and 5.4, and a couple of hours after the release of the notebook 5.1, we are happy to announce the release of IPython 6.2 (Python 3 only), and it’s cousin IPython 5.5 still compatible with Python 2.7.

You can update now by using:

pip install --upgrade ipython

If you have a recent enough version of pip you will get the latest compatible version of IPython regardless of the version of Python you are running.

The conda packages are on their way; once available you will be able to update with:

conda install ipython

New Features

As IPython 6.2 and 5.5 are minor releases you will only find a small number of new features. When API additions were done on IPython 6.2 they were backported on 5.5 to simplify the maintenance of code compatible both with Python 2.7 and 3+. You can find the full list of new features in the changelog.

As a quick teaser, IPython 6.2 can now:

Show function signature in the terminal while completing.
Assignments can trigger the display mechanism
IPdb can be called recursively
Support for system-wide configuration
Built-in support for Progress Bar.

Head there for more complete description.

Slowing down backports to 5.x

As stated on our roadmap, we’ll keep releasing a 5.x for some time, though starting at end of year. However, we will decrease our active involvement in fixing bugs affecting the 5.x branch. We will still accept PRs, and backport if you nicely ask us. Releases will happen occasionally if fixes are available, but we will be sunsetting the Python 2 support slowly.

If you are interested in further maintenance of the 5.x branch, we would love help with that work. Feel free to contact us on GitHub.

What’s next ?

We are going to start thinking about IPython 7, and start to embrace more of the Python 3 only features. Slowing down backports should allow us to be more confident that changes will not affect the automatic application of patches on old branches. Trimming down old legacy code may also help to regain some speed on interpreter startup, and should lead to plenty of opportunities for new contributors to join.

We will also try to simplify our documentation, and make often requested sections easier to find.

If you are looking for a project to contribute to – code, documentation, example, design, helping others, feel free to contact us so we can guide you through the process.

Enjoy this new release, and hope to see you around the mailing list and bug tracker!

JupyterCon NYC: August 22nd-25th

2017-07-14T18:56:00+00:00

Dear fellow Jovyans,

We’re just weeks away our first Jupyter community conference, JupyterCon. It will take place from August 22nd to 25th (and Sprints 26th), in the beautiful city of New York, at the Hilton Midtown, a spectacular location just steps from Central Park, Times Square, MOMA.

If you haven’t registered yet there’s still time. There are a number of pass options to choose from including 2, 3, or 4 day passes. Discounts are available for students, academic instructors, and government and non-profit employees. Don’t qualify for any of those? We have a special 20% discount for you, just use the code JUPCORE20 when you register.

What is JupyterCon?

The four day event will feature two days of training and tutorials and two days of Keynotes and Sessions. Topics include the Jupyter platform’s core architecture, kernels, extensions & customizations, usage and application of Jupyter Notebooks, as well as sessions on Jupyter’s development and community from the core Jupyter team.

We’ve left plenty of time for networking, including attendee receptions, Speed Networking, Poster Sessions, community group meetups, as well as the chance to meet some of the speakers in small group settings. Check out the event page. The core Jupyter team will also be present and ready to answer your trickiest questions about the Jupyter platform and what’s in-store for the future.

The preconference starts on Monday 21st with a Solar Eclipse from 1:23pm to 4:00 pm in NYC.

A fantastic line-up of speakers

JupyterCon is chaired by Fernando Pérez, creator and BDFL of Jupyter, and Andrew Odewahn, CTO of O’Reilly.

Some of the speakers joining us at JupyterCon include:

Lorena Barba from the George Washington University
Nadia Eghbal from GitHub
Wes McKinney from Two Sigma Investment
Safia Abdalla from the nteract project
Rachel Thomas from Fast.ai
Brett Cannon from Microsoft / the Python Software Fundation

These are only a few of the speakers joining us. We are looking forward to hearing how Jupyter is being used in education, finance, machine learning, and what the future holds for the Jupyter ecosystem. See the full line-up here.

Sprints

Sprints are happening on Saturday 26th, if you want to come hack on Jupyter and related project get more information on the GitHub JupyterCon repository. You do not need to have registered to the main conference — but you must registered via Eventbrite even if you have already register for the main conference.

Financial aid

We had a number of really good applicants for financial help, and the selection process was tough. If you have not been selected this time, don’t be discouraged and we hope to see you at the next JupyterCon.

BOFS

We will have a special “Jupyter for Teaching & Learning BOF“ organized by Lorena Barba And Robert Talbert on Thursday at 7pm. This BOF is for anyone interested in using Jupyter for teaching and learning. Topics for discussion include incorporating Jupyter in the classroom, using Jupyter tools like nbgrader and JupyterHub, connecting with other Jupyter educators, and more. For more information and to let us know you’re interested in participating, please see this flyer and fill out the Call for Participation form at http://bit.ly/jupyter-ed-bof.

Solar Eclipse

If you are coming from outside of the United States please remember to be in NYC on the 21st from 1:23pm as there is a (partial) solar eclipse in NYC, which ends at 4:00.pm. Do not forget your eclipse glasses!

Thanks

This is our first JupyterCon! We do welcome feedback and will be looking for help to organize another one next year. Please contact us if you are interested in helping organizing the next conference.

We’ve partnered with O’Reilly Media to develop the JupyterCon conference. O’Reilly Media is a long-time supporter of the project and active publishers in the Python/Data Science space. O’Reilly has extensive experience running conferences and we’ve been working with them for the past year to bring you a great inaugural JupyterCon.

I also want to thank the members of the Jupyter Con committee as well as the Jupyter Community Members and the team at O’Reilly Media, and all our sponsors for making this conference possible.

You can learn more on the conference website

Release of IPython 5.4, 6.1, and rlipython

2017-05-31T22:47:00+00:00

Slightly over a month ago we released IPython 6.0: the first version of IPython to be compatible only with Python 3. The reception of IPython 6.0 was great, and the codebase way more maintainable. So while 6.0 was mostly focused on internal refactoring and working upstream to make the release of a Python 3 only version possible, the next releases can focus on bug fixes and new features. Today we not only have the new release of IPython 6.1, but IPython 5.4 as well (the LTS branch targeting Python 2.7). Read on for more information! As usual, you can upgrade using pip and the 🔥 Conda Forge 🔥 builder is warming up: IPython 5.4, and 6.1 should be available via these channels soon.

IPython 5.4 (and 6.1) major new features

IPython 5.x is the LTS branch which is still compatible with Python 2. The IPython team ♥️ Python 2 users so we continue to make fixes to our 5.x branch. You can install it today by making sure you have pip 9 or above:

$ pip --version
pip 9.0.1

Then upgrade IPython:

$ pip install ipython --upgrade

If you do not have pip 9 and are using Python 2, pip may download IPython 6, which will not work on your system. If you are on Python 3, see the next section to upgrade to IPython 6.1 but read on in this section for the new features.

IPython 5.4 got a number of new features that went into 6.0, plus a number of API improvements from 6.1 allowing the underlying library to target both 5.4+ (Python 2+3) and 6.1+ (Python 3) without having conditional branches. This should make the transition easier for users and developers

You can see the what’s new for:

`TerminalIPythonApp.interactive_shell_class`

In 6.0 and 5.4 IPython gained a new c.TerminalIPythonApp.interactive_shell_class option, to customize the class used to start the terminal frontend. This should enable you to use custom interfaces, such as reviving the former readline interface which is now a separate package. See the next section for more.

Define `_repr_mimebundle_`

In IPython 6.1+ and 5.4+ object can now define _repr_mimebundle_ in place of multiple _repr_*_ methods and return a dictionary of MIME types and MIME data for rich display. IPython provides _ipython_display_ but this mechanism was complex and difficult to use for frontends with different architectures, such as nteract and hydrogen. This new _repr_mimebundlde_ methods are thus simpler for both backend and frontends developers to implement, and are already handled in projects like nteract. Our own JupyterLab, the next-generation evolution of the Jupyter Notebook, takes full advantage of these capabilities to expose rich and flexible data display capabilities that work not only in notebook documents, but across all of JupyterLab.

This new model will simplify things for client implementations and will facilitate the publication of custom mimetypes (GeoJson, Plotly, DataFrames, etc.). See the Custom Display Logic example notebook for examples and technical details.

Execution Heuristics

In IPython 5.4+ and 6.1 +, the heuristic for execution in the command line interface is now more biased toward executing for single statement. Use Ctrl-O to force the insertion of a new line. The execution semantics of Enter were still debated, so if you like to configure that you can define your own function which decides what to do.

Display IDs

IPython 5.4 and 6.1+ implement display ids, which can be used to update a displayed object. This should provide simple patterns to implement common cases of dynamic display updates (see Updating Displays.ipynb notebook). It will also simplify the implementation of some types of interactive user interface elements, such as progress-bars. Display objects with IDs can be updated from other cells or frontends.

The display function is now always available

The display() function is now available by default in the IPython session, without users having to import anything. display() is the "moral equivalent” of print(), but takes advantage of the rich display capabilities of Jupyter (and defaults to printing a string for objects lacking rich representations). By making display() appear, in practice, as if it was built-in, we hope to encourage library authors to provide their objects with rich and informative representations that users can now more conveniently access, whether at the IPython REPL or in any of the richer clients in the Jupyter ecosystem (JupyterLab, classic Notebook, nteract, etc.). Note that this automatic import of display() is only done in the live IPython kernel; scripts and libraries that rely on display and may be run outside of IPython still need to import it using from IPython.display import display.

Rlipython

Many users were pleased with the transition from readline to prompt_toolkit for IPython 5.0, this gave IPython multiline editing, nice completion and highlighting as you type. Still some users are missing the old readline interface (for its reactiveness an custom keybindings) though are still using IPython 4.x series.

While prompt toolkit is still the default interface, you can now use the new configuration option c.TerminalIPythonApp.interactive_shell_class .With this you can now replace the Prompt_Toolkit interface. For one of your liking.

Rlipython is one of these alternative interface that resurect readline and is lighter, and more responsive on some system than default IPython. Using it still give you most of the new features of IPython 5.x or 6.x ! This should also please many of our users who use vi keybinding in terminals as this should respect .inputrc.

To try rlipython, install it with pip,

$ pip install rlipython

And follow the instructions.

We look forward to your contribution and please give lots of 💖 by to @ivanov (tweet) for making this possible.

IPython 6.1 and Python 3 only features

IPython 6.1 is the first minor release of our Python 3 only branch, it is the branch that get the most work, and start to use great new features such as type annotations, async/await [more to be added]. You can install it today please upgrade pip to pip 9 if you can, then upgrade IPython:

$ pip install ipython --upgrade

You’ll get all the new features in 5.4, some of which are already in IPython 6.0, plus a number of additional enhancements:

You can read more on the what’s new for IPython 6 what’s new.

In this release, we had a number of bug fixes and improvements from developers at TwoSigma, who participated in an internal open-source hackathon for Jupyter/IPython in May. Hackathon attendees tackled some long standing, hard to debug issues, and feature requests. Among these:

Quotes in a filename are always escaped during tab-completion on non-Windows, which should make many of our windows users happy. As none of the core devs use windows this one needed someone to simply use some elbow grease!
Variables now shadow magics in autocompletion which make autocompletion of things like matplotlib simpler and less surprising.
Magic aliases can now have parameters.
?? /pinfo2 machinery will show docstrings if source can’t be retrieved.

The completer also got a number of small improvements, and is smarter in many cases. For example, it understands the arguments to the %config and %colors magics. Long methods and snake_case function names can now be completed from abbreviations. For example, if foo_bar_kitten is an existing object, f_b<tab> will find it as a possible completion.

Look out for a blog post from TwoSigma.

Only a small number of improvements are describe here, and we encourage you to read the full change logs if you want to know more.

We hope that all these releases will make you more productive. We welcome your contributions, and questions. Feel free to open issues on GitHub, we’ll do our best to respond.

Release of IPython 6.0

2017-04-19T22:37:00+00:00

It is with great pleasure that today we released IPython 6.0 — almost a year after the 5.0 version.
Users on Python 3.3 and above can get this latest version with all its new features by asking your package manager to upgrade IPython. If you are using pip:

Ensure you have pip 9+:

pip --version

if not upgrade pip:

pip install pip --upgrade

Then upgrade IPython.

pip install ipython --upgrade

or if you have conda:

conda update ipython

It might take a few hours/days for the various conda channels to be updated.

The What’s New page has an extensive description of the changes, which includes new, modified and deprecated features.

Highlighted feature: Jedi completion

One of the biggest changes is integration with Jedi, which provides completions using static analysis, and can display completion types. This will provide a foundation for a richer tab completion experience (e.g., it allows extracting and displaying function signatures in the completer). As of now, these features are available only for command line users, but work is underway in IPykernel and in various frontends (like nteract) to make use of these features.

Sunsetting Python 2 support

As we announced last year when we released IPython 5.0, IPython 6 and above require Python 3. The 5.x branch is still maintained and will be regularly released, so Python 2 users will still see updates, but new features in IPython 6.x will not be backported.

We have worked to make installing IPython on Python 2 and 3 as seamless as possible, but you may still encounter errors. If you do, please check the readme and if you still have a problem, open an issue on Github.

If you want to hear more about the process of creating a seamless Python 2 sunset and are planning on attending PyCon 2017, we encourage you to attend our talk “Ending Py2/Py3 compatibility in a user friendly manner” on Saturday, May 20, 2017 at 15:15–16:00 PDT.

Make sure you have Pip 9 or above

For most end users the only thing you should pay attention is to have pip version 9, or above. To check your pip version, run:

pip --version

If you have a version of pip older than 9.0.1, you should upgrade pip to its latest version before upgrading IPython:

pip install pip --upgrade

Pip 9+ will install the latest compatible version of IPython – that is to say IPython 5.x if you are on Python 2, and IPython 6.x if you are on Python 3. Make sure this is the case for all your Python environments, and build scripts. Users of Ubuntu Trusty 14.04 LTS should be aware that the default pip is too old and will not install the correct version of IPython. Don’t forget to upgrade pip in your various install script, continuous integration settings and docker images.

The Python 3 statement

We care deeply about the seamless coexistence of Python 2 and Python 3 software. This is one of the reasons we started this transition with a new release that is only available for Python 3, while still maintaining the 5.x release series, which is compatible with Python 2. If you want to read more about these reasons you can find more information at the python 3 statement.

As expected, we discovered many challenges and pitfalls as we moved from a single source to a Python3-only source. In the course of overcoming these obstacles, we learned a great deal. We have documented the the risks and errors you can encounter (and their solutions) at the practicalities section of the python 3 statement.

If you are a library maintainer planning to stop support for an older version of Python (even a minor version), and you want to limit the frustration faced by users who use your library on the older version of Python, we could use your help! If you want to add to the resources available for trouble-shooting this transition, please join the conversation on our GitHub repository.

Migrating to Python3-only source

The migration of the IPython source code to be Python3-only was not an easy task, and was – as far as we can tell – unexplored territory. But we’ve bitten the bullet and hopefully anticipated most of the problems that can arise. With this, we hope to make the transition easier for other projects who want to follow our lead.

First, let’s speak about our personal experience writing Python3-only source code.

The size of the IPython codebase has decreased by about 1500 lines of Python code relative to the last release. Of course, that’s not solely due to the removal of Python 2 support, but a non-negligible amount is. And this reduction is even more remarkable in light of completely new features that required adding hundreds of lines of code. A large number of conditionals are gone, making the code more straightforward, easier to read, and simpler to maintain. Across the codebase we saw improvements in code compactness just from removing utility functions that existed only to provide identical behavior across Python 2 and Python 3. Even then, parts of the codebase remain affected by “Python 2 code”, so the potential gains are even greater. We will continue our quest to remove things as we go, and, as usual, contributions are welcome.

This change eases the burden on contributors to IPython. Contributors can can spend less time thinking “what about Python 2”, or rewriting a pull request because the Python 2 test suite fails. At the same time, our tests now complete more quickly on continuous integration services because they need to run on fewer versions of Python.

A couple of new APIs are using type annotations, Python3-only syntax which, properly used, make the code clearer and easier to refactor, and allow the documentation to focus on usage / reason than to describe the types of the function. We can also make use of nifty Python 3 features like keyword-only arguments, which is definitely appreciated when designing APIs.

From a developer point of view we are extremely pleased with having the possibility to write Python3-only code, and are looking forward to even more improvements like pathlib.

Keeping a Python 2 stable branch

Keeping a Python 2 stable branch, and making the install experience as seamless as possible for Python 2 users was a key point in our decision to migrate to requiring Python 3. One of our first questions was how could we make sure that Python 2 users using pip install would get the latest 5.x version of IPython still compatible with their system (and not the latest >6.x version that would not be compatible)? This lead us to a number of upstream patches (we’re giving a talk at PyCon, come see us) and a number of guides as to how to make the transitions as a seamless as possible. We patched pip, pypi-legacy, warehouse (we had to learn postgres for that one), amended peps and made use of recent improvements in setuptools to allow Python users to peacefully coexist. The only thing the user should need to do is to make sure to use pip 9+, and to use pip install . and pip install -e . instead of invoking setup.py directly.

Migrating more code to Python3-only

IPython is the first of the Python packages we maintain which has been migrated from Python 2+3 compatibility to requiring Python 3. Some of our more recent projects (like JupyterHub) have required Python 3 from their beginnings. We expect more Jupyter & IPython projects will move to requiring Python 3 in the future.

As the Jupyter protocol is language agnostic, it is perfectly possible to run Python 2 code in a Python 2 kernel using a Notebook Server running on Python 3. Especially since JupyterHub already requires Python 3, we envisage that the single-user notebook server will follow suit soon.

Conclusion

We hope you will enjoy this release. It will be the base for some awesome features, like async/await REPL. We hope our work in the packaging ecosystem will minimise the inevitable teething difficulties from our first release to require Python 3. We look forward for your feedback.

HN and reddit

Release of IPython 5.0

2016-07-08T06:56:00+00:00

IPython 5.0-LTS is out!

We are pleased to announce the release of IPython 5.0 LTS (or Long Term Support). IPython is the Python kernel for Jupyter and the interactive Python shell; it provides a rich set of features for fluid interactive computation in Python at the terminal, in the Jupyter Notebook and across all other clients that support the Jupyter architecture.

This release has some exciting new features and lots of new development (227 commits by 27 contributors over 191 PRs). Most importantly, there have been significant improvements to the classic IPython command line interface.

As usual you can try this new release with:

pip install ipython --upgrade

The package should also be available through conda and other package managers in the next few days.

Note: IPython is now developed under the umbrella of the broader Project Jupyter, but like other components of Jupyter, with its own independent schedule. Therefore, this release does not impact the Jupyter Notebook, Qt Console, nbconvert, or other packages that were formerly part of IPython.

A brand new terminal interface

Decoupling IPython from the Jupyter Notebook package has allowed the core team to focus on improving the command line interface independently of the notebook. The awkward dependencies on pyreadline for Windows and gnureadline for Mac prompted Thomas Kluyver to replace the old machinery with a brand new pure-python readline replacement: prompt_toolkit.

The prompt_toolkit package is an amazing library from Jonathan Slenders and recently reached version 1.0. Going beyond readline, prompt_toolkit provides many advanced features for editing text in the terminal that significantly improve the user experience. Since it is a cross-platform library, all our users on Linux/Unix, macOS and Windows benefit from these improvements. Thanks to prompt_toolkit, IPython now supports:

Syntax highlighting as you type
Real multi-line editing (up and down arrow keys move between lines)
Multi-line paste without breaking indentation or immediately executing code
Better code completion interface (we plan to improve that more)
Optional mouse support

We’re not using all of the features of prompt_toolkit yet, but after working with it for a few weeks, it already feels strange to go back to older versions of IPython without these improvements. We are hopeful that you will enjoy them. We’re extremely grateful to Jonathan Slenders, who has been super responsive with all our questions and feature requests!

You can get a more detailed list of changes to expect by reading the “What’s new in IPython 5.0” document.

Jupyter Console

The Jupyter Console provides the interactive client-side experience of IPython at the terminal, but with the ability to connect to any Jupyter kernel instead of only to IPython. This lets you test any Jupyter Kernel you may have installed at the terminal, without needing to fire up a full-blown Notebook for it. The Jupyter console gained also most of the functionality described above and also makes use of prompt_toolkit.

A few days ago we released Jupyter Console 5.0 as well, which brings compatibility with IPython 5. If you are a Jupyter Console user you will need to upgrade as well.

$ pip install jupyter_console --upgrade

Long Term Support (LTS)

You might have picked this up from the title of this blog post: IPython 5.x will be the first release series to get Long Term Support (hence the LTS name).

With IPython, we usually only offer support for a single major release at a time; once a new major release comes out, previous major releases stop getting bug fixes. For the 5.x series releases we are making an exception to that rule: until the end of 2017 the core team will do its best to provide fixes for critical bugs in the 5.x release series. Beyond that, we will deprioritise this work, but we will continue to accept pull requests from the community to fix bugs through 2018 and 2019, and make releases when necessary.

We hope that this will help organisations that need long term support for IPython version 5.x.

End of support for Python 2

IPython has been compatible with Python 3 for several years, since Thomas Kluyver ported the codebase to be Python 3 compatible using 2to3 in 2011, and we moved to a single-source codebase for Python 2 and 3 in 2013. The day to day development of IPython is now completely done using Python 3, and we’re starting to accidentally break Python 2 compatibility until tests or users flag it. We’re also keen to use many of the new Python 3 features, such as type annotation, yield from, asyncio, async def, await and other improvements the language and its standard library have gained in recent years.

We have therefore decided that IPython 5.x will be the last major version to support Python 2.

This is, of course, why we are planning to support IPython 5.x for much longer than usual. We recognise that many people still use Python 2, and they will be able to continue with a supported version of IPython for several years, and transition at a time that suits them. Beyond the end of 2017 we are willing to provide minor bug fix releases in the 5.x with community contributed patches. Most importantly, no new features will be added to a Python 2 supported IPython beyond the upcoming 5.0 release.

Thus, the next major version of IPython, IPython 6.x will require Python 3. It will start to make use of new syntax, and shed the compatibility layer we have in place.

If you are a Python 2 user, be reassured, we will make sure that upgrading does not unexpectedly install IPython 6.x and break your system. You can decide to stay for a longer period of time on IPython 5.x LTS and decide to leapfrog a few IPython versions once you migrate to Python 3, though we recommend keeping up to date with the latest stable versions as they are released, and of course to migrate to Python 3 when possible.

IPython is the first IPython/Jupyter project to drop support for Python 2, but you can expect other components of IPython/Jupyter to follow. Since its inception, JupyterHub for example as always been Python 3 only.

It is important to note that users will always be able to use a Python 2 kernel with the Jupyter Notebook, even when all of our projects have transitioned to Python 3: as part of our LTS commitment, we will make any necessary updates to the IPython kernel so it can continue to work in a Jupyter Notebook for the duration of our LTS support.

Help us with the Python 3 transition

We understand that migrating to Python 3 can be difficult for various reasons, and that planning ahead is often necessary. For this reason we are helping to gather a non-exhaustive list of projects that have decided to drop support for Python 2 in or before 2020, when support for Python 2.7 itself ends. Projects such as Matplotlib and SymPy plan to drop support in the next few years, while a few projects like Scikit-Bio are already ahead of us, and should be Python 3 only soon.

Thus we decided to sign the Python3 Statement that lists projects that are taking this step, as well as — when possible — provide a planned release schedule for which versions will still be Python 2 compatible, and which versions will be Python 2 only.

If you’d like to add your project to this page, or you know a project that’s thinking about the Python 3 transition, please get in touch there. We believe that giving enough information to Python users as early as possible will help ease the transition.

See you at SciPy!

Some of us will be at SciPy this year in Austin. We’ll be happy to meet with you, and hopefully run sprints on IPython and Jupyter projects. We hope to see you there.

[Update] Friday July 8, 11 Pacific

Hackernews and Reddit threads

Notebook 4.1 release

2016-01-08T23:36:00+00:00

We are excited to announce the release of the latest minor revision of the Jupyter Notebook, version 4.1.

As usual, you can update using pip:

pip install notebook --upgrade

If you are using Anaconda, you will have to wait for conda to update their copy of the notebook. Once they do, you can run the following:

conda upgrade notebook

The big split

A little over four months ago we released version 4.0, the first release of Jupyter after the Big Split. The transition from the monolitic IPython to the many pieces that we have today was not smooth, but it was well worth it. It will enable us to make many smaller releases of individual subprojects, rather than massive monolithic releases across the entire project. As a result, we are starting to ship smaller releases more often. Today’s release is the notebook, the most visible project in the Jupyter organization.

An overdue update

This minor release was almost four months in the making, much longer than the other Jupyter projects. Usually we only include bug fixes in minor revisions, but in this releases we include a few new features. The addition of these new features meant more testing and design work for 4.1 than minor releases we’ve made in the past. In this blog post we will highlight some of the new features you can expect in this release!

Multi-cells selection

Multi-cell selection is an often requested feature. The new version of the notebook includes the ability to select multiple cells and perform commands on them. Not all cell commands can be performed on multiple cells yet, but the basic API exists and can be used for extensions.

To select a contiguous block of cells while in command mode, press Shift-J or Shift-K (also Shift-Down or Shift-Up) to extend and shrink the selection. Or you can hold Shift and left click on a cell to select a range of cell. The selected cells can be visually distinguished from the unselected ones by their soft blue background.

Many cell commands operate now on the selected cells. For example, merging cells (Shift-M) used to merge current with the cell below it. It still does that for a single selected cell, but now it also merges all selected cells together.

Tell us what you think of multi-select, how you use it, and share the extensions you write which make use of it!

Command palette

Atom and Sublime Text users might be familiar with the Command Palette. As users of Atom and Sublime Text ourselves, we find this is a useful capability as it expands the number of commands that can be performed quickly and efficiently via the keyboard. Thus we have implemented a Command Palette which looks and behaves like the ones found in those editors. Now, using the shortcut Cmd-Shift-P (or Ctrl-Shift-P on Linux and Windows), you can bring up the Command Palette, which is a Spotlight-like dialog box that allows you to quickly search through all the commands available in the notebook and execute them. Regardless of whether the commands originated in the Menu, the Toolbar, or were only exposed through the notebook APIs, you can find them in the Command Palette. Additionally, the Command Palette will show you the keyboard shortcuts bound to the commands you use. This helps you to have a faster workflow as you become familiar with the keyboard shortcuts.

The Command Palette brings the notebook closer to your fingertips. We hope it improves your experience both now and in future revisions where we plan to add fuzzy matching and other features. If you’d like to help implement these features, stop by the jupyter/notebook github repository, we’d be excited to welcome you!

Restart kernel and run-all

One often requested feature is the ability to restart the kernel and run all cells at once. This is a simple way of making sure your notebook does what you think. Now you can access this new feature from the Kernel Menu:

In attempt to get this feature right, we took our time designing the UI and also tested alternatives using a third-party user testing system. You can view the results of the study on GitHub or below:

The option on the left leads to a faster completion of the restarting kernel task.

We haven’t added a keyboard shortcut, but just like the other notebook functions, you can bind your own keyboard shortcut using custom JavaScript. The various restart commands are also available in the Command Palette.

Find and Replace

The new Find and Replace dialog is found under the Edit menu of the notebook. If you’re in command mode, you can press the F key to bring up the dialog.

The dialog displays a live preview of the find and replace action you specify. We have also added toggle buttons for case sensitivity, regular expressions, and the ability to only replace within selected cells.

En route for 5.0

We hope you enjoy this new release and the accompanying features. As always, we welcome questions, feedback, contributions.

We have already started the process of working on 5.0, in collaboration with developers from Continuum Analytics, Bloomberg, and IBM, who are all doing fantastic jobs. The comunity is growing and now is a perfect time for new contributors. A few needed improvements have already been mentioned in this blog post, but many more can be found on our issue tracker on GitHub!

Rendering Notebooks on GitHub

2015-05-07T19:47:00+00:00

We are pleased to announce that, starting today, and as announced on the GitHub blog, Jupyter/IPython notebook (.ipynb) files will render directly on GitHub. This feature works for notebooks in any of the supported Jupyter programming languages on both public and private repos. This capability will complement nbviewer and make it easier for GitHub users to create, view and share notebooks on GitHub.

Here are some great examples of notebook based content that can now be viewed directly on GitHub:

Cameron Davidson-Pilon’s Probabilistic Programming and Bayesian Methods for Hackers.
Lorena Barba’s Aerodynamics-Hydrodynamics with Python.
Benjamin Laken’s Monsoon Analysis.

We want to thank the GitHub team for bringing this new feature to GitHub. In particular @tclem, @arfon, @sshirokov, and @skalnik have all done an incredible job building this at GitHub. The first discussion of how to bring notebook rendering to GitHub happened almost 18 months ago, and a lot of work made this feature possible today. With over 200,000 notebooks available on GitHub, you already have a lot to read!

We are excited to see how this feature will impact science education and research on GitHub, as well as encourage the adoption of the Jupyter notebook as a standard file format across a wide range of fields. Jupyter notebook files are a JSON-based open document format that supports code and results, narrative text, images, and equations in one file. These documents simplify the creation and communication of computational narratives and make the sharing and replication of scientific results and data analyses simpler.

nbviewer

One key piece of infrastructure in sharing notebooks has been the nbviewer service, which started as a side project of mine in the late summer of 2012. Nbviewer has grown through deep integration with GitHub, with the ability to browse all public users, organizations, repositories, branches, tags, and even Gists. This would not have been possible without @rgbkrk and @bollwyvl, who handle the development and deployment of nbviewer. It would also not have been possible without Rackspace and Fastly, who provide free hosting and fast static asset distribution. Nbviewer has steadily grown to render hundreds of thousands of notebooks every week, and usage is still increasing. Based on this, we believe that rendering on GitHub will be a highly used and appreciated feature.

Although you can now view notebooks directly on GitHub, we are not planning to stop the development of nbviewer. While rendering directly in GitHub repositories is convenient and allows viewing notebooks in private repositories, there are number of reasons that nbviewer will continue to remain an important part of project Jupyter.

First, there are many notebooks not hosted on GitHub. As before, these notebooks can still be viewed on nbviewer. Furthermore, the Jupyter/IPython team are actively working on improving integration with other content providers, building on our experience with GitHub. For example, a recent pull request of @bollwyvl streamlines the process of adding new providers to nbviewer, with proposals for DropBox, GitLab, Google Drive, Stash, and other hosting and collaboration platforms.

Second, because of security concerns, some features available on nbviewer will not be available on GitHub. For example, GitHub will not render any dynamic output display that uses JavaScript, custom CSS, and most custom HTML embedded in Markdown or in outputs. When this is the case, and GitHub cannot fully render your notebooks, you will see an icon that allows you to view the full notebook on nbviewer.

Because nbviewer does not handle authentication, it will continue to support custom CSS, HTML and JavaScript in notebook. Thus, for highly customized or dynamic notebooks, nbviewer will still be the ideal place to share them.

We would love your feedback! If you have comments or find bugs with notebook rendering on GitHub or nbviewer, please open an issue on our nbviewer repo. Also, please send a few tweets to thank all the people without whom this would have not happened.

Jupyter Blog - Matthias Bussonnier

Release of IPython 8.0

Many new features

Moving forward by shedding weight

Moving toward pyproject.toml

Monthly Release and following NEP 29

Looking forward to your contributions

Thanks

Jupyter’s role in #ChaosDB

What the Jupyter team knows

What are we doing internally

CVE-2021-32797 and CVE-2021-32798 Remote Code execution in JupyterLab and Jupyter Notebook

Jupyter Security Model

CVE Timeline

What we learned

What we’ll do better

IPython 7.0, Async REPL

A Primer on concurrency

Async in the Python world

IPython AsyncIO Integration

Other Async Libraries (aka: curio and trio integration)

Async in Notebooks (and other Jupyter Clients)

Future improvements

I Python, You R, We Julia

Follow along on Binder

The Tail of Fibonacci

I, Python

You R

Let’s C

Julia to bind them all

JupyterCon 2018: Registration Open

Community Sprint Day, August 25th.

Jupyter Pop-Up, DC, May 15th

Jupyter Pop-Up, March 21, Boston

JupyterCon 2018: Call For Proposal

JupyterCon 2018, CFP Open

Community Day

Thanks

Incident Report: Jupyter services down

How are Jupyter services hosted?

Temporary resolution

Low bus factor

How can you help

Enjoy JupyterCon 2017 on YouTube

Did you miss JupyterCon 2017?

See you next August for JupyterCon 2018

[Update Jan 2018] Missing Talks

Release of IPython 5.5 and 6.2

New Features

Slowing down backports to 5.x

What’s next ?

JupyterCon NYC: August 22nd-25th

What is JupyterCon?

A fantastic line-up of speakers

Sprints

Financial aid

BOFS

Solar Eclipse

Thanks

Release of IPython 5.4, 6.1, and rlipython

IPython 5.4 (and 6.1) major new features

TerminalIPythonApp.interactive_shell_class

Define _repr_mimebundle_

Execution Heuristics

Display IDs

The display function is now always available

Rlipython

IPython 6.1 and Python 3 only features

Release of IPython 6.0

Highlighted feature: Jedi completion

Sunsetting Python 2 support

Make sure you have Pip 9 or above

The Python 3 statement

Migrating to Python3-only source

Keeping a Python 2 stable branch

Migrating more code to Python3-only

Conclusion

Release of IPython 5.0

IPython 5.0-LTS is out!

A brand new terminal interface

`TerminalIPythonApp.interactive_shell_class`

Define `_repr_mimebundle_`