Performance and memory profiling Jupyter with Sciagraph

Jupyter notebook running too slowly, or using too much memory? You can get results faster—but only if you can find the bottlenecks and fix them.

Sciagraph can help: it can give you performance and memory profiling reports to help identify the bottlenecks, and it comes with Jupyter integration built in.

In order to use Sciagraph with Jupyter, you need to:

  1. Ensure Sciagraph is installed.
  2. Make sure your notebook is using the Sciagraph kernel.
  3. Enable profiling using %load_ext sciagraph.
  4. Profile a cell using %%sciagraph_profile.

Sciagraph has been tested with JupyterLab and Jupyter Notebook.

1. Installing Sciagraph

Supported operating systems:

  • Linux on x86_64, Python 3.7-3.11 (ARM can be made available if there’s interest; email me).
  • macOS with Python 3.9-3.11, Catalina (10.15) or later, on both x86_64 and ARM.
  • Windows is not currently supported directly, so you need to use Linux emulation of some sort:
    • WSLv2 is likely to work, but still untested.
    • Or, you can use Docker to start a Linux-based shell prompt: 
      docker run -it -v python:3.11 bash

To install Sciagraph, you’ll need to run the following inside the environment where Jupyter is installed:

pip install sciagraph

2. Make sure your notebook is using the Sciagraph kernel

The kernel for a Jupyter notebook typically determines what programming language it is using, for example Python 3. To use Sciagraph with your notebook, you need to use a special kernel that adds Sciagraph to Python 3; it shouldn’t otherwise have any noticeable impact on running your code.

  • When creating a new notebook, just choose “Python 3 with Sciagraph” as your kernel, instead of Python 3.
  • To change an existing notebook, go to the “Kernel” menu, choose “Change Kernel…” and then select the “Python 3 with Sciagraph” kernel.

3. Enable profiling

Once your notebook is started, run the following in one of your cells:

%load_ext sciagraph

This will start Sciagraph.

If you haven’t already stored your access token, the first time you run this, you will be required to enter your SCIAGRAPH_ACCESS_KEY and SCIAGRAPH_ACCESS_SECRET. You will need to sign up for the free Sciagraph plan to get these. The keys will then be stored in a local config file so you don’t need to store them again.

Alternatively, you can run the python -m sciagraph.store_token ... command in the account page to store the token.

4. Profile a cell

To profile the contents of a cell, just add %%sciagraph_profile as the first line of the cell, for example:

%%sciagraph_profile
x = call_a_function()
call_another_function(x, 123)

This will generate a report on disk, for future reference, and also embed it into the notebook itself. On JupyterLab you might want to disable cell scrolling so it’s easier to read the report: hold down the right mouse button in the notebook and choose “Disable Scrolling for Outputs”.

Note that memory profiling will only profile code run during the execution of the cell; anything allocated before the cell ran won’t show up in the profiling report. If the code in the cell calls functions defined in other cells, that will be profiled, since it was still triggered by the particular cell you are profiling. Put another way, you will get profiling for any code that ran after %%sciagraph_profile, until the cell finishes executing. The same is true for the performance profiling.

To learn more about what Sciagraph’s output means, read the introductory documentation.