Pascal T. Merz

A GitHub action to turn Jupyter notebooks into Minimal Mistakes Jekyll blog posts

2021-12-17T00:00:00+00:00

Starting point

As mentioned in Turning Jupyter notebooks into Jekyll blog posts, I was deploying my Minimal Mistakes Jekyll site to the gh_pages branch using a simple workflow. I now wanted to add an action which would take all notebooks stored in a specific folder and turn them into markdown posts.

name: Deployment

# Controls when the workflow will run
on:
  # Triggers the workflow on push or pull request events but only for the main branch
  push:
    branches: [ main ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

jobs:
  build-site:
    runs-on: ubuntu-latest
    steps:
      # Check out current repo
      - uses: actions/checkout@v2

      # TODO HERE:
      # Action which turns notebooks into Markdown posts

      # Build and deploy the site
      - name: Build and deploy
        uses: EdricChan03/action-build-deploy-ghpages@v2.5.0
        env:
          JEKYLL_ENV: 'production'
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}

I had also written a simple script consisting of a few lines of bash and perl and a call to jupyter nbconvert to convert notebooks to Markdown posts, taking care of moving the file and the images and fixing image links. Finally, I found that creating a new layout type notebook was the easiest way to include badges to view the notebook on GitHub or Google Colab.

I decided to bundle the two files which are required on top of the typical Jekyll files into the action, such that it could be used with zero installation.

One seemingly small detail that doesn’t seem to be too clearly documented is how data can be accessed in the different stages of the action. Here is what I think to have understood:

When setting up the docker container (when running the Dockerfile), the files in the action repository are available. We can therefore copy any files into the container at that point.
The container will start in the ${GITHUB_WORKSPACE} which is mounted at /github/workspace. The files in the action repository are not available anymore, unless we have copied them previously.
${GITHUB_WORKSPACE} is also where, for example, actions/checkout is checking out the repository in. The workspace remains available throughout the steps of a job, which means that it can be used to store files which will be used by later actions.

Setting up a docker container

The action has relatively few requirements: We need an environment to run bash and perl and we need to be able to use jupyter nbconvert. I decided to use jupyter/base-notebook, which is a basic installation of Jupyter on Ubuntu. It is pretty light-weight but has all the features we will need pre-installed.

The Dockerfile is relatively easy, we only need to specify our starting point (the current base-notebook release), add the notebook layout and the action script, and set the action script as the entry point.

# Container image that runs our code.
# base-notebook is a ubuntu image with a light-weight Jupyter installation
FROM jupyter/base-notebook:2021-12-16
USER root

# The action needs the special notebook layout and the action script
# We'll just copy the entire folder there
COPY . /action

# Run the action script when starting the docker container
ENTRYPOINT ["/action/convert_notebooks.sh"]

A few notes:

jupyter/base-notebook is released every day. For reproducibility, I picked a recent version. It’s unlikely I’ll have to change it any time soon.
By default, the jupyter/base-notebook has no root privileges. GitHub actions do, however, normally run using the root user. The non-privileged default user does not have access to the ${GITHUB_WORKSPACE} folder which is the default work directory for GitHub actions. To play well with other actions, I decided to use the root user rather than trying to work in a different folder.
When building the Docker container, we have full access to the files of the action. Since we need some of them later, I decided to simply copy the full folder to the /action location in the container.
The entry point is the script which does the entire conversion. The container will be run from ${GITHUB_WORKSPACE}.

Action metadata file

The action metadata file defines the name and description of the action, as well as the run environment, the inputs, and the outputs. For our action, it is extremely easy since we need no inputs and no outputs. All we need to define is the name and the description, and that we are using docker to run it.

The minimal metadata file looks like this:

name: 'Jupyter Minimal Mistakes'
description: 'Convert Jupyter notebooks to Markdown blog posts ready for use with Jekyll and the Minimal Mistakes theme'
runs:
  using: 'docker'
  image: 'Dockerfile'

Action script

The action script itself is a simple bash script which does some sanity checks, and then loops over the notebooks found in the _notebooks folder and creates post for each of them. It also creates the notebook layout in the _layouts folder if it doesn’t exist. The full action script is here.

Tests

I added some basic functionality and regression tests. The test checks out the current repository and transforms a simple Jupyter notebook into Markdown. The resulting Markdown file is checked against a reference file, and the existence of the notebook layout file and the image file is verified. This is unlikely to catch all errors, but does at least perform some basic functionality check.

Notes

To run a base image locally and play around a bit, this command is very helpful:

docker run -it --entrypoint /bin/bash jupyter/base-notebook

This will start a docker container with the chosen base image (in the example above jupter/base-notebook) with a bash session as entry point. This is very handy to run some commands interactively and make sure that everything works as intended.

References

GitHub Docs: Creating a Docker container action

Turning Jupyter notebooks into Jekyll blog posts

2021-12-17T00:00:00+00:00

One functionality I really want for my blog is to be able to easily turn Jupyter notebooks into blog posts. I do much exploration in notebooks, so being able to use them as the base of blog posts would be great. I am hoping to find a solution which works well with the Minimal Mistakes theme I’m using to make the result fit nicely with the remaining design of the website. I don’t think I need a very complicated solution - a static version of the notebook like it is for example used by nbsphinx should be enough. I used nbsphinx recently to include usage examples into the physical_validation documentation (see e.g. here), and quite liked the result. If I get this to work nicely, I do have a few additional features on my wish list, however:

An automatic link to the source notebook on GitHub, which would allow readers to easily download the notebook and run it locally.
An integration with Google Colab, which would allow readers to run the notebook online.
Hiding certain input or output cells would allow me to highlight specific things while hiding less interesting boilerplate code without completely removing it.

Getting started

There are multiple blog posts explaining how to turn Jupyter notebooks into either Markdown or HTML files (e.g. 1, 2, 3) using nbconvert. If there are plots in the notebook, nbconvert turns them into png images which are then embedded in the Markdown document. Using the resulting files as Jekyll posts does then only require adjusting a few things like headers or image paths, moving the files in the right folder, and upload the result. This sounds like a viable option!

There are several additional options I found which don’t perfectly fit for what I’m trying to achieve, but look very interesting:

fastpages, a blogging platform which supports Markdown and Jupyter notebooks (and even Word docs!). This looks extremely powerful, but after some exploration I found that trying to fit the result into my existing site did not yield satisfactory results. I think this is a great project, though!
Jupyter Book turns notebooks into beautiful books or articles. This looks amazing for standalone projects, but I don’t see an easy way to fit the result into my existing page.
Jekyll Jupyter Notebook plugin allows to embed Jupyter notebooks in Markdown documents used by Jekyll. This seems really useful, but is not exactly what I’m looking for here - I would prefer to write the entire post in a notebook rather than including a notebook into a Markdown post.

I decided to use nbconvert to create Markdown files (and not directly HTML). My reasoning was that this approach would have a higher likelihood to create posts with the right look, as they would be processed by Jekyll in the same way as the rest of the site. This also meant that I could write the YAML front matter for the posts in a Markdown cell, which should then be left alone by nbconvert. Finally, it would also mean that the automatic excerpt function of the Minimal Mistakes theme (taking the first paragraph as an excerpt if it isn’t defined differently) should work properly.

Turning a Jupyter notebook into a Markdown Jekyll post

Naming and file organization

When converting a Jupyter notebook to Markdown, nbconvert will by default write a Markdown file with the same base name as the notebook, changing only the file extension. The Markdown conversion of a notebook named basename.ipynb will be saved in a file called basename.md. Since Jekyll blog posts are required to be named as YEAR-MONTH-DAY-title.md ( reference), it is easiest to follow that naming for notebooks. A notebook which should become a blog post should hence be named YEAR-MONTH-DAY-title.ipynb.

Blog posts on Jekyll sites live in the _posts folder. I decided to follow the example of fastpages and create a _notebooks folder in which to store the notebooks. I will then move the resulting Markdown file into the _posts folder, where it will be picked up by Jekyll. I will also move the images into the assets/images/ folder, which is the default location for images.

A bash script to convert Jupyter notebooks to Markdown Jekyll posts

As a recap, here’s the steps we need to take:

Use nbconvert to convert Jupyter notebook to Markdown
Change image paths from local paths to absolute paths in assets/images/
Move the resulting Markdown file into the _posts folder
Move the image folder to assets/images/.

This can easily be done by a few lines of bash with some perl support (and a call to jupyter nbconvert which does the heavy-lifting, of course):

# Get file name without extension
filename="${notebook_file%.*}"
# Name of Markdown file will be base name + ".md"
md_notebook="${filename}.md"
# Convert notebook to Markdown
jupyter nbconvert "$notebook_file" --to markdown
# Fix image links to point to a place where Jekyll will pick up the images
perl -pi -e 's/\[png\]\(/[png]({{ site.url }}{{ site.baseurl }}\/assets\/images\//g' "$md_notebook"
# Move Markdown notebook to _posts/ directory
mv "$md_notebook" "${root_directory}/_posts/"
# Move images to a assets such that Jekyll finds them
mv "${filename}_files" "${root_directory}/assets/images/"

Front matter

Since nbconvert does not change the content of Markdown cells, I can simply define the first cell of the notebook to be a Markdown cell and write the YAML front matter there. For example, the first cell of this notebook reads

---
title: "Turning Jupyter notebooks into Jekyll blog posts"
toc: true
category: "Blogging platform"
tags: ["Jekyll", "Jupyter", "Minimal Mistakes"]
badges: true
---

nbconvert will simply print that as the first lines of the created Markdown file. Jekyll will then interpret these lines as the YAML front matter, and will render the rest of the post based on the settings set here. I’m pretty satisfied with this solution: It’s easy and intuitive to write, and it doesn’t impair the functionality of the notebook in any way.

Links to GitHub and Google Colab

Having a way to get the source notebook on GitHub or run in on Google Colab is a great feature that I really enjoy on other pages, so I’d love to have that on my blog. fastpages has built-in functionality to add badges that I think works and looks great, so I will try to replicate some of that behavior here.

Links

Links to files on GitHub do generally have the form https://github.com/<GitHub username or organization>/<repository name>/blob/<branch>/<file path>.

To run notebooks stored on GitHub in Google Colab, the link has the form https://colab.research.google.com/github/<GitHub username or organization>/<repository name>/blob/<branch>/<file path>.

Knowing this, it’s easy to create links to the notebook generating a blog post. I am serving my site from the main branch, and in my file organization mentioned above the notebook is going to be under _notebooks/<filename>. Note that the repository name can be set in the Jekyll configuration, so if the repository tag was set in _config.yml, the repository information in the link can be filled by Jekyll. To achieve this, I simply replaced <GitHub username or organization>/<repository name> by {{ site.repository }}. So my links are going to read
https://github.com/{{site.repository}}/blob/main/_notebooks/<filename>
for GitHub and
https://colab.research.google.com/github/{{site.repository}}/blob/main/_notebooks/<filename>
for Google Colab.

Badges

Google Colab offers a badge which looks like this:

I didn’t find a comparable badge for GitHub, but using shields.io it’s easy to create one:

Including badges in blog posts

To include the badges in blog posts, I decided, again inspired by fastpages, to add a new layout called notebook to Minimal Mistakes. That new layout is the same as the default single layout, but adds the badges. This solution has a couple of nice properties:

It’s update friendly: If the single layout is changed, notebook is updated accordingly.
It works with the default excerpts: By default, the excerpt which is used to create the pagination links is done by taking the first few lines of the content of the post. If the badges were part of the content, the standard excerpt of all posts would start with the (non-rendered) code for the badges. By introducing the new layout and making the badges part of the layout rather than the content, we are avoiding this problem.

The new layout is only a few lines long and lives in _layouts/notebook.html:

---
layout: single
---

{% if page.badges %}
  <hr>
  <div style="display: flex; justify-content: center;">
    <div style="padding-left: 0.5em; padding-right: 0.5em;">
      <a href="https://github.com/{{ site.repository }}/blob/main/_notebooks/{{ page.filename }}" target="_blank">
        <img src="https://img.shields.io/static/v1?label=&message=Open%20on%20GitHub&logo=github&color=lightgray&labelColor=gray" alt="Open on GitHub">
      </a>
    </div>
    <div style="padding-left: 0.5em; padding-right: 0.5em;">
      <a href="https://colab.research.google.com/github/{{ site.repository }}/blob/main/_notebooks/{{ page.filename }}" target="_blank">
        <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open in Colab">
      </a>
    </div>
  </div>
  <hr>
{% endif %}

{{ content }}

The first line makes the layout inherit all properties of the single layout. Before including the page content on the last line, there are a few lines of HTML code with some Jekyll tags. The HTML code gets only included if page.badges is true. If it gets included, it adds the two tags with links as described above between two horizontal lines <hr>.

To make this work seamlessly, I extended the bash code above by a few lines to add the notebook layout and the file name to the YAML front matter automatically when converting Jupyter notebook.

Running the conversion as a GitHub action

I am already using GitHub actions to deploy my website to the gh_pages branch. The workflow is very simple, it is checking out the current repository, building the website and pushing the result to gh_pages. I therefore decided that all the machinery of turning Jupyter notebooks in blog posts should live in a separate action, meaning that there wouldn’t need to be any changes to my site repository to enable this functionality - except for adding notebooks to the _notebooks folder, of course.

The action I wrote lives at ptmerz/jupyter-minimal-mistakes and is ready to be used by others (but has only been tested by myself, so has certainly tons of potential improvements and probably a few bugs). The README has more details on its use. I also wrote some more details about creating that action at A GitHub action to turn Jupyter notebooks into Minimal Mistakes Jekyll blog posts.

Running the conversion locally

While I think that the GitHub action is the perfect way to handle this upstream, I sometimes want to run the conversion locally, mostly to be able to preview how a post looks using my local Jekyll installation. To do that, I have a clone of the repository of the GitHub action next to my local clone of my website repository. The first time I am using it, I need to copy the layout file into my website repository:

mkdir -p username.github.io/_layouts
cp jupyter-minimal-mistakes/_layouts/notebook.html username.github.io/_layouts/notebook.html

Then, from the site folder, I can simply call the convert_notebooks.sh script:

cd username.github.io
sh ../jupyter-minimal-mistakes/convert_notebooks.sh

This feels a tad hack-y, but it works perfectly, so I’m OK with that for now!

Features

Math mode

Math mode works in the Markdown part of the notebook, both inline mode $a^2 + b^2 = c^2$ and display mode

\[i \hbar \frac{\partial}{\partial t}\Psi(\mathbf{r},t) = \hat H \Psi(\mathbf{r},t)\]

Matplotlib plot

import numpy as np
import matplotlib.pyplot as plt

x = np.arange(-np.pi, np.pi, 0.01)
fig, ax = plt.subplots(1, 2, figsize=(12, 3))
ax = ax.flatten()

ax[0].plot(x, np.sin(x))
ax[1].plot(x, np.cos(x))

plt.show()

Code with output

for word in ["one", "two", "three"]:
    print(word)

one
two
three

Future work

There are two small details I might look into some day:

I am not completely satisfied by the design of the code output. I would like to make it easier to distinguish between the input of a code cell and its output.
I would love to have a way to hide / collapse input or output of code cells like Jupyter Book. I think this feature would allow to write concise posts focusing on the essential parts. It is challenging, though, as it would require to either convert the notebook directly into HTML, or to further process the Markdown file created by jupyter nbconvert.

Conclusion

I am really happy with this solution. I can write blog posts as Jupyter notebooks, needing only to add a first cell with the YAML front matter. I can commit these notebooks to GitHub, and the GitHub actions take care of the rest. The results fit perfectly in the Minimal Mistake theme, and allow readers to view the original notebook on GitHub or in Colab.

Enabling blog comments using `giscus`

2021-12-11T00:00:00+00:00

Minimal Mistakes offers built-in support for commenting via Disqus, Discourse, Facebook, utterances, giscus and Staticman. The first three are external services. I didn’t pursue these options further, because I would prefer to avoid external trackers and heavy assets on my website. Staticman allows adding comments as pushes or pull requests to the static site, making them effectively part of the static site. This would certainly be the most light-weight option, but the setup process and especially the spam handling are a bit intimidating.

utterances and giscus leverage GitHub functionality to provide comments. Both are open source and promise to have no tracking, no ads, and be always free. utterances (mis)uses GitHub issues on the repository for the comments, while giscus uses the new GitHub discussions functionality. Since the latter sounds more like using functionality how it was meant to be used and less like a hack, I decided to go for giscus.

The Minimal Mistakes documentation does a great job at explaining the steps, which roughly consist of

Turn on the “Discussions” feature of the repository under Settings > Options > Features.
Install the giscus app in the repository.
Head to https://giscus.app and fill out the details. Most of it is self-explanatory and for the rest I went with the default choice hoping for the best. Specifically, I wasn’t sure about the “Page ↔️ Discussions Mapping”, so I left the default “Discussion title contains page pathname”. I picked the “Announcements” category and chose to enable reactions. These settings are populating a JavaScript snippet that appears at the bottom of the page. Since Minimal Mistakes has built-in functionality for giscus, we will actually not copy that snippet, but use the settings in the Minimal Mistakes configuration file.
Adapt the Minimal Mistakes _config.yml: Make sure that the repository tag is set, set the provider in the comments section to "giscus", and fill the variables in the giscus section by using the values in the JavaScript snippet.
Finally, set comments: true in the YAML front matter of the posts. This can either be done on a per-post-basis, by adding this in the front matter of the posts that should have comments, or globally by changing the default in the _config.yml file for the posts category.

Having done all that, I pushed and reloaded the page and - nothing showed up! It turns out that Minimal Mistakes only turns comments on when Jekyll is run in a production environment, while Jekyll is in a development environment by default (reference). The GitHub action I’m using to build and deploy the Jekyll site was running Jekyll in a development environment. Fixing that did the trick.

You can see the results of my efforts under this post! I am actually really happy with how that looks and feels. I think that a lot of people that could want to interact with these posts will have a GitHub account, so using that account to comment seems like a very low barrier. If people prefer not to grant giscus the required GitHub OAuth authorization, they can post directly on the GitHub Discussions. The comments support the same rich Markdown and reactions as is being used all over GitHub, and the option to add post reactions is a nice little addition. Finally, I also think that the looks fit well with the rest of the site layout.

Minimal Mistakes and MathJax

2021-12-07T00:00:00+00:00

After setting up this website with the beautiful Minimal Mistakes theme, I noticed that it wasn’t supporting typesetting math equations. Well, it can’t be too hard to get this to work, right? Right…

Note that I am using Minimal Mistakes as a remote theme, and I would really like to keep it that way if possible. Using a remote theme reduces the bloat of my repository, and means I’m always getting the latest version. So any solution I’m finding should, if possible, not break updates!

After a bit of digging, it decided I’d try to set up the MathJax JavaScript library. Unfortunately, none of the suggestions (1, 2) I found seemed to work properly - the equations would either not render at all or render with weird typesetting errors. Finally, I found that Academic Pages, a fork of Minimal Mistakes, has working MathJax support which is enabled by a few lines which are added to the header of each file. I copied the last few lines from here and put them into a newly created _includes/head/custom.html in my repo. This is a file which gets added automatically to the header of each Markdown file when it is converted to HTML. This file exists in the upstream Minimal Mistakes repo, but it is empty. If Jekyll finds a file both in the remote theme and in the local repo, it will prefer the local file. So as long as future versions of Minimal Mistakes will not start using that file, using it locally won’t cause any problems when updating.

Here’s how my _includes/head/custom.html looks now:

<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    tex2jax: {
      inlineMath: [ ['$','$'], ["\\(","\\)"] ],
      processEscapes: true
    }
  });
</script>
<script src='https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.4/latest.js?config=TeX-MML-AM_CHTML' async></script>

Note that I left out one line from the Academic Pages header, which reads

<script type="text/x-mathjax-config"> MathJax.Hub.Config({ TeX: { equationNumbers: { autoNumber: "all" } } }); </script>

This line would enable equation numbering by default. I don’t want equation numbers as a default, so I left this out. I should also play around a bit to see if different MathJax version would work, but for now I’m just happy I can typeset nice equations inline, like $a^2 + b^2 = c^2$, and in display mode

\[i \hbar \frac{\partial}{\partial t}\Psi(\mathbf{r},t) = \hat H \Psi(\mathbf{r},t)\]