Conda, Jupyter and Emacs

Jupyter is great. Yet, I find myself missing all the little tweaks I made to Emacs whenever I have Jupyter open in a browser. The obvious solution is to have Jupyter in Emacs. One solution is EIN, the Emacs IPython Notebook. However, I had a mixed experience with it: it would often hang and eat up memory (I never bothered to try to debug this behaviour). A neat alternative, for me, is emacs-jupyter. Here’s my setup.

Virtual Environments with Conda

I’ve moved my Python virtual environments over to conda. Virtualenv in combination with virtualenvwrapper has served me well, but my projects often also involve some C/C++ libraries such as FPLLL for which conda is a better choice. The FPyLLL virtualenv instructions are pretty hacky and this sort of hackery can be avoided by using conda.

The official installation instructions for Debian are:

# Install our public GPG key to trusted store
curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg
install -o root -g root -m 644 conda.gpg \
        /usr/share/keyrings/conda-archive-keyring.gpg

# Check whether fingerprint is correct (will output an error message otherwise)
gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg \
    --no-default-keyring --fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806

# Add our Debian repo
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \
     > /etc/apt/sources.list.d/conda.list
apt update
apt install conda

This will install conda in /opt/conda/. You will need to add source /opt/conda/etc/profile.d/conda.sh to your shell’s init files to get the conda command etc in your PATH. In addition, if you’re running oh-my-zsh you may want to install conda-zsh-completion. Run

git clone https://github.com/esc/conda-zsh-completion \
    ${ZSH_CUSTOM:=~/.oh-my-zsh/custom}/plugins/conda-zsh-completion

then add plugins=(… conda-zsh-completion) and autoload -U compinit && compinit to your ~/.zshrc to enable tab completion for commands and environments.

Creating an environment for, say, G6K development then is as easy as

conda create -n g6k python=3.7 fpylll
conda activate g6k
git clone https://github.com/fplll/g6k
cd g6k
conda install --file requirements.txt
./rebuild.sh
python setup.py install
conda install jupyter # optional

Note that you can also install SageMath using conda (but this can be slow).

Jupyter Kernels in one Place

Now, a cool thing you can do with Jupyter and virtual environments is to add the different virtual environments as Jupyter kernels. In particular, running

python -m ipykernel install --user --name=$CONDA_DEFAULT_ENV

inside your conda environment will install a kernel of the same name, e.g. my ~/.local/share/jupyter/kernels/g6k/kernel.json reads:

{
    "argv": [
        "/HOME-DIR/.conda/envs/g6k/bin/python",
        "-m", "ipykernel_launcher",
        "-f", "{connection_file}"
    ],
    "display_name": "g6k",
    "language": "python"
}                    

To list your kernels, run jupyter kernelspec list. To remove the kernel named foo, run: jupyter kernelspec uninstall foo.

This also works for Sage. To install a SageMath kernel run:

jupyter kernelspec install --user $SAGE_ROOT/local/share/jupyter/kernels/sagemath

from inside sage -sh (this can take a while because it copies 1.7GB of documentation for some reason). You’ll then need to fix

{
    "argv":
    [
-        "YOUR-SAGE-ROOT/local/bin/sage",
+        "YOUR-SAGE-ROOT/sage",
        "--python", "-m", "sage.repl.ipython_kernel",
        "-f", "{connection_file}"
    ],
    "display_name": "SageMath 9.1", "language": "sage"
} 

and you’re set.

After doing this, you can run jupyter console --kernel=sagemath or jupyter console --kernel=g6k to open an iPython shell for your environment without activating a conda virtual environment first.

Emacs

Time to hook this all into Emacs.

For conda, I use conda.el. My config is pretty straight forward:

(use-package conda
  :config (progn
            (conda-env-initialize-interactive-shells)
            (conda-env-initialize-eshell)
            (conda-env-autoactivate-mode t)
            (setq conda-env-home-directory (expand-file-name "~/.conda/"))
            (custom-set-variables '(conda-anaconda-home "/opt/conda/"))))

Then, in each project related to G6K, I have a .dir-locals.el file containing ((nil . ((conda-project-env-path . "g6k")))) which is then picked up by conda-env-autoactivate-mode to activate the “g6k” conda environment. I’ve also added a segment to the doom-modeline to show the current conda environment and tweaked my toggle shells to automatically activate the right conda environment (same for Python shells, which I should migrate over to emacs-jupyter).

Next, to make use of emacs-jupyter, we need Emacs built with support for dynamic modules. Debian does not currently ship binaries which fulfil this criterion, so we need to build our own.

1. Get the sources with apt source -t unstable emacs

2. Enable modules in /debian/rules

   confflags += --with-sound=alsa
   +confflags += --with-modules
   confflags += --without-gconf
   

3. Add a new version number with dch --local malb
4. Build with dpkg-buildpackage -us -uc
5. Install the produced Debian packages

My config then is again pretty simple:

(use-package jupyter
  :commands (jupyter-run-server-repl
             jupyter-run-repl
             jupyter-server-list-kernels)
  :init (eval-after-load 'jupyter-org-extensions ; conflicts with my helm config, I use <f2 #>
          '(unbind-key "C-c h" jupyter-org-interaction-mode-map)))

This already gives you a Jupyter REPL (i.e. what iPython used to be) for each of the kernels we installed above: upon running jupyter-run-repl you’re prompted with a choice of kernels. This repl is rich. For example, typing plot(sin(x) 0, 2*pi) into a SageMath kernel will show the plot in the Emacs buffer directly.

Finally, to get that Jupyter notebook feeling, we can make use of the org-babel integration of emacs-jupyter. My babel config is

(use-package ob
  :ensure nil
  :config (progn
            ;; load more languages for org-babel
            (org-babel-do-load-languages
             'org-babel-load-languages
             '((python . t)
               (shell . t)
               (latex . t)
               (ditaa . t)
               (C . t)
               (dot . t)
               (plantuml . t)
               (makefile . t)
               (jupyter . t)))          ; must be last

            (setq org-babel-default-header-args:sh    '((:results . "output replace"))
                  org-babel-default-header-args:bash  '((:results . "output replace"))
                  org-babel-default-header-args:shell '((:results . "output replace"))
                  org-babel-default-header-args:jupyter-python '((:async . "yes")
                                                                 (:session . "py")
                                                                 (:kernel . "sagemath")))

            (setq org-confirm-babel-evaluate nil
                  org-plantuml-jar-path "/usr/share/plantuml/plantuml.jar"
                  org-ditaa-jar-path "/usr/share/ditaa/ditaa.jar")

            (add-to-list 'org-src-lang-modes (quote ("plantuml" . plantuml)))))

which enable to type type code into jupyter blocks (with tab completion etc) into org-mode files, e.g.

#+begin_src jupyter-python :kernel sagemath
2^3
#+end_src

#+RESULTS:
: 8

Again, the interface is a rich interface, showing plots etc in the buffer directly. To create new blocks, to move them around, etc. there is jupyter-org-hydra/body which I’ve bound to <f2> #.