Conda, Jupyter and Emacs

Jupyter is great. Yet, I find myself missing all the little tweaks I made to Emacs whenever I have Jupyter open in a browser. The obvious solution is to have Jupyter in Emacs. One solution is EIN, the Emacs IPython Notebook. However, I had a mixed experience with it: it would often hang and eat up memory (I never bothered to try to debug this behaviour). A neat alternative, for me, is emacs-jupyter. Here’s my setup.

Virtual Environments with Conda

I’ve moved my Python virtual environments over to conda. Virtualenv in combination with virtualenvwrapper has served me well, but my projects often also involve some C/C++ libraries such as FPLLL for which conda is a better choice. The FPyLLL virtualenv instructions are pretty hacky and this sort of hackery can be avoided by using conda.

The official installation instructions for Debian are:

# Install our public GPG key to trusted store
curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg
install -o root -g root -m 644 conda.gpg \
        /usr/share/keyrings/conda-archive-keyring.gpg

# Check whether fingerprint is correct (will output an error message otherwise)
gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg \
    --no-default-keyring --fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806

# Add our Debian repo
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \
     > /etc/apt/sources.list.d/conda.list
apt update
apt install conda

This will install conda in /opt/conda/. You will need to add source /opt/conda/etc/profile.d/conda.sh to your shell’s init files to get the conda command etc in your PATH. In addition, if you’re running oh-my-zsh you may want to install conda-zsh-completion. Run

git clone https://github.com/esc/conda-zsh-completion \
    ${ZSH_CUSTOM:=~/.oh-my-zsh/custom}/plugins/conda-zsh-completion

then add plugins=(… conda-zsh-completion) and autoload -U compinit && compinit to your ~/.zshrc to enable tab completion for commands and environments.

Creating an environment for, say, G6K development then is as easy as

conda create -n g6k python=3.7 fpylll
conda activate g6k
git clone https://github.com/fplll/g6k
cd g6k
conda install --file requirements.txt
./rebuild.sh
python setup.py install
conda install jupyter # optional

Note that you can also install SageMath using conda (but this can be slow).

Jupyter Kernels in one Place

Now, a cool thing you can do with Jupyter and virtual environments is to add the different virtual environments as Jupyter kernels. In particular, running

python -m ipykernel install --user --name=$CONDA_DEFAULT_ENV

inside your conda environment will install a kernel of the same name, e.g. my ~/.local/share/jupyter/kernels/g6k/kernel.json reads:

{
    "argv": [
        "/HOME-DIR/.conda/envs/g6k/bin/python",
        "-m", "ipykernel_launcher",
        "-f", "{connection_file}"
    ],
    "display_name": "g6k",
    "language": "python"
}                    

To list your kernels, run jupyter kernelspec list. To remove the kernel named foo, run: jupyter kernelspec uninstall foo.

This also works for Sage. To install a SageMath kernel run:

jupyter kernelspec install --user $SAGE_ROOT/local/share/jupyter/kernels/sagemath

from inside sage -sh (this can take a while because it copies 1.7GB of documentation for some reason). You’ll then need to fix

{
    "argv":
    [
-        "YOUR-SAGE-ROOT/local/bin/sage",
+        "YOUR-SAGE-ROOT/sage",
        "--python", "-m", "sage.repl.ipython_kernel",
        "-f", "{connection_file}"
    ],
    "display_name": "SageMath 9.1", "language": "sage"
} 

and you’re set.

After doing this, you can run jupyter console --kernel=sagemath or jupyter console --kernel=g6k to open an iPython shell for your environment without activating a conda virtual environment first.

Emacs

Time to hook this all into Emacs.

For conda, I use conda.el. My config is pretty straight forward:

(use-package conda
  :config (progn
            (conda-env-initialize-interactive-shells)
            (conda-env-initialize-eshell)
            (conda-env-autoactivate-mode t)
            (setq conda-env-home-directory (expand-file-name "~/.conda/"))
            (custom-set-variables '(conda-anaconda-home "/opt/conda/"))))

Then, in each project related to G6K, I have a .dir-locals.el file containing ((nil . ((conda-project-env-path . "g6k")))) which is then picked up by conda-env-autoactivate-mode to activate the “g6k” conda environment. I’ve also added a segment to the doom-modeline to show the current conda environment and tweaked my toggle shells to automatically activate the right conda environment (same for Python shells, which I should migrate over to emacs-jupyter).

Next, to make use of emacs-jupyter, we need Emacs built with support for dynamic modules. Debian does not currently ship binaries which fulfil this criterion, so we need to build our own.

  1. Get the sources with apt source -t unstable emacs
  2. Enable modules in /debian/rules

    confflags += --with-sound=alsa
    +confflags += --with-modules
    confflags += --without-gconf
    
  3. Add a new version number with dch --local malb
  4. Build with dpkg-buildpackage -us -uc
  5. Install the produced Debian packages

My config then is again pretty simple:

(use-package jupyter
  :commands (jupyter-run-server-repl
             jupyter-run-repl
             jupyter-server-list-kernels)
  :init (eval-after-load 'jupyter-org-extensions ; conflicts with my helm config, I use <f2 #>
          '(unbind-key "C-c h" jupyter-org-interaction-mode-map)))

This already gives you a Jupyter REPL (i.e. what iPython used to be) for each of the kernels we installed above: upon running jupyter-run-repl you’re prompted with a choice of kernels. This repl is rich. For example, typing plot(sin(x) 0, 2*pi) into a SageMath kernel will show the plot in the Emacs buffer directly.

Finally, to get that Jupyter notebook feeling, we can make use of the org-babel integration of emacs-jupyter. My babel config is

(use-package ob
  :ensure nil
  :config (progn
            ;; load more languages for org-babel
            (org-babel-do-load-languages
             'org-babel-load-languages
             '((python . t)
               (shell . t)
               (latex . t)
               (ditaa . t)
               (C . t)
               (dot . t)
               (plantuml . t)
               (makefile . t)
               (jupyter . t)))          ; must be last

            (setq org-babel-default-header-args:sh    '((:results . "output replace"))
                  org-babel-default-header-args:bash  '((:results . "output replace"))
                  org-babel-default-header-args:shell '((:results . "output replace"))
                  org-babel-default-header-args:jupyter-python '((:async . "yes")
                                                                 (:session . "py")
                                                                 (:kernel . "sagemath")))

            (setq org-confirm-babel-evaluate nil
                  org-plantuml-jar-path "/usr/share/plantuml/plantuml.jar"
                  org-ditaa-jar-path "/usr/share/ditaa/ditaa.jar")

            (add-to-list 'org-src-lang-modes (quote ("plantuml" . plantuml)))))

which enable to type type code into jupyter blocks (with tab completion etc) into org-mode files, e.g.

#+begin_src jupyter-python :kernel sagemath
2^3
#+end_src

#+RESULTS:
: 8

Again, the interface is a rich interface, showing plots etc in the buffer directly. To create new blocks, to move them around, etc. there is jupyter-org-hydra/body which I’ve bound to <f2> #.

One thought on “Conda, Jupyter and Emacs

  1. Hello,

    The subject is really appealing thank y you ou for this great article.

    Personally I tend not to like having anaconda (conda) on my machine. Conda not having the same path for the library storage I find the dual set up (Python + anaconda) cumbersome.

    I tend to use org mode with python enclose in « +SRC » tags, you can pass variables between different source codes inside your orgfile, create a sort of index that keep track of your topics, tasks and so on.

    That’s the most efficient usage I have found so far.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s