Jupyter is great. Yet, I find myself missing all the little tweaks I made to Emacs whenever I have Jupyter open in a browser. The obvious solution is to have Jupyter in Emacs. One solution is EIN, the Emacs IPython Notebook. However, I had a mixed experience with it: it would often hang and eat up memory (I never bothered to try to debug this behaviour). A neat alternative, for me, is emacs-jupyter. Here’s my setup.
Virtual Environments with Conda
I’ve moved my Python virtual environments over to conda. Virtualenv in combination with virtualenvwrapper has served me well, but my projects often also involve some C/C++ libraries such as FPLLL for which conda is a better choice. The FPyLLL virtualenv instructions are pretty hacky and this sort of hackery can be avoided by using conda.
The official installation instructions for Debian are:
# Install our public GPG key to trusted store curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg install -o root -g root -m 644 conda.gpg \ /usr/share/keyrings/conda-archive-keyring.gpg # Check whether fingerprint is correct (will output an error message otherwise) gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg \ --no-default-keyring --fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806 # Add our Debian repo echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \ > /etc/apt/sources.list.d/conda.list apt update apt install conda
This will install conda in /opt/conda/
. You will need to add source /opt/conda/etc/profile.d/conda.sh
to your shell’s init files to get the conda
command etc in your PATH
. In addition, if you’re running oh-my-zsh you may want to install conda-zsh-completion. Run
git clone https://github.com/esc/conda-zsh-completion \ ${ZSH_CUSTOM:=~/.oh-my-zsh/custom}/plugins/conda-zsh-completion
then add plugins=(… conda-zsh-completion)
and autoload -U compinit && compinit
to your ~/.zshrc
to enable tab completion for commands and environments.
Creating an environment for, say, G6K development then is as easy as
conda create -n g6k python=3.7 fpylll conda activate g6k git clone https://github.com/fplll/g6k cd g6k conda install --file requirements.txt ./rebuild.sh python setup.py install conda install jupyter # optional
Note that you can also install SageMath using conda (but this can be slow).
Jupyter Kernels in one Place
Now, a cool thing you can do with Jupyter and virtual environments is to add the different virtual environments as Jupyter kernels. In particular, running
python -m ipykernel install --user --name=$CONDA_DEFAULT_ENV
inside your conda environment will install a kernel of the same name, e.g. my ~/.local/share/jupyter/kernels/g6k/kernel.json
reads:
{ "argv": [ "/HOME-DIR/.conda/envs/g6k/bin/python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "g6k", "language": "python" }
To list your kernels, run jupyter kernelspec list
. To remove the kernel named foo
, run: jupyter kernelspec uninstall foo
.
This also works for Sage. To install a SageMath kernel run:
jupyter kernelspec install --user $SAGE_ROOT/local/share/jupyter/kernels/sagemath
from inside sage -sh
(this can take a while because it copies 1.7GB of documentation for some reason). You’ll then need to fix
{ "argv": [ - "YOUR-SAGE-ROOT/local/bin/sage", + "YOUR-SAGE-ROOT/sage", "--python", "-m", "sage.repl.ipython_kernel", "-f", "{connection_file}" ], "display_name": "SageMath 9.1", "language": "sage" }
and you’re set.
After doing this, you can run jupyter console --kernel=sagemath
or jupyter console --kernel=g6k
to open an iPython shell for your environment without activating a conda virtual environment first.
Emacs
Time to hook this all into Emacs.
For conda, I use conda.el. My config is pretty straight forward:
(use-package conda :config (progn (conda-env-initialize-interactive-shells) (conda-env-initialize-eshell) (conda-env-autoactivate-mode t) (setq conda-env-home-directory (expand-file-name "~/.conda/")) (custom-set-variables '(conda-anaconda-home "/opt/conda/"))))
Then, in each project related to G6K, I have a .dir-locals.el
file containing ((nil . ((conda-project-env-path . "g6k"))))
which is then picked up by conda-env-autoactivate-mode
to activate the “g6k” conda environment. I’ve also added a segment to the doom-modeline to show the current conda environment and tweaked my toggle shells to automatically activate the right conda environment (same for Python shells, which I should migrate over to emacs-jupyter).
Next, to make use of emacs-jupyter, we need Emacs built with support for dynamic modules. Debian does not currently ship binaries which fulfil this criterion, so we need to build our own.
- Get the sources with
apt source -t unstable emacs
-
Enable modules in
/debian/rules
confflags += --with-sound=alsa +confflags += --with-modules confflags += --without-gconf
- Add a new version number with
dch --local malb
- Build with
dpkg-buildpackage -us -uc
- Install the produced Debian packages
My config then is again pretty simple:
(use-package jupyter :commands (jupyter-run-server-repl jupyter-run-repl jupyter-server-list-kernels) :init (eval-after-load 'jupyter-org-extensions ; conflicts with my helm config, I use <f2 #> '(unbind-key "C-c h" jupyter-org-interaction-mode-map)))
This already gives you a Jupyter REPL (i.e. what iPython used to be) for each of the kernels we installed above: upon running jupyter-run-repl
you’re prompted with a choice of kernels. This repl is rich. For example, typing plot(sin(x) 0, 2*pi)
into a SageMath kernel will show the plot in the Emacs buffer directly.
Finally, to get that Jupyter notebook feeling, we can make use of the org-babel integration of emacs-jupyter. My babel config is
(use-package ob :ensure nil :config (progn ;; load more languages for org-babel (org-babel-do-load-languages 'org-babel-load-languages '((python . t) (shell . t) (latex . t) (ditaa . t) (C . t) (dot . t) (plantuml . t) (makefile . t) (jupyter . t))) ; must be last (setq org-babel-default-header-args:sh '((:results . "output replace")) org-babel-default-header-args:bash '((:results . "output replace")) org-babel-default-header-args:shell '((:results . "output replace")) org-babel-default-header-args:jupyter-python '((:async . "yes") (:session . "py") (:kernel . "sagemath"))) (setq org-confirm-babel-evaluate nil org-plantuml-jar-path "/usr/share/plantuml/plantuml.jar" org-ditaa-jar-path "/usr/share/ditaa/ditaa.jar") (add-to-list 'org-src-lang-modes (quote ("plantuml" . plantuml)))))
which enable to type type code into jupyter blocks (with tab completion etc) into org-mode files, e.g.
#+begin_src jupyter-python :kernel sagemath 2^3 #+end_src #+RESULTS: : 8
Again, the interface is a rich interface, showing plots etc in the buffer directly. To create new blocks, to move them around, etc. there is jupyter-org-hydra/body
which I’ve bound to <f2> #
.
Hello,
The subject is really appealing thank y you ou for this great article.
Personally I tend not to like having anaconda (conda) on my machine. Conda not having the same path for the library storage I find the dual set up (Python + anaconda) cumbersome.
I tend to use org mode with python enclose in « +SRC » tags, you can pass variables between different source codes inside your orgfile, create a sort of index that keep track of your topics, tasks and so on.
That’s the most efficient usage I have found so far.
This is solid gold. I’ve been really struggling with jupyter and cond virtual environments for some time now and this fixes everything.