sqrtminusone/sqrtminusone.github.io
1
0
Fork 0
mirror of https://github.com/SqrtMinusOne/sqrtminusone.github.io.git synced 2025-12-10 15:53:03 +03:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat(org-python): more

Browse source
This commit is contained in:
Pavel Korytov 2021-04-28 20:57:43 +03:00
parent 5ae4f92d08
commit eab1bdd471
2 changed files with 712 additions and 16 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

483
content/posts/2021-04-07-org-python.md Normal file
View file

@ -0,0 +1,483 @@
+++
title = "Replacing Jupyter Notebook with Org Mode"
author = ["Pavel"]
date = 2021-04-08
tags = ["emacs", "org"]
draft = true
+++
## Why? {#why}
## Basic setup {#basic-setup}
There are multiple ways of doing literate programming with Python in Emacs, [ein](https://github.com/millejoh/emacs-ipython-notebook) being one of the notable alternatives.
However, I go with the [emacs-jupyter](https://github.com/nnicandro/emacs-jupyter) package. Installing it is pretty straightforward, e.g. `use-package` with `straight.el`:
```emacs-lisp
(use-package jupyter
:straight t)
```
Then, we have to enable languages for `org-babel`. The following isn't the best practice for startup performance time, but the least problematic in my experience.
```emacs-lisp
(org-babel-do-load-languages
'org-babel-load-languages
'((emacs-lisp . t) ;; Other languages
(shell . t)
;; Python & Jupyter
(python . t)
(jupyter . t)))
```
That adds Org source blocks with names like `jupyter-LANG`, e.g. `jupyter-python`. To use just `LANG` src blocks, call the following function after `org-babel-do-load-languages`:
```emacs-lisp
(org-babel-jupyter-override-src-block "python")
```
That overrides built-in `python` block with `jupyter-python`.
If you use [ob-async](https://github.com/astahlman/ob-async), you have to set `jupyter-LANG` blocks as ignored by this package, because emacs-jupyter has async execution of its own.
```emacs-lisp
(setq ob-async-no-async-languages-alist '("python" "jupyter-python"))
```
## Environments {#environments}
So, we've set up a basic emacs-jupyter configuration.
The catch here is that Jupyter should be available on Emacs startup (at the time of evaluation of the `emacs-jupyter` package, to be precise). That means, if you are launching Emacs with something like an application launcher, global Python & Jupyter will be used.
```python
import sys
sys.executable
```
```text
/usr/bin/python3
```
Which is probably not what we want. To resolve that, we have to make the right Python available at the required time.
### Anaconda {#anaconda}
If you were using Jupyter Lab or Notebook before, there is a good change you used it via [Anaconda](https://anaconda.org/). If not, in a nutshell, it is a package & environment manager, which specializes on Python & R, but also supports a whole lot of stuff like Node.js. In my opinion, it is the easiest way to manage multiple Python installations if you don't use some advanced package manager like Guix.
As one may expect, there is an Emacs package called [conda.el](https://github.com/necaris/conda.el) to help working with conda environments in Emacs. We have to put it somewhere before `emacs-jupyter` package and call `conda-env-activate`:
```emacs-lisp
(use-package conda
:straight t
:config
(setq conda-anaconda-home (expand-file-name "~/Programs/miniconda3/"))
(setq conda-env-home-directory (expand-file-name "~/Programs/miniconda3/"))
(setq conda-env-subdirectory "envs"))
(unless (getenv "CONDA_DEFAULT_ENV")
(conda-env-activate "base"))
```
If you have Anaconda installed on a custom path, as I do, you'd have to add these 3 `setq` in the `:config` section. Also, there is no point in activating environment if Emacs is somehow already lauched in an environment.
That'll give us Jupyter from a base conda environment.
### virtualenv {#virtualenv}
TODO
### Switching an environment {#switching-an-environment}
However, as you may have noticed, `emacs-jupyter` will always use the Python kernel found on startup. So if you switch to a new environment, the code will still be ran in the old one, which is not too convinient.
Fortunately, to fix that we have only to refresh the jupyter kernelspecs:
```emacs-lisp
(defun my/jupyter-refresh-kernelspecs ()
"Refresh Jupyter kernelspecs"
(interactive)
(jupyter-available-kernelspecs t))
```
Calling `M-x my/jupyter-refresh-kernelspecs` after a switch will give you a new kernel. Just keep in mind that the kernelspec seems to be attached to a session, so you'd also have to change the session name to get a new kernel.
```python
import sys
sys.executable
```
```text
/home/pavel/Programs/miniconda3/bin/python
```
```emacs-lisp
(conda-env-activate "ann")
```
```python
import sys
sys.executable
```
```text
/home/pavel/Programs/miniconda3/bin/python
```
```emacs-lisp
(my/jupyter-refresh-kernelspecs)
```
```python
import sys
sys.executable
```
```text
/home/pavel/Programs/miniconda3/envs/ann/bin/python
```
## Programming {#programming}
To test if everything is working correctly, run `M-x jupyter-run-repl`, which should give you a REPL with a chosen kernel. If so, we can finally start using Python in org mode.
```text
#+begin_src python :session hello :async yes
print('Hello, world!')
#+end_src
#+RESULTS:
: Hello, world!
#+end_src
```
To avoid repeating similar arguments for the src block, we can set the `header-args` property at the start of the file:
```text
#+PROPERTY: header-args:python :session hello
#+PROPERTY: header-args:python+ :async yes
```
When a kernel is initialized, an associated REPL buffer is also created with a name like `*jupyter-repl[python 3.9.2]-hello*`. That may also come in handy, although you may prefer running a standalone REPL, doing which will be discussed further.
Also, one advantage of emacs-jupyter is that kernel requests for input are queried through the minibuffer. So, you can run a code like this:
```text
#+begin_src python
name = input('Name: ')
print(f'Hello, {name}!')
#+end_src
#+RESULTS:
: Hello, Pavel!
```
without any additional setup.
## Code output {#code-output}
### Images {#images}
Image output should work out of box. Run `M-x org-toggle-inline-images` (`C-c C-x C-v`) after the execution to see the image inline.
```text
#+begin_src python
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.plot([1, 2, 3, 4], [1, 4, 2, 3])
pass
#+end_src
#+RESULTS:
[[file:./.ob-jupyter/86b3c5e1bbaee95d62610e1fb9c7e755bf165190.png]]
```
However, there is some room for improvement. First, you can add the following hook if you don't want press this awkward keybinding every time:
```emacs-lisp
(add-hook 'org-babel-after-execute-hook 'org-redisplay-inline-images)
```
Second, we may override the image save path like this:
```text
#+begin_src python :file img/hello.png
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.plot([1, 2, 3, 4], [1, 4, 2, 3])
pass
#+end_src
#+RESULTS:
[[file:img/hello.png]]
```
That can save you a `savefig` call if the image has to be used somewhere further.
Finally, by default the image has tranparent background and ridiculously small size. That can be fixed with some matplotlib settings:
```python
import matplotlib as mpl
mpl.rcParams['figure.dpi'] = 200
mpl.rcParams['figure.facecolor'] = '1'
```
At the same time, we can set image width to prevent images from becoming too large. I prefer to do it inside a `emacs-lisp` code block in the same org file:
```emacs-lisp
(setq-local org-image-actual-width '(1024))
```
### Basic tables {#basic-tables}
If you are evaluating something like pandas DataFrame, it will be outputted in the HTML format, wrapped in the `begin_export` block. To view the data in text format, you can set `:display plain`:
```text
#+begin_src python :display plain
import pandas as pd
pd.DataFrame({"a": [1, 2], "b": [3, 4]})
#+end_src
#+RESULTS:
: a b
: 0 1 3
: 1 2 4
```
Another solution is to use something like the [tabulate](https://pypi.org/project/tabulate/) package:
```text
#+begin_src python
import pandas as pd
import tabulate
df = pd.DataFrame({"a": [1, 2], "b": [3, 4]})
print(tabulate.tabulate(df, headers=df.columns, tablefmt="orgtbl"))
#+end_src
#+RESULTS:
: | | a | b |
: |----+-----+-----|
: | 0 | 1 | 3 |
: | 1 | 2 | 4 |
```
### HTML & other rich output {#html-and-other-rich-output}
Yet another solution is to use emacs-jupyter's option `:pandoc t`, which invokes pandoc to convert HTML, LaTeX and Markdown to Org. Predictably, this is slower than the options above.
```text
#+begin_src python :pandoc t
import pandas as pd
df = pd.DataFrame({"a": [1, 2], "b": [3, 4]})
df
#+end_src
#+RESULTS:
:RESULTS:
| | a | b |
|---+---+---|
| 0 | 1 | 3 |
| 1 | 2 | 4 |
:END:
```
Finally, every once in a while I have to view an actual, unconverted HTML in a browser, e.g. when using [folium](https://python-visualization.github.io/folium/) or [displaCy](https://spacy.io/usage/visualizers).
To do that, I've written a small function, which performs `xdg-open` on the HTML export block under the cursor:
```emacs-lisp
(setq my/org-view-html-tmp-dir "/tmp/org-html-preview/")
(use-package f
:straight t)
(defun my/org-view-html ()
(interactive)
(let ((elem (org-element-at-point))
(temp-file-path (concat my/org-view-html-tmp-dir (number-to-string (random (expt 2 32))) ".html")))
(cond
((not (eq 'export-block (car elem)))
(message "Not in an export block!"))
((not (string-equal (plist-get (car (cdr elem)) :type) "HTML"))
(message "Export block is not HTML!"))
(t (progn
(f-mkdir my/org-view-html-tmp-dir)
(f-write (plist-get (car (cdr elem)) :value) 'utf-8 temp-file-path)
(start-process "org-html-preview" nil "xdg-open" temp-file-path))))))
```
`f.el` is used by a lot of packages, including the above mentioned `conda.el`, so you probably already have it installed.
Put a cursor on the `begin_export html` block and run `M-x my/org-view-html`.
There also [seems to be widgets support](https://github.com/nnicandro/emacs-jupyter#building-the-widget-support-experimental) in emacs-jupyter, but I wasn't able to make it work.
### DataFrames {#dataframes}
Last but not least option I want to mention here is specifically about pandas' DataFrames. There aren't many good options to view the full dataframe inside Emacs. The way I can think of is to save the dataframe in csv and view it with `csv-mode`.
However, there are standalone packages to view dataframes. My favorite one is [dtale](https://github.com/man-group/dtale), which is a Flask + React app designed just for that purpose. It has a rather extensive list of features, including charting, basic statistical instruments, filters, etc. [Here](http://alphatechadmin.pythonanywhere.com/dtale/main/1) is an online demo.
And example usage:
```python
import dtale
d = dtale.show(df)
d.open_browser() # Or get an URL from d._url
```
Another notable alternative is [PandasGUI](https://github.com/adamerose/pandasgui), which, as one can guess, is a GUI (PyQt5) application, although it uses QtWebEngine inside.
The obvious downside is, of course, that these applications are huge ones with lots of dependencies, and they have to be installed in the same environment as your project.
## Remote kernels {#remote-kernels}
There are yet some problems in the current configuration.
- Input/output handling is far from perfect. For instance, (at least in my configuration) Emacs tends to get slow for log-like outputs, e.g. Keras with `verbose=2`. It may even hang if an output is a one long line.
- `ipdb` behaves rather awkwardly if called from an `src` block, although it at least will let you type `quit`.
- Whenever you close Emacs, kernels are stopped, so you'd have to execute the code again on the next start.
### Using a "remote" kernel {#using-a-remote-kernel}
For the reasons above I prefer to use a standalone kernel. To do that, execute the following command in the path and environment you need:
```bash
jupyter kernel --kernel=python
```
After the kernel is launched, put the path to the connection file into the `:session` header and press `C-c C-c` to refresh the setup:
```text
#+PROPERTY: header-args:python :session /home/pavel/.local/share/jupyter/runtime/kernel-e770599c-2c98-429b-b9ec-4d1ddf5fc16c.json
```
To open a REPL, run `M-x jupyter-connect-repl` and select the given JSON. Or launch a standalone REPL like this:
```bash
jupyter qtconsole --existing kernel-e770599c-2c98-429b-b9ec-4d1ddf5fc16c.json
```
### Some automation {#some-automation}
Now, I wouldn't use Emacs if it was impossible to automate at least some the listed steps. So here are some functions I've written.
First, we need to get open ports on the system:
```emacs-lisp
(defun my/get-open-ports ()
(mapcar
#'string-to-number
(split-string (shell-command-to-string "ss -tulpnH | awk '{print $5}' | sed -e 's/.*://'") "\n")))
```
Then, list the available kernel JSONs:
```emacs-lisp
(setq my/jupyter-runtime-folder (expand-file-name "~/.local/share/jupyter/runtime"))
(defun my/list-jupyter-kernel-files ()
(mapcar
(lambda (file) (cons (car file) (cdr (assq 'shell_port (json-read-file (car file))))))
(sort
(directory-files-and-attributes my/jupyter-runtime-folder t ".*kernel.*json$")
(lambda (x y) (not (time-less-p (nth 6 x) (nth 6 y)))))))
```
And query the user for an running kernel:
```emacs-lisp
(defun my/select-jupyter-kernel ()
(let ((ports (my/get-open-ports))
(files (my/list-jupyter-kernel-files)))
(completing-read
"Jupyter kernels: "
(seq-filter
(lambda (file)
(member (cdr file) ports))
files))))
```
After which we can use the `my/select-jupyter-kernel` function however we want:
```emacs-lisp
(defun my/insert-jupyter-kernel ()
"Insert a path to an active Jupyter kernel into the buffer"
(interactive)
(insert (my/select-jupyter-kernel)))
(defun my/jupyter-connect-repl ()
"Open an emacs-jupyter REPL, connected to a Jupyter kernel"
(interactive)
(jupyter-connect-repl (my/select-jupyter-kernel) nil nil nil t))
(defun my/jupyter-qtconsole ()
"Open Jupyter QtConsole, connected to a Jupyter kernel"
(interactive)
(start-process "jupyter-qtconsole" nil "setsid" "jupyter" "qtconsole" "--existing"
(file-name-nondirectory (my/select-jupyter-kernel))))
```
The first function, which simply inserts the path to the kernel, is meant to be used on the `:session` header. I can go even further and locate the header automatically, but that's an idea for the next time.
The second one opens a REPL provided by emacs-jupyter. The `t` argument is necessary to pop up the REPL immediately.
The last one launches Jupyter QtConsole. `setsid` is required to run a console in a new session, so it won't close together with Emacs.
### Cleaning up {#cleaning-up}
I've also noticed that there are JSON files left in the runtime folder whenever kernel isn't stopped correctly. So here is a cleanup function.
```emacs-lisp
(defun my/jupyter-cleanup-kernels ()
(interactive)
(let* ((ports (my/get-open-ports))
(files (my/list-jupyter-kernel-files))
(to-delete (seq-filter
(lambda (file)
(not (member (cdr file) ports)))
files)))
(when (and (length> to-delete 0)
(y-or-n-p (format "Delete %d files?" (length to-delete))))
(dolist (file to-delete)
(delete-file (car file)))))
```
## Export {#export}
A lot of articles were written on the subject of Org Mode export, so I will just cover my particular setup.
### HTML {#html}
Export to html is pretty straightforward and should work out of box with `M-x org-html-export-to-html`. However, we can improve the output a bit.
First, we can add a custom CSS to the file:
```text
#+HTML_HEAD: <link rel="stylesheet" type="text/css" href="https://gongzhitaao.org/orgcss/org.css"/>
```
### LaTeX {#latex}
### ipynb {#ipynb}

245
org/2021-04-07-org-python.org
View file

@ -1,7 +1,7 @@
#+HUGO_SECTION: posts
#+HUGO_BASE_DIR: ../
#+TITLE: Replacing Jupyter Notebook with Org Mode
#+DATE: 2021-04-07
#+DATE: 2021-04-08
#+HUGO_DRAFT: true
#+HUGO_TAGS: emacs
#+HUGO_TAGS: org
@ -9,20 +9,20 @@
#+PROPERTY: header-args:python+ :exports both
#+PROPERTY: header-args:python+ :tangle yes
#+PROPERTY: header-args:python+ :async yes
#+PROPERTY: header-args :exports both
#+PROPERTY: header-args:python+ :eval never-export
#+PROPERTY: header-args:emacs-lisp+ :eval never-export
* Why?
* Basic setup
There are multiple ways of doing literate programming with Python & Org Mode, [[https://github.com/millejoh/emacs-ipython-notebook][ein]] being one of the notable alternatives.
There are multiple ways of doing literate programming with Python in Emacs, [[https://github.com/millejoh/emacs-ipython-notebook][ein]] being one of the notable alternatives.
I go with the [[https://github.com/nnicandro/emacs-jupyter][emacs-jupyter]] package. Installing it is pretty straightforward, I use =use-package= with =straight.el=:
However, I go with the [[https://github.com/nnicandro/emacs-jupyter][emacs-jupyter]] package. Install it however you install packages in Emacs, here is my preffered way with =use-package= and =straight.el=:
#+begin_src emacs-lisp :eval no
(use-package jupyter
:straight t)
#+end_src
Then, we have to enable languages for =org-babel=. The following isn't the best practice for startup performance time, but the least problematic in my experience.
Then, we have to enable languages for =org-babel=. Put the following in your org mode config section:
#+begin_src emacs-lisp :eval no
(org-babel-do-load-languages
'org-babel-load-languages
@ -38,7 +38,7 @@ That adds Org source blocks with names like ~jupyter-LANG~, e.g. ~jupyter-python
(org-babel-jupyter-override-src-block "python")
#+end_src
That overrides built-in ~python~ block with ~jupyter-python~.
That overrides the built-in ~python~ block with ~jupyter-python~.
If you use [[https://github.com/astahlman/ob-async][ob-async]], you have to set ~jupyter-LANG~ blocks as ignored by this package, because emacs-jupyter has async execution of its own.
#+begin_src emacs-lisp :eval no
@ -47,7 +47,7 @@ If you use [[https://github.com/astahlman/ob-async][ob-async]], you have to set
* Environments
So, we've set up a basic emacs-jupyter configuration.
The catch here is that Jupyter should be available on Emacs startup (at the time of evaluation of the =emacs-jupyter= package, to be precise). That means, if you are launching Emacs with something like application launcher, global Python & Jupyter will be used.
The catch here is that Jupyter should be available on Emacs startup (at the time of evaluation of the =emacs-jupyter= package, to be precise). That means, if you are launching Emacs with something like an application launcher, global Python & Jupyter will be used.
#+begin_src python :eval no
import sys
@ -136,6 +136,7 @@ print('Hello, world!')
#+RESULTS:
: Hello, world!
#+end_src
#+end_example
To avoid repeating similar arguments for the src block, we can set the =header-args= property at the start of the file:
@ -162,7 +163,7 @@ without any additional setup.
* Code output
** Images
Image output show work out of box. Run =M-x org-toggle-inline-images= (=C-c C-x C-v=) after the execution to see the image inline.
Image output should work out of box. Run =M-x org-toggle-inline-images= (=C-c C-x C-v=) after the execution to see the image inline.
#+begin_example
#+begin_src python
import matplotlib.pyplot as plt
@ -203,11 +204,11 @@ mpl.rcParams['figure.dpi'] = 200
mpl.rcParams['figure.facecolor'] = '1'
#+end_src
Then, we can set image width to prevent images from becoming too large. I prefer to do it inside a =emacs-lisp= code block in the same org file:
At the same time, we can set image width to prevent images from becoming too large. I prefer to do it inside a =emacs-lisp= code block in the same org file:
#+begin_src emacs-lisp
(setq-local org-image-actual-width '(1024))
#+end_src
** Tables
** Basic tables
If you are evaluating something like pandas DataFrame, it will be outputted in the HTML format, wrapped in the =begin_export= block. To view the data in text format, you can set =:display plain=:
#+begin_example
#+begin_src python :display plain
@ -221,7 +222,7 @@ pd.DataFrame({"a": [1, 2], "b": [3, 4]})
: 1 2 4
#+end_example
Another solution is to use the [[https://pypi.org/project/tabulate/][tabulate]] package:
Another solution is to use something like the [[https://pypi.org/project/tabulate/][tabulate]] package:
#+begin_example
#+begin_src python
import pandas as pd
@ -254,7 +255,9 @@ df
:END:
#+end_example
Finally, every once in a while I have to view an actual HTML in a browser, e.g. when using [[https://python-visualization.github.io/folium/][folium]]. To do that, I've written a small function, which performs =xdg-open= on the HTML export block under the cursor:
Finally, every once in a while I have to view an actual, unconverted HTML in a browser, e.g. when using [[https://python-visualization.github.io/folium/][folium]] or [[https://spacy.io/usage/visualizers][displaCy]].
To do that, I've written a small function, which performs =xdg-open= on the HTML export block under the cursor:
#+begin_src emacs-lisp :eval no
(setq my/org-view-html-tmp-dir "/tmp/org-html-preview/")
@ -277,11 +280,221 @@ Finally, every once in a while I have to view an actual HTML in a browser, e.g.
#+end_src
=f.el= is used by a lot of packages, including the above mentioned =conda.el=, so you probably already have it installed.
Put a cursor on an export block and run =M-x my/org-view-html=.
Put a cursor on the =begin_export html= block and run =M-x my/org-view-html=.
There also [[https://github.com/nnicandro/emacs-jupyter#building-the-widget-support-experimental][seems to be widgets support]] in emacs-jupyter, but I wan't able to make it work.
There also [[https://github.com/nnicandro/emacs-jupyter#building-the-widget-support-experimental][seems to be widgets support]] in emacs-jupyter, but I wasn't able to make it work.
** DataFrames
Last but not least option I want to mention here is specifically about pandas' DataFrames. There aren't many good options to view the full dataframe inside Emacs. The way I can think of is to save the dataframe in csv and view it with =csv-mode=.
However, there are standalone packages to view dataframes. My favorite one is [[https://github.com/man-group/dtale][dtale]], which is a Flask + React app designed just for that purpose. It has a rather extensive list of features, including charting, basic statistical instruments, filters, etc. [[http://alphatechadmin.pythonanywhere.com/dtale/main/1][Here]] is an online demo.
And example usage:
#+begin_src python :eval no
import dtale
d = dtale.show(df)
d.open_browser() # Or get an URL from d._url
#+end_src
Another notable alternative is [[https://github.com/adamerose/pandasgui][PandasGUI]], which, as one can guess, is a GUI (PyQt5) application, although it uses QtWebEngine inside.
The obvious downside is, of course, that these applications are huge ones with lots of dependencies, and they have to be installed in the same environment as your project.
* Remote kernels
There are yet some problems in the current configuration.
- Input/output handling is far from perfect. For instance, (at least in my configuration) Emacs tends to get slow for log-like outputs, e.g. Keras with ~verbose=2~. It may even hang if an output is a one long line.
- =ipdb= behaves rather awkwardly if called from an =src= block, although it at least will let you type =quit=.
- Whenever you close Emacs, kernels are stopped, so you'd have to execute the code again on the next start.
** Using a "remote" kernel
For the reasons above I prefer to use a standalone kernel. To do that, execute the following command in the path and environment you need:
#+begin_src bash
jupyter kernel --kernel=python
#+end_src
#+RESULTS:
#+begin_example
[KernelApp] Starting kernel 'python'
[KernelApp] Connection file: /home/pavel/.local/share/jupyter/runtime/kernel-e770599c-2c98-429b-b9ec-4d1ddf5fc16c.json
[KernelApp] To connect a client: --existing kernel-e770599c-2c98-429b-b9ec-4d1ddf5fc16c.json
#+end_example
After the kernel is launched, put the path to the connection file into the ~:session~ header and press =C-c C-c= to refresh the setup:
#+begin_example
#+PROPERTY: header-args:python :session /home/pavel/.local/share/jupyter/runtime/kernel-e770599c-2c98-429b-b9ec-4d1ddf5fc16c.json
#+end_example
To open a REPL, run =M-x jupyter-connect-repl= and select the given JSON. Or launch a standalone REPL like this:
#+begin_src bash
jupyter qtconsole --existing kernel-e770599c-2c98-429b-b9ec-4d1ddf5fc16c.json
#+end_src
** Some automation
Now, I wouldn't use Emacs if it wasn't possible to automate at least some the listed steps. So here are some functions I've written.
First, we need to get open ports on the system:
#+begin_src emacs-lisp
(defun my/get-open-ports ()
(mapcar
#'string-to-number
(split-string (shell-command-to-string "ss -tulpnH | awk '{print $5}' | sed -e 's/.*://'") "\n")))
#+end_src
Then, list the available kernel JSONs:
#+begin_src emacs-lisp
(setq my/jupyter-runtime-folder (expand-file-name "~/.local/share/jupyter/runtime"))
(defun my/list-jupyter-kernel-files ()
(mapcar
(lambda (file) (cons (car file) (cdr (assq 'shell_port (json-read-file (car file))))))
(sort
(directory-files-and-attributes my/jupyter-runtime-folder t ".*kernel.*json$")
(lambda (x y) (not (time-less-p (nth 6 x) (nth 6 y)))))))
#+end_src
And query the user for an running kernel:
#+begin_src emacs-lisp
(defun my/select-jupyter-kernel ()
(let ((ports (my/get-open-ports))
(files (my/list-jupyter-kernel-files)))
(completing-read
"Jupyter kernels: "
(seq-filter
(lambda (file)
(member (cdr file) ports))
files))))
#+end_src
After which we can use the ~my/select-jupyter-kernel~ function however we want:
#+begin_src emacs-lisp
(defun my/insert-jupyter-kernel ()
"Insert a path to an active Jupyter kernel into the buffer"
(interactive)
(insert (my/select-jupyter-kernel)))
(defun my/jupyter-connect-repl ()
"Open emacs-jupyter REPL, connected to a Jupyter kernel"
(interactive)
(jupyter-connect-repl (my/select-jupyter-kernel) nil nil nil t))
(defun my/jupyter-qtconsole ()
"Open Jupyter QtConsole, connected to a Jupyter kernel"
(interactive)
(start-process "jupyter-qtconsole" nil "setsid" "jupyter" "qtconsole" "--existing"
(file-name-nondirectory (my/select-jupyter-kernel))))
#+end_src
The first function, which simply inserts the path to the kernel, is meant to be used on the ~:session~ header. One can go even further and locate the header automatically, but that's an idea for the next time.
The second one opens a REPL provided by emacs-jupyter. The =t= argument is necessary to pop up the REPL immediately.
The last one launches Jupyter QtConsole. =setsid= is required to run the program in a new session, so it won't close together with Emacs.
** Cleaning up
I've also noticed that there are JSON files left in the runtime folder whenever kernel isn't stopped correctly. So here is a cleanup function.
#+begin_src emacs-lisp
(defun my/jupyter-cleanup-kernels ()
(interactive)
(let* ((ports (my/get-open-ports))
(files (my/list-jupyter-kernel-files))
(to-delete (seq-filter
(lambda (file)
(not (member (cdr file) ports)))
files)))
(when (and (length> to-delete 0)
(y-or-n-p (format "Delete %d files?" (length to-delete))))
(dolist (file to-delete)
(delete-file (car file)))))
#+end_src
* Export
A lot of articles have been written already on the subject of Org Mode export, so I will just cover my particular setup.
** HTML
** LaTeX
Export to html is pretty straightforward and should work out of box with =M-x org-html-export-to-html=. However, we can improve the output a bit.
First, we can add a custom CSS to the file. I like this one:
#+begin_example
#+HTML_HEAD: <link rel="stylesheet" type="text/css" href="https://gongzhitaao.org/orgcss/org.css"/>
#+end_example
To get a syntax highlighting, we need the =htmlize= package:
#+begin_src emacs-lisp
(use-package htmlize
:straight t
:after ox
:config
(setq org-html-htmlize-output-type 'css))
#+end_src
If you use the [[https://github.com/Fanael/rainbow-delimiters][rainbow-delimeters]] package, like I do, default colors for delimiters may not look good with the light theme. The easiest way I see to fix that is to put an HTML snippet like this in a =begin_export html= block:
#+begin_src html
<style type="text/css">
.org-rainbow-delimiters-depth-1, .org-rainbow-delimiters-depth-2, .org-rainbow-delimiters-depth-3, .org-rainbow-delimiters-depth-4 {
color: black
}
</style>
#+end_src
Of course, you can also modify the custom CSS, but that looks like a good way to provide a standalone HTML.
Which brings me to the point of this option - exporting to a standalone HTML is an easy way to share a code with someone who doesn't use Emacs, at least one way.
** LaTeX -> pdf
Despite the fact that I use LaTeX quite extensively, I don't like to add another layer of complexity here and 98% of the time write plain =.tex= files. LaTeX by itself provides many good options whenever you need to write a document together with some data or source code, contrary to "traditional" text processors.
Nevertheless, I want to get at least a tolerable pdf, so here is piece of my config with some inline comments.
#+begin_src emacs-lisp
(defun my/setup-org-latex ()
(setq org-latex-compiler "xelatex") ;; Probably not necessary
(setq org-latex-pdf-process '("latexmk -outdir=%o %f")) ;; Use latexmk
(setq org-latex-listings 'minted) ;; Use minted to highlight source code
(setq org-latex-minted-options ;; Some minted options I like
'(("breaklines" "true")
("tabsize" "4")
("autogobble")
("linenos")
("numbersep" "0.5cm")
("xleftmargin" "1cm")
("frame" "single")))
;; Use extarticle without the default packages
(add-to-list 'org-latex-classes
'("org-plain-extarticle"
"\\documentclass{extarticle}
[NO-DEFAULT-PACKAGES]
[PACKAGES]
[EXTRA]"
("\\section{%s}" . "\\section*{%s}")
("\\subsection{%s}" . "\\subsection*{%s}")
("\\subsubsection{%s}" . "\\subsubsection*{%s}")
("\\paragraph{%s}" . "\\paragraph*{%s}")
("\\subparagraph{%s}" . "\\subparagraph*{%s}"))))
;; Make sure to eval the function when org-latex-classes list already exists
(with-eval-after-load 'ox-latex
(my/setup-org-latex))
#+end_src
In the document itself, add the following headers:
#+begin_example
#+LATEX_CLASS: org-plain-extarticle
#+LATEX_CLASS_OPTIONS: [a4paper, 14pt]
#+end_example
14pt size is required by certain state standards here for some reason.
After which you can put whatever you want in the preamble with =LATEX_HEADER=. My workflow with LaTeX is to write a bunch of =.sty= files beforehand and import the necessary ones in the preamble. [[https://github.com/SqrtMinusOne/LaTeX_templates][Here]] is the repo with these files, although quite predictably, it's a mess. At any rate, I have to write something like the following in the target Org file:
#+begin_example
#+LATEX_HEADER: \usepackage{styles/generalPreamble}
#+LATEX_HEADER: \usepackage{styles/reportFormat}
#+LATEX_HEADER: \usepackage{styles/mintedSourceCode}
#+LATEX_HEADER: \usepackage{styles/russianLocale}
#+end_example
** ipynb
One last export backend I want to mention is [[https://github.com/jkitchin/ox-ipynb][ox-ipynb]], which allows exporting Org documents to Jupyter notebooks. Sometimes it works, sometimes it doesn't.
Also the package isn't on MELPA, so you have to install it from the repo directly.
#+begin_src emacs-lisp :eval no
(use-package ox-ipynb
:straight (:host github :repo "jkitchin/ox-ipynb")
:after ox)
#+end_src
To (try to) do export, run =M-x ox-ipynb-export-org-file-ipynb-file=.