diff --git a/content/posts/2021-02-27-gmail.md b/content/posts/2021-02-27-gmail.md new file mode 100644 index 0000000..1694aa3 --- /dev/null +++ b/content/posts/2021-02-27-gmail.md @@ -0,0 +1,408 @@ ++++ +title = "Multiple Gmail accounts & labels with Emacs" +author = ["Pavel"] +date = 2021-02-27 +draft = true ++++ + +## Intro {#intro} + +For quite some time, e-mail seemed like an anomaly in my workflow. I am a long time Gmail user, and my decade-old account has a somewhat formidable quantity of labels and filters. My messages are often assigned multiple labels, and I also like to keep only a bunch of messages in the inbox. + +Although, in my opinion, Gmail web UI was and still is leagues ahead of many of its competitors and even allows keyboard-centric workflow, it's awkward to use with keyboard-driven browser, and for no money on Earth I would enable browser notifications. + +Any classical IMAP/SMTP client is hard to use in my case, because a message with multiple labels is copied to IMAP folders for each of the label plus the inbox folder, and the copies look like different messages from the client side. For example, a message can be read in one label and unread in another. + +For a few years my solution was [Mailspring](https://getmailspring.com/), which provides first-class support for labels. However, it has a feature to deploy [spy pixels](https://www.bbc.com/news/technology-56071437) on emails (and offers no protection from them, obviously), the client is Electron-based with mouse-driven interface, and the sync engine was closed-source at the time. + +So, I found an alternative in Emacs+notmuch+lieer and ditched one more proprierary app (the last big one I can't let go is DataGrip). + +{{< figure src="/ox-hugo/main.png" >}} + +{{< figure src="/ox-hugo/mail.png" >}} + +Notmuch's tags are just as advanced as Gmail's labels, so I have basically the same mail structure acccessible from Emacs, Gmail Android client and even the web UI, when I don't have access to the first two. + +Also, I think the setup I describe here is pretty straightforward and less complex than many I encountered, but my impression is not the most reliable source of such knowledge. + +At any case, what follows is a description of my current workflow with instructions of varying levels of precision of how to get there. + + +## Setting up {#setting-up} + + +### Gmail {#gmail} + +Before we start, some setup is required for Gmail account. + +First, as there is no way to enable SMTP without IMAP on Gmail, you have to set "Enable IMAP" in the "Forwarding and POP/IMAP" tab in the settings. If you use two-factor auth, generate an [app password](https://support.google.com/accounts/answer/185833?hl=en). + +Also, make sure your labels do not contain whitespaces, because if they do, you will have to type them in quotes all the time. + + +### lieer {#lieer} + +[lieer](https://github.com/gauteh/lieer) (formerly gmailieer) is a program which uses Gmail API to download email and synchronize Gmail labels with notmuch tags. Because of its usage of Gmail API instead of IMAP, there are no problems with duplicating emails in different labels, etc. + +As I need to use multiple versions of Python & Node.js for other reasons, I manage my installations of them with [Anaconda](https://anaconda.org) (Miniconda, to be precise). You may instead use [venv](https://docs.python.org/3/library/venv.html) or even the system-wide installation of Python and omit the `conda` clauses, but in my experience Anaconda makes life easier in that regard. + +```bash +# Create an environment with name "mail" +conda create --name mail +# Activate the environment +conda activate mail +# Install Python +conda install python +# Download and install lieer +git clone https://github.com/gauteh/lieer.git +cd lieer +pip install . +``` + +After which we may check if the `gmi` executable is available: + +```bash +which gmi +``` + + +### Notmuch {#notmuch} + +[Notmuch](https://notmuchmail.org/) is present in most of the package repositories, so you can install it with your package manager, which is `pacman` in my case. + +```bash +sudo pacman -S notmuch +``` + +After the installation, run `notmuch setup`. That will inquire the parameters and create the `.notmuch-config` file with the answers. + +```bash +Your full name [Pavel]: Pavel Korytov +Your primary email address [pavel@pdsk.(none)]: thexcloud@gmail.com +Additional email address [Press 'Enter' if none]: +Top-level directory of your email archive [/home/pavel/mail]: /home/pavel/Mail +Tags to apply to all new messages (separated by spaces) [unread inbox]: new +Tags to exclude when searching messages (separated by spaces) [deleted spam]: +``` + +It is important to set the `new` tag for the new messages instead of the default `unread` and `inbox`. + +Next, add the rule to ignore JSON files to the `[new]` section of the `.notmuch-config` file, so it would look like this: + +```bash +[new] +tags=new +ignore=/.*[.](json|lock|bak)$/ +``` + +That is needed to ignore the lieer config files. Although, as I have noticed, notmuch is generally pretty good at detecting wrong files in its directories, an explicit ignore rule won't hurt. + +Now, create the mail directory and run the [notmuch new](https://notmuchmail.org/manpages/notmuch-new-1/) command. As notmuch has probably already noticed you, it uses the [maildir](https://en.wikipedia.org/wiki/Maildir) format, which basically means that one message is stored in one file. + +```bash +# The same directory mentioned in the 4th question +mkdir ~/Mail +# Initialize notmuch +notmuch new +``` + + +### Add an account {#add-an-account} + +After that, we can create a directory for a mail account and initialize lieer. + +```bash +cd ~/Mail +# Use whatever name you want +mkdir thexcloud +cd thexcloud +# Intialize lieer +gmi init thexcloud@gmail.com +``` + +Running `gmi init` will run an OAuth authentication to your Gmail account. The credentials will be stored in `.credentials.gmailieer.json` file, so make sure not to expose it somewhere. + +We also can add a few settings for lieer, which will make the life easier. First, dots seem to be less awkward to type than slashes for the nested tags: + +```bash +gmi set --replace-slash-with-dot +``` + +Then, we don't want the `new` tag to be pushed back to Gmail + +```bash +gmi set --ignore-tags-local new +``` + +Now we can finally download the mail directory. To initiate the download, run + +```bash +gmi sync +``` + +The first download can easily take several hours, depending on the size of your email and the speed of your internet connection, but subsequent runs will be much faster. + +The last thing to do here is to add the `gmi sync` command to notmuch's [pre-new hook](https://notmuchmail.org/manpages/notmuch-hooks-5/), so that the email will be synchronized on the `notmuch new` command. + +```bash +# Create the hooks folder +mkdir -p ~/Mail/.notmuch/hooks +# Create the file +cd ~/Mail/.notmuch/hooks +cat > pre-new < post-new < $CHECK_FILE +``` + +The script is launched with cron every 5 minutes: + +```bash +*/5 * * * * bash /home/pavel/bin/scripts/check-email +``` + +Here's how the notification looks like: +![](/ox-hugo/notification.png) + + +## Caveats {#caveats} + +- [lieer](https://github.com/gauteh/lieer#caveats) has an extensive list of caveats concerning Gmail API +- Make sure that you understand the [implications](https://github.com/gauteh/lieer#changing-ignored-tags-and-translation-after-initial-sync) of lieer's `--ignore-tags-locally` and `--ignore-tags-remote` +- If two of your accounts receive the same email, it will be stored as one email in notmuch, so tags from these accounts will be merged and pushed back on the next sync. To solve that, you can set tags from one account to be ignored on the rest of accounts +- A sent email is being downloaded again on the next sync. Not a great deal, but it is somewhat annoying to download recently sent attachments. diff --git a/content/posts/hello-world.md b/content/posts/hello-world.md index 45baf4c..9f8330b 100644 --- a/content/posts/hello-world.md +++ b/content/posts/hello-world.md @@ -1,7 +1,7 @@ +++ title = "Hello, world!" author = ["Pavel"] -date = 2020-02-01 +date = 2021-02-01 draft = false +++ diff --git a/org/2021-02-27-gmail.org b/org/2021-02-27-gmail.org new file mode 100644 index 0000000..ee839e8 --- /dev/null +++ b/org/2021-02-27-gmail.org @@ -0,0 +1,342 @@ +#+HUGO_SECTION: posts +#+HUGO_BASE_DIR: ../ +#+TITLE: Multiple Gmail accounts & labels with Emacs +#+DATE: 2021-02-27 +#+HUGO_DRAFT: true + +#+PROPERTY: header-args :exports both + +* Intro +For quite some time, e-mail seemed like an anomaly in my workflow. I am a long time Gmail user, and my decade-old account has a somewhat formidable quantity of labels and filters. My messages are often assigned multiple labels, and I also like to keep only a bunch of messages in the inbox. + +Although, in my opinion, Gmail web UI was and still is leagues ahead of many of its competitors and even allows keyboard-centric workflow, it's awkward to use with keyboard-driven browser, and for no money on Earth I would enable browser notifications. + +Any classical IMAP/SMTP client is hard to use in my case, because a message with multiple labels is copied to IMAP folders for each of the label plus the inbox folder, and the copies look like different messages from the client side. For example, a message can be read in one label and unread in another. + +For a few years my solution was [[https://getmailspring.com/][Mailspring]], which provides first-class support for labels. However, it has a feature to deploy [[https://www.bbc.com/news/technology-56071437][spy pixels]] on emails (and offers no protection from them, obviously), the client is Electron-based with mouse-driven interface, and the sync engine was closed-source at the time. + +So, I found an alternative in Emacs+notmuch+lieer and ditched one more proprierary app (the last big one I can't let go is DataGrip). + +[[file:images/gmail/main.png]] + +[[file:images/gmail/mail.png]] + +Notmuch's tags are just as advanced as Gmail's labels, so I have basically the same mail structure acccessible from Emacs, Gmail Android client and even the web UI, when I don't have access to the first two. + +Also, I think the setup I describe here is pretty straightforward and less complex than many I encountered, but my impression is not the most reliable source of such knowledge. + +At any case, what follows is a description of my current workflow with instructions of varying levels of precision of how to get there. +* Setting up +** Gmail +Before we start, some setup is required for Gmail account. + +First, as there is no way to enable SMTP without IMAP on Gmail, you have to set "Enable IMAP" in the "Forwarding and POP/IMAP" tab in the settings. If you use two-factor auth, generate an [[https://support.google.com/accounts/answer/185833?hl=en][app password]]. + +Also, make sure your labels do not contain whitespaces, because if they do, you will have to type them in quotes all the time. +** lieer +[[https://github.com/gauteh/lieer][lieer]] (formerly gmailieer) is a program which uses Gmail API to download email and synchronize Gmail labels with notmuch tags. Because of its usage of Gmail API instead of IMAP, there are no problems with duplicating emails in different labels, etc. + +As I need to use multiple versions of Python & Node.js for other reasons, I manage my installations of them with [[https://anaconda.org][Anaconda]] (Miniconda, to be precise). You may instead use [[https://docs.python.org/3/library/venv.html][venv]] or even the system-wide installation of Python and omit the =conda= clauses, but in my experience Anaconda makes life easier in that regard. + +#+begin_src bash :eval no +# Create an environment with name "mail" +conda create --name mail +# Activate the environment +conda activate mail +# Install Python +conda install python +# Download and install lieer +git clone https://github.com/gauteh/lieer.git +cd lieer +pip install . +#+end_src + +After which we may check if the =gmi= executable is available: +#+begin_src bash +which gmi +#+end_src + +#+RESULTS: +: /home/pavel/Programs/miniconda3/envs/mail/bin/gmi +** Notmuch +[[https://notmuchmail.org/][Notmuch]] is present in most of the package repositories, so you can install it with your package manager, which is =pacman= in my case. +#+begin_src bash :eval no +sudo pacman -S notmuch +#+end_src + +After the installation, run =notmuch setup=. That will inquire the parameters and create the =.notmuch-config= file with the answers. +#+begin_src bash :eval no +Your full name [Pavel]: Pavel Korytov +Your primary email address [pavel@pdsk.(none)]: thexcloud@gmail.com +Additional email address [Press 'Enter' if none]: +Top-level directory of your email archive [/home/pavel/mail]: /home/pavel/Mail +Tags to apply to all new messages (separated by spaces) [unread inbox]: new +Tags to exclude when searching messages (separated by spaces) [deleted spam]: +#+end_src +It is important to set the =new= tag for the new messages instead of the default =unread= and =inbox=. + +Next, add the rule to ignore JSON files to the =[new]= section of the =.notmuch-config= file, so it would look like this: +#+begin_src bash :eval no +[new] +tags=new +ignore=/.*[.](json|lock|bak)$/ +#+end_src + +That is needed to ignore the lieer config files. Although, as I have noticed, notmuch is generally pretty good at detecting wrong files in its directories, an explicit ignore rule won't hurt. + +Now, create the mail directory and run the [[https://notmuchmail.org/manpages/notmuch-new-1/][notmuch new]] command. As notmuch has probably already noticed you, it uses the [[https://en.wikipedia.org/wiki/Maildir][maildir]] format, which basically means that one message is stored in one file. +#+begin_src bash :eval no +# The same directory mentioned in the 4th question +mkdir ~/Mail +# Initialize notmuch +notmuch new +#+end_src +** Add an account +After that, we can create a directory for a mail account and initialize lieer. +#+begin_src bash :eval no +cd ~/Mail +# Use whatever name you want +mkdir thexcloud +cd thexcloud +# Intialize lieer +gmi init thexcloud@gmail.com +#+end_src +Running =gmi init= will run an OAuth authentication to your Gmail account. The credentials will be stored in =.credentials.gmailieer.json= file, so make sure not to expose it somewhere. + +We also can add a few settings for lieer, which will make the life easier. First, dots seem to be less awkward to type than slashes for the nested tags: +#+begin_src bash :eval no +gmi set --replace-slash-with-dot +#+end_src + +Then, we don't want the =new= tag to be pushed back to Gmail +#+begin_src bash :eval no +gmi set --ignore-tags-local new +#+end_src + +Now we can finally download the mail directory. To initiate the download, run +#+begin_src bash :eval no +gmi sync +#+end_src + +The first download can easily take several hours, depending on the size of your email and the speed of your internet connection, but subsequent runs will be much faster. + +The last thing to do here is to add the =gmi sync= command to notmuch's [[https://notmuchmail.org/manpages/notmuch-hooks-5/][pre-new hook]], so that the email will be synchronized on the =notmuch new= command. +#+begin_src bash :eval no +# Create the hooks folder +mkdir -p ~/Mail/.notmuch/hooks +# Create the file +cd ~/Mail/.notmuch/hooks +cat > pre-new < post-new < $CHECK_FILE +#+end_src + +The script is launched with cron every 5 minutes: +#+begin_src bash :eval no +*/5 * * * * bash /home/pavel/bin/scripts/check-email +#+end_src + +Here's how the notification looks like: +[[file:images/gmail/notification.png]] +* Caveats +- [[https://github.com/gauteh/lieer#caveats][lieer]] has an extensive list of caveats concerning Gmail API +- Make sure that you understand the [[https://github.com/gauteh/lieer#changing-ignored-tags-and-translation-after-initial-sync][implications]] of lieer's =--ignore-tags-locally= and =--ignore-tags-remote= +- If two of your accounts receive the same email, it will be stored as one email in notmuch, so tags from these accounts will be merged and pushed back on the next sync. To solve that, you can set tags from one account to be ignored on the rest of accounts +- A sent email is being downloaded again on the next sync. Not a great deal, but it is somewhat annoying to download recently sent attachments. diff --git a/org/2021/hello-world.org b/org/hello-world.org similarity index 93% rename from org/2021/hello-world.org rename to org/hello-world.org index fd655b2..0574838 100644 --- a/org/2021/hello-world.org +++ b/org/hello-world.org @@ -1,7 +1,7 @@ #+HUGO_SECTION: posts -#+HUGO_BASE_DIR: ../.. +#+HUGO_BASE_DIR: ../ #+TITLE: Hello, world! -#+DATE: 2020-02-01 +#+DATE: 2021-02-01 #+PROPERTY: header-args:python :session *123* #+PROPERTY: header-args:python+ :exports both #+PROPERTY: header-args:python+ :tangle yes diff --git a/org/images/gmail/mail.png b/org/images/gmail/mail.png new file mode 100644 index 0000000..ed29000 Binary files /dev/null and b/org/images/gmail/mail.png differ diff --git a/org/images/gmail/main.png b/org/images/gmail/main.png new file mode 100644 index 0000000..439e060 Binary files /dev/null and b/org/images/gmail/main.png differ diff --git a/org/images/gmail/notification.png b/org/images/gmail/notification.png new file mode 100644 index 0000000..33f32d0 Binary files /dev/null and b/org/images/gmail/notification.png differ diff --git a/static/ox-hugo/mail.png b/static/ox-hugo/mail.png new file mode 100644 index 0000000..ed29000 Binary files /dev/null and b/static/ox-hugo/mail.png differ diff --git a/static/ox-hugo/main.png b/static/ox-hugo/main.png new file mode 100644 index 0000000..439e060 Binary files /dev/null and b/static/ox-hugo/main.png differ diff --git a/static/ox-hugo/notification.png b/static/ox-hugo/notification.png new file mode 100644 index 0000000..33f32d0 Binary files /dev/null and b/static/ox-hugo/notification.png differ