Browse Source

workable mailsync readded to enable autosync

the new script manually searches for environmental variables
so that it can work from a cronjob
and additionally searches for all Xorg displays on Linux
so that the display notification should appear correctly
no doubt there will be more bugs, but this is what the people want
and it's worth working out
tags/v3.1
Luke Smith 3 years ago
parent
commit
f589cd4dfd
No known key found for this signature in database GPG Key ID: 4C50B54A911F6252
4 changed files with 221 additions and 43 deletions
  1. +82
    -25
      README.md
  2. +98
    -0
      bin/mailsync
  3. +28
    -18
      bin/mw
  4. +13
    -0
      mw.1

+ 82
- 25
README.md View File

@@ -3,17 +3,24 @@
Get this great stuff without effort:

- A full-featured and autoconfigured email client on the terminal with neomutt
- Mail stored offline so you can view and write email while you're away from internet and keep backups
- Mail stored offline so you can view and write email while you're away from
internet and keep backups
- Provides a `mailsync` script that can be scheduled to run as often as you
like, which downloads/syncs mail and notifies you when new mail has arrived.

Specifically, this wizard:

- Determines your email server's IMAP and SMTP servers and ports
- Creates dotfiles for `neomutt`, `isync`, and `msmtp` appropriate for your email address
- Encrypts and locally stores your password for easy remote access, accessible only by your GPG key
- Creates dotfiles for `neomutt`, `isync`, and `msmtp` appropriate for your
email address
- Encrypts and locally stores your password for easy remote access, accessible
only by your GPG key
- Handles as many as nine separate email accounts automatically
- Auto-creates bindings to switch between accounts or between mailboxes
- Provides sensible defaults and an attractive appearance for the neomutt email client
- If mutt-wizard doesn't know your server's IMAP/SMTP info by default, it will prompt you for them and will put them in all the right places.
- Provides sensible defaults and an attractive appearance for the neomutt email
client
- If mutt-wizard doesn't know your server's IMAP/SMTP info by default, it will
prompt you for them and will put them in all the right places.

## Install and Use

@@ -33,6 +40,8 @@ The mutt-wizard is run with the command `mw`. Once everything is setup, you'll u
- `mw -Y` -- sync all configured email accounts
- `mw -d` -- choose an account to delete
- `mw -D your@email.com` -- delete account settings without confirmation
- `mw -t 30` -- toggle automatic mailsync to every 30 minutes
- `mw -T` -- toggle mailsync without specifying minutes (default is 10)
- `pass edit mw-your@email.com` -- revise an account's password

### Options usable when adding an account
@@ -70,6 +79,7 @@ There's a chance of errors if you use a slow-release distro like Ubuntu, Debian
- `notmuch` - index and search mail. Install it and run `notmuch setup`, tell it that your mail is in `~/.local/share/mail/` (although `mw` will do this automatically if you haven't set notmuch up before). You can run it in mutt with `ctrl-f`. Run `notmuch new` to process new mail.
- `abook` - a terminal-based address book. Pressing tab while typing an address to send mail to will suggest contacts that are in your abook.
- `urlview` - outputs urls in mail to browser.
- `cronie` - (or any other major cronjob manager) to set up automatic mail syncing.

## Neomutt user interface

@@ -91,43 +101,90 @@ To give you an example of the interface, here's an idea:
-
## New stuff and improvements since the original release

- `mw` is now scriptable with command-line options and can run successfully without any interaction, making it possible to deploy in a script.
- `isync`/`mbsync` has replaced `offlineimap` as the backend. Offlineimap was error-prone, bloated, used obsolete Python 2 modules and required separate steps to install the system.
- `mw` is now an installed program instead of just a script needed to be kept in your mutt folder.
- `dialog` is no longer used (le bloat) and the interface is simply text commands.
- More autogenerated shortcuts that allow quickly moving and copying mail between boxes.
- More elegant attachment handling. Image/video/pdf attachments without relying on the neomutt instance.
- `mw` is now scriptable with command-line options and can run successfully
without any interaction, making it possible to deploy in a script.
- `isync`/`mbsync` has replaced `offlineimap` as the backend. Offlineimap was
error-prone, bloated, used obsolete Python 2 modules and required separate
steps to install the system.
- `mw` is now an installed program instead of just a script needed to be kept
in your mutt folder.
- `dialog` is no longer used (le bloat) and the interface is simply text
commands.
- More autogenerated shortcuts that allow quickly moving and copying mail
between boxes.
- More elegant attachment handling. Image/video/pdf attachments without relying
on the neomutt instance.
- abook integration by default.
- The messy template files and other directories have been moved or removed, leaving a clean config folder.
- msmtp configs moved to `~/.config/` and mail default location moved to `~/.local/share/mail/`, reducing mess in `~`.
- The messy template files and other directories have been moved or removed,
leaving a clean config folder.
- msmtp configs moved to `~/.config/` and mail default location moved to
`~/.local/share/mail/`, reducing mess in `~`.
- `pass` is used as a password manager instead of separately saving passwords.
- Script is POSIX sh compliant.
- Error handling for the many people who don't read or follow directions. Less errors generally.
- Error handling for the many people who don't read or follow directions. Less
errors generally.
- Addition of a manual `man mw`

## Help the Project!

- Try mutt-wizard out on weird machines and weird email addresses and report any errors.
- Open a PR to add new server information into `domains.csv` so their users can more easily use mutt-wizard.
- Try mutt-wizard out on weird machines and weird email addresses and report
any errors.
- Open a PR to add new server information into `domains.csv` so their users can
more easily use mutt-wizard.
- If nothing else, [Donate!](https://paypal.me/LukeMSmith)

See Luke's website [here](https://lukesmith.xyz). Email him at [luke@lukesmith.xyz](mailto:luke@lukesmith.xyz).
See Luke's website [here](https://lukesmith.xyz). Email him at
[luke@lukesmith.xyz](mailto:luke@lukesmith.xyz).

mutt-wizard is free/libre software, licensed under the GPLv3.

## Details for Tinkerers

- The critical `mutt`/`neomutt` files are in `~/.config/mutt/`.
- Put whatever global settings you want in `muttrc`. mutt-wizard will add some lines to this file which you shouldn't remove unless you know what you're doing, but you can move them up/down over your personal config lines if you need to. If you get binding conflict errors in mutt, you might need to do this.
- Each of the accounts that mutt-wizard generates will have custom settings set in a separate file in `accounts/`. You can edit these freely if you want to tinker with settings specific to an account.
- In `/usr/share/mutt-wizard` are several global config files, including `mutt-wizard`'s default settings. You can overwride this in your `muttrc` if you wish.
- Put whatever global settings you want in `muttrc`. mutt-wizard will add some
lines to this file which you shouldn't remove unless you know what you're
doing, but you can move them up/down over your personal config lines if you
need to. If you get binding conflict errors in mutt, you might need to do
this.
- Each of the accounts that mutt-wizard generates will have custom settings set
in a separate file in `accounts/`. You can edit these freely if you want to
tinker with settings specific to an account.
- In `/usr/share/mutt-wizard` are several global config files, including
`mutt-wizard`'s default settings. You can overwride this in your `muttrc` if
you wish.

## Watch out for these things:
- Gmail accounts can now create 'App Password' to use with """less secure""" applications. This password is single use (ie. for setup) and will be stored and encrypted locally. Enabling third-party applications requires turning off two-factor authentication and this will circumvent that. You might also need to manually "Enable IMAP" in the settings.
- Protonmail accounts will require you to set up "Protonmail Bridge" to access PM's IMAP and SMTP servers. Configure that before running mutt-wizard. Note that when mutt-wizard asks for a password, you should put in your [bridge password](https://protonmail.com/bridge/thunderbird#3), not your account password.
- Protonmail bridge is prone to timing out. Watch out for this while adding an account. If the bridge times out, try again. It might help to [increase the timeout](https://protonmail.com/support/knowledge-base/thunderbird-connection-server-timed-error/) in your `mbsyncrc`.
- If you have a university email, or enterprise-hosted email for work, there might be other hurdles or two-factor authentication you have to jump through. Some, for example, will want you to create a separate IMAP password, etc.
- `isync` is not fully UTF-8 compatible, so non-Latin characters may be garbled (although sync should succeed). `mw` will also not autocreate mailbox shortcuts since it is looking for English mailbox names. I strongly recommend you to set your email language to English on your mail server to avoid these problems.

### Gmail

Gmail accounts should require an
[application password](https://support.google.com/accounts/answer/185833) to
work unless you allow Gmail to access "less secure" applications. You may also
need to "Enable IMAP" in your Gmail settings.

### Protonmail

Protonmail accounts will require you to set up "Protonmail Bridge" to access
PM's IMAP and SMTP servers. Configure that before running mutt-wizard. Note
that when mutt-wizard asks for a password, you should put in your
[bridge password](https://protonmail.com/bridge/thunderbird#3), not your
account password.

Protonmail bridge is prone to timing out. Watch out for this while adding an
account. If the bridge times out, try again. It might help to
[increase the timeout](https://protonmail.com/support/knowledge-base/thunderbird-connection-server-timed-error/)
in your `mbsyncrc`.

### Other

- If you have a university email, or enterprise-hosted email for work, there
might be other hurdles or two-factor authentication you have to jump through.
Some, for example, will want you to create a separate IMAP password, etc.
- `isync` is not fully UTF-8 compatible, so non-Latin characters may be
garbled (although sync should succeed). `mw` will also not autocreate
mailbox shortcuts since it is looking for English mailbox names. I strongly
recommend you to set your email language to English on your mail server to
avoid these problems.

## To-do



+ 98
- 0
bin/mailsync View File

@@ -0,0 +1,98 @@
#!/bin/sh

# - Syncs mail for all accounts, or a single account given as an argument.
# - Displays a notification showing the number of new mails.
# - Displays a notification for each new mail with its subject displayed.
# - Runs notmuch to index new mail.
# - This script can be set up as a cron job for automated mail syncing.

# There are many arbitrary and ugly features in this script because it is
# inherently difficult to pass environmental variables to cronjobs and other
# issues. It also should at least be compatible with Linux (and maybe BSD) with
# Xorg and MacOS as well.

# Run only if user logged in (prevent cron errors)
pgrep -u "${USER:=$LOGNAME}" >/dev/null || { echo "$USER not logged in; sync will not run."; exit ;}
# Run only if not already running in other instance
pgrep -x mbsync >/dev/null && { echo "mbsync is already running." ; exit ;}

# First, we have to get the right variables for the mbsync file, the pass
# archive, notmuch and the GPG home. This is done by searching common profile
# files for variable assignments. This is ugly, but there are few options that
# will work on the maximum number of machines.
eval `grep -h -- \
"^\(export \)\?\(MBSYNCRC\|PASSWORD_STORE_DIR\|NOTMUCH_CONFIG\|GNUPGHOME\)=" \
"$HOME/.profile" "$HOME/.bash_profile" "$HOME/.zprofile" "$HOME/.bashrc" "$HOME/.zshrc" "$HOME/.pam_environment" 2>/dev/null`
# One alternative to this kind of command would be marking the script for
# /bin/sh -l. That might cause other problems on other particular setups that
# do more complicated things on login, or those people who assign environmental
# variables in shell rc files.

# This variable might be required for soysdemd users, but it will break the
# script on Artix runit.
# export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u)/bus
export GPG_TTY=$TTY

[ -n "$MBSYNCRC" ] && alias mbsync="mbsync -c $MBSYNCRC" || MBSYNCRC="$HOME/.mbsyncrc"

# Settings are different for MacOS (Darwin) systems.
case "$(uname)" in
Darwin)
notify() { osascript -e "display notification \"$2 in $1\" with title \"You've got Mail\" subtitle \"Account: $account\"" && sleep 2 ;}
messageinfo() { osascript -e "display notification with title \"📧 $from\" subtitle \"$subject\"" ;}
;;
*)
displays="$(ps axo user,cmd | grep "^$USER\s\+.*Xorg" | grep -wo "[0-9]*:[0-9]\+")"
notify() { for x in $displays; do
export DISPLAY=$x
notify-send --app-name="mutt-wizard" "mutt-wizard" "📬 $2 new mail(s) in \`$1\` account."
done ;}
messageinfo() { for x in $displays; do
export DISPLAY=$x
notify-send --app-name="mutt-wizard" "📧$from:" "$subject"
done ;}
;;
esac

# Check account for new mail. Notify if there is new content.
syncandnotify() {
acc="$(echo "$account" | sed "s/.*\///")"
if [ -z "$opts" ]; then mbsync "$acc"; else mbsync "$opts" "$acc"; fi
new=$(find "$HOME/.local/share/mail/$acc/INBOX/new/" "$HOME/.local/share/mail/$acc/Inbox/new/" "$HOME/.local/share/mail/$acc/inbox/new/" -type f -newer "$HOME/.config/mutt/.mailsynclastrun" 2> /dev/null)
newcount=$(echo "$new" | sed '/^\s*$/d' | wc -l)
if [ "$newcount" -gt "0" ]; then
notify "$acc" "$newcount" &
for file in $new; do
# Extract subject and sender from mail.
from=$(awk '/^From: / && ++n ==1,/^\<.*\>:/' "$file" | perl -CS -MEncode -ne 'print decode("MIME-Header", $_)' | awk '{ $1=""; if (NF>=3)$NF=""; print $0 }' | sed 's/^[[:blank:]]*[\"'\''\<]*//;s/[\"'\''\>]*[[:blank:]]*$//')
subject=$(awk '/^Subject: / && ++n == 1,/^\<.*\>: / && ++i == 2' "$file" | head -n 1 | perl -CS -MEncode -ne 'print decode("MIME-Header", $_)' | sed 's/^Subject: //' | sed 's/^{[[:blank:]]*[\"'\''\<]*//;s/[\"'\''\>]*[[:blank:]]*$//' | tr -d '\n')
messageinfo &
done
fi
}

# Sync accounts passed as argument or all.
if [ "$#" -eq "0" ]; then
accounts="$(awk '/^Channel/ {print $2}' "$MBSYNCRC")"
else
for arg in "$@"; do
[ "${arg%${arg#?}}" = '-' ] && opts="${opts:+${opts} }${arg}" && shift 1
done
accounts=$*
fi

#( kill -46 "$(pidof "${STATUSBAR:-dwmblocks}")" >/dev/null 2>&1 ) 2>/dev/null

# Parallelize multiple accounts
for account in $accounts
do
syncandnotify &
done

wait
#( kill -46 "$(pidof "${STATUSBAR:-dwmblocks}")" >/dev/null 2>&1 ) 2>/dev/null

notmuch new 2>/dev/null

#Create a touch file that indicates the time of the last run of mailsync
touch "$HOME/.config/mutt/.mailsynclastrun"

+ 28
- 18
bin/mw View File

@@ -21,7 +21,7 @@ cachedir="${XDG_CACHE_HOME:-$HOME/.cache}/mutt-wizard"
muttrc="$muttdir/muttrc"
msmtprc="${XDG_CONFIG_HOME:-$HOME/.config}/msmtp/config"
msmtplog="${XDG_CONFIG_HOME:-$HOME/.config}/msmtp/msmtp.log"
MARKER="# mw-autogenerated"
marker="# mw-autogenerated"
alias mbsync='mbsync -c "$mbsyncrc"'

for x in "/etc/ssl/certs/ca-certificates.crt" "/etc/pki/tls/certs/ca-bundle.crt" "/etc/ssl/ca-bundle.pem" "/etc/pki/tls/cacert.pem" "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem" "/etc/ssl/cert.pem" "/usr/local/share/ca-certificates/"
@@ -151,7 +151,7 @@ delete() { if [ -z "${fulladdr+x}" ]; then

[ -z "$fulladdr" ] && echo "Invalid account name given." && return 1

sed -ibu "/IMAPStore $fulladdr-remote$/,/# End profile/d" "$mbsyncrc" ; rm -rf "$mbsyncrc"bu
sed -ibu "/IMAPStore $fulladdr-remote$/,/# End profile/d" "$mbsyncrc" ; rm -f "$mbsyncrc"bu
rm -rf "${cachedir:?}/${fulladdr:?}" "$accdir/"[1-9]"-$fulladdr.muttrc"
sed -ibu "/[0-9]-$fulladdr.muttrc/d" "$muttrc" ; rm -f "$muttrc"bu
sed -ibu "/account $fulladdr/,/^\(\s*$\|account\)/d" "$msmtprc"; rm -f "$msmtprc"bu
@@ -205,9 +205,9 @@ writeinfo() { mkdir -p "$muttdir" "$accdir" "$cachedir/$fulladdr/bodies" "${XDG_
# Create a muttrc for viewing mail.
echo "$mutt_profile" > "$accdir/$idnum-$fulladdr.muttrc"
[ ! -f "$muttrc" ] && echo "# vim: filetype=neomuttrc" > "$muttrc"
! grep -q "^source.*mutt-wizard.muttrc" "$muttrc" && echo "source $mwconfig $MARKER" >> "$muttrc"
! grep "^source.*.muttrc" "$muttrc" | grep -qv "$mwconfig" && echo "source $accdir/$idnum-$fulladdr.muttrc $MARKER" >> "$muttrc"
echo "macro index,pager i$idnum '<sync-mailbox><enter-command>source $accdir/$idnum-$fulladdr.muttrc<enter><change-folder>!<enter>;<check-stats>' \"switch to $fulladdr\" $MARKER" >> "$muttrc"
! grep -q "^source.*mutt-wizard.muttrc" "$muttrc" && echo "source $mwconfig $marker" >> "$muttrc"
! grep "^source.*.muttrc" "$muttrc" | grep -qv "$mwconfig" && echo "source $accdir/$idnum-$fulladdr.muttrc $marker" >> "$muttrc"
echo "macro index,pager i$idnum '<sync-mailbox><enter-command>source $accdir/$idnum-$fulladdr.muttrc<enter><change-folder>!<enter>;<check-stats>' \"switch to $fulladdr\" $marker" >> "$muttrc"

notmuchauto # Create a notmuch config file if not present already.
}
@@ -216,12 +216,12 @@ getpass() { while : ; do pass rm -f "$pass_prefix$fulladdr" >/dev/null 2>&1
pass insert "$pass_prefix$fulladdr" && break; done ;}

formatShortcut() { toappend="$toappend
macro index,pager g$1 \"<change-folder>=$3<enter>\" \"go to $2\" $MARKER
macro index,pager M$1 \";<save-message>=$3<enter>\" \"move mail to $2\" $MARKER
macro index,pager C$1 \";<copy-message>=$3<enter>\" \"copy mail to $2\" $MARKER" >> "$accdir/$idnum-$fulladdr.muttrc" ;}
macro index,pager g$1 \"<change-folder>=$3<enter>\" \"go to $2\" $marker
macro index,pager M$1 \";<save-message>=$3<enter>\" \"move mail to $2\" $marker
macro index,pager C$1 \";<copy-message>=$3<enter>\" \"copy mail to $2\" $marker" >> "$accdir/$idnum-$fulladdr.muttrc" ;}

setBox() { toappend="$toappend
set $1 = \"+$2\" $MARKER" ;}
set $1 = \"+$2\" $marker" ;}

getboxes() { [ -n "${force+x}" ] && mailboxes="INBOX
Drafts
@@ -239,7 +239,7 @@ Archive" && return 0
fi ;}

finalize() { echo "$mailboxes" | xargs -I {} mkdir -p "$maildir/$fulladdr/{}/cur" "$maildir/$fulladdr/{}/tmp" "$maildir/$fulladdr/{}/new"
sed -ibu "/$MARKER/d" "$accdir/$idnum-$fulladdr.muttrc" ; rm -f "$accdir/$idnum-$fulladdr.muttrcbu"
sed -ibu "/$marker/d" "$accdir/$idnum-$fulladdr.muttrc" ; rm -f "$accdir/$idnum-$fulladdr.muttrcbu"
toappend="mailboxes $(echo "$mailboxes" | sed "s/^/\"=/;s/$/\"/" | paste -sd ' ' - )"
for x in $mailboxes; do
case "$x" in
@@ -257,12 +257,6 @@ finalize() { echo "$mailboxes" | xargs -I {} mkdir -p "$maildir/$fulladdr/{}/cur
command -V urlview >/dev/null 2>&1 && [ ! -f "$HOME/.urlview" ] && echo "COMMAND \$BROWSER" > "$HOME/.urlview"
return 0 ;}

syncwrapper() { mbsync "${1:--a}" &
( kill -46 "$(pidof "${STATUSBAR:-dwmblocks}")" >/dev/null 2>&1 ) 2>/dev/null
wait
( kill -46 "$(pidof "${STATUSBAR:-dwmblocks}")" >/dev/null 2>&1 ) 2>/dev/null
notmuch new ;}

notmuchauto() { \
[ -z "$NOTMUCH_CONFIG" ] && NOTMUCH_CONFIG="$HOME/.notmuch-config"
[ -f "$NOTMUCH_CONFIG" ] && return 0
@@ -282,6 +276,17 @@ synchronize_flags=true
gpg_path=$GPG"
echo "$nmbasic" > "$NOTMUCH_CONFIG" ;}

togglecron() { cron="$(mktemp)"
crontab -l > "$cron"
if grep -q mailsync "$cron"; then
echo "Removing automatic mailsync..."
sed -ibu /mailsync/d "$cron"; rm -f "$cron"bu
else
echo "Adding automatic mailsync every ${cronmin:-10} minutes..."
echo "*/${cronmin-10} * * * * /usr/local/bin/mailsync" >> "$cron"
fi &&
crontab "$cron"; rm -f "$cron" ;}

setact() { if [ -n "${action+x}" ] && [ "$action" != "$1" ]; then
echo "Running $1 with $action..."
echo "Incompatible options given. Only one action may be specified per run."
@@ -292,7 +297,7 @@ setact() { if [ -n "${action+x}" ] && [ "$action" != "$1" ]; then

trap 'echo -e "\033[0m\n"; exit' INT ABRT

while getopts "fplhodYD:y:i:I:s:S:u:a:n:x:m:" o; do case "${o}" in
while getopts "fplhodTYD:y:i:I:s:S:u:a:n:x:m:t:" o; do case "${o}" in
l) setact list || exit 1 ;;
d) setact delete || exit 1 ;;
D) setact delete || exit 1 ; fulladdr="$OPTARG" ;;
@@ -309,6 +314,8 @@ while getopts "fplhodYD:y:i:I:s:S:u:a:n:x:m:" o; do case "${o}" in
o) setact add || exit 1 ; online=True ;;
f) setact add || exit 1 ; force=True ;;
x) setact add || exit 1 ; password="$OPTARG" ;;
t) setact toggle || exit 1 ; cronmin="$OPTARG" ;;
T) setact toggle || exit 1 ;;
p) echo "NOTE: Protonmail users must install and configure Protonmail Bridge first for the first sync to work."
imap="127.0.0.1"
iport="1143"
@@ -329,6 +336,8 @@ Main actions:
-D your@email.com Force remove account without confirmation
-y your@email.com Sync mail for account by name
-Y Sync mail for all accounts
-t number Toggle automatic mailsync every <number> minutes
-T Toggle automatic mailsync

Options allowed with -a:
-u Account login name if not full address
@@ -357,5 +366,6 @@ case "$action" in
list) list ;;
add) askinfo && getprofiles && writeinfo && getboxes && finalize || { delete ; exit 1 ;} ;;
delete) delete $fulladdr ;;
sync) syncwrapper $fulladdr ;;
sync) mailsync $fulladdr ;;
toggle) togglecron ;;
esac

+ 13
- 0
mw.1 View File

@@ -35,6 +35,12 @@ download and upload mail for an email account
.TP
.B -Y
sync all email accounts
.TP
.B -t 15
toggle a cronjob that syncs your mail every 15 minutes (or any other number under 60)
.TP
.B -T
toggle a cronjob without specifying minutes between sync
.SH OPTIONS FOR ADDING ACCOUNTS
These can be specified on the command line, otherwise, you will be prompted for what is necessary. mutt-wizard knows the IMAP/SMTP server information for most email providers, so specifying them is usually redundant.
.TP
@@ -75,6 +81,13 @@ connection will still be attempted in setup to discover mailboxes.
Add a Protonmail account. Protonmail Bridge must be installed and set up.
.SH DETAILS
.TP
.B mailsync
mutt-wizard calls a script
.I mailsync
to sync mail. This script additionally indexes new mail with notmuch and gives you a notification if new mail has arrived. If you want to bypass its additional features, you can always just run
.I mbsync -a
to sync your mail directly.
.TP
.B Mail location
If the user chooses to keep offline email with
.B isync,


Loading…
Cancel
Save