Hello all,

Just to chime in here. I wrote that email on the aerc mailing list. For
some context. This specific work around is in regards to using MAILDIR
and notmuch with aerc. You can use notmuch and aerc without MAILDIR by
removing maildir:// and copy-to from accounts.conf but you can only send
from one account without some interesting solutions, If you have multiple
accounts this isn't ideal. The aerc team has decided this approach is
best practice when using aerc, MAILDIR and notmuch. This work around was 
to provide a working solution for people who want to follow the
developments team best practice while using a BSD system. Running
without a matching openfile results in "Too many open file errors" and
the application not functioning.

Aerc uses fsnotify for watching files. Linux backend is inotify  which
supports recurive watching. OpenBSD uses kqueue which doesn't.

This was never meant to be an ideal solution. I also mentioned to
Johannes, that a better approach would be to shrink your inbox and have
a smaller openfile.

I'm using aerc on a OpenBSD system as a desktop and I'm the sole user.
So I'm not concerned with people opening multiple instances of aerc. I
realise everyone might not be in that boat.

On Thu Sep 5, 2024 at 11:35 PM AEST, Johannes Thyssen Tishman wrote:
> 2024-09-05T12:55:06Z Stuart Henderson:
> > On 2024/09/05 14:23, Johannes Thyssen Tishman wrote:
> > > 2024-09-05T12:09:51Z Stuart Henderson:
> > > > On 2024/09/05 12:52, Johannes Thyssen Tishman wrote:
> > > > > The email below was sent to the aerc-discuss mailing list a couple of
> > > > > days ago. Is it worth adding a pkg/README for this?
> > > > > 
> > > > > 2024-09-03T23:12:23Z "Dylan D'silva" <dylan@dsilva.email>:
> > > > > Hello all,
> > > > > 
> > > > > After some digging I've come to a solution. For those on a BSD system
> > > > > with large MAILDIR, you'll require something similar. Aerc uses
> > > > > github.com/fsnotify/fsnotify to monitor changes to files. BSD system
> > > > > rely on Kqueue which doesn't support recursive watching of files.
> > > >                                                              ^^
> > > > 							     directories
> > > > > Therefore you require one file descriptor for each file.
> > > >
> > > > This does not seem to be a good approach for monitoring emails in
> > > > Maildirs on kqueue systems.
> > > >
> > > > Most of the common changes to a Maildir result in either a new file,
> > > > rename, or deletion and not actually changes to the contents of the
> > > > files. AFAIK new files won't show up without rescanning the whole
> > > > directory - for renamed files I don't think you can find the new
> > > > name so again you need to rescan - it seems like really it's just
> > > > going to need an efficient way to pick up changes in the dir.
> > > >
> > > > (I suppose this is not going to work well with NFS either which is
> > > > not an uncommon place to store Maildirs..)
> > > >
> > > > > Assuming you are running aerc from the current account. You can
> > > > > change the limit to match your maildir size.
> > > > > 
> > > > > For me, I store my maildir in ~/Mail 
> > > > > find . -type f | wc -l #counts number of files
> > > > > 71514
> > > > > 
> > > > > I run aerc from a staff account. Therefore I updated my /etc/login.conf
> > > > > under staff to increase the openfile-cur size to 71600 and openfile-max
> > > > > to 72000
> > > > >  :openfiles-cur=71600:\
> > > > >  :openfiles-max=72000:\
> > > >
> > > > If you're going to do this, I would recommend a "normal" openfiles-cur,
> > > > just raise openfiles-max. Then start such software from a script or
> > > > shell alias which uses ulimit -n to raise the limit just for that
> > > > software.

What do you consider normal? In your comment below, you made comment
about not many people running computers from the 80's. 
Here is a commit for inotify in 2020 making the openfile limit dynamic
between 8192 and 104576.

From the commit:
"The default value of inotify.max_user_watches sysctl parameter was set
to 8192 since the introduction of the inotify feature in 2005 by
commit 0eeca28 ("[PATCH] inotify"). Today this value is just too
small for many modern usage. As a result, users have to explicitly set
it to a larger value to make it work."

https://github.com/torvalds/linux/commit/92890123749bafc317bbfacbe0a62ce08d78efb7


It's worth mentioning but the syncthing pkg-readme suggest a openfile of
4096. 

> > > >
> > > > I think it would generally be better to run aerc pointed at an IMAP
> > > > server though (obviously the IMAP server can just run on the machine
> > > > running aerc if wanted).

I agree this is a better solution. This was only meant as a work around
to follow what the dev team are using. 
> > > >
> > > > > If you use a large login.conf you might want create a database version
> > > > > with cap_mkdb /etc/login.conf see login.conf(5).
> > > >
> > > > We are mostly not running on computer systems from the 80s and the file
> > > > is not really slow to parse - however having the db files around does
> > > > cause trouble when someone forgets to rerun cap_mkdb after changing the
> > > > file later. I would not recommend this.

The man page for login(5).conf has that comment.

> > > 
> > > Thanks for the feedback Stuart. Should this advice be included in a
> > > pkg/README as it is done for syncthing?
> >
> > It would be better if the situation was improved upstream, but we
> > could do...
>
> I'll check with upstream.
>
> > Rather than documenting the workaround I think it's probably better if
> > we go for this approach. Setting the OS to allow opening of tens of
> > thousands of files is very much not ideal, and imagine what it would
> > be like if you have a couple of users on the machine, all of whom want
> > to run aerc...

How about improving Kqueue to support recursive watching? 

> I agree, this seems more sane. Thank you Stuart :)
>
> > Index: Makefile
> > ===================================================================
> > RCS file: /cvs/ports/mail/aerc/Makefile,v
> > diff -u -p -r1.16 Makefile
> > --- Makefile	31 Jul 2024 18:36:02 -0000	1.16
> > +++ Makefile	5 Sep 2024 12:52:52 -0000
> > @@ -7,6 +7,7 @@ MODGO_MODNAME =	git.sr.ht/~rjarry/aerc
> >  MODGO_VERSION =	v0.0.0-20240729220005-e037c095a049
> >  
> >  V =		0.18.2
> > +REVISION =	0
> >  DISTNAME =	aerc-$V
> >  
> >  CATEGORIES =	mail
> > Index: pkg/PLIST
> > ===================================================================
> > RCS file: /cvs/ports/mail/aerc/pkg/PLIST,v
> > diff -u -p -r1.9 PLIST
> > --- pkg/PLIST	31 Jul 2024 18:36:02 -0000	1.9
> > +++ pkg/PLIST	5 Sep 2024 12:52:52 -0000
> > @@ -43,4 +43,5 @@ share/aerc/templates/forward_as_body
> >  share/aerc/templates/new_message
> >  share/aerc/templates/quoted_reply
> >  share/applications/aerc.desktop
> > +share/doc/pkg-readmes/${PKGSTEM}
> >  @tag update-desktop-database
> > Index: pkg/README
> > ===================================================================
> > RCS file: pkg/README
> > diff -N pkg/README
> > --- /dev/null	1 Jan 1970 00:00:00 -0000
> > +++ pkg/README	5 Sep 2024 12:52:52 -0000
> > @@ -0,0 +1,12 @@
> > ++-------------------------------------------------------------------------------
> > +| Running ${PKGSTEM} on OpenBSD
> > ++-------------------------------------------------------------------------------
> > +
> > +aerc's Maildir support on BSD uses kqueue(2), which is really intended for
> > +watching individual files rather than large directories or trees. It requires
> > +a file handle to be opened for each file in the monitored directories, which
> > +will soon exceed default limits in the system.
> > +
> > +It is therefore recommended to run an IMAP daemon (for example Dovecot,
> > +courier-imap, imap-uw) and connect over IMAP, even if this is just to the
> > +local machine.