Wednesday 10 November 2021

Building an LDAP address book for Thunderbird and Evolution

Overview

Households—and small companies—where there is more than one email user can derive considerable benefit from a shared address book.  Aside from proprietary systems such as Microsoft Exchange, implementation of an LDAP server is a well-recommended way to do it.

However, LDAP is “as user-friendly as a cornered rat”, and Thunderbird’s support of LDAP is more or less broken (and has been so for decades, without any obvious plan to fix it); together these make the whole process far harder than one would like.

It can be done, and almost all the benefits gained, but documentation on how to do it is sketchy and hard to find, and some of it is out of date with respect to current levels of software.  This document sets down my own route to success, in the hopes that others may find it helpful.

The detailed instructions assume a linux server (I’m using Debian 10); Windows users will, I hope, find that the outline of what to do is the same, just with different command syntax.

Henry Law, Manchester UK, November 2021

A note on LDAP itself

It’s important to note that LDAP is not a directory system.  It isn’t even a piece of software: it’s a description of the protocol that can be used to interact with a directory. So the word “LDAP” does the same job as “TCP/IP”; it describes something that has to be implemented and configured by software developers and users.

And like all protocols it has a very complex definition covering every eventuality and use case; it’s got lots of specific jargon which you have to learn, and configuring it isn’t immediately straightforward.

Fortunately you don’t need to know more than the bare bones of it to implement and configure an LDAP server for a well-understood requirement like a directory (or “address book” if you prefer). You’ll find what you need in the sections below.

If you do want to know all about LDAP, you can find it in a host of technical documents elsewhere on the web (and good luck).

A note on OpenLDAP

The commonest open-source LDAP server is OpenLDAP.  It is available on all the popular Linux distributions and also on Windows.  That’s what I used; if you want to use something else then a lot of the specifics of what follows may not be helpful.

Thunderbird and LDAP

If you read the program description you will find that Mozilla Thunderbird “supports” LDAP, and indeed it does.  You can use an LDAP directory as one of your address books, and get it to auto-complete when sending an email; it can also use LDIF (a text format used when loading information into an LDAP directory) for export and import.  But despite this apparent goodness the LDAP support in Thunderbird is very poor, and in some places broken.  More particularly:

  • LDAP directories in Thunderbird are read-only; the user cannot directly create or change an entry in an LDAP directory.
  • The LDIF that Thunderbird emits (via the “export” function) is not valid for input to an LDAP directory.  It needs to be reformatted before it can be used.
  • Thunderbird does not support multiple email addresses in a single address book entry.  It does permit a second email address, using a Mozilla-specific field to do it, but this is special to Thunderbird; some other mail client using the same directory is unlikely to see that field.
  • Because mailing lists (lists of people to whom a single email may be sent) cannot span different address books, a Thunderbird user cannot create a list which contains entries from a connected LDAP directory.  In fairness this is a standard limitation of Thunderbird, not a particular facet of its LDAP support

Placeholders in this document

The detailed instructions that follow do, of course, have to give values for various parameters which will not match your system.  Here are the values which you will need to change everywhere:

Your domain:            example.org
Your organisation name    myplace
Your master user        ldapuser
Master user’s password    ldaptest
Your address book name    abook

Install the code

sudo apt install slapd ldap-utils
For some reason, this doesn’t perform a configuration of LDAP; it just asks for an admin password and that’s it.  Re-configuring it asks you all the questions:
sudo dpkg-reconfigure slapd
Omit OpenLDAP config?: no (of course not!)
Domain name: example.org
Organisation name: myplace
Administrator password: (as before)
Backend: MDB
Remove when slapd is purged? No
Move old database? Yes

You can test your installation with this command:

ldapsearch -x -b dc=example,dc=org

The last lines of the output should be “search: 2 … result: 0 Success”.

Create ou for address book

Create defineAddressBook.ldif with the following contents

dn: ou=abook,dc=example,dc=org
ou: abook
objectClass: organizationalUnit

Add it to the directory with

ldapadd -vx -D "cn=admin,dc=example,dc=org" -H ldapi:/// -w ldaptest -f defineAddressBook.ldif

The configuration may be queried thus

sudo slapcat -b cn=config

Add Thunderbird mail schema

Download the Mozilla schema from the Mozilla web site.  and copy it to /etc/ldap/schema.

In a work directory, create slapd.conf containing the following lines:

include /etc/ldap/schema/core.schema
include /etc/ldap/schema/cosine.schema
include /etc/ldap/schema/nis.schema
include /etc/ldap/schema/inetorgperson.schema
include /etc/ldap/schema/mozillaAbPersonAlpha.schema

Change to the work directory and generate the LDIF files thus:

sudo slaptest -f slapd.conf -F $PWD
But the ldif file that is generated is not suitable for loading as a schema, for some unfathomable reason.  It needs to be modified; the following sed script mozillaAbPersonAlpha.sed should do it:
/dn:/ s|\{[0-9]\}||
/dn:/ s|$|,cn=schema,cn=config|
/cn:/ s|\{[0-9]\}mozillaabpersonalpha|mozillaabpersonalpha|
/structuralObjectClass/d
/entryUUID/d
/creatorsName/d
/createTimestamp/d
/entryCSN/d
/modifiersName/d
/modifyTimestamp/d

Modify the generated LDIF file by running the sed script (in the work directory).

sudo sed -i -rf mozillaAbPersonAlpha.sed './cn=config/cn=schema/cn={4}mozillaabpersonalpha.ldif'

Now you can add the schema

sudo ldapadd -Y EXTERNAL -H ldapi:/// -f './cn=config/cn=schema/cn={4}mozillaabpersonalpha.ldif' -w ldaptest

No, I don’t understand why we need to specify an SASL method (with -Y) here, but it didn’t work for me without.

Verify the addition with

sudo ldapsearch -Y EXTERNAL -H ldapi:/// -w ldaptest -b 'cn={4}mozillaabpersonalpha,cn=schema,cn=config'

Check that the mozillaabpersonalpha schema has been loaded.

Add Thunderbird address book

The Thunderbird address book must be exported from Thunderbird itself, and the resulting ldif file heavily reformatted.  See the appendix for this.  Then it can be loaded into the directory:

sudo ldapadd -vx -D "cn=admin,dc=example,dc=org" -H ldapi:/// -w ldaptest -c -f thunderbird.ldif

Use the -c option; otherwise the load will stop on any error, including finding a duplicate entry.

Configuring remote access for update

In order to use Evolution to create entries, and in order to use a remote LDAP management applications (I use JXplorer) you need a user which has the ability to create and delete address book entries; it’s bad practice to use the root user.  So create AddMasterUser.ldif with the following contents:

dn: cn=ldapuser,ou=abook,dc=example,dc=org
cn: ldapuser
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
objectclass: mozillaAbPersonAlpha
sn: ldapuser

Likewise, create AddWriteAccessForMasterUser.ldif thus:

dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcAccess
olcAccess: {0}to *
  by dn="cn=ldapuser,ou=abook,dc=example,dc=org" write
  by self write
  by * read

Use these two ldif files to add the master user and configure read-write access:

sudo ldapadd -vx -D "cn=admin,dc=example,dc=org" -H ldapi:/// -w ldaptest -c -f AddMasterUser.ldif
sudo ldapmodify -Y EXTERNAL -D "cn=admin,dc=example,dc=org" -H ldapi:/// -f AddWriteAccessForMasterUser.ldif

Finally, set the master user’s password like this:

sudo ldappasswd -D "cn=admin,dc=example,dc=org" -H ldapi:/// -w ldaptest -S "cn=ldapuser,ou=abook,dc=example,dc=org"

Testing the LDAP server

You should now be able to see the LDAP address book from Thunderbird by configuring it appropriately.  There are lots of web pages which will tell you how to do this.  

Once configured, Thunderbird should use the LDAP directory for auto-completion, and you should be able to search it.

Configuring Evolution is similar, but if you use the “ldapuser” account and supply its password you will be able to create entries also.

Appendix: Editing the Thunderbird address book file

Thunderbird provides a means of exporting an address book to an LDIF file, but the data in the LDIF is not suitable for loading into an LDAP database with the mozilla schema. Deficiencies include, for example:

  • It doesn’t have a version line.
  • The attribute mozillaUseHtmlMail has numeric values (corresponding, presumably, to the three possible values in the Tbird address book, that is “No”, “Yes” and “Don’t care”), whereas Mozilla’s own address book schema requires TRUE/FALSE. You couldn't make this stuff up, really.

In order to reformat the exported ldif file I hacked up a utility program abookrft, which will do a lot of the heavy lifting for you.  It's not marvellous Perl, in case you worry about that.

Installing abookrft

Download the code from here.  By way of pre-requisite modules, you will need to install Email::Address; everything else is in the standard list.

You can run abookrft from any directory.  For help, enter ./abookrft --help

Features of abookrft

Using a Thunderbird LDIF export as its source, abookrft will:
  • Extract a value for cn from the dn line if it’s not supplied elsewhere.  If nothing else, the cn will be set to an email address.
  • Detect a cn in the form “Blatz, Joe” and reformat it to “Joe Blatz”.
  • Strip out unwanted stuff from most of the fields, especially quote characters, backslash escapes and leading/trailing spaces.
  • Fix the value of mozillaUseHTMLMail if it’s present.
  • Removes the modifytimestamp attribute which, being read-only, cannot be loaded.
  • Reformat member attributes in mailing lists to use the appropriate dn’s; it also checks whether the members in a list exist elsewhere in the address book.
  • Handles multiple entries for a single individual containing different email addresses, in the following slightly convoluted fashion in order to satisfy Thunderbird’s lame email handling as much as possible.
    • Every unique entry results in an LDAP entry.  Numbers in square brackets are appended to the cn to distinguish multiple entries.
    • The first entry found—the one with no number appended—is treated as the “master” entry.
    • The last primary email found in multiple entries for the same person is established as the primary email for the master entry, and all the others are also added as separate mail: lines.
    • The “secondary email” attribute for the last entry found is also placed in the mozillaSecondEmail attribute of the master entry.  Thunderbird will not see the multiple mail: lines, but it will at least see the "second email" field.
    • Thus the first (“master”) entry contains all the mail addresses for that person which were found; the other—numbered—entries each contain just the mail address which appeared in their particular LDIF stanza.
  • Rejects duplicate entries within a single LDIF file—those that are the same in every respect.  It does not query the LDAP directory to detect duplicates.
  • Establish surname (sn) and given name (givenName) from the cn wherever possible.

Appendix: converting a .mab file to ldif for loading

Later note: I don't use Thunderbird any more (migrated to Evolution) so this change passed me by.  But I now know that recent versions of Thunderbird don't store the address book information in abook.mab but in an SQLite database, the structure of which is opaque to me. So what follows pertains only to Thunderbird releases before 78.

If you can’t obtain an exported LDIF version of the Thunderbird address book, you can use the original Thunderbird address book, usually called abook.mab.  As a companion to the ldif-reformatting program referred to above, you may download my even less elegant Perl program which will read the mab file and produce LDIF.
To run it, enter ./thunderbird-ldif abook.mab

Appendix: Configuring Evolution mail for LDAP access

Follow this tutorial, with the following additions and changes.

Encryption: none
Authentication method: Using distinguished name (DN)
Username: cn=ldapuser,ou=abook,dc=example,dc=org
Search base: ou=book,dc=example,dc=org

Evolution is able to add entries to the LDAP address book, which Thunderbird is not.  If you are a Thunderbird user you will have to use something else to add your entries: JXplorer works well for me.



Tuesday 18 May 2021

Installing Manjaro Linux on logical volumes (LVM)

The default Manjaro installer looks as if it will allow you to define LVM partitions during the installation process but it actually won't. I like LVM a lot, and being determined to have a Manjaro system with LVMs underneath I spent some time researching and have what I think is a foolproof way of doing it.  

I'm assuming here that you want separate root and home volumes; if you don't these instructions should still work, but you'll need to adapt them.

Install Manjaro on ordinary partitions

I'll leave you to work out the sizes, and to decide whether you need an EFI partition and so on.

Boot a live CD/USB medium

I like sysrescueCD.

Dump the root and home partitions to external storage

I use fsarchiver to do the dumping.

fsarchiver savefs /mnt/sdb1/prospero/prospero-manjaro-nolvm.fsa /dev/nvme0n1p5 /dev/nvme0n1p5

Delete the root and home partitions and set up a PV

I use parted.  Something like this should work:

parted /dev/nvme0n1
(parted) p                                                                
Model: UMIS RPITJ1TBVME2HWD (nvme)
Disk /dev/nvme0n1: 1024GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system     Name  Flags
 1      1049kB  538MB   537MB   fat32                 boot, esp
 2      538MB   1075MB  537MB   ext4
 3      1075MB  35.4GB  34.4GB  linux-swap(v1)        swap
 4      35.4GB  193GB   157GB   ext4
 5      193GB   1024GB  831GB   ext4


(parted) rm 5                                                             
(parted) rm 4                                                             
(parted) mkpart primary 35.4GB 100%                                       
(parted) set 4 lvm on                                                     
(parted) quit 
                                                             

Set up the volume group and the LVs

[root@sysrescue ~]# pvcreate /dev/nvme0n1p4                               
WARNING: ext4 signature detected on /dev/nvme0n1p4 at offset 1080. Wipe it? [y/n]: y
  Wiping ext4 signature on /dev/nvme0n1p4.
  Physical volume "/dev/nvme0n1p4" successfully created.
[root@sysrescue ~]# vgcreate vg01 /dev/nvme0n1p4
  Volume group "vg01" successfully created
[root@sysrescue ~]# lvcreate -L 100G --name root vg01
  Logical volume "root" created.
[root@sysrescue ~]# lvcreate -L 500G --name home vg01
  Logical volume "home" created.

Restore the dumped partitions

fsarchiver restfs /mnt/sdb1/prospero/prospero-manjaro-nolvm.fsa id=0,dest=/dev/mapper/vg01-root id=1,dest=/dev/mapper/vg01-home

Mount the root partition

mkdir /linux
mount /dev/mapper/vg01-root /linux
 

Get the UUIDs of the partitions

blkid /dev/mapper/vg01-root /dev/mapper/vg01-home

 Now use an editor (I like sed for this) to change the UUIDs for the root and home filesystems in /linux/etc/fstab to be the UUIDs displayed above.  You may not need to make any changes.

 chroot into the mounted root filesystem

modprobe efivarfs
arch-chroot /linux

(That "modprobe" is in there because of this Stack Exchange article.)

Mount required other filesystems and reinstall grub

mount /dev/nvme0n1p2 /boot
mount /dev/nvme0n1p1 /boot/efi
mount -t efivarfs efivarfs /sys/firmware/efi/efivars
grub-install /dev/nvme0n1

Add LVM2 to the hooks in the initramfs

(Otherwise the LVM filesystems aren't seen by the kernel, which doesn't end well)

sed -i 's/base udev autodetect modconf block/base udev autodetect modconf block lvm2/' /etc/mkinitcpio.conf

 The LVM2 hook must be after "block".

mkinitcpio -P

Reboot and your system should be running again


 

Monday 19 October 2020

Removing non-Latin fonts from Mint/Ubuntu

The default installation of Mint (now 20) includes a wide range of fonts for non-Latin alphabets - Chinese, Gujarati and so forth.  It's excellent that Ubuntu provides these, of course, but it seems odd that they are automatically included in an installation carried out after specifying a Latin-character language (UK English in my case).  It's also rather inconvenient because lists of fonts (in word-processing applications, for example) contain large numbers of fonts which will be inappropriate for most UK, US and European users virtually all of the time. 

During my recent Mint upgrade I spent a bit of time creating a list of the font packages which can be removed.  With one or two exceptions the list I came up with is this:

fonts-beng fonts-beng-extra fonts-gubbi fonts-gujr fonts-gujr-extra fonts-guru fonts-guru-extra fonts-kacst fonts-kacst-one fonts-kalapi fonts-khmeros-core fonts-lao fonts-lohit-beng-assamese fonts-lohit-beng-bengali fonts-lohit-deva fonts-lohit-gujr fonts-lohit-guru fonts-lohit-knda fonts-lohit-mlym fonts-lohit-orya fonts-lohit-taml fonts-lohit-taml-classical fonts-lohit-telu fonts-mlym fonts-noto-cjk fonts-pagul fonts-sahadeva fonts-samyak-deva fonts-samyak-gujr fonts-samyak-mlym fonts-samyak-taml fonts-sarai fonts-sil-abyssinica fonts-sil-padauk fonts-smc fonts-smc-anjalioldlipi fonts-smc-chilanka fonts-smc-dyuthi fonts-smc-gayathri fonts-smc-karumbi fonts-smc-keraleeyam fonts-smc-manjari fonts-smc-meera fonts-smc-rachana fonts-smc-raghumalayalamsans fonts-smc-suruma fonts-smc-uroob fonts-tamil fonts-telu fonts-telu-extra fonts-thai-tlwg fonts-tibetan-machine fonts-tlwg-garuda fonts-tlwg-garuda-ttf fonts-tlwg-kinnari fonts-tlwg-kinnari-ttf fonts-tlwg-laksaman fonts-tlwg-laksaman-ttf fonts-tlwg-loma fonts-tlwg-loma-ttf fonts-tlwg-mono fonts-tlwg-mono-ttf fonts-tlwg-norasi fonts-tlwg-norasi-ttf fonts-tlwg-purisa fonts-tlwg-purisa-ttf fonts-tlwg-sawasdee fonts-tlwg-sawasdee-ttf fonts-tlwg-typewriter fonts-tlwg-typewriter-ttf fonts-tlwg-typist fonts-tlwg-typist-ttf fonts-tlwg-typo fonts-tlwg-typo-ttf fonts-tlwg-umpush fonts-tlwg-umpush-ttf fonts-tlwg-waree fonts-tlwg-waree-ttf fonts-yrsa-rasa

If you want to do the same, use the list above with sudo apt remove.

Friday 9 October 2020

NEC Multisync 1970NX display not centred, no controls to move it

I have a trusty NEC LCD1970NX 17" display which is in regular use with the domestic Linux workhorse, connected via the DVI interface.  Recently I've been building a Windows 10 machine, for occasional use, which will be connected to the same monitor; for a number of reasons the new machine will be connected via the VGA socket on the display: there's a handy "select source" button on the screen so I can switch between the two when necessary.

But when I brought the Windows system up, the display image was offset about 15mm to the right: quite unusable.  Worse, the "Menu" button on the display did not bring up the image-adjustment controls referred to in the manual: instead it offered only brightness/contrast adjustment.

The solution, admittedly a sledge-hammer-scale fix for what should be a nut-sized problem, was this:

  • On the Windows machine, install NEC's "NaViSet Administrator" application, a 200MB-odd download from NEC.  This is a tool for systems administrators, allowing them remote access to NEC displays and projectors across a network.
  • Also install the "DDC/CI WMI Provider" client on the same machine; this is the component that talks to NaViSet to provide the control functions.  Without it you get no information about the current state of the monitor, and can make no changes.
  • Fire up NaViSet, again on the same machine, add the PC to the network, navigate to it and find its display.
  • There is now a "Geometry" tab which allows modification of the image position on the screen.  For me the "Auto Setup" function did the trick.

Here's what the NaViSet window looks like:



Monday 26 August 2019

LibreOffice can't open files on NFS server

Trying to open a LibreOffice spreadsheet on an NFS-mounted directory within my home network.  LibreOffice comes up with "locked for editing by unknown user". 

Many posts will tell you to look for a hidden lock file in the same directory: I had none.  In fact there were no files anywhere of any consequence with the names *lck* or *lock*.

I found a comment on an Ubuntu LibreOffice bug 1751005 which corresponded with some AppArmor messages I found in the local client log, and took the requisite action: I still had the problem.  I'm now running LibreOffice 6 so that action may have been unnecessary, actually.

I tailed the server log and attempted the failing access: instantly there came a lockd: cannot monitor myclient message.

Finally I found the problem: rpc-statd.service was not running on the Ubuntu 18.04 server which was providing the NFS mounts.  I enabled the service and all is now well.

Friday 14 December 2018

Why has emacs stopped running in a window?

I use several editors (and also Eclipse, which has its own) and manage to be reasonably productive with all of them, but all other things being equal emacs is my favourite.I have it installed as an icon on the Linux Mint start bar.

Imagine my annoyance and distress when yesterday clicking on the emacs icon did nothing; no error message, no window display, just a momentary blink and then nothing.  emacs would run in a terminal window, but not in its own emacs window on the desktop.  Reinstalling the "emacs" package didn't help.

I traced the problem to a mistake I made yesterday.  I have a list of packages to install on a remote Ubuntu server, which has no X windows installed and which runs headless.  In error I typed the "apt install" command into a window which was not ssh-ed into the local machine, thus installing all those packages on my local laptop (the one where emacs was failing to run).  I do my development and testing locally so in fact installing those packages was a no-op, with one important exception: I use emacs on the Ubuntu Server system, but to avoid problems install only the terminal-only version, called emacs-nox.  So of course I had installed emacs-nox on my laptop, which had faithfully done what was required of it: stop emacs running in an X Window.  I purged it, reinstalled emacs and all was well. 

Saturday 6 January 2018

A run-once system for linux

Current development of the Linux image for the community centre I'm involved with requires me to be able to carry out tasks just once (a "run once" capability). Moreover, I have need to run such tasks at three different stages in the overall process:
  1. At start up before the network is active (to manipulate network definitions)
  2. At start up after the network is active but before users log in (in my case to incorporate the IP address into a workspace wallpaper)
  3. At shut down, after the users are logged off but while the filesystems are still mounted.

The run-once script

I have a run-once script /usr/local/bin/runonce, based on one I found here.
RUNONCE_DIR=$1
[ -z $RUNONCE_DIR ] && RUNONCE_DIR=/etc/local/runonce.d

[ ! -d $RUNONCE_DIR/ran ] && mkdir -p $RUNONCE_DIR/ran

for FILE in $RUNONCE_DIR/*; do
    [ -d "$FILE" ] && continue
    COMMAND_NAME=$(basename $FILE)
    logger -it runonce -p local3.info "Running: $FILE"
    "$FILE"
    mv "$FILE" "$RUNONCE_DIR/ran/$COMMAND_NAME-$$-$(date +%Y%m%d-%H%M%S)"
    logger -it runonce -p local3.info "Ran: $FILE"
done


I use it in all three places, with three directories /etc/local/runonce-a.d, /etc/local/runonce-b.d and so on.  The parameter to the runonce job specifies which one to use.

The three jobs are then controlled by systemd unit files (I'm just getting used to systemd ..), as follows:

runonce-a.service

# Run-once service; runs any executable in /etc/local/runonce-a.d
# Ensuring that it runs before the network is active
[Unit]
Description=Run once programs (pre network)

Before=network-pre.target
Wants=network-pre.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/runonce /etc/local/runonce-a.d

[Install]
WantedBy=network.target

runonce-b.service

# Run-once service; runs any executable in /etc/local/runonce-b.d
# Ensuring that it runs after the network is active
[Unit]
Description=Run once programs (post-network)

After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/runonce /etc/local/runonce-b.d

[Install]
WantedBy=multi-user.target

runonce-c.service

# Run-once service; runs any executable in /etc/local/runonce-c.d
# before the system is shut down or rebooted.
[Unit]
Description=Run once programs (before shutdown)
RequiresMountsFor=/home

[Service]
Type=oneshot
ExecStop=/usr/local/bin/runonce /etc/local/runonce-c.d
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target


It took quite a while to assemble the various bits of this from around the web; hope it helps someone else.