Metadata-Version: 2.4
Name: aa-ldap-directory
Version: 0.1.0a1
Summary: Project the Alliance Auth core into an LDAP directory
Keywords: allianceauth,ldap,directory,eve-online
Author: Boris Talovikov
Author-email: Boris Talovikov <boris.t.66@gmail.com>
License-Expression: GPL-2.0-or-later
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Web Environment
Classifier: Framework :: Celery
Classifier: Framework :: Django
Classifier: Framework :: Django :: 5.2
Classifier: Intended Audience :: System Administrators
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: System :: Systems Administration :: Authentication/Directory :: LDAP
Requires-Dist: allianceauth>=5.0.0,<6
Requires-Dist: django-redis>=5.4,<8
Requires-Dist: python-ldap>=3.4.8,<4
Maintainer: Boris Talovikov
Maintainer-email: Boris Talovikov <boris.t.66@gmail.com>
Requires-Python: >=3.10, <3.15
Project-URL: Repository, https://gitlab.com/eveo7/aa-ldap-directory
Project-URL: Changelog, https://gitlab.com/eveo7/aa-ldap-directory/-/blob/main/CHANGELOG.md
Project-URL: Issues, https://gitlab.com/eveo7/aa-ldap-directory/-/issues
Description-Content-Type: text/markdown

# aa-ldap-directory

Project the Alliance Auth core into an LDAP directory (OpenLDAP or 389
Directory Server): users with the attributes of their main character,
corporations, alliances, groups and states, so that LDAP clients read
the community from one place. The directory is a read model: nobody
authenticates against it.

An Alliance Auth module.

## Status

Alpha. The package validates its settings, prints its LDAP schema,
writes the entries of changed objects through Celery tasks and the
whole tree with the sync marker in a scheduled full pass, and checks a
running export.

## Requirements

- Alliance Auth 5.x, Python 3.10 to 3.14.
- `CACHES['default']` on `django_redis.cache.RedisCache`, as the
  Alliance Auth settings configure it; `django-redis` 5.4 to 7.x is a
  dependency of the plugin.
- A C compiler and the OpenLDAP client and Cyrus SASL headers: the
  LDAP client `python-ldap` ships no wheels and is compiled on
  install. On Debian 12/13 and Ubuntu 24.04:

  ```bash
  apt-get install --no-install-recommends build-essential libldap-dev libsasl2-dev
  ```

  The Alliance Auth Docker image ships gcc but neither header
  package, and its `custom.dockerfile` installs requirements as a
  non-root user, so the `apt-get` line has to run as root before
  that step. Without the headers the install fails with
  `fatal error: lber.h: No such file or directory`.

## Installation

The steps run in this order: the plugin writes as soon as it is
configured, so the server has to be ready first. The examples use the
suffix `dc=example,dc=org` and the accounts of "Server setup".

1. On the LDAP server, create the suffix and the accounts, load the
   schema, then create `ou=allianceauth` as the server administrator,
   set the access rules and the size limits (see "Server setup" and
   "LDAP schema"). Leave `ou=characters` closed until step 8.
2. Install the package into the virtual environment of Auth, with the
   build headers of "Requirements" present:
   `pip install aa-ldap-directory==0.1.0a1`.
3. Add the settings of "Configuration" to `local.py` and
   `aa_ldap_directory` to `INSTALLED_APPS`, after the Alliance Auth apps.
4. `python manage.py check` prints no `aa_ldap_directory` error.
5. Add the beat entry of "Schedule". With
   `AA_LDAP_DIRECTORY_TASK_QUEUE` set, start a worker for that queue
   (see "Celery worker").
6. Restart Auth, its workers and beat, so they load the plugin.
7. Run one pass and the check:

   ```bash
   python manage.py ldapdirectory_sync
   python manage.py ldapdirectory_check
   ```

   The first prints `full pass finished:` and exits 0; the second prints
   one line per check and exits 0 when none is an `error` (see
   "Checking a running export").
8. Only for the characters branch: first give `ou=characters` its read
   rule on the server, then set `AA_LDAP_DIRECTORY_EXPORT_CHARACTERS`
   and `AA_LDAP_DIRECTORY_CHARACTERS_ACL_CONFIRMED` and restart. The
   other order publishes the alts of every main to every reader of the
   subtree until the rule exists. The rule does not cover
   `ou=corporations` and `ou=alliances`: with the branch on they also
   list the corporations and alliances of alts, so one that only alts
   belong to shows every reader where the community keeps alts.

A failure at step 7 leaves the entries written so far; the next pass
completes them. To back out, follow "Removing the plugin".

## Configuration

Settings go into the `local.py` of the Auth project. Five are required;
with any of them absent every `manage.py` command prints a system check
error. Add `aa_ldap_directory` to `INSTALLED_APPS` after the Alliance
Auth apps.

```python
import os

AA_LDAP_DIRECTORY_URI = "ldaps://ldap.example.org"
AA_LDAP_DIRECTORY_BIND_DN = "cn=aa-writer,dc=example,dc=org"
AA_LDAP_DIRECTORY_BIND_PASSWORD = os.environ["AA_LDAP_DIRECTORY_BIND_PASSWORD"]
AA_LDAP_DIRECTORY_BASE_DN = "dc=example,dc=org"
AA_LDAP_DIRECTORY_ACCESS_PERMISSIONS = ["app_label.codename"]
```

| Setting | Type | Default | Meaning |
|---|---|---|---|
| `AA_LDAP_DIRECTORY_URI` | `str` | required | `ldaps://host[:port]`, or `ldap://host[:port]` with StartTLS; one server, no path, query or credentials; scheme in lower case |
| `AA_LDAP_DIRECTORY_START_TLS` | `bool` | `False` | upgrade an `ldap://` connection to TLS; `True` with `ldaps://` is an error |
| `AA_LDAP_DIRECTORY_ALLOW_PLAINTEXT` | `bool` | `False` | accept `ldap://` without StartTLS; for a development server only, reported as a warning |
| `AA_LDAP_DIRECTORY_CA_FILE` | `str` or `None` | `None` | PEM bundle that verifies the server certificate; `None` uses the system trust store; must be a file on every host that runs Auth or its workers |
| `AA_LDAP_DIRECTORY_CONNECT_TIMEOUT` | number, seconds | `5.0` | limit for the TCP connect and the TLS handshake; above zero |
| `AA_LDAP_DIRECTORY_OPERATION_TIMEOUT` | number, seconds | `30.0` | limit for one LDAP operation; above zero |
| `AA_LDAP_DIRECTORY_BIND_DN` | `str`, DN | required | identity the plugin binds as |
| `AA_LDAP_DIRECTORY_BIND_PASSWORD` | `str` | required | password of the bind DN; no message of the plugin quotes it |
| `AA_LDAP_DIRECTORY_BASE_DN` | `str`, DN | required | suffix under which the plugin owns `ou=allianceauth` |
| `AA_LDAP_DIRECTORY_ACCESS_PERMISSIONS` | list of `str` | required, non-empty | permissions, `app_label.codename`, that set `aaCommunityAccess` |
| `AA_LDAP_DIRECTORY_PERMISSION_ALLOWLIST` | list of `str` | `()` | permissions exported as `aaPermission`; empty exports none |
| `AA_LDAP_DIRECTORY_EXPORT_CHARACTERS` | `bool` | `False` | export the `ou=characters` branch, which links mains to alts; it also adds the corporations and alliances of alts to `ou=corporations` and `ou=alliances`, which every reader sees (see step 8 of "Installation") |
| `AA_LDAP_DIRECTORY_CHARACTERS_ACL_CONFIRMED` | `bool` | `False` | the operator confirms a read ACL on `ou=characters`; without it the branch is not exported |
| `AA_LDAP_DIRECTORY_GROUP_DENYLIST` | list of `int` or `str` | `()` | groups never exported, by primary key or by exact name, case-sensitive on every database; a name holds no NUL; empty exports every group. Prefer the primary key: a name entry stops matching, without a warning, once the group is renamed, and the group is exported; use a name only for a group that is never renamed |
| `AA_LDAP_DIRECTORY_EXPORT_EMAIL` | `bool` | `False` | export the user's email as `mail`; a placeholder address is never exported; Heddle does not read `mail` |
| `AA_LDAP_DIRECTORY_TASK_QUEUE` | `str` or `None` | `None` | Celery queue of the plugin's tasks; `None` is Celery's default queue |

A list setting accepts a list or a tuple; a bare string is a type error.
A number setting refuses `True` and `False`.

Silent failure: a `AA_LDAP_DIRECTORY_TASK_QUEUE` that no worker consumes
raises no error anywhere; the tasks wait in the broker. Start a worker
with `-Q` and that name (see "Celery worker"); `ldapdirectory_check`
reports a queue without one.

### System checks

The checks read only the settings: they do not dial the LDAP server and
do not query the database, so they do not see a wrong password, an
unreachable server or a permission name that does not exist. A Celery
worker runs no system checks; it refuses the same values when it reads
them, with `ImproperlyConfigured` carrying the same id.

| Id | Level | Raised when |
|---|---|---|
| `aa_ldap_directory.E001` | error | a required setting is absent, blank, or the access permission list is empty |
| `aa_ldap_directory.E002` | error | a setting has the wrong type; the message names the type, never the value |
| `aa_ldap_directory.E003` | error | the URI is not `ldap://` or `ldaps://` followed by one host and an optional port |
| `aa_ldap_directory.E004` | error | `ldap://` without StartTLS and without `ALLOW_PLAINTEXT` |
| `aa_ldap_directory.E005` | error | StartTLS on an `ldaps://` URI |
| `aa_ldap_directory.E006` | error | the CA file is not a file on this host |
| `aa_ldap_directory.E007` | error | a timeout is zero, negative, infinite or NaN |
| `aa_ldap_directory.E008` | error | the bind DN or the base DN does not parse as an RFC 4514 DN |
| `aa_ldap_directory.E009` | error | a permission is not of the form `app_label.codename` |
| `aa_ldap_directory.E010` | error | `EXPORT_CHARACTERS` without `CHARACTERS_ACL_CONFIRMED` |
| `aa_ldap_directory.E011` | error | a group denylist entry is neither an int from 1 to 2147483647 (a primary key) nor a string with a character other than whitespace and without NUL |
| `aa_ldap_directory.E012` | error | the task queue name is blank |
| `aa_ldap_directory.E013` | error | `CACHES['default']` is not `django_redis.cache.RedisCache` or a subclass, or its `OPTIONS['CLIENT_CLASS']` is neither `DefaultClient` nor a subclass of it, or is `HerdClient`, `ShardClient` or a subclass of either (both subclass `DefaultClient`); `SentinelClient` passes. The task locks need Redis and a client whose stored value a Lua script can compare |
| `aa_ldap_directory.W001` | warning | `ldap://` without StartTLS, accepted by `ALLOW_PLAINTEXT` |

## What is exported

The plugin owns `ou=allianceauth,<AA_LDAP_DIRECTORY_BASE_DN>` and
writes these entries under it. The tree is built from the database
alone; tasks write the entries of changed objects (see "Writing on
change"), the full pass writes all of them and deletes everything else
in the subtree (see "Full pass").

| Branch | RDN | Object classes | One entry per |
|---|---|---|---|
| `ou=users` | `aaUserId=<User.pk>` | `inetOrgPerson`, `aaUser` | every Django user, active or not |
| `ou=characters` | `aaCharacterId=<EVE id>` | `aaCharacter` | character with a `CharacterOwnership`; only with `EXPORT_CHARACTERS` and `CHARACTERS_ACL_CONFIRMED` |
| `ou=corporations` | `aaCorporationId=<EVE id>` | `organization`, `aaOrganization` | corporation of a main, or of an owned character while `ou=characters` is exported |
| `ou=alliances` | `aaAllianceId=<EVE id>` | `organization`, `aaOrganization` | alliance of such a character or of such a corporation |
| `ou=groups` | `cn=<Group.name>`, see below | `groupOfNames`, `aaGroup` | Django group outside `GROUP_DENYLIST` |
| `ou=states` | `cn=<State.name>`, see below | `aaState` | Alliance Auth state |
| `cn=sync-status` | | `applicationProcess`, `aaSyncStatus` | the generation marker of the last completed full pass |

Every DN an entry names (`aaOwner`, `aaCorporation`, `aaAlliance`,
`member`) is an entry of the same tree, and no DN occurs twice:

- A corporation or alliance comes from its `EveCorporationInfo` or
  `EveAllianceInfo` row. Without a row, the entry exists while an
  exported character names the organization, and it is built from the
  cached affiliation fields of the character of that organization with
  the highest `character_id`, exported or not (name, ticker, and for a
  corporation the alliance id): a change of main or of ownership does
  not move the source. Alliance Auth keeps no affiliation timestamp common to 5.0 to
  5.5, so the cache of the most recent update cannot be told apart;
  the highest id only makes the choice deterministic. When characters
  disagree, the corporation entry follows that one character. Rows and
  cache are merged by id, the row winning.
- A character is exported only while its owner has a user entry, and
  a group lists only members that have one. `aaCorporation` of a
  character and `aaAlliance` of a corporation are written only while
  the named entry is in the tree; both attributes are optional in the
  schema.

The rule holds without relying on transaction isolation: Django opens
MySQL, MariaDB and PostgreSQL connections in READ COMMITTED, so a
change committed while the plugin reads the database can be half
visible. Such a change may leave a reference out for one pass; it never
leaves a reference to a missing entry.

### Group and state names

Django keeps groups apart by exact name, a server compares `cn` without
case and with some Unicode and space normalisation. The plugin compares
group names by a key built per RFC 4518 for `caseIgnoreMatch`: the
compatibility caseless match of Unicode (definition D145: case folded
twice, each fold between compatibility decompositions), composed to
NFKC, leading and trailing spaces dropped, inner runs of spaces
collapsed. The key of a key is the key itself. The key is no finer than
the server's comparison: two names
the server compares as equal always share a key, while names with one
key may still be two RDNs to the server, so a group can carry
`+aaGroupId` without need. Measured on OpenLDAP 2.6.10 and 389
Directory Server 3.1.2 with 33 `cn` pairs under one parent (case,
NFKC forms, spaces, tabs, sharp s against `ss`, mathematical and
modifier capitals); no pair with distinct keys
collided on a server. `tests/test_name_key_ldap.py` (tag `ldap`, run by
`tests_ldap`) pins 22 pairs per server. The key is deliberately coarser
than both servers, never finer: it folds together pairs a server keeps
apart. On both servers these are a tab against a space, U+001C to
U+001F against a space, sharp s against `ss`, a mathematical capital
such as U+1D400 against the plain small letter, U+1D2C MODIFIER LETTER
CAPITAL A against `a` and U+037A against the small iota; on 389ds also a
no-break space against a space, a combining accent against the composed
letter, the U+FB01 ligature against `fi`, and a name of one space
against a name of two (measured 2026-11-02). Such a rare pair costs an
extra `+aaGroupId`. The key itself is pinned by
`NameCollisionProjectionTests` in `tests/unit/test_projection.py`.

Every group whose key collides with another exported group's gets the
multi-valued RDN `cn=<name>+aaGroupId=<pk>`; every other group keeps
`cn=<name>`. The empty name collides with none, as its RDN is its pk
(below): a name of spaces next to it keeps `cn=<name>`. States follow
the same rule with `cn=<name>+aaStateId=<pk>`. A generic client that
builds a group DN from a name breaks on a colliding name; look entries
up by `(aaGroupId=<pk>)` or `(aaStateId=<pk>)` instead. A rename that
creates or removes a collision changes the DN of every group involved.

Most of the names below reach a group or state only from code
(`Group.objects.create`, an import, another app). The admin form strips
leading and trailing whitespace, which includes TAB, LF, CR, VT, FF and
U+001C to U+001F, and refuses an empty name and a NUL; it keeps
U+0001 to U+0008 and U+000E to U+001B at either end.

- A name with a C0 control character (U+0000 to U+001F: TAB, LF and
  CR among them) at either end, or a NUL anywhere, is not exported to
  any server. Both servers accept such a DN, but 389 Directory Server
  3.1.2 returns it altered: an edge TAB raw, an edge LF or CR dropped,
  `\00` as `000` next to a second `cn` value, and refuses to rename an
  entry with `\00`. Such a group is left out like a denylisted one: its
  entry is deleted, also when an exported group is renamed to such a
  name, and no other group collides with it. A user in such a state has
  neither `aaState` nor `aaStateId`.
- Any other control character, and DEL anywhere, is written escaped as
  a hex pair in the DN (`cn=Fleet\09Ops`) and kept as it is in `cn`;
  both servers return it unchanged.
- An empty name gets the RDN `cn=<pk>+aaGroupId=<pk>`
  (`cn=<pk>+aaStateId=<pk>` for a state) whether or not it collides,
  and `cn` holds the pk; so does `aaState` of a user in such a state.
- A name of one space gets the RDN `cn=\20` (`cn=\20+aaGroupId=<pk>`
  when it collides), and `cn` holds the space. 389 Directory Server
  3.1.2 refuses the spelling with an escaped space, `cn=\` and a
  space, with result code 21; OpenLDAP 2.6.10 accepts both spellings as
  one RDN. Measured 2026-11-02.

Measured 2026-10-30 on OpenLDAP 2.6.10 and 389 Directory Server 3.1.2
with every C0 control and DEL inside a name and TAB, LF, CR and DEL at
its ends.

### User entries

- `uid` is the username, `aaUserId` the primary key; the DN never
  changes on a rename.
- A user with an empty username, which only code can store (Django's
  `create_user` and the admin form refuse it), is not exported to any
  server: its `uid`, `cn`, `sn` and `displayName` would be empty, and
  both servers refuse an empty value. It has no entry, no group lists
  it in `member`, and its characters are not exported, as they would
  name a missing owner. A user renamed to the empty name loses its
  entry, its `member` values and its characters; renamed back, it gets
  them again. Its main and characters still count for the
  corporations and alliances exported, whose entries name no user.
  `ldapdirectory_check` names such users (`unexported_names`).
- `cn`, `sn` and `displayName` are the name of the main, or the
  username for a user without a main.
- The attributes of the main, `aaMainCharacterId`,
  `aaMainCharacterName`, `aaCorporation{Id,Name,Ticker}`,
  `aaAlliance{Id,Name,Ticker}` and `aaFaction{Id,Name}`, are copied
  from the cached fields of the main `EveCharacter`, the same fields
  the Alliance Auth OIDC provider reads for its `eve_*` claims. A
  blank field is omitted, as the provider omits the claim; a user
  without a main has none of them.
- `aaActive` is `User.is_active`; `aaState` and `aaStateId` are the
  state of the profile. For an inactive user that is whatever Alliance
  Auth stored, not necessarily Guest: deactivation sets Guest, but a
  later save of the profile or of the main (an ESI update), or a change
  of a state definition, reassesses the state without looking at
  `is_active`.
- `aaCommunityAccess` is `TRUE` when the user holds any permission of
  `ACCESS_PERMISSIONS`, `FALSE` otherwise. Permissions are evaluated
  as `user.has_perm` does with the Alliance Auth backends: user, group
  and state permissions; an active superuser holds every permission,
  an inactive user none.
- Fail-safe: when a permission of `ACCESS_PERMISSIONS` has no row in
  the database (a typo, an app removed), `aaCommunityAccess` is not
  written at all. Values already in LDAP stay, a new user entry is
  created without the attribute, and its absence means "not
  processed", never "no access".
- `aaPermission` lists the permissions of `PERMISSION_ALLOWLIST` the
  user holds.
- `preferredLanguage` is the language of the profile, omitted when
  unset.
- `mail` is exported only with `EXPORT_EMAIL = True`, and never for a
  placeholder address (next section).

A group lists its users in `member`. A group without members names
itself, because OpenLDAP requires one `member` value on
`groupOfNames`; a reader counts only members under `ou=users`.

### Placeholder email addresses

With `EXPORT_EMAIL = True`, `mail` carries `User.email` stripped of
surrounding whitespace, except when it is a placeholder:

- blank, or without a local part and a domain around the last `@`;
- with a character outside ASCII, such as `pilot@bücher.de`: `mail` is
  an IA5String, so OpenLDAP and 389 Directory Server refuse the whole
  user entry with `Invalid syntax (21)`; an internationalized domain is
  not converted to punycode;
- at the domain of `AA_SKIP_EMAIL_DOMAIN`, compared without case, the
  setting the aa-skip-email module fills placeholders from; its own
  default `no-email.invalid` is covered by the next rule;
- at a domain reserved from mail delivery, or a subdomain of one:
  the top-level names `invalid`, `test`, `example`, `localhost`
  (RFC 2606, RFC 6761) and `example.com`, `example.net`,
  `example.org` (RFC 2606).

The rule has no setting of its own. A placeholder scheme at another
domain is not recognised and is exported as a real address.

## Writing on change

Signal receivers queue Celery tasks when the transaction that changed an
object commits; the tasks read the object from the database again and
write, rename or delete its entry. Tasks are best effort: a change a
task cannot write, or one no signal reports, reaches LDAP with the next
full pass (see "Full pass"). Start a worker that consumes
`AA_LDAP_DIRECTORY_TASK_QUEUE`.

### Celery worker

With `AA_LDAP_DIRECTORY_TASK_QUEUE = None` the tasks go to Celery's
default queue, which the workers of Alliance Auth already consume. With
a queue of its own, say `"ldap"`, start a worker for it next to them,
from the directory of the Auth project:

```bash
celery -A myauth worker -Q ldap --hostname ldap@%h
```

In a supervisor `command=` line `%h` is written `%%h`. The worker has to
load the Auth project, as the other workers do: it imports the plugin's
tasks and answers the worker check of `ldapdirectory_check`.

### Receivers

Every receiver of the plugin, as `aa_ldap_directory.receivers.build_receiver_table`
connects it. "Lock" is the lock the task holds per object. The last
column is what the task does when the object is gone from the database
or no longer exported.

| Signal | Sender | Receiver | Task, lock | Object gone |
|---|---|---|---|---|
| `post_save` | `User` | `on_user_saved` | `sync_users`, `user:<pk>`; skipped when `update_fields` names none of `username`, `email`, `is_active`, `is_superuser` (a login); an entry that appears or disappears queues `update_group_members` per group of the user, `group:<pk>`, and `sync_characters` for its characters | entry deleted for an empty username (see "User entries") |
| `pre_delete` | `User` | `on_user_deleting` | none: remembers the user's groups, whose rows the cascade deletes without `m2m_changed` | - |
| `post_delete` | `User` | `on_user_deleted` | `sync_users`, `user:<pk>`; `update_group_members` per remembered group, `group:<pk>` | user entry deleted, its `member` values removed |
| `post_save` | `UserProfile` | `on_profile_changed` | `sync_users`, `user:<pk>`; skipped when `update_fields` names none of `state`, `main_character`, `language` | - |
| `post_delete` | `UserProfile` | `on_profile_changed` | `sync_users`, `user:<pk>` | entry without state, main and language |
| `state_changed` | `UserProfile` | `on_state_changed` | `sync_users`, `user:<pk>` | - |
| `post_save` | `CharacterOwnership` | `on_ownership_changed` | `sync_characters`, `character:<EVE id>`; only while `ou=characters` is exported | - |
| `post_delete` | `CharacterOwnership` | `on_ownership_changed` | `sync_characters`, `character:<EVE id>`; only while `ou=characters` is exported | character entry deleted |
| `pre_save` | `EveCharacter` | `on_character_saving` | nothing; remembers the character's corporation and alliance (one query) for the row below | - |
| `post_save` | `EveCharacter` | `on_character_saved` | `sync_characters`, `character:<EVE id>`, while `ou=characters` is exported; `sync_users` for the user whose main it is; `sync_corporations` and `sync_alliances` for its corporation and alliance when they have no `EveCorporationInfo` or `EveAllianceInfo` row and this character is their name source: their character with the highest EVE id, exported or not, whose cached name and ticker (a corporation's alliance too) the entry carries (one query per organization for its row, one more without); the same two for a corporation or alliance without a row that the character left | - |
| `post_delete` | `EveCharacter` | `on_character_deleted` | `sync_characters`, `character:<EVE id>`; only while `ou=characters` is exported; `sync_corporations` and `sync_alliances` for its corporation and alliance without a row | character entry deleted |
| `post_save` | `EveCorporationInfo` | `on_corporation_changed` | `sync_corporations`, `corporation:<EVE id>` | - |
| `post_delete` | `EveCorporationInfo` | `on_corporation_changed` | `sync_corporations`, `corporation:<EVE id>` | entry from the character cache, or deleted when no exported character names it |
| `post_save` | `EveAllianceInfo` | `on_alliance_changed` | `sync_alliances`, `alliance:<EVE id>` | - |
| `post_delete` | `EveAllianceInfo` | `on_alliance_changed` | `sync_alliances`, `alliance:<EVE id>` | entry from the character cache, or deleted when nothing names it |
| `post_save` | `Group` | `on_group_changed` | `sync_group`, `groups` and `group:<pk>` | entry deleted when the group is denylisted |
| `post_delete` | `Group` | `on_group_changed` | `sync_group`, `groups` and `group:<pk>` | entry deleted |
| `post_save` | `State` | `on_state_definition_changed` | `sync_state`, `states` and `state:<pk>`; a change of the entry's `cn` (a rename, a name that stops or starts being exported, a new state) also queues `fan_out_users` for the state's users | - |
| `post_delete` | `State` | `on_state_definition_changed` | `sync_state`, `states` and `state:<pk>`; also queues `fan_out_users` for the state, which finds no user: Alliance Auth moves the profiles to another state before the delete | entry deleted |
| `m2m_changed` | `User.groups.through` | `on_membership_changed` | `update_group_members` per group, `group:<pk>`; `post_add`, `post_remove`, `post_clear` with the pks read at `pre_clear` | value removed |
| `m2m_changed` | `User.user_permissions.through` | `on_user_permissions_changed` | `sync_users`, `user:<pk>` | - |
| `m2m_changed` | `Group.permissions.through` | `on_group_permissions_changed` | `fan_out_users`: `sync_users` for the group's members in batches of 100, only when a changed permission is exported | - |
| `m2m_changed` | `State.permissions.through` | `on_state_permissions_changed` | `fan_out_users`: `sync_users` for the state's users in batches of 100, only when a changed permission is exported | - |

What a task writes changes what other entries depend on. A task
compares the entry before and after its write and queues the entries
that follow: a user or character whose `aaCorporationId` or
`aaAllianceId` changed queues both the old and the new organization,
which is deleted once no exported character names it; a user whose
`aaMainCharacterId` changed queues both characters for `aaIsMain`; a
member update of a group that grants an exported permission queues the
users, also when the group is denylisted or its entry is not written
yet: effective permissions count every group. A group or state task
also moves the entries of its namesakes: a rename, a deletion or a
denylisting that creates or ends a name collision changes their RDN
too (see "Group and state names").

Group membership is written one value at a time: `update_group_members`
adds or removes the `member` value of each named user as the database
says now, so tasks that run out of order converge. The arrival of the
first member removes the group's own DN, and the departure of the last
adds it, in the same modify, because OpenLDAP refuses a `groupOfNames`
without `member`. A value another writer added or removed between the
read and the modify fails the whole modify; the task then reads the
entry again and computes the modify anew, once, and after a second
failure sends itself again under the lock rules below. Only the full
pass replaces `member` as a whole.

### Locks and retries

A task holds a lock per object in the django-redis cache
`CACHES['default']`, which every worker shares: `cache.add` with a
random token. Releasing and extending a lock are each one Lua script on
the Redis server that compares the token first, so a lock that expired
and was taken by another worker is never deleted or extended. Another
cache backend is the system check error `aa_ldap_directory.E013`, and
so is a django-redis client other than `DefaultClient` and its
subclasses, or `HerdClient`, `ShardClient` or a subclass of either,
although both subclass `DefaultClient`; `SentinelClient` passes.
`HerdClient` could be supported only by taking and checking the locks
around its value packing (`SET NX EX` and the scripts on the raw Redis
client instead of `cache.add`); the plugin does not do that. A
worker refuses such a cache at the first task with `setting refused`. A cache
that fails while a task takes its locks gets the locks of that call
released as far as it still answers; a lock it cannot release expires
with its timeout. A lock
lives for the connect timeout plus 16 operation timeouts, 485 seconds
with the defaults; the lock of a worker that died expires after that
time. A group or state task that moves namesakes also takes the lock of
each moved group or state, so no member update of it runs meanwhile,
and extends all its locks to 12 plus 7 operation timeouts per move
before its first write; a lock it no longer holds stops it, and it is
sent again.

| Event | What the task does |
|---|---|
| lock held by another worker | sends the object again after 15 seconds, at most 40 times (10 minutes), counted apart from LDAP retries; then logs `lock busy, task given up` at WARNING |
| server unavailable or timed out | Celery retry after 30, 60, 120, 240 and 480 seconds; after the fifth, `ldap unavailable, task given up` at ERROR |
| bind refused (wrong DN or password, or result code 53, which 389ds answers for an account locked with `nsAccountLock`), StartTLS certificate fault | `ldap bind refused, task not retried` at ERROR |
| setting refused | `setting refused, task not retried` at ERROR, with the `check_id` of the system check |
| server schema older than the package, schema violation, a rename target held by an entry outside the plugin's moves, any other directory error | `ldap operation failed, task not retried` at ERROR; an error on one object of a multi-object task names that object in `object_ids`, and the others are written |

Log messages are constant; the values are fields of the record
(`ldap_task`, `object_ids`, `error_type`, `result_code`,
`error_message`, `attempt`, `countdown`, `lock_attempt`, `check_id`,
`missing_permissions`, `lock_name`). Besides the messages of the table,
a task logs `ldap unavailable, task retried` at WARNING before each
retry, `member values changed meanwhile, entry read again` at DEBUG,
and `lock release failed, lock expires with its timeout` at ERROR when
the cache cannot release a lock. The default Alliance Auth log format
prints the message only; a formatter that renders extra fields shows
the rest.

Fail-safe: every task checks that all permissions of
`ACCESS_PERMISSIONS` exist in the database, the result cached for 10
seconds. While one is missing, tasks write no `aaCommunityAccess`
(values in LDAP stay, new user entries are created without it) and log
`access permissions missing, aaCommunityAccess not written` at WARNING
once per check. A task that writes a user checks again, uncached, after
reading the user: a permission deleted takes its grants along, so a
read after the deletion would give `FALSE`, and the later check then
finds the permission missing and leaves `aaCommunityAccess` as it is.

### Changes no signal reports

The full pass is the only writer of these; until it runs, the entries
keep the state before the change:

- `QuerySet.update()` and `bulk_create()` send no signal: the
  affiliation update of `EveCharacter` and `EveCorporationInfo` rows
  in bulk, the alliance assignment of `populate_alliance`, a profile
  changed with `update()`.
- Deleting or creating a `Permission` (an app removed or migrated):
  `aaPermission`, `aaCommunityAccess` and the fail-safe mode of
  existing entries.
- Deleting a `Group`: its entry is deleted, but its members lose the
  permissions it granted without a signal, so their `aaPermission` and
  `aaCommunityAccess` stay.
- Deleting an `EveAllianceInfo` sets the alliance of its corporations
  to null without a signal; their `aaAlliance` stays.
- Deleting the `EveCharacter` of a main the user does not own through
  a `CharacterOwnership`: the profile's main is cleared without a
  signal.
- A change of the plugin settings (denylist, allowlist, access list,
  the characters branch, email export) reaches an entry with the next
  change of its object, at the latest with the next full pass.

## Full pass

The full pass writes the whole subtree from the database, deletes what
nothing projects into any more and, when it completes, writes the
generation marker `cn=sync-status`. The task
`aa_ldap_directory.tasks.resync_directory` runs it; the plugin adds no
schedule of its own.

### Schedule

Add the entry to the `local.py` of the Auth project:

```python
from celery.schedules import crontab

CELERYBEAT_SCHEDULE["aa_ldap_directory_resync"] = {
    "task": "aa_ldap_directory.tasks.resync_directory",
    "schedule": crontab(minute="*/30"),
}
```

The task goes to `AA_LDAP_DIRECTORY_TASK_QUEUE`, like the other tasks
of the plugin. The period is the longest delay of the changes no signal
reports, and a reader that treats the marker as stale after a multiple
of the period (Heddle: twice the period) needs the same value. A beat
run that starts while another pass runs does nothing (see "Lock").

`python manage.py ldapdirectory_sync` runs one pass in the calling
process, with the same lock, settings and log records. It prints
`full pass finished:` with the fields below and exits 0; a pass that
is skipped or fails exits 1 with `another full pass holds the lock` or
`full pass failed, marker not written:` and the fields.

### What a pass does

1. It takes the lock `resync` (see "Lock") and its generation: the time
   it starts, UTC, to the second, from the clock of the host it runs on.
2. It checks that every permission of `ACCESS_PERMISSIONS` exists, binds
   and checks the server schema version.
3. It loads the database (12 queries, 13 with `ou=characters`, one more
   with a non-empty `GROUP_DENYLIST`) and reads the subtree with paged
   searches. Each object whose entry it writes in step 6 costs more
   queries: a user 10, one of them the access check again (measured on
   SQLite, `tests/test_task_volume.py`). A pass over an unchanged
   database makes 16 queries in all, 17 with a non-empty denylist.
4. It compares every entry with the database: DNs by their parsed form,
   the managed attributes as the server compares them, `objectClass` as
   a subset. An entry that matches gets no write: a pass over an
   unchanged database writes the marker and nothing else.
5. It adds the missing containers (`ou=allianceauth`, the branches) and
   deletes the entries no object owns, the deepest first: any other DN
   in the subtree, an entry below another entry. An entry in a branch
   that names an object by its id at another DN than the object's own
   (written by hand or by another writer), or a second entry with the
   `aaGroupId` or `aaStateId` of one already found, waits for step 7,
   unless it holds a DN or a parent DN an entry of the pass needs. Only
   an entry of the kind's class (`aaUser`, `aaCharacter`,
   `aaOrganization`, `aaGroup`, `aaState`) names an object: a user entry
   copied into `ou=states` carries `aaStateId` but is foreign there, in
   the pass as in a task.
6. For each object whose entry differs, is missing or is no longer
   projected, it takes the task lock of the object (`user:<pk>` and so
   on; groups and states also `groups` or `states`). A lock held by a
   task leaves the object to that task (`objects_skipped`). Under the
   lock it reads the object from the database again and writes or
   deletes the entry as the task does. An entry is deleted only when
   this read confirms that nothing projects into it: an object created
   after the load keeps its entry, and a change a task wrote after the
   load is not overwritten by the older load. Alliances, corporations,
   users and characters go first, then groups and states, those to be
   deleted before the others, so a name they free is free for the
   renames; then the characters, users and organizations no longer
   projected. The pass is the only writer that replaces `member` of a
   group as a whole. Of several entries with the id of one group or
   state, the write of that object, in the pass as in its task, keeps
   the one at the DN the object needs, else the first by DN, whatever
   order the server returns them in; the pass keeps the same one. Before
   it renames anything the write deletes the others, and any such copy
   of another group or state that holds the DN it or a moved namesake
   needs.
7. It deletes the entries that wait from step 5, now that the object's
   own entry is written: a failure before this step leaves the object
   with two entries, never with none. The entry of an object the pass
   skipped or failed to write stays until the next pass. While
   `ou=characters` is not exported, it then deletes the branch, unless
   the pass skipped or failed one of its characters: then the branch
   stays until the next pass, and the pass is not failed for it.
   These deletions, and those of step 5, take no lock and read no object
   again, unlike the deletions of step 6: the plugin itself writes no
   such entry, so they only remove what was written by hand or by
   another writer, even after the load.
8. It checks the access permissions again and writes the marker (see
   "Marker").

A write that fails for one object or entry logs
`ldap operation failed, object left to the next pass` (fields
`object_kind`, `object_ids`) or
`ldap operation failed, entry left to the next pass` (field
`entry_dn`) at WARNING, with `error_type`, `result_code` and
`error_message`; the pass goes on and ends failed. An unavailable
server, a refused bind or StartTLS, an old schema or a refused setting
stops the pass at once. Another exception (the database, the cache)
stops it too: `unexpected error, full pass stopped` at ERROR with the
traceback and `error_type`, then the failed record below. A cache
fault while the pass takes the lock `resync` comes before that: it
fails the Celery task without either record and leaves the last
recorded pass in place.

An entry is written with an add or a modify, never deleted and added
again, and is renamed only when a group or state name changes or a
name collision begins or ends, as a task renames it.

### Marker

At the end of a completed pass one modify deletes the
`aaSyncGeneration` value it read and adds the pass's generation, with
`aaSyncFinishedAt`, `aaSyncEntryCount` (the `ou=users` entries the
pass projected) and `aaSyncSchemaVersion` (the schema version of the
package). A writer that changed the generation in between makes the
server refuse the whole modify; the pass reads the marker again and
compares anew, as often as the marker holds an older generation, so of
several writers at once the newest one wins. A missing marker is added.
A marker the bind may add but not read (access rules with write and no
read) makes that add fail as existing with nothing to read: the pass
stops with `directory_error`, `the marker exists, but this bind cannot
read it`, instead of adding again. `aaSyncAttemptStartedAt` is not
written.

The pass writes no marker when the held generation is not older than
its own: a pass started later has completed already, or the clock of
the host went back (`marker_not_newer`). A generation from the future
thus blocks every pass until the clock passes it; delete the attribute
`aaSyncGeneration` of `cn=sync-status` and the next pass adds its own.
A generation in a form the plugin does not write (a fraction, an
offset) is `marker_unreadable` and is left alone in the same way.

Fail-safe: while a permission of `ACCESS_PERMISSIONS` is missing from
the database, at the start or at the end of the pass, the pass writes
everything except `aaCommunityAccess` (the values in LDAP stay, a new
user entry is created without it), logs
`access permissions missing, aaCommunityAccess not written` at WARNING
and ends with `access_permissions_missing`: the marker keeps its old
generation and ages, so a reader sees the directory stale instead of
every user without access. A permission deleted during the pass, after
the check at its start, gives no user `FALSE` either: each user the
pass writes is checked again after it is read, as a task does, and the
check at the end fails the pass.

### Lock

The pass holds the lock `resync` in `CACHES['default']`, as the tasks
hold theirs: the connect timeout plus 100 operation timeouts, 3005
seconds with the defaults. It extends the lock after each step (the
load, the subtree read, each object written), only while the lock still
holds its own token. A pass that finds the lock taken logs
`full pass skipped, another pass holds the lock` at INFO and opens no
connection. A pass that finds its lock lost, because one step outlasted
it and another pass took over, stops with `lock_lost` and writes no
marker; the marker swap keeps such a pass from rolling the marker back
even if it got that far. A Celery redelivery of a running pass (a
worker lost, a broker visibility timeout shorter than the pass) is a
skipped pass.

Measured on the test servers (OpenLDAP 2.6.10 and 389 Directory Server
3.1.2 on the same host as the pass, SQLite, bound as the rootdn): 1000
users with a main each in 20 groups and 20 corporations, 1050 entries;
`uv run python tests/bench/measure_pass_time.py openldap 1000` measures
it again.

| Pass | OpenLDAP | 389 Directory Server |
|---|---|---|
| first, every entry added | 12.0 s | 38.1 s |
| unchanged database, the marker only | 0.7 s | 0.8 s |
| every user renamed, 1000 modifies | 10.7 s | 14.1 s |

### Log record of a pass

| Outcome | Message | Level |
|---|---|---|
| completed | `full pass finished` | INFO |
| another pass holds the lock | `full pass skipped, another pass holds the lock` | INFO |
| failed | `full pass failed, marker not written` | ERROR |

Fields of the completed and the failed record: `generation`,
`user_entry_count`, `ldap_adds`, `ldap_modifies`, `ldap_renames` and
`ldap_deletes` (write operations on entries, the marker not counted),
`objects_skipped`, `error_count`, `duration_seconds`; the failed record
adds `failure` and its own fields. Every other record of a pass that
took the lock (the WARNING records above) carries `generation` too, so
they join the record that ends the pass; the skipped record has none.

| `failure` | Cause | Fields |
|---|---|---|
| `access_permissions_missing` | a permission of `ACCESS_PERMISSIONS` is missing | `missing_permissions` |
| `entry_errors` | a write failed for at least one object or entry; the WARNING records before it name them | |
| `marker_not_newer` | the marker holds a generation not older than the pass's | |
| `marker_unreadable` | the marker holds a generation the plugin did not write | `error_message` |
| `lock_lost` | the lock `resync` is no longer the pass's | |
| `directory_error` | server unavailable, bind or StartTLS refused, server schema older than the package, another directory error | `error_type`, `result_code`, `error_message` |
| `setting_refused` | a setting the system checks refuse | `check_id`, `error_message` |
| `unexpected_error` | another exception, such as a database or cache fault | `error_type`, `error_message` |

### Delays

How long a change of Alliance Auth takes to reach LDAP:

- A change a signal reports: one task, queued when its transaction
  commits, so seconds while a worker is free and LDAP answers.
- A change of a character's corporation, alliance or name in EVE:
  Alliance Auth finds it with `run_model_update`, every 6 hours in its
  default `CELERYBEAT_SCHEDULE`, plus the random delay of up to 10
  minutes of its update tasks; their saves then queue the plugin's
  tasks. Lag: up to that period plus the task.
- A change no signal reports (see "Changes no signal reports"): up to
  the period of the full pass. Such a change made by `run_model_update`
  (the alliance of corporations set by `populate_alliance`) waits for
  both: up to the two periods added.

Tasks queued by `run_model_update`, pinned on the receivers
(`tests/test_task_volume.py`, Alliance Auth 5.0.0 and 5.5.0): one
`sync_corporations` per corporation it updates, one `sync_alliances` per
alliance, and per character whose affiliation or name changed two
`sync_users` for the user whose main it is plus two `sync_characters`
while `ou=characters` is exported, one per save of `update_character`
and `update_character_other`. 30 users with a main and an alt each, all
renamed, 3 corporations, one alliance: 184 tasks with the branch, 64
without. A permission granted to or taken from a state or group queues
one `fan_out_users`, then one `sync_users` per 100 of its users when the
permission is exported, none otherwise.

## Checking a running export

`python manage.py ldapdirectory_check` checks what the system checks
cannot: it dials the cache, the Celery broker, the LDAP server and the
database, and writes to neither LDAP nor the database (the cache gets
one probe key, deleted again). Run it where Auth runs, with the same
settings. It prints one line per check, `<level> <check>: <message>`
followed by `key=value` details, the levels `ok`, `warning` and
`error`; it exits 0 without an `error`, and 1 with
`CommandError: checks failed: <checks>` on stderr otherwise.

| Check | `error` when | Details |
|---|---|---|
| `cache` | `CACHES['default']` fails `aa_ldap_directory.E013`, does not answer, or loses the probe value | `check_id`, `error_type` |
| `schedule` | no `CELERYBEAT_SCHEDULE` entry runs `aa_ldap_directory.tasks.resync_directory`; `warning` when its period cannot be read (a `solar` schedule) | `entry`, `period_seconds` |
| `worker` | the broker does not answer within 5 seconds, or no worker with the plugin consumes `AA_LDAP_DIRECTORY_TASK_QUEUE` (Celery's default queue when `None`) | `queue`, `workers`, `clock_offset_seconds`, `other_workers` |
| `bind` | the bind DN does not bind, the TLS handshake fails; `warning` for `ldap://` without StartTLS | `uri`, `tls`, `error_type`, `result_code` |
| `schema` | the server schema is older than the package, or not readable | `server_version`, `package_version` |
| `access_permissions` | a permission of `ACCESS_PERMISSIONS` is missing from the database (the fail-safe mode) | `missing_permissions` |
| `unexported_names` | never; `warning` when a user, group or state is not exported for its name: an empty username (see "User entries"), a C0 control character at either end of a group or state name or a NUL in it (see "Group and state names") | `groups`, `states`, `users` (primary keys) |
| `marker` | no marker; a generation or finish time in a form the plugin does not write; a generation more than 2 seconds later than the clock of a worker; the marker older than twice the period | the marker attributes, `age_seconds`, `stale_after_seconds`, `clocks_behind` |
| `last_pass` | the last pass recorded in the cache failed; `warning` when none is recorded | the fields of "Log record of a pass", `status`, `recorded_at` |

`schema` and `marker` need the bind: after a failed `bind` they report
`not checked: no bind`. A setting the system checks refuse is the
`error` of the check that reads it, with its `check_id`.

The worker check broadcasts the remote control command
`ldapdirectory_clock`, which a worker has once it imported the plugin's
tasks; a worker that runs without the plugin answers with an error and
is listed in `other_workers`. Each worker returns its clock: the full
pass takes its generation from the clock of the worker that runs it, so
a generation later than the clock of any worker on the queue blocks
every pass that worker starts (`marker_not_newer`) until the clock
passes it; the check reports the workers in `clocks_behind` once the
generation is more than 2 seconds ahead of their clock (the generation
is cut to the second, and hosts on NTP drift by about a second more), so
a check right after a pass on a worker a second behind stays `ok`.
Without a
worker answering, the generation is compared with the clock of the host
the check runs on. `clock_offset_seconds` is each worker's clock minus
the time the broadcast was sent; keep the hosts on NTP.

The stale age follows the reader: Heddle treats the directory as stale
when `aaSyncFinishedAt` is older than twice the period of the full pass.
The period is the longest gap between two runs of the beat entry, for a
`crontab` evaluated in UTC over 32 days; the check does not apply the
`apply_offset` of Alliance Auth's scheduler. The entry is read from
`CELERYBEAT_SCHEDULE` in the settings, not from the `PeriodicTask` row
the database scheduler runs: a row disabled in the Django admin still
passes `schedule`, and only the age of the marker shows it, as an
`error` once the marker is stale.

The last pass is recorded by the pass itself, in the cache key
`aa_ldap_directory:last_pass` without expiry: a completed or failed pass
replaces it, a skipped one leaves it. A flushed cache shows the
`warning` until the next pass.

## Server setup

What the LDAP server needs besides the schema: a suffix, three
accounts, access rules on the subtree and size limits for the accounts
that read all of it. OpenLDAP and 389 Directory Server are supported
alike. Every LDIF here is applied by the server administrator; replace
`dc=example,dc=org` with the suffix of `AA_LDAP_DIRECTORY_BASE_DN`.

`tests/test_server_setup_ldap.py` applies every LDIF block of this
section but the one of "Usernames on 389 Directory Server", as
written, to the test servers (OpenLDAP 2.6.10, 389 Directory Server
3.1.2) and checks: the writer adds, renames and deletes inside
`ou=allianceauth` and is refused outside it; a full pass bound as the
writer completes, and a second one writes only the marker; an anonymous
search of the subtree finds nothing; the reader reads `ou=users` but
nothing in `ou=characters`, which only the characters reader sees; no
account reads another one's `userPassword`; the limits below lift a
server limit.

### Suffix

The plugin writes below an existing suffix and creates nothing above
`ou=allianceauth`.

- OpenLDAP: the suffix is the `olcSuffix` of a database, with its own
  entry (`dcObject` and `organization`, or `domain`).
- 389 Directory Server: a backend with its suffix entry, created by
  Directory Manager, for example with `dsconf <instance> backend create
  --suffix dc=example,dc=org --be-name userRoot --create-suffix` (the
  test server creates it over LDAP instead; the command follows the 389
  Directory Server manual and was not run here).

Then add `ou=allianceauth` as the administrator: the writer below gets
no right on the suffix entry, so it cannot create the subtree itself.

```ldif
dn: ou=allianceauth,dc=example,dc=org
objectClass: top
objectClass: organizationalUnit
ou: allianceauth
```

### Accounts

| Account | Binds as | Needs |
|---|---|---|
| `cn=aa-writer,dc=example,dc=org` | the plugin, `AA_LDAP_DIRECTORY_BIND_DN` | write in `ou=allianceauth`, no size limit |
| `cn=heddle-reader,dc=example,dc=org` | Heddle and other readers | read in `ou=allianceauth` except `ou=characters`, no size limit |
| `cn=characters-reader,dc=example,dc=org` | a reader allowed to see which characters belong to whom | read in `ou=allianceauth` including `ou=characters` |

The accounts sit outside `ou=allianceauth`: the full pass deletes every
entry in it that the plugin does not write. On OpenLDAP an account entry
is `applicationProcess` plus `simpleSecurityObject` with `userPassword`;
on 389 Directory Server a `person` with `sn` and `userPassword`.

### Access rules on OpenLDAP

slapd applies the first `to` clause that matches an entry, and within
it the first matching `by` clause. Insert these rules before any rule of
the database that grants read on the whole suffix. The `by * none` of
each rule closes the subtree to anonymous and to every other account,
whatever follows.

```ldif
dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcAccess
olcAccess: {0}to dn.subtree="dc=example,dc=org" attrs=userPassword
  by anonymous auth
  by * none
olcAccess: {1}to dn.subtree="ou=characters,ou=allianceauth,dc=example,dc=org"
  by dn.exact="cn=aa-writer,dc=example,dc=org" write
  by dn.exact="cn=characters-reader,dc=example,dc=org" read
  by * none
olcAccess: {2}to dn.subtree="ou=allianceauth,dc=example,dc=org"
  by dn.exact="cn=aa-writer,dc=example,dc=org" write
  by dn.exact="cn=heddle-reader,dc=example,dc=org" read
  by dn.exact="cn=characters-reader,dc=example,dc=org" read
  by * none
```

```bash
ldapmodify -Y EXTERNAL -H ldapi:/// -f aa-ldap-directory-acl.ldif
```

The database DN `olcDatabase={1}mdb,cn=config` is the one of the
suffix; find it with
`ldapsearch -Q -LLL -Y EXTERNAL -H ldapi:/// -b cn=config '(olcSuffix=dc=example,dc=org)' dn`.
The `{N}` of each rule is its position: with rules already in place,
number the new ones so they come first. The plugin reads the schema
version from the subschema entry `cn=Subschema`, which has to stay
readable for the writer; a slapd without frontend rules, as the test
server, lets every account read it.

A database without `olcAccess` lets everyone read everything. Once it
has rules, slapd ends them with an implicit `to * by * none`: the rest
of the suffix is closed to every account but the rootdn. In such a
database, end with a rule that keeps the rest as it was, or with one the
site needs:

```ldif
dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcAccess
olcAccess: {3}to *
  by * read
```

### Access rules on 389 Directory Server

The access control instructions go on `ou=allianceauth`; the server
denies what no instruction allows. The third one takes `ou=characters`
away from the reader.

```ldif
dn: ou=allianceauth,dc=example,dc=org
changetype: modify
add: aci
aci: (targetattr="*")(version 3.0; acl "aa-ldap-directory writer"; allow (all) userdn="ldap:///cn=aa-writer,dc=example,dc=org";)
aci: (targetattr="*")(version 3.0; acl "aa-ldap-directory readers"; allow (read,search,compare) userdn="ldap:///cn=heddle-reader,dc=example,dc=org || ldap:///cn=characters-reader,dc=example,dc=org";)
aci: (target="ldap:///ou=characters,ou=allianceauth,dc=example,dc=org")(targetattr="*")(version 3.0; acl "aa-ldap-directory characters"; deny (read,search,compare) userdn="ldap:///cn=heddle-reader,dc=example,dc=org";)
```

```bash
ldapmodify -x -H ldaps://ldap.example.org -D "cn=Directory Manager" -W -f aa-ldap-directory-aci.ldif
```

The writer's `allow (all)` on `ou=allianceauth` includes deleting that
entry once it is empty; the plugin never deletes it. An instruction
elsewhere in the suffix that grants anonymous read, such as the ones
`dsconf ... --create-entries` adds, opens the subtree too: check
`ldapsearch -x -H ldaps://ldap.example.org -b ou=allianceauth,dc=example,dc=org`
finds nothing. Per the 389 Directory Server manual,
`nsslapd-allow-anonymous-access: rootdse` in `cn=config` limits
anonymous access to the root DSE; not tried here.

### Usernames on 389 Directory Server

389 Directory Server 3.1.2 ships its 7-bit check plugin enabled for
`uid` and `mail`. It refuses an entry whose `uid` holds a character
outside ASCII with `Constraint violation (19)`, `The value is not 7-bit
clean`. The plugin writes the username to `uid`, and Django accepts
non-ASCII letters in a username: such a user gets no entry, and every
full pass fails with `entry_errors` and leaves the marker unchanged.
OpenLDAP has no such check. Take `uid` out of the plugin's arguments,
as Directory Manager. `mail` stays among them: its IA5 syntax refuses a
non-ASCII address anyway (21), and the plugin writes only an ASCII
address there:

```ldif
dn: cn=7-bit check,cn=plugins,cn=config
changetype: modify
replace: nsslapd-pluginarg0
nsslapd-pluginarg0: mail
-
replace: nsslapd-pluginarg1
nsslapd-pluginarg1: ,
-
replace: nsslapd-pluginarg2
nsslapd-pluginarg2:
-
replace: nsslapd-pluginarg3
-
```

The plugin reads its arguments when the server starts: the modify alone
changes nothing until the instance restarts, for example with `dsctl
<instance> restart` (the command follows the 389 Directory Server
manual and was not run here). Measured 2026-11-02 on 389 Directory
Server 3.1.2. The test server gets the same block from
`tests/_ldapbench.py` without a restart: it switches
`nsslapd-dynamic-plugins` on in `cn=config` and disables and enables
the plugin around the modify, then sets the switch back.

The block replaces the whole argument list, written for the one 389
Directory Server 3.1.2 ships: `uid`, `mail`, `,` and an empty subtree.
An instance with other arguments loses them; read them first:

```bash
ldapsearch -LLL -x -H ldaps://ldap.example.org -D "cn=Directory Manager" -W \
  -b "cn=7-bit check,cn=plugins,cn=config" -s base \
  nsslapd-pluginarg0 nsslapd-pluginarg1 nsslapd-pluginarg2 nsslapd-pluginarg3
```

Run again, the block changes nothing: a `replace` without a value
removes `nsslapd-pluginarg3` and passes once it is gone.

### Size limits

The full pass reads the whole subtree with paged searches of 500
entries, and Heddle reads it the same way. A limit of the server that
cuts the read fails it as a whole, `Size limit exceeded (4)` or
`Administrative limit exceeded (11)`: the pass stops with
`directory_error` and writes nothing, and no entry is lost. Give the
writer and the readers no limit:

| Server | Default for a plain account | Measured on the test server, `tests/bench/measure_size_limits.py` |
|---|---|---|
| OpenLDAP | `olcSizeLimit` 500, paged searches counted in total (`size.prtotal`) | 604 entries: refused, paged or not |
| 389 Directory Server | `nsslapd-sizelimit` 2000 per search, paged searches counted per page; `nsslapd-lookthroughlimit` 5000 | 2104 entries paged: read; 5204 entries paged: refused with 11 |

OpenLDAP, per account:

```ldif
dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcLimits
olcLimits: dn.exact="cn=aa-writer,dc=example,dc=org" size.soft=unlimited size.hard=unlimited size.prtotal=unlimited
olcLimits: dn.exact="cn=heddle-reader,dc=example,dc=org" size.soft=unlimited size.hard=unlimited size.prtotal=unlimited
olcLimits: dn.exact="cn=characters-reader,dc=example,dc=org" size.soft=unlimited size.hard=unlimited size.prtotal=unlimited
```

389 Directory Server, on each account entry (operational attributes,
written by Directory Manager):

```ldif
dn: cn=aa-writer,dc=example,dc=org
changetype: modify
replace: nsSizeLimit
nsSizeLimit: -1
-
replace: nsPagedSizeLimit
nsPagedSizeLimit: -1
-
replace: nsLookThroughLimit
nsLookThroughLimit: -1
-
replace: nsPagedLookThroughLimit
nsPagedLookThroughLimit: -1
```

and the same for the two readers. The test lowers the server limit to
5 entries and reads the subtree as each account, refused before these
limits and complete after them.

### memberOf

The plugin writes `member` on groups and never reads or writes
`memberOf`. A client that wants `memberOf` on user entries gets it from
the server: on 389 Directory Server from the MemberOf plugin, scoped to
`ou=allianceauth`; on OpenLDAP from the memberof overlay, deprecated
since 2.5, or the `memberOf` of the dynlist overlay. Not tried with the
plugin; the vendor manuals describe the setup. An object class such a
server adds to user entries (389 Directory Server: `nsMemberOf`) does
not make the plugin rewrite them: it compares `objectClass` as a subset.
A group without members names itself in `member`, so it shows up in its
own `memberOf`; count only the `member` values below `ou=users`.

### Differences between the servers

Measured on OpenLDAP 2.6.10 and 389 Directory Server 3.1.2 with the
contract tests of the adapter; the plugin writes the same tree on both.

| Behaviour | OpenLDAP | 389 Directory Server |
|---|---|---|
| `groupOfNames` without `member` | refused (65) | accepted; the plugin names the group itself anyway |
| `objectClass` read back | as written | with the superclasses added (`person`, `organizationalPerson`) |
| inner runs of spaces in a DN | kept | collapsed to one in the DN, kept in `cn` |
| size limit of a paged search | counted over the whole search | counted per page |
| paged search below a missing base | `No such object (32)` | success without entries; the plugin checks the base |
| attribute names in a reply | as in the schema | as in the request |
| second schema install | `Other (80)`, `Duplicate attributeType` | accepted, nothing changes |
| RDN value of one space written as `\` and a space | accepted, returned as `\20` | refused (21); `\20` accepted, which the plugin writes |

## LDAP schema

The plugin writes its own attribute types and object classes (`aaUser`,
`aaOrganization`, `aaGroup`, `aaSyncStatus`, `aaCharacter`, `aaState`
and the `aa*` attributes), so the server has to hold the plugin schema
before the first write. The package defines schema version 1. It builds
on the standard `core`, `cosine` and `inetorgperson` schemas, which
both servers ship.

`ldapdirectory_schema` prints the LDIF to stdout and opens no
connection. It runs no system checks, so it works before the plugin
settings are filled in.

```bash
python manage.py ldapdirectory_schema --format openldap --mode install > aa-ldap-directory.ldif
python manage.py ldapdirectory_schema --format 389ds --mode install > aa-ldap-directory.ldif
```

| Option | Values | Meaning |
|---|---|---|
| `--format` | `openldap`, `389ds` | `openldap` adds the entry `cn=aa-ldap-directory,cn=schema,cn=config` at install and modifies the stored entry at upgrade; `389ds` modifies `cn=schema` |
| `--mode` | `install`, `upgrade` | `install` loads every version into a server without the schema; `upgrade` adds only what the server lacks |
| `--from` | integer | the schema version the server holds; required with `upgrade`, refused with `install` |
| `--schema-dn` | DN | the DN slapd stores the entry under, `cn={N}aa-ldap-directory,cn=schema,cn=config`; required with `--format openldap --mode upgrade`, refused otherwise |

A missing or unknown option exits with status 2 and the usage. A
request no LDIF exists for exits with status 1 and one
`CommandError: ...` line on stderr. Neither prints LDIF. Version 1 is
the only version, so `--mode upgrade --from 1` reports
`nothing to upgrade from 1: it is the current schema version`.

Install once. A second install fails on OpenLDAP with
`Other (80)` and `Duplicate attributeType`; 389 Directory Server
accepts it with status 0 and changes nothing.

### Loading into OpenLDAP

The server has to run with `cn=config` (`slapd.d`). As root on the
LDAP host, where the `ldapi:///` socket maps root to the `cn=config`
administrator:

```bash
ldapadd -Y EXTERNAL -H ldapi:/// -f aa-ldap-directory.ldif
```

slapd renames the entry to `cn={N}aa-ldap-directory,cn=schema,cn=config`
and applies it without a restart.

### Loading into 389 Directory Server

As Directory Manager, over TLS:

```bash
ldapmodify -x -H ldaps://ldap.example.org -D "cn=Directory Manager" -W -f aa-ldap-directory.ldif
```

The server writes the definitions into `99user.ldif` of the instance
and applies them without a restart.

### Version of a loaded schema

The server holds version N when its subschema carries the object class
`aaSchemaVersionN`; the highest N is the version. Read the name, not
`X-ORIGIN`: 389 Directory Server rewrites `X-ORIGIN` to
`( 'aa-ldap-directory schema N' 'user defined' )`. The subschema entry
differs per server.

OpenLDAP, `cn=Subschema`:

```bash
ldapsearch -Q -LLL -Y EXTERNAL -H ldapi:/// -o ldif-wrap=no \
  -b cn=Subschema -s base objectClasses | grep -o "aaSchemaVersion[0-9]*"
```

389 Directory Server, `cn=schema`:

```bash
ldapsearch -LLL -x -H ldaps://ldap.example.org -D "cn=Directory Manager" -W -o ldif-wrap=no \
  -b cn=schema -s base objectClasses | grep -o "aaSchemaVersion[0-9]*"
```

No output means the schema is not loaded, provided `ldapsearch` itself
succeeded: a wrong base fails with `No such object (32)` on stderr and
prints nothing to the pipe either. With no version, use
`--mode install`; otherwise pass the highest N as `--from`.

### Upgrading

On OpenLDAP the upgrade modifies the stored entry, whose DN carries the
index `{N}` slapd assigned; a modify of the name without the index
fails with `No such object (32)`. Find the DN, pass it as
`--schema-dn`, then apply the LDIF:

```bash
ldapsearch -Q -LLL -Y EXTERNAL -H ldapi:/// -b cn=schema,cn=config -s one '(cn=*aa-ldap-directory)' dn
python manage.py ldapdirectory_schema --format openldap --mode upgrade --from 1 \
  --schema-dn 'cn={3}aa-ldap-directory,cn=schema,cn=config' > aa-ldap-directory-upgrade.ldif
ldapmodify -Y EXTERNAL -H ldapi:/// -f aa-ldap-directory-upgrade.ldif
```

On 389 Directory Server:

```bash
python manage.py ldapdirectory_schema --format 389ds --mode upgrade --from 1 > aa-ldap-directory-upgrade.ldif
ldapmodify -x -H ldaps://ldap.example.org -D "cn=Directory Manager" -W -f aa-ldap-directory-upgrade.ldif
```

With version 1 the only version, both print no LDIF and report
`nothing to upgrade from 1`.

Released definitions are never changed or removed: a new version only
adds attribute types and object classes, so a server with an older
version keeps the meaning of every definition it holds.

## Removing the plugin

In this order: a worker that still runs the plugin writes the entries
back.

1. Remove the `CELERYBEAT_SCHEDULE` entry of the plugin from
   `local.py`. Alliance Auth's database scheduler keeps the row it made
   from it; delete that too:

   ```bash
   python manage.py shell -c "from django_celery_beat.models import PeriodicTask; print(PeriodicTask.objects.filter(task='aa_ldap_directory.tasks.resync_directory').delete())"
   ```

   It prints the number of rows deleted, `(0, {})` when there was none.
2. Remove `aa_ldap_directory` from `INSTALLED_APPS` and the
   `AA_LDAP_DIRECTORY_*` settings, stop a worker started for its queue,
   restart Auth, its workers and beat. Tasks still waiting in the broker
   then fail as unregistered and are dropped by Celery.
3. `pip uninstall aa-ldap-directory`.
4. Delete the subtree as the server administrator, then the accounts,
   the access rules and the limits of "Server setup":

   ```bash
   ldapdelete -r -x -H ldaps://ldap.example.org -D "cn=Directory Manager" -W "ou=allianceauth,dc=example,dc=org"
   ```

   On OpenLDAP bind as its administrator instead. A reader that still
   polls sees the directory stale, as its marker is gone.
5. The cache keys of the plugin start with `aa_ldap_directory:`; the
   locks expire by themselves, the record of the last pass does not:

   ```bash
   python manage.py shell -c "from django.core.cache import cache; cache.delete('aa_ldap_directory:last_pass')"
   ```

The schema stays loaded: without entries that use it, it changes
nothing, and removing a schema from a running server is not covered
here.

## Development

```bash
make dev     # uv sync + pre-commit hooks
make check   # the aggregate gate: lint, types, tests, migrations, Markdown, Makefile
make help    # every target
```

Tests run under the Django test runner inside a full Alliance Auth
settings module (`tests/settings.py`).

### Test LDAP servers

The contract tests of the LDAP adapter run in every `make check` on an
in-memory directory; `tests_ldap` runs the same cases on OpenLDAP 2.6.10
or 389 Directory Server 3.1.2 in a container. They need Docker with
Compose v2, network access for the images, and the loopback ports 3890
and 6360 (OpenLDAP) and 3389 (389ds) free:

```bash
docker compose -f compose.test.yml --profile openldap up -d --wait
uv run nox -s "tests_ldap(server='openldap')"
docker compose -f compose.test.yml --profile openldap down -v
```

For 389ds, put `389ds` in all three commands. The OpenLDAP container
installs slapd from the Debian archive on every start, about 25 seconds
and 3 to 4 minutes on a slow mirror; `up --wait` waits up to its start
period of 10 minutes, then fails with the container unhealthy. The
389ds container creates its instance on first start. The tests then
create the 389ds database of the suffix, load the plugin schema when
the server holds none, add `dc=example,dc=org` and work below their own
`ou=contract-<hex>` entries, which they remove. `down -v` discards the
server with everything in it.

The tests bind over plain LDAP with the password `test-only-ldap`: as
the rootdn, and for the size-limit cases as `cn=bench-limited` under the
suffix, an account the tests create with a server size limit of 3. The
OpenLDAP container also listens on `ldaps://127.0.0.1:6360` and offers
StartTLS, with a self-signed certificate for `127.0.0.1` made anew on
every start; the TLS cases (tag `ldap-tls`, excluded on 389ds, whose
container has no such listener) read it from the listener and check
that the adapter binds with it and refuses without it or for another
host name.
`AA_LDAP_TEST_URI` replaces the URI the session uses; point it only at
a throwaway server, since the tests load a schema, add that account and
its limit and, on 389ds, create a database.

The Lua scripts of the task locks run on a real Redis (8.10.2) under a
session of their own; the ordinary run uses a fake client that keeps the
same contract in Python. It needs the loopback port 6380 free:

```bash
docker compose -f compose.test.yml --profile redis up -d --wait
uv run nox -s tests_redis
docker compose -f compose.test.yml --profile redis down -v
```

The tests flush database 0 of the server `AA_REDIS_TEST_URL` names
(default `redis://127.0.0.1:6380/0`); point it only at a throwaway
server.

The Redis session also starts a Celery worker in the test process, with
the test Redis as broker, and checks that it answers the worker check of
`ldapdirectory_check`.

`tests/data/directory.ldif` is the fixture of the Heddle package,
vendored byte for byte; its source commit and git blob id are in
`_nox/_fixture.py`, and every test run checks the copy against the blob
id. The tests load it into the servers with the plugin schema. To see
whether Heddle changed the fixture since, point the session at a
checkout of the Heddle package, fetched first:

```bash
HEDDLE_REPO=../heddle uv run nox -s fixture_drift
```

It compares the copy with the fixture at the source commit and at
`main` of the checkout (another ref as the argument after `--`) and
fails naming each difference.

In CI, `tests_ldap_openldap` runs in every merge request, and
`tests_redis` in the same job against its Redis service.
`tests_ldap_389ds` runs on a manual start or when the merge request
carries the label `ldap-389ds`. Both run on scheduled pipelines and on
tags, where a failure stops the release.

### Demo site

A local Alliance Auth site to try the plugin by hand: the admin of
Alliance Auth on `127.0.0.1:8000`, an OpenLDAP instance of its own on
`127.0.0.1:3892` and database 2 of the test Redis on `127.0.0.1:6380`.
It needs the loopback ports 3892 and 8000 free; port 6380 is the Redis
of the bench, profile `redis`:

```bash
docker compose -f compose.test.yml --profile openldap-demo --profile redis up -d --wait
uv run nox -s demo
docker compose -f compose.test.yml --profile openldap-demo --profile redis down -v
```

The session checks that both servers answer and refuses before it
writes anything otherwise, naming the server and the first command. It
keeps its state in `.demo/`: a SQLite database and `manifest.json`. The
first run seeds the graph of the test generator (`tests/_aa_graph.py`)
for seed 0, the graph of the test runs, with the plugin settings of that
graph and the export of characters and email on; the manifest records
the seed, the commit and those settings. Every run then prepares the
LDAP instance as "Server setup" describes, below `dc=example,dc=org`
(the plugin schema, the suffix, the three accounts, `ou=allianceauth`
and the access rules), runs a full pass and starts `runserver`. The
container loses its data on every start; the next run of the session
writes the tree again from the database.
The first run with `--no-serve` takes under a minute, seeding included
(the seeding drops the tasks of the receivers: the pass writes the
tree anyway); a later run takes seconds.

The session prints the address, the user to log in with and the
`ldapsearch` command that reads the tree as `heddle-reader`, run inside
the container:

```bash
docker compose -f compose.test.yml exec openldap-demo ldapsearch -x -LLL -H ldap://127.0.0.1:3890 -D cn=heddle-reader,dc=example,dc=org -w demo-only-ldap -b ou=allianceauth,dc=example,dc=org
```

The user is the active superuser of the graph; the passwords are
`demo-only-admin` for `/admin/` and `demo-only-ldap` for the LDAP
accounts. They are public: keep the site on loopback and never expose
it.

A change in the admin runs the tasks of the plugin in the request
(Celery eager, no broker), and the tree shows it at once. A task that
fails for good is logged in the console, as on a worker, and the request
does not fail. The admin actions "Update from ESI" and "Update Main
Character from ESI", in every form, reach ESI over the network and
achieve nothing: the EVE ids of the graph are made up.

Every command of the demo prints the warning
`aa_ldap_directory.W001`: the demo instance speaks plain LDAP on
loopback, which is expected. `ldapdirectory_check` and
`ldapdirectory_sync` from a second terminal, run in the repository
root, share the pass status and the locks of the site:

```bash
DJANGO_SETTINGS_MODULE=tests._demo_settings uv run python -m django ldapdirectory_check
```

The check finds the pass of the session and reports only that no worker
answers.

After `--`, `reset` removes `.demo/` and starts over, `--seed N` seeds
another graph (by default the seed of the manifest, else 0; another seed
than the manifest's needs `reset`), and `--no-serve` stops before
`runserver`:

```bash
uv run nox -s demo -- reset --seed 7
```

The demo and `tests_redis` share the Redis server: the tests flush only
database 0 and use database 1 as a broker, the demo keeps database 2.
`down -v` of the profile `redis`, after `tests_redis` too, removes the
pass status and the locks of the demo; the next run of the session
writes the status again.

A defect seen in the demo becomes a test:

1. Note the seed and the commit from `.demo/manifest.json` and the steps
   taken in the admin.
2. Write the steps as a `GraphChange` of an existing `ChangeKind` on
   `build_graph(SeededChoices(seed), variants=[...])` with the fewest
   variants that still show the defect, and the plugin settings of
   `build_plugin_settings` with the same export flags. A step none of
   the kinds covers is an ORM write in the test or a new kind of the
   generator, with tests of its own.
3. Put the test in a mixin class of its own in
   `tests/_task_scenarios.py` or `tests/_resync_scenarios.py`: it runs on
   the in-memory directory in `make check` and on the servers under
   `tests_ldap`. Behaviour of the server alone goes into a test tagged
   `ldap`.
4. The test fails before the fix and passes after it.

## License

GPL-2.0-or-later, see [LICENSE](https://gitlab.com/eveo7/aa-ldap-directory/-/blob/main/LICENSE).

Russian: [README.ru.md](https://gitlab.com/eveo7/aa-ldap-directory/-/blob/main/README.ru.md).
