Metadata-Version: 2.4
Name: splent_feature_session_filesystem
Version: 1.2.0
Requires-Python: >=3.13
Description-Content-Type: text/markdown
Requires-Dist: cachelib>=0.13

# splent_feature_session_filesystem

Server-side sessions for a SPLENT product, kept as files on disk through
Flask-Session and cachelib. The browser holds only a random session id, and
everything the session carries stays on the server.

Pick this feature or `splent_feature_session_redis`, never both. Files need
no extra service and suit a product that runs on one host. Redis suits a
product whose workers run on more than one.

## Settings

Every setting has a default that works in production, so a product needs no
`.env` entry to run. They are declared under `[tool.splent.config]` in this
feature's `pyproject.toml`, which is how `product:env` learns about them.

| Variable | Default | What it does |
|---|---|---|
| `SESSION_FILE_DIR` | `/tmp/flask_sessions` | Where session files are written |
| `SESSION_FILE_THRESHOLD` | `100000` | How many sessions are kept before some are evicted. `0` means no limit |
| `SESSION_FILE_ANONYMOUS_LIFETIME` | `86400` (a day) | Seconds a session nobody signed in to is kept. `0` gives it a member's lifetime |
| `SESSION_FILE_CLEANUP_INTERVAL` | `3600` (an hour) | Least seconds between two sweeps for expired sessions, counted for the whole directory and not per worker. `0` sweeps only when the store is over its threshold |

A value that is not a whole number of zero or more stops the app at start
with an error that names the variable. A blank line in the `.env` means the
default.

Keep `SESSION_FILE_ANONYMOUS_LIFETIME` at least as long as
`WTF_CSRF_TIME_LIMIT`, an hour unless the product changes it. A visitor's
CSRF token lives in its session, so a session that expires before the token
does makes a slow visitor's form fail with "The CSRF session token is
missing". The app logs a warning at start when the two disagree.

A member's session lives for Flask's `PERMANENT_SESSION_LIFETIME`, 31 days
unless the product changes it, counted from the last request.

### Put the directory on a volume

Under `/tmp` the sessions live in the container, so recreating it signs
everybody out, which is what happens on every deploy of a new image. In
production point `SESSION_FILE_DIR` at a path on a named volume.

```yaml
services:
  web:
    environment:
      SESSION_FILE_DIR: /var/lib/splent/sessions
    volumes:
      - sessions:/var/lib/splent/sessions
volumes:
  sessions:
```

Every worker of the product has to see the same directory. Two containers
on different hosts each with a directory of their own would sign people out
whenever a load balancer sent them to the other one. That case is what
`splent_feature_session_redis` is for.

### Sizing the threshold

The store fills up with visitors, not members. Every anonymous visit to a
page with a form leaves a session that lives for
`SESSION_FILE_ANONYMOUS_LIFETIME`, so at any moment the directory holds
about one day's worth of such visits with the default lifetime, plus every
member who signed in within the last month. The threshold is reached when
that sum passes it. A site with a few thousand form views a day never gets
close to the default of 100000. A crawler hammering `/login` at more than
one request a second does.

A session file is a few hundred bytes. Even counted at a 4 KB disk block
each, the default stays under half a gigabyte. Raise it for a product with
more than a hundred thousand form views a day, lower it on a host with very
little disk.

When the store does reach its threshold it scans the directory once,
deletes what has expired, and evicts the sessions closest to expiry down to
a tenth below the threshold. Visitors' sessions are given a day and
members' a month, so visitors go first and members are kept. The next scan
waits until another tenth of the threshold in new sessions has arrived,
10000 with the default.

### What the housekeeping costs

A scan opens every session file to read its expiry, so it takes time in
proportion to the number of files. On a laptop with an SSD it took about
five seconds for 90000 sessions and seven and a half when 20000 of them had
expired and were deleted. A Linux server with the directory on a local
volume is usually quicker. That cost is paid

* once per `SESSION_FILE_CLEANUP_INTERVAL` for the whole directory, by
  whichever worker writes a session first once the interval is up, and
* once per tenth of the threshold in new sessions while the store is full.

Neither is paid by a request. The scan runs in a thread of its own inside
the worker, the request that started it carries on at once, and a second
scan is never started while one is running. Writing a session costs about a
millisecond however full the directory is. Starting a worker scans nothing,
so a deploy does not wait on the directory either.

## Why visitors have sessions at all

Every page with a form, which includes `/login`, puts a CSRF token in the
session, and this feature stores the session. A crawler or a passer-by that
opens the login page leaves a file behind.

With cachelib's own default of 500 files that signed people out. During the
marketplace's production rehearsal 560 anonymous visits to `/login`, made
in 16 seconds, filled the store and cachelib evicted the members' sessions,
which were the oldest. The fix has three parts.

1. The threshold is a setting, `100000` by default instead of `500`.
2. A visitor's session is kept for a day instead of a month. It holds a
   CSRF token that is good for an hour anyway, and with the shorter expiry
   a full store evicts visitors before members.
3. Expired sessions are swept once an hour. cachelib only looks for them
   once it is over the threshold, so with a threshold high enough to be
   safe they would pile up for ever.

Raising the threshold alone was not enough. Once over its threshold
cachelib scans the directory twice on every write and evicts a single
file, so with a hundred thousand sessions each new visitor waited about
twelve seconds. Its sweep also rewrites the count of files once per file it
deletes. This feature does its own housekeeping, in one scan, outside the
request, as the section on costs describes.

Not storing a session for visitors at all was considered and turned down.

* flask_wtf has no stateless CSRF mode. Its token is checked against a
  value kept in the session, so a visitor with no session cannot submit
  any form, the login form included.
* Flask-Session replaces the whole session interface of the app. It has no
  way to keep a visitor's session in a signed cookie and a member's on the
  server, and building that would mean a second cookie and moving the
  session across at sign in, in the framework, for every product.
* Flask-Session already stores nothing for a visit that leaves the session
  empty. Visitors only get a file when a page really needs a token.

Short lived visitor sessions protect members just as well, keep CSRF as it
is, and change nothing outside this feature.

## How it is wired

`config.py` sets `SESSION_TYPE` to `cachelib` and hands Flask-Session a
`SessionFileCache` from `cache.py`. That class is cachelib's
`FileSystemCache` with the changes above. It writes the same file format,
so sessions written by earlier versions of this feature, which used
Flask-Session's deprecated `filesystem` type, survive the upgrade. It has
been tested with cachelib 0.14 and 0.17. A scan that fails for any reason
is logged and retried later, and never fails the request that started it.

The directory also holds two files that are not sessions. One is
cachelib's count of files, the other a stamp whose modification time
records when the last sweep started, which is how workers in different
containers share one interval.

A session counts as a member's when it holds Flask-Login's `_user_id`.

## Tests

```bash
splent feature:test session_filesystem
```

The unit tests build a small app wired like a product, with Flask-Session,
CSRFProtect and this feature's config. They sign a member in, make 600
anonymous visits to a login form, and check the member is still signed in.
A control test runs the same visits against cachelib's defaults and checks
that the member is signed out there, so the first test cannot pass for the
wrong reason. Other unit tests count the scans a full store makes, check
that a sweep writes the count once, and that scans run outside the
request. The functional tests check that the product's own session
interface uses this store and keeps a member through 600 anonymous
sessions, with its own threshold and with one small enough to evict.
