gpajot/photon repository overview

⁠photon

Web server and application to view photos organized in directories.

Features:

• Minimal increase in storage requirements compared to storing only originals (<2%).
• Uses filesystem structure: the app is an enhanced view of the filesystem.
• Grouped by date: in a given stream, photos are grouped by date using an adapted DBSCAN algorithm.
• Reverse geocoding: photos with location metadata will be assigned a city and country.
• Face identification: detect, cluster and identify faces.

⁠Running

Using the docker image:

docker run \
  -v /path/to/originals:/data:rw \
  -v /path/to/store:/app/store:rw \
  --user $(stat -c '%u:%g' ~) \
  gpajot/photon:latest

CLI documentation⁠

⁠Deploying

The easiest way to serve is to use this Caddy configuration:

localhost:8000 {
    encode
    handle /api/* {
        reverse_proxy <HOST>
    }
    handle /originals/* {
        header Cache-Control "max-age=31536000, immutable"
        root <PATH_TO_ORIGINALS>
        file_server
    }
    handle /thumbs/* {
        header Cache-Control "max-age=31536000, immutable"
        root <PATH_TO_THUMBNAILS>
        file_server
    }
    handle {
        root <PATH_TO_FRONT>
        route {
            try_files {path} /index.html
            header                Cache-Control "max-age=31536000, immutable"
            header /index.html    Cache-Control "no-cache"
            header /favicon.png   Cache-Control "no-cache"
            header /manifest.json Cache-Control "no-cache"
        }
        file_server
    }
}

⁠Originals

⁠Reverse geocoding

Reverse geocoding is done using a kDTree in 2 dimension to find the closest city. Cities are downloaded from Geonames⁠ on first use and cached in the database.

⁠Face identification

Face detection and recognition is made using ONNX models.

⁠Sources

Both pretrained models correspond to buffalo_l of deepinsight/insightface⁠:

• detection is done using SCRFD, trained on Wider face⁠
• recognition is done using ArcFace, trained on WebFace12M, a subset of WebFace260M⁠

Pretrained models are downloaded from Immich at Hugging Face⁠ and cached in /app/store/models-cache.

⁠Clustering

Clustering is done with an adapted, incremental DBSCAN algorithm. 4 passes are made, the first 3 are very close to the original DBSCAN algorithm.

⁠Pass 0

Simple caching of closest neighbors with distances to avoid recomputing in each phase.

⁠Pass 1

Identify core faces. A core face is defined by having at least K neighbors within EPS distance. Here we suppose that EPS is conservative (low) to only create meaningful clusters.

In addition to that, 2 conditions are added:

• face detection score must be above a stricter threshold (--core.faces.clustering.core-min-confidence)
• the actual face area defined by eyes and mouth corners must be big enough (avoid faces not front facing enough) ( --core.faces.clustering.core-front-facing-threhold)

These conditions are added to avoid creating big clusters of unclear faces that would swallow other clusters.

⁠Pass 2.0

Incremental support.

When adding new faces to already clustered faces (as the library grows), we first try to assign new core faces to existing clusters. We assign a face to the cluster of the closest clustered core face within EPS distance.

After adding points, we must check if clusters should be merged as new points might have bridged the gap.

⁠Pass 2.1

Assign each core face to a cluster. This is done by selecting a non-clustered core face, adding it to a cluster and recursively adding all core faces within EPS distance. When no more faces can be added a new cluster is created, and this is repeated until all core faces are assigned to a cluster.

⁠Pass 3

Assign non-core faces to clusters. This is done by assigning to the cluster of the closest core face within a certain distance, if any. Here we deviate from the original algorithm by using EPS * R (because EPS is conservative). R is controlled via the --core.faces.clustering.non-core-multiplier.

⁠Pass 4

This pass is an addition to the original algorithm. We add even more non-core faces to clusters by adding all unclustered faces that have at least K neighbors (core and non-core) in the same cluster within EPS * R distance. This pass should help to compensate differences in density inside clusters, and the fact that we are conservative with core faces.

⁠Tuning

The most important parameter to tune is the number of neighbors K (--core.faces.clustering.min-neighbors), as it will depend on the number of faces and their density:

• if many small clusters that correspond to the same person are created, it should be decreased
• if too few clusters containing multiple people are created, it should be increased

The clustering logs should help with that by checking how many core faces there are, how many clusters are created and how many are merged.

⁠Mapping clusters to people

After face embeddings have been computed and clustered, identification is done by assigning each cluster to a person (admin page).