keptdream/hibpd repository overview

⁠What It Does

This project creates a lightweight webserver written in Rust that allows you to query your large local pwnedpasswords.txt file using an API similar to https://api.pwnedpasswords.com/range/FFFFF.

Note: While it mimics the behavior of the original API, it is not a complete drop-in replacement. For example, the server does not truncate the first 5 characters of the hash in the response.

⁠Why Does It Exist?

Importing a massive file like pwnedpasswords.txt into an existing database (e.g., MySQL or PostgreSQL) can be time-consuming and require significant disk space. With hibpd, you can work directly with your existing text file and a small index file, avoiding the need for full database imports.

⁠Why Do I Need It?

You might need this if you:

1. Prefer not to rely on the official Have I Been Pwned (HIBP) API for privacy or availability reasons.
2. Want to avoid importing large datasets into a database, saving both time and disk space.

hibpd compresses and optimizes the file to reduce storage requirements further.

⁠How Fast Is It?

hibpd is designed to be fast when running on an SSD. Although formal benchmarks aren't available (yet), it responds to queries almost instantly.

⁠Performance Details:

• No caching is implemented; only the necessary parts of the file are read during a request.
• Speed is mainly limited by your disk drive and CPU.
• The in-memory index requires around 100 MB of RAM at all times.

⁠Security Notes

The server is written in Rust, which provides strong memory safety guarantees. However, it is not designed to be publicly accessible in its current state.

⁠Key Considerations:

• No HTTPS Support: As of now, the server does not support HTTPS. This feature is planned for future updates.
• No Access Logs: The server does not produce any access logs by design, prioritizing security and privacy. While this minimizes the risk of sensitive data being logged, it also means there is no built-in way to monitor or audit usage.
• Intended Use: It is recommended to run the server within your internal network or behind a reverse proxy server (e.g., NGINX or traefik) to add an additional layer of security.
• Source Code: The source code will be made publicly available in the future, allowing for further inspection and security hardening.

⁠Available Docker Tags

• keptdream/hibpd:latest – The most recent version (recommended).
• keptdream/hibpd:1 – The latest patch release for version 1.
• keptdream/hibpd:1.0 – Specific release for version 1.0.

⁠How to Create an Index

1. Download the pwnedpasswords.txt file. Ensure the hashes in the file are sorted alphabetically (they usually are by default).

2. Run the following command:

   Standalone hibpd index pwnedpasswords.txt plain pwnedpasswords.index

   Using Docker docker run -v /directory/with/your/pwnedpasswords.txt/:/data keptdream/hibpd index /data/pwnedpasswords.txt plain /data/pwnedpasswords.index

• This process takes some time.
• The resulting index file will be approximately 23–24 MB in size.

⁠How to Run the Server

Standalone hibpd serve

Using Docker docker run -v /directory/with/your/pwnedpasswords.txt/:/var/lib/hibpd/ -p 8080:8080 keptdream/hibpd

⁠Usage

Query the API just like the original HIBP API:

http://localhost:8080/range/FFFFF

The output will look like this:

000000005AD76BD555C1D6D771DE417A4B87E4B4:10
00000000A8DAE4228F821FB418F59826079BF368:4
00000000DD7F2A1C68A35673713783CA390C9E93:876
00000001E225B908BAC31C56DB04D892E47536E0:6
00000006BAB7FC3113AA73DE3589630FC08218E7:3
00000008C4037D3E893F8E1FA7BAD32B9F60948C:3
00000008CD1806EB7B9B46A8F87690B2AC16F617:6
...

This implementation is not a complete drop-in replacement for the original HIBP API. It does not truncate the first 5 characters of the hash in the response. Keep this in mind when integrating it into your workflows.

⁠Using the CLI for Testing

The tool can also be used directly from the command line to query specific hash ranges:

Standalone hibpd range 00000

Using Docker docker run -v /directory/with/your/pwnedpasswords.txt/:/var/lib/hibpd/ -p 8080:8080 keptdream/hibpd range 00000

⁠Environment Variables

HIBPD_INDEX=/var/lib/hibpd/pwnedpasswords.index     # Index file location
HIBPD_DATABASE=/var/lib/hibpd/pwnedpasswords.txt    # Database file location
HIBPD_DATABASE_TYPE=plain                           # Options: plain or bgzf
HIBPD_SERVER_X_HOST=0.0.0.0                         # Server host
HIBPD_SERVER_X_PORT=8080                            # Server port

⁠Compression:

hibpd supports bgzf-compressed files, which can reduce the size of pwnedpasswords.txt by approximately 50%.

1. Compress the file:

   Standalone hibpd convert pwnedpasswords.txt pwnedpasswords.bgzf bgzf

   Using Docker docker run -v /directory/with/your/pwnedpasswords.txt/:/data keptdream/hibpd convert /data/pwnedpasswords.txt /data/pwnedpasswords.bgzf bgzf

2. Create a new index for the compressed file:

   Standalone hibpd index pwnedpasswords.bgzf bgzf pwnedpasswords.bgzf.index

   Using Docker docker run -v /directory/with/your/pwnedpasswords.txt/:/data keptdream/hibpd index /data/pwnedpasswords.bgzf bgzf /data/pwnedpasswords.bgzf.index

3. Update your configuration and enjoy the reduced file size!

⁠Additional Configuration

You can modify the configuration file at /etc/hibpd/config.toml if needed:

database_type = "plain"
database = "/var/lib/hibpd/pwnedpasswords.txt"
index = "/var/lib/hibpd/pwnedpasswords.index"

[server]
host = "0.0.0.0"
port = 8080

⁠Quick Links

• Developer: GitHub Profile⁠
• Original Have I Been Pwned Service: 'https://haveibeenpwned.com/'⁠

⁠Important Disclaimer

This tool is provided as-is and free of charge, without any warranty or guarantee of functionality, accuracy, or security. Use it at your own risk. The developer assumes no responsibility for any damages, data loss, or issues that may arise from using this software.

Additionally, this project is not affiliated with Troy Hunt or the official Have I Been Pwned (HIBP) service. It is an independent tool inspired by the concept of querying password breach data.