redirhub/watchmen_worker repository overview

⁠watchmen

Watchmen is a distributed monitoring system, it contains gaffer and worker to distribute and run highly configurable monitoring tasks in many different regions.

Watchmen use Tonrado, aio-pika and other asyncio libs to implement asynchronous message processing, monitoring program and HTTP APIs by coroutine⁠. It can provide the above multiple functions in one thread, so it's lightweight and uses less resources to meet higher concurrent tasks.

⁠How it works

There are many workers deployed in different regions to ensure the wide range of monitoring tasks, and at least one gaffer to distribute tasks, collect data from workers and provide APIs for human.

Watchmen use RabbitMQ to distribute tasks and collect data. RabbitMQ is the most widely deployed open source message broker. It has many good and detailed documents, reliable and easy-to-use plugins and management tools, and high reliability which is the key point to make sure our data will not be lost. Also, there is a good asyncio lib aio-pika⁠ to use Asynchronous message processing⁠.

When gaffer senses task is triggered, it wrap and send it to the specified queues according to task's configuration. Workers that subscribed these queues take the message data of mission from RabbitMQ, run monitoring program and publish result to data queue. Gaffer take the result data of mission from data queue, save it and notify users by api callback, email and other social apps.

Each working queue arranges tasks in one region, such as a country, a city. Each worker will subscribe at least one working queue and publish data to one data queue depending on its configuration. Each gaffer will subscribe one data queue to collect results from workers. A mission means that one worker will run monitoring once.

⁠Notice

1. A task may need to be executed multiple times, so this type of task will be sent multiple times, so there may be some same mission in queue, worker will not run one mission twice in one task round.
2. Priority queue⁠ ensure more users' tasks are executed when the number of workers is tight.
3. Time-To-Live message⁠ make queue clean when next round reached.