undp/ms-graph-exporter repository overview

⁠MsGraphExporter

MsGraphExporter is an application that performs periodic export of time-domain data like Azure AD user signins from Microsoft Graph API⁠ into a buffer key-value store (currently supports Redis⁠) for subsequent processing. It uses Celery⁠ task queue for parallel processing, gevent⁠ greenlets for concurrent uploads, relies on the Graph API pagination⁠ to control memory footprint and respects Graph API throttling⁠. The application could be deployed as a single-container worker or as a set of multiple queue-specific workers for high reliability and throughput.

⁠Getting Started

Follow these instructions to use the application.

⁠Installing

MsGraphExporter is distributed through the Python Package Index⁠ as ms-graph-exporter⁠. Run the following command to:

• install a specific version

  pip install "ms-graph-exporter==0.1"
  

  Copy

• install the latest version

  pip install "ms-graph-exporter"
  

  Copy

• upgrade to the latest version

  pip install --upgrade "ms-graph-exporter"
  

  Copy

• install optional DEV dependencies like pytest framework and plugins necessary for performance and functional testing

  pip install "ms-graph-exporter[test]"
  

  Copy

MsGraphExporter is also distributed as the Alpine-based docker container and published on Docker Hub⁠ as undp/ms-graph-exporter⁠.

Service Principal credentials with correct access rights to Graph API must be defined for docker/docker-compose.yamlexample deployment to work. Following .env file should be placed at the root of the project folder, or following ENV variables defined in other way:

GRAPH_CLIENT_ID=<clientId>
GRAPH_CLIENT_SECRET=<clientSecret>
GRAPH_TENANT=<tenant>

Then, docker deployment could be initiated with:

docker-compose --file docker/docker-compose.yaml up

⁠Requirements

• Python >= 3.6

⁠Built using

• Celery⁠ - Distributed task queue
• gevent⁠ - concurrent data upload to Redis⁠
• redis-py⁠ - Python interface to Redis⁠

⁠Versioning

We use Semantic Versioning Specification⁠ as a version numbering convention.

⁠Release History

For the available versions, see the tags on this repository⁠. Specific changes for each version are documented in CHANGELOG.md⁠.

Also, conventions for git commit messages are documented in CONTRIBUTING.md⁠.

⁠Authors

• Oleksiy Kuzmenko - OK-UNDP@GitHub⁠ - Initial design and implementation

⁠Acknowledgments

• Hat tip to all individuals shaping design of this project by sharing their knowledge in articles, blogs and forums.

⁠License

Unless otherwise stated, all authors (see commit logs) release their work under the MIT License⁠. See LICENSE.md⁠ for details.

⁠Contributing

There are plenty of ways you could contribute to this project. Feel free to:

• submit bug reports and feature requests
• outline, fix and expand documentation
• peer-review bug reports and pull requests
• implement new features or fix bugs

See CONTRIBUTING.md⁠ for details on code formatting, linting and testing frameworks used by this project.