bmeares/meerschaum repository overview

PyPI	GitHub	License	Stats

⁠What is Meerschaum?

Meerschaum is a tool for quickly synchronizing time-series data streams called pipes. With Meerschaum, you can have a data visualization stack running in minutes.

The power of the Meerschaum system comes from projects like pandas⁠, sqlalchemy⁠, fastapi⁠, and more⁠.

⁠Why Meerschaum?

If you've worked with time-series data, you know the headaches that come with ETL. Data engineering often gets in analysts' way, and when work needs to get done, every minute spent on pipelining is time taken away from real analysis.

Rather than copy / pasting your ETL scripts, simply build pipes with Meerschaum! Meerschaum gives you the tools to design your data streams how you like⁠ ― and don't worry — you can always incorporate Meerschaum into your existing systems!

⁠Features

• 📊 Built for Data Scientists and Analysts
  • Integrate with Pandas, Grafana and other popular data analysis tools⁠.
  • Persist your dataframes and always get the latest data.
• ⚡️ Production-Ready, Batteries Included
  • Synchronization engine⁠ concurrently updates many time-series data streams.
  • One-click deploy a TimescaleDB and Grafana stack⁠ for prototyping.
  • Serve data to your entire organization through the power of uvicorn, gunicorn, FastAPI
• 🔌 Easily Expandable
  • Ingest any data source with the plugin system⁠. Just return a DataFrame, and Meerschaum handles the rest.
  • Add any function as a command⁠ to the Meerschaum system.
  • Include Meerschaum in your projects with its easy-to-use Python API⁠.
• ✨ Tailored for Your Experience
  • Rich CLI makes managing your data streams surprisingly enjoyable!
  • Web dashboard for those who prefer a more graphical experience.
  • Manage your database connections with Meerschaum connectors⁠
  • Utility commands with sensible syntax let you control many pipes with grace.
• 📾 Portable from the Start
  • The environment variable $MRSM_ROOT_DIR lets you emulate multiple installations and group together your instances⁠.
  • No dependencies required; anything needed will be installed into a virtual environment.
  • Required packages for plugins

⁠Installation

For a more thorough setup guide, visit the Getting Started⁠ page at meerschaum.io⁠.

⁠TL;DR

pip install -U --user meerschaum
mrsm stack up -d db grafana
mrsm bootstrap pipes

⁠Usage Documentation

Please visit meerschaum.io⁠ for setup, usage, and troubleshooting information. You can find technical documentation at docs.meerschaum.io⁠.

⁠Plugins

Here is the list of community plugins⁠.

For details on installing, using, and writing plugins, check out the plugins documentation⁠ at meerschaum.io⁠.

⁠Support Meerschaum's Development

For consulting services and to support Meerschaum's development, please considering sponsoring me on GitHub sponsors⁠.

Additionally, you can always buy me a coffee☕⁠!

⁠License

Copyright 2021 Bennett Meares

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0⁠

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.