Public Repository

Last pushed: 2 days ago
Short Description
Queue based data collector and transmitter. Online/offline JSON collection, buffering, queuing.
Full Description








rxtx source

rxtx by txn2

rxtx is a queue based data collector > data transmitter. Useful for online/offline data collection, back pressure buffering or general queuing. rxtx uses bbolt maintained by CoreOs, a single file database for storing messages before they can be sent.

Test on MacOs

Install with brew

brew tap txn2/homebrew-tap
brew install rxtx

Help

rxtx -h

Test Docker Container

Help

docker run --rm -it txn2/rxtx -h

on arm 6/7 based device:

docker run --rm -it txn2/rxtx:arm32v6-1.0.2 -h

Test Source

Help

go run ./rxtx.go -h

Usage of rxtx:
  -batch int
        Batch size. (default 5000)
  -ingest string
        Ingest server. (default "http://localhost:8081/in")
  -interval int
        Seconds between intervals. (default 30)
  -maxq int
        Max number of message in queue. (default 2000000)
  -name string
        Service name. (default "rxtx")
  -path string
        Directory to store database. (default "./")
  -port string
        Server port. (default "8080")

Start server on 8080

go run ./rxtx.go 

Add message to queue

The rxtx services accepts HTTP POST data to an API endpoint in the following form /rx/PRODUCER/KEY/LABEL/.../. One label is required, however as many labels as necessary may be added, separated by a forward slash.

curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/data

Add message to queue every second

 while true; do curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/data; sleep 1; done

Add 1000 messages to the queue.

 time for i in {1..1000}; do curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/data; done

Building and Releasing

rxtx uses GORELEASER to build binaries and Docker containers.

Test Release Steps

Install GORELEASER with brew (MacOS):

brew install goreleaser/tap/goreleaser

Build without releasing:

goreleaser --skip-publish --rm-dist --skip-validate

Release Steps

  • Commit latest changes
  • Tag a version git tag -a v2.0 -m "Version 2.0"
  • Push tag git push origin v2.0
  • Run: GITHUB_TOKEN=$GITHUB_TOKEN goreleaser --rm-dist

Resources

License

Docker Pull Command
Owner
txn2