syscomiddleware/dup-backend-processor repository overview

⁠Some conventions for the file

• Each uploaded file should have header.
• File should be csv, seperated by ';'
• Each header item should be prefixed with its data type.
• If no prefix is provided, its assumed to be text-data.
• Allowed datatypes are
  • text
  • date
  • number
  • decimal
  • lat (latitude)
  • lon (longitude)

⁠How it works internally

• Create a JobQueue
• Initiate Parsers and Dispatchers
• Dispatchers create worker pool. WorkPool is shared between dispatcher and workers
• Start polling the input directory
  • Send the polled file info to workers
  • Workers receive jobs from dispatcher
  • Workers call Consume method on Parser
  • Consume method monitors local channels and goroutines