koddr/csv2api repository overview

⁠csv2api – Parse CSV with filtering and sending to API

The csv2api parser reads the CSV file with the raw data, filters the records, identifies fields to be changed, and sends a request to update the data to the specified endpoint of your REST API.

All actions take place according to the settings in the configuration file.

Features:

• 100% free and open source.
• Works with any size of input CSV file.
• Use any configuration file format: JSON, YAML, TOML, or HCL (Terraform).
• Ability to keep a configuration file (and, in the future, input data file) on a remote server with HTTP access, it will be read as if it was in a folder on your local machine.
• Configure any request body for the REST API endpoint directly in the configuration file (in a clear declarative style).
• Extensive options for filtering incoming data from a CSV file.
• Provides extensive capabilities for constructing multiple filters to accurately perform actions on selected fields.

⁠⚡️ Quick start

Prepare your config and input data files:

• In your config.yml:
  • Make sure that the first column name in the columns_order section is a primary key (PK) for your process.
  • Set up your API (base URL, token, endpoints, etc) in the api section.
  • Set up the filter for your fields in the filter_columns section.
  • Set up fields to be updated in the update_fields section.
• In your input data.csv:
  • Make sure that the first line of your CSV file contains the correct field names.

And now, run csv2api with options:

docker run --rm -it -v ${PWD}:${PWD} -w ${PWD} koddr/csv2api:latest \
  -c /path/to/config.yml \
  -d /path/to/data.csv \
  -e CONFIG

Done! 🎉 Your transactions have been performed:

Hello and welcome to csv2api! 👋
                                
– According to the settings in './config.yml', 5 transactions were filtered out of 10 to start the process.
– Only 3 transactions got into the final set of actions to be taken... Please wait!
                                                                                                                                
 ✓ Field 'tags' with values '[paid]' in the transaction '2' has been successfully updated (HTTP 200)
 ✓ Field 'tags' with values '[paid]' in the transaction '8' has been successfully updated (HTTP 200)
 ✓ Field 'tags' with values '[paid]' in the transaction '10' has been successfully updated (HTTP 200)
                                                                                                
– Saving filtered transactions to CSV file './filtered-1686993960.csv' in the current dir... OK!
                                
All done! 🎉 Time elapsed: 0.11s

⁠🧩 Options

⁠✨ Solving case

In my work, I often have to work with large amounts of raw data in CSV format.

Usually it goes like this:

1. Unload a file with data from one system.
2. Clean up this file from duplicates and unnecessary columns.
3. Make some changes in some columns of some rows.
4. Mapping the processed lines from CSV file to the database structure fields.
5. Write a function to bypass the CSV file and form the request body.
6. Write an HTTP client that will send requests to the REST API endpoint.
7. Send prepared request body to the REST API endpoint in other system for specified DB records.

To ease this whole process, I created this parser that takes absolutely any data file as input, does the conversions and filtering, and is set up in one single configuration file.

Just prepare the data, set the configuration to your liking, run csv2api and wait a bit! Yes, it's that simple.

⁠⚠️ License

csv2api⁠ is free and open-source software licensed under the Apache 2.0 License⁠, created and supported with 🩵 for people and robots by Vic Shóstak⁠.