A tool to insert QuoteManager-exported CSV files into a PostgreSQL database

Questions about MultiCharts .NET and user contributed studies.
User avatar
dataheck
Posts: 30
Joined: 19 Nov 2019
Location: Amherst, Nova Scotia
Been thanked: 5 times
Contact:

A tool to insert QuoteManager-exported CSV files into a PostgreSQL database

Postby dataheck » 11 Jul 2020

I'm into quantitative trading and data science, and so naturally I wanted my market data in a database. I used this as an opportunity to learn the Rust programming language. This currently only inserts minute and daily data, it will ignore other files. That reminds me that I did not document that fact yet!

I've shared my code on GitHub, and have included a release build for Windows:

https://github.com/isosphere/finance_csv_to_postgresql

It's a concurrent application, you'll want to tweak the --threads option as it defaults to 10. You'll need to let it create the table structure it expects with --create on the first run.

It should hopefully be pretty fast. The bottleneck is insertion. I could probably improve how I am doing that significantly, it's possible I've configured the indexes poorly and have a massive insertion overhead.

I'd also like to make it smart about not inserting existing data. Right now it uses the ON CONFLICT DO NOTHING option but it would be more performant to retrieve dates that we don't have data for and only insert if this is in that set (or greater than the max, etc.).

Return to “MultiCharts .NET”