+1 888 340 6572
MultiCharts Project Management
previous_open_issue.png
Go to the previous open issue
previous_issue.png
Go to the previous issue (open or closed)
star_faded.png
Please log in to bookmark issues
feature_request_small.png
Open Feature request MC-2630

Quote Manager Export Scales Poorly When Exporting Large Data Sets - Make Parallel?

action_vote_minus_faded.png
2
Votes
action_vote_plus_faded.png
next_issue.png
Go to the next issue (open or closed)
next_open_issue.png
Go to the next open issue
Description

If I export a great deal of data from Quote Manager, it exports each item one-by-one in serial. While this is underway, my CPU, memory, and disk are all under-utilized. It takes a great deal of time to process all of the symbols for export. It looks like it could drastically improve performance by processing the items in parallel.

Comments (1)
#1
user-offline.png  dataheck (dataheck)
Mar 14, 2020 - 15:37

I have an AMD Ryzen 7 3700X 8-Core processor, 64 GB of RAM, and two SSDs. My MC database is on one SSD, and I dump data to the other one.

It takes 12068 seconds (3.4 hours) to export 400 of my fields. I regularly export 1431, as of this writing, but it will likely grow.

QuoteManager is utilizing 7.5% of my CPU and less than 1% of my disk's bandwidth.

Whatever its doing, it should be able to do it faster. I process the exported data on my own using Python (package it up into a bunch of pickled dataframes) and it takes 50 minutes to process with my (multiprocessing) code. Total export took around 4.5 hours from QuoteManager.

e:
I seem to also be exporting so many records it overflows the record counter - probably using a signed integer:

(-211800000000 records in 15840 sec)

History
Issue basics
  • Type of issue
    Feature request
  • Category
    Not determined
  • Targeted for
    Not determined
  • Status
    Under Review
User pain
  • Type of bug
    Not triaged
  • Likelihood
    Not triaged
  • Effect
    Not triaged
Affected by this issue (0)
There are no items
People involved
Times and dates
  • Posted at
  • Last updated
Issue details
  • Resolution
    Not determined
Attachments (0)
There is nothing attached to this issue
Commits (0)
There are no code checkins for this issue
Duplicate issues (0)
This issue does not have any duplicates