+1 888 340 6572

GET STARTED

Open Feature request MC-2630

Quote Manager Export Scales Poorly When Exporting Large Data Sets - Make Parallel?

Votes

Description

If I export a great deal of data from Quote Manager, it exports each item one-by-one in serial. While this is underway, my CPU, memory, and disk are all under-utilized. It takes a great deal of time to process all of the symbols for export. It looks like it could drastically improve performance by processing the items in parallel.

Comments (1)

Toggle system-generated comments

dataheck

Matthew Scheffel
Last seen online at Aug 25, 2021 - 20:12
Show user details

Mar 14, 2020 - 18:37

I have an AMD Ryzen 7 3700X 8-Core processor, 64 GB of RAM, and two SSDs. My MC database is on one SSD, and I dump data to the other one.

It takes 12068 seconds (3.4 hours) to export 400 of my fields. I regularly export 1431, as of this writing, but it will likely grow.

QuoteManager is utilizing 7.5% of my CPU and less than 1% of my disk's bandwidth.

Whatever its doing, it should be able to do it faster. I process the exported data on my own using Python (package it up into a bunch of pickled dataframes) and it takes 50 minutes to process with my (multiprocessing) code. Total export took around 4.5 hours from QuoteManager.

e:
I seem to also be exporting so many records it overflows the record counter - probably using a signed integer:

(-211800000000 records in 15840 sec)

Edit comment #1

Comment visibility (whether to hide this comment for "regular users")

Comment

Selected syntax: Markdown ▼

Don't worry about URLs - they will automatically be transformed to links.
See more formatting tips in MarkdownFormatting.

To create a bulleted list, start one or more lines with a star ("* bulleted list item"). You can create a numbered list by using the hash/pund sign instead ("# numbered list item").
See more formatting tips in MarkdownFormatting.

If you want to add text that is not supposed to be parsed, put it inside tags, or start the line with a space.
See more formatting tips in WikiFormatting.

or cancel

Reply to comment #1

Comment visibility (whether to hide this comment for "regular users")

Selected syntax: Markdown ▼

To auto-link to an existing issue, write "issue", "bug", "ticket" or the issue type (like "bug report" or "enhancement") followed by the issue number (ex: "ticket #3" or "bug report MYPROJ-1"). Enclose sourcecode samples between two "~~~~" lines.
See more formatting tips in MarkdownFormatting.

Code samples are best presented between two "~~~~"-lines. The Bug Genie uses GeSHi for syntax highlighting with support for over 100 languages! (<source lang="php"><?php echo "fu"; ?></source>)
See more formatting tips in MarkdownFormatting.

If you want to add text that is not supposed to be parsed, put it inside tags, or start the line with a space.
See more formatting tips in WikiFormatting.

or cancel

History

Issue basics

Type of issue

Feature request Unknown issue type
Issue label

No label set
Category

Not determined
Targeted for

Not determined
Status

Under Review

Status not determined
Progress
Priority

Not determined

People involved

Posted by
dataheck

Matthew Scheffel
Last seen online at Aug 25, 2021 - 20:12

Show user details
Owned by

Not owned by anyone
Assigned to

Not assigned to anyone
Subscribers

1 subscriber(s)

Click here to show the list of subscribers

Times and dates

Posted at

Feb 02, 2020
Last updated

Feb 02, 2020
Estimated time

Not estimated
Time spent

No time spent

Click here to see time logged against this issue

Issue details

Resolution

Not determined
Reproducability

Not determined
Severity

Not determined

Commits (0)

There are no code checkins for this issue

Quote Manager Export Scales Poorly When Exporting Large Data Sets - Make Parallel?

Welcome to Project Management