Possible DataLoader overflow?  [SOLVED]

Questions about MultiCharts .NET and user contributed studies.
User avatar
jwebster503
Posts: 24
Joined: 13 Mar 2014
Has thanked: 9 times
Been thanked: 14 times

Possible DataLoader overflow?

Postby jwebster503 » 01 Sep 2015

Hi, Everyone. I'm running into an issue with DataLoader.

I'm asking DataLoader for ESU15 from Jun 1st through Sep 1st with 1-tick resolution. I do get a response back, but it's malformed. I know this is a HUGE amount of data, so I cut my date range and got back expected results:

Starting 6/1 through:
- 8/15: 20.7 million ticks in the result.Data array -- result.Event is DataLoadedEvent.History as expected -- result.RTData is null as expected.
- 8/16: 21.1M ticks, History, null
- 8/18: 21.4M ticks, History, null
- 8/19: 22.1M ticks, History, null
- 8/20: 22.9M ticks, History, null
- 8/21: 24.0M ticks, History, null
- 8/23: 24.2M ticks, History, null
- 8/24: 26.2M ticks, History, null
- 8/25 -- Craziness ensues. The result.Data array is empty (zero elements), the Event is still History, but RTData has a current-tick value. ?!?

I'd give a code example, but I don't think the issue is with my code -- it's run successfully thousands of times, just not with this much data. Also, the fact that the IDataLoaderResult.Data array should have a growing size with increasing ticks, and it comes back empty, and that a realtime tick is given for a historical event?!? Something else is going on inside DataLoader.

Has anyone ever run into issues with DataLoader overflowing?
If so, is there a way to detect that a request will result in an overflow?
Or, instead, is there a way to request the data to be returned in chunks?

Thanks!

Jeff

User avatar
jwebster503
Posts: 24
Joined: 13 Mar 2014
Has thanked: 9 times
Been thanked: 14 times

Re: Possible DataLoader overflow?

Postby jwebster503 » 08 Sep 2015

I have created a minimal study which demonstrates the probable DataLoader overflow issue, along with a workspace and screenshot of the output.

When loading the workspace, set the date range from Jun 1 2015 to Aug 25 2015 -- you'll see that just under 26.2 million ticks are returned. Push the end date out one more day to Aug 26 and you'll see that a malformed DataLoaderResult is given with 0 ticks (and on my full-sized code, a populated RTData even though the result.Event is History). When I call EndLoadData on it, I get an exception "Array dimensions exceeded supported range."
DataLoaderTest Output.png
Output screenshot
(14.54 KiB) Downloaded 2075 times
MultiCharts, can you please confirm that there is an issue with DataLoader? Also, will I miss any ticks if I create partial calls, say Jun 1 to Jun 30, a second call for Jul 1 to Jul 31, a third call for Aug 1 to Aug 26, etc? Lastly, given that the DataLoaderResult has a property "IsCompleted" which "Returns whether data loading is completed.", can I have the DataLoader automatically split these results if the requested data is too large?

Thanks,

Jeff
Attachments
DataLoaderTest.wsp
(69.67 KiB) Downloaded 627 times
DataLoaderTest.pln
(1.69 KiB) Downloaded 597 times

User avatar
jwebster503
Posts: 24
Joined: 13 Mar 2014
Has thanked: 9 times
Been thanked: 14 times

Re: Possible DataLoader overflow?

Postby jwebster503 » 15 Sep 2015

*bump* (sorry for the repeated ask for assistance, but I'm not sure where else to turn, and I don't want the initial request to age into oblivion... :( ) Nobody has had issues with loading > 26 million ticks via DataLoader?

Thanks,

Jeff

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Possible DataLoader overflow?

Postby Henry MultiСharts » 29 Sep 2015

Hello jwebster503,

What exact version and build number of MultiCharts are you running? (in MultiCharts go to Help tab-> About)

User avatar
jwebster503
Posts: 24
Joined: 13 Mar 2014
Has thanked: 9 times
Been thanked: 14 times

Re: Possible DataLoader overflow?

Postby jwebster503 » 29 Sep 2015

Hi, Henry.

I am using MultiCharts .NET64 Version 9.0 Release (Build 10761).

Thanks,

Jeff

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Possible DataLoader overflow?

Postby Henry MultiСharts » 05 Oct 2015

Hello jwebster503,

This issue has been confirmed. The fix is targeted to MultiCharts .NET 9.1 Release.
There is no ETA for the version yet. Please follow our blog to get the latest updates:
https://www.multicharts.com/traders-blog/

User avatar
jwebster503
Posts: 24
Joined: 13 Mar 2014
Has thanked: 9 times
Been thanked: 14 times

Re: Possible DataLoader overflow?

Postby jwebster503 » 25 Oct 2015

Hi, Henry (et al),

I am running "MultiCharts .NET64 Version 9.1 Release (Build 11817)", and I have verified that the issue still exists in this version, with symptoms unchanged -- I'm still receiving the malformed DataLoaderResult with an exception "Array dimensions exceeded supported range" when I try to EndLoadData on the result.

Perhaps the fix didn't make it into this release?

Thanks,

Jeff

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Possible DataLoader overflow?

Postby Henry MultiСharts » 27 Oct 2015

Hi Jeff,

The fix has been postponed until MultiCharts .NET 9.1 Release 2.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Possible DataLoader overflow?  [SOLVED]

Postby Henry MultiСharts » 01 Dec 2015

Hi Jeff,

We have studied this case. That was impossible to load the data as the system was unable to provide enough memory for this operation, but the corresponding error message did not appear. Since MultiCharts .NET 9.1 Release 2 DataLoader returns "Out of Memory" exception when accessing Data(N) if the system is unable to provide enough memory for loading the data. You need to catch and handle this exception in your code, example below:

Code: Select all

protected void DataCB(IDataLoaderResult Result, Test_DataLoader _this)
{
try
{
if (Result.IsCompleted)
{
if (Result.Request.QuoteField == RequestQuoteField.Trade)
{
Output.WriteLine("DataCount: {0}, {1}", Result.Data == null ? -1 : Result.Data.Length, Result.Event);
}
}
Output.WriteLine("{0}", Result.IsCompleted);
}
catch(System.Exception e){
Output.WriteLine("Error {0}", e);
}
}
It is also worth to mention that if one, for example, calls ExecControl.Abort in the exception handler - the exception will not be caught as it appears in a different thread.


Return to “MultiCharts .NET”