Hi, Everyone. I'm running into an issue with DataLoader.
I'm asking DataLoader for ESU15 from Jun 1st through Sep 1st with 1-tick resolution. I do get a response back, but it's malformed. I know this is a HUGE amount of data, so I cut my date range and got back expected results:
Starting 6/1 through:
- 8/15: 20.7 million ticks in the result.Data array -- result.Event is DataLoadedEvent.History as expected -- result.RTData is null as expected.
- 8/16: 21.1M ticks, History, null
- 8/18: 21.4M ticks, History, null
- 8/19: 22.1M ticks, History, null
- 8/20: 22.9M ticks, History, null
- 8/21: 24.0M ticks, History, null
- 8/23: 24.2M ticks, History, null
- 8/24: 26.2M ticks, History, null
- 8/25 -- Craziness ensues. The result.Data array is empty (zero elements), the Event is still History, but RTData has a current-tick value. ?!?
I'd give a code example, but I don't think the issue is with my code -- it's run successfully thousands of times, just not with this much data. Also, the fact that the IDataLoaderResult.Data array should have a growing size with increasing ticks, and it comes back empty, and that a realtime tick is given for a historical event?!? Something else is going on inside DataLoader.
Has anyone ever run into issues with DataLoader overflowing?
If so, is there a way to detect that a request will result in an overflow?
Or, instead, is there a way to request the data to be returned in chunks?
Thanks!
Jeff
Possible DataLoader overflow? [SOLVED]
- jwebster503
- Posts: 24
- Joined: 13 Mar 2014
- Has thanked: 9 times
- Been thanked: 14 times
- jwebster503
- Posts: 24
- Joined: 13 Mar 2014
- Has thanked: 9 times
- Been thanked: 14 times
Re: Possible DataLoader overflow?
I have created a minimal study which demonstrates the probable DataLoader overflow issue, along with a workspace and screenshot of the output.
When loading the workspace, set the date range from Jun 1 2015 to Aug 25 2015 -- you'll see that just under 26.2 million ticks are returned. Push the end date out one more day to Aug 26 and you'll see that a malformed DataLoaderResult is given with 0 ticks (and on my full-sized code, a populated RTData even though the result.Event is History). When I call EndLoadData on it, I get an exception "Array dimensions exceeded supported range."
MultiCharts, can you please confirm that there is an issue with DataLoader? Also, will I miss any ticks if I create partial calls, say Jun 1 to Jun 30, a second call for Jul 1 to Jul 31, a third call for Aug 1 to Aug 26, etc? Lastly, given that the DataLoaderResult has a property "IsCompleted" which "Returns whether data loading is completed.", can I have the DataLoader automatically split these results if the requested data is too large?
Thanks,
Jeff
When loading the workspace, set the date range from Jun 1 2015 to Aug 25 2015 -- you'll see that just under 26.2 million ticks are returned. Push the end date out one more day to Aug 26 and you'll see that a malformed DataLoaderResult is given with 0 ticks (and on my full-sized code, a populated RTData even though the result.Event is History). When I call EndLoadData on it, I get an exception "Array dimensions exceeded supported range."
MultiCharts, can you please confirm that there is an issue with DataLoader? Also, will I miss any ticks if I create partial calls, say Jun 1 to Jun 30, a second call for Jul 1 to Jul 31, a third call for Aug 1 to Aug 26, etc? Lastly, given that the DataLoaderResult has a property "IsCompleted" which "Returns whether data loading is completed.", can I have the DataLoader automatically split these results if the requested data is too large?
Thanks,
Jeff
- Attachments
-
- DataLoaderTest.wsp
- (69.67 KiB) Downloaded 630 times
-
- DataLoaderTest.pln
- (1.69 KiB) Downloaded 600 times
- jwebster503
- Posts: 24
- Joined: 13 Mar 2014
- Has thanked: 9 times
- Been thanked: 14 times
Re: Possible DataLoader overflow?
*bump* (sorry for the repeated ask for assistance, but I'm not sure where else to turn, and I don't want the initial request to age into oblivion... ) Nobody has had issues with loading > 26 million ticks via DataLoader?
Thanks,
Jeff
Thanks,
Jeff
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Possible DataLoader overflow?
Hello jwebster503,
What exact version and build number of MultiCharts are you running? (in MultiCharts go to Help tab-> About)
What exact version and build number of MultiCharts are you running? (in MultiCharts go to Help tab-> About)
- jwebster503
- Posts: 24
- Joined: 13 Mar 2014
- Has thanked: 9 times
- Been thanked: 14 times
Re: Possible DataLoader overflow?
Hi, Henry.
I am using MultiCharts .NET64 Version 9.0 Release (Build 10761).
Thanks,
Jeff
I am using MultiCharts .NET64 Version 9.0 Release (Build 10761).
Thanks,
Jeff
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Possible DataLoader overflow?
Hello jwebster503,
This issue has been confirmed. The fix is targeted to MultiCharts .NET 9.1 Release.
There is no ETA for the version yet. Please follow our blog to get the latest updates:
https://www.multicharts.com/traders-blog/
This issue has been confirmed. The fix is targeted to MultiCharts .NET 9.1 Release.
There is no ETA for the version yet. Please follow our blog to get the latest updates:
https://www.multicharts.com/traders-blog/
- jwebster503
- Posts: 24
- Joined: 13 Mar 2014
- Has thanked: 9 times
- Been thanked: 14 times
Re: Possible DataLoader overflow?
Hi, Henry (et al),
I am running "MultiCharts .NET64 Version 9.1 Release (Build 11817)", and I have verified that the issue still exists in this version, with symptoms unchanged -- I'm still receiving the malformed DataLoaderResult with an exception "Array dimensions exceeded supported range" when I try to EndLoadData on the result.
Perhaps the fix didn't make it into this release?
Thanks,
Jeff
I am running "MultiCharts .NET64 Version 9.1 Release (Build 11817)", and I have verified that the issue still exists in this version, with symptoms unchanged -- I'm still receiving the malformed DataLoaderResult with an exception "Array dimensions exceeded supported range" when I try to EndLoadData on the result.
Perhaps the fix didn't make it into this release?
Thanks,
Jeff
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Possible DataLoader overflow?
Hi Jeff,
The fix has been postponed until MultiCharts .NET 9.1 Release 2.
The fix has been postponed until MultiCharts .NET 9.1 Release 2.
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Possible DataLoader overflow? [SOLVED]
Hi Jeff,
We have studied this case. That was impossible to load the data as the system was unable to provide enough memory for this operation, but the corresponding error message did not appear. Since MultiCharts .NET 9.1 Release 2 DataLoader returns "Out of Memory" exception when accessing Data(N) if the system is unable to provide enough memory for loading the data. You need to catch and handle this exception in your code, example below:
It is also worth to mention that if one, for example, calls ExecControl.Abort in the exception handler - the exception will not be caught as it appears in a different thread.
We have studied this case. That was impossible to load the data as the system was unable to provide enough memory for this operation, but the corresponding error message did not appear. Since MultiCharts .NET 9.1 Release 2 DataLoader returns "Out of Memory" exception when accessing Data(N) if the system is unable to provide enough memory for loading the data. You need to catch and handle this exception in your code, example below:
Code: Select all
protected void DataCB(IDataLoaderResult Result, Test_DataLoader _this)
{
try
{
if (Result.IsCompleted)
{
if (Result.Request.QuoteField == RequestQuoteField.Trade)
{
Output.WriteLine("DataCount: {0}, {1}", Result.Data == null ? -1 : Result.Data.Length, Result.Event);
}
}
Output.WriteLine("{0}", Result.IsCompleted);
}
catch(System.Exception e){
Output.WriteLine("Error {0}", e);
}
}