We ran into an issue with a data export file created from the Quote Manager. The millisecond timestamps are not being exported with leading zeros in the millisecond portion of the timestamp.
In the timestamps, if the millisecond portion of the timestamp has only one or two significant digits, MC is not providing the leading zeros in the timestamp field in the export file. An example is below.
The problem this will cause, if the file was used as an import file to another program, like Excel for example, is that the times will not be imported correctly.
Below, the 00:12:40.92 timestamp would be imported as 00:12:40.920, when it should be 00:12:40.092.
The timestamps will all be off by as much as almost a whole second in some cases (.9 vs. .009 = difference of .891 second).
Can a fix be issued for this right away, to correct this formatting problem? If so, thank you! Much appreciated!
eegroup
From a file exported from the Quote Manager (tick file, trade data):
Date, Timestamp, Price, Quantity
11/21/2014, 00:12:33.355, 1.5677, 1
11/21/2014, 00:12:33.355, 1.5677, 2
11/21/2014, 00:12:40.92, 1.5677, 1
11/21/2014, 00:12:40.92, 1.5677, 1
11/21/2014, 00:12:40.92, 1.5677, 1
11/21/2014, 00:12:40.102, 1.5677, 3
11/21/2014, 00:12:40.176, 1.5677, 8
QM Export File Leading Zeros Error, Millisecond Timestamps [SOLVED]
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2958 times
Re: QM Export File Leading Zeros Error, Millisecond Timestam
Hello eegroup,
We are currently analyzing this behavior.
We are currently analyzing this behavior.
Re: QM Export File Leading Zeros Error, Millisecond Timestam
Hi Henry,
Thank you for investigating and analyzing this issue.
Here is another example of a data error in the export file. In this case, we have a single digit millisecond timestamp. The 09:22:36.7 timestamp should be 09:22:36.007. If imported into another program, like Excel, it converts the timestamp to 09:22:36.700. The conversion is correct based upon how the data is exported from the Quote Manager. The data, however, is not correct as it is exported out of the Quote Manager.
Of interest, if we build a 1 tick chart of the data in the main MultiCharts program, and export the bar data directly from the workspace, the millisecond portion of the tick timestamp is exported correctly. This is a little impractical to accomplish as a workaround, however, when we have over 12 years of Future tick data on British Pound contract alone, which represent over 112 million tick records. MultiCharts has a little issue building a workspace with 112+ million bars on it. Not a practical solution to obtain the correct timestamps on all of the tick records.
If your analysis proves that there is an issue with the exported data from the Quote Manager, what would be your estimate of a fix/patch being made available? Can one be made available prior to when the next MC 9.0 r4 is released?
Thank you for your continue help and assistance with this issue. Much appreciated!
eegroup
From a file exported from the Quote Manager (tick file, trade data):
Date, Timestamp, Price, Quantity
6/13/2014, 09:22:35.970, 1.6942, 1
6/13/2014, 09:22:35.977, 1.6942, 1
6/13/2014, 09:22:35.979, 1.6942, 2
6/13/2014, 09:22:35.984, 1.6942, 1
6/13/2014, 09:22:36.7, 1.6942, 1
6/13/2014, 09:22:36.369, 1.6941, 1
6/13/2014, 09:22:36.369, 1.6941, 1
6/13/2014, 09:22:36.369, 1.6941, 1
6/13/2014, 09:22:36.369, 1.6941, 1
6/13/2014, 09:22:36.370, 1.6941, 1
6/13/2014, 09:22:36.370, 1.6941, 2
6/13/2014, 09:22:36.370, 1.6941, 2
6/13/2014, 09:22:36.370, 1.6941, 2
6/13/2014, 09:22:36.370, 1.6941, 1
6/13/2014, 09:22:36.370, 1.6941, 4
6/13/2014, 09:22:36.428, 1.6941, 2
6/13/2014, 09:22:37.157, 1.6941, 2
Thank you for investigating and analyzing this issue.
Here is another example of a data error in the export file. In this case, we have a single digit millisecond timestamp. The 09:22:36.7 timestamp should be 09:22:36.007. If imported into another program, like Excel, it converts the timestamp to 09:22:36.700. The conversion is correct based upon how the data is exported from the Quote Manager. The data, however, is not correct as it is exported out of the Quote Manager.
Of interest, if we build a 1 tick chart of the data in the main MultiCharts program, and export the bar data directly from the workspace, the millisecond portion of the tick timestamp is exported correctly. This is a little impractical to accomplish as a workaround, however, when we have over 12 years of Future tick data on British Pound contract alone, which represent over 112 million tick records. MultiCharts has a little issue building a workspace with 112+ million bars on it. Not a practical solution to obtain the correct timestamps on all of the tick records.
If your analysis proves that there is an issue with the exported data from the Quote Manager, what would be your estimate of a fix/patch being made available? Can one be made available prior to when the next MC 9.0 r4 is released?
Thank you for your continue help and assistance with this issue. Much appreciated!
eegroup
From a file exported from the Quote Manager (tick file, trade data):
Date, Timestamp, Price, Quantity
6/13/2014, 09:22:35.970, 1.6942, 1
6/13/2014, 09:22:35.977, 1.6942, 1
6/13/2014, 09:22:35.979, 1.6942, 2
6/13/2014, 09:22:35.984, 1.6942, 1
6/13/2014, 09:22:36.7, 1.6942, 1
6/13/2014, 09:22:36.369, 1.6941, 1
6/13/2014, 09:22:36.369, 1.6941, 1
6/13/2014, 09:22:36.369, 1.6941, 1
6/13/2014, 09:22:36.369, 1.6941, 1
6/13/2014, 09:22:36.370, 1.6941, 1
6/13/2014, 09:22:36.370, 1.6941, 2
6/13/2014, 09:22:36.370, 1.6941, 2
6/13/2014, 09:22:36.370, 1.6941, 2
6/13/2014, 09:22:36.370, 1.6941, 1
6/13/2014, 09:22:36.370, 1.6941, 4
6/13/2014, 09:22:36.428, 1.6941, 2
6/13/2014, 09:22:37.157, 1.6941, 2
Re: QM Export File Leading Zeros Error, Millisecond Timestam
As long as you don't rape Multicharts I think there is nothing wrong with that. You easily could separate the exports via separation in years, months and so on and obtain from there......
Of interest, if we build a 1 tick chart of the data in the main MultiCharts program, and export the bar data directly from the workspace, the millisecond portion of the tick timestamp is exported correctly. This is a little impractical to accomplish as a workaround, however, when we have over 12 years of Future tick data on British Pound contract alone, which represent over 112 million tick records. MultiCharts has a little issue building a workspace with 112+ million bars on it. Not a practical solution to obtain the correct timestamps on all of the tick records.
.....
Are you saying the export of data in the Quote Manager via qmd-file is correct, but the export to a different 3rd party file format (in your case Excel-csv) with single or double digit millisecond time stamps is incorrect ? If so, please correct your entry post and make this more clear from there......
Here is another example of a data error in the export file. In this case, we have a single digit millisecond timestamp. The 09:22:36.7 timestamp should be 09:22:36.007. If imported into another program, like Excel, it converts the timestamp to 09:22:36.700. The conversion is correct based upon how the data is exported from the Quote Manager. The data, however, is not correct as it is exported out of the Quote Manager.
.....
How do you exported these data to Excel ? Via csv-file ? Via dll-interaction ?
Regards.
Ben
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2958 times
Re: QM Export File Leading Zeros Error, Millisecond Timestam
eegroup, the issue has been confirmed. The fix is targeted to MultiCharts 9.0 Release 4.
I will get back to you if it is technically possible to provide a hotfix.
I will get back to you if it is technically possible to provide a hotfix.
Re: QM Export File Leading Zeros Error, Millisecond Timestam
Hi Henry,eegroup, the issue has been confirmed. The fix is targeted to MultiCharts 9.0 Release 4.
I will get back to you if it is technically possible to provide a hotfix.
Thank you very much for your confirmation and for the impending correction for this issue. Much appreciated!
eegroup
Re: QM Export File Leading Zeros Error, Millisecond Timestam
Henry, not quite related but something I would like to know: are these millisecond timestamps in QM based on timestamps in datafeed or local computer clock? Thanks.eegroup, the issue has been confirmed. The fix is targeted to MultiCharts 9.0 Release 4.
I will get back to you if it is technically possible to provide a hotfix.
Re: QM Export File Leading Zeros Error, Millisecond Timestam
Of interest, if we build a 1 tick chart of the data in the main MultiCharts program, and export the bar data directly from the workspace, the millisecond portion of the tick timestamp is exported correctly. This is a little impractical to accomplish as a workaround, however, when we have over 12 years of Future tick data on British Pound contract alone, which represent over 112 million tick records. MultiCharts has a little issue building a workspace with 112+ million bars on it. Not a practical solution to obtain the correct timestamps on all of the tick records.
First, bad choice of words, Ben. I was not criticizing MultiCharts in any way. I was simply pointing out that using two different methods of exporting data, where one exported the millisecond timestamps correctly (the main MultiCharts program), the other did not (the Quote Manager). Second, the issue is that exporting tick data from a one tick chart in the main MultiCharts program is not the preferred or acceptable way of exporting large volumes of data. By using your suggestion, while it might work in theory by doing maybe a year at a time, building 12 separate workspaces to contain one tick bar data on each of them (representing the 12 years of Futures historical data), and then having to concatenate the 12 different files together is not quite the ultimate way do doing things, especially across 45 different Future symbols. One extract file created per symbol from the Quote Manager is the only practical way to do what we need to accomplish. I am also not sure that even a bar chart built with 10+ million single tick bars is that practical, even if the files could be built a year a time. Not really interested in going there, for the reasons mentioned above.As long as you don't rape Multicharts I think there is nothing wrong with that. You easily could separate the exports via separation in years, months and so on and obtain from there.
I have no idea if a qmd-file extract is formatted correct or not. We only export .csv files from the Quote Manager, so I have not checked that, nor do I have a reason to. Sorry I cannot be of more assistance in answering your question/concern. My original post was correct, in the context of what I was reporting. I thought it was very clear as initially presentedAre you saying the export of data in the Quote Manager via qmd-file is correct, but the export to a different 3rd party file format (in your case Excel-csv) with single or double digit millisecond time stamps is incorrect ? If so, please correct your entry post and make this more clear from there.
How do you exported these data to Excel ? Via csv-file ? Via dll-interaction ?
We really do not import the data into Excel. Excel was simply an example of how the milliseconds would not be imported properly the way the data was being formatted coming out of the Quote Manager. That example would have been true for any application in which the exported file would have been imported, and there are many applications which can accept .csv files as input besides Excel.
We hope that helps to further your understanding of our previous posts. Thank you for the opportunity to share these explanations with you.
eegroup
Re: QM Export File Leading Zeros Error, Millisecond Timestam
Hi Orion,Henry, not quite related but something I would like to know: are these millisecond timestamps in QM based on timestamps in datafeed or local computer clock? Thanks.eegroup, the issue has been confirmed. The fix is targeted to MultiCharts 9.0 Release 4.
I will get back to you if it is technically possible to provide a hotfix.
You will need to confirm this with Henry, but I believe based upon our past experience, that the timestamps in the MultiCharts database (and when the timestamps are subsequently used in the QM) are all based upon the Exchange time of the contract. This time can then be adjusted as to your specific needs (Local or Exchange), while accessing the data in a workspace in the main MultiCharts program, or when exporting the data from the Quote Manager. MultiCharts provides lots of flexibility regarding the basis of the timestamp time zone, so it really does not matter too much regarding how the data is store in the database.
Hope that helps...
eegroup
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2958 times
Re: QM Export File Leading Zeros Error, Millisecond Timestam
orion, it depends on the data provider you are using.Henry, not quite related but something I would like to know: are these millisecond timestamps in QM based on timestamps in datafeed or local computer clock? Thanks.eegroup, the issue has been confirmed. The fix is targeted to MultiCharts 9.0 Release 4.
I will get back to you if it is technically possible to provide a hotfix.
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2958 times
Re: QM Export File Leading Zeros Error, Millisecond Timestam [SOLVED]
This issue has been resolved in MultiCharts 9.0 Release 4 builds 10359/10360. You can download it here.