QM Export File Leading Zeros Error, Millisecond Timestamps  [SOLVED]

Questions about MultiCharts and user contributed studies.
eegroup
Posts: 78
Joined: 04 Aug 2008
Has thanked: 11 times
Been thanked: 9 times

QM Export File Leading Zeros Error, Millisecond Timestamps

Postby eegroup » 21 Nov 2014

We ran into an issue with a data export file created from the Quote Manager. The millisecond timestamps are not being exported with leading zeros in the millisecond portion of the timestamp.

In the timestamps, if the millisecond portion of the timestamp has only one or two significant digits, MC is not providing the leading zeros in the timestamp field in the export file. An example is below.

The problem this will cause, if the file was used as an import file to another program, like Excel for example, is that the times will not be imported correctly.

Below, the 00:12:40.92 timestamp would be imported as 00:12:40.920, when it should be 00:12:40.092.

The timestamps will all be off by as much as almost a whole second in some cases (.9 vs. .009 = difference of .891 second).

Can a fix be issued for this right away, to correct this formatting problem? If so, thank you! Much appreciated!

eegroup

From a file exported from the Quote Manager (tick file, trade data):

Date, Timestamp, Price, Quantity
11/21/2014, 00:12:33.355, 1.5677, 1
11/21/2014, 00:12:33.355, 1.5677, 2
11/21/2014, 00:12:40.92, 1.5677, 1
11/21/2014, 00:12:40.92, 1.5677, 1
11/21/2014, 00:12:40.92, 1.5677, 1
11/21/2014, 00:12:40.102, 1.5677, 3
11/21/2014, 00:12:40.176, 1.5677, 8

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: QM Export File Leading Zeros Error, Millisecond Timestam

Postby Henry MultiСharts » 21 Nov 2014

Hello eegroup,

We are currently analyzing this behavior.

eegroup
Posts: 78
Joined: 04 Aug 2008
Has thanked: 11 times
Been thanked: 9 times

Re: QM Export File Leading Zeros Error, Millisecond Timestam

Postby eegroup » 23 Nov 2014

Hi Henry,

Thank you for investigating and analyzing this issue.

Here is another example of a data error in the export file. In this case, we have a single digit millisecond timestamp. The 09:22:36.7 timestamp should be 09:22:36.007. If imported into another program, like Excel, it converts the timestamp to 09:22:36.700. The conversion is correct based upon how the data is exported from the Quote Manager. The data, however, is not correct as it is exported out of the Quote Manager.

Of interest, if we build a 1 tick chart of the data in the main MultiCharts program, and export the bar data directly from the workspace, the millisecond portion of the tick timestamp is exported correctly. This is a little impractical to accomplish as a workaround, however, when we have over 12 years of Future tick data on British Pound contract alone, which represent over 112 million tick records. MultiCharts has a little issue building a workspace with 112+ million bars on it. Not a practical solution to obtain the correct timestamps on all of the tick records.

If your analysis proves that there is an issue with the exported data from the Quote Manager, what would be your estimate of a fix/patch being made available? Can one be made available prior to when the next MC 9.0 r4 is released?

Thank you for your continue help and assistance with this issue. Much appreciated!

eegroup

From a file exported from the Quote Manager (tick file, trade data):

Date, Timestamp, Price, Quantity
6/13/2014, 09:22:35.970, 1.6942, 1
6/13/2014, 09:22:35.977, 1.6942, 1
6/13/2014, 09:22:35.979, 1.6942, 2
6/13/2014, 09:22:35.984, 1.6942, 1
6/13/2014, 09:22:36.7, 1.6942, 1
6/13/2014, 09:22:36.369, 1.6941, 1
6/13/2014, 09:22:36.369, 1.6941, 1
6/13/2014, 09:22:36.369, 1.6941, 1
6/13/2014, 09:22:36.369, 1.6941, 1
6/13/2014, 09:22:36.370, 1.6941, 1
6/13/2014, 09:22:36.370, 1.6941, 2
6/13/2014, 09:22:36.370, 1.6941, 2
6/13/2014, 09:22:36.370, 1.6941, 2
6/13/2014, 09:22:36.370, 1.6941, 1
6/13/2014, 09:22:36.370, 1.6941, 4
6/13/2014, 09:22:36.428, 1.6941, 2
6/13/2014, 09:22:37.157, 1.6941, 2

User avatar
bensat
Posts: 331
Joined: 04 Oct 2014
Has thanked: 46 times
Been thanked: 104 times

Re: QM Export File Leading Zeros Error, Millisecond Timestam

Postby bensat » 24 Nov 2014

.....

Of interest, if we build a 1 tick chart of the data in the main MultiCharts program, and export the bar data directly from the workspace, the millisecond portion of the tick timestamp is exported correctly. This is a little impractical to accomplish as a workaround, however, when we have over 12 years of Future tick data on British Pound contract alone, which represent over 112 million tick records. MultiCharts has a little issue building a workspace with 112+ million bars on it. Not a practical solution to obtain the correct timestamps on all of the tick records.

.....
As long as you don't rape Multicharts I think there is nothing wrong with that. You easily could separate the exports via separation in years, months and so on and obtain from there.
.....

Here is another example of a data error in the export file. In this case, we have a single digit millisecond timestamp. The 09:22:36.7 timestamp should be 09:22:36.007. If imported into another program, like Excel, it converts the timestamp to 09:22:36.700. The conversion is correct based upon how the data is exported from the Quote Manager. The data, however, is not correct as it is exported out of the Quote Manager.

.....
Are you saying the export of data in the Quote Manager via qmd-file is correct, but the export to a different 3rd party file format (in your case Excel-csv) with single or double digit millisecond time stamps is incorrect ? If so, please correct your entry post and make this more clear from there.

How do you exported these data to Excel ? Via csv-file ? Via dll-interaction ?

Regards.

Ben

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: QM Export File Leading Zeros Error, Millisecond Timestam

Postby Henry MultiСharts » 24 Nov 2014

eegroup, the issue has been confirmed. The fix is targeted to MultiCharts 9.0 Release 4.
I will get back to you if it is technically possible to provide a hotfix.

eegroup
Posts: 78
Joined: 04 Aug 2008
Has thanked: 11 times
Been thanked: 9 times

Re: QM Export File Leading Zeros Error, Millisecond Timestam

Postby eegroup » 25 Nov 2014

eegroup, the issue has been confirmed. The fix is targeted to MultiCharts 9.0 Release 4.
I will get back to you if it is technically possible to provide a hotfix.
Hi Henry,

Thank you very much for your confirmation and for the impending correction for this issue. Much appreciated!

eegroup

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: QM Export File Leading Zeros Error, Millisecond Timestam

Postby orion » 25 Nov 2014

eegroup, the issue has been confirmed. The fix is targeted to MultiCharts 9.0 Release 4.
I will get back to you if it is technically possible to provide a hotfix.
Henry, not quite related but something I would like to know: are these millisecond timestamps in QM based on timestamps in datafeed or local computer clock? Thanks.

eegroup
Posts: 78
Joined: 04 Aug 2008
Has thanked: 11 times
Been thanked: 9 times

Re: QM Export File Leading Zeros Error, Millisecond Timestam

Postby eegroup » 25 Nov 2014

Of interest, if we build a 1 tick chart of the data in the main MultiCharts program, and export the bar data directly from the workspace, the millisecond portion of the tick timestamp is exported correctly. This is a little impractical to accomplish as a workaround, however, when we have over 12 years of Future tick data on British Pound contract alone, which represent over 112 million tick records. MultiCharts has a little issue building a workspace with 112+ million bars on it. Not a practical solution to obtain the correct timestamps on all of the tick records.
As long as you don't rape Multicharts I think there is nothing wrong with that. You easily could separate the exports via separation in years, months and so on and obtain from there.
First, bad choice of words, Ben. I was not criticizing MultiCharts in any way. I was simply pointing out that using two different methods of exporting data, where one exported the millisecond timestamps correctly (the main MultiCharts program), the other did not (the Quote Manager). Second, the issue is that exporting tick data from a one tick chart in the main MultiCharts program is not the preferred or acceptable way of exporting large volumes of data. By using your suggestion, while it might work in theory by doing maybe a year at a time, building 12 separate workspaces to contain one tick bar data on each of them (representing the 12 years of Futures historical data), and then having to concatenate the 12 different files together is not quite the ultimate way do doing things, especially across 45 different Future symbols. One extract file created per symbol from the Quote Manager is the only practical way to do what we need to accomplish. I am also not sure that even a bar chart built with 10+ million single tick bars is that practical, even if the files could be built a year a time. Not really interested in going there, for the reasons mentioned above.
Are you saying the export of data in the Quote Manager via qmd-file is correct, but the export to a different 3rd party file format (in your case Excel-csv) with single or double digit millisecond time stamps is incorrect ? If so, please correct your entry post and make this more clear from there.
How do you exported these data to Excel ? Via csv-file ? Via dll-interaction ?
I have no idea if a qmd-file extract is formatted correct or not. We only export .csv files from the Quote Manager, so I have not checked that, nor do I have a reason to. Sorry I cannot be of more assistance in answering your question/concern. My original post was correct, in the context of what I was reporting. I thought it was very clear as initially presented

We really do not import the data into Excel. Excel was simply an example of how the milliseconds would not be imported properly the way the data was being formatted coming out of the Quote Manager. That example would have been true for any application in which the exported file would have been imported, and there are many applications which can accept .csv files as input besides Excel.

We hope that helps to further your understanding of our previous posts. Thank you for the opportunity to share these explanations with you.

eegroup

eegroup
Posts: 78
Joined: 04 Aug 2008
Has thanked: 11 times
Been thanked: 9 times

Re: QM Export File Leading Zeros Error, Millisecond Timestam

Postby eegroup » 25 Nov 2014

eegroup, the issue has been confirmed. The fix is targeted to MultiCharts 9.0 Release 4.
I will get back to you if it is technically possible to provide a hotfix.
Henry, not quite related but something I would like to know: are these millisecond timestamps in QM based on timestamps in datafeed or local computer clock? Thanks.
Hi Orion,

You will need to confirm this with Henry, but I believe based upon our past experience, that the timestamps in the MultiCharts database (and when the timestamps are subsequently used in the QM) are all based upon the Exchange time of the contract. This time can then be adjusted as to your specific needs (Local or Exchange), while accessing the data in a workspace in the main MultiCharts program, or when exporting the data from the Quote Manager. MultiCharts provides lots of flexibility regarding the basis of the timestamp time zone, so it really does not matter too much regarding how the data is store in the database.

Hope that helps...

eegroup

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: QM Export File Leading Zeros Error, Millisecond Timestam

Postby Henry MultiСharts » 25 Nov 2014

eegroup, the issue has been confirmed. The fix is targeted to MultiCharts 9.0 Release 4.
I will get back to you if it is technically possible to provide a hotfix.
Henry, not quite related but something I would like to know: are these millisecond timestamps in QM based on timestamps in datafeed or local computer clock? Thanks.
orion, it depends on the data provider you are using.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: QM Export File Leading Zeros Error, Millisecond Timestam  [SOLVED]

Postby Henry MultiСharts » 02 Dec 2014

This issue has been resolved in MultiCharts 9.0 Release 4 builds 10359/10360. You can download it here.


Return to “MultiCharts”