Many of us create requests for one feature or another because we have in mind some goal but often don't state that goal.
Rather than ask for specific features, I would like to ask TSS to recommend how we should solve the problem of implementing some custom processing on historical data using MC.
Let's use an example of a continuous futures contract. I want to use the current live data to extend the price series forward but I want to use my own price series up to some point in time, say the most recent weekend or beginning of the month. How should I go about this with MC?
Recommend how to implement custom history data processing
- Marina Pashkova
- Posts: 2758
- Joined: 27 Jul 2007
I want to do some arbitrary computation on the past values (say last month and earlier) but use the real-time data for recent values (this month and going forward).
I'm asking for a recommendation to accomplish this.
Importing and overwriting the stored historical data is one idea.
Creating a merge is another.
IMPORTING:
If you import the data for a symbol for values up until the previous month, will the recent data for this month be retained or lost?
Can the import be done while real-time data is being collected?
This would be much simplified if you could provide a simple tool that can be run from the command line to export and import trade data. Something like:
quotemanager -export -symbol ZB -resolution minute -field trade -exchange cbot -source TS -timezone exchange -starttime "11/8/2007 8:00am" -endtime "11/9/2007 8:00am"
and
quotemanager -import -symbol ZB -resolution minute -field trade -exchange cbot -source TS -timezone exchange -starttime "11/8/2007 8:00am" -endtime "11/9/2007 8:00am"
This ought to be fairly easy to implement, I think, and would greatly help dealing with data stored in the database.
It would then be possible to create a script that exports a list of symbols, runs some process, and then imports the result.
MERGING:
My understanding from another thread is that merging won't solve this problem because the real-time data is used only from the time I start MC. Thus I would have gaps in my data for this month for any time that MC wasn't running while the market was producing data.
I'm asking for a recommendation to accomplish this.
Importing and overwriting the stored historical data is one idea.
Creating a merge is another.
IMPORTING:
If you import the data for a symbol for values up until the previous month, will the recent data for this month be retained or lost?
Can the import be done while real-time data is being collected?
This would be much simplified if you could provide a simple tool that can be run from the command line to export and import trade data. Something like:
quotemanager -export -symbol ZB -resolution minute -field trade -exchange cbot -source TS -timezone exchange -starttime "11/8/2007 8:00am" -endtime "11/9/2007 8:00am"
and
quotemanager -import -symbol ZB -resolution minute -field trade -exchange cbot -source TS -timezone exchange -starttime "11/8/2007 8:00am" -endtime "11/9/2007 8:00am"
This ought to be fairly easy to implement, I think, and would greatly help dealing with data stored in the database.
It would then be possible to create a script that exports a list of symbols, runs some process, and then imports the result.
MERGING:
My understanding from another thread is that merging won't solve this problem because the real-time data is used only from the time I start MC. Thus I would have gaps in my data for this month for any time that MC wasn't running while the market was producing data.
- Marina Pashkova
- Posts: 2758
- Joined: 27 Jul 2007
Dear jek,
Below are the answers to your questions about data import:
And thank you for your suggestions. We will consider the feature you are talking about.
Below are the answers to your questions about data import:
The recent data should be still there.If you import the data for a symbol for values up until the previous month, will the recent data for this month be retained or lost?
Yes.Can the import be done while real-time data is being collected?
And thank you for your suggestions. We will consider the feature you are talking about.
Excellent. Please PM me if you need any help with testing or discussing command options.And thank you for your suggestions. We will consider the feature you are talking about.
Another useful command line option would dump the list of symbols stored in the database with their sources/exchanges/dates covered/etc such as
quotemanager -listsymbols
- Marina Pashkova
- Posts: 2758
- Joined: 27 Jul 2007
Please reconsider this feature.In the nearest future we are not planning to implement export/import through the command line. As a solution I could suggest using a Macro program.
It would be very very useful to list symbols and export/import data.
Just imagine how hard it would be to write some macro program to list the symbols in your database. How would you pick them out of the lists? Very hard.
Also, macro programs are so very fragile. Windows decides to pop up some "Peform system upgrade" dialog on you and everything gets messed up.
And trying to figure out reliably when an operation is done can be really difficult for some things.
Please. Anyone else out there feel this need?
Since today is JEK's birthday I request the command-line market data export/import feature and an API rather than a command line interface even if it just for such high end individuals as JEK.
I think this would be a fantastic birthday present and even Christmas present for him.
Here here JEK.
HAPPY BIRTHDAY!!!! from the Prof.
I think this would be a fantastic birthday present and even Christmas present for him.
Here here JEK.
HAPPY BIRTHDAY!!!! from the Prof.
Thanks for the support.
With Windows, the feared response to a problem is "reinstall the operating system and all the applications".
With MC, the feared response to a problem is "export all your symbols and re-import them into a fresh database".
Both are enormous tasks.
Glitches in data do happen, often beyond the ability of MC to prevent, such as network or computer crashes. We need the ability to fix things when they go awry.
With Windows, the feared response to a problem is "reinstall the operating system and all the applications".
With MC, the feared response to a problem is "export all your symbols and re-import them into a fresh database".
Both are enormous tasks.
Glitches in data do happen, often beyond the ability of MC to prevent, such as network or computer crashes. We need the ability to fix things when they go awry.
Any tools to help us deal with large amounts of data, either modifying or importing or exporting would be tremendously helpful, currently everythingSince today is JEK's birthday I request the command-line market data export/import feature and an API rather than a command line interface even if it just for such high end individuals as JEK.
I think this would be a fantastic birthday present and even Christmas present for him.
Here here JEK.
HAPPY BIRTHDAY!!!! from the Prof.
is a daunting task.....