Recommend how to implement custom history data processing

Questions about MultiCharts and user contributed studies.
jek
Posts: 163
Joined: 24 Dec 2006
Been thanked: 2 times

Recommend how to implement custom history data processing

Postby jek » 03 Nov 2007

Many of us create requests for one feature or another because we have in mind some goal but often don't state that goal.

Rather than ask for specific features, I would like to ask TSS to recommend how we should solve the problem of implementing some custom processing on historical data using MC.

Let's use an example of a continuous futures contract. I want to use the current live data to extend the price series forward but I want to use my own price series up to some point in time, say the most recent weekend or beginning of the month. How should I go about this with MC?

jek
Posts: 163
Joined: 24 Dec 2006
Been thanked: 2 times

Postby jek » 03 Nov 2007

At the risk of making a suggestion, a really valuable addition would be a market data API that we can plug into that can both read and write the database as well as hook in as a filter.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 08 Nov 2007

Hi jek,

Are you talking about adjusting data or simply merging it?

If you simply want to merge data together, this can be done through data import.

If you would like to adjust the data, this feature will be available next year.

Regards.

jek
Posts: 163
Joined: 24 Dec 2006
Been thanked: 2 times

Postby jek » 08 Nov 2007

I want to do some arbitrary computation on the past values (say last month and earlier) but use the real-time data for recent values (this month and going forward).

I'm asking for a recommendation to accomplish this.

Importing and overwriting the stored historical data is one idea.

Creating a merge is another.

IMPORTING:

If you import the data for a symbol for values up until the previous month, will the recent data for this month be retained or lost?

Can the import be done while real-time data is being collected?

This would be much simplified if you could provide a simple tool that can be run from the command line to export and import trade data. Something like:

quotemanager -export -symbol ZB -resolution minute -field trade -exchange cbot -source TS -timezone exchange -starttime "11/8/2007 8:00am" -endtime "11/9/2007 8:00am"

and

quotemanager -import -symbol ZB -resolution minute -field trade -exchange cbot -source TS -timezone exchange -starttime "11/8/2007 8:00am" -endtime "11/9/2007 8:00am"

This ought to be fairly easy to implement, I think, and would greatly help dealing with data stored in the database.

It would then be possible to create a script that exports a list of symbols, runs some process, and then imports the result.


MERGING:

My understanding from another thread is that merging won't solve this problem because the real-time data is used only from the time I start MC. Thus I would have gaps in my data for this month for any time that MC wasn't running while the market was producing data.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 14 Nov 2007

Dear jek,

Below are the answers to your questions about data import:

If you import the data for a symbol for values up until the previous month, will the recent data for this month be retained or lost?


The recent data should be still there.


Can the import be done while real-time data is being collected?


Yes.

And thank you for your suggestions. We will consider the feature you are talking about.

jek
Posts: 163
Joined: 24 Dec 2006
Been thanked: 2 times

Postby jek » 14 Nov 2007

And thank you for your suggestions. We will consider the feature you are talking about.


Excellent. Please PM me if you need any help with testing or discussing command options.


Another useful command line option would dump the list of symbols stored in the database with their sources/exchanges/dates covered/etc such as

quotemanager -listsymbols

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 15 Nov 2007

Hi jek,

In the nearest future we are not planning to implement export/import through the command line. As a solution I could suggest using a Macro program.

Best regards.

jek
Posts: 163
Joined: 24 Dec 2006
Been thanked: 2 times

Postby jek » 28 Oct 2008

In the nearest future we are not planning to implement export/import through the command line. As a solution I could suggest using a Macro program.


Please reconsider this feature.

It would be very very useful to list symbols and export/import data.

Just imagine how hard it would be to write some macro program to list the symbols in your database. How would you pick them out of the lists? Very hard.

Also, macro programs are so very fragile. Windows decides to pop up some "Peform system upgrade" dialog on you and everything gets messed up.

And trying to figure out reliably when an operation is done can be really difficult for some things.

Please. Anyone else out there feel this need?

proflogic
Posts: 12
Joined: 20 Dec 2006

Postby proflogic » 28 Nov 2008

Since today is JEK's birthday I request the command-line market data export/import feature and an API rather than a command line interface even if it just for such high end individuals as JEK.

I think this would be a fantastic birthday present and even Christmas present for him.

Here here JEK.

HAPPY BIRTHDAY!!!! from the Prof.

ed roche
Posts: 11
Joined: 01 Apr 2008

Postby ed roche » 30 Nov 2008

At the risk of making a suggestion, a really valuable addition would be a market data API that we can plug into that can both read and write the database as well as hook in as a filter.

jek
Posts: 163
Joined: 24 Dec 2006
Been thanked: 2 times

Postby jek » 30 Nov 2008

Thanks for the support.

With Windows, the feared response to a problem is "reinstall the operating system and all the applications".

With MC, the feared response to a problem is "export all your symbols and re-import them into a fresh database".

Both are enormous tasks.

Glitches in data do happen, often beyond the ability of MC to prevent, such as network or computer crashes. We need the ability to fix things when they go awry.

ed roche
Posts: 11
Joined: 01 Apr 2008

Postby ed roche » 30 Nov 2008

Since today is JEK's birthday I request the command-line market data export/import feature and an API rather than a command line interface even if it just for such high end individuals as JEK.

I think this would be a fantastic birthday present and even Christmas present for him.

Here here JEK.

HAPPY BIRTHDAY!!!! from the Prof.


Any tools to help us deal with large amounts of data, either modifying or importing or exporting would be tremendously helpful, currently everything
is a daunting task.....


Return to “MultiCharts”