MC12 and researching stock data, implications

So MC12 is great for research and strategy execution of futures, however when it regards equities in portfolio, i have not found a great way yet to go about it.

Research of equities requires taking into account usually whole sectors, specific exchanges etc, and also being able to look at current and past constituents to remove survivorship bias etc.

How can one solve this in MC12? if i for example want to use NORGATE data, and have possibility to feed different databses to mc for research how can one actually practically solve this? In quote manager one cant handle external dynamic databases from example Norgate api, and i dont know a realistic way of keeping different databases in MC for being able to look at current and past constituents etc..

Anyone has been able to configure or define in quotemanager different databases for stocks for this purpose?

If MC12 could work with dynamic external databases from supplier as Norgate api, it would solve a lot for equities traders.
Perhaps i am missing something.

Regards

I do this kind of analysis but not within Multicharts. It is not really the right tool for the job. Take a look at Rapidminer and MariaDB as a database. Anything missing can be filled in with Python or R. There are very few limitations. That is how I do it.

Kind regards,

Alex

I do this kind of analysis but not within Multicharts. It is not really the right tool for the job. Take a look at Rapidminer and MariaDB as a database. Anything missing can be filled in with Python or R. There are very few limitations. That is how I do it.

Kind regards,

Alex

Hi Alex, thank you for your feedback.

So as i sunderstand this is your setup for gathering and structuring data, which is helpful. Not sure if you mean that this is how you maintain data but test on another software than MC?


However biggest problem as i see it, is accessing it from MC, and updating it going ahead and testing different versions of if..
For example making sure that you keep a portfolio to test on and potentially live trade on, that contains all sp500 stocks, ie adding new ones, and replacing some.

And then being able to test on adjusted data, testing on total return series not only adjusted for splits and corp actions, but also dividends.

And best question would be, how to handle in MC that one will making sure to only test on sp 500 symbols making sure that symbol was actually a member of the index when the trade happened.

Is this even possible in MC? i am not sure.
Maintaining equity data like this is very difficult and only option is manually updating it, which is not even possible when you are dealing with a large index.


Any help is appreciated

Hi Salzburg,

I use Portfolio Trader on the top 150 SP500 stocks. I do however download unadjusted data, split it on the fly and build selections outside of Multicharts. These get written to text files that Portfolio Trader then imports and rotates daily. Data and trade selections get stored in MariaDB. Making minimal changes to the symbol list in PT is the way to go as it is a slow process to add a symbol, set the session template and then do the symbol mapping. It does not lend itself to creating portfolios on the fly quickly. Having a master list and controlling the selections with text files works really well for me. This way you can use your external database to track SP500 members, dividends etc. It does take time to setup but it is not impossible. One Python script does everything. Takes just a few minutes every morning.

regards,

Alex

Hi Alex, Many thanks for providing some insight.
Ok then i understand, you actually input all sp500 stocks in portfolio trader, then you allow or do not allow specific symbols, and this information is from a text file locally on your trading server this is working without having to stop and start the portfolio,, and this text file in changed by a Python script that uses information that you manually handle externally in MariaDB.... kindly confirm so i have understood correctly.

You mention that you use unadjusted data, but you split and adjust it on the fly, how would you go about that ? you mean you manually or by a script change the datafile for each symbol?

So you also use this text file methodology to define index constituents during backtests since the text file will contain dates when the symbol was a member of the index or not ? somehow one needs to make sure we test only trades from symbols that during when the trade actually happened was actually a member of the index.. I have not found a way to solve this with MC yet...


Many thanks Alex for sharing

Yes Salzburg, the symbol selection is managed with text files. At the open of the next trading day, the strategy reads the file and then submits the orders. This is all automatic. I download unadjusted data and split adjust it using a python script. I keep unadjusted, split adjusted and dividend + split adjusted datasets. The data is stored to a database and the text files written. This is all done with some python.

Basically a symbol stays trading until it is dropped from the text file. You can manage changes to the index the same way.

regards,

Alex

Yes Salzburg, the symbol selection is managed with text files. At the open of the next trading day, the strategy reads the file and then submits the orders. This is all automatic. I download unadjusted data and split adjust it using a python script. I keep unadjusted, split adjusted and dividend + split adjusted datasets. The data is stored to a database and the text files written. This is all done with some python.

Basically a symbol stays trading until it is dropped from the text file. You can manage changes to the index the same way.

regards,

Alex

Alex very much, thank you, gave me something to think about..

One thing though that confused me, i get that you maintain your own database using MariaDB, you here keep track of when a certain symbol was included in a index, financials etc... and this controls on a daily basis which symbols to allow... but what i dont get is, database that contains symbols for MC normally, is maintained by MC? and is stored locally arriving from dataprovider.... unless your are somehow your own dataprovider to MC?

Your own MariaDB database where you store your tape data, and info, splits etc...
How are you utilising this data for your backtests in MC? and how are you doing to include and exclude symbols for trading during a certain time in the backtest when they was not part of the chosen index? and how are you able to switch from adjusted to not adjusted etc, for MC backtests?

Again, thank you for your very kind help and being open on how you have solved these things...

I have thought about maintaining my own database, however i have not figured out how to utilise it in MC, since using ASCII is not really optimal ..

Best Regards

Hi Salzburg, just to be clear, I also have IQFeed for equities which actually drives Multicharts. The downloaded data is for the symbol selection. That data never gets imported into Multicharts. IQFeed automatically split adjusts their data. Over time, I had some issues going back to a particular day and recreating the exact conditions the symbol selection was made. The way I do it now is to make sure that everything is 100 percent reproducible. The database allows me to go back to any day in the past to see the exact data I used independently of the Multicharts database.

I see Alex, thank you. Then its more clear.

So you have a clever way of selecting from all symbols used in portfolio trader, what symbols to tactually trade on in the portfolio day to day, based on your own SQL MariaDB.

The below two issues i am still scratching my head trying to solve ..

Have you managed to solve the problem, to being able to backtest on specific index symbols that actually was a member of a particular index at the trade date, during a backtest to avoid biassed backtest ?
I know other solutions uses a indicator to allow or not allow a symbol by using a 0 or 1 depending on what dates the actual symbol was a member of a certain index or not.

And have you been able to solve our problem that you want to include delisted stocks also in your tests, and not only stocks that survived to the point when you did the backtest ?

Many thanks again

Hi Salzburg, if you manage the selection day by day in text files you can handle de-listings quite easily. Recently I had to drop CELG but I have data for CELG until it stopped trading so it is still part of the back tests even though it does not currently trade. As long as you can bring in the delisted trade data into PT you are fine. I am not sure how long IQFeed will maintain that data. I will have to check. I guess I could just import it and not update it. I am sure that there are ways to make it work.

regards,

Alex

Contact me via PM if you would like any more details.