(yet another thread about) Complete Symbol Lists [UN-SOLVED]

Questions about MultiCharts and user contributed studies.
biffhero
Posts: 47
Joined: 22 Sep 2020
Has thanked: 29 times
Been thanked: 7 times

(yet another thread about) Complete Symbol Lists [UN-SOLVED]

Postby biffhero » 09 Nov 2020

I have read way too many threads on this forum about how MC "doesn't do symbols right", and I agree, having to add each symbol by hand is a real pain. I also realize that asking a company to keep the "symbol download, translation and import service" up-to-date to a professional level is too much to ask for.

That being said, I think that we users might be able to come up with something that is way better than what we have now, but not something that MC could officially support.

Here is how I am thinking about augmenting my list of symbols, but I don't want to do it if it is going to cause heartburn moving forward. I use Trade_Station as my data provider.

I currently use an external CSV file, and import it continuously as new symbols come up throughout the day. This is non-negotiable, as I require that there be a way to bring in the symbols in case of a computer error. I do not believe using QMD files is appropriate, as they are opaque blobs, without an open standard. If I am wrong, please let me know how to parse that file. I think I found a way tonight to come up with a more complete list than my current one-at-a-time approach.

1. Bring up the symbol importer, using ctrl-alt-t (I'm using Trade_Station)
2. Make sure I'm on the 'Stock' tab
3. Change the 'Country' to "United States"
4. Put 'A' in the "Symbol" field.
5. Click on "Lookup"
6. Wait for them all to be loaded.
7. Import them all.
8 - 32. Repeat this 25 times, one for each letter in the alphabet.
33. Back in the QuoteManager, click on "All Symbols"
34. Click on the top symbol in your QuoteManager
35. Press Ctrl-a to select all symbols.
36. Right-click, choose "Export Data, ASCII"
37. In "Select Resolutions and Fields, chose _only_ "Day"
38. Select "Export Symbol Name"
39. Select "Export Description"
40. In the field export box, deselect every field.
41. Click on "OK"
42. Select a directory that is empty. This is important, as this directory will be full of files. I use Desktop/exports/

There will be one file per symbol in this directory. The good news is that we get the symbol and the broker in the filename, so it shouldn't be hard to write a program that grabs those and builds the CSV file for us. One filename is as follows:
*AAAU-Trade_Station-NYSE-Stocks-Day-Trade.txt*

The second piece of good news is that the top two rows of one of the files looks like the following:

"Symbol","Description","TotalVolume"
AAAU,Perth Mint Physical Gold ETF,433293

It should be easy for us to parse that file and get the information we need.

The bad news is that we don't get that information unless the symbol has been loaded in the UI and downloaded some data.

98. (do some magical things here that I don't know how to fix right now)

Is this something that we can fix with EasyLanguage? I'm envisioning running through a list of all symbols available to the system, loading data for one day of data, and then moving forward. Of course, then I would do this once a month for a *blistering* fun Friday night. ;-)

Thanks,
Rob
Last edited by biffhero on 14 Nov 2020, edited 1 time in total.

User avatar
JoshM
Posts: 2195
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1544 times
Been thanked: 1565 times
Contact:

Re: (yet another thread about) Complete Symbol Lists

Postby JoshM » 09 Nov 2020

Is this something that we can fix with EasyLanguage?
PowerLanguage unfortunately doesn't have the capability to generate a symbol list.

With MultiCharts .NET you can query symbols from the QuoteManager. But this is only retrieving the symbols that are already in the database.

As far as I know, neither regular MultiCharts nor MultiCharts .NET have the capability to retrieve a list of symbols from a data feed provider.
I'm envisioning running through a list of all symbols available to the system, loading data for one day of data, and then moving forward. Of course, then I would do this once a month for a *blistering* fun Friday night.
Can the historical data downloader in MultiCharts 14 help in this situation? That would allow for an automatic download of data. (You'd still need to add every symbol by hand tough.)

Salzburg
Posts: 140
Joined: 19 Sep 2018
Has thanked: 9 times
Been thanked: 35 times

Re: (yet another thread about) Complete Symbol Lists

Postby Salzburg » 09 Nov 2020

Yes this is a pain for Stock traders... to maintain a list for example all SP500 stocks going forward is problematic in MC... but most of all, it does not support to be able to backtest without hindsight bias, ie its not possible in MC to decide to use or not use a symbol that at the time of the trade was a member if an index or not..

I know its not exactly the same as OP had an issue with, but its closely related, handling equities in MC is poor, we cant even make sublists in QM to separate type of stocks within a market..

This below is what we need to communicate with MC, a DYNAMIC list functionality.. utilising a data provider that actually maintains lists and data for use in MC..
https://norgatedata.com/ndu-overview.php

This has been mentioned several times and asked for, but been scuffed to the sideline with the usual political correct... please ad a feature request.... that never happens.. in this case data provider was willing to make this dynamic list functionality work with MC, but the thing died when MC demanded to much funds in order for them to allow connecting to MC api... makes no sense since its a solution of great benefit for all

Anyway, like and use MC daily for all my algo trading.., but please ad a functionality that makes sense for all MC users that is not only trading futures... does not have to be this solution ... but ANY solution would be great.

Cheers

biffhero
Posts: 47
Joined: 22 Sep 2020
Has thanked: 29 times
Been thanked: 7 times

Re: (yet another thread about) Complete Symbol Lists

Postby biffhero » 14 Nov 2020

Is this something that we can fix with EasyLanguage?
With MultiCharts .NET you can query symbols from the QuoteManager. But this is only retrieving the symbols that are already in the database.
I'm envisioning running through a list of all symbols available to the system, loading data for one day of data, and then moving forward. Of course, then I would do this once a month for a *blistering* fun Friday night.
Can the historical data downloader in MultiCharts 14 help in this situation? That would allow for an automatic download of data. (You'd still need to add every symbol by hand tough.)
1. query symbols from the QuoteManager.

That is nice, as that would allow me to do the "load one day of 1d data" step.

2. historical data downloader.

That's a good idea, as I could "add all symbols", and say to get very little data.

Thank you,
Rob

biffhero
Posts: 47
Joined: 22 Sep 2020
Has thanked: 29 times
Been thanked: 7 times

Re: (yet another thread about) Complete Symbol Lists

Postby biffhero » 14 Nov 2020


I know its not exactly the same as OP had an issue with, but its closely related, handling equities in MC is poor, we cant even make sublists in QM to separate type of stocks within a market..

This below is what we need to communicate with MC, a DYNAMIC list functionality.. utilising a data provider that actually maintains lists

This has been mentioned several times and asked for, but been scuffed to the sideline with the usual political correct... please ad a feature request.... that never happens.. in this case data provider was willing to make this dynamic list functionality work with MC, but the thing died when MC demanded to much funds in order for them to allow connecting to MC api... makes no sense since its a solution of great benefit for all

Anyway, like and use MC daily for all my algo trading.., but please ad a functionality that makes sense for all MC users that is not only trading futures... does not have to be this solution ... but ANY solution would be great.
1. Sublists. I have thought, while editing my instruments.csv file the other day, that it would be interesting to use different exchanges, that it might be interesting to put different instrumens into different exchanges, so that they show up in different parts of the QM. I'm not sure of the value in this, so I haven't done it yet.

2. A data provider that maintains lists to work with MultiCharts. I think it will be a very big problem to solve, if they try to solve it across the entire universe of symbols, past and present. I think if there was an API we could use to communicate with MultiCharts, then we would have a way to write connectors to do what we need.

It is obvious that MC has code to connect to TS and pull the list of symbols. I presume they can do this with many different providers, some which search with "*", some which do not. If we had access to be able to do that from the Operating System, we could move forward on our own.

Even if it were something as simple as the little helper applications that some data providers hand out as proof of concepts for connecting to their API, we would have something to go on.

Thank you,
Rob

biffhero
Posts: 47
Joined: 22 Sep 2020
Has thanked: 29 times
Been thanked: 7 times

Re: (yet another thread about) Complete Symbol Lists

Postby biffhero » 14 Nov 2020

I have come to the conclusion that this is a *bad idea* (tm), and I am no longer actively pursuing this solution.

I did my steps for the letters a, b, c, and did not ever get to the part where I created a huge instruments.csv file.

From that point onward, whenever I used the chart user interface to type in a symbol, it would take way too long. It completely rendered the feature unusable.

Until MultiCharts comes up with a faster way to build that dropdown (along with possibly setting up the top section with 'recent' symbols), then this is not something that should be done in the MultiCharts user interface.

Thank you,
Rob
Last edited by biffhero on 15 Nov 2020, edited 1 time in total.

bomberone1
Posts: 310
Joined: 02 Nov 2010
Has thanked: 26 times
Been thanked: 23 times

Re: (yet another thread about) Complete Symbol Lists [UN-SOLVED]

Postby bomberone1 » 14 Nov 2020

TS automatically updates the indices when components change and also allows you to create custom lists. It would take an identical function in the multicharts quotemanger that is unrelated to a specific data provider or api. The auto-update function of the components of an index and its components should be native to the multicharts quotemanger. The quotemanger should contain the list of all world indices and automatically change as the various baskets change.

Salzburg
Posts: 140
Joined: 19 Sep 2018
Has thanked: 9 times
Been thanked: 35 times

Re: (yet another thread about) Complete Symbol Lists

Postby Salzburg » 15 Nov 2020



Thank you,
Rob

Hi

Its this provder i am referring to and they provide DYNAMIC index constituents lists that MC could utilise if they wanted to.
https://norgatedata.com/ndu-overview.php

They provide an awesome api, that maintains dynamic lists, newly added and all all delisted stocks including providing markers in the list for the dates if a specifc symbol was a member of a specific index or not... so with this you can select test your strategy X on SP500 index or Nasdaq 100 stocks from 2000 to Today.. and you then are able to allow trades only on symbols that was a member of the index AT THE TIME of the trade... ¨
If you dont have a database that supplies delisted symbols , and markers like 0 or 1 for symbols if they was a member of the index per date.

If MC would you to test select the whole index for example sp500, so instead of having the hassle to input or import csv files, to portfolio trader.. and you could off course deselect symbols of you wanted to etc, but this index symbols list is maintained by the API on an ongoing basis.

As of today its more or less impossible to backtest equities in MC, survivor biased free.
And yes this can all be solved with other software and maintaining this database structure on my own, but i am after to handle it in MC.

Cheers

bomberone1
Posts: 310
Joined: 02 Nov 2010
Has thanked: 26 times
Been thanked: 23 times

Re: (yet another thread about) Complete Symbol Lists [UN-SOLVED]

Postby bomberone1 » 15 Nov 2020

Mc should provide DYNAMIC index constituents lists witound inside quotemanger from their server and to avoided the linking of a specific datafeed or api.

biffhero
Posts: 47
Joined: 22 Sep 2020
Has thanked: 29 times
Been thanked: 7 times

Re: (yet another thread about) Complete Symbol Lists [UN-SOLVED]

Postby biffhero » 17 Nov 2020

I agree that this is a thing that could be made more robust. However, I don't think that it should stop there.

I think that they should keep a production level backend that provides the symbols at each broker up-to-date.

This would be a contractual go live gate with each broker that there would be an API set up for the consumption by our individual QMs. I presume this would be run by the TradingView production team, and the symbols that we would use on our platform would be the TradingView symbols, with seamless mapping to each broker on the backend.

I don't see symbol queries as having that much data requirement. I think a bigger deal would be designing and implementing a forward-compatible system so that the QM could not only be tied to brokers (legacy), but also un-tied to brokers (new way), in such a way that we would see the new symbols as appropriate.

And of course, being able to see the old symbols for old data, too. That's going to be an interesting problem to solve, from a data provider point of view. It would be simpler to aggregate and provide at the MC side. Ugh. I'm glad I don't have to do it. ;-)

Have fun,
Rob

User avatar
Mark Brown
Posts: 181
Joined: 29 Nov 2016
Has thanked: 111 times
Been thanked: 17 times

Re: (yet another thread about) Complete Symbol Lists [UN-SOLVED]

Postby Mark Brown » 05 Dec 2020



Return to “MultiCharts”