Scanner seems too slow.  [SOLVED]

Questions about MultiCharts and user contributed studies.
bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Scanner seems too slow.

Postby bowlesj3 » 30 Jul 2017

Hi,
Update:
I found a great solution. See 2nd last and last posts.
I have about 82 stocks to scan in the scanner. It takes just over 4 minutes to complete the scan. I am testing the speed of the scanner on-line but when the market is closed. I will test it again today on-line but with live data coming in. I have 1 study only. The resolution is 5 minute bars. The study is tiny. It is %R with a few small modifications (the %R is going back only 14 bars). Basically for each stock it processes it grabs 4 GVs which I set in another program, it totals them, does a few comparisons to set my waitlist indicator plot and plots 6 other items. I apply this study to both pre-scanning and watchlist. In the next paragraph I describe what I do to test it. I have some questions at the end.

I uncheck the checkbox that causes the scanner to scan every minute. I stop the scanner. I start the scanner with a stopwatch ready and I start the stopwatch. It shows me this message "Scanning... Left scan ## tickers RT slots for "Interactive Brokers": Total 20; Available 0.". When I start it the ## part runs through a few numbers quickly then gets stuck on value 81, 61, 41, 21, 1 and it finishes. The good news is it works :-) in that it puts the correct entries in the watchlist.

Q: What does the bold+italic message mean (in detail)?

Q: I have to assume it is running slow because of the every 1 minute default checkbox which implies that it should complete in less than 1 minute. Am I correct?

Q: I am not sure just how valuable it is running at this speed. Is there any way to speed it up or at least figure out why it might be going so slow so I can speed it up. I guess I could write out a fileappend statement to see when it starts each new stock to see if it is taking a lot longer on some than others.

Q: How does this speed compare to what other's are experiencing?

Thanks,
John
Last edited by bowlesj3 on 01 Aug 2017, edited 1 time in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Scanner seems too slow.

Postby bowlesj3 » 31 Jul 2017

Running the "rescan now" seems to be running faster when live data is coming in. I would say it has dropped from the 4 minutes down to around 1 minute (sometimes as low as maybe 30 seconds and sometimes over a minute). The scanner seems to run fast at the start then as it gets near the end it takes longer and longer to process the last 5 or so symbols. I tried putting the file append in. It works okay for doing a scan outside market hours to determine how long it is taking. However it does not work so good when live data is coming in because the script is getting updated even though the rescan is not running. It is interesting that the rescan runs about 4 times as slow outside market hours. So If this is normal behavior it seems that setting the automatic rescan rate to higher than 1 minute would be required since maybe the program does not detect that a rescan is currently running and maybe it will start a new rescan before the first one is even finished.

I thought recompiling the study was making it go very fast but then I noticed that it was not loading the watchlist properly. It is just replacing whatever was already on the screen once the compile is finished (not restanning).

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Scanner seems too slow.

Postby bowlesj3 » 31 Jul 2017

So my concern was correct. I decided to try turning on the checkbox to have the scanner run every 1 minute. So I noticed that the message changes to also show the amount of time left before the next scan starts. I noticed that the current scan had 1 symbol left to process. The processing of the symbols gets slower and slower as less and less are left to process. So one symbol was left to process while the time left before the next scan is started was counting down very close to being ready to start the next scan. So it happened. The next scan started without the prior scan finishing and the watchlist never got updated. So there are two options. Set the auto restart rate to 5 minutes or start the scans manually. In my case it would be better to have 2 minutes unless these scans can consistently complete in less than 1 minute.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Scanner seems too slow.

Postby bowlesj3 » 31 Jul 2017

Nothing has changed in the above messages. However I have some interesting stuff to add.

Now that the markets are closed I decided to run the scanner off line again (IB's TWS is down). Once again if I click the "Rescan Button" the scanner takes about 4 minutes to run while off line. I tightened my code to try and speed this up by moving all the new code I inserted into the %R to the last bar on chart so they only run once. That includes getting the 4 GV data items, calculating the extra plots and putting them out with the plot statement. The %R needed to be done as normal so that the overbought and oversold colours could be set. This change had no impact on the speed at which the scanner ran.

I tried this test this morning and then again tonight. I 100% confirmed it. If I recompile the study the scanner produces the results almost instantly. To be 100% sure I did two things. First I cleared the watch list and I did the recompile and it gives the same results. Second I used my external program to change one of the GVs such that one entry should be missing and this occurred exactly as it should have. Maybe the MC support team can answer the question as to why recompiling speeds the scanner up so much. Tomorrow while the live feed is coming into MultiCharts I will try this test. If it works I have a way to really speed up the scanner which isn't that much harder to run.

I inserted my append results which are written during the last bar on the chart part of the study. It shows how the scanner inserts a delay of 1 minute 4 times during the scan. If recompiling during the day speeds it up hopefully they don't change it so recompiling slows it down :-)
Attachments
A_My_%R_WithShiftScores.txt
(9.81 KiB) Downloaded 412 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Scanner seems too slow.  [SOLVED]

Postby bowlesj3 » 01 Aug 2017

As expected the recompile works during live trading. So I guess that solves it.


Return to “MultiCharts”