notes on Global Variable

Studies that have been contributed to the community by other users. If you’ve got something useful to share, that’s great!
bowlesj3
Posts: 1980
Joined: 21 Jul 2007
Has thanked: 156 times
Been thanked: 395 times

notes on Global Variable

Postby bowlesj3 » 26 Feb 2008

This is a little long so here is a bit of organization to help you decide if it is worth reading.

1a/ Short Background history leading to work to test GV speed.
1b/ Background Specifics.
2/ A method to test the Named GV speed.
3/ Questions about how the Named GVs work and what can be done to make it faster.
4/ What about numbered GVs instead of named GVs.

1a/ General Background:
I am a discretionary trader. I do not attempt to program a full trading system and thus use MC primarily as a work load reduction tool. I am an application programmer, I use MS-Access VBA as part of my overall trading setup. I use MS-Access as a control software to control MC and also IB’s Traders Work Station. I also obviously get data from MC via named global variables for this process. After 7 months of working with MC and MS-access I have my trading system software basically done and I have finally started to notice enough delay in getting data from MC to MS-Access through Named GV methods that I am having to now both look for ways to speed up the EL code, reduce unneeded EL code, and also ask myself if any new code will just cause more slowdown with little or no value in return which contributes to increased income.

1b/ Background Specifics:
My EL system has to count ticks and feed these counts to MS-Access so it knows that certain data is available for the “trigger wait mode” portion of my trading cycle. MS-Access has a timer function in every form. In one of my forms I use the timer to go after the tick count that the EL code puts out. It does this every second. If a parameter setting of 10 seconds passes and it does not get a tick count MS-Access gives me a popup warning. This warning happens a lot with my current level of system code and 10 seconds is a lot. I also see the last prices changing in the MC chart but I do not get the ticks in MS-Access even though I see them.

2/ Method to test:
Okay so there is a problem. However before I go looking for code to get rid of or make more efficient I need a good way to measure if these changes are working. So I plan on writing out a print statement in the EL code at the exact same point that I put the tick count out to the GV. I put the current time on this print statement along with the actual current tick count. In MS-Access I use the GV_GetNamedInt command to get the tick and I write out a log record exactly when I get it with the tick count and the current time. After running for a while I run in MS-Access a matching record batch process that is sorted by tick count and when the two logs (the EL print statements and the MS-Access log writes) get a matching tick number it does a subtraction of the current times on each to get the number of seconds or milli second difference. At the end of the batch run I have the average, max and min delays giving me a feel for how bad is it. Now I can start putting in changes and see how much each change helps. I can also run this test on a stripped down single chart workspace to see what the best possible result could be. I have a list of 10+ possible improvements to my EL code to make and I can measure each improvement and the total improvement.

3/ Questions about how the GVs work:
To the best of my knowledge the named GV calls should be executing code to do a binary search to get the info. This would seem to be the fastest method to get the data out of the named tables. In order to do this the named items must be sorted. The fastest sort for tables of 200+ elements is to the best of my knowledge what is called a “comb sort”. I think it is actually faster than the shell sort. So the question is “do new elements get loaded in and sorted with a comb sort or maybe an even better one”. Maybe even faster, are they loaded in with a binary search to find the proper position with a subsequent mass calculated move that shifts everything down using an assembler type low level move statement. Anyway, what I currently do, just in case it is doing a binary search, is I preload all my GVs at the very start of the trading day in sorted sequence. I use MS-Access for that. This way the searches should be faster and the dll code should not have to do any insertions during trading time.

4/ What about numbered GVs instead of named GVs.
I am thinking that numbered GVs may be a much faster approach since they have a direct reference not needing any form of search and I could use MS-Access to keep all this organized as new ones come in with future programming/trading enhancements. Maybe that is the best approach now that my trading systems are basically done and I am not likely to make too many changes to it (no major ones).


Thanks for any comments or suggestions.
John.
Last edited by bowlesj3 on 06 Mar 2008, edited 9 times in total.

bowlesj3
Posts: 1980
Joined: 21 Jul 2007
Has thanked: 156 times
Been thanked: 395 times

Thoughts on production testing for the GV delay.

Postby bowlesj3 » 27 Feb 2008

Hi,

It occurred to me that while waiting for a trade (especially during fast markets) I need to have almost immediate updates/warnings of excess GV delay across apps (MC and MS-Access). I am thinking that I could bundle last-price & current-time & tick-cnt into a single field (single GV field) to be shipped out to be picked up by MS-Access which would unbundle it to test the time it was shipped and compute the delay to decide if a warning need be given. Okay so now I need to figure out two things. 1/ an algorithm for this and 2/ figure out how to get this output of the GV to be the first thing that MC does after it updates the last price on the chart (I suspect that is a very high priority with MC since it appears to keep up with Traders Work Station but the study info gets done later). I don't suppose anyone knows how to do #2 or even better has done it already.

bowlesj3
Posts: 1980
Joined: 21 Jul 2007
Has thanked: 156 times
Been thanked: 395 times

I guess I should have posted this one as a reply.

Postby bowlesj3 » 27 Feb 2008

Later yesterday I went to a few websites to check to look up the names and purpose of all processes running in my computer to ensure that no viruses were running (there are a few good sites actually for this - see list below if interested). All looks well there. I ran avg spyware and removed about 15 items and will likely run that each morning before trading. So tomorrow I figure the GV delay problem can only be blamed on the EL coding or the DLL code. I ran a defrag and disk cleanup. Maybe this will help.

Websites where one can get info on the processes running in their computer.
www.processlibrary.com
www.onecomputerguy.com
www.ppedia.com
www.softwarepatch.com
Maybe someone knows of other good sites.

bowlesj3
Posts: 1980
Joined: 21 Jul 2007
Has thanked: 156 times
Been thanked: 395 times

Here is the solution.

Postby bowlesj3 » 28 Feb 2008

I guess no one who has read this has a solution to the GV slow down or info on how to optimize it.

I came up with something. It is actually faster than MC can ever be but uses what it can from MC and uses MS-Access to do the rest. It involves putting out the Interactive Brokers bracket order in advance (ACTIVE) so no matter how fast prices are moving the order is there ready and waiting (can't get faster than that). MS-Access picks up the entry point information from MC's EL code. This information moves fairly slowly (changes fairly slowly) even though prices may be going very fast in a fast market. So MS-Access feeds this info to Traders Work Station and makes the adjustments to the bracket order as needed (this occurs very fast - way faster than any human can do it and it never makes an error). If I choose to change my entry point because prices are stalling ahead of my original entry point, I enter the new entry point information through MS-Access and both TWS and MC are updated as needed. Therefore there is no double entry which greatly increase the chances of human error. Actually in my MS-Access program it is a simple button click. So the slowest part is the GV transfer, the next slowest is MC. I will still review my code to make the other approach as fast as possible but this new approach reduces the need to do this review significantly.

Thats it. Problem solved.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 29 Feb 2008

Great job! It should help others to avoid icebergs of GV!

bowlesj3
Posts: 1980
Joined: 21 Jul 2007
Has thanked: 156 times
Been thanked: 395 times

Postby bowlesj3 » 29 Feb 2008

Thanks Andrew. I think there is a niche for this type of programming but it is highly custom. Either the trader has to be a good programmer or rich enough to pay to have it done. It is tricky too since the IB TWS bracket order once filled inverts itself with each update. I am doing it by sendkeys since I think taking the time to get the API will make it impractical. To make it commercial would require the API approach of course.

Regarding the GV slow down problem (and how to optimize this particular DLL call) a web search seems to pull up some info but I have not had a chance to find anything relevant. Maybe TS-Support could put this on their list "How to optimize GV usage". Yah, I know. As if you don't have enough to do right! I would think your team is really busy to say the least.

bowlesj3
Posts: 1980
Joined: 21 Jul 2007
Has thanked: 156 times
Been thanked: 395 times

I think it is MC (or my overloading it)

Postby bowlesj3 » 29 Feb 2008

Hi, I think the slowdown is MC rather than the GVs. Various plots, lines, etc, are really slow in being updated on the chart so this would have nothing to do with the GVs. I have to implement my strategy above "Method To Test" to try and figure out where it is getting bogged down. A week or two of work.

bowlesj3
Posts: 1980
Joined: 21 Jul 2007
Has thanked: 156 times
Been thanked: 395 times

Send data as soon as possible from all charts.

Postby bowlesj3 » 02 Mar 2008

I think I will send out current price from each chart (at the very start before all other code) and bring them into MS-Access via GVs into a special form that shows them all to see if they are keeping up with IB's TWS and if some of them are falling behind suggesting there is an order to the processing of the charts that I can consistently tap into. I may be able to solve the problem this way (much easier). The timer for this form could be adjusted on the form itself. What I mean by this is the MS-Access forms timer could be sped up to as fast as I would ever need it to be for this test. I would watch the IB TWS price changes and see if the prices coming in from MC to MS-Access via the GVs are keeping up.
Last edited by bowlesj3 on 03 Mar 2008, edited 1 time in total.

drwar
Posts: 218
Joined: 31 Jul 2005

Re: I think it is MC (or my overloading it)

Postby drwar » 02 Mar 2008

[quote]Hi, I think the slowdown is MC rather than the GVs. Various plots, lines, etc, are really slow in being updated on the chart so this would have nothing to do with the GVs. I have to implement my strategy above "Method To Test" to try and figure out where it is getting bogged down. A week or two of work.[/quote]

I think the real solution is for MC to implement GV's as part of their programming language. Thusly I am sure they would optimize it for their environment. It would also be easy to use since it would be built in.

User avatar
TJ
Posts: 6523
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Postby TJ » 03 Mar 2008

WOW... GV in MC... that would make it one unbeatable powerhouse.

bowlesj3
Posts: 1980
Joined: 21 Jul 2007
Has thanked: 156 times
Been thanked: 395 times

GVs I think are very fast.

Postby bowlesj3 » 03 Mar 2008

Hi, I am about to do a test in access to actually log the time it takes to send a GV out and immediately get it back. I suspect this is actually really fast. I am thinking maybe 1 millisecond. Now, having said that I find that at times the named string GV returns some funny characters after the letter I sent out. So in MS-Access I have to use a command called "left(string,1)" to make sure I use only the good part of the string. I have not noticed this problem when I send a string from MS-Access out to MC.

For my trading system I need the last price and the RSI to keep up with TWS so I can get as little slippage as possible. Unfortuantely a slowdown makes the work around programming more complex than I had hoped for.

bowlesj3
Posts: 1980
Joined: 21 Jul 2007
Has thanked: 156 times
Been thanked: 395 times

Found the problem

Postby bowlesj3 » 03 Mar 2008

Gee, I found the problem and the solution is a little strange.

First off my 1 minute bar study which is in the main chart was not picking up global variables and bogging way down (I mean really really boggind down). I had an RSI study in subchart #2 which had a lot of plots. It was the RSI study in subchart #2 that was causing the problem. However I solved it by moving the RSI study to another 1 minute bar chart and also in subchart #2 of that second 1 minute bar chart. All of a sudden the main 1 minute bar chart is working fine and the really strange part is that the RSI study is working fine in the second 1 minute bar chart.

bowlesj3
Posts: 1980
Joined: 21 Jul 2007
Has thanked: 156 times
Been thanked: 395 times

Its confirmed.

Postby bowlesj3 » 05 Mar 2008

It confirmed for sure now. What I talked about in the previous post solved the problem. MC is now running faster than I have ever seen it run. So if you have a study in a subchart #2 that you suspect may be a possible source of a major slow down in MC then try moving it to another chart. I figure there is some sort of conflict that was happening in the first chart. Some day if I find some time I can try and figure out what it was.

bowlesj3
Posts: 1980
Joined: 21 Jul 2007
Has thanked: 156 times
Been thanked: 395 times

RSI on subchart #2 blanks and redraws.

Postby bowlesj3 » 06 Mar 2008

Hi,

There was another thing going on with that RSI study on subchart #2 that I forgot to mention was happening (something in addition to its slowing MC down to a crawl).

When I put the extra plots in that RSI study, it started to do a total blank out and redraw. It would do this every 1 to 3 minutes. This problem also appears to have been fixed by simply moving that study to another chart window which also has 1 minute bars.

John.

arjfca
Posts: 1263
Joined: 23 Nov 2010
Has thanked: 717 times
Been thanked: 221 times

Re: notes on Global Variable

Postby arjfca » 20 Jun 2012

John, I know this is an old post but here is the way that I get bit more speed from Global variable data transfer. I post it in case it may help others.


I'm sending some data from MC to Excel via Global Variable, All data sent is transformed and placed in one unique data CSV string. All value transform in a string and all string put together to create a bigger string. All portion of data is separated with a comma

Like "O,H,L,C,LastH,LastLow" So one string include all data

Then In Excel, I read, back the string value If the string is different than the previous one, I read every segment of it and replace the value in the appropriate cell

To separate each segment of information I use the little function that I create and publish in MC call SplitString
viewtopic.php?f=5&t=8950&p=42264#p42264

Again, only one string include all info, one transfer only instead of the multiple one I was doing before for each data needed.

Martin

bowlesj3
Posts: 1980
Joined: 21 Jul 2007
Has thanked: 156 times
Been thanked: 395 times

Re: notes on Global Variable

Postby bowlesj3 » 20 Jun 2012

That split string is a good thing to have. I am not sure if it would be faster but it is worth having something like that. Regarding the speed of GV transfers I think the delays we see are actually delays in the study getting executed (more of an MC problem than a GV problem). In other words the Tick comes in but it takes time to execute all the studies that need processing and by the time the last study gets executed another tick has arrived for processing. A backlog occurs then you see a big catch-up. In my case it may be more of a "machine that needs upgrading" problem. I did test a GV transfer when I was having slow downs and it was amazingly fast. I did it with the external program by setting the GV then immediately getting the value back (basically instant). It proved to me that the slowdowns in MC 3.0 were MC and not the GVs. I am trying to remember if I recorded the speed and sent it back in the 2nd popup (not hard to do). I guess you could do something like that in excel actually. It would be interesting to try that with your string routine against not using that string routine. I think you could do it in MC too by sending it back in an alert. As far as testing the delay in studies being executed that could be done. I did that by writing to a file once. The order at that time was alpha within chart but across charts it was random. However the bottom line is that tick has to be processed by all studies (although now they have a bypass duplicate priced ticks feature which can be used to reduce backlogs if you do not use volume in your studies). I am not sure how to label the start and end of a tick because they are numbered in the study itself one by one I think. I suggested an audit for this so we could find any slowdowns. However although I figure that may be done some day It could be a long time before we see that type of debugging feature for MC. The other reason might be a delay between ticks but now we have that command to submit fake ticks to execute our studies so it is less of a problem now.


Return to “User Contributed Studies and Indicator Library”