New Subject: Studies not getting executed on every tick.

Questions about MultiCharts and user contributed studies.
bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

New Subject: Studies not getting executed on every tick.

Postby bowlesj3 » 17 Nov 2009

Old Subject: Understanding Ticks command & controlling study execution

Start of New Special Note:
This thread
http://forum.tssupport.com/viewtopic.php?t=6871
on the very last post, has a document you can download which at the very end has a good short explaination of the ticks reserve word as well as good explaination of the intrabarpersist command.
End of New Special Note:



There are two scripts in this thread that report on what appear to be 3 bugs in MC (Andrew confirmed they are bugs and said the will likely be fixed in MC 6.0). Specifically:
1/ Every so often a tick does not get processed by a study when "execute on every tick" is checked.
2/ BarStatus = 0 does not occur at all every so often.
3/ More often BarStatus = 0 occurs 2 times at the start of a bar.
I don't know if it is symbol specific so you might want to run the reports.

Two warnings about running the study:

Warning1: This code is designed for 1 minute bars or second bars such as 10 second bars. It is not designed for tick charts. It is a time based report. If you want to run against tick charts use the version attached to the post which is three posts below this one.

Warning2: As the help for Ticks states "Build Volume On" must be set to "Tick Count". This is found in the "format Symbol", "Settings" drop down box. I forgot to do this when I first applied it to the minute bars and the “Ticks” gaps were huge (often as high as 3000). When you set it properly the gaps in the “Ticks” are more reasonable (0, 2, 3). A gap of 1 is normal and is not reported.

I learned a lot from this problem such as thinking out where the tick data is likely to be centrally stored and accessed by each chart on a tick by tick basis without duplicating the data. In other words the studies are likely called with pointers back to the central data in the buffer. I realized this as part of dealing with this missing ticks problem and building the tables below in my mind to think out what was going on. As the table below demonstraites, it made me realize that this central buffer pool of ticks has to have a master Tick Number and I am guessing it has a master tick Arrival time too (both of which would be handy if we could get at them to rewrite our studies more efficiently).

Code: Select all

Below is a diagram of the potential reality of tick arrival and processing (or at least I am concerned about it because I have on occasion written loops in error that stop MC dead with an immediate freeze- I immediately realize it, fix it and the freeze is gone). In the table below one study can keep up with tick arrival speed and the other study can not (other factors outside of bugs in a study or inefficient studies could be the speed of the computer, number of studies, MC/TS itself, operating system catchup, whatever).

Studying the table creates these questions:
“How does MC currently handle it when a big study can not keep up?”
“How should MC report the problem to the trader so they can adjust their studies to fix the problem?”
“What tools can the trader be given to allow them to bypass a low priority study if it falls behind?”
"For slow ticking symbols can Studies have an option to execute every X.X seconds in addition to every tick (and know how it was executed).
For TSS “How can we market this to make traders realize MC handles this reality better than TS/NT? - In other words, helps the traders find studies that need to be made more efficient.”

Symbol ES tick receive buffer Info:
Note: Tick# is not the ticks command value (for demo only)
Tick# ReceiveTime Price, Volume
1 10:00:01 1000.00 ?
2 10:00:02 1000.25 ?
3 10:00:03 1000.50 ?
4 10:00:04 1000.75 ?
5 10:00:05 1001.00 ?
6 10:00:06 1001.25 ?

(Here is a study that can keep up to the ticks as fast as they arrive)
The study uses a pointer to the buffer table info.
At 10:00:01 Call Study2(Using Tick1 table entry) done at 10:00:01
At 10:00:02 Call Study2(Using Tick2 table entry) done at 10:00:02
At 10:00:03 Call Study2(Using Tick3 table entry) done at 10:00:03
At 10:00:04 Call Study2(Using Tick4 table entry) done at 10:00:04
At 10:00:05 Call Study2(Using Tick5 table entry) done at 10:00:05
At 10:00:06 Call Study2(Using Tick6 table entry) done at 10:00:06


(Here is a study that can not keep up in that it takes 2 times as long to execute as the speed of tick arrival)
The study uses a pointer to the buffer table info.
At 10:00:01 Call Study1(Using Tick1 table entry) done at 10:00:03
At 10:00:03 Call Study1(Using Tick3 table entry) done at 10:00:05
At 10:00:05 Call Study1(Using Tick5 table entry) done at 10:00:07

Code: Select all

TSS may want to verify my conclusions here.

Based upon the attached study results against the ES symbol, I think that when you start MC or TS it starts grabbing ticks (trades) and assigning a master number internally to each new tick (for each symbol probably). A small sample of these is shown on the left side of the little table below. Now at the start of each bar MC internally gets the master tick number and stores it for use by the ticks command. If you use the ticks command on each new tick it could only show the missing ticks by using this calculation shown here in brackets which is associated with what I diagram below in the table. (Master# 12 minus Master# 8 + a constant 1 Gives tick 5 within the current bar). In other words it stores the master number 8 at the start of the bar in the example below and works with that to compute the ticks command output. Again it properly computes the occasional gap which Andrew said they would fix in MC 6.0.

Main..................Study
Tick..................Starts
Receive...............at
Buffer................Base Tick# 8
Tick..................Ticks Command
Number................Value=1
1
2
3
4
5
6
7.........Constant
8..........minus 7......1
9..........minus 7......2
10.........minus 7......3
11.........................(Gap of 2 here as the study shows)
12.........minus 7......5

If we can get at the master tick number in the buffer from all our studies we can make good use of that to audit our studies. Also if we can get at the master arrival time that would be even more useful.

Note that MC does not count study executions to calculate the ticks command output or it would not be able to detect missing ticks. It has to use the master tick number from the buffer. Users can count study executions. No need for a reserve word command to do that.

New features of the attached study which creates the missing ticks report:

1/. Bar type, bar size and symbol are part of the report name so you can run it on seconds bars and minutes bars at the same time and have separate sets of reports being created.

2/ The “?” which shows a gap in the “ticks” reserve word is replaced with the size of the gap. Each tick should be an increment of 1 (a gap of 1) and when this occurs nothing is reported in that column. If it is less than 1 or greater than 1 it is reported. Note that the Stats report only counts the number of bad gaps that occurred (it does not total the size of the gaps).

3/ “<<” is placed beside large gaps and you can control the size for this. You also have the option to have it play a buzz sound if you want when this happens. I used it during testing.

4/ The stats report now reports every X number of bars. Doing it by minutes was too hard to verify.

5/ If you want you can have a sound come out when a stats record is written. I needed it for testing.

6/ A raw report is optional (no formating). I needed it to test the study.


The zip file now has the .pla file of the code. No more cut and paste.
Attachments
A_Test_BarStatusZeroAndTicks.zip
(5.38 KiB) Downloaded 291 times
Last edited by bowlesj3 on 22 Jun 2010, edited 70 times in total.

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 17 Nov 2009

John,

Thanks for sharing, it will take me few days just to figure out how to use it in a strategy.....if I can only manage to understand your wounderful coding technique.

Regards
Super

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 17 Nov 2009

Thanks Super.

Unfortunately this code was really rushed. Now I can see things that could be simplified. It is just a quick and dirty report (stuff grabbed from other code and stripped to fit). Maybe in time after TSS fixes the bugs I will clean it up and new MC users could use it to understand BarStatus in time based charts. For now it can be used to teach that you need to use a change in Time or Time_s to detect the start of a bar rather then BarStatus = 0; I did put the file path in a single variable and updated the code.


It is unfortunate that the Code markers in this forum do not work perfectly. They still seem to misallign stuff here and there.

John.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 18 Nov 2009

There are two scripts in this thread that report on what appear to be 3 bugs in MC. Specifically:
1/ Every so often a tick does not get processed by a study when "execute on every tick is check".
2/ BarStatus = 0 does not occur at all.
3/ More often BarStatus = 0 occurs 2 times.
I don't know if it is symbol specific so you might want to run the reports.

Here is the version of this test code which is used for tick charts. Do not use this version for Minute bars or Second Bars (it is not time based).

Warning2: As the help for Ticks states "Build Volume On" must be set to "Tick Count". This is found in the "format Symbol", "Settings" drop down box. I forgot to do this when I first applied it to the minute bars and the “Ticks” gaps were huge (often as high as 3000). When you set it properly the gaps in the “Ticks” are more reasonable (0, 2, 3). A gap of 1 is normal and is not reported.

I put in an extra report called "Raw" which is a test of the detail report. The raw report just dumps the records. It will show ticks dumped from one partial bar ahead of the easier to read formated detail report.

The zip file now has the .pla file of the code. No more cut and paste.
Attachments
A_Test_BarStatusZeroAndTicks2.zip
(5.85 KiB) Downloaded 270 times
Last edited by bowlesj3 on 20 Nov 2009, edited 8 times in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 19 Nov 2009

zipped compressions of the .pla files are available now for both versions of the test code above. Super is doing a larger test of the second one today (tick charts version).

brodnicki steven
Posts: 407
Joined: 01 Jan 2008
Been thanked: 3 times

Postby brodnicki steven » 19 Nov 2009

zipped compressions of the .pla files are available now for both versions of the test code above. Super is doing a larger test of the second one today (tick charts version).
** Please make sure that Andrew is aware of the results(if a bug exists) and give him the code to verify it. We all want a bug free program to rely on.

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Postby RobotMan » 19 Nov 2009

Hi Steven,
Andrew already confirmed the bug and estimated a fix in the Ver 6 release. That seems like a long way away, but oh well.


Hi John,
WOW. I go away for a day or so and then I come back to this post. Awesome. This forum is really lucky to have you. ( and TJ and SUPER and on and on...)

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 19 Nov 2009

Hi All,

I removed the cut and paste. It is all zips now.

I made some changes. The stats is a count based upon X number of bars. Status every Minute was to hard for me to verify. It is set to 100 bars. You can change it. The heading indicates your selection now.

I turned off the raw report. All seems to be working now.

Super is testing against TS and MC. It seems that TS does not have a BarStatus = 0 at all. If that is true then you have to do what I am doing in the 2nd test code if you want to check things on the first tick of the bar.

John.
Last edited by bowlesj3 on 19 Nov 2009, edited 2 times in total.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 19 Nov 2009

according to the TradeSation EasyLanguage help file:


BarStatus = 0
the opening tick of a bar
(relevant only for strategies using Open Next Bar order actions)

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 19 Nov 2009

this is what I got with some test settings on the e-mini. A run on 1 minute bars, 10 second bars and a tick chart run of 10 bars.
Attachments
Outputs.zip
(151.63 KiB) Downloaded 269 times

brodnicki steven
Posts: 407
Joined: 01 Jan 2008
Been thanked: 3 times

Postby brodnicki steven » 19 Nov 2009

Hi Steven,
Andrew already confirmed the bug and estimated a fix in the Ver 6 release. That seems like a long way away, but oh well.


Hi John,
WOW. I go away for a day or so and then I come back to this post. Awesome. This forum is really lucky to have you. ( and TJ and SUPER and on and on...)
*Robot- That's great news that V6 will have it fixed.
Thanks for contacting Andrew about it.
This is a great list and we are lucky to have terrific support from Andrew and the rest of the MC support staff, as well as help from some super programmers here.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 20 Nov 2009

Thanks guys for the compliments about the list report. It should make it easier for TSS to confirm any bug fixes.

At this point we do not know if this is symbol specific (or at least I do not). I have changed the first post to state the three bugs more directly right up front in the hope that more users will run the study on different symbols.

I put a table in the very first post which gives a picture of the problem. I set it inside the code markers so it stands out. It replaces the purpose of this thread (giving it a better visual to help understand what is to be learned and understood).

If the version you downloaded does not have 5 input parameters you are not running the latest version.

John.
Last edited by bowlesj3 on 20 Nov 2009, edited 2 times in total.

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 20 Nov 2009

Here's test file run on MC Version 5.5 Beta 2 (Build 2536) and TS 8.6 on symbole ECZ09 on 1 minute and 10 ticks.

TS on tick data works perfectly unlike MC but on the other hand with 1 Minute data MC stands out, though not perfect.
Attachments
TSLog_ECZ09.zip
(169.56 KiB) Downloaded 297 times
MCLog.zip
(49.1 KiB) Downloaded 275 times

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 20 Nov 2009

Here's addition information on recent run on ES symbol based on 1 Minute and 10 tick charts on MC and TS platform.
Attachments
MCLogES_Detail.zip
(110.1 KiB) Downloaded 264 times
TSLog_ES.zip
(703.47 KiB) Downloaded 244 times

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 20 Nov 2009

TS works perfectly because it is single-threaded. We didn't have the issue until we introduced the multi-threading. However we know how ot fix it.

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 20 Nov 2009

TS works perfectly because it is single-threaded. We didn't have the issue until we introduced the multi-threading. However we know how ot fix it.
Andrew,

Thanks for your prompt response and assurance that it will be fixed, I sincerely hope that it will be fixed in your uncoming version.

We all want MC to be the best product out in the Market.....and I am sure we are getting closer each day with your(Teams) sincere efforts.

Regards
Super

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Postby Nick » 23 Nov 2009

TS works perfectly because it is single-threaded. We didn't have the issue until we introduced the multi-threading. However we know how ot fix it.
As this is a pretty fundamental bug (it effects the accurate reception, storage or display of data) any chance of a hotfix?

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 23 Nov 2009

Hi Andrew,

Regarding
TS works perfectly because it is single-threaded. We didn't have the issue until we introduced the multi-threading. However we know how ot fix it.
TS skips ticks with Minute bars just like MC. So single thread does not correct it in TS.

From the sounds of Super's tests it appears that TS is even worse than MC with minute bars. (assuming it was done on the same symbol at the same time but maybe Super was unable to do a direct test).

For me it is not a problem if the code is bypassing duplicate priced ticks on purpose when the code detects that factors such as (too slow a computer, excess study load and very fast market load) are causing the study to fall behind. It is probably just an old forgotten TS feature (rather than a bug) which was installed years back when computers were slower. If it is I personally would prefer it be uncovered, documented for the traders and adjusted so it is an option (defaulting to process every tick and report an any slow downs to help the trader decide what to do). In my other post I ask for TickArrivalTime and MasterTickNumber comands to supplement this effort so we can audit to improve our studies and decide if we need a faster computer, whatever. Seems like a great idea to me (but then again I thought of it) :D

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Postby Nick » 03 Feb 2010

Can anyone confirm whether this is in fact fixed in version 6?

I see no mention in Version 6 release notes so presumably it has not been? A rather serious problem for the project I am currently planing.

Cheers.

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Postby RobotMan » 03 Feb 2010

Hi Nick

Not yet in Ver 6 Beta 2. I test it every time a new one comes out.

Maybe, for real-time trading, go to a single core CPU and turn off Multi-threading? (j/k :wink: )

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 03 Feb 2010

Andrew said It would not be fixed until the final release.


Return to “MultiCharts”