Why I don't trust "IntraBarPersist"

Questions about MultiCharts and user contributed studies.
User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Why I don't trust "IntraBarPersist"

Postby RobotMan » 06 May 2009

I am posting two .jpg screens. They are two separate charts of identically the same data: ES 500 tick bars, volume built on volume, same amount of bars back. I essentially copied and pasted a window inside of multicharts and then "compiled" the indicator to get it to start at the exact same time on both charts.

Look at CumulativeVolume-01.jpg
The indicator is cumulative (upticks - downticks).
The yellow one is calling the upticks and downticks (see the code).
The skyblue one is using intrabarpersist and calling upticks and downticks (see the code).
Both plots should be on top of each other. There should only be one line because both plots are essentially measuring the same thing.

One of them is lying

Now note the totals in the Price Scale:
Yellow = 63,504
Skyblue = 39,013

Now look at CumulativeVolume-02.jpg
Note the totals in the Price Scale again.
Yellow = 63,504
Skyblue = 16,964

Intrabarpersist cannot be trusted.

Please! Someone who knows what they are talking about prove me wrong. Don't guess. Show me.

Or I shall go completely insane.

Code: Select all

// This test compares (UPticks - Downticks) VS (IntrabarPersist Cumulative Volume)
// Best tested on Tick charts with "Build Volume On" set to "Trade Volume"
// Theoretically, Tick based bars include all the volume for the bar, whereas
// Volume based bars cut off any left over volume once the bar reaches it's total.
// ie: for a 5000 contract bar, if the bar is at 4975 contracts and a trade comes in
// at 100 contracts, only 25 contracts are used and the other 75 are ignored.

Inputs: MinLotSize(1), MaxLotSize(9999) ;

Var: Cumulativeticks(0) ;
Var: CumulativeSum(0);
Var: intrabarpersist PreviousTicks(0);
Var: intrabarpersist TSize(0);
Var: intrabarpersist UpSum(0);
Var: intrabarpersist DnSum(0);

if BarStatus(1) = 2 then Value1 = 1; //this prevents code from starting in mid-bar
if LastBarOnChart_s and Value1 = 1 then begin
TSize = Ticks - PreviousTicks;
If TSize >= MinLotSize and TSize <= MaxLotSize then Begin
if Close = CurrentBid then DnSum = DnSum + TSize;
if Close = CurrentAsk then UpSum = UpSum + TSize;
End;
PreviousTicks = Ticks;

if BarStatus(1) = 2 then begin
CumulativeSum = CumulativeSum + UpSum - DnSum ; //calculate RT Volume from trades at bid and ask
Plot1(CumulativeSum,"CumulativeSum") ;

Cumulativeticks = Cumulativeticks + Upticks - DownTicks ; //calculates tabulated volume
Plot2(Cumulativeticks, "Cumulativeticks") ;

PreviousTicks = 0;
UpSum = 0;
DnSum = 0;
end;
End;
Attachments
CumulativeVolume-02.jpg
(89.88 KiB) Downloaded 8666 times
CumulativeVolume-01.jpg
(85.98 KiB) Downloaded 8736 times

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 06 May 2009

// Volume based bars cut off any left over volume once the bar reaches it's total.
// ie: for a 5000 contract bar, if the bar is at 4975 contracts and a trade comes in
// at 100 contracts, only 25 contracts are used and the other 75 are ignored.
slightly OT:
I know some software do it the way you described.
My understanding is, MultiCharts has taken extra effort to program it the proper way.

i.e. what is left over from one bar is used to start the next bar.

you can confirm this by adding a volume indicator to a volume bar chart.


anyway, back to the main program...
I would be interested to learn more about Intrabarpersist.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 07 May 2009

I have not studied your code (got to trade now) but does this post about the barstatus bug help at all?

http://forum.tssupport.com/viewtopic.ph ... =barstatus

John.

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 07 May 2009

I have not studied your code (got to trade now) but does this post about the barstatus bug help at all?

http://forum.tssupport.com/viewtopic.ph ... =barstatus

John.
BarStatus(1)=2 does not have bug,

it's BarStatus(1)=1 which has problem

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 07 May 2009

Robotman,

Here's screen shot of your code on TS 8.5 for comparision purpose,

Both charts were identical for a while (image r-1) and later on they looked different (image r-2).

I guess we have to keep scratching our head to figure this out.

Regards
Super
Attachments
r-2.JPG
(87.73 KiB) Downloaded 8534 times
r-1.JPG
(99.8 KiB) Downloaded 8484 times

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Postby RobotMan » 07 May 2009

TJ said:
I know some software do it the way you described.
My understanding is, MultiCharts has taken extra effort to program it the proper way.

i.e. what is left over from one bar is used to start the next bar.
Well, I'm a little less insane. But like Super says:
I guess we have to keep scratching our head to figure this out.
It would be nice for Marina or Andrew to weigh in after they get a chance to check it out.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 14 May 2009

Hi Bob,

Please find our comments below.
1) Both plots should be on top of each other. There should only be one line because both plots are essentially measuring the same thing.
Plot 1 cannot be equal to Plot 2, as they are calculated in different ways - as specified by their respective codes.

2) As to the difference of values on Plot 2 (bluesky) of the indicator on two charts:
You did manage to ensure the same starting point on either chart. However, for this particular plot even if the starting point and symbol settings are identical on two charts, it does not guarantee identical results.

This can be explained by the following:
The script uses the keywords: InsideBid, InsideAsk. The history for these values is not saved, but taken from the chart’s Status Line each time the information is required. Indicators are not calculated simultaneously, and thus, indicators on different charts receive different values when calculating InsideBid and InsideAsk. Hence the difference in Cumulative value for Plot 2 (skyblue) on different charts.

P.S. keywords InsideBid, InsideAsk are called from the CurrentBid, CurrentAsk functions.

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 14 May 2009

Bob,

it seems from Marina's reply you will have to find a way to save the InsideBid, InsideAsk may be using Global Variables or ADE functionality.

You have more midnight oil to burn now.....

Regards
Super

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Postby RobotMan » 14 May 2009

Hi Marina,

Thanks for getting back and answering some questions about the reserved word, IntraBarPersist.
I realize that this answer not only reflects Multicharts, but probably TS also.

I find it rather odd, that, using "build volume on": Volume, and then using the reserved words, "Upticks" and "DownTicks" I can get an accurate accumulation of real time "up volume" and "down volume". I can see it happen intrabar, tick by tick, in Plot2. It is accurate and repeatable in a separate identical chart. The program itself must be keeping track somehow of the volume acquired on an uptick and the volume acquired on a downtick, otherwise how could these two reserved words work in real time, tick by tick? How do these programs do this?

There must be some kind of algorithm that works reliably inside, silently sorting volume like two workers, named "uptick" and the other named "downtick" on an a fish sorting assembly line. "oh, these ten fish go with the upfish bucket, oh, these 43 fish go with the downfish bucket." And on and on it goes, until the bar finally closes and the assembly line workers dump their two buckets into a special divided fish bin for that bar, and start over again. The Line Supervisor can walk down the line to each divided bin in the past and count the Total UpFish and DownFish for that particular bar, if he wishes.

Now, what you are telling me, is this same program, has another identical assembly line, with two assembly line workers that are told, "do the same thing that other assembly line is doing, only this time only select upfish and downfish that are greater than 0 kilos and less than 9999 kilos. Keep track and report in real time just like the other line, but don't use divided bins because we would need too many, instead, throw all the fish away at the end of the bar."

Now, at the end of the shift, with identical fish given to both assembly lines, one assembly line can account for every single fish on the tote-board (plot2) and the other assembly line has no idea what they did and their totals are absolutely incomprehensible (plot1). Plot1 assembly line workers can do whatever the hell they want at what ever time they want to and they answer to no one and are still considered as competent as the Plot2 assembly line workers? (Sounds like union workers!)

My conclusion:
In my mind, I find it inconceivable that IntraBarPersist cannot do this simple thing that it promises it can do!
IntraBarPersist is a joke and should never be used by any serious Easylanguage/PowerLanguage programmer whether in MultiCharts or TS. (Just my opinion.)

There should be a reliable and accurate way to access the same info that Upticks and Downticks have access to intrabar. Maybe introduce a reserved word "VolUpTick" and VolDownTick". I don't think I should have to "kluge" something up like a GV or .dll to do something so basic that it is the number one requested feature on the TS Forums for YEARS.

Please see:
http://forum.tssupport.com/viewtopic.php?p=16835#16835

ps - I still think MC is awesome!
Last edited by RobotMan on 15 May 2009, edited 2 times in total.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 14 May 2009

does this variance has to do with aggregated ticks by the datafeed?

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Postby Nick » 27 May 2009

What would be great is if MC would

Have the option to store (in its database) best bid and best ask changes with size and time stamp.
Have the option to store deeper levels of the order book with size changes and time stamp.

The time stamp would probably need to be locally generated.

The ability to calculate indicators based on tick by tick changes of all these values.

Some of the cutting edge studies in use nowadays detect things like order flow, people 'bailing' by pulling orders, people supporting levels using icebergs (refreshing the order book) need access to this sort of information.

Incidentally upticks and downticks are slightly different as price can tick up by traders bidding up the price. Put another way just because price ticked up dosent mean that the last order hit the ask. Having said that they are a decent proxy. The big advantage of Uptick Downtick is it can be determined on historical tick data of price alone.

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Postby RobotMan » 28 May 2009

Hi Nick,

Well, the emphasis of development for MC is more for systems and portfolio testing and not for discretionary scalping. It's the law of diminishing returns really. So I don't expect to see any 5 level deep bids and offers displayed soon. I agree with you about Upticks and Downticks - they are good enough. I have done LOTS of testing with the intrabarpersist vs. UpTicks/DownTicks - If I really thought that I wanted to filter small vs larger traders, I could do it by using intrabarpersist and upticks/downticks instead of close=bid and close=ask.

If you really need it, hire a programmer for NT or PhotonTrader. NT started life several years ago as a front end for scalpers. Charting was an afterthought. Both NT and PT have access to the DOM 5 levels deep in the bid and ask. My friend, Bill Shepard in Seattle of ViperSpeedTrader.com has done lots of studies on just this topic and has had many studies programmed that use the information there. If it gives you an edge, I say go for it and spare no expense. But, in the end, what makes him a good trader is the hours and hours he has spent honing his skills at reading the order flow and then just trying to make that thinking process easier.

There are now, and, will be in the future, huge discussions on using intrabarpersist and close=bid and close=ask. Just do a search on Traderslabratory.com or TS Forums. Mostly from guys that don't want to put in the effort to do the testing and learning. I personally, have my own OPINION, and that's all it is.

If I had a software company like TSS, I could come out with a module just for scalpers or intraday traders.
(Infact, sometimes I wish MC was modular and you just buy the module that you needed. Just need charting? - get a charting module. Just need Systems development? -get the systems module. Want portfolio analysis? -Portfolio module. Radar Screen Module. Scalper module. Replay order entry module with DOM and depth of market module. Genetic algorithm and neural network module. All add-ons to the central core engine.)

But then the reality of marketability steps in and you have to ask "is it justified?" Would people buy them? You gotta put the development into the stuff that gives a quick payoff before you get into the esoteric stuff.

Heck, Mike Barna, over at tradingsystemlab.com (he lives about 15 minutes away from me) has a brainiac pushbutton systems generator module (for only $20,000) and people are buying it like crazy. So, you never know.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 30 May 2009

i do not fully understand intrabarpersist.

I can't see any difference between these 2 codes in real time.

is there any differences?

Code: Select all

var:
intrabarpersist avg(0);

avg = averagefc(close, 10);

Code: Select all

var:
avg(0);

avg = averagefc(close, 10);

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 30 May 2009

Hi Bob,

I finally had a chance to read this post a little better.

Try reading both Marina's response and my final post in the thread below (actually my last two responses since I did an update today). I think this may explain the problem you are having with insidebid and insideask.

In short, your study is probably executing when a tick comes in (the trade comes in) but bids and asks are changing in a totally different timing. As an experiment, you might want to try what I suggest in the last post (I have never done that experiement actually). It may not solve the problem but it may help you trust intrabarpersist more. If you do that test (pretty easy test) I would like the hear the results.

Actually, come to think of it, if I understood your long discussion above correctly and your comments about volume being calcualated correctly then this post probably explains that as well. It is because the volume is attached to the ticks and thus will be calculated correctly. Bids and asks are a totally different timing. Again the experiment I suggest may lead to a solution.

John.

http://forum.tssupport.com/viewtopic.ph ... =insidebid
Last edited by bowlesj3 on 30 May 2009, edited 2 times in total.

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 30 May 2009

i do not fully understand intrabarpersist.

I can't see any difference between these 2 codes in real time.

is there any differences?

Code: Select all

var:
intrabarpersist avg(0);

avg = averagefc(close, 10);

Code: Select all

var:
avg(0);

avg = averagefc(close, 10);
Tj,

Use it in a strategy with Intrabarordergeneration=true to see the differences.

Variable declared Intrabarpersist will update on every tick.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 30 May 2009

Tj,
Use it in a strategy with Intrabarordergeneration=true to see the differences.
Variable declared Intrabarpersist will update on every tick.

ok, thanks.

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Postby RobotMan » 08 Jun 2009

Hi John,

Been gone for awhile mountain biking in Moab, UT. and visiting friends. Just now catching up.

You said
In short, your study is probably executing when a tick comes in (the trade comes in) but bids and asks are changing in a totally different timing.
That is exactly my point. An emini trade comes in during regular trading hours. It came in on the bid or the ask. That's it. (or in the VERY rare case, in between.) It's not a snapshot of a moving target like watching the order book numbers flipping around all over the place. It is a trade. It is a certain size and occurred at the bid or ask. It doesn't matter if it occurred on a 1 tick chart or a 500 tick chart, or two identical charts. It should be consistent on both because it has nothing to do intrinsically with the chart.
Look at time and sales from just about any charting package. (Maybe I am way off base here - but that is what I was hoping someone could point out. "Hey Bob, you are way off base. This is the way it really happens..." and I would be all, "Hey thanks for setting me straight".

To me intrabarpersist: close=bid, close=ask is just black magic that changes depending on what chart you are using. Even if two charts are set up identical! I think people are deceiving themselves if they use things they do not understand and just take for granted. Besides, there are software packages that are dedicated to exactly this one thing: order flow (futuresScalper, OrderFlowAnalytics, MD among others). If I was really that dissatisfied with what I use I would probably go to someone who specializes in the thing I was interested in and has a proven track record.

I don't think I should jump through hoops to get it to work right (not to mention, to even understand it. No GV or DLLs or ADE. I am too old and cranky. I would rather believe intrabarpersist: if upticks>upticks[1] then...etc
At least upticks and downticks are consistant.

I was following a thread on TradersLaboratory called "Volume Splitter" and you would think people had discovered the holy grail and were trying to reverse engineer an indicator that EOTPro uses (and seems to have gotten right). I even chuckled when Richard Todd, who I totally respect (yea, doth envy his mathematical prowess), could not hold back anymore and tried to help out. Here was the original author of the EOT indicator and even he wasn't able to change the direction of the discussion. I stopped following it because I was just too frustrated.

I don't use intrabar anything in my trading. And I am a happier man for it. (although it probably doesn't sound like it :D )

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 08 Jun 2009

Glad you got some time off Bob.


My understanding is that intrabarpersist has changed somehow when using
Intrabarordergeneration. I don't use that and to me it is straight forward. Ironically I don't use it that much and my systems work. I don't want to touch them if they are not broken.

John.

kh_model
Posts: 43
Joined: 13 Jul 2005
Has thanked: 12 times

just a guess...

Postby kh_model » 09 Jun 2009

Could the intriguing discrepency described above have anything to do with "tsShaper"? Somewhere here I found a very nice discussion about it, including how various registry settings can be used to control how much volume gets redistributed across bars whenever "shaping" is necessary. According to the discussion I read, there is a tradeoff between accuracy and memory useage, and the default values for "tsShaper" err on the side of preserving memory to avoid crashes. I can't pretend to be any kind of expert here, just thought i'd offer a "possible" clue.

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Postby RobotMan » 10 Jun 2009

Hi kh_model,
Could the intriguing discrepency described above have anything to do with "tsShaper"?
I have no idea. Marina did not mention that.
What we do know is that MC and TS do not take a "hard look" at what bid/ask trades are being executed at. They take a guess by looking at a "snapshot" whenever an indicator calls them. The price/trade size, and whether it was a bid/ask are all separate entities and not linked together in a usable way. When I say usable, I mean constantly in step with each other and independent of the indicator that calls them. If you have 2 different charts and you use intrbarpersist: if close=bid or close=ask, you are never going to get consistent, reliable results between the two.
All we really know for sure, is if a certain size of trade is at a greater or lower price than the previous trade. In other words: Intrabarpersist: Upticks and Downticks are consistent and reliable.
That's where we are at, without going through hoops by using an external dll or some other trick that keeps track of trade size at the bid and ask at particular times.
What I am saying: People on these other forums and indicator websites that are trying to use perceived information derived from bid/ask as actual information are deceiving themselves and are not doing their homework, IMO. Unless they are using a program that is specifically built to track this info.
On a lighter note, I don't want to discourage them as they are my main source of income when I trade. :D

kh_model
Posts: 43
Joined: 13 Jul 2005
Has thanked: 12 times

thanks

Postby kh_model » 13 Jun 2009

Thank you Bob for your excellent comments on this topic, including those at other sites.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 14 Jun 2009

Everything you say Rob is correct. What would assist us is for the data feed provider to send all tick data as triplets. That is, trade, ask and bid together even when one or two of these don't change. This is not the total solution but it would do most of the work for us if one wanted a more consistent snapshot of the market within a single study. After all, that's what one sees on the quote window in say IB. Of course I doubt they will ever be convinced to provide tick data is such a format.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 14 Jun 2009

Bob,

My need to try this is a ways off but you might want to give this a shot now. It appears that there are questions here that no one can answer yet.

The experiment is based upon the assumption that #1,#2 and #3 below are correct and it would require that you have a plot for (ask and bid and last) on the same chart (also something I have never had the need to do but it seems that Janus is doing that) and that you force these three studies to execute in the proper order on that chart.

If
1/ I change ESM9 to bid-series via: format, symbol, settings, field, "bid".
2/ set the study to be based upon this symbol running on the Bid-Series,
3/ I set the study to execute on every tick.
then does the study execute on every change in inside bid and do I plot using CurrentBid?

I have noticed that the studies are executed in alpha sequence within one chart. That is the key to this idea (the bid and ask studies are executed before the study based upon the last price). So if the bid study is set up to execute when the bid is updated and the ask study is set up to execute when the ask is changed and both of these update a GV (this is really fast) then if your 3rd study which is executed when the last price comes through it should always have the accurate bid and ask in those GVs when it is executed (getting the GVs is very fast).

It is worth a try. It may not be 100% perfect but if it works it would catch it correctly 99% of the time I would think. I think the chances of it being accurate are greater if you have less charts on the workspace and less workspaces open as well. Here is the big question that relates to this. "If a study is based upon bid or ask does it gather the changes in a buffer for processing when the study is eventually executed or are all those changes lost if there is a bit of a pause in the executing of the study. In other words is there a catch up going on to eventually process the changes in bid or ask so you can actually plot them accurately a bit late if needed.". Here is another thing to ask. If there is a buffer for the bid and ask is the buffer processed for catchup before the next study on the chart is executed? Here is something else to really get one thinking. If the bid or ask or last does fall behind in its feed or MC falls behind in its processing then logically (in theory) everything needs a data/time stamp and the catch up should be in proper sequence.

I need this when I trade the SXF which trades very thin but right now I am not doing that so I have delayed this experiment. In my case I do not need a plot of the bid or ask. I just want to be warned if the spead is very wide so one study running under MC is probably more than adequate. I think that if one is basing their trades on some sort of pattern in the last price and it is auto executing then testing close = currentbid or close = currentask as a way to ensure there is not going to be a big jump in price it probably good enough. A plot would not be needed here. That is all I am looking for. Having said that, I haven't traded anything very thin before so I probably have some learning to do here. In time maybe a plot would seem handy. I do not use time and sales for very small scalps. I won't trade unless I see a pattern for a much larger move. In essence I anticpate what time and sales would be before it actually happens.

John.
Last edited by bowlesj3 on 14 Jun 2009, edited 1 time in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 14 Jun 2009

Another thing about this. I remember Marina saying that currentbid and currentask do not have history (no square bracket backward offset is available). However this does not say that they are not accurate when sought after regardless of whether the study is based off of bid or ask or last price. If one wants history the EL collections has some commands that can be used for this which according to the documentation are very fast. For sure the study needs to be set to execute based upon the bid/ask rather than the last price (kind of obvious if #1#2#3 above are correct). Otherwise instead of using EL collection commands you could set up a rolling table (rolling array) with a special offset calculation to go back and get old entries. The rolling table could be GVs too. This second approach is getting back to real programming.

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Postby RobotMan » 14 Jun 2009

Hi John,

Nice thought process of continually thinking things out. I am only good at discovering a problem and not so good at understanding it (or solving it) until Marina or Andrew or someone like you explains it to me. That is why I asked the initial question at the beginning of the thread and gave it such a provocative (yet probably , misleading, title). I initiated it after watching some videos from Bill Dennis about smart/dumb money and an indicator they were using at EOTPro that was written by Richard Todd.

I studied my own version to see if dumb money actually traded opposite of the direction of the trend, as I have read some research papers that indicated that large intraday spec traders enter a position "all in" and then scale out as the trade goes in their favor. Hence the appearance of selling on the way up and buying on the way down [by smaller traders]. I brought this up to Bill in another thread and he insists that it is dumb money. So we disagree. But, if he can make money trading it, then he is a hero and that is all that matters and I don't care if he believes that the moon is made of green cheese. (which it is. ;) )

Then, I started seeing big inconsistencies using intrabarpersist and did not know what was causing it. So I started searching around the internet to find others that were using it. There are large discussions in several other forums but I did not dig deep enough to discover the flaws. I only knew there were some. Thus, the beginning of this thread.

Subsequently, through Marina and Richard Todd (posting on TradersLaboratory) an understanding of the phenomena was explained. The problem, as Richard explains, is called "race conditions" when trying to correlate two independent data streams in "event driven" programs. IntraBarPersist works fine and can be misused when a programmer does not understand the limitations.

Richard knew this when writing his indicator (or at least stumbled upon it like I did). He understood and worked around the problem. He's a pretty smart guy. That's why, when I have a problem, I search the internet for smart guys who have had the same problem and solved it. (Otherwise, I stay away from discussion forums except this one.)

Sometimes, there are others who have already solved for the bid/ask problems and it is better to just pay them. Hence the reference in my other posts to programs that keep track of such granularity: "(FuturesScalper, OrderFlowAnalytics, MD among others)".

You might be able to use EL Collections to do what you need. I don't use it and have not delved into it extensively. It seems to fill a niche that TS programmers abandoned long ago.

To me, time is money, and I look to those whose specialty is solving specific problems (unless of course, you like to tinker, then it becomes your hobby). I always tell people when they have car problems to go to a specialist. If you have radiator or brake problems don't go to a muffler shop. If you have transmission problems don't go to an oil change place. Go to a specialist. For myself, my specialty is not writing .dlls or using GVs. I would rather just pony up for a real, solid solution. I don't mind paying for quality, especially when it will pay for itself many times over.

For those who need a solution and are not so inclined to figure this out on your own...

I would think that contracting and paying TSSupport to write a program to solve the bid/ask/tradesize would be very effective. Who knows more about this stuff than they do?

Additionally, here is a guy (Steve Johns) that has a solution at a good price: (I hope Andrew doesn't mind me posting this, I have no affiliation - I think the original author, SwissTrader, posts to this forum)
http://www.codefortraders.com/Tradestat ... kDelta.htm
He might even customize it for MC.

Richard Todd (if I can be presumptuous - I have never spoken or emailed him) might also give direction if contacted privately by email. He is a programmer, wrote the code for EOTPro and is an MC expert.
http://www.movethemarkets.com/blog/richard/

I, myself, feel that there is merit in pursuing a solution individually and sharing it with the forum. It is good to understand how things like this work so that we are not deceiving ourselves by unintentional misuse. I have satisfied myself in the answers that have been given. I greatly appreciate those that have contributed here and on other forums that I have searched. I probably will not pursue it anymore as I, like you, watch price patterns and only confirm with volume and do not use this small granularity of information.

But we are all looking for an edge, No?

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 14 Jun 2009

Yes, I agree Bob. I think intrabarpersist is really very simple as explained by the test code and print statements in the post below. But when it comes to lining up separate data sources like bid/ask/last things get harder (and even harder to explain). I suspect it is not really intrabarpersist that is the problem but other deeper factors.

http://forum.tssupport.com/viewtopic.ph ... barpersist

Hopefully I can try my experiment in the future and post the results.

John.

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Postby RobotMan » 12 Nov 2009

I thought I would tag this on to an old thread because I got curious about using intrabarpersist to do some statistical analysis of data. I wrote some simple code to test a theory and kept getting weird results. One of the basic trouble shooting things I do when I write code is to make sure my assumptions are correct and that the individual components of the code actually do what the they are supposed to.

So I wrote an indicator that just counts ticks. Every time a tick comes in the counter increments by one. It doesn't matter if "Build Volume On" is set for Trade Volume or Tick Count.

Code: Select all

Var: intrabarpersist TradeCount(0);
Var: intrabarpersist TradesThisBar(0);

If LastBarOnChart_s then Begin

TradeCount = TradeCount + 1 ;
TradesThisBar = TradeCount ;

If Barstatus(1) = 2 then TradeCount = 0;

Plot1(TradesThisBar ,"Trades") ;

End;
I then set it to count on a 500 tick bar chart on 3 separate subcharts, then let it run all day. Theoretically, each bar stands on it's own but should be identical to the barinterval (which in this case is 500).

Results:
Not only is each count different but each subchart bar is different for the same chart bar (see data window in attached .jpg). Never in the entire day did any count even get close to reaching the barinterval. Maybe this code needs to run on it's own computer with one chart running, no other data or indicators running and super powerful CPU, 32 GB of memory and all Windows services turned off.

Conclusion:
Unless someone can poke a hole in my code, I think I have proven once and for all that IntraBarPersist is useless and untrustable for even the most basic of needs. IN MY OPINION; IntraBarPersist should never be used by anyone under any circumstances for anything. It simply does not work. This reserved word should be changed to "RandomNumberGenerator".

Can anyone prove me wrong?
Attachments
TradeCounter.jpg
(104.01 KiB) Downloaded 8439 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 13 Nov 2009

Hi Bob,


I checked the help on this
IntraBarPersist

Used in variable and array declaration statements, before a variable or array name, to specify that the value of the variable or array elements are to be updated on every tick.

If IntraBarPersist is not specified, the value will be updated at the close of each bar.
I wish they had something like this instead.

SimpleVariables; {these have no underlying table - full user control}
MySimpleVariable1(0),
MySimpleVariable2(0);

Variables; {these have an associated table built which is maxbarsback size}
MyUnderlyingTableVariable1(0),
MyUnderlyingTableVariable2(0);

I created two posts below showing two tests.
1/ the first shows some sort of problem with either the execution of the study for each tick or a problem with the "ticks" reserve word. I can't remember the person's name but they reported a problem with BarStatus = 0 and this appears in that test. It oculd be that when BarStatus = 0 is not appearing the study is not executing that tick.
2/ the second test shows what happens with and without intrabarpersist. To see the results just run it twice and include or exclude the IntraBarPersist command on the variable called BN (which stands for BarNumber).

John.
Last edited by bowlesj3 on 13 Nov 2009, edited 23 times in total.

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 13 Nov 2009

Robotman,

I tried out your code on MC and TS 8.6, I did see inconsistency with regard to MC but not to the scale as you have demonstrated, mind you I had a smaller run on your code.

TS 8.6 showed perfect results....clearly pointing to a bug with MC's intrbarpersit.

Look forward to Andrews view now.
Attachments
mc-1.jpg
(124.69 KiB) Downloaded 8442 times
MC-2.JPG
(77.91 KiB) Downloaded 8425 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 13 Nov 2009

removed, see next post.
Last edited by bowlesj3 on 13 Nov 2009, edited 3 times in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 13 Nov 2009

This study shows something interesting about the ticks reserve word on 10 second bars when the "build Volume On" is set to "ticks". If the ticks reserve word is correct then it seems the study is being executed twice on the first tick and sometimes is being skipped on other ticks.

I have also notice that sometimes when you run it you do not even get a bar where the barStatus = 0 exists and this is when you do not get the spacer printout at all. This is why in situations where you want to know the first tick of the bar you have to test Time_S against a PrevTime_S variable which has IntraBarPersist on it as I did in the test study in the post below.

When you run this test, ignore the first bar of the print out because the "diff" should be there. After that it really should not be showing on any lines the way I see it. However it is showing because there is a difference between your count of the ticks for the bar and what the "ticks" reserve word is saying. There is a problem here and TSS should check into it.

Symbol is Emini ES

Code: Select all



Var: intrabarpersist TC(0);
{TradesThisBar, same as your code exactly}

Var: intrabarpersist TTB(0);
{TradeCount, same as your code exactly}

var: marker("");
{used to mark the last tick of the bar in the printout for easier reading}

var: Problem("");
{used in the printout to show when your trade count & the MC ticks disagree}



If LastBarOnChart_s then
Begin
If Barstatus = 0 then
Print( File("C:\ALog_BobsTest.txt"),
" ");

TC = TC + 1 ;
TTB = TC ;

If Barstatus = 2 then
begin
TC = 0;
marker = "2";
end
else
begin
marker = " ";
end;

if TTB <> Ticks then
problem = "Diff"
else
problem = " ";


Print( File("C:ALog_BobsTest.txt"),
" BarNumber", " " ,
BarNumber , " " ,

" Time_s", " " ,
Time_S , " " ,

" Barstatus", " " ,
Barstatus , " " ,

" TC", " " ,
TC, " " ,

" TTB", " " ,
TTB, " " ,

" Ticks", " " ,
Ticks, " " ,

" ", " " ,
Marker, " " ,

" ", " " ,
Problem, " " ,
" ");

End; {If LastBarOnChart_s then }
Last edited by bowlesj3 on 14 Nov 2009, edited 11 times in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 13 Nov 2009

Here is a very good test to study.
I left intrabarstatus off for a few items.
it is run on 10 second bars of the emini ES
There are three print statements in the study.
The first is a spacer to make it easier to read.
The 2nd and 3rd print statements are exactly the same.
the "BN = BarNumber;" is stuck right in between the 2nd and 3rd prints.
This test tells you several things.

1/ to test barstatus = 0 you have to use time because barstatus = 0 is duplicated on the first and second executions of the study within the bar. Either that or the study is being executed 2 times for the first tick as the printout of the test above shows.

2/ When you run it study the BN variables and how the variable during the first tick of a new bar has the old barnumber value until the move command "BN = BarNumber;" takes place. This explains what they mean in the help for the IntraBarPersist command (I copied the help out to the 3rd post above)

3/ It shows that BN and BN[0] are exactly the same thing in memory because the move command "BN = BarNumber;" is only used on BN yet BN[0] gets its value immediately.

4/ Getting back to observation #2 above, notice how the BN is different between the print statements because of the move which is between them. this difference stays the same for every single tick within the bar. However if you put IntraBarPersist on BN it changes and the BN value is only different between the pair of printouts for the first tick. Give it a try (redo it with intrabarpersist and keep both printouts to compare them). This explains what IntraBarPersist does with an actual example. (This gets a little confusing and that is why I think it would be simpler to have a separate set of variables which are simple with no underlying tables).

Anyway, that is what IntraBarPersist does. I think it does work. I think the test from the post above may explain the problem you are having (again, it seems to suggest that the study is executing twice on tick #1 with barstatus = 0 and sometimes it is not getting executed for a tick and this at times appears to be the first tick of the bar too which would explain some of my runs where Barstatus = 0 could not be found for a bar).

Maybe the results from the test above and below puts all this to rest finally.

One more note. Before this test I was thinking that variables without IntraBarPersist get their current bar table value reset to their initial value at the start of the study with each new tick. This test shows their current bar table value gets set to the previous bar's value at the start of the study. This effects some of the statements I made when I created a post suggesting the commands TL_Persist, Arw_Persist, and Text_Persist be created.

Code: Select all



var: marker("");
{used in the first study to mark the last tick of the bar in the printout - make it easier to analyse}


var: IntraBarPersist TimePrev(0);
{needed to be sure we have a new bar since BarStatus = 0 is not trustable}

var: BN(0);
{BarNumber, a variable to store bar number across ticks}
{study it in the print out carefully to fully understand how inclusion and exclusion of IntraBarPersist effects your run.}
{In other words run this study once with BN not having IntrabarPersist then run it again with BN having Intrabarpersist and study the differences between the printouts very carefully}


If LastBarOnChart_s then
Begin

marker = "";
If Barstatus = 2 then
marker = "LastTick";
if Time_s > TimePrev then
marker = "FirstTick";
TimePrev = Time_s;


Print( File("C:\ALog_A_BobTestNoIntraBarPersist.txt"),
" ");

Print( File("C:\ALog_A_BobTestNoIntraBarPersist.txt"),

" Barstatus", " " ,
Barstatus , " " ,

" BarNumber", " " ,
BarNumber , " " ,

" BN", " " ,
BN, " " ,

" BN[0]", " " ,
BN[0], " " ,

" BN[1]", " " ,
BN[1], " " ,

" BN[2]", " " ,
BN[2], " " ,

Marker, " " ,

" ");

BN = BarNumber;

Print( File("C:\ALog_A_BobTestNoIntraBarPersist.txt"),

" Barstatus", " " ,
Barstatus , " " ,

" BarNumber", " " ,
BarNumber , " " ,

" BN", " " ,
BN, " " ,

" BN[0]", " " ,
BN[0], " " ,

" BN[1]", " " ,
BN[1], " " ,

" BN[2]", " " ,
BN[2], " " ,

Marker, " " ,

" ");


End; {If LastBarOnChart_s then }
Last edited by bowlesj3 on 14 Nov 2009, edited 10 times in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 13 Nov 2009

It occured to me that my test results are under MC 2.1.999.999. Someone should run the first test code under the latest version to see if the problems with barstatus = 0 and the ticks reserve word are occuring there too. If they are not occuring then it would be interesting to find out if your workspaces are running light (very few charts and very few studies).

Someone might want to total up the internal count of ticks versus the ticks command count of ticks at the end of each bar to get a running total over a period of hours to print out. Maybe the totals and their difference at the end of each hour could be written to a file. That might raise some interest.

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Postby RobotMan » 14 Nov 2009

SUPER, Wow - I wish I could even get close to the results you got! It would be totally acceptable because it is so consistent not only from bar to bar but also between each subchart. I hope that I can find resolution for this as I am very frustrated with this situation. Maybe Andrew can shed more light on this issue than Marina did. Not holding my breath.

John,
Thank you so much for all the work you have put in! I will need to test your ideas out further during live data conditions and it will probably take a day or two for me just to understand where you are now. I honestly am not a programmer, so it takes me a little longer. But I am very anxious to use MC to do more meaningful statistical research on Volume.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 14 Nov 2009

Hi Bob,

Yah, I realize you need live ticks to run my tests. It is too bad this did not come in on Thursday night. I double checked the compile and you can just drop them in, compile, assign them and do the run and go to the C root directory to find the log.

I forgot to mention that my first study is basically your study but I shortened the names to make the print statement not so wide. I went back and put comments in at the top to explain the variables better. My study just adds a bit of analysis in a detailed form to isolate the problem versus yours which is a batch run which compares totals to signal that there is a problem. In other words my variation does a tick by tick compare to find it.

Here is my concern after analyzing my test results. I think MC is counting ticks correctly in the "ticks" reserve word and I think that some of these ticks are not triggering the execution of the study. It is a theory and TSS should get in on it since it is a bit of a jaw dropper. In my case I don't use volume at all and being a discretionary trader I don't care if the odd tick is dropped due to a bug (or to speed up the charts to capture only the highs and lows and close of the bar during a very fast market) but for systems traders if my theory is correct this could be a major problem. What I am hoping for is the "ticks" reserve word is in fact incorrect and the study is being executed on every tick coming in. Hey, if the market is too fast for MC (or I have overloaded MC with charts and studies) I would rather get a faster computer. Now if in reality markets can get so fast that MC does not have a hope in hell of keeping up well an option for smart "ticks bypass" may just be a good option the trader can choose to use (but the trader needs to be aware of it).

I will probably eventually put run stats into the first study and keep this in my 10 second bars for analysis over a few weeks or during fast markets. I will have it picked up in my database program.

John.
Last edited by bowlesj3 on 14 Nov 2009, edited 4 times in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 14 Nov 2009

Hey Bob,

I have written a simple study to test and hopefully prove that IntraBarPersist does work. It is below. It is a variation on your study that updates a GV in parallel with your study's count but it does not zero the count which has IntraBarPersist associated with it. It will total all day and issue print when/if they go out of sync. We will be sure either way after that run as to whether intrabarpersist works or not. If you want to make bets send me a private email :D

After that is done, If my theory that there is no IntraBarPersist problem is true and the theory I have that the "study is not getting executed on every tick" is true, then I think you will need to change the subject of this thread or maybe I will create a new one and copy some of the test code into it.

It would be interesting to (as a last resort) study the TWS api to see if I can get the tick count from there and somehow get it from MC and detect a difference.

Again if my theory turns out to be correct, this idea of skipping study execution during high volume if the user has a lot of charts and studies is interesting. It would be handy to have an "ExitStudy" command that tests a GV up front to determine if extremely high volume is present and shortcuts need to be taken just so the charts can keep up. The GV could be updated by the first study that MC processes which determines if this tick is a new high/low within the bar or a close within the bar (and thus decide how important it is for charting purposes). It could also run stats on volume and use this as a factor as to whether it should change the GV to signal other studies that they should be processing differently or maybe even bypassed all together. In my case I could make a manual decision just watching the charts and set the GV via the database program.

John.

Code: Select all

{This Studytests the IntraBarPersist command.
If IntraBarPersist is working properly TC and TickCnt should never differ
unless the MC variable TC and the GV Named Variable TickCnt have different upper limits.}

Input: AlertFrequency(100); {this limits the number of alerts you get to every 100 TC <> TickCnt errors}

Var: intrabarpersist TC(0);
Var: TickCnt(0);
var: TickCntAlertLimit(0);

if CurrentBar = 1 then
Begin
DefineDLLFunc: "C:\Program Files\TS Support\MultiCharts\GlobalVariable.dll", int, "GV_SetNamedInt", lpstr, int ;
DefineDLLFunc: "C:\Program Files\TS Support\MultiCharts\GlobalVariable.dll", int, "GV_GetNamedInt", lpstr, float ;
value1 = GV_SetNamedInt("TickCnt",0);
value1 = GV_SetNamedInt("TickCntAlertLimit",0);
end;


If LastBarOnChart_s then
Begin
TC = TC + 1 ;
TickCnt = GV_GetNamedInt("TickCnt",0);
TickCnt = TickCnt + 1;
value1 = GV_SetNamedInt("TickCnt",TickCnt);

if TC <> TickCnt then
begin
Print( File("C:\ALog_Test_IntraBarPersist.txt"),
" BarNumber", " " ,
BarNumber , " " ,

" Date", " " ,
Date, " " ,

" Time_s", " " ,
Time_S , " " ,

" Barstatus", " " ,
Barstatus , " " ,

" TC", " " ,
TC, " " ,

" TickCnt", " " ,
TickCnt, " " ,

" IntraBarPersistt Error");
TickCntAlertLimit = GV_GetNamedInt("TickCntAlertLimit",0);
TickCntAlertLimit = TickCntAlertLimit + 1;
value1 = GV_SetNamedInt("TickCntAlertLimit",TickCntAlertLimit);
if TickCntAlertLimit = 1 then
begin
Alert("IntraBarPersist Error.");
end;
if TickCntAlertLimit > AlertFrequency then
begin
value1 = GV_SetNamedInt("TickCntAlertLimit",0);
end;
TC = TickCnt; {Reset them to equal to 1/ limit the prints and 2/ see if there is a pattern to when they go out of sync}
end; {if TC <> TickCnt then}

End; {If LastBarOnChart_s then }
Last edited by bowlesj3 on 15 Nov 2009, edited 1 time in total.

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Postby RobotMan » 15 Nov 2009

Hi John,
I think MC is counting ticks correctly in the "ticks" reserve word and I think that some of these ticks are not triggering the execution of the study.
This is what I believe is true also.

I will be watching the new code execute this coming week and checking the log that the GV creates. No bets yet! :wink:

However, SUPER seemed to have no problems with my code in TS and his MC results did not look bad either. I mean it was usable because it was consistent. I have asked and asked about getting the internal MC code to take care of this problem for years. It is very frustrating not to be able to do intrabar volume studies in real time without jumping through GV hoops or learning another platform's coding (ie: NT).

You would be surprised how little is on my trading charts and the the stuff I do have is simple statistical studies on raw volume. But I do a hell of a lot of research - mostly on raw volume. Not on indicators or oscillators anymore (I just don't use them). So, I have hit the wall on what MC has to offer unless there is a more consistent and reliable way to get code that executes on every tick (like I thought it was supposed to.)

Again, I appreciate the time you have put into writing this code to help better understand this phenomena.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 15 Nov 2009

Hi Bob,
get code that executes on every tick (like I thought it was supposed to.)
For sure all traders using MC are going to assume what you have assumed and I too. In time I hope MC can
1/ report the order that workspaces are executed, order that charts are excuted within workspaces and the order that studies are executed within charts.
2/ allow the trader to actually change the order and get the report again if they want.
3/ and detect when ticks are getting to the computer late using an exchange based time stamps and various adjustable alerts.
4/ and also detect if the studies the user has in place are having trouble keeping up with the rate of ticks coming in (I would purposely write a study that would delay things buy creating a long to process internal loop yet not an out of control loop). I could do this unintentionally too and I would like to know about it some how to decide if maybe I should pass on an idea that is delaying the charts too much (or write code to use the stats that MC provides which would automatically decide to bypass some studies and report it to me via my database).

Someday my princely MC will come :D (For those who do not know "Someday My Prince Will Come" is a song that the late great piano player Bill Evans did).

John.

P.S. Maybe in time I will write a study to apply that will purposely bog down MC studies (with a long controlled loop) and see if the first analysis study I wrote seems to get a lot of gaps in the reserve word "Ticks" indicating the test study is not getting executed on every tick.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 16 Nov 2009

Robotman,

I tried out your code on MC and TS 8.6, I did see inconsistency with regard to MC but not to the scale as you have demonstrated, mind you I had a smaller run on your code.

TS 8.6 showed perfect results....clearly pointing to a bug with MC's intrbarpersit.

Look forward to Andrews view now.
I confirm that we do have the issue. We will try to address it in MultiCharts 6 release (not beta).

aljafp
Posts: 184
Joined: 28 Oct 2005
Been thanked: 1 time

Postby aljafp » 16 Nov 2009

Guys, thanks for putting in the effort to troubleshoot.

I assumed that studies are executed on every tick as long as you set "update on every tick".

I expected this this to be basic functionality. And fundamental to how things are supposed to work in MC.

MC.. please get this fix. Thanks

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 16 Nov 2009

Thanks Andrew for getting in on this post.

This morning I will be finished a study that counts the number of missing ticks per bar and which gives an hourly stats on it. Like I said earlier, I am not sure it is a bad thing in situations where the user has overloaded MC beyond what can be reasonably expected of it. However it is something that needs to be fully out in the open with MC options to tweek how many ticks are skipped so MC can keep its charts up to date (in short the trader sets the priorities for the studies and knows exactly what is going on).

Got a few other things also on the go at the minute.

John.

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Postby RobotMan » 16 Nov 2009

Andrew,
Thanks! You don't know how much I look forward to the release!

John, SUPER,
Thanks for helping out with your time, effort and insight!

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 16 Nov 2009

Andrew,
Thanks! You don't know how much I look forward to the release!

John, SUPER,
Thanks for helping out with your time, effort and insight!
Bob,

John, is a real Gem out here....along with your persistance.....the problem was highlighted and now solution is on its way.

Regards
Super

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 16 Nov 2009

Hi Bob,

Here is an improved version of the test with some simple stats. Cut and paste the code into a new study and it should compile. The two reports are in the root directory.

The Question marks just after "Ticks Diff" shows the real problem and how often it occurs (I counted this one). The very last column points out problems with BarStatus = 0.

Regarding the detail and stats (if you open your files in notepad with a wide screen) it will be all lined up. The stats printout shows the totals every minute (I could not wait for an hour). It could use work but I think it is time to let this fun little project go for now. I have better things to do like trading. Interesting change of pace however.

I have been running the IntraBarPersist test for over an hour and it is working perfectly.

John.

Code: Select all


Var: intrabarpersist TC(0);
Var: intrabarpersist TTB(0);
var: marker("");
var: Problem("");
var: ProblemA("");
var: Problem2("");
var: IntraBarPersist TimePrev(0);
var: IntraBarPersist TotDiff(0);
var: IntraBarPersist TotZero(0);
var: IntraBarPersist BarCnt(0);
var: IntraBarPersist PrevTicks(0);
var: IntraBarPersist Status0Cnt(0);


var: IntraBarPersist CurrentTimePrev(0);




If LastBarOnChart_s then
Begin

TC = TC + 1 ;
TTB = TC ;

if TTB <> Ticks then
begin
problem = "Ticks Diff ";
end
else
problem = ".......... ";

problem2 = "................";
marker = "...";
if Time_s > TimePrev then
begin
Status0Cnt = 0;
PrevTicks = 0;
BarCnt = BarCnt + 1;
if BarCnt = 2 then
playsound("C:\WINDOWS\Media\ringin.wav");
if BarCnt = 1 then
begin
Print( File("C:\ALog_BarStatusZeroAndTicks_Stats.txt"),
" This gives a total every minute. The first row will be incorrect.");

Print( File("C:\ALog_BarStatusZeroAndTicks_Detail.txt"),
" This first row is your header row ");
Print( File("C:\ALog_BarStatusZeroAndTicks_Detail.txt"),
" Bar ", " " ,

" Time_s ", " " ,

" Barstatus ", " " ,

" TC ",

" TTB ",

" Ticks ",

"B/E ",

"Diff ",

"StatusZeroErrs",
" ");
Print( File("C:\ALog_BarStatusZeroAndTicks_Detail.txt"),
" ");
end;
if BarCnt > 1 then
begin
Print( File("C:\ALog_BarStatusZeroAndTicks_Detail.txt"),
" ");
If Barstatus <> 0 then
begin
problem2 = "Status 0 missing";
TotZero = TotZero + 1;
end
else
begin
problem2 = "................";
end;
end;
marker = "Beg";
end;
If Barstatus = 0 then
begin
Status0Cnt = Status0Cnt + 1;
if Status0Cnt > 1 then
begin
Status0Cnt = -999999999;
problem2 = "TooMany";
end;
end;
If Barstatus = 2 then
begin
marker = "End";
TC = 0;
end;
TimePrev = Time_s;

if BarCnt > 1 then
begin
if TickS <> PrevTicks + 1 then
begin
ProblemA = "?";
TotDiff = TotDiff + 1;
end
else
ProblemA = " ";

PrevTicks = Ticks;


Print( File("C:\ALog_BarStatusZeroAndTicks_Detail.txt"),
" Bar", " " ,
BarNumber , " " ,

" Time_s", " " ,
Time_S , " " ,

" Barstatus", " " ,
Barstatus , " " ,

" TC", " " ,
TC, " " ,

" TTB", " " ,
TTB, " " ,

" Ticks", " " ,
Ticks, " " ,

marker, " " ,

Problem,
ProblemA, " " ,

Problem2, " " ,
" ");
if CurrentTime > CurrentTimePrev then
begin
Print( File("C:\ALog_BarStatusZeroAndTicks_Stats.txt"),
" Currenttime_s", " " ,
Currenttime_s, " " ,

" CurrentHour", " " ,
HoursFromDateTime(computerdatetime) , " " ,



" Tot ? Ticks =",
TotDiff, " " ,

" BarStatus 0 is missing=",
TotZero, " " ,
" ");
TotDiff = 0;
TotZero = 0;
end;
CurrentTimePrev = CurrentTime;
end;


End; {If LastBarOnChart_s then }

This first row is your header row
Bar Time_s Barstatus TC TTB Ticks B/E Diff StatusZeroErrs


Bar 30226.00 Time_s 95140.00 Barstatus 0.00 TC 1.00 TTB 1.00 Ticks 1.00 Beg .......... ................
Bar 30226.00 Time_s 95140.00 Barstatus 0.00 TC 2.00 TTB 2.00 Ticks 1.00 ... Ticks Diff ? TooMany
Bar 30226.00 Time_s 95140.00 Barstatus 1.00 TC 3.00 TTB 3.00 Ticks 2.00 ... Ticks Diff ................
Bar 30226.00 Time_s 95140.00 Barstatus 1.00 TC 4.00 TTB 4.00 Ticks 4.00 ... .......... ? ................
Bar 30226.00 Time_s 95140.00 Barstatus 1.00 TC 5.00 TTB 5.00 Ticks 6.00 ... Ticks Diff ? ................
Bar 30226.00 Time_s 95140.00 Barstatus 1.00 TC 6.00 TTB 6.00 Ticks 8.00 ... Ticks Diff ? ................
Bar 30226.00 Time_s 95140.00 Barstatus 1.00 TC 7.00 TTB 7.00 Ticks 9.00 ... Ticks Diff ................
Bar 30226.00 Time_s 95140.00 Barstatus 1.00 TC 8.00 TTB 8.00 Ticks 10.00 ... Ticks Diff ................
Bar 30226.00 Time_s 95140.00 Barstatus 1.00 TC 9.00 TTB 9.00 Ticks 12.00 ... Ticks Diff ? ................
Bar 30226.00 Time_s 95140.00 Barstatus 1.00 TC 10.00 TTB 10.00 Ticks 14.00 ... Ticks Diff ? ................
Bar 30226.00 Time_s 95140.00 Barstatus 1.00 TC 11.00 TTB 11.00 Ticks 15.00 ... Ticks Diff ................
Bar 30226.00 Time_s 95140.00 Barstatus 1.00 TC 12.00 TTB 12.00 Ticks 16.00 ... Ticks Diff ................
Bar 30226.00 Time_s 95140.00 Barstatus 1.00 TC 13.00 TTB 13.00 Ticks 18.00 ... Ticks Diff ? ................
Bar 30226.00 Time_s 95140.00 Barstatus 1.00 TC 14.00 TTB 14.00 Ticks 19.00 ... Ticks Diff ................
Bar 30226.00 Time_s 95140.00 Barstatus 1.00 TC 15.00 TTB 15.00 Ticks 20.00 ... Ticks Diff ................
Bar 30226.00 Time_s 95140.00 Barstatus 2.00 TC 0.00 TTB 16.00 Ticks 20.00 End Ticks Diff ? ................

Bar 30227.00 Time_s 95150.00 Barstatus 0.00 TC 1.00 TTB 1.00 Ticks 1.00 Beg .......... ................
Bar 30227.00 Time_s 95150.00 Barstatus 0.00 TC 2.00 TTB 2.00 Ticks 1.00 ... Ticks Diff ? TooMany
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 3.00 TTB 3.00 Ticks 2.00 ... Ticks Diff ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 4.00 TTB 4.00 Ticks 3.00 ... Ticks Diff ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 5.00 TTB 5.00 Ticks 4.00 ... Ticks Diff ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 6.00 TTB 6.00 Ticks 6.00 ... .......... ? ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 7.00 TTB 7.00 Ticks 7.00 ... .......... ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 8.00 TTB 8.00 Ticks 8.00 ... .......... ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 9.00 TTB 9.00 Ticks 10.00 ... Ticks Diff ? ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 10.00 TTB 10.00 Ticks 11.00 ... Ticks Diff ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 11.00 TTB 11.00 Ticks 12.00 ... Ticks Diff ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 12.00 TTB 12.00 Ticks 13.00 ... Ticks Diff ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 13.00 TTB 13.00 Ticks 14.00 ... Ticks Diff ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 14.00 TTB 14.00 Ticks 15.00 ... Ticks Diff ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 15.00 TTB 15.00 Ticks 17.00 ... Ticks Diff ? ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 16.00 TTB 16.00 Ticks 18.00 ... Ticks Diff ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 17.00 TTB 17.00 Ticks 20.00 ... Ticks Diff ? ................
Bar 30227.00 Time_s 95150.00 Barstatus 1.00 TC 18.00 TTB 18.00 Ticks 21.00 ... Ticks Diff ................
Bar 30227.00 Time_s 95150.00 Barstatus 2.00 TC 0.00 TTB 19.00 Ticks 21.00 End Ticks Diff ? ................

Bar 30228.00 Time_s 95200.00 Barstatus 1.00 TC 1.00 TTB 1.00 Ticks 2.00 Beg Ticks Diff ? Status 0 missing
Bar 30228.00 Time_s 95200.00 Barstatus 1.00 TC 2.00 TTB 2.00 Ticks 2.00 ... .......... ? ................
Bar 30228.00 Time_s 95200.00 Barstatus 1.00 TC 3.00 TTB 3.00 Ticks 3.00 ... .......... ................
Bar 30228.00 Time_s 95200.00 Barstatus 1.00 TC 4.00 TTB 4.00 Ticks 5.00 ... Ticks Diff ? ................
Bar 30228.00 Time_s 95200.00 Barstatus 1.00 TC 5.00 TTB 5.00 Ticks 6.00 ... Ticks Diff ................
Bar 30228.00 Time_s 95200.00 Barstatus 1.00 TC 6.00 TTB 6.00 Ticks 7.00 ... Ticks Diff ................
Bar 30228.00 Time_s 95200.00 Barstatus 1.00 TC 7.00 TTB 7.00 Ticks 8.00 ... Ticks Diff ................
Bar 30228.00 Time_s 95200.00 Barstatus 1.00 TC 8.00 TTB 8.00 Ticks 9.00 ... Ticks Diff ................
Bar 30228.00 Time_s 95200.00 Barstatus 1.00 TC 9.00 TTB 9.00 Ticks 10.00 ... Ticks Diff ................
Bar 30228.00 Time_s 95200.00 Barstatus 1.00 TC 10.00 TTB 10.00 Ticks 12.00 ... Ticks Diff ? ................
Bar 30228.00 Time_s 95200.00 Barstatus 1.00 TC 11.00 TTB 11.00 Ticks 13.00 ... Ticks Diff ................
Bar 30228.00 Time_s 95200.00 Barstatus 1.00 TC 12.00 TTB 12.00 Ticks 15.00 ... Ticks Diff ? ................
Bar 30228.00 Time_s 95200.00 Barstatus 2.00 TC 0.00 TTB 13.00 Ticks 15.00 End Ticks Diff ? ................

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Postby RobotMan » 16 Nov 2009

Hi John,

That is a very good test to show that the internal tick total and the indicator counter do not agree. I am going to hang onto that code because it actually shows *where* the problem occurs.

thanks

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 16 Nov 2009

I learned more than a few things in this thread

Thanks guys!

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Postby Nick » 20 Nov 2009

Here is why things don't match :-

I didnt read the whole thread and only scanned your original code. Having written similar I am quite familiar with what you are doing :). The issue is with CurrentBid CurrentAsk not intrabarpersist. They are read asynchronously. Therefore it is quite possible for them to change whilst you are processing the incoming ticks volume before you get to read them.

This is something I have banging on about for years (though I don't think anyone at TS understand the issue). It would be great if best bid best ask (may as well go for the whole order book) where available synchronously. As it stands all these MD type indicators that are flavour of the day are actually inaccurate on the MC/TS platform.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 20 Nov 2009

Hi Nick,

I agree, For sure there is no intrabarpersist problem and the bids and asks can change totally independed of the trades. However the ticks problem may effect Bob's stuff. A new thread was created for it.

Regarding the ticks, if you study the run of the latest study created for that thread it appears that a lot of ticks are slipping buy with absolutely no execution of the code (I have seen as high as 6 on the emini) - there is no catch up to match the check box that says to execute the code on every tick. This while at the same time 99.9% of the ticks are in exact proper sequence relative to the execution of the study as if there is some sort of direct link (as the checking of the "execute on every tick" check box suggests should be happening). If the studies are doing a catch up then this would cause the ticks command to report them at least as repeats. This does happen but it only happens at the start of a new bar and happens almost every single time. So that thread is a real issue. Andrew said there will be a fix for it but it is a real curiosity at this point.


John.
Last edited by bowlesj3 on 20 Nov 2009, edited 4 times in total.

Dug
Posts: 31
Joined: 26 Aug 2009
Location: London

Postby Dug » 20 Nov 2009

While I don't look at ticks, and i am not a programmer, and hence have a little trouble following the thread.
Thankyou to those guys doing this. As an MC user, anything that can make it bug free, or more accurate is fantastic.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 20 Nov 2009

Hi Dug,

We have to thank Bob for is pesky persistence with his volume related problems. I don't look at ticks either but last friday I looked at Bob's script and thought "the ticks command should match his count right?". The fact that it did not caught my interest so I created the parallel thread about ticks and execution of the study (not volume problems or intrabarpersist which has no problem). More is coming there (a new type of script by next week to let you know what studies, charts, workspaces are executing in what order - easy to install and shut off too).

John.

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Postby Nick » 23 Nov 2009

Hi John, That's not something I had noticed but it is not something I have looked for! Seems to suggest a fairly fundamental architectural issue handling data.

Incidentally the issue with inside bid is not so much that best bid and ask change independently it is that insidebid insideask functions read the current values rather than the value when the ticks occurred.

Edit: I wonder if TS plan to offer some way to synchronously deal with the whole order book? NT and Neoticker both do decent jobs of this if you need ideas how to implement it.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 23 Nov 2009

Hi Nick, your correct regarding the ticks. What gets me is programmers who program compilers are very intelligent (it is not an easy thing to do). For a trader who does not need volume skipping ticks if they are the same price as the prior tick is not a bad idea if the studies are falling behind the rate of tick arrival (there is not way to slow them down in Traders Work Station). However the charting program must keep up and the studies are separate than the chart and they have to do thier best. If slow downs occur the trader needs to know all available options (ways to speed their studies, eliminate studies, check box options in MC to bypass duplicate ticks or maybe an exitstudy command used when the EL code detects a duplicate priced tick and also knows it is falling behind tick arrival times, and of course a faster computer, addware cleanup, etc). I have basically assumed that TSS picked up the rights to an old TS and are making it a better product as MC (and seem to be doing a great job in my opinion - I am very happy with MC). The fact that TS is also missing ticks supports my theory. TSS probably didn't even know of the missing ticks problem at some point. It may have been done to TS years back when computers were slower (again programmers who program compilers are very bright!!!!!). So I think that if it is a speed up feature as opposed to a bug it should be kept as an option and TSS should program MC to provide info to the trader allowing them to audit their studies and make wise choices (traders will love them for this approach I am sure).

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Postby RobotMan » 23 Nov 2009

Hi Nick,
Incidentally the issue with inside bid is not so much that best bid and ask change independently it is that insidebid insideask functions read the current values rather than the value when the ticks occurred.
I don't look at best bid/ask. I was doing some research on trade size frequency and use the reserve words: "Ticks", "Upticks", "DownTicks" to filter on an IntraBarPersist basis. I have already ranted on best bid/ask in previous posts on this thread and only care about actual trades rather than worrying if the trade occurred at the best bid/ask. I figure it evens out over time when using "ticks".
My point is; other software like eSignal and NT do a fairly good job of allowing you to see time and sales in tabular form in real time. The info is there in the data. I was just trying to grab it to do some statistical analysis and could never get consistent results. It has bugged me for a long time. I recently had some ideas that I wanted to test and ended up just seeing if I could get a tick counter to work inside MC. It has nothing to do with best bid/ask - totally independent from it. - This is the basis for John and SUPER's recent research. We postulated 'does our code execute on each tick'. Andrew has explained that, because of multi-threading, MC is not performing as expected in the intrabarpersist mode. John and Super have tested both TS and MC under similar conditions and continue to have questions about this subject.

In the end, MC will be well profiled and better understood resulting in a solid fix as opposed to a "hotfix" that relies on quick assumptions. The more John and Super and others "beat up the software" the better understood the problem and the better the solution will be. MC is an awesome charting package and Andrew has been very responsive to user input - we all want it to succeed.

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Postby Nick » 25 Nov 2009

Hi Bob, Missing ticks of course is a much more fundamental issue than synchronously storing and making available order book updates (the latter won't work properly if the former doesn't anyway). Having said that if they are looking at how ticks are processed may as well look at removing some of the limits imposed by the current architecture. An opportunity to provide other events to the application (e.g. full order book) and the full and synchronous processing of multiple data streams.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 25 Nov 2009

I am preparing the logic and will be coding up a set of 3 studies that will test for missing ticks after TSS gets their fix in. Based upon Super's run, TS has problems with time based bars but processes ticks correctly with tick charts. These studies will be run on both TS and MC to create a file of print statements for every tick out of both programs. At night a manual lineup of the files will be done to ensure the very first few records match. After that a matching record study (no need for an ascending sequence key here) will look for any differences in those files. At the same time we will run the existing test again on the TS tick charts to ensure that it is still working. Regarding the MC output file it will be processed with all charts types (it should not matter because all types should not miss ticks). Again, when TSS gets thier MC6.0 fix in these tests will be run. Once I have completed my initial tests and Super has tested against TS and MC I will put the instructions and scripts out for download (in 2 or 3 weeks).

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Re: Why I don't trust "IntraBarPersist"

Postby RobotMan » 04 Aug 2010

Bumping this thread.

I ran the same test with the following code that I have run before.
I must be doing something wrong, because Andrew said that this would be fixed with the release of Version 6 build 3553. So my question to you MC gurus (Andrew? Anastassia? Dave?) is this. Why does this simple code not work as expected?

Code: Select all

// Format Instrument:
// Set Chart type to "Tick"
// "Build Volume On" is irrelevant
//
// Each Histogram bar of Plot1 should be the same as tick bar Interval

Var: intrabarpersist TradeCount(0);
Var: intrabarpersist TradesThisBar(0);

If LastBarOnChart_s then Begin

TradeCount = TradeCount + 1 ;
TradesThisBar = TradeCount ;

If Barstatus(1) = 2 then TradeCount = 0;

Plot1(TradesThisBar ,"TradesThisBar") ; //set to Histogram

End;
What I am expecting:
1. All histogram bars should be at or near 500.
2. All bars should be the same.

(see the attachments)

"Build Voluime On" is set for ticks just for reference. Ie: The program seems to know that there are upticks (Up Vol) and downticks (Dn Vol) and can keep track of them internally - why can't this code keep track of Ticks?

I don't want an explaination of how IBP works. I want to fix the code so it works. When my air conditioner is not working and it is 95°F inside, I don't want a discussion of thermodynamics or Boyle's Ideal Gas Law. I just want it fixed and my room to be 75°F.

Do I need to restrict this program to one core of my CPU? Do I need to turn off Hyperthreading (it is AMD, so I don't know what it is called)?

So. Does anyone know what am I doing wrong?
Attachments
TradeCounter01.jpg
(91.93 KiB) Downloaded 8407 times
TradeCounter02.jpg
(86.87 KiB) Downloaded 8386 times
TradeCounter03.jpg
(90.67 KiB) Downloaded 8399 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why I don't trust "IntraBarPersist"

Postby bowlesj3 » 05 Aug 2010

Hi Bob,

Yes Andrew said it would be fixed for MC 6.0 official release but Dave said that they could not get a 100% fix due to the fact that a full fix would require profound changes to MC. Unfortunately I can't find the quotes for this quickly at this exact moment (I think I set them aside somewhere figuring I may need them). My test runs show that they did something to greatly improve it.

Yes your histogram shows bars less than 500 (all of them and a fair bit lower). I guess on BarStatus=2 MC detects its own internal tick count is at your 500 ticks amount and thus MC finalizes the price bar and ticks histogram bar so you can see it. However your count is much lower because it was not added to by your EL code and thus your bar is short of the 500 you wish to see. So a tick comes into MC, MC detects it and adds one to a count somewhere to keep track of it but your "TradeCount = TradeCount + 1 ;" never gets executed and it falls behind (lower count) and thus the histogram bar is less than 500.

TS does it. MC does it. MC 2.1.999.999 did it more than MC 6.0 beta2.

When I can't understand something I create a very short small data example to clarify it in my mind. Here is my understanding of what is happening.

If 3 ticks occur at the exchange as follows during the bar (lets say 1 minute bar) This is what shows up at the MC doorway.

Price/volume.
1125.00 / 100
1125.00 / 100
1125.00 / 100

The price bars will correctly show open high low close of 1125.00.

However if MC somehow failed to execute your study on the 2nd tick and does however execute your study on the 1st tick and 3rd tick then the volume display will only be 200 rather than the proper display of 300 for the bar. This is exactly what is happening with your histograms. The price bar is perfect but the histogram for volume is wrong.

One question. Why do you have 3 histograms showing?

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why I don't trust "IntraBarPersist"

Postby bowlesj3 » 05 Aug 2010

Bob, I applied your code to 10 second bars then I revised it to count the number of times your count "TradeCount = TradeCount + 1 ;" is not executed by MC for the bar that has just completed. Maybe this is a better way to look at it. Doing it this way you can apply it to minute bars, second bars or any bar type and still see the number of times your study was not executed when it should have been.

Code: Select all

Var: intrabarpersist TradeCount(0);
Var: intrabarpersist TradesThisBar(0);

If LastBarOnChart_s then
Begin
TradeCount = TradeCount + 1 ;
TradesThisBar = TradeCount ;
If Barstatus(1) = 2 then
begin
TradeCount = 0;
if TradesThisBar < Ticks then
begin
Plot1(Ticks - TradesThisBar ,"Missing Executions");
end
else
begin
Plot1(0,"Missing Executions");
end;
end;
End;
I have been running it for a few minutes on the ESU0 (on the 10 second bars) and I am getting very few missed executions. On MC 2.1.999.999 I would have lots.

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 53 times

Re: Why I don't trust "IntraBarPersist"

Postby Tresor » 05 Aug 2010

One question. Why do you have 3 histograms showing?
On the Data Window to the left you will see that ''!TradeCounter'' study has different values for the same bar for each histogram.

My guess is these values should be the same?
Attachments
TradeCounter_bar.jpg
(90.91 KiB) Downloaded 8407 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why I don't trust "IntraBarPersist"

Postby bowlesj3 » 05 Aug 2010

Thanks Tresor,

I don't use the data window at all so I was a little lost there.

You are correct. They should all be showing 500 I figure. But it just goes to show you that MC is missing execution of these studies randomly (rather than what it should be doing which is execute all studies on the chart for every single tick that becomes available to that chart). I bet you that if Bob is to create a histogram of the reserve word "Ticks" (rather than his count) all 3 histograms will show 500 on every bar. This is because the ticks reserve word is returning its value from a common count that is added to independent of the study that is about to be executed (hopefully executed that is).

My question is why are Bob's histograms so bad when My 10 second bar charts are not missing many executions.
Last edited by bowlesj3 on 05 Aug 2010, edited 1 time in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why I don't trust "IntraBarPersist"

Postby bowlesj3 » 05 Aug 2010

Here is a simple way to prove that intrabarpersist is not the problem. Just drop this code into Bob's study near the bottom of the lastbaronchart section (include the variable of course). Now, turn off the study, clear the file that is written to and start the study running. It will write a line for every execution of the study (but it will only do it for one bar). The number of lines should match the histogram (you have to figure out the correct histogram bar so hopefully the time will help you do that). I included Bobs count in the file write to help out. You can read the file with MS-word and use Tools Wordcount to get the number of lines in the file if needed. It should match the count inside the file itself.

Code: Select all

var: Startwrites("N");


If StartWrites = "Y" then
FileAppend("C:\MyFile.txt",NumToStr(TradesThisBar,0)+ " - " + NumToStr(Time_s,0) + newline);
If BarStatus = 2 then
begin
if StartWrites = "Y" then {If we have written for a full bar then stop}
Startwrites = "X";
if StartWrites = "N" then {Start writing only at the start of a bar}
Startwrites = "Y";
end;

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Re: Why I don't trust "IntraBarPersist"

Postby RobotMan » 05 Aug 2010

John said
then the volume display will only be 200 rather than the proper display of 300 for the bar
I am only counting ticks, not volume for each bar. So, in your example the proper answer would be a total of 3 ticks would be displayed by the histogram.

John said
One question. Why do you have 3 histograms showing?
To show the exact same code running on the exact same bar during the exact time gives different results. None of them correct.

Tresor said
My guess is these values should be the same?
Yes. Exactly.

John said
I bet you that if Bob is to create a histogram of the reserve word "Ticks" (rather than his count) all 3 histograms will show 500 on every bar. This is because the ticks reserve word is returning its value from a common count that is added to independent of the study that is about to be executed
I think a viable and solid solution would be that TSS create a reserve word for intrabarpersist volume that uses their own internal data (it is shown in the data window and thus must exist - I mean, how do they create tick bars if they are not keeping track?). Ie: iUpTicks, iDownTicks, iTicks. Additionally, iBidVol, iAskVol when close is at bid or ask respectively intrabar.

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Re: Why I don't trust "IntraBarPersist"

Postby RobotMan » 05 Aug 2010

John said
Here is a simple way to prove that intrabarpersist is not the problem.
IBP is not the issue as far as I can tell. If it sees the tick, it executes. It is just not seeing the tick and I am pointing this out to TSS and asking if I am not correct. Bottom line: My code should work as is.
Either MultiCharts has some further issues that have not been addressed or my computer (AMD 5000 x2 (2.8 GHz), 4 GB ram) is not adequate to process their code.
Last edited by RobotMan on 05 Aug 2010, edited 1 time in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why I don't trust "IntraBarPersist"

Postby bowlesj3 » 05 Aug 2010

I agree Bob.

You seem to be getting really bad numbers in your histogram but I am not getting as bad numbers in my 10 second bars. Do you get bad numbers if you apply it to 1 minute bars or 10 second bars? Try my new variant on your code that reports missing executions rather than the number of executions as your code does. You may find that more useful.

I don't know why your numbers are that bad Bob but here is one reality that needs to be considered. To make my point I need to take an extreme extreme. If a user tries to run 100 workspaces and has 15 charts on every workspace and 30 studies on every chart MC is going to have to start skipping executions to keep up (pretty dam smart programming if it can do this effectively). So it may be that I am not having problems because I am running light (one workspace, one symbol, 8 charts, each chart with 6 studies and only 2 charts that have more) and thus I am not getting many missed executions (translation MC can keep up because I have realistic expectations for MC). Being a programmer I know that programs have limits (programming is not the be all and end all solution to trading in my mind). This is why I will never fully automate as well. I learned most of what I know of the market without any programming facility at all (too extreme the other way). So for me now this is almost a none issue. However it is still interesting. I think TSS could actually market MC's ability to intelligently skip executions when the market ticks are comming extremely fast or the user has overloaded MC. It would need to properly report it and provide a way for the user to audit and fix the situation somehow. (so to me skipping executions it is a strong point actually if it can do that properly - to do it without full disclosure reporting is not good).

Regarding the above paragraph I guess we need to test this problem on a slower computer to see if it gets worse on average (or maybe create a few extra workspaces and run them to compare the numbers.).

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 53 times

Re: Why I don't trust "IntraBarPersist"

Postby Tresor » 05 Aug 2010

Guys,

I am getting very interested in this issue (bug?). Can I trouble you a bit?

Could you please make one more ES screenshot with 3 histograms:
(i) 2 histograms of !TradeCounter study, plus
(ii) 1 histogram of Volume (please select ''Build Volume on: Tick Count'' in Format Instrument - Settings)

At least we will know if your MC versions build bars of equal number of ticks. If the bars are built of equal number of ticks then the volume bars should all have value of 500, if you select ''Build Volume on: Tick Count''.

Do you both have the same data feed?

My MC is not connected to any live data feed at the moment (I trade via MT). I tried your studies in MC, but they appear to work only live on real-time data. If this issue can be tested on historic data, please let me know how. I have lots of historic tick data in csv format, so I can do some testing.

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Re: Why I don't trust "IntraBarPersist"

Postby Nick » 06 Aug 2010

I have noticed some inconsistencies too, they are particularly apparent on multiple data streams. My testing had 3 - bid, ask, and last, of the same instrument. Far too many trades that are not at best bid or best ask.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why I don't trust "IntraBarPersist"

Postby bowlesj3 » 06 Aug 2010

Hi Tresor,
If this issue can be tested on historic data, please let me know how.
I don't think you can test this on historic. All code uses "if lastbaronchart then" meaning ticks need to be coming in after that bar is finally displayed on the chart.

I will reread your post again after the trading day and maybe add more here.

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Re: Why I don't trust "IntraBarPersist"

Postby RobotMan » 06 Aug 2010

I ran Tradecounter again this morning. MultiCharts was the only program running. I closed all workspaces except one. I only had one chart open, the same one seen in the previous .jpg attachments.

1. Histograms all still way, way below where expected - no change from previous results.

2. Using Process Explorer, I looked at System Information and both CPU cores averaged around 1% load, spiking to the occasional 3% maximum for only a second or two. This was during the open.

3. I average around 23 mSec round trip ping time and 0% packet loss to the eSignal Main Ticker Plant (199.217.10.36 ). A very good and solid connection.

I have nothing further to say unless someone has a fix for my code. I await comment from TSS.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why I don't trust "IntraBarPersist"

Postby bowlesj3 » 06 Aug 2010

I can see only 2 differences relative to my setup.
1/ I am using IB's TWS for my feed.
2/ I do not use tick charts (only 10 second bars).

They fixed the search. Here are the threads I started and Super's comment below.

Ticks Missing Test using TS & MC with GV Data Sharing.
viewtopic.php?f=1&t=6919&hilit=missing+execution

New Subject: Studies not getting executed on every tick.
viewtopic.php?f=1&t=6877&hilit=missing+execution
Here's test file run on MC Version 5.5 Beta 2 (Build 2536) and TS 8.6 on symbole ECZ09 on 1 minute and 10 ticks.
TS on tick data works perfectly unlike MC but on the other hand with 1 Minute data MC stands out, though not perfect.
So it appears MC was stronger with Minute bar data and seconds bar data than it is with tick data charts. Based upon my tests with seconds bars, it has improved a lot between MC 2.1.999.999 and MC 6.0 beta 2. However, Super will be using my scripts to compare TS and MC some time in September roughtly.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Re:

Postby janus » 06 Aug 2010

If I had a software company like TSS, I could come out with a module just for scalpers or intraday traders.
(Infact, sometimes I wish MC was modular and you just buy the module that you needed. Just need charting? - get a charting module. Just need Systems development? -get the systems module. Want portfolio analysis? -Portfolio module. Radar Screen Module. Scalper module. Replay order entry module with DOM and depth of market module. Genetic algorithm and neural network module. All add-ons to the central core engine.)
Yes I came to realise too late that MC is not the right package for me as I'm a day trader; sometimes a scalper, at other times a swing trader. I've stayed with MC by working around some of the deficiencies, partly because I like the charting features, and partly because I like the language despite some of odd features that could be handled much better. I'm now thinking of using my own DLL plug-ins to interact with IB directly thus bypassing MC's order management system. This will overcome all of the issues I'm currently having with MC handling real-time orders in a dynamic market. In the end I might even develop my own trading system that does all the work for preparing and submitting orders but still plug into MC to plot the prices and signals. I tend to think MC is designed for portfolio traders, not day or swing traders. I hope they realise this and one day accommodate the other types of users as it's really where I see the market is going. The only drawback for them doing this is they will have to perform some major changes to the way MC operates. For example, a study based on one data series has to be made to update on ticks not just for that data series but also all other data series used by the study. Another major change is to allow a study to run at periodic intervals even when there is no tick update. There are several other major changes that need to be made. I don't see them doing this though as I think TSS is focusing on other areas, which are just as important, such as manual trading, which by the way I'm already doing using my own DLL plug-ins, including the use of manually drawn trend lines to add and remove stops and limit orders.

I have often thought of moving to another platform, such as AB, which does a better job for my style of trading (and I may still do one day) but for now I'm happy to stay with MC as it performs most of what I want provided my programming skills can handle the workarounds. I also have in the back of my mind that MC will continue to evolve and one day will do all the things I need in the years to come; I hope.
Last edited by janus on 07 Aug 2010, edited 1 time in total.

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Re: Why I don't trust "IntraBarPersist"

Postby RobotMan » 06 Aug 2010

Hi Janus,

MultiCharts is the best charting software I have ever used.
Like you, I'm a "scalper".
MultiCharts is not for scalpers.

NT was originally just a DOM and order entry for scalpers.
I switched over to it from J-Trader.
Now it has charting, mediocre drawing tools, scripts, crude back testing and replay, and a bunch of other junk.

I knew the demographic I fit into when I took a chance and bought MC four and a half years ago and started to participate in this forum. I have always juggled development in MC with real trading in NT. I like both very much, but have always hoped that one day I would be able to use just MC as my "go to" program of choice. I realize that it will probably still be a year or two before that happens. Even if MC introduces manual order entry and management, there will still be a beta period.

(And don't get me started about real-time volume analysis.)

Even though I find it would be hard to part with NT's "Chart Trader" and "Automated Trade Management" along with excellent tools like Final's "Market Balance" or the Gomi "Delta Ladder", I want MC to succeed very much and I will continue to wave the flag for them as long as Andrew continues to work hard. I actually miss Marina's hard nosed aggressiveness in the forums.

I agree with pretty much everything in your post. Though, using DLLs is way beyond my understanding unless I have some hand holding. I could never write one - I am not a programmer. And implementing even the simplest of "Global Variables" is hit or miss with me. It seems too "klugey". - I like the way Microsoft Office has all these different programs that are separate, but that they all understand and work with each other seamlessly. You only have to install the ones you need and leave the rest behind.

As far as I can tell, portfolio and risk analysis, could be a whole separate module just like Quote manager and PLE. Wouldn't it be cool if Stan one day announces, "MultiScalper module now available for download for $50/mo with a 30 day free trial". And it included all the great features that NT, ATCtrader, AMPtrader, and X_Trader had, and fully integrated with the awesomeness of MultiCharts Charting. BAM! they'd own the market.

I really like what Andrew has done and hope he continues with the retail side inspite of the big money with the institutions. I, too, hope you stick with MC and continue to give your insights here in this forum. There are a lot of users, but not that many that contribute here like those with the huge presence of corporate moderators like in the NT or TS forums.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Re: Why I don't trust "IntraBarPersist"

Postby janus » 07 Aug 2010

I, too, hope you stick with MC and continue to give your insights here in this forum. There are a lot of users, but not that many that contribute here like those with the huge presence of corporate moderators like in the NT or TS forums.
I will stay with MC for now as I've put in too much time and effort into using the EL coding. I'm not a heavy duty scalper/day trader, not yet anyway. If I become one I'm sure I will either need to move to another platform or develop my own add-ons. One thing is for sure. If MC didn't have the ability to use custom DLLs, I would have left a long time ago. Fortunately, it has allowed me to satisfy most of my needs thus far.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why I don't trust "IntraBarPersist"

Postby bowlesj3 » 07 Aug 2010

Interesting conversation guys.

Having MC modular would be nice. I can think of a few functions I would not use.

Scalping is always an interesting topic to me. I always get interested when someone says MC is not a good scalping program. In that case I must be an "almost scalper" since MC seems 100% perfect for me (LOL, along with my secondary program(s) which have probably 100,000 to 300,00 lines of code for tons of direct and indirect functions to help me out with interactive flexibility, record keeping, etc). Am I a scalper? I have no idea. I do switch long/short a fair bit each day. I trade when the wave I tend to like to trade is suggesting an up or down swing of $100 per contract minimum can be had if I trade it 100% perfectly (and that is all). Slowly I am perfecting the automated code to get good entry prices without missing the trade too often (goal is 1 to 2 units from top/bottom on both ends to leave at least $50 per contract profit before commissions). When in a trade I try to milk it for all it is worth with a constant manual read on the market (not a scalper I guess). Pretty hard to get that 1 to 2 unit goal every time (as would be expected) so if one does not milk it the occasional loss could wipe out 4 of those $50 scalps. Besides why waste that big move on another risk that I may miss getting in on it again. I feel no need for anything beyond MC to achieve this However I tend to forget that other program so I guess in the end I have my own way of working around what MC can't do just like most others. I just accept it and tend to forget that I use MC plus my outside programming. Could MC do everything I do! LOL, In my dreams maybe. TSS will never get MC to do all I am doing during the trading day. But like Janus said that is what the DLLs feature is for (not to mention the GVs too). So my impression is that some scalpers use time and sales and try to scalp 1 spread unit profit. I could never do that (preferring chart pattern reading) and maybe this is why I think MC is just great for my type of scalping (or 1/2 half scalping if that is what it is).

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Re: Why I don't trust "IntraBarPersist"

Postby janus » 08 Aug 2010

I take your point that MC does offer a way out to allow a scalper/day trader to do pretty much everything, even for high frequency trading. However, the issue is that one has to be an experienced programmer to do so and even then has to invest a lot of time. I'm an experienced programmer (over 20 years) I've spent days trying to perform a relatively simple exercise, which in another platform I've used took only minutes. The reason it was simple is because the platform provides complete functions to place, modify and cancel orders directly with the broker exposing all the options that are available by the broker interface. The strategy order nomenclature and procedure in MC (and TS) has so many limitations and strange ways it makes something that is supposed to be simple far too complicated for a scalper or heavy duty daytrader (and in may cases for any trader) who has little or no programming experience. That's the issue I'm having. Perhaps a solution is for TSS to develop a special module for such traders. It would have to be specific to the broker as each one have slightly different approaches to placing and handling orders. In fact the other platform I use only supports IB although other brokers can be accommodated by developing custom DLLs using their very extensive toolkits. I'm almost at the point of doing this for MC myself but don't have the time at the moment - perhaps later.

Why don't I just use the other platform? Two reasons really. One, I've invested too much time and effort into developing studies in EasyLanguage so I don't relish the idea of porting them to another language. Two, I like MC's programmable charting techniques too much to leave them.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why I don't trust "IntraBarPersist"

Postby bowlesj3 » 09 Aug 2010

It is a complex issue.
Many brokers MC can run with.
Many types of traders.
Some trying full automation. Some not.
Special wants of each trader.

Regarding the last one, my control program is designed to tuck in the lower right corner of the right most screen directly under IB's TWS and out of the way of all the charts for maximum viewing. Being a small trader I have no fear of Market Makers coming after my pocky little orders so I can run a bracket in TWS (stop and target exposed to them). I control the bracket via my program with specialty button clicks which are directly tied to my strategy bounce points and various other strategy related ideas (I never touch TWS unless my software approach fails for some reason which happens but is rare). What I am saying is no one can design what I want (part of which is the ability to reprogram very fast for changing neads and ideas).

In the end I think it is pretty hard for TSS to take on a new task direction without getting lots of input to find what most would want. I think their main strategy is to make it easy for TS users to come over to MC and offer extras that make it hard for these users to go back because they would loss too much. So in a way TS is largely in the drivers seat until MC gets a much larger user base than TS.

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Re: Why I don't trust "IntraBarPersist"

Postby RobotMan » 14 Aug 2010

Interesting Thread over at TradersLab
http://www.traderslaboratory.com/forums ... -8311.html
Nice if the rumor was true.

Of course, it would take Andrew or Dave all of 5 minutes to comment on this topic here in this thread. What's up guys?

Is it possible that Stan will one day announce "MultiScalper add-on module now available for download"?

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 53 times

Re: Why I don't trust "IntraBarPersist"

Postby Tresor » 14 Aug 2010

Is it possible that Stan will one day announce "MultiScalper add-on module now available for download"?
Is it possible that Stan (i) will one day announce that MC crew had a look at TradeVec platform mentioned in this thread and (ii) will announce that MC will be from now on built with traders in mind? - yes, I know it's a different kettle of fish.

One of many features written with traders in mind is an excellent Order Tracking functionality with (i) ergonomic ''Exit (position)'' button and (ii) sortable columns (iii) and what not :?:

http://www.tradevec.com/KB/Knowledgebas ... 50050.aspx

Compare it to the clumsy stone-age Tracking feature we have in MC

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Re: Why I don't trust "IntraBarPersist"

Postby janus » 14 Aug 2010

will announce that MC will be from now on built with traders in mind? - yes, I know it's a different kettle of fish.
One of many features written with traders in mind is an excellent Order Tracking functionality with (i) ergonomic ''Exit (position)'' button and (ii) sortable columns (iii) and what not
It's early days yet but I'm sure MC will have a better order tracking feature in future releases. What interests me more though is the first point you make. There are different types of traders. Apart form the approach, that is scalper all the way to long term position trading in stocks, traders have varying degree of programming and trading skills. It's extremely hard to have one single package to satisfy all these different types. If there was such a package everyone would be using it and all others would go broke in quick order. In reality, each platform has it's good and bad points. I know many traders who use more than one package to obtain the best of two or more worlds. Others, like myself prefer to focus on one but keep the option open to move to another if and when necessary. At the moment I'm happy with MC despite some of its failings for my style of trading. Fortunately for me, I have enough programming experience to get around most of the issues using DLLs. I suspect this is not the case for most traders, hence the eternal search for the perfect platform and/or plug-ins. In fact, if I didn't have the programming experience I do have, I would still be searching myself. I feel sorry for those who have little or no programming experience. The search for their ideal platform will most likely be filled with many disappointments. Perhaps all serious traders have to learn how to program first before using these packages.

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 53 times

Re: Why I don't trust "IntraBarPersist"

Postby Tresor » 15 Aug 2010

BTW - Guys I apologise for grumbling and putting the topic off track with my last post. I just culdn't resist.
I know many traders who use more than one package to obtain the best of two or more worlds. Others, like myself prefer to focus on one but keep the option open to move to another if and when necessary. At the moment I'm happy with MC despite some of its failings for my style of trading.
My MC has been lying idle for more than a year. I am trading via MT.
Fortunately for me, I have enough programming experience to get around most of the issues using DLLs.
Yes. I saw how you tackled manual orders. Really impressive stuff :D

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 53 times

Re: Why I don't trust "IntraBarPersist"

Postby Tresor » 16 Aug 2010

Hi Guys,

I ran test with !TradeCounter and !dummy2. It looks much better on my end with Dukascopy data feed.

Either:
(i) only 1-5% ticks (trades) are missing, or
(ii) only 1% ticks (trades) are added.

Regards
Attachments
Duka intrabar.png
(29.35 KiB) Downloaded 8401 times
Duka intrabar time.png
(29.82 KiB) Downloaded 8393 times

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Re: Why I don't trust "IntraBarPersist"

Postby Nick » 03 Sep 2010

The biggest problem in my book is the high precision handling of live and historic data. Sure you can write code to interface to a data provider and store ticks in your own database (MC's 1 second resolution is no good for applications that require accuracy) - however at this stage you have to ask why am I using MC for this task. The thing is without this 'core precision' a lot of higher level functionality (that MC has) is redundant. I do hope that things are re architected sooner rather than later, it will become more difficult and more limiting as time goes by.

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 489 times

Re: Why I don't trust "IntraBarPersist"

Postby Dave Masalov » 01 Feb 2011

Intrabarpersist cannot be trusted ...

So I wrote an indicator that just counts ticks. Every time a tick comes in the counter increments by one. It doesn't matter if "Build Volume On" is set for Trade Volume or Tick Count.

Code:
Var: intrabarpersist TradeCount(0);
Var: intrabarpersist TradesThisBar(0);

If LastBarOnChart_s then Begin

TradeCount = TradeCount + 1 ;
TradesThisBar = TradeCount ;

If Barstatus(1) = 2 then TradeCount = 0;

Plot1(TradesThisBar ,"Trades") ;

End;


I then set it to count on a 500 tick bar chart on 3 separate subcharts, then let it run all day.
Theoretically, each bar stands on it's own but should be identical to the barinterval (which in this case is 500).
The issue is not in "Intrabarpersist", this keyword works fine. The reason of the issue is skipping calculation on some ticks. Let's take 1 second resolution: 10 ticks came, but the study has been calculated only twice - on 5th and on 7th tick. This is a downside of MultiCharts multithreading capability. We have the solution to this issue and it will be implemented in MC 7 Beta: https://www.multicharts.com/pm/viewissu ... _no=MC-155 Thus, the issue of skipping ticks in real-time calculation will be solved.

Regarding InsideBid and InsideAsk, they get values from Status Line. Example 1:

plot1(close);
plot2(last);

On 1 tick resolution, if the symbol is ticking fast, this indicator will have different plots. The reason is that the sources of data in this case are independent from each other: Status Line vs current tick.

"Last" price will be taken from current snapshot of status-line in the moment of calculation.

Example 2:

plot1(last);

On 1 tick resolution, if the symbol is ticking fast, this indicator applied to different charts may have different values. The reason is the parallel calculation of indicators. In one moment of time on one calculation stream the snapshot of status-line will have configuration A. When calculating on the same tick in the other stream, it can have configuration B (updated).

To make sure that the calculation on different charts will be the same, you should use tick data (Close, O, H, L, Volume ...), as indicators calculation is based on the arrival of the new tick.

As for the synchronization issue, ticks are saved in the database in the same sequence as they were received and you can recall tick data in the correct sequence if you use one data series on your chart.
However, if you plot several data series on the same chart synchronized with each other, there is no way to know in which order they should be placed – they can all come in one second. The solution is to use sequencing ID and millisecond or microsecond granularity. This will be implemented in MC 7 Beta: https://www.multicharts.com/pm/viewissu ... _no=MC-156

User avatar
RobotMan
Posts: 375
Joined: 12 Jul 2006
Location: Los Altos, California, USA
Has thanked: 31 times
Been thanked: 13 times
Contact:

Re: Why I don't trust "IntraBarPersist"

Postby RobotMan » 01 Feb 2011

This is AWESOME!!!!!

As cutting edge as MC is, to me, this is the biggest news of the last 5 years.
I realize the difficulty in grabbing last bid and ask in real time and that it would take a computer and data stream working at sub-internet connection ping-time granularity. However I always noticed that my eSignal (now IQfeed) and NT seemed to have a Time and Sales ticker window running where they already had grabbed the data and tabulated it. I have been baffled why MC could not do the same internally and make it available to Powerlanguage (IntraBarPersist) access.


Thank you Dave, Stan, Anastassia, and Andrew! :)

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Re: Why I don't trust "IntraBarPersist"

Postby Nick » 01 Feb 2011

Super Awesome!!! I'm rather excited, which is a sad state of affairs over a piece of software.

User avatar
geizer
Posts: 375
Joined: 16 Jun 2008
Has thanked: 40 times
Been thanked: 38 times

Re: Why I don't trust "IntraBarPersist"

Postby geizer » 01 Feb 2011

Thanks Team,
Very encouraging news.

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Re: Why I don't trust "IntraBarPersist"

Postby SUPER » 01 Feb 2011

Excellent news, keep up the good work TS-Team.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Re: Why I don't trust "IntraBarPersist"

Postby janus » 01 Feb 2011

Great news. Glad to see MC finally is listening to us and taking action on such a major weak point in the software. Keep up the good work.


Return to “MultiCharts”