Mysterious line deletions explained & TL_Persist needed!

Questions about MultiCharts and user contributed studies.
bowlesj3
Posts: 2016
Joined: 21 Jul 2007
Has thanked: 171 times
Been thanked: 402 times

Mysterious line deletions explained & TL_Persist needed!

Postby bowlesj3 » 09 Oct 2009

START: 2013/Feb/24th Update.
Regarding this post, a new [RecoverDrawings = false] parameter, has been created that is inserted in your power langauge scipt to solve this issue. See this link for more details.
viewtopic.php?f=1&t=11974&p=59597#p58425
END: 2013/Feb/24th Update.





Here is a suggestion but before I begin I want to mention that the first paragraph properly explains what is actually causing the mysterious line deletions (and lines moving). It is very easy to misunderstand this problem and blame the line delete command but that is not the problem.

Mysterious line deletions explained:
For anyone new to this problem here is a simple example of what is happening. Lets say the user has study1 and study2 that both create lines on the same chart. The last bar on the chart being processed at the moment has 10 ticks to process. Study1 has an if statement that triggers a TL_New command on tick 1 to create the new line. At the very start of tick2 coming into that same study MC automatically deletes the line that study1 created then later study1's if statement is triggered and the study creates the line again and captures the line #1 ID in a variable which will be used later to delete the line using TL_Delete. Tick 3 comes through and MC as usual automatically deletes study1's line it just created but this time the if statement used to execute the TL_new command is not tripped and it is also not tripped for any of the remaining ticks in that bar (that is the problem). Study 1 is finished creating the line but MC has no way of knowing the user wanted to retain this line with the use of only two TL_New commands rather than having to keep reexecuting the command on the remaining ticks until the end of the bar when barstatus = 2. It makes a lot of sense that a new user will not realize that MC deletes the line on every new tick up until the end of the bar for one reason (from a user point of view it makes no sense for MC to delete the line yet it does just that and it happens so fast the user can not see it occurring). Okay to continue the example study2 has been creating its line with line ID #2 with this same add-a-line delete-a-line add-a-line battle between the user code and MC process. However now that study 1 is no longer creating a line on every tick, MC now thinks that ID #1 is now available for study 2. Yep the user has stored ID #1 in the variable but now study #2 is using line ID #1. Later study 1 decides to delete the line #1 thinking it should still be there but it is not and it deletes line #1 which study 2 is using now (that explains your mystery delete/move of the wrong line - well the second part of it - I will summarize later). The only solution that avoids this problem is for the user to keep executing TL_New on every single tick until the end of the bar when barstatus = 2 and only then should they store the ID for the line in a variable. So if the code that actually gives rise to creating line #1 only occurs on tick #1 through #9 of the 10 ticks in that bar, then the user has to trick their code into thinking the TL_new command is suppose to keep executing until barstatus = 2. However the user only has to do this because of this automatic line deletion flaw in MC's EL which I just explained. Also note that two TL_new commands in one study will cause the same problem if they do not keep getting executed until barstatus = 2. To summarize, there are two types of bad deletions that can occur here. 1/ The line may appear for a while during the initial bar then disappears if the TL_new command is not continued until the end of that initial bar. This is because of the MC automatic deletion at the start of each new tick until barstatus = 2. and 2/ the study has stored that ID number of the line that MC automatically deleted and the study later has a delete command executed that deletes another line which has later used this ID number. Note that if you are using this incorrect ID to move lines you will see the wrong line moving (maybe you are using that ID to change a line's color or size or whatever. It is the same problem.). Also note that it does not matter how far back the lines are being placed on the chart. The problem still exists. If you want to test for yourself and see that MC is deleting Text on every tick up until barstatus = 2 you can use the code in the 3rd posting below. Is is the test code which I used to first determine that it was doing this.

To fix this problem I am requesting that new commands be created that do not automatically delete these drawings at the start of each new tick. Below are 5 reasons I am requesting TL_Persist, Arw_Persist & Text_Persist commands.

Reason 1/ Most who are new to MC, will not realize that MC keeps automatically deleting on a new tick any drawings previously created by TL_new Arw_new and Text_new up until barstatus = 2 (meaning the last tick in the bar) and thus they will likely have lots of risky code to deal with that tends to delete the wrong lines/arrows/text. This problem helps no one.

Reason 2/ TSS can not change the existing commands (TL_New, Arw_New, Text_New) to immediately retain the lines/arrows/text because it would create lots of duplicate lines/arrows/text with the existing user code which was written to keep issuing these commands up until barstatus = 2.

Reason 3/ anyone who has tried correcting this problem by replacing their delete commands with SetLocation commands used to set the line/arrow/text location to zero will find it does not work. It will just move the wrong line to zero. If it does work then something else is going on in that example.

Reason 4/ The existing commands make the users logic work more complex than it needs to be. These new commands I am suggesting would make things work as they should work (reducing user code bugs).

Reason 5/ Having the drawing retained immediately allows the user to immediately use the drawing interactively rather than having to wait for the bar to complete.

So why not consider creating three new commands called TL_Persist Arw_Persist and Text_Persist which do immediately retain the drawings (they do not get deleted by MC on the next tick). That way the users can learn the difference and gradually convert their old code to properly handle the new commands when they have time. If they happen to have code that made the assumption that the lines persisted immediately as I was doing for a long time before I finally figured out what was going on (there was no help when I bought MC actually) then maybe all they have to do is switch to the new command. Of course it would be helpful if the help is revised to make reference to the other commands and explain the differences and what can happen if the old version commands are used incorrectly.

Thanks,
John.
Last edited by bowlesj3 on 24 Feb 2013, edited 61 times in total.

bowlesj3
Posts: 2016
Joined: 21 Jul 2007
Has thanked: 171 times
Been thanked: 402 times

Postby bowlesj3 » 09 Oct 2009

Regarding the problem described in paragraph #1 in the previous post here is how one would find the problem TL_new statement if you do not already suspect one. For every TL_new command that has a related deletion command for that line create a print statement that writes to a file immediately after the TL_new command (a new file for each one is best). Make it a standard format with date,time,LineID,barstatus. Now write another program to process these files and loop through the records in the sequence they were written out (date/time sequence) until there is a break in line number. At this point you have to have stored the previous line number and the previous bar status and have them available for inspection. Inspect these previous values and if the barstatus is not #2 then that means your last write for that line number was barstatus 0 or 1 and thus if you did a delete with this line number it will have deleted a line from a different TL_New command. You have to go back to this TL_new command and somehow get it to keep creating lines until barstatus = 2. If your code is complex it could be hard to do this.

Here is the simple logic for this scan process just incase the paragraph is not clear.

Code: Select all


Start:
    read first record.
    set PrevID to first record ID.
    set PrevBarStatus = record BarStatus
ReadLoop:
    read next record.
    if end of file goto done.
    if RecordID not equal to PrevID then
           If PrevBarStatus not = 2 then report this as an error;
    set PrevID to record ID.
    set PrevBarStatus to record BarStatus
    goto ReadLoop.
Done:
     If PrevBarStatus not = 2 then report this as an error;


Most who do not have a fair amount of programming experience would have trouble figuring this problem out. After reading these two posts to this thread I suspect that TSS will agree that creating a TL_Persist, Arw_Persist, Text_Persist would be a good idea. The user should not have to make their code more complex (trick their code) into constantly issuing unneeded drawing commands such as these just because of this MC EL flaw.

bowlesj3
Posts: 2016
Joined: 21 Jul 2007
Has thanked: 171 times
Been thanked: 402 times

Postby bowlesj3 » 10 Oct 2009

A simpler test would be to not issue the TL_New command if barstatus = 2. Your line should disappear from the chart. However you need a switch to stop it from creating another line on the next bar.

Code: Select all

{This code can be used by new users to prove that MC's is deleting Lines, Text, arrows on every new tick until barstatus = 2. Study it then put it on a new minute bar chart.}

variables:
   IntraBarPersist DateLoc(0),   
    IntraBarPersist TimeLoc(0),   
    IntraBarPersist MyCount(0),
    IntraBarPersist TextID(0);

if LastBarOnChart then
   begin
   if MyCount = 0 then
      begin
        TimeLoc = MinutesToTime(TimeToMinutes(Time) + 10);
      if BarStatus < 2 then
         begin
         {
         Here it puts out a new one on every tick so it gets
         blanked out on the next tick
         }
         value1 = Text_New(Date,TimeLoc,Close,Numtostr(MyCount,0));
         value2 = text_setstring(Value1,NumToStr(Value1,0));
          end
       else
          begin
         {
         Here it puts out a new one on every tick again but since
         it is the last tick it can stop putting out new ones
         You will notice that it stops following the close for
         a little while but it is still out front
         }
           TextID = Text_New(Date,TimeLoc,Close,Numtostr(MyCount,0));
         value2 = text_setstring(TextID,NumToStr(TextID,0));
          MyCount = MyCount  + 1;
            end;
      end;
   if MyCount > 0 then
      begin
        MyCount = MyCount  + 1;
        if MyCount = 20 then
           begin
           {
           This command is to prove that it has held it by moving
           it back at tick 20 while the next bar is still being built.
           }
           TimeLoc = MinutesToTime(TimeToMinutes(Time) - 10);
              value1 = Text_SetLocation(TextID,Date,TimeLoc,Close);
              end;
        end;
   end;
   
Last edited by bowlesj3 on 11 Oct 2009, edited 3 times in total.

bowlesj3
Posts: 2016
Joined: 21 Jul 2007
Has thanked: 171 times
Been thanked: 402 times

Postby bowlesj3 » 11 Oct 2009

To TTS,
Because the TL_Persist command does not exist, when a user starts issuing more than one TL_New command per chart in a complex code situation and especially if they have to immediately get the line ID, it appears that the extra code below is what is needed to avoid the mysterious line deletions/moves that start occuring if the user does what naturally makes sense (that being just issuing the TL_new command and logically assuming MC will not delete the line on the next tick). As you can see from the code below it becomes really obvious that providing a TL_Persist command would save a lot of users a lot of trouble. IMO this is a lot of extra work that the user should not have to do.

Code: Select all

{Note: the "build volume on" drop down box
 has to be set to "tick count" It is found in
 the settings tab of the symbol for your chart.}

variables
       {create variables if needed to store
        your parameters so you can keep calling the
        Place1Line function until barstatus = 2}
       Line1_ID(0),
       Place1LineTicks(-1);

if LastBarOnChart then
       begin
       {
       tons of very complex code
       }
       If I should call place1line right now and capture the ID then
                begin
               {store the parameters needed to keep
                issuing Place1Line until barstatus = 2}
               {if this variable is 0 or greater it allows the
                place1line function below to start issuing
                the TL_Keep command}
                Place1LineTicks = 0;
                {this code has to be executed here because
                 the Line1_ID is needed now.}
                Line1_ID = place1line(parameters);
                end;
      {
      tons of very complex code, some of which uses the Line1_ID.
      }

      {A series of calls to keep issuing
       TL_New, Arw_New, Text_New until the end of the barstatus = 2}
       value1 = Place1line(Parameters that were stored);
      {notice that I do not store the line1_ID here}

      end; {if LastBarOnChart then}


Code: Select all

{this is the Place1Line function}

   {Place1LineTicks will be a reference parameter or a GV}
     if Place1LineTicks = -1 then
          exit function; {to TSS, Please include an exit function}

     {this IF prevents issuing the command twice
      within the same tick}
     {I am not sure if it really matters other than saving MC work}
     if Place1LineTicks = Ticks then
              exit function
     Line1_ID_Temp = TL_New(parameters);
     if Line1_ID <> 0 and Line1_ID_Temp <> Line1_ID then
              error: ID has changed.;
     Line1_ID = Line1_ID_Temp;
     Place1LineTicks = Ticks;


    {print to a file for checking later by a batch edit program
     to ensure the last TL_New for the bar was
     issued during barstatus = 2}
    print "date,time,Line1_ID,barstatus"
    {Note: you might want to also print the tick count
              and the call location to study how this is working}

    {My TL_New, Arw_New, Text_New wrapper functions all have
     code which allow them to delete a line using the Line ID,
     A parameter "C"=create is changed to "D"=delete
      to delete the line later.
     I have not shown this code here because
     it is outside the issue
     regarding requesting a TL_Persist command.
     }

    {this command stops TL_Keep from being issued in the next bar}
     If barstatus = 2 then
            Place1LineTimeTicks = -1;

Place1Line = Line1_ID; {send the ID back out of the function}
Last edited by bowlesj3 on 12 Oct 2009, edited 1 time in total.

SUPER
Posts: 622
Joined: 03 Mar 2007
Has thanked: 99 times
Been thanked: 80 times

Postby SUPER » 11 Oct 2009

bowlesj3,

"For sure endurance and perseverance combined to win in the end".

bowlesj3
Posts: 2016
Joined: 21 Jul 2007
Has thanked: 171 times
Been thanked: 402 times

Postby bowlesj3 » 11 Oct 2009

Absolutely. However this is the proper way to think of it. When TSS finally gets a chance (in their priority list) to make these commands available (which I am sure they will) everyone wins including TSS. Maybe TS isn't currently ahead of MC on this issue but I bet the other competitors are.

bowlesj3
Posts: 2016
Joined: 21 Jul 2007
Has thanked: 171 times
Been thanked: 402 times

Postby bowlesj3 » 12 Oct 2009

I have to laugh at this. There seems to be some sort of inconsistency in this barstatus = 2 thing. I am creating a pair of studies to be applied to price (subchart #1) and RSI (subchart #2). I am creating them in two phases. Both studies were created last week for phase 1 which is fairly simple and only the RSI had the vanishing lines and only occasionally. Over the weekend I completed the addition of more code to get them ready for phase 2 testing. There was very little change to the phase 1 code (mostly addition of new code). Now it seems that this vanishing line problem is in full force on the price study and is not effecting the RSI study that I have noticed yet (probably about the same as last week). So I put in a print of the barstatus on the price line TL_new command and sure enough the lines vanish when the write is done on barstatus = 1 rather than 2 (I have forced the write to occur only once and the print statements prove this is working - just like I suggested in the prior posts above, why would I want to keep repeating it just because of this problem). So in other words it is the first deletion type (that being that MC is deleting the line on the next tick). So now the really big question is "why was it not happening last week?" It should have been happening last week.

1/ Does this only happen if the code gets over a certain size?

2/ Does it only happen if you have more than one TL_New command?
This week the code has some more TL_new commands (two more in a function) and two setlocation commands for one of the new lines. Maybe that has something to do with bringing this problem out. There are no delete commands.

I have to ask myself why any of my studies work. So I guess this suggests that the batch process I developed in post #2 above for checking these TL_new commands should be eventually done to try and figure out if sometimes these lines actually do stick without barstatus = 2 and why?

It just occured to me that I guess I have to try the phase1 price code again (which I kept exactly as I had it last week) and see if it still works okay.

For this phase 2 version I guess I have to use the logic above to get this to work by forcing these lines to keep writing until the end of the bar.

bowlesj3
Posts: 2016
Joined: 21 Jul 2007
Has thanked: 171 times
Been thanked: 402 times

Postby bowlesj3 » 12 Oct 2009

No, I just applied the price subchart code which I was running last week and this week it is removing the lines if the TL_new command does not occur when barstatus = 2. It is anyone's guess as to why it was not doing this last week. Nothing has changed that I am aware of but obviously something is different.

bowlesj3
Posts: 2016
Joined: 21 Jul 2007
Has thanked: 171 times
Been thanked: 402 times

Postby bowlesj3 » 12 Oct 2009

I just fixed the problem of the vanishing lines I am currently having by creating a "VL_Needed" swtich, setting to "Y" where the original TL_New command was being issed and then taking the TL_New command down to where it would not be blocked by any if statements and having it execute on evry single tick as below until barstatus = 2 at which point is is switched off. I checked and there are no duplicate lines on the chart so MC is deleting them if barstatus is not 2. It is just lucky that none of the parameters were changed and the ID is not needed earlier. In some situations is may not be this easy. One line fixed. Two more to go.

Code: Select all


if VL_Needed = "Y" then
   begin
   LVL_ID[X] = TL_new_s(date,Arw_Active_Time,LineStart,Date,Arw_Active_Time,LineEnd);
   if BarStatus = 2 then
      VL_Needed = "N";
   end;


SUPER
Posts: 622
Joined: 03 Mar 2007
Has thanked: 99 times
Been thanked: 80 times

Postby SUPER » 12 Oct 2009

bowlesj3,

Please do try to remember that barstatus(1)=1 in MC is not same as TS.

http://forum.tssupport.com/viewtopic.ph ... ght=#26855

bowlesj3
Posts: 2016
Joined: 21 Jul 2007
Has thanked: 171 times
Been thanked: 402 times

Postby bowlesj3 » 12 Oct 2009

Hi Super,

I never actually had the real time TS. Just EOD way back.

The code in the above post which has that "VL_Needed" variable is working for my current study I am working on. Unfortunately I wrote a lot of studies when I first got MC that have TL_New commands buried in very complex code and inside functions. While I was writing them things seemed okay except a line would vanish or move strangly every so often but not so much so that it was all that significant so I never really understood why. Eventually I finally hit a road block because of this problem and I just could not get a study to work at all. That led to my creating some tests where I discovered that the lines were being deleted until BarStatus = 2. Knowing what I know now I am amazed the code I first wrote actually works. Getting that complex code fixed so this does not happen could be a lot harder than doing what I did just above with the "VL_Needed" variable.

In this post and a few others I created the elaborate writeup(s) hoping it would help someone new avoid learning it the hard way as I did. Also hoping TS support would take note of it.

I think TS support knows of the problem but maybe are not as aware as they could be that it really should be fixed with some new commands. My guess is they problably would need to have the New commands flag a database entry for the line indicating that MC should not delete the line at the start of a new tick within the bar.

John.

bowlesj3
Posts: 2016
Joined: 21 Jul 2007
Has thanked: 171 times
Been thanked: 402 times

Re: Mysterious line deletions explained & TL_Persist needed!

Postby bowlesj3 » 24 Feb 2013

START: 2013/Feb/24th Update.
Regarding this thread, a new [RecoverDrawings = false] parameter, has been created that is inserted in your power langauge scipt to solve this issue of new drawings being removed on every tick until BarStatus=2. See this link for more details.
viewtopic.php?f=1&t=11974&p=59597#p58425
END: 2013/Feb/24th Update.


Return to “MultiCharts”