Exit Discrepancy: Automated vs. BackTested  [SOLVED]

Questions about MultiCharts and user contributed studies.
Doctor Al
Posts: 24
Joined: 15 Aug 2019
Has thanked: 4 times

Exit Discrepancy: Automated vs. BackTested

Postby Doctor Al » 27 Oct 2019

I created a strategy for trading the NASDAQ 100 emini futures on 23 minute price bars. It was running as an automated strategy on 10/24/19.
There was one trade that day which lost $870. The actual entry is fairly close to what shows up in the current back-test (i.e. short entry at 7920.75
in the back-test, while the actual short entry was at 7920 - so 3 ticks of slippage). However, the exit does not agree at all with the back-test. The back-test shows an exit labelled with the designation "ExStp-S2". The Positions History tab of the Orders and Positions Tracker Window shows that the short position was actually entered at 10:06:00 AM. This agrees with the closing time of the price bar before the entry on the chart, and corresponds to the opening time of the price bar on which the entry occurs in the back-test. I do not use Intrabar Order Generation in the strategy. I slightly modified the strategy to print out the stop-loss values bar by bar on the day of interest. According to the strategy code, the stop-loss values will be active on the next price bar - not the current price bar. The output window shows these stop-loss values for the trade:

Time = 1029 BarsSinceEntry = 0 StopLoss_S = 7953.25000
Time = 1052 BarsSinceEntry = 1 StopLoss_S = 7971.25000
Time = 1115 BarsSinceEntry = 2 StopLoss_S = 7985.25000
Time = 1138 BarsSinceEntry = 3 StopLoss_S = 7995.25000
Time = 1201 BarsSinceEntry = 4 StopLoss_S = 8001.25000
Time = 1224 BarsSinceEntry = 5 StopLoss_S = 8003.25000
Time = 1247 BarsSinceEntry = 6 StopLoss_S = 8001.25000
Time = 1310 BarsSinceEntry = 7 StopLoss_S = 7995.25000
Time = 1333 BarsSinceEntry = 8 StopLoss_S = 7985.25000
Time = 1356 BarsSinceEntry = 9 StopLoss_S = 7971.25000
Time = 1419 BarsSinceEntry = 10 StopLoss_S = 7953.25000

The back-test shows the trade being stopped out on the open of the bar following the 14:19 price bar at a price of 7954, which makes sense. Adjusting the printed stop-loss values 3 ticks lower because of slippage of the actual entry price will not change the outcome that the trade should be stopped out on the open of the 11th bar of the trade at 7954.

However, in reality, the trade was not stopped out and continued until 14:42 (2:42 PM), another 23 minutes (the period of another price bar). My recollection is that the chart on that day showed that there was a market exit ("ExMark-S") based on a different exit criteria - if barssinceentry >= NBarEx1 (which is equal to 11) then buytocover next bar at market. The question is why the stop-loss did not work in the real world. I do not understand why? Can anyone explain this? Instead of losing $700 (as in the back-test) , I lost $870 (reality) - 24% more. I really need to determine why this discrepancy occurred.

Here is the exit-related code. I did not remove the declarations of some of the entry-related inputs and variables, but the entry logic is not shown:

Code: Select all

Inputs:
a2(-40),
b2(400),
c2(650);
{ Strategy inputs }
Inputs: Wgt1 (-0.7884), { Neural network weight value (-1 to 1) }
FirstEntryTm (943), { Earliest allowable trade entry time (HHMM) }
LastEntryTm (1505), { Latest allowable trade entry time (HHMM) }
NBarEn1 (15), { Indicator look-back length (bars) }
NBarEn3 (24), { Price pattern look-back length (bars) }
EntFr (2.5632), { Multiple of price difference (e.g., ATR); entry }
NBarEx1 (11), { Number of bars from entry for market exit }
NBarEx2 (30), { Number of bars from entry for market exit if unprofitable }
NATR (59), { Indicator look-back length (bars) }
ATRFrTrail (1.2481), { Multiple of price difference (e.g., ATR); exit }
TrailPct (11.0000), { Trailing stop percentage }
PSParam (1.00), { Position sizing parameter value }
RoundPS (true), { Round-to-nearest (true/false) }
RoundTo (1), { Round-to position size value }
MinSize (1), { Minimum allowable position size }
SizeLimit (50);
{ Variables for entry and exit prices }
Var: EntPrS (0),
SStop (0),
NewSStop (0),
STrailOn (false);

{ Variables for average true range for entry and exit orders }
Var: AveTR (0),
MyBound2(0),
StopLoss_S(0);

{ Average true range }
AveTR = AvgTrueRange(NATR);

{ Exit orders, short trades }
If MarketPosition = -1 then begin
If BarsSinceEntry = 0 then begin
SStop = Power(10, 10);
STrailOn = false;
end;

If EntryPrice - C > ATRFrTrail * AveTR then
STrailOn = true;

If STrailOn then begin
NewSStop = EntryPrice - TrailPct * (EntryPrice - C)/100.0;
SStop = MinList(SStop, NewSStop);
end;

If STrailOn then
Buy to cover("ExTrail-S") next bar at SStop stop;

If BarsSinceEntry >= NBarEx1 or (BarsSinceEntry >= NBarEx2 and C > EntryPrice) then
Buy to cover("ExMark-S") next bar at market;

MyBound2 = a2 * (BarsSinceEntry * BarsSinceEntry) + b2 * BarsSinceEntry + c2;

StopLoss_S = EntryPrice + MyBound2/BigPointValue;

BuyToCover("ExStp-S2") next bar at {(EntryPrice + MyBound2/BigPointValue)} StopLoss_S Stop;

If Date = 1191024 then begin
Print("Time = ",Time:0:0," BarsSinceEntry = ",BarsSinceEntry:0:0," StopLoss_S = ", StopLoss_S:0:5);
end;
end;

If MarketPosition = 0 then begin
SetStopShare;
SetStopLoss(c2);
end;

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Exit Discrepancy: Automated vs. BackTested

Postby TJ » 27 Oct 2019

We don't know your background and experience, so it is difficult to find a starting point for this discussion.

Have you checked out the FAQ in this forum?

Have you read any of these articles?

Relevant articles in Wiki: Fundamentals
- Backtesting vs Live Trading
- Understanding Backtesting
- Why is Data Playback strategy performance different from Backtesting results?
- Intra-bar Price Movement Assumptions

Doctor Al
Posts: 24
Joined: 15 Aug 2019
Has thanked: 4 times

Re: Exit Discrepancy: Automated vs. BackTested

Postby Doctor Al » 28 Oct 2019

Hello TJ,

I have a lot of experience with TS, but am relatively new to Multicharts.
I will look into the Relevant articles and FAQ, but in my opinion you are not answering the question that I have posed in pretty reasonable
detail. If you need more information from me in regard to the specifics of my question please let me know what else I need to give you.

I also have a question:

My question is whether you have actually carefully read my post, looked at the code that I have included in the post, looked at the output of my print statement, and made a significant effort to determine why the stop-loss did not execute.

If you can make a determination as to why the stop-loss did not execute or have a probable speculation as to why, I would greatly appreciate your sharing the information with me.

Thank you so much for your time and effort.

User avatar
rrams
Posts: 128
Joined: 10 Feb 2011
Location: USA
Has thanked: 7 times
Been thanked: 70 times
Contact:

Re: Exit Discrepancy: Automated vs. BackTested

Postby rrams » 28 Oct 2019

Hi Doctor AI,
It requires more than the code to determine what happened. The time and sales for that period and the logs as a start. Was the stop order rejected or just not sent? Did the broker connection or market data stop transmitting at any time?
nq.png
(54.77 KiB) Downloaded 78 times
Also, SetStopLoss(c2); should not be inside a condition.

Doctor Al
Posts: 24
Joined: 15 Aug 2019
Has thanked: 4 times

Re: Exit Discrepancy: Automated vs. BackTested

Postby Doctor Al » 29 Oct 2019

Looking under the "Logs" tab of the Orders and Position Tracker Window, there is no indication of a rejected order for that symbol during that time period. There was a rejected order for that symbol on the prior day (10/23/19) , related to "insufficient margin", but we are discussing a trade that opened and closed on 10/24/19.

Did the broker connection or market data stop transmitting at any time? I see no mention of that having happened in the logs tab of the order and positions tracker window. I am not aware of any such occurrence. There is nothing in the Logs tab that indicates, for example, either a loss of a connection or re-establishment of a connection to my dedicated server during the relevant time period.


After reading TJ's post, viewtopic.php?t=10811 (# 5), I thought of the possibility of a problem related to the last few lines of code, but it would seem that that could , at worst, only effect the entry bar of the trade, since after the entry bar MarketPosition is no longer 0, and there is code (above that) to establish a regular stop-loss if MarketPosition = -1. Below is the section of code, related to the entry bar, that we are talking about:

If MarketPosition = 0 then begin
SetStopShare;
SetStopLoss(c2);
end;

Also, I have the same lines of code in several other strategies, including one strategy in the ten year note futures that currently has an open trade ongoing, and I see no problem showing up with my stop loss order for that trade - the stop-loss order is showing as having been "sent" to my broker
and has a typical "broker order number" in the Orders and Position Tracker Window. At this time, the ten year note futures trade is well past the entry bar. So, I don't think that this section of code explains the problem.

Regarding the "Time and Sales" point, my regular stop-loss never executed - it is not a case where the price moved a great distance from my stop-loss and I got a bad fill. I was taken out of the trade as a result of an entirely different order 23 minutes later [that other order says in English: exit on the open of the next bar, after the 11th bar (where the entry bar to the trade is counted as the "0th" bar and then sequentially count upward each bar].

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Exit Discrepancy: Automated vs. BackTested

Postby wilkinsw » 05 Nov 2019

What mode were you auto trading in?

How did you demonstrate that the backtest disagreed with live trading? Did you copy and paste the live chart, for example?

If trading in SA mode: Try trading in AA mode and see if the discrepancy happens again. That way you remove the possibility that live entry fill slippage and timings isn't causing downstream deviations in outcomes.

Doctor Al
Posts: 24
Joined: 15 Aug 2019
Has thanked: 4 times

Re: Exit Discrepancy: Automated vs. BackTested

Postby Doctor Al » 09 Nov 2019

I saw on the chart after the close of the trade the label "ExMark-S", which is my label (see code above from the initial post for this topic) for a market order exit; there is a different label for stop orders , namely "ExStp-S2" (again, see the code above). Also, the stop-loss result, had it worked, would have been a significantly lower loss then I actually sustained - as per the output of my print statement - repeated here:

Time = 1029 BarsSinceEntry = 0 StopLoss_S = 7953.25000
Time = 1052 BarsSinceEntry = 1 StopLoss_S = 7971.25000
Time = 1115 BarsSinceEntry = 2 StopLoss_S = 7985.25000
Time = 1138 BarsSinceEntry = 3 StopLoss_S = 7995.25000
Time = 1201 BarsSinceEntry = 4 StopLoss_S = 8001.25000
Time = 1224 BarsSinceEntry = 5 StopLoss_S = 8003.25000
Time = 1247 BarsSinceEntry = 6 StopLoss_S = 8001.25000
Time = 1310 BarsSinceEntry = 7 StopLoss_S = 7995.25000
Time = 1333 BarsSinceEntry = 8 StopLoss_S = 7985.25000
Time = 1356 BarsSinceEntry = 9 StopLoss_S = 7971.25000
Time = 1419 BarsSinceEntry = 10 StopLoss_S = 7953.25000

I believe that I now understand why this happened, and how to prevent it from happening again. I will create a new topic - let's say
"empirical observation re complex stop-loss coding" to discuss

Doctor Al
Posts: 24
Joined: 15 Aug 2019
Has thanked: 4 times

Empirical Observation Re Complex Stop-Loss Coding

Postby Doctor Al » 09 Nov 2019

Although what follows seemingly to me does not adhere entirely to the "party line" I have seen on this forum, I have nevertheless found it to be accurate.

It is possible to use the keyword "SetStopLoss" within a condition; however, you need to be very careful when doing so.

Example
Suppose you want to create a stop-loss that will depend on the entry price of your trade. Also, suppose that your strategy does not use intrabar order generation - it creates orders only on the close of a bar. Suppose that you are using 60 minute price bars with closing times of 0900, 1000,
1100, 1200, ... . Now at 0900 you generate an order to enter the market long on a buy stop at a price above the current market. You want a stop-loss to be immediately active from the outset of your trade. Notice that at the time of your entry order the strategy does not have an actual trade entry price - there is a buy stop level, but there could be a gap up that might result in a higher entry price then your buy stop. So, how can you code a stop-loss that will be immediately effective as soon as you enter long (using the estimated entry fill price on the first bar of the trade), but will use the actual entry fill price on subsequent price bars, when that fill price will be known? Let's assume that we will use a 2% stop-loss.

My Suggestion :

Code: Select all

Vars: BuyStopLevel(0),
StopPcnt(2.00),
StopAmount(0);

{ Strategy calculates a value for "BuyStopLevel" and generates an entry order}

{ Add a Stop-Loss}
If MarketPosition = 0 then {This applies to the entry bar only}
Begin
SetStopShare;
StopAmount = (StopPcnt/100 * BuyStopLevel * BigPointValue);
SetStopLoss(StopAmnt);
End
Else
Begin {This applies to bars of trade after the entry bar}
SetStopShare;
StopAmount = (StopPcnt/100 * EntryPrice * BigPointValue);
SetStopLoss(StopAmnt);
End;
-------------------------------------------------------------------------------------------------------------------------------------------------------------
In the above , the Keyword "SetStopLoss" is within a condition (i.e. depending on MarketPosition).

A caution is that you cannot use just the entry bar stop (i.e. the if / then / without the else portion of the statement ) and then attempt to
generate a stop-loss in another section of code. In that case your stop-loss will back-test okay, but it will not be executed in real time.

In other words do not code this

Code: Select all


If MarketPosition > 0 then begin
StopLevel = (1 - StopPcnt/100) * EntryPrice;
Sell next bar at StopLevel Stop;
end;

If MarketPosition = 0 then {This applies to the entry bar only}
Begin
SetStopShare;
StopAmount = (StopPcnt/100 * BuyStopLevel * BigPointValue);
SetStopLoss(StopAmnt);
End;



I believe that the above will back-test okay, but will not work in real time automation - you gave up the ability to insert a stop-loss after the entry bar of the trade by removing the "else" section of the original code. When MarketPosition is no longer equal to zero, a stop-loss can now not be reliably created.
Last edited by Doctor Al on 09 Nov 2019, edited 1 time in total.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Empirical Observation Re Complex Stop-Loss Coding

Postby wilkinsw » 09 Nov 2019

I don’t use setstoploss.

I simply maintain variables equal to entry price next bar.

You can then simply say

“sell next bar at entry_price_var stop next bar” and you will have a stop sent the same bar as entry.

You can look ahead a check if your entry fill was an aggressor at open or not using “open next bar”.

Ie:

for a long stop entry.....

if marketposition<1 then begin
entry_price_var=xxxxxx
if open next bar>=entry_price_var then entry_price_var=open next bar;
End;

sell next bar at entry_price_var - x*ticks stop;


something like that anyway.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Empirical Observation Re Complex Stop-Loss Coding  [SOLVED]

Postby TJ » 09 Nov 2019

Although what follows does not adhere entirely to the "party line" I have seen on this forum, I have found it to be accurate.

It is possible to use the keyword "SetStopLoss" within a condition; however, you need to be very careful when doing so.

。。。
Nobody ever said it is impossible.
If you read the recommendation again, you might save yourself (and other innocent newbies) some precious coding time and trading opportunities.


Return to “MultiCharts”