Incorrect backtesting results with SetProfitTarget?

Questions about MultiCharts and user contributed studies.
Xyzzy
Posts: 162
Joined: 19 Mar 2011
Has thanked: 43 times
Been thanked: 79 times

Incorrect backtesting results with SetProfitTarget?

Postby Xyzzy » 18 Nov 2011

I've coded a strategy to trade the S&P 500 e-mini. It uses a bracket order like this:

Code: Select all

If ... then begin
SellShort ("Partial Up SE") next bar at close limit ;
end ;

if ... then begin
Buy ("Partial Down LE") next bar at close limit ;
end ;

SetStopContract ;
SetProfitTarget( 225 ) ;
SetStopLoss( 350 ) ;
The profit target of $225 represents a gain of 4.5 points for the e-mini, while the stop loss of $350 represents a loss of 7 points. In my strategy properties, I've set a commission of $2/contract, and $0 for slippage.

When I'm backtesting this, I'm seeing two results in the list of trades that don't make sense to me. I've attached an Excel file with the performance report for reference.

First, the strategy hits the profit target numerous times during backtesting. However, each time it exits with a "Profit Target" exit, it shows a profit of $233.50 -- not the $225 that is coded in the strategy. For example, on June 24, 2003, it shows a long entry at 1067.25 followed by a "Profit Target" exit at 1072, for a gain of 4.75 points or $233.50. I.e., it is essentially showing positive slippage of $12.50 (one tick for the e-mini).

Second, the strategy only deducts commissions from the trades when hitting the stop loss, not when hitting the profit target. For example, in the above June 24, 2003 transaction, it shows a profit of exactly $233.50 (4.75 points), without deducting $4.00 for the round-trip commission. However, when it hit the stop loss in a transaction on June 27, 2003, it shows a loss of $354 -- i.e., the $350 stop-loss amount plus the $4.00 round-trip commission.

Note, however, that Strategy Performance Summary page shows the correct total amount of commissions. It reports "Commissions Paid: $3000," based on 750 round-trip trades (i.e., $4.00 per round-trip). Does this mean that the Strategy Performance Report might be double-counting some commissions -- including them as a loss in the stop-loss calculations, plus including them a second time in the Strategy Performance Summary?

I ran this strategy during automated trading on Interactive Brokers this morning, and it hit the profit target. I ended up with the expected results -- a gain of $225 per contract, less a round-trip commission of about $4.00 per contract. However, when I disabled the automated trading and instead looked at the backtesting results, the strategy likewise showed a gain of $233.50 for today (i.e., $12.50 higher, with no commission). Consequently, the backtesting results are not matching reality.

Can anyone explain why these two issues are happening? Are these bugs, or are they expected behavior? If it's expected behavior, is there a way to improve the code so that the backtesting results are more accurate? Thanks in advance.
Attachments
@ES# _ _Bean Gap Fill 1 Back-Testing Strategy Performance Report.pdf
PDF of trade listing
(42.83 KiB) Downloaded 149 times
@ES# _Bean Gap Fill 1 Back-Testing Strategy Performance Report.zip
(1.41 MiB) Downloaded 98 times

Xyzzy
Posts: 162
Joined: 19 Mar 2011
Has thanked: 43 times
Been thanked: 79 times

Re: Incorrect backtesting results with SetProfitTarget?

Postby Xyzzy » 18 Nov 2011

By the way, here's the complete strategy code for reference. It's based on a strategy from the book "Seven Trading Systems for the S&P Futures" by David Bean (highly recommended).

For this strategy, Data1 and Data2 are both set to the ES continuous contract. Data1 is a two-minute chart with a custom session of 08:30-15:00 exchange time (Central time). Data2 is a 390-minute chart with the same custom session (i.e., the equivalent of a daily bar for the same custom time period).
Attachments
_Bean Gap Fill 1.xml
(5.29 KiB) Downloaded 394 times

vking
Posts: 235
Joined: 21 May 2009
Has thanked: 51 times
Been thanked: 41 times

Re: Incorrect backtesting results with SetProfitTarget?

Postby vking » 20 Nov 2011

Xyzzy - I have observed the same issue with another strategy as well but haven't reported.

In my opinion - This is incorrect calculation and should be fixed.

Thanks.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Incorrect backtesting results with SetProfitTarget?

Postby TJ » 21 Nov 2011

I've coded a strategy to trade the S&P 500 e-mini. It uses a bracket order like this:

Code: Select all

If ... then begin
SellShort ("Partial Up SE") next bar at close limit ;
end ;

if ... then begin
Buy ("Partial Down LE") next bar at close limit ;
end ;

SetStopContract ;
SetProfitTarget( 225 ) ;
SetStopLoss( 350 ) ;
The profit target of $225 represents a gain of 4.5 points for the e-mini, while the stop loss of $350 represents a loss of 7 points. In my strategy properties, I've set a commission of $2/contract, and $0 for slippage.

When I'm backtesting this, I'm seeing two results in the list of trades that don't make sense to me. I've attached an Excel file with the performance report for reference.

First, the strategy hits the profit target numerous times during backtesting. However, each time it exits with a "Profit Target" exit, it shows a profit of $233.50 -- not the $225 that is coded in the strategy. For example, on June 24, 2003, it shows a long entry at 1067.25 followed by a "Profit Target" exit at 1072, for a gain of 4.75 points or $233.50. I.e., it is essentially showing positive slippage of $12.50 (one tick for the e-mini).

Second, the strategy only deducts commissions from the trades when hitting the stop loss, not when hitting the profit target. For example, in the above June 24, 2003 transaction, it shows a profit of exactly $233.50 (4.75 points), without deducting $4.00 for the round-trip commission. However, when it hit the stop loss in a transaction on June 27, 2003, it shows a loss of $354 -- i.e., the $350 stop-loss amount plus the $4.00 round-trip commission.

Note, however, that Strategy Performance Summary page shows the correct total amount of commissions. It reports "Commissions Paid: $3000," based on 750 round-trip trades (i.e., $4.00 per round-trip). Does this mean that the Strategy Performance Report might be double-counting some commissions -- including them as a loss in the stop-loss calculations, plus including them a second time in the Strategy Performance Summary?

I ran this strategy during automated trading on Interactive Brokers this morning, and it hit the profit target. I ended up with the expected results -- a gain of $225 per contract, less a round-trip commission of about $4.00 per contract. However, when I disabled the automated trading and instead looked at the backtesting results, the strategy likewise showed a gain of $233.50 for today (i.e., $12.50 higher, with no commission). Consequently, the backtesting results are not matching reality.

Can anyone explain why these two issues are happening? Are these bugs, or are they expected behavior? If it's expected behavior, is there a way to improve the code so that the backtesting results are more accurate? Thanks in advance.
What is the resolution of your backtest?
Did you use backtest precision?

Xyzzy
Posts: 162
Joined: 19 Mar 2011
Has thanked: 43 times
Been thanked: 79 times

Re: Incorrect backtesting results with SetProfitTarget?

Postby Xyzzy » 21 Nov 2011

TJ,

Thanks for asking. I normally backtest this strategy on two-minute bars, without tick precision.

However, I just re-ran the backtesting under two different conditions: (1) on one-tick bars, and (2) on two-minute bars with the bar magnifier enabled. In both cases, the same thing occurred -- the strategy incorrectly reported a gain of $233.50, even though the SetProfitTarget was set to $225.00.

This happens every time it hits the profit target on the ES. This isn't a case of the price suddenly gapping past my profit target, so that I would actually get some favorable slippage. Rather, it just ignores the actual SetProfitTarget and reports an extra tick of profit.

I'm not sure how tick precision should matter for backtesting purposes in this case. If my code sends a SetProfitTarget order for $225.00, then it should report a profit of $225.00 as soon as the strategy hits the profit target. Instead, it's off by this one tick.

Xyzzy
Posts: 162
Joined: 19 Mar 2011
Has thanked: 43 times
Been thanked: 79 times

Re: Incorrect backtesting results with SetProfitTarget?

Postby Xyzzy » 21 Nov 2011

By the way, I realized that my math was wrong about the second issue above concerning commissions. In fact, MultiCharts is correctly deducting the $4.00 commission from the Profit Target orders -- i.e., it's calculating a total of $237.50 in profit, less $4.00 in commission, which equals the $233.50 reported profit.

However, this doesn't explain the first issue -- why it's calculating $237.50 in profit (less commission) for a SetProfitTarget order of $225.00.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Incorrect backtesting results with SetProfitTarget?

Postby Henry MultiСharts » 22 Nov 2011

The amount of profit/loss plus comission should be an integer value set in points without decimals.
Profit target + Comission =$225+$4=$229.
The price step on ES symbol is $12.5.
$229/$12.5=18.32 points
18.32 points are rounded to 19 points.
19 points x $12.5= $237.5
$237.5-$4 comission=$233.5
The profit is calculated correctly using the specified values.

If you want to set 18 points profit you need to use SetProfitTarget( 221 ) ;
ProfitTarget+Comission/price step=221+4/12.5=18
Last edited by Henry MultiСharts on 28 Nov 2011, edited 1 time in total.

Xyzzy
Posts: 162
Joined: 19 Mar 2011
Has thanked: 43 times
Been thanked: 79 times

Re: Incorrect backtesting results with SetProfitTarget?

Postby Xyzzy » 22 Nov 2011

Henry,

Thank you for the response. I think I understand now what is happening. However, this seems to be incorrect behavior for how SetProfitTarget is determined in backtesting.

In actual live trading, the SetProfitTarget keyword appears to set a limit order for the amount of profit before commission. For example, if I have the statement "SetProfitTarget(225)" in my strategy for the ES, it will set a limit order for 4.5 big points (18 ticks), which equals $225 in gross profit before commission. My net profit will be $225 - $ 4 commission = $221.

This is how my strategy actually performs in live trading through IB. Last week, the strategy placed a short order at 1219.75, and then covered the short at 1215.25 through SetProfitTarget, for a gain of 4.5 points or $225.00. My actual net profit was $225 - $4 = $221.

However, in backtesting, the SetProfitTarget keyword appears to generate a limit order for the amount of profit after commission. I.e., the statement "SetProfitTarget(225)" will attempt to generate a profit of $225 or higher after commission. To do this, it simulates a limit order for 4.75 points (19 ticks), which equals $237.50 in gross profit. The calculated net profit will be $237.50 - $4 commission = $233.50.

Is my understanding correct? If so, then the calculated results in backtesting won't match the actual results in live trading. I.e., the SetProfitTarget order will perform differently in backtesting then it will in live trading.

If so, can this be fixed, so that the SetProfitTarget keyword in backtesting matches the actual trading results?

vking
Posts: 235
Joined: 21 May 2009
Has thanked: 51 times
Been thanked: 41 times

Re: Incorrect backtesting results with SetProfitTarget?

Postby vking » 22 Nov 2011

Agree with Xyzzy on this.

SetProfitTarget - Shouldn't take commission calculations into consideration to determine the target. it should work independent of the commissions defined - and only this way historical and realtime results are going to match in this space.

Thanks

Xyzzy
Posts: 162
Joined: 19 Mar 2011
Has thanked: 43 times
Been thanked: 79 times

Re: Incorrect backtesting results with SetProfitTarget?

Postby Xyzzy » 22 Nov 2011

Thanks. For what it's worth, the strategy generates the correct backtesting results if I set commissions to zero in strategy properties (except that it doesn't calculate commissions, of course). When I set a commission, then the backtesting result do not match the actual results.

Put another way, the SetProfitTarget keyword only seems to take commissions into account during backtesting. During live trading, it ignores the commissions.

vking
Posts: 235
Joined: 21 May 2009
Has thanked: 51 times
Been thanked: 41 times

Re: Incorrect backtesting results with SetProfitTarget?

Postby vking » 22 Nov 2011

I used the same work around as well in the past and continue to use it that way as the current way of including commissions in SetProfitTarget - I don't think it's the correct approach.

Thanks.

User avatar
JoshM
Posts: 2195
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1544 times
Been thanked: 1565 times
Contact:

Re: Incorrect backtesting results with SetProfitTarget?

Postby JoshM » 26 Nov 2011

If so, can this be fixed, so that the SetProfitTarget keyword in backtesting matches the actual trading results?
I personally don't use the SetProfitTarget keyword, but a possible workaround (untested) might be:

Code: Select all

Variables:
myProfitTarget(225), myPositionSize(1);

{
The value of '1' from the 'GetAppInfo(aiRealTimeCalc)' below occurs when
"the calling application's calculations are based on real-time data".
In the example below, the commission is added to the profit target when there is
real-time data (or PlayBack mode), while the commission is not added when there
is no real-time data (i.e. backtesting mode).
}
if GetAppInfo(aiRealTimeCalc) = 1 then
value1 = myProfitTarget + (Commission * myPositionSize)
else
value1 = myProfitTarget;

SetProfitTarget(value1);
Regards,
Josh

Xyzzy
Posts: 162
Joined: 19 Mar 2011
Has thanked: 43 times
Been thanked: 79 times

Re: Incorrect backtesting results with SetProfitTarget?

Postby Xyzzy » 27 Nov 2011

Josh,

Thanks very much for the help! This is a great idea. I made a few modifications and tested it.

First, when in backtesting mode, the commission needs to be subtracted from the "real" profit target amount, rather than added. It also needs to use double the commission, since the commission amount is for each transaction, and it needs to subtract the full round-trip amount.

Second, I'm setting the stop loss amount on a per-contract basis using SetStopContract. Consequently, I don't need to multiply the commission again for each contract. (If someone is using SetStopPosition instead, they might need to modify the code below.)

Third, the code only needs to determine the amount of the profit target once, rather than on each bar, so I enclosed it in a "Once" block for performance reasons.

Here's the code I ended up with:

Code: Select all

Inputs:
myProfitTarget(225) ;

Variables:
ProfitTargetAmount( 0 );

Once begin
ProfitTargetAmount = IFF( GetAppInfo(aiRealTimeCalc) = 1, myProfitTarget, myProfitTarget - Commission * 2 ) ;
end ;

...

SetStopContract ;
SetProfitTarget( ProfitTargetAmount ) ;
This backtests correctly. The upshot is that if the code is in backtesting mode, it sets the profit target to $221 (i.e., $225 minus the $4 round-trip commission). If it's in live mode, it sets the profit target to $225. This way, the live mode with match the backtested results.

Henry, I still view this issue as a bug. The SetProfitTarget keyword doesn't perform the same way in backtesting as it will perform in live trading, which will give misleading backtesting results (unless you use this workaround). Can you please consider whether to address this? I can submit a bug request if you'd like.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Incorrect backtesting results with SetProfitTarget?

Postby Henry MultiСharts » 28 Nov 2011

This issue cannot be reproduced in our environment.
Please come to our live chat Monday-Friday 6:30 am - 4 pm EST to investigate the issue.
We will do our best to help you.


Return to “MultiCharts”