Walk Forward optimization - results are not coherent

Questions about MultiCharts and user contributed studies.
anfo0000
Posts: 4
Joined: 02 Feb 2015

Walk Forward optimization - results are not coherent

Postby anfo0000 » 10 Jan 2019

Hi,

I have been running several Walk-Forward optimizations in the past days and I have found issues on the data shown as result.

1) For the IS window, the WF optinization results are often different compared with the result I got running the standard otimization for the same IS period. (as optimized parameters have same value and data window is the same, results should match).

2) Taking the optimized paramenters from the WF report and using then in the OOS period, the value of the equity (profit) provided by the FW report is often different from what I got from the Perfomance Report (adain here, as the used optimized parameters have same value and data window is the same, results should match).

This is not evident for long IS data window (i.e. several months), but when I occasionaly realised about the issue, I run the WF with much smaller IS and OOS data window (i.e. 10 days or less), and there the differences are sometime huge.
Not sure if it depends on how MC splits the data samples in IS and OOS (maybe data are split within a given day), but it should not be an issues when I select days as IS window instead of bars.

I am using MC 12.0 (build 17002).

Anyone else experienced the same?

Thanks
Antonello

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Walk Forward optimization - results are not coherent

Postby Svetlana MultiCharts » 16 Jan 2019

Hello, Antonello,

This inquiry has already been forwarded to the engineers. We will get back to you by email once any results are available.

paulc
Posts: 59
Joined: 26 Sep 2012
Has thanked: 13 times
Been thanked: 2 times

Re: Walk Forward optimization - results are not coherent

Postby paulc » 29 Nov 2021

Has there been any update to this question raised by anfo0000?

Thanks.

User avatar
joebone
Posts: 175
Joined: 05 Sep 2018
Has thanked: 53 times
Been thanked: 4 times

Re: Walk Forward optimization - results are not coherent

Postby joebone » 09 Dec 2021

Has there been any update to this question raised by anfo0000?

Thanks.
interested as well

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Walk Forward optimization - results are not coherent

Postby Svetlana MultiCharts » 09 Dec 2021

Hello,

Anfo0000 was using the signal that strongly depended on the initial calculation point, because OpenD, HighD, LowD functions were used. That’s why, the signal, that started calculating at least one bar earlier than session start, produced strongly different result.

If you would like our team to investigate your case, please provide:
- Screenshots demonstrating the issues with inconsistent results highlighted;
- Exported WFO, Optimization and Performance reports;
- Test case to reproduce the behaviour on our end:
1. The workspace where the behaviour is reproduced.
2. Export of used symbols (with data) from QuoteManager in .qmd archive.
3. The exported scripts with all dependent functions that are used on the workspace. Please reproduce the behaviour on simplified signals or pre-built signals if possible.

User avatar
joebone
Posts: 175
Joined: 05 Sep 2018
Has thanked: 53 times
Been thanked: 4 times

Re: Walk Forward optimization - results are not coherent

Postby joebone » 14 Dec 2021

Hello,

Anfo0000 was using the signal that strongly depended on the initial calculation point, because OpenD, HighD, LowD functions were used. That’s why, the signal, that started calculating at least one bar earlier than session start, produced strongly different result.

If you would like our team to investigate your case, please provide:
- Screenshots demonstrating the issues with inconsistent results highlighted;
- Exported WFO, Optimization and Performance reports;
- Test case to reproduce the behaviour on our end:
1. The workspace where the behaviour is reproduced.
2. Export of used symbols (with data) from QuoteManager in .qmd archive.
3. The exported scripts with all dependent functions that are used on the workspace. Please reproduce the behaviour on simplified signals or pre-built signals if possible.

Thats what I thought. I wanted to make sure. Thanks again

paulc
Posts: 59
Joined: 26 Sep 2012
Has thanked: 13 times
Been thanked: 2 times

Re: Walk Forward optimization - results are not coherent

Postby paulc » 15 Dec 2021

Hi Svetlana,

I'd like to ask a more general question if I may.

In relation to the OOS results report in the WF optimisation:

- does the calculation of the OOS P&L shown in the OOS report for each OOS period start from the start time/date of each OOS period or from the start of any trade which is already open from the prior OOS period?
- does the calculation of the OOS P&L shown in the OOS report for each OOS period, end on the end of the OOS period or does it include P&L in the next period(s) if the trade remains open at the OOS period end?
- does the above in any way change based on whether a trade is open from the prior OOS period and would remain open (or be closed) in the current OOS period?


Thanks.

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Walk Forward optimization - results are not coherent

Postby Svetlana MultiCharts » 16 Dec 2021

Hi paulc,

All OOSs calculate independently of each other.

Taking this into account,
- does the calculation of the OOS P&L shown in the OOS report for each OOS period start from the start time/date of each OOS period or from the start of any trade which is already open from the prior OOS period?
Not applicable, as there is no P&L before the first trade of the OOS.
- does the calculation of the OOS P&L shown in the OOS report for each OOS period, end on the end of the OOS period or does it include P&L in the next period(s) if the trade remains open at the OOS period end?
The calculation ends on the end of the OOS period. Its results are not applicable to the next OOSs.
- does the above in any way change based on whether a trade is open from the prior OOS period and would remain open (or be closed) in the current OOS period?
Neither the previous nor the next OOS influence on the current OOS.

paulc
Posts: 59
Joined: 26 Sep 2012
Has thanked: 13 times
Been thanked: 2 times

Re: Walk Forward optimization - results are not coherent

Postby paulc » 16 Dec 2021

Thanks Svetlana for your reply.

To clarify then, since only P&L within the OOS period is included in the OOS P&L result, the OOS trade must be opened and closed within the period (or on the first and/or last bar)?

paulc
Posts: 59
Joined: 26 Sep 2012
Has thanked: 13 times
Been thanked: 2 times

Re: Walk Forward optimization - results are not coherent

Postby paulc » 16 Dec 2021

Hi Svetlana,

To follow up again please, in relation to the image attached:

If the IS optimisation runs across the period indicated by the yellow line and the OOS period is across the red line, where would the trade be opened per the OOS results? Would the trade be opened at point A (at the start of the OOS period) or point B where the strategy parameters identify a new trade?

And secondly, please confirm the OOS P&L calculates the trade P&L until point C; i.e. the end of the OOS period.

Thanks.

Screenshot 2021-12-17 013603.jpg
(116.55 KiB) Not downloaded yet

paulc
Posts: 59
Joined: 26 Sep 2012
Has thanked: 13 times
Been thanked: 2 times

Re: Walk Forward optimization - results are not coherent

Postby paulc » 17 Dec 2021

28 Jan trade WF test.jpg
(1.16 MiB) Not downloaded yet
I have been working on providing an example of how the WF optimisation report does not appear to be providing logical results. In order to do this I have run a WF optimisation using just 1 variable which is not linked to the strategy code. In this way I can run an optimisation report which gives me results for IS and OOS which only uses 1 parameter; i.e. does not change over time. In this way, it is simple to review the period results against a chart knowing the input parameter does not need to change in order to generate results.

I hope this is clear in the attached images which show the same report. You can see in the right hand column all parameter results are 1 across all periods. You can also see on the chart I have set the parameter of the applied strategy to 1 (status line top left). You can see the dates on the chart which correspond to the periods shown on the report.

The results on the chart should therefore match those in the report. The report only shows the OOS periods but you can see in the report these are continuous.

I have checked all other settings to ensure they are consistent - the settings on the chart have not changed since I ran the walk-forward optimisation.

I have also set the OOS periods to 1 day so it is again easy to compare and discuss the results with the view on the chart. There are costs of $2 per contract and $12.50 slippage but these are not material to the discussion.

What I think is happening is the OOS period where a trade occurs includes P&L from trades which are CLOSED in the OOS period and only includes the P&L from those closed trades which occurs in the OOS period.

So taking 28 Jan as an example, we can see the report and chart shows 1 closed trade. The report shows a profit. The chart shows the closed trade itself from 24 Jan is a loss in total but a profit on the day.

Then looking back to 24 Jan, 2 trades are reported as closed in the OOS period and shown on the chart. The reports shows these both as a profit.

If the report is calculating the results on 24 Jan from trades CLOSED in that period then we can see we are talking about the open trade from 20 Jan and the trade (both open and closed) on 24 Jan. Both show a profit so this matches.

But in the OOS period of 24 Jan there was also a long trade opened (not closed) which is not reported in the OOS period of 24 Jan since this trade was not closed on 24 Jan. This trade was closed on 28 Jan at a large loss but the 28 Jan period only shows a profit from the P&L on the day of 28 Jan.

In order to actually use the walk-forward optimisation (indeed the new dynamic self-adjusting parameters!) I need to understand why Multicharts excludes the large loss incurred on the 24 Jan trade whilst it was open.

The OOS sample report for the OOS period seems to assume there is no trade opened in the period unless it was closed in the period and hence does not include P&L for the OOS period from trades open at the end of the OOS period. These trades would occur in real trading if applying the optimisation results.

Question: Where does the WF report show P&L in respect of trades opened in the OOS period?

Question: Why does the WF report include P&L from trades opened in prior periods (P&L for the OOS period) given these trades may not have been opened with different parameter results from the prior period.

Question: Where parameter results do not change (e.g. in the image) and there are no trades closed in a OOS period e.g. 27 Jan, why does the report not include the P&L from this period?

Question: If it assumes the trade was not open then why does it include P&L from the open trade in the OOS period when it is closed?

Question: Why not just show the P&L for the OOS period based on closed, opened and trades (which were open from prior periods but not closed in this period) in the OOS period (where the P&L falls in the period)?

Thanks.

User avatar
Tammy MultiCharts
Posts: 200
Joined: 06 Aug 2020
Has thanked: 6 times
Been thanked: 65 times

Re: Walk Forward optimization - results are not coherent

Postby Tammy MultiCharts » 04 Feb 2022

Hello Paul,

During Walk Forward Optimization it is assumed that no orders were placed before or after each OOS or IS interval.
Here's how WFO works:
The strategy is optimized using a data range from IS Start minus MaxBarsBack value to IS End. The inputs with best performance are then used to backtest the strategy on a brand-new data range from OOS Start minus MaxBarsBack to OOS End. The backtesting results are then presented in the Optimization report.
The calculation is done independently on each interval.
Comparing the trades on the chart with the WFO results is irrelevant.
If you would like to check the WFO results like that, you could use Anchored optimization. In this case the closed trades of IS intervals may be compared with the backtesting results on the chart. However, the OOS trades will not match the trades on the chart anyway, as the starting point of each OOS interval is offset.

paulc
Posts: 59
Joined: 26 Sep 2012
Has thanked: 13 times
Been thanked: 2 times

Re: Walk Forward optimization - results are not coherent

Postby paulc » 07 Feb 2022

Thanks for the reply Tammy.

The issue I still have however is that, as I said in my above post on 17 December, I set the variable to 1 for all WF results. So all of the WF results were set to come out exactly as if fixed parameters were used. In this case the single parameter was set to 1. See screenshot in my post above.

Therefore, with each WF result being a continuous period using a single parameter of 1, why would the results not reflect the trades shown on a chart with the same strategy and parameter set to 1.


The only reason I can think of is that the WF results are not continuous from one period to the next. The dates are continuous as shown in the screenshot which would suggest this is not the answer but otherwise I am still lost.

How can WF results give the same parameter result of 1 in all periods and not reflect the position on the chart with the single parameter set to 1?

Thanks.

User avatar
joebone
Posts: 175
Joined: 05 Sep 2018
Has thanked: 53 times
Been thanked: 4 times

Re: Walk Forward optimization - results are not coherent

Postby joebone » 30 Nov 2022

Hey Paul,

COuld it be your Maxbarsback?

From Tammy's post
The strategy is optimized using a data range from IS Start minus MaxBarsBack value to IS End.


Return to “MultiCharts”