Walk forward /Matrix explained for an idiot

Questions about MultiCharts and user contributed studies.
ScottishSpeculator
Posts: 47
Joined: 03 Jan 2023
Been thanked: 3 times

Walk forward /Matrix explained for an idiot

Postby ScottishSpeculator » 01 Feb 2023

Hey, guys I have read the basics on walk forward but perhaps TJ or Henry or other experience people can explain how MC uses the theory in its software. Been playing around with the matrix optimisation. But not sure about the efficiency setting and how interpret the matrix results.

So I really don't understand this walk forward efficiency setting. You would think on the robust page if you set this high then you are only going to see the input parameters for a given strategy that behave similarly in IS and OOS. I mean that is my understanding I could be wrong. But when you look into a pass box of Runs vs OOS% you see many examples of IS making a profit and OSS losing money and vice versa so how can they be regarded as similar behaviour and a pass?

I also don't understand why the boxes use the average results to determine a pass or fail on robustness when surely you are looking for one strategy input which pass across a range of runs vs OOS%. It would seem you can have good strategies within a failed box just because the average has failed. Why wouldn't you just have an output which shows the strategies which pass all of the Runs vs OOS% ? Isn't that what we are trying to find to prove robustness.?

Attached is a picture of what I am looking at a very simple matrix output. I set the robust pass of walk-forward efficiency to 75%. I have clicked on one of the boxes which "pass" yet when you look at IS net profit vs OOS net profit most 7/10 are opposite behaviour so I don't understand how you could say they behave similar or am I missing something obvious here?
Attachments
Matrix walkforward example.JPG
Snapshot of walkforward matrix output
(116.63 KiB) Not downloaded yet

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Walk forward /Matrix explained for an idiot

Postby Svetlana MultiCharts » 02 Feb 2023

Hi ScottishSpeculator,

Strategy Efficiency parameter shows how the backtesting results differ from the OOS results. It is calculated in percents and normalized by an interval.
Example: If during the strategy optimization the best Net Profit is 100k in a year, and the strategy performance of the next quarter is 20k, then Robustness is calculated the following way:
Net Profit in a quarter = 100К / 12 * 3 = 25К;
Efficiency = 20К / 25К * 100% = 80%
It is calculated according to average Net Profit values of the IS and OOS.
In your example, the average values are IS = 1879.93 and OOS = 222.53.
The picture does not show what IS intervals you have; but if you normalize the ISS averages taking into account the interval and calculate, then most likely it will satisfy your criterion.
We see that the results are poorly correlated by Net Profit. Correlation is -0.67.
Would you like us to add an additional criteria for Robustness — Correlation? If not, please offer a different Efficiency formula.

ScottishSpeculator
Posts: 47
Joined: 03 Jan 2023
Been thanked: 3 times

Re: Walk forward /Matrix explained for an idiot

Postby ScottishSpeculator » 02 Feb 2023

Hello thanks for the reply , ok I think I understand where this efficiency Calculation is coming from now. Its the avg profit from IS and compares with average OOS. over same time length interval. Correct? In that picture I was using 20% OOS data.

It would be useful to have the average IS printed somewhere and yes the correlation as you mentioned as a robust setting. Because it seems to me you could have one or two runs in which the OOS makes quite a bit of profit turning the average from negative to positive but in reality if 8/10 are showing opposite behaviour in terms of +/- profit to the IS results then its not really doing a good job or is robust . Instead of average perhaps comparing median is is better . Or how many runs both pair up in terms of positive or negative profit.

What's your thoughts?

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Walk forward /Matrix explained for an idiot

Postby Svetlana MultiCharts » 03 Feb 2023

It would be useful to have the average IS printed somewhere
It can be displayed in the bottom of the report:
image).png
(45.98 KiB) Not downloaded yet
As for other suggestions, we have forwarded them to the developers.

ScottishSpeculator
Posts: 47
Joined: 03 Jan 2023
Been thanked: 3 times

Re: Walk forward /Matrix explained for an idiot

Postby ScottishSpeculator » 04 Feb 2023

Ok I have also read that the matrix box in which you should select is the one in the middle of a grid of say 9 boxes. Cant find anything on Multicharts saying a similar line of thinking or theory behind picking the best run vs OOS % . Any thoughts on this evaluation process?

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Walk forward /Matrix explained for an idiot

Postby Svetlana MultiCharts » 07 Feb 2023

Ok I have also read that the matrix box in which you should select is the one in the middle of a grid of say 9 boxes. Cant find anything on Multicharts saying a similar line of thinking or theory behind picking the best run vs OOS % . Any thoughts on this evaluation process?
Hi,

The latest comment is not quite clear. Could you please rephrase?

ScottishSpeculator
Posts: 47
Joined: 03 Jan 2023
Been thanked: 3 times

Re: Walk forward /Matrix explained for an idiot

Postby ScottishSpeculator » 07 Feb 2023

I guess for the layman here it isn't clear what you do after running the matrix optimisation.

You run a matrix optimisation test picking a range of run lengths and OOS % and some robust parameters. You get back 20 boxes some are green some are red for example. Then what?
There is no material saying what the logical next step is in picking the best parameter to go live with. Do you pick the single box with the best OOS profit or as this link below is suggesting pick an area with 3 x3 green boxes?
Where is the MC statistical/math advice on running this software and picking optimal parameters after the test?

Still not 100% convinced that Walk forward is better than running Monte Carlo simulations. It seems just like another optimisation test every run based on an even smaller dataset of one potential "our/history" sequence.

https://strategyquant.com/doc/strategyq ... rd-matrix/

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Walk forward /Matrix explained for an idiot

Postby Svetlana MultiCharts » 09 Feb 2023

Matrix Optimization represents several WFOs.

The meaning of different colors
The table shows the target parameter that’s selected next to the table. Target parameter is calculated as an average value in the OOS column of the corresponding parameter for each cell in the Walk-Forward report.
Cells where the total robustness fails are red.
Cells where the total robustness passes are green.
The farther the value from the minimum target parameter – the more intensive is green color.
The farther the value from the max target parameter – the more intensive is red color.
When you hover the cursor over a cell, a tooltip is displayed, in which all the target parameter values for this cell are displayed.

WFO shows what results inputs, found by the optimization, give in a future interval. The purpose of testing a strategy on historical data is to prepare it for real-time trading thoroughly. Therefore WFO cuts the data series into pieces – one part for optimization (IS), others for applying the results in backtesting (OOS). One can forecast how the strategy behaves in real market by comparing IS and OOS.

Forecast definition
After the last run, optimization is carried out on a data series, the length of which equals to IS (or the entire range, if the calculation is Anchored), and which ends with the latest bar in the chart. The set of inputs obtained as a result of such optimization is a forecast.
You can select the best parameters according to your preferences.

ScottishSpeculator
Posts: 47
Joined: 03 Jan 2023
Been thanked: 3 times

Re: Walk forward /Matrix explained for an idiot

Postby ScottishSpeculator » 09 Feb 2023

Ok, I get the logic of walk forward although I am not convinced it improves live trading yet. It was more the forecast part and determining which cell to pick going forward.
So in theory you would look for the cell with the highest profit and then go look at the parameters within that cell which are used in the forecast. Go "live with this"

What are your thoughts on the link from strategyquant. Regarding you need an area in which the cells are all green around each other to say the strategy is robust then pick the cell in the middle of that to use.
Thanks

o2cool
Posts: 9
Joined: 29 Nov 2022
Has thanked: 1 time
Been thanked: 1 time

Re: Walk forward /Matrix explained for an idiot

Postby o2cool » 17 Mar 2023

If you want to do meaningful robustness testing you need to use something else besides MC. The robustness testing tools in MC range from bad to incomplete.

For example, WF and WFM is missing basically all but one of the normalized efficiency metrics (which are really important for WFM testing). There should be efficiency calculations for max intraday drawdown, return/drawdown ratio, win percentage, avg trade, profit factor, etc. As well, there should be a metric to measure the performance of the original strategy vs. each run as given by a percentage etc.

The custom fitness function doesn't work for any of this as there are no keywords for time. This means you can't normalize metrics between the IS and OOS periods. Besides that there is only space for one custom function.

The Monte Carlo testing isn't useful for anything. You can't apply any pass/fail metrics, you can't calculate risk of ruin, you can't even get a chart that shows net profit for each confidence level to define the cone.

Try out StrategyQuantX for robustness testing. They have all of this stuff included and more. Especially they have System Parameter Permutation which is basically a better type of WFM test.

ScottishSpeculator
Posts: 47
Joined: 03 Jan 2023
Been thanked: 3 times

Re: Walk forward /Matrix explained for an idiot

Postby ScottishSpeculator » 17 Mar 2023

O2cool , yeah I did put a feature request in for more functionality with Monto carlo. I have somewhat gone off walk forward recently as I don't see how it gets around the curve fitting, just smaller periods of curve fitting.

- I think ill just do sensible backtesting looking for robust stable ranges on the 3D plots . Then I pull that data out and use in Python for monto carlo simulations of 5000 runs and then look at the median values to get a sense of the strategy.

o2cool
Posts: 9
Joined: 29 Nov 2022
Has thanked: 1 time
Been thanked: 1 time

Re: Walk forward /Matrix explained for an idiot

Postby o2cool » 17 Mar 2023

IMO you might be asking for trouble with that approach.

One of the ideas behind WFM is to INTENTIONALLY try to curve fit your strategy as much as possible with your inputs and input ranges on IS data. If these ideal settings still work on OOS data, then chances are your system is modelling true market behavior and not just the data (and is therefore robust). If your systems ideal settings do not work on OOS data, then the IS portion was probably curve fit and not robust. WFM repeats this same test over and over again for each run with the data you have. Just backtesting and picking stable areas is really the definition of curve fitting.

ScottishSpeculator
Posts: 47
Joined: 03 Jan 2023
Been thanked: 3 times

Re: Walk forward /Matrix explained for an idiot

Postby ScottishSpeculator » 17 Mar 2023

o2cool, sure but you would always keep some OOS data at the end to run on or see how it performs before going live. I feel a monte carlo showing median values or max/min of potential 5000 outcomes seems like a reasonable approach. Would be interested to hear other people's methods.

The thing I can't really understand about WF is for every run you're going to have different input parameters "optimal" from the IS which then run on OOS as you move forward in the dataset.
Ok I understand your looking for good OOS performance but then you are left with 10 different input parameters if you have 10 runs are you not? Still not sure how you decide to go ahead with final input parameters.

o2cool
Posts: 9
Joined: 29 Nov 2022
Has thanked: 1 time
Been thanked: 1 time

Re: Walk forward /Matrix explained for an idiot

Postby o2cool » 28 Feb 2024

The focus is more on testing the strategy rather than the inputs themselves. For each different data segment, the optimal inputs will vary. Additionally, the choice of optimization criteria (e.g., net profit or return/drawdown) affects the inputs selected. This means that no matter what inputs you choose, introducing new data will almost certainly render them suboptimal. A robust strategy should be adaptable enough to perform well across a broad spectrum of suboptimal inputs. The walk-forward test evaluates this adaptability. It identifies the optimal inputs based on your selected criteria, then applies new data where those inputs may not be ideal, revealing the outcome. If your strategy remains effective across various inputs, it's likely capturing genuine market behavior. Conversely, if it underperforms with new data, it might have been overfitted to past data.

This underscores that the selection of inputs, while necessary, is secondary to the quality of the strategy itself, provided the inputs are chosen reasonably. The exact method for selecting input values can vary—some might use recent data, others a broader data range, and some may optimize for net profit vs. other metrics, etc. Over time and with experience, you will develop an intuition for selecting inputs that best suit your strategy, acknowledging that inputs will need periodic adjustment when new data is introduced.


Return to “MultiCharts”