New Monte Carlo simulator

Questions about MultiCharts and user contributed studies.
wilkinsw
Posts: 445
Joined: 21 Apr 2013
Has thanked: 111 times
Been thanked: 74 times

New Monte Carlo simulator

Postby wilkinsw » 05 Jan 2017

Hi,

I've been playing around with the new MC (monte carlo) simulator.

When running on individual signals applied to one instrument, it seems fine. As an example I typically find the 5 percentile worse case drawdown to be about 2-3 times the backtest drawdown. That feels right to me.

However, when I run the MC on the portfolio level, it seems to produce very optimistic 5 percentile drawdown figures. As an example, maybe only 1.25 times the backtest drawdown.

To give more background on the portfolio I'm using: 1 instrument with 3 highly correlated but different signals. The signals are all the product of optimisation using a fitness function featuring max drawdown as a criteria. So how can the 5 percentile drawdown figure, from MC resampling be so close to the backtest base drawdown figure? In fact it shows the base backtest drawdown figure as itself being just below the 20 percentile! That can't be right when max drawdown has been optimised?? You'd expect is to be at least the 50 percentile?

Please correct me if I'm wrong, but I believe what is happening is the following........

The MC simulator is reshuffling trades without any regard for if the trades happened at the same time or not. Therefore, when running multiple signals the simulator is actually helping to artificially smooth out run ups and drawdowns, skewing results optimistically. It is basically using the correlation of signals and instruments in a portfolio and artificially introducing diversification through reshuffling.

For example, if I simply created a portfolio of identical signals with different order names I would end up with a MC median drawdown figure that is very very optimistic and much smaller than if I had just used one signal with a larger order size.

Suggestions:

I think the Monte Carlo simulator needs redesigning. I'd almost argue it's dangerous in it's current form as it promotes under capitalisation.
Might I suggest that the Monte Carlo sim has a user defined button introduced to allow the choice of what is being resampled: Trades or Daily P&L?

Allowing the user to define "daily P&L" as the dataseries to be resampled allows the preservation of correlation between signals and instruments (that can/will lead to a worse/more realistic drawdown).

Other users: please chip in with your thoughts.
These users thanked the author wilkinsw for the post:
tcat

User avatar
Henry MultiСharts
Posts: 8833
Joined: 25 Aug 2011
Has thanked: 1228 times
Been thanked: 2844 times

Re: New Monte Carlo simulator

Postby Henry MultiСharts » 06 Jan 2017

Hello wilkinsw,

Thanks for your feedback. We will discuss it with our engineers.

User avatar
Angelina MultiСharts
Posts: 260
Joined: 28 Dec 2016
Has thanked: 28 times
Been thanked: 66 times

Re: New Monte Carlo simulator

Postby Angelina MultiСharts » 24 Jan 2017

wilkinsw,

This case has been forwarded to development team for analysis.

wilkinsw
Posts: 445
Joined: 21 Apr 2013
Has thanked: 111 times
Been thanked: 74 times

Re: New Monte Carlo simulator

Postby wilkinsw » 01 Feb 2017

Very much looking forward to it getting released.

Currently the Monte Carlo sim only produces relevant data for a single signal, that can only hold one position at a time.

I'm hoping my suggestion of providing an option to resample "daily P&L," instead of trade list, will be a very simple one to implement?

It would then mean you could run simulations on large, complex portfolios.

Can MC provide me with an eta on this fix?

If it's going to take a while then I'll look at purchasing an excel solution instead.
These users thanked the author wilkinsw for the post:
NothingDone

pablo gonzalez vidal
Posts: 35
Joined: 25 Aug 2007
Has thanked: 6 times
Been thanked: 1 time

Re: New Monte Carlo simulator

Postby pablo gonzalez vidal » 10 Nov 2017

Is it possible the shuffled trades monte-carlo analysis is using more of one time the same trade? I´ve been observed it and there is a big difference between MC Monte-Carlos shuffled analysis and the same analysis done with other software like Market System Analyzer. If the trades are used once there woudn't have any difference.

On the other hand the distriution analysis coincides with MSA monte-carlo analysis. It´s like they were changed.

Can someone confirm this observation??
Last edited by pablo gonzalez vidal on 10 Nov 2017, edited 1 time in total.

wilkinsw
Posts: 445
Joined: 21 Apr 2013
Has thanked: 111 times
Been thanked: 74 times

Re: New Monte Carlo simulator

Postby wilkinsw » 10 Nov 2017

pablo gonzalez vidal wrote:Is it possible the shuffled trades monte-carlo analysis is using more of one time the same trade? I´ve been observed it and there is a big difference between MC Monte-Carlos shuffled analysis and the same analysis done with other software like Market System Analyzer. If the trades are used once there woudn't have any difference.

Can someone confirm this observation??


Sampling with replacement is the methodology used. So yes, the same trade can be used twice. Nothing wrong with that IMHO


Multicharts:

When are you adding the feature to shuffle daily P&Ls and not individual trades? I raised this flaw ages ago now.

In its current form, if I had a signal that buys 4 seperate clips of euroFX on a Monday and sells on a Friday..........

The Monte Carlo sim will artificially spread out those 4 clips, improve consistency and reduce risk in the process.

It really is a major limitation and potentially a dangerous one. Please please please add the function to shuffle daily p&l and not trade list.

pablo gonzalez vidal
Posts: 35
Joined: 25 Aug 2007
Has thanked: 6 times
Been thanked: 1 time

Re: New Monte Carlo simulator

Postby pablo gonzalez vidal » 10 Nov 2017

wilkinsw wrote:
pablo gonzalez vidal wrote:Is it possible the shuffled trades monte-carlo analysis is using more of one time the same trade? I´ve been observed it and there is a big difference between MC Monte-Carlos shuffled analysis and the same analysis done with other software like Market System Analyzer. If the trades are used once there woudn't have any difference.

Can someone confirm this observation??


Sampling with replacement is the methodology used. So yes, the same trade can be used twice. Nothing wrong with that IMHO




Captura.JPG
That isn't what they say in the MC guide
Captura.JPG (62.08 KiB) Viewed 1000 times

wilkinsw
Posts: 445
Joined: 21 Apr 2013
Has thanked: 111 times
Been thanked: 74 times

Re: New Monte Carlo simulator

Postby wilkinsw » 10 Nov 2017

Hi Pablo,

Apologies, I should've read that you were talking about shuffled trades and not distribution analysis.

I ran a test for you. A signal that generates lots of 1 lot entries and exits. But on one day only.... generates orders for 1000000 lots.

If the shuffled curves were indeed replacing as you suspect (even though MC state it is not) then you'd expect worse case drawdowns much much larger than baseline. Which I didn't find. Worst case was about the same as baseline. Ie we never generated a curve that sampled the massive trades more frequently than baseline....which would appear to be representative of sampling without replacement.

demo of no replacement.PNG
demo of no replacement.PNG (174.46 KiB) Viewed 992 times


If you disagree, I welcome evidence.
These users thanked the author wilkinsw for the post:
pablo gonzalez vidal

wilkinsw
Posts: 445
Joined: 21 Apr 2013
Has thanked: 111 times
Been thanked: 74 times

Re: New Monte Carlo simulator

Postby wilkinsw » 17 Jan 2018

Angelina MultiСharts wrote:wilkinsw,

This case has been forwarded to development team for analysis.


Hi,

Just chasing up for any updates please?

https://www.multicharts.com/pm/public/m ... es/MC-2397

Just to repeat the issue:

MC monte carlo sim can ONLY be used when a signal(s) has one position at a time. Otherwise it discounts the negative effects of signal correlation and artificially smoothens run ups and drawdowns.

Simple way to demo it: Apply a strategy to PT and collect the 5% drawdown metric. Then clone the strategy so you now have 4 in total. Again, collect the 5% drawdown metric. You'll see that the latter metric, divided by 4, will be significantly less that the former metric.

Please add the option to "Sample from" "Tradelist" or "Daily P&Ls" as radio buttons. Should be a simple fix?

Thanks!

User avatar
Henry MultiСharts
Posts: 8833
Joined: 25 Aug 2011
Has thanked: 1228 times
Been thanked: 2844 times

Re: New Monte Carlo simulator

Postby Henry MultiСharts » 25 Jul 2018

Hello wilkinsw,

Thanks for your feedback. We are going to implement your suggestions in MultiCharts 13.
These users thanked the author Henry MultiСharts for the post:
wilkinsw

wilkinsw
Posts: 445
Joined: 21 Apr 2013
Has thanked: 111 times
Been thanked: 74 times

Re: New Monte Carlo simulator

Postby wilkinsw » 25 Jul 2018

That's awesome Henry.

I only though if this yesterday, but could MC look at generating annualised figures from the monte carlo sim too?

The excel monte carlo sim I currently use annualises all the metrics which allows side by side comparison between various strategies and instruments:

It would be great to be able to gather the 1 year expected 5th percentile max drawdown metric for any strategy/instrument, for example.

Estimating profit/loss or adverse/favourable excursion using monte carlo ideally needs to be put into the context of time.

One could simply only use a chart with a year of trading history on it, but then the monte carlo figures won't be as accurate as annualising data from 10 years of performance data.

User avatar
Henry MultiСharts
Posts: 8833
Joined: 25 Aug 2011
Has thanked: 1228 times
Been thanked: 2844 times

Re: New Monte Carlo simulator

Postby Henry MultiСharts » 26 Jul 2018

wilkinsw wrote:That's awesome Henry.

I only though if this yesterday, but could MC look at generating annualised figures from the monte carlo sim too?

The excel monte carlo sim I currently use annualises all the metrics which allows side by side comparison between various strategies and instruments:

It would be great to be able to gather the 1 year expected 5th percentile max drawdown metric for any strategy/instrument, for example.

Estimating profit/loss or adverse/favourable excursion using monte carlo ideally needs to be put into the context of time.

One could simply only use a chart with a year of trading history on it, but then the monte carlo figures won't be as accurate as annualising data from 10 years of performance data.
wilkinsw, please provide more details regarding this idea and the "excel monte carlo sim" that you currently use for calculating these values. How do you suggest calculating these values?

wilkinsw
Posts: 445
Joined: 21 Apr 2013
Has thanked: 111 times
Been thanked: 74 times

Re: New Monte Carlo simulator

Postby wilkinsw » 26 Jul 2018

My excel sheet would take me ages to explain to someone.

But I can explain the methodology for annualising:

Trade data date range (years): Last possible trade date - earliest possible trade date.

Using the trade list:

Sample year = Total trades / trade data date range.

Simulate 100000 sample years (randomly generate a list of trades totalling the sample year value).

Using the daily mark to market performance

Sample year = Total trading days / trade data date range.

Simulate 100000 sample years (randomly generate a list of trading days totalling the sample year value).



Output metrics and percentiles.

Extra notes:
-I would have earliest possible trade default to date range start + maxbarsback. Also lastest to default date range end. But have both editable by the user also.


Return to “MultiCharts”