Monte Carlo produces incorrect results for many signals and setups - [OPEN] MC-2397

#1

MultiCharts Support
Last seen online at Apr 17, 2024 - 12:37
Show user details

Jun 01, 2018 - 21:03

Hello wilkinsw,
There is no way to "add the function to shuffle daily p&l and not trade list" as the resolutions higher than 1 day may not have enough data points for shuffling.
Currently the Monte Carlo simulation is calculated based on the Equity Curve Close to Close.
That is possible to use the Equity Curve Detailed - using the “per bar” equity values should eliminate the symbol correlation effect and provide a smoother line. Do you consider it a working solution?

#2

Will (wilkinsw)

wilkinsw
Last seen online at Jul 09, 2021 - 19:47
Show user details

Jun 04, 11:32, in reply to comment #1

"resolutions higher than 1 day may not have enough data points for shuffling"
Can't you simply mark to market daily as is done in the performance report?

"Currently the Monte Carlo simulation is calculated based on the Equity Curve Close to Close.
That is possible to use the Equity Curve Detailed - using the “per bar” equity values should eliminate the symbol correlation effect and provide a smoother line. Do you consider it a working solution? "

I don't know, have you tested it? "Smoother line" was quite the opposite of what I'm after.

Please look again at "steps to reproduce this issue". All MC has to do is correct the simulator (or add a user defined switch) that stops the false improvement of the drawdown metric when cloning a signal. The simulator is currently nonsense unless you are only using one signal.

I suggest a time bucket solution so that buckets are shuffled, which will mean cloned signals (using 100% cloned signals reproduces the correlation issue at its extreme) will correctly include the negative effects of correlation on the drawdown metric.

I've built a monte carlo sim in excel that uses daily values (I had to, because Multicharts' is so very wrong). I suggest MC test solutions that solve the problem in "Steps to reproduce this issue" OR place a disclaimer in the Wiki that the monte carlo simulator is for single signals only!

Thanks.

Edit comment #2

Comment visibility (whether to hide this comment for "regular users")

Comment

Selected syntax: Plaintext ▼

"resolutions higher than 1 day may not have enough data points for shuffling"
Can't you simply mark to market daily as is done in the performance report?

"Currently the Monte Carlo simulation is calculated based on the Equity Curve Close to Close.
That is possible to use the Equity Curve Detailed - using the “per bar” equity values should eliminate the symbol correlation effect and provide a smoother line. Do you consider it a working solution? "

I don't know, have you tested it? "Smoother line" was quite the opposite of what I'm after.

Please look again at "steps to reproduce this issue". All MC has to do is correct the simulator (or add a user defined switch) that stops the false improvement of the drawdown metric when cloning a signal. The simulator is currently nonsense unless you are only using one signal.

I suggest a time bucket solution so that buckets are shuffled, which will mean cloned signals (using 100% cloned signals reproduces the correlation issue at its extreme) will correctly include the negative effects of correlation on the drawdown metric.

I've built a monte carlo sim in excel that uses daily values (I had to, because Multicharts' is so very wrong). I suggest MC test solutions that solve the problem in "Steps to reproduce this issue" OR place a disclaimer in the Wiki that the monte carlo simulator is for single signals only!

Thanks.

To auto-link to an existing issue, write "issue", "bug", "ticket" or the issue type (like "bug report" or "enhancement") followed by the issue number (ex: "ticket #3" or "bug report MYPROJ-1"). Enclose sourcecode samples between two "~~~~" lines.
See more formatting tips in MarkdownFormatting.

The Bug Genie uses an enhanced flavor of the markdown syntax, which makes each newline count. If you want to add a linebreak, add a new line and it will be presented just like you type.
See more formatting tips in MarkdownFormatting.

If you want to add text that is not supposed to be parsed, put it inside tags, or start the line with a space.
See more formatting tips in WikiFormatting.

or cancel

Reply to comment #2

Comment visibility (whether to hide this comment for "regular users")

Selected syntax: Plaintext ▼

'''Will wrote:'''
>"resolutions higher than 1 day may not have enough data points for
> shuffling"
> Can't you simply mark to market daily as is done in the performance report?
> 
> "Currently the Monte Carlo simulation is calculated based on the Equity
> Curve Close to Close.
> That is possible to use the Equity Curve Detailed - using the “per bar”
> equity values should eliminate the symbol correlation effect and provide a
> smoother line. Do you consider it a working solution? "
> 
> I don't know, have you tested it? "Smoother line" was quite the opposite of
> what I'm after.
> 
> Please look again at "steps to reproduce this issue". All MC has to do is
> correct the simulator (or add a user defined switch) that stops the false
> improvement of the drawdown metric when cloning a signal. The simulator is
> currently nonsense unless you are only using one signal. 
> 
> I suggest a time bucket solution so that buckets are shuffled, which will
> mean cloned signals (using 100% cloned signals reproduces the correlation
> issue at its extreme) will correctly include the negative effects of
> correlation on the drawdown metric.
> 
> I've built a monte carlo sim in excel that uses daily values (I had to,
> because Multicharts' is so very wrong). I suggest MC test solutions that
> solve the problem in "Steps to reproduce this issue" OR place a disclaimer
> in the Wiki that the monte carlo simulator is for single signals only!
> 
> 
> Thanks.
> 
> 
> 
> 
>

Don't worry about URLs - they will automatically be transformed to links.
See more formatting tips in MarkdownFormatting.

To create a bulleted list, start one or more lines with a star ("* bulleted list item"). You can create a numbered list by using the hash/pund sign instead ("# numbered list item").
See more formatting tips in MarkdownFormatting.

You can create a link directly to a wikipedia page by using the "WIKIPEDIA:" link namespace: "Here is a link to the wikipedia article about [[WIKIPEDIA:Norway|Norway]].
See more formatting tips in WikiFormatting.

or cancel

#3

Will (wilkinsw)

wilkinsw
Last seen online at Jul 09, 2021 - 19:47
Show user details

Jan 12, 2019 - 02:25

https://www.multicharts.com/discussion/viewtopic.php?t=50422&p=129969#p129969

Annualising the results is also essential else can't make side by side comparisons (and is the norm for monte carlo).

To Annualise: create a fictitious year= Total trades / Trade date range (years) .

e.g. 1000 trades / 10 years = 100 trades = 1 year.

Create X* 1 years via resampling (probably with replacement). X probably >= 100000.

Output percentiles.

#4

MultiCharts Support (MultiCharts)

MultiCharts Support
Last seen online at Apr 17, 2024 - 12:37
Show user details

Jan 15, 2019 - 14:03

William, I'm afraid we are not getting your idea. Please provide an example of the calculations in Excel.

#5

Will (wilkinsw)

wilkinsw
Last seen online at Jul 09, 2021 - 19:47
Show user details

Jan 15, 14:37, in reply to comment #4

I'll email you my whole sheet. It may take some time for your to understand it though (as it was built only for my eyes).

I'll also have another bash here in describing how I run monte carlo sims. I'm presuming you're stuck on how to annualise??

Currently Multicharts outputs percentiles from their monte carlo sims. But they do so without the context of time.

If I have performance report derived from a signal that started trading in 1/1/2003 (chart start date + maxbarsback) and ends 14/1/2019, I have a report for 16.04 years of trading.

I then want to construct a "typical year" trading. If my performance report has 1150 trades, I do 1150 divided by 16.04 which gives me: 72 trades in an average year.

So, I now want to to randomly sample (with replacement) my performance report and stop once I've reached 72 samples** . I then collect performance metrics of interest. I then repeat this 100,000 times. I finally output the percentiles for the metrics that were collected.

sample could equal trades or could equal daily P&L (I specifically use the daily mark to market P&L in the daily performance of Multicharts' performance report). I recommend the definition of "sample" be user definable in Multicharts as simply using trades will only produce reliable results on one signal at a time (and nonsense for portfolios).

Edit comment #5

Comment visibility (whether to hide this comment for "regular users")

Comment

Selected syntax: Mediawiki ▼

I'll email you my whole sheet. It may take some time for your to understand it though (as it was built only for my eyes).

I'll also have another bash here in describing how I run monte carlo sims. I'm presuming you're stuck on how to annualise??

Currently Multicharts outputs percentiles from their monte carlo sims. But they do so without the context of time.

If I have performance report derived from a signal that started trading in 1/1/2003 (chart start date + maxbarsback) and ends 14/1/2019, I have a report for 16.04 years of trading.

I then want to construct a "typical year" trading. If my performance report has 1150 trades, I do 1150 divided by 16.04 which gives me: 72 trades in an average year.

So, I now want to to randomly sample (with replacement) my performance report and stop once I've reached 72 samples** . I then collect performance metrics of interest. I then repeat this 100,000 times. I finally output the percentiles for the metrics that were collected.

**sample could equal trades or could equal daily P&L (I specifically use the daily mark to market P&L in the daily performance of Multicharts' performance report). I recommend the definition of "sample" be user definable in Multicharts as simply using trades will only produce reliable results on one signal at a time (and nonsense for portfolios).

To auto-link to an existing issue, write "issue", "bug", "ticket" or the issue type (like "bug report" or "enhancement") followed by the issue number (ex: "ticket #3" or "bug report MYPROJ-1"). Enclose sourcecode samples between two "~~~~" lines.
See more formatting tips in MarkdownFormatting.

Code samples are best presented between two "~~~~"-lines. The Bug Genie uses GeSHi for syntax highlighting with support for over 100 languages! (<source lang="php"><?php echo "fu"; ?></source>)
See more formatting tips in MarkdownFormatting.

To create a bulleted list, start one or more lines with a star ("* bulleted list item"). You can create a numbered list by using the hash/pund sign instead ("# numbered list item").
See more formatting tips in WikiFormatting.

or cancel

Reply to comment #5

Comment visibility (whether to hide this comment for "regular users")

Selected syntax: Mediawiki ▼

'''Will wrote:'''
>I'll email you my whole sheet. It may take some time for your to understand
> it though (as it was built only for my eyes).
> 
> I'll also have another bash here in describing how I run monte carlo sims.
> I'm presuming you're stuck on how to annualise??
> 
> 
> Currently Multicharts outputs percentiles from their monte carlo sims. But
> they do so without the context of time.
> 
> If I have performance report derived from a signal that started trading in
> 1/1/2003 (chart start date + maxbarsback) and ends 14/1/2019, I have a
> report for 16.04 years of trading.
> 
> I then want to construct a "typical year" trading. If my performance report
> has 1150 trades, I do 1150 divided by 16.04 which gives me: 72 trades in an
> average year.
> 
> So, I now want to to randomly sample (with replacement) my performance
> report and stop once I've reached 72 samples** . I then collect performance
> metrics of interest. I then repeat this 100,000 times. I finally output the
> percentiles for the metrics that were collected.
> 
> 
> **sample could equal trades or could equal daily P&L (I specifically use
> the daily mark to market P&L in the daily performance of Multicharts'
> performance report). I recommend the definition of "sample" be user
> definable in Multicharts as simply using trades will only produce reliable
> results on one signal at a time (and nonsense for portfolios).
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>

To auto-link to an existing issue, write "issue", "bug", "ticket" or the issue type (like "bug report" or "enhancement") followed by the issue number (ex: "ticket #3" or "bug report MYPROJ-1"). Enclose sourcecode samples between two "~~~~" lines.
See more formatting tips in MarkdownFormatting.

To create a bulleted list, start one or more lines with a star ("* bulleted list item"). You can create a numbered list by using the hash/pund sign instead ("# numbered list item").
See more formatting tips in MarkdownFormatting.

Code samples are best presented in tags. The Bug Genie uses GeSHi for syntax highlighting with support for over 100 languages! (<source lang="php"><?php echo "fu"; ?></source>)
See more formatting tips in WikiFormatting.

or cancel