Why do different versions yield different optimizations?

Hello,

I've been sticking to one multicharts version for awhile to keep my optimization results consistant.
Could someone explain to me the reason versions such as DT(I don't think is supported I know) and lets say the Multicharts64 has different optimization results?

Wouldn't we want them always to be consistant?

What (if any) are the differences in optimization rules for the versions? How does it affect the out come? Why are they different? Is one version better than another?

Just so it's clear the tests are with identical methods and identical parameters, et cetera.

Hello Mobiius,

The versions from different release cycles can produce different results as there are constant fixes and improvements introduced in each new version. There should be no significant difference in the optimization results between different versions of MultiCharts from the same release cycle.

Due to some architectural peculiarities (of Win x86 and x64) functions using values with decimal parts may give slightly different results in 32 and 64 bit versions. There are 2 possible ways to make the results the same:
1. Use only integer numbers for inputs for each script in both MultiCharts.
2. Use the same bitness of MultiCharts on 2 PCs.

It is recommended to use the latest available build of MultiCharts to have the latest improvements.

Thanks for some clarification Henry.
Yes, our optimizations have strict rules that use only integer numbers.

I'm surprised I'm the only one concerned in this regard. In another sense I am happy because it is verification in the methods we use to trade are indeed unique (with MS users anyway).

My concern is (I'd think you can appreciate) is consistancy. Here is a scenario:

If we started our 'systems' 8 years ago with whatever build that was.
We utilized a 'rolling' optimization for our systems that continually change them.

If the MS systems optimization rules and architecture change at all during the last 8 years, then we will not have consistant results. It is not in fact a problem in that we use an old MS without all the latest 'improvements' but that we have results that are consistant.

There would be a huge shift in 'results' that may have systems that have had constant negative profitability after optimization, to (after using MS64) having systems that have now positive profitability. It is not so much the ability of MS64 to have changed the system from positive to negative that is attractive, but it is the consistancy of having 8 years of constant negative profitability that is more attractive as we have 'consistancy' in the results that we are able to utilize.

The previous build had the exact same results to the DT version. We retested everything recently on the DT version as we can have a few computers using that version while our main version (MS previous before MS64, I'm not sure what version that was. 8?) is not usable for optimizations but for utilization of our systems. If we were to use MS64 for our optimizations then we would have to retest 7-8 years of results which would take approximately 1000 staff hours.

This is not a critisism, but an observation. Obviously this is not a problem for your other 99.9% of customers, but it is something we now have to be aware of. In fact we were aware of them recently (one year ago?) when we noticed that our past and current optimizations were not consistant. We thought at the time that this was user or data error. So we retested everything then. Now we know it was not data or user error, but bitness or version error.

Unfortunately, we have to continue doing what we do with DT unless we do a complete re-test for our optimizations (which sucks as it takes half as long for the MS64 to test).

Still, I don't understand why an optimization should change in results. If you use very strict (and simple) guidelines with the same parameters trying to get the best result, such as the best % profitable (for example). Wouldn't you want all versions to have the same result? After optimization you can have completely different trades... That in my opinion is not a true 'best' % profitability test. What version was right then? Were all your previous versions incorrect and only the new MS64 shows the 'true' % profitability that the previous versions were 'supposed' to give you for results?

Do you understand my concern?

Maybe this topic has come up before? Is there other member/clients on this board not find this troubling? Of maybe they have already dealt with this and there has been alot of discussion about this else where. If so, I would like see the discussion in this regard.

cheers.

Hi All / Henry,

Sorry to be such a pain on this subject, but I really do think this is a major issue. If optimizations are changing with the same parameters that means that anyone who has done any optimizations in the past 8 years has had results that are not indicative of the current build. Does this mean that your past negative testing actually should have been good? Or does that mean your past great results actually should have (and will in the future) be garbage?

Open for comments.

Hello Mobiius,

In order to address your inquiry and provide comprehensive analysis results we need to have more information. Please send me (support@multicharts.com) the following information for analysis:
- what are the exact version and build numbers of MultiCharts you are comparing? (in MultiCharts go to Help tab-> About).
- workspace you are using
- in QuoteManager select the symbol you are using, make a right click on it->Export data->Export instrument. Send us the Qmd export file for analysis.
- in Power Language editor->File->Export->export with dependent functions the studies you are using the workspaces you are providing. Send us the study export file.
Please attach a detailed problem description and highlight the problem on the screenshots.

If the size of the file is > 10mb please use Dropbox or any file sharing hosting to upload the file.

*that is not obligatory to send us your exact strategy. Just create a simplified study that will replicate the same behavior.

Hey Henry,

I don't mind doing that; it'll take a bit of time. But, I'm wondering why I need to? You've already admitted that this is happening.

I'm not sure my data would clarify the issue or just delay the enviable justification response that may or may not be required?

Regardless, what versions would you like me to test with? I'd suppose three or four versions would be preferable for a good analysis. I'm not sure I have the link to all the past downloadable versions so if you can post that as well, I'd appreciate it. Or I can just use the DT and MC64 that I have now?

Thanks Henry.

As I've stated above:

There should be no significant difference in the optimization results between different versions of MultiCharts from the same release cycle.

Per your description:

There would be a huge shift in 'results' that may have systems that have had constant negative profitability after optimization, to (after using MS64) having systems that have now positive profitability.

Please provide the information requested in my post above demonstrateing such case. You can replicate it on any MultiCharts version you have. If you are comparing MCDT and MC64-that is ok.

The versions from different release cycles can produce different results as there are constant fixes and improvements introduced in each new version.

Is what I read so that's why I ask.

Never-mind though, I am commencing the testing. I'll get the info to you probably next week.

Cheers.

OK!

Found some very interesting things along the testing pathway...

1. (SLIPPAGE ERROR?) DT adds or calculates slippage X 2. So if you put in slippage at $20 for MCDT and $20 for MC64 you will have a profit on a trade for MCDT @ $2622.5, where as the profit for MC64 will be $2642.5, this is consistent throughout all trades. Although, what is interesting is that the first trade does not do that... see picture:
2. There is a main reason there is an error when it comes to optimization results. We did optimization tests where there was two results that had the same # (lets say you were testing for Return On Assets). Well DT and 64 reverse the posted optimization result so that if you were testing for RSI LE and SE and you had 2 top ROA results of 1000;

DT would post it like:
ROA RSI LE RSI SE
1000 40 80
1000 42 80

BUT 64 would post it like:
ROA RSI LE RSI SE
1000 42 80
1000 40 80

If you are in the habit of always choosing the top optimization result then you can see where there will be continual errors if you do rolling optimizations!

So, I am doing some more testing on our proprietary systems now to re-test the issues I found and I'm sure what the issues stem from is #2. Issue #1 is curious but I do not think it causes an issue with our actual systems or results.

Comments though on them would be good. Thanks.

OK, I believe I have solved the issue.

In our proprietary systems when we do optimizing from DT to 64 we will get a similar amount of results that again as per #2 above are mixed up and in no particular order.

Again, for example you will have:

MC DT:

ROA____________RSI LE__________RSI SE
1000____________40______________80
1000____________42______________80
1000____________48______________80
1000____________40______________82
1000____________42______________82
1000____________42______________84
1000____________42______________78
1000____________44______________76
1000____________40______________88

For MC64:

ROA____________RSI LE__________RSI SE
1000____________48______________80
1000____________42______________78
1000____________40______________82
1000____________42______________84
1000____________44______________76
1000____________40______________80
1000____________40______________88
1000____________42______________80
1000____________42______________82

If you are someone who choses the top result which is identical in any other respect (profit %, et cetera) then you will start to have messed up results if you start using MC64 as the top result. For MCDT is 40 - 80 where as that is line # 6 in MC64.

So, the question now is how do we stay consistent without knowing which result to chose in MC64 as they are always random (such as one top line optimization test in MCDT would be the 6th line in MC64 in this example, but in another test it may be 2nd out of the top 10 or 8th, or ?).

So the problem is solved, but how do we now us MC64 to have consistent results compared to MCDT? Is this a bug or is an update required to set it so the same pattern of optimization results are shown?

Thanks.

Mobiius, make sure you have the same columns and sorting options selected in optimization reports of both MultiCharts.

yes, all (three) columns set to ROA - descending on all versions.

If you want us to check this case please provide information that was requested in post #7