Custom Criteria Optimization  [SOLVED]

Questions about MultiCharts .NET and user contributed studies.
tradetree
Posts: 81
Joined: 29 Apr 2013
Location: www.threefoldmarkets.com
Has thanked: 12 times
Been thanked: 16 times
Contact:

Custom Criteria Optimization

Postby tradetree » 08 Dec 2013

I have spent years optimizing strategies on other platforms. I therefore have extensive knowledge about optimization criteria, Monte Carlo simulations etc. I can't for the life of me figure out what is going on with the MC optimizer. I am open to seeing how I could be using it incorrectly, but so far I'm puzzled. Being that I'm porting systems / signals from other platforms, I know the best set of inputs for the arguments a-priori. I also imported the exact same data set, so the data it is working against is identical.

So here is the problem. I can't get MC to find the best set of inputs, even though I do an "Exhaustive" optimization, and the best input set is part of the search range! In other words, it bypasses the best combinations. If I put the best combination into the inputs (Format the signal with them) and let it run in MC, the performance results are better than any that MC finds! Not only that, but the MC optimizer actually finds worse and worse sets when the number of inputs are increased, even though they intersect this "best set".

Can anyone give some ideas of what could be going wrong?

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Custom Criteria Optimization

Postby Henry MultiСharts » 09 Dec 2013

Hello tradetree,

Please send me (support@multicharts.com) the following information for further investigation:
1) What exact version and build number of MultiCharts are you running? (in MultiCharts go to Help tab-> About)
2) workspace you are using;
3) in QuoteManager select the symbol you are using, make a right click on it->Export data->Export instrument (with data). Send me the Qmd export file for analysis;
4) in PowerLanguage editor->File->Export->export the studies you are using in the workspaces you are providing. Send me the study export file;
5) create screenshots of all optimization windows with the parameters you are using;
6) Please attach a detailed problem description and highlight the problem on the screenshots.

If the file size is >10 mb please upload it to any file sharing hosting and send me the download link.

tradetree
Posts: 81
Joined: 29 Apr 2013
Location: www.threefoldmarkets.com
Has thanked: 12 times
Been thanked: 16 times
Contact:

Re: Custom Criteria Optimization

Postby tradetree » 09 Dec 2013

Henry,
The problem I face is that I'm a professional system developer. My studies are proprietary and I can't afford to take a chance with sharing them, even though I'm a partner with Multicharts. I would have to figure out how to reproduce the problem with a "faked" version of the study. If I can figure out a way to do that, and it reproduces the problem, then I'll send that. But that will most likely take some time and effort.
I believe it may be related to my imported data. What I may try next is to use a trial I have of IQfeed and see if it works better on vendor historical data rather than imported data. What is strange about imported data is that I have to keep re-importing it each time I log into Multicharts. It is like it is not stored into the database. My concern is that the optimizer is not seeing the full historical data or something strange like that.

tradetree
Posts: 81
Joined: 29 Apr 2013
Location: www.threefoldmarkets.com
Has thanked: 12 times
Been thanked: 16 times
Contact:

Re: Custom Criteria Optimization

Postby tradetree » 10 Dec 2013

I have some new information about the symptoms. What I found is that after a few optimization runs, I get into a state where the original study is no longer functioning. I then find if I rebuild the study in the editor it will clear it up. Another symptom is that sometimes it stops printing out the starting input values for the optimizer inputs, and shows old ones. This is a hint that it is in a bad state. Lastly, too many inputs at once, lets say 4 inputs being optimized at once will cause it to fail.

None of these "symptoms" will help isolate the problem, but they have another purpose. I have found after years of running on platforms, that they each have quirks that you have to learn. Multicharts is known to be quite stable at run-time. I get the feeling it may be less stable at optimization. There are a lot of cool features, like the 3D optimization graph, but all those features result in instabilities. It is just a fact of life.

Like I said, if I find a way to dumb-down my algorithm, so I can submit it, I will do that.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Custom Criteria Optimization

Postby Henry MultiСharts » 10 Dec 2013

tradetree, thank you for your observations. Please provide the sample script that will replicate this behavior and all of the other information requested in post #2 of this thread so that we can study this case on our end and help you.

tradetree
Posts: 81
Joined: 29 Apr 2013
Location: www.threefoldmarkets.com
Has thanked: 12 times
Been thanked: 16 times
Contact:

Re: Custom Criteria Optimization

Postby tradetree » 10 Dec 2013

Henry,
I had a thought. I did some tests and read more details in the Programmers Guide. I think the problem I'm having is related to multiple threads during optimization. The Guide says that it is possible to turn off multi-threading. On page 47 it says,
  • "Several strategy instances are created in parallel during the optimization. And they are calculated in
    different threads, that is why global (static) variables should be used with care. MultiCharts .NET for
    optimization creates the amount of threads equal to the amount of cores of your computer. If it is
    necessary to do the optimization in one thread, you can use the following key:
    «NumberOfThreadsOnOptimization», upon the register path «HKEY_CURRENT_USER\Software\TS
    Support\Power Language\StudyRunner». "
I am having trouble with this as my registry entries are not totally consistent with this path. Is there an example registry entry for this? The other question is what is the boundary for static globals to be shared between threads? I assume everything within the SignalObject is thread safe?

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Custom Criteria Optimization

Postby Henry MultiСharts » 11 Dec 2013

I am having trouble with this as my registry entries are not totally consistent with this path. Is there an example registry entry for this?
Please save and unzip the attached file.
While all MultiCharts processes are closed please run the One_Thread_ForOptimize.reg file. This will disable multi core usage for optimization.
To enable multi core calculation please use the All_Thread_ForOptimize.reg file.
The other question is what is the boundary for static globals to be shared between threads?
That is a general .Net programming/multi threading question that is not related to MultiCharts in particular.
I assume everything within the SignalObject is thread safe?
Not all objects of SignalObject are thread safe.
Attachments
All_Thread_ForOptimize.zip
(523 Bytes) Downloaded 239 times

tradetree
Posts: 81
Joined: 29 Apr 2013
Location: www.threefoldmarkets.com
Has thanked: 12 times
Been thanked: 16 times
Contact:

Re: Custom Criteria Optimization

Postby tradetree » 11 Dec 2013

Henry,
That didn't solve it, but I'm getting closer to isolating it. I am still not sure if it is a thread-safe issue or an initialization issue. What I know is that my code gets the identical inputs (CalcBar bar data) during optimization as during regular running. But my C# code produces different computations between the two cases. Being that my C# code is extensive and complex, this is still not an easy problem to solve, but now I'm confident I have a path-way to find it.
Questions for you:
1) can I debug during optimization?
2) I found the minimum number of test cases is 2 for the optimizer, or it won't run. Is there any way you know to run the optimizer with one test case?
3) Any other suggestions?
4) Can you expand on the statement that not all objects in SignalObject are threadsafe? I create instances of classes that I need to keep state in for CalcBar calls. How else can you keep state than to instantiate an object within SignalObject? In my other platform that is also C# this same concept works fine. Others must have hit this problem before I did.

tradetree
Posts: 81
Joined: 29 Apr 2013
Location: www.threefoldmarkets.com
Has thanked: 12 times
Been thanked: 16 times
Contact:

Re: Custom Criteria Optimization  [SOLVED]

Postby tradetree » 11 Dec 2013

I found the problem! This is actually really important and I'd suggest a review of the programmers guide documentation. When I read about the Create method, it stated, "The constructor and the Create() method for study classes will be executed once when a study is selected, now indicators will create plots and signals will create orders." This implies something I did not realize until now. If you put the wrong things in Create you will get into big trouble, but only with optimization (or mainly). When the optimizer runs, it must reuse existing SignalObjects, which have already had the Create method called on them. As a result, only during optimization, if you initialize something you do not get it re-initialized with each optimization pass.

So early on when I was learning to port code to MC, I put some initialization code from the other platform into Create. The tests of my SignalObjects passed with flying colors, as the Create method was called and performed my initialization during single runs. But then when I went to optimize, all hell broke loose. Thus my original post where I thought the MC optimizer was possessed. And in a sense it was, as initialization failures can really cause great confusion. I would simply suggest you put a bold-typeface warning about this in the guide.

Regarding my other questions, I was able to debug with the optimizer running. This is how I found the initialization problem. The remaining questions were mainly fishing for answers, but anything more about thread-safe coding would help.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Custom Criteria Optimization

Postby Henry MultiСharts » 12 Dec 2013

I would simply suggest you put a bold-typeface warning about this in the guide.
tradetree, probably you have missed the How Indicators and Signals are Calculated section of our Programming guide that repeats this warning multiple times ;-)
but anything more about thread-safe coding would help.
Objects in SignalObject are not thread safe because when the study is calculated - it is calculated in a single thread. During the optimization several strategy instances are created in parallel. Each is calculated in different thread. When MultiCharts operates with the strategies that have MultiCharts variables (VariableSeries and VariableObject) it keeps everything safe and sound. But if a global (static) variable is used in the code - this is when you need to think about thread-safety.

tradetree
Posts: 81
Joined: 29 Apr 2013
Location: www.threefoldmarkets.com
Has thanked: 12 times
Been thanked: 16 times
Contact:

Re: Custom Criteria Optimization

Postby tradetree » 12 Dec 2013

tradetree, probably you have missed the How Indicators and Signals are Calculated section of our Programming guide that repeats this warning multiple times ;-)
Well I mentioned in my post that I had not seen the implications of the Create method until hitting the optimizer problems. Even after re-reading these sections this relationship still does not stand out. The only thing stated, if you know directly what to look for, is that the method is "called first and only one time, immediately after the creation of a new study object". This is very clear, but if you didn't realize that each optimizer run is a reuse of the object and not a "creation of a new study object" you would miss this, as I did. Later on it actually talks about seeing errors if you put the wrong items in the Create method, giving you the feeling that you will at least know when you do something wrong. So my point is simply that it is very easy to have really bad things go wrong without putting these facts together. Anyway, I don't mind your having some fun, it certainly was my mistake in how I read the document. - But it could help others, which is why I mention it.

I'll tell you something else. I almost gave up on Multicharts thinking it was buggy. It is working great now that this is fixed and I'm having fantastic results with the optimizer. I think the feature where you can specify a Custom Criteria is worth its weight in gold.


Return to “MultiCharts .NET”