Real life optimization speed
Real life optimization speed
Hi,
I'm a long time TS user and I'm thinking of migrating to MC because of the exciting multi-core optimization feature.
For those who have done optimizations on MC, may I ask what's the real life optimization speed you're getting compared to TS? I'm interested in both the multi-core speed and single core speed as compared to TS optimization. The single core speed would tell me whether MC optimization is more efficient than TS on a single core.
I'm very happy with TS except for the optimization speed. I have no problem with the lack of portfolio back-test or lack of genetic optimization cos I have bought RINA and other 3rd party genetic optimizers. What really gets me is the ridiculous single core usage in TS optimization. I'm a very heavy user of optimization and anything that could potentially give me 4x the speed (on 4 cores machine) is definitely worth looking at...
I'm a long time TS user and I'm thinking of migrating to MC because of the exciting multi-core optimization feature.
For those who have done optimizations on MC, may I ask what's the real life optimization speed you're getting compared to TS? I'm interested in both the multi-core speed and single core speed as compared to TS optimization. The single core speed would tell me whether MC optimization is more efficient than TS on a single core.
I'm very happy with TS except for the optimization speed. I have no problem with the lack of portfolio back-test or lack of genetic optimization cos I have bought RINA and other 3rd party genetic optimizers. What really gets me is the ridiculous single core usage in TS optimization. I'm a very heavy user of optimization and anything that could potentially give me 4x the speed (on 4 cores machine) is definitely worth looking at...
I just downloaded MC beta and played around with it.
I loaded up a few signals and optimize to test for speed. The first time the optimization went ok, but from the 2nd time on the optimizer just quit on me. It jumps out from the optimization window. First it jumps out after a few thousands iterations, then it jumps out sooner and to the point of jumping out of optimiztion immediately after I click to optimize 'ok' button.
Obviously there is a bug with MC optimization.
While my first impression with MC is great and looks like it picks up from where TS has left this piece of software has some serious bug issues to solve before it can be used for research in serious day to day operation let alone to be used as an ATS environment...
I loaded up a few signals and optimize to test for speed. The first time the optimization went ok, but from the 2nd time on the optimizer just quit on me. It jumps out from the optimization window. First it jumps out after a few thousands iterations, then it jumps out sooner and to the point of jumping out of optimiztion immediately after I click to optimize 'ok' button.
Obviously there is a bug with MC optimization.
While my first impression with MC is great and looks like it picks up from where TS has left this piece of software has some serious bug issues to solve before it can be used for research in serious day to day operation let alone to be used as an ATS environment...
I played around with a few optimizations (keltner channel, bollingerband) and compared the speed between TS and MC...
What I found is basically the speed of TS using 1 core is the same as MC using 2 cores...
This is quite surprising because it suggest EL code compiled into C++ on MC is only half efficient as TS implementation...
With a 2 core machine I get no advantage in optimization speed...
Tssupport, could you see if there is anything you can do with that? I'm counting on MC to give me multi core optimization that TS could'n't give us for over 5 years...
But having code run half as efficient as TS is a great penalty - I get no improvement with 2 cores, and with 4 cores I only get <2x improvement after all the migration trouble
What I found is basically the speed of TS using 1 core is the same as MC using 2 cores...
This is quite surprising because it suggest EL code compiled into C++ on MC is only half efficient as TS implementation...
With a 2 core machine I get no advantage in optimization speed...
Tssupport, could you see if there is anything you can do with that? I'm counting on MC to give me multi core optimization that TS could'n't give us for over 5 years...
But having code run half as efficient as TS is a great penalty - I get no improvement with 2 cores, and with 4 cores I only get <2x improvement after all the migration trouble
BUG report
I found that everytime I cancel an optimization, MC runs into optimization premature exit for the next run.
TSsupport pls look into the problem...
scenario reproduce:
1) run an exhaustive optimization and cancel it
2) run another optimization under genetic mode
what to expect:
In the second run of optimization (genetic mode), either the optimizer exit prematurely or it doesn't show optimization report
I found that everytime I cancel an optimization, MC runs into optimization premature exit for the next run.
TSsupport pls look into the problem...
scenario reproduce:
1) run an exhaustive optimization and cancel it
2) run another optimization under genetic mode
what to expect:
In the second run of optimization (genetic mode), either the optimizer exit prematurely or it doesn't show optimization report
- Andrew Kirillov
- Posts: 1589
- Joined: 28 Jul 2005
- Has thanked: 2 times
- Been thanked: 31 times
- Contact:
glam_100,
The problems you encounter make me thinks that you use wrong version.
What version of the product are you running?
The latest beta works faster than TS8.3 both on a single core or a dual core.
Please confirm your version and we will proceed.
The problems you encounter make me thinks that you use wrong version.
What version of the product are you running?
The latest beta works faster than TS8.3 both on a single core or a dual core.
Please confirm your version and we will proceed.
Last edited by Andrew Kirillov on 19 Jul 2007, edited 1 time in total.
Hi Andrew,
I just downloaded the latest Beta and performed the following test:
Symbol: SPY
Date: 7/18/2005 - 7/17/2007 (2 years back)
Strategy and Optimization parameters:
MovAvg 2 Line Cross LE ( FastLength: 1..9; Slow Length: 1..18 )
MovAvg 2 Line Cross SE ( FastLength: 1..9: Slow Length: 1..18 )
Total Number of tests: 26,244
Hardware: Core 2 Duo e6600 @ 3.7Ghz with 2GB memory
Result:
TS 8.2 Build 3896 - 1 min 19 seconds
Multi Chart 2.1.909.909 - 1 min 21 seconds
So MC running on 2 cores is roughly the speed of TS running 1 core.
Could you please take a look at what goes wrong? I am really excited to learn that MC is suuposed to run faster than TS core to core...
If MC is working as you suggest with faster per core speed than TS and support multi-core I'm sure many many many TS users will jump right at MC in no time... it has been a big issue for many TS users for at least 4 years and TS has not made any response at all!
I just downloaded the latest Beta and performed the following test:
Symbol: SPY
Date: 7/18/2005 - 7/17/2007 (2 years back)
Strategy and Optimization parameters:
MovAvg 2 Line Cross LE ( FastLength: 1..9; Slow Length: 1..18 )
MovAvg 2 Line Cross SE ( FastLength: 1..9: Slow Length: 1..18 )
Total Number of tests: 26,244
Hardware: Core 2 Duo e6600 @ 3.7Ghz with 2GB memory
Result:
TS 8.2 Build 3896 - 1 min 19 seconds
Multi Chart 2.1.909.909 - 1 min 21 seconds
So MC running on 2 cores is roughly the speed of TS running 1 core.
Could you please take a look at what goes wrong? I am really excited to learn that MC is suuposed to run faster than TS core to core...
If MC is working as you suggest with faster per core speed than TS and support multi-core I'm sure many many many TS users will jump right at MC in no time... it has been a big issue for many TS users for at least 4 years and TS has not made any response at all!
Hi Andrew,
I tried another optimization and encountered 2 bugs:
The first bug I tried to run the following optimization:
BollingerBand LE ( Length: 1..50; Num StdD: 1..5)
BollingerBand SE( Length: 1..50; Num StdD: 1..5)
Total tests: 62500
When I run this optimization MC hangs up every time when it just gets past 30000 tests... I restared computer and retry and it's the same.
Could you please take a look?
The second bug I tried to run another optimization with different parameters after the Mov Avg 2 Line Cross optimization mentioned above. MC just jump to the optimization report page without doing the actual optimization.
Could you also please take a look.
I really hope the MC optimization functions will work perfectly with no problem. Having stable bug free and multi core fast optimization will make MC a much more powerful research tool over TS... not to mention the integrated genetic optimizer...
Very good job indeed TS Support...
I tried another optimization and encountered 2 bugs:
The first bug I tried to run the following optimization:
BollingerBand LE ( Length: 1..50; Num StdD: 1..5)
BollingerBand SE( Length: 1..50; Num StdD: 1..5)
Total tests: 62500
When I run this optimization MC hangs up every time when it just gets past 30000 tests... I restared computer and retry and it's the same.
Could you please take a look?
The second bug I tried to run another optimization with different parameters after the Mov Avg 2 Line Cross optimization mentioned above. MC just jump to the optimization report page without doing the actual optimization.
Could you also please take a look.
I really hope the MC optimization functions will work perfectly with no problem. Having stable bug free and multi core fast optimization will make MC a much more powerful research tool over TS... not to mention the integrated genetic optimizer...
Very good job indeed TS Support...
Hi Andrew,
I did further testing. Both bugs seem to relate to the same source. Everytime I run an optimization with # simulation > 20000 - 30000 runs the above two problems appear.
You can 100% recreate the scenarios by running a BollingerBand LE and BollingerBand SE strategies with optimization parameter Length 1..50 and Std Dev 1..5 totaling 62500 tests...
I did further testing. Both bugs seem to relate to the same source. Everytime I run an optimization with # simulation > 20000 - 30000 runs the above two problems appear.
You can 100% recreate the scenarios by running a BollingerBand LE and BollingerBand SE strategies with optimization parameter Length 1..50 and Std Dev 1..5 totaling 62500 tests...
- Andrew Kirillov
- Posts: 1589
- Joined: 28 Jul 2005
- Has thanked: 2 times
- Been thanked: 31 times
- Contact:
- Andrew Kirillov
- Posts: 1589
- Joined: 28 Jul 2005
- Has thanked: 2 times
- Been thanked: 31 times
- Contact:
Glam_100,
Our research is coming to its completion so I can tell you some things...
First of all, we've found that there is a bug (a memory leak) in 909 version. It caused two problems - slow-down and the weird behavior you and O66 explained. I appreciate your help.
Also, it is a fact that the next MC beta version will be quicker than TS8 on dual core PC for all studies. However, it is important to understand that speed improvement varies: Bollinger bands signals work two times faster whereas Moving Averages will work just a little quicker on dual cores since we calculate it slower than TS8 does on a single core.
There is more room for improvement we can see and it is planned for MC 2.2
In a few days we will release a new beta build that doesn't contain the problems you encountered.
Our research is coming to its completion so I can tell you some things...
First of all, we've found that there is a bug (a memory leak) in 909 version. It caused two problems - slow-down and the weird behavior you and O66 explained. I appreciate your help.
Also, it is a fact that the next MC beta version will be quicker than TS8 on dual core PC for all studies. However, it is important to understand that speed improvement varies: Bollinger bands signals work two times faster whereas Moving Averages will work just a little quicker on dual cores since we calculate it slower than TS8 does on a single core.
There is more room for improvement we can see and it is planned for MC 2.2
In a few days we will release a new beta build that doesn't contain the problems you encountered.
- Andrew Kirillov
- Posts: 1589
- Joined: 28 Jul 2005
- Has thanked: 2 times
- Been thanked: 31 times
- Contact:
I just downloaded the new beta. The bugs are gone but the optimization speed is still a disappointment.
I ran a few benchmarks:
1) Moving Avg 2 Lines
Moving Avg 2 Line Cross LE ( Fast Len 1..9; Slow Len: 1..18 )
Moving Avg 2 Line Cross LE ( Fast Len 1..9; Slow Len: 1..18 )
TS time (with 1 core): 1 min 21 Sec
MC time (with 2 cores): 1 min 40 Sec
2) Bollinger Band
Bollinger Bands LE ( Len: 1..30; StdDev: 1..5 )
Bollinger Bands SE ( Len: 1..30; StdDev: 1..5)
TS time (with 1 core): 1 min 19 sec
MC time (with 2 cores): 46 sec
3) Price Channel with ATR Trailing
Price Channel LE ( Len: 1..2 )
Price Channel SE ( Len: 1..2 )
ATR Trailing LX ( Len: 1..20; NumATR: 1..3 )
ATR Trailing SX ( Len: 1..20; NumATR: 1..3 )
TS time (with 1 core): 21 secs
MC time (with 2 cores): 1 min 4 secs
4) MACD
MACD LE ( Fast: 1..12, Slow: 26; Len: 1..9 )
MACD LE ( Fast: 1..12, Slow: 26; Len: 1..9 )
TS time (with 1 core): 33 secs
MC time (with 2 core): 51 secs
It seems to me that most functions on MC is slower or significantly slower except for bollinger bands. More importantly, TS is running with just 1 core and MC is running with 2 cores.
While MC has given and still gives me dreams about faster optimization speed, I have to say the current optimization performance is a lot slower than TS. The most striking result is the third test where TS outperform MC 3 times with a single core while MC is using both cores. That would suggest MC is 6x less efficient on a core to core basis (at least for Prince channel and ATR trailing functions).
That shatters my dream of benefiting from faster optimzation in the near future. It looks like I'm stuck with TS, at least for the moment...
MC would make a very valuable research tool if its optimization speed can get at least on par with TS on a core to core basis due to MC's support on multi-core... but right now it just doesn't do it
I ran a few benchmarks:
1) Moving Avg 2 Lines
Moving Avg 2 Line Cross LE ( Fast Len 1..9; Slow Len: 1..18 )
Moving Avg 2 Line Cross LE ( Fast Len 1..9; Slow Len: 1..18 )
TS time (with 1 core): 1 min 21 Sec
MC time (with 2 cores): 1 min 40 Sec
2) Bollinger Band
Bollinger Bands LE ( Len: 1..30; StdDev: 1..5 )
Bollinger Bands SE ( Len: 1..30; StdDev: 1..5)
TS time (with 1 core): 1 min 19 sec
MC time (with 2 cores): 46 sec
3) Price Channel with ATR Trailing
Price Channel LE ( Len: 1..2 )
Price Channel SE ( Len: 1..2 )
ATR Trailing LX ( Len: 1..20; NumATR: 1..3 )
ATR Trailing SX ( Len: 1..20; NumATR: 1..3 )
TS time (with 1 core): 21 secs
MC time (with 2 cores): 1 min 4 secs
4) MACD
MACD LE ( Fast: 1..12, Slow: 26; Len: 1..9 )
MACD LE ( Fast: 1..12, Slow: 26; Len: 1..9 )
TS time (with 1 core): 33 secs
MC time (with 2 core): 51 secs
It seems to me that most functions on MC is slower or significantly slower except for bollinger bands. More importantly, TS is running with just 1 core and MC is running with 2 cores.
While MC has given and still gives me dreams about faster optimization speed, I have to say the current optimization performance is a lot slower than TS. The most striking result is the third test where TS outperform MC 3 times with a single core while MC is using both cores. That would suggest MC is 6x less efficient on a core to core basis (at least for Prince channel and ATR trailing functions).
That shatters my dream of benefiting from faster optimzation in the near future. It looks like I'm stuck with TS, at least for the moment...
MC would make a very valuable research tool if its optimization speed can get at least on par with TS on a core to core basis due to MC's support on multi-core... but right now it just doesn't do it
- Andrew Kirillov
- Posts: 1589
- Joined: 28 Jul 2005
- Has thanked: 2 times
- Been thanked: 31 times
- Contact:
- Andrew Kirillov
- Posts: 1589
- Joined: 28 Jul 2005
- Has thanked: 2 times
- Been thanked: 31 times
- Contact:
Unfortunately I must admit that my conclusions grounded on wrong assumption. MultiCharts calculates indicators quicker than TS8.3 does and I trusted that it is true for signals as well. However signal engine is totally different and includes calculation of performance ratios such as net profit and similar. We have significant inefficiencies here and will optimize it in MC 2.2.
So I confirm that your tests reflect performance of MultiCharts and TS8.3 is still quicker on dual core PC.
However if you use Quad-core should outperform TS8.3 because it uses all 4 cores in spite of calculation inefficiency.
We believe MC 2.2 beta will be out in September. So it will be relatively soon.
So I confirm that your tests reflect performance of MultiCharts and TS8.3 is still quicker on dual core PC.
However if you use Quad-core should outperform TS8.3 because it uses all 4 cores in spite of calculation inefficiency.
We believe MC 2.2 beta will be out in September. So it will be relatively soon.
Hi Andrew,
Thanks for the update.
You guys have a great product and I'm sure you can overcome the signal study inefficiency problem. I look forward to your optimized version with Multichart 2.2. If you can get on par or better performance than TS on a per core basis, Multichart will easily become the top strategy development tool available.
I look forward to MultiChart 2.2, and will check back regularly.
Cheers
Thanks for the update.
You guys have a great product and I'm sure you can overcome the signal study inefficiency problem. I look forward to your optimized version with Multichart 2.2. If you can get on par or better performance than TS on a per core basis, Multichart will easily become the top strategy development tool available.
I look forward to MultiChart 2.2, and will check back regularly.
Cheers
- Andrew Kirillov
- Posts: 1589
- Joined: 28 Jul 2005
- Has thanked: 2 times
- Been thanked: 31 times
- Contact:
- Andrew Kirillov
- Posts: 1589
- Joined: 28 Jul 2005
- Has thanked: 2 times
- Been thanked: 31 times
- Contact:
Andrew,
If possible pls consider adding an option to set the number of optimization thread the spawn in case some users would want to leave one core for other work.
The computer would get very unresponsive when all cores are used.
I have no problem with it personally cos I want fastest speed possible but would be nice for other users.
If possible pls consider adding an option to set the number of optimization thread the spawn in case some users would want to leave one core for other work.
The computer would get very unresponsive when all cores are used.
I have no problem with it personally cos I want fastest speed possible but would be nice for other users.
- Andrew Kirillov
- Posts: 1589
- Joined: 28 Jul 2005
- Has thanked: 2 times
- Been thanked: 31 times
- Contact: