+1 888 340 6572 GET STARTED
MultiCharts Project Management
previous_open_issue.png
Go to the previous open issue
previous_issue.png
Go to the previous issue (open or closed)
star_faded.png
Please log in to bookmark issues
bug_report_small.png
Open Bug report MC-2362

Portfolio Backtester: for multiple strategies set on different symbols, backtester loops number = product of all params across all strategies = Grossly unexpected and missleading! This is a fault, not a lack of optimization. Strategy param sets should be

action_vote_minus_faded.png
6
Votes
action_vote_plus_faded.png
next_issue.png
Go to the next issue (open or closed)
next_open_issue.png
Go to the next open issue
Description

Portfolio Backtester: for multiple strategies set on different symbols, backtester loops number = product of all params across all strategies = Grossly unexpected and missleading! This is a fault, not a lack of optimization.

Take a Portfolio with 2 different strategies defined, each with their own signal and their own different instrument.
The problem I see is following:

Optimizing Strategy 1 with the 4 parameters (Parameter 1A, 1B, 1C, 1D), we get 384 cycles when we run optimization only on this strategy (having Strategy 2 disabled)
Optimizing Strategy 2 with the 4 parameters (Parameter 2A, 2B, 2C, 2D), we get 384 cycles when we run optimization only on this strategy (having Strategy 1 disabled)

Running the Optimization for both in paralel ... I'm surprised to see that total number of combinations is full mesh across all 8 parameters ... with a total number of combinations of 147456 ... Unexpected ... why?
- There is no way parameters 1A-D can influence performance on Strategy 2 ...
- There is no way parameters 2A-D can influence performance on Strategy 1 ...
Hence, I would have expected that:
1) Both strategies remain independent in their Optimization combinations.
2) Total number of combinations when optimizing both Strategies in paralel is: 384 + 384 ... it should be their sum, not their product.

Steps to reproduce this issue
  • Optimizing Strategy 1 with the 4 parameters (Parameter 1A, 1B, 1C, 1D), we get 384 cycles when we run optimization only on this strategy (having Strategy 2 disabled)

    • Optimizing Strategy 2 with the 4 parameters (Parameter 2A, 2B, 2C, 2D), we get 384 cycles when we run optimization only on this strategy (having Strategy 1 disabled)

Running the Optimization for both in paralel ... I'm surprised to see that total number of combinations is full mesh across all 8 parameters ... with a total number of combinations of 147456 ... Unexpected ... why?
- There is no way parameters 1A-D can influence performance on Strategy 2 ...
- There is no way parameters 2A-D can influence performance on Strategy 1 ...

So, the expected number of iterations should be: 384 (total loops corresponding to params permutation for Strategy 1) + 384 (total loops corresponding to params permutation for Strategy 2) = 768.

Also, the optimization jobs should output:
- Best params values for Stragegy1 (out of the 384 possible combinations)
- Best params values for Stragegy2 - fully independent from Strategy1 (out of the 384 possible combinations for Strategy 2)

Currently optimization engine treats all params defined across all strategies set on all symbols present into a Portfolio Trade window, as belonging to the same Strategy ... and doesn't treat the component Strategies independent.

Comments (5)
#1
user-offline.png  Zheka (Zheka)
Oct 24, 2017 - 17:43

This is a major drawback - and obvious too - that limits productivity!

Please let's do this change!

#2
user-offline.png  MultiCharts Support (MultiCharts)
Nov 10, 2017 - 14:16
Hello andb,

Strategies in Portfolio are not that independent from one another. They operate common funds, and therefore they influence each other: after the first strategy buys something, there might be not enough money left for the second to buy, though it will still generate an order. As the algorithm for money distribution between the strategies is not known all of them are optimized simultaneously in order to find optimal parameters for each signal. This is why the total number of combinations is defined as 384*384 =147456, and not as 384+384=768.
Changes:
  •   icon_status.pngStatus changed: Under Review => Not a bug
#3
user-offline.png  andb (andb)
Dec 01, 2017 - 22:48

Hi Alex,

Listen - if I sell you a blue Lada 1200S and tell you it's a red 2016 Ferrari, that doesn't make it a red Ferrari at all. You'll take a look at the product and conclude my description is not conforming with the product appearance.

The same applies with how you guys are marketing a product, how it looks like in the GUI vs. what you're saying in your last message.

Maybe it's not clear:

Opening the Portfolio Trader & running an optimization on 2 Strategies running on 2 different symbols MAKES YOU THINK these two strategies are independent in the way you optimize parameters on each.
Whouldn't you agree PARAM 1A has not bearing on results of Strategy 2? I think you do, otherwise you wound't be in this industry.

So we established: Optimizing PARAM 1A, 1B, 1C, 1D has NOTHING to do with Strategy 2. Making a LOGICAL inference, then from the point of Strategy 2. the two below iterations:

Iteration X:
Param 1A = 1; 1B = 1; 1C = 1; e1D = 1; Param 2A = 1; 2B = 1; 2C = 1; 2D = 1

Iteration X + 1:
Param 1A = 2; 1B = 1; 1C = 1; e1D = 1; Param 2A = 1; 2B = 1; 2C = 1; 2D = 1

Are practically the same. Meaning, variation in PARAM 1A has NO BEARING on Strategy 2.

Logical conclusion:

THERE IS NO POINT IN DOING A FULL MESH TESTING ACROSS ALL PARAMETERS OF THESE 2 STRATEGIES.
I fully understand that Portfolio Management needs to aggregate and do funds allocation. But that's beyond the parameter optimization. You might run 2 separate pools of Threads (one Pool for each strategy), then choose optimal for each strategy, then choose what signals you follow at portfolio level. But Portfolio resource allocation has NOTHING to do with how you're meshing PARAMS across the two strategies.

If your Software design imposes you this full mesh against the above logical principles, this means your design is severely limited and UN-optimal.

#4
user-offline.png  MultiCharts Support (MultiCharts)
Jun 22, 2018 - 18:35
We do not plan changing this behavior at the moment. As specified above - the strategies can influence each other. if the first strategy with a set of parameters invests all funds into buying 100000000 blue Ladas 1200S, then the second strategy may not have enough funds to buy even a single red 2016 Ferrari.
Changes:
  •   icon_status.pngStatus changed: Under Review => Not a bug
#5
user-offline.png  Riccardo (Riccardo)
Jul 11, 2019 - 13:10

Maybe not a bug, it depends on the point of view, anyway surely a desired feature: I need to scan and/or optimize the same strategy on a basket of instruments to understand on which ones it performs better.

For example a simple Bollinger Band strategy 20 periods 2 std devs: I have found it works on 10 instruments, now I want to understand for each of them - independently from each other - what is the best set of parameters for period lenght, stoploss and trailing stop.
Today I have to run 10 optimizations, one for each instrument. It would be useful to run it once for the ten (or more) instruments at the same time.

History
Issue basics
  • Type of issue
    Bug report
  • Category
    Not determined
  • Targeted for
    Not determined
  • Status
    Not a bug
  • Priority
    Not determined
User pain
  • Type of bug
    Not triaged
  • Likelihood
    Not triaged
  • Effect
    Not triaged
Affected by this issue (0)
There are no items
People involved
Times and dates
  • Posted at
  • Last updated
Issue details
  • Reproducability
    Not determined
Attachments (0)
There is nothing attached to this issue
Commits (0)
There are no code checkins for this issue
Duplicate issues (0)
This issue does not have any duplicates