faulty backtest results when dataindex is used

Questions about MultiCharts and user contributed studies.
krzysiaczek99
Posts: 19
Joined: 23 Sep 2012
Been thanked: 2 times

faulty backtest results when dataindex is used

Postby krzysiaczek99 » 23 Jun 2023

res15_1.jpg
trades with wrong prices
(832.19 KiB) Not downloaded yet
test1.jpg
just one instrument USDJPY and setting
(231.79 KiB) Not downloaded yet
instr1.jpg
1 instrument
(252.28 KiB) Not downloaded yet
res1.jpg
backtest result 1 instrument
(945.8 KiB) Not downloaded yet
res1_1.jpg
ttrades 1 instrument
(844.4 KiB) Not downloaded yet
Hi,

I have following code:

Code: Select all

Inputs: dataindex(1), RSILength(14); Vars: oResult1(0), ObLevel(0.8), OsLevel(-0.8), Price(0); Price = Close of Data(dataindex); oResult1 = JohnEhler_My_RSI( RSILength, Price ); if oResult1 crosses under 0 then SellShort("0crossSell") next bar at Market else if oResult1 crosses over 0 then Buy("0crossBuy") next bar at Market ; if oResult1 crosses under ObLevel then SellShort("ObCross") next bar at Market else if oResult1 crosses over OsLevel then Buy("OsCross") next bar at Market ;
Using of this code gives wrong backtest results if dataindex > 1 so in case when I have more then one serie attached to the chart. This fault manifest
in differen situations e.g. during simple backtest when chart have many series and not 1st serie is chosen, in this case trades shown in result file refer to prices from serie 1 even differen serie is used. See attached screenshots - serie 15 (USDJPY) is used but trade prices refer to prices of AUDJPY. In comparision chart with one serie only (USDJPY) gives different backtest results and trade prices. Debugging more proper prices (as per selected datatindex) are sent to EL script however internally backtest results are messed up.

See attachements.

Krzysztof
Last edited by krzysiaczek99 on 23 Jun 2023, edited 2 times in total.

krzysiaczek99
Posts: 19
Joined: 23 Sep 2012
Been thanked: 2 times

Re: faulty backtest results when dataindex is used

Postby krzysiaczek99 » 23 Jun 2023

additiona attachements
instr15.jpg
USDJPY has index 15
(638.95 KiB) Not downloaded yet
test15.jpg
setting of dataindex to 15
(193.5 KiB) Not downloaded yet
res15.jpg
backtests results
(840.07 KiB) Not downloaded yet

krzysiaczek99
Posts: 19
Joined: 23 Sep 2012
Been thanked: 2 times

Re: faulty backtest results when dataindex is used

Postby krzysiaczek99 » 23 Jun 2023

the same bug occurs also when such code is used:
if idx = 1 then Price = Close of Data1;
if idx = 15 then Price = Close of Data15;
and idx set yo 15

User avatar
TJ
Posts: 7743
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2222 times

Re: faulty backtest results when dataindex is used

Postby TJ » 24 Jun 2023


krzysiaczek99
Posts: 19
Joined: 23 Sep 2012
Been thanked: 2 times

Re: faulty backtest results when dataindex is used

Postby krzysiaczek99 » 24 Jun 2023

See post #5
viewtopic.php?t=6929
I simple want to iterate over dataindex to make multiple instruments backtest using optimizer. Optimizer results are faulty as well when you change
dataindex to use different instruments. If I use the trick as in my last post and iterate over idx i.e. if idx = xxx then Price = Close of datax results are faulty also even data to EL script is feed properly. So is any workaround to iterate over different instruments in optimizer and get good backtest results ?? This product is very old and its a MAJOR fault !!!!

krzysiaczek99
Posts: 19
Joined: 23 Sep 2012
Been thanked: 2 times

Re: faulty backtest results when dataindex is used

Postby krzysiaczek99 » 24 Jun 2023

See post #5
viewtopic.php?t=6929
Can you just give example how my code should be changed to be able to iterate over different instruments using optimizer ?

krzysiaczek99
Posts: 19
Joined: 23 Sep 2012
Been thanked: 2 times

Re: faulty backtest results when dataindex is used

Postby krzysiaczek99 » 28 Jun 2023

See post #5
viewtopic.php?t=6929
Unfortunatelly your advice is wrong. After modifying code as below, backtest is still done on data1 even after setting idx = 2.
So please advice how it is possible to change input data inside the script to get correct backtest / optimization results.

Code: Select all

Inputs: idx(1); //dataindex(1), Inputs: RSILength(14); Vars: oResult1(0), ObLevel(0.8), OsLevel(-0.8), Price(0), Price1(0, Data1), Price2(0, Data2); if idx = 1 then begin Price1 = Close data1; Price = Price1; end; if idx = 2 then begin Price2 = Close data2; Price = Price2; end; //Var: Close2( 0, data2 ); //Close2 = Close data2; //Price = Close of Data(dataindex); oResult1 = JohnEhler_My_RSI( RSILength, Price ); if oResult1 crosses under 0 then SellShort("0crossSell") next bar at Market else if oResult1 crosses over 0 then Buy("0crossBuy") next bar at Market ; if oResult1 crosses under ObLevel then SellShort("ObCross") next bar at Market else if oResult1 crosses over OsLevel then Buy("OsCross") next bar at Market;

User avatar
TJ
Posts: 7743
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2222 times

Re: faulty backtest results when dataindex is used

Postby TJ » 29 Jun 2023

is my advice wrong?
or
your code does not work?

there is a difference.

krzysiaczek99
Posts: 19
Joined: 23 Sep 2012
Been thanked: 2 times

Re: faulty backtest results when dataindex is used

Postby krzysiaczek99 » 29 Jun 2023

is my advice wrong?
or
your code does not work?

there is a difference.
reply was made as a solution to problem described in this thread and its not working. It feeds data from different instrument to EL script but it don't affect backtest or optimization results what is a subject of this thread

User avatar
rrams
Posts: 128
Joined: 10 Feb 2011
Location: USA
Has thanked: 7 times
Been thanked: 70 times
Contact:

Re: faulty backtest results when dataindex is used

Postby rrams » 30 Jun 2023

Hi krzysiaczek99,
The FAQ that TJ pointed out does have the correct information for handling multiple data series, including function calls. The topic is not the most apparent to figure out for an easy language. It is ok to ask how to write the code but you included a function from John Ehler that we don't have.

You should start with basic print statements to show what you want and what is not happening when you try to iterate over data values. I wrote some example code to help you understand.

Code: Select all

inputs: dataindex(2), RSILen(5); var: Price(0); if BarStatus(dataindex)<0 then raiseruntimeerror("Need datastream "+NumToStr(dataindex, 0)); for value1=1 to dataindex begin Price=C Data(value1); // wrong print("Close of Data", value1:0:0, ": ", C Data(value1):0:0, ",", " RSIa Data", value1:0:0, ": ", RSI(Price, RSILen):0:0, ",", // wrong " RSIb Data", value1:0:0, ": ", RSI(C Data(value1), RSILen):0:0, // sometimes wrong " RSIc Data", value1:0:0, ": ", RSI(C Data(value1), RSILen) Data(value1):0:0, // correct " RSId Data1: ", RSI(C, RSILen):0:0); end;
I wrote it quickly, so be patient and understanding if it has a mistake.

krzysiaczek99
Posts: 19
Joined: 23 Sep 2012
Been thanked: 2 times

Re: faulty backtest results when dataindex is used

Postby krzysiaczek99 » 30 Jun 2023

Hi krzysiaczek99,
The FAQ that TJ pointed out does have the correct information for handling multiple data series, including function calls. The topic is not the most apparent to figure out for an easy language. It is ok to ask how to write the code but you included a function from John Ehler that we don't have.

You should start with basic print statements to show what you want and what is not happening when you try to iterate over data values. I wrote some example code to help you understand.

Code: Select all

inputs: dataindex(2), RSILen(5); var: Price(0); if BarStatus(dataindex)<0 then raiseruntimeerror("Need datastream "+NumToStr(dataindex, 0)); for value1=1 to dataindex begin Price=C Data(value1); // wrong print("Close of Data", value1:0:0, ": ", C Data(value1):0:0, ",", " RSIa Data", value1:0:0, ": ", RSI(Price, RSILen):0:0, ",", // wrong " RSIb Data", value1:0:0, ": ", RSI(C Data(value1), RSILen):0:0, // sometimes wrong " RSIc Data", value1:0:0, ": ", RSI(C Data(value1), RSILen) Data(value1):0:0, // correct " RSId Data1: ", RSI(C, RSILen):0:0); end;
I wrote it quickly, so be patient and understanding if it has a mistake.
Hi,

I believe its misunderstanding what the fault is, please see my fist messages.

to reproduce it.

1) create chart with 2 instruments.
2) apply the script with ANY strategy which allows to change input series used by selecting data series like code below. You can use ordinary RSI. (see code below)
3) set idx to 1
4) check strategy performace results in view/ strategy performance
5)set idx to 2
6) check strategy performace results in view/ strategy performance. You will see that results are wrong i.e. trades are with prices from idx=1 (data1)
if you make another chart with series data2 only and apply this strategy, backtest will give you different results then from fist chart when idx=2.
7) the same bug occurs when you will use optimizer i.e. when you iterate over idx, chart one for idx=2 gives different results than strategy with data2 in second chart

I detected this bug when I tried to reproduce optimizer backtest results using Python backtester. I made EL code which makes a backtest in EL script
and data is feeded to EL script correctly but at the same time reserved variables GrossProfit and GrossLoss are set incorrectly internally so when you make a view in strategy performance or optimizer results it give wrong results .

Please see my initial message and screenshots.

Krzysztof

Code: Select all

Inputs: idx(1); //dataindex(1), Inputs: RSILength(14); Vars: oResult1(0), ObLevel(0.8), OsLevel(-0.8), Price(0), Price1(0, Data1), Price2(0, Data2); if idx = 1 then begin Price1 = Close data1; Price = Price1; end; if idx = 2 then begin Price2 = Close data2; Price = Price2; end; //Var: Close2( 0, data2 ); //Close2 = Close data2; //Price = Close of Data(dataindex); oResult1 = JohnEhler_My_RSI( RSILength, Price ); if oResult1 crosses under 0 then SellShort("0crossSell") next bar at Market else if oResult1 crosses over 0 then Buy("0crossBuy") next bar at Market ; if oResult1 crosses under ObLevel then SellShort("ObCross") next bar at Market else if oResult1 crosses over OsLevel then Buy("OsCross") next bar at Market;

User avatar
rrams
Posts: 128
Joined: 10 Feb 2011
Location: USA
Has thanked: 7 times
Been thanked: 70 times
Contact:

Re: faulty backtest results when dataindex is used

Postby rrams » 01 Jul 2023

krzysiaczek99,
It is very possible I don't understand what it is that you are trying to do. I can't tell by the pictures.

I have to assume you know that only the primary symbol (the symbol in the first data slot) can be traded. Those are the prices that show up in the backtest results. The AUDJPY pair is labeled at the top of the report and the prices match. You can only use the other data series for triggers to trade the primary series.

If you are trying to optimize for best symbol to trade, you might want to try the Portfolio Trader.

Your code was wrong for calling a function which uses the data from a secondary series and I provided the example of how to do it correctly like you asked.

krzysiaczek99
Posts: 19
Joined: 23 Sep 2012
Been thanked: 2 times

Re: faulty backtest results when dataindex is used

Postby krzysiaczek99 » 03 Jul 2023

Hi,

I know that only primary instrument can be traded but we are talking here about backtest and optimization results not trading.

The pourpose of this script was to make fully automated backtest on multiple symbols using exhausive optimization, by iterating over dataindex i wanted to do if for multiple symbols and it seems not to be possible.

I know I can use Portfolio Trader but in this case I have to do it manually symbol by symbol as output for me are optimization reports and its not possible to generate stacked optimization report e.g for two strategies, optimizer wants to do combination of all parameters for two strategies
rather then do it optimization stategy by stategy( another bug). Setting custom fitness to differentiate in combined optimization report (for two symbols) to differentiate between results to know which belong to which symbol also fails (another bug) - always last fitness is used.

Statement 'my code was wrong' points that you are amatoeur software eng. 1st If such usage is not allowed it should not compile, 2nd where is the documentation which describes that such usage is not allowed.

MC sells this product since many years and there is still plenty of major bugs. This bug costed me that I must repeat 150 000 backtests with 200 000 000 trades as results are false.

Including screens showing random optimization result for code above, maybe it will be more clear.

Krzysztof
Attachments
USDJPY_res.jpg
correct backtest results for USDJPY
(733.46 KiB) Not downloaded yet
opt.jpg
wrong backtest results for USDJPY
(276.47 KiB) Not downloaded yet
symbols.jpg
two symbols attached to the chart
(212.21 KiB) Not downloaded yet


Return to “MultiCharts”