portfolio backtesting - signals not called in correct order

Questions about MultiCharts and user contributed studies.
ts2mc
Posts: 122
Joined: 12 Jan 2009
Been thanked: 1 time

portfolio backtesting - signals not called in correct order

Postby ts2mc » 29 Jul 2009

Hello MC team,
i setup a portfolio backtesting over 4 symbols.
each symbol has a 2. symbol as datarow.

the range over all symbols is 50 days back, based on 1 min bar.
in a chart window, where i have attached all symbols involved,
with exactly same settings as in portfolio (50 days back, 1 min bar),
i can say that all symbols start and ends at the same date.

now the problem:
the portfolio backtesting shows, that the signal is not called
bar by bar for all symbols. instead, signal is called for instrument 1
and 2 up to bar 30 before calling for instrument 3 and 4 with bar 1.

i checked this by loading a dll doing logging.
below you can see the logs. the format is

log_timestamp [date time][symbol][barnumber] comment
(find attached the complete log along with pf setup)

20090729 15:36:32:330 [1090712 1850][AUD.CHF][30] enter short at 0.846600 cor=55.00
20090729 15:36:32:330 [1090712 1850][GBP.JPY][30] enter long at 149.725000 cor=25.00
20090729 15:36:32:330 [1090712 1851][EUR.USD][1] === init ===
20090729 15:36:32:330 [1090712 1851][EUR.USD][1] Param: pip=0.000050 corr=1.000 length=20 TargetPips=40 LossPips=40
20090729 15:36:32:330 [1090712 1851][EUR.USD][1] pair=<AUD.USD-EUR.USD hi=1
20090729 15:36:32:330 [1090712 1851][EUR.USD][1] === init end ===
20090729 15:36:32:330 [1090712 1851][AUD.USD][1] === init ===
20090729 15:36:32:330 [1090712 1851][AUD.USD][1] Param: pip=0.000050 corr=1.000 length=20 TargetPips=40 LossPips=40
20090729 15:36:32:330 [1090712 1851][AUD.USD][1] pair=<AUD.USD-EUR.USD hi=-1
20090729 15:36:32:330 [1090712 1851][AUD.USD][1] === init end ===
20090729 15:36:32:330 [1090712 1851][AUD.CHF][31] cover short
20090729 15:36:32:330 [1090712 1851][GBP.JPY][31] close long


if portfolio backtesting should represent the real world, it should
call (for time based signals)

signal (symbol 1) bar[1]
...
signal (symbol n) bar[1]
....
signal (symbol 1) bar[k]
...
signal (symbol n) bar[k]
Attachments
tmp.rar
logfile and pf setup
(477.07 KiB) Downloaded 130 times

ts2mc
Posts: 122
Joined: 12 Jan 2009
Been thanked: 1 time

Postby ts2mc » 29 Jul 2009

a solution seems to be, that you have to include each symbol
in each row of traded symbol.

tradeable symbol | informational symbol
A | B C D
B | A C D
C | D A B
D | A B C

ts2mc
Posts: 122
Joined: 12 Jan 2009
Been thanked: 1 time

Postby ts2mc » 29 Jul 2009

must correct myself.
i had the hope it is solved but in fact it is not.

my solution above works, since you have only 1 strategy attached.
since you can only attach 4 more informational symbols at the maximum
per symbol, you can only implement a strategy with a maximum
of 4 dependent symbols.

if you need more symbols which depend on each other (for instance hedge), then you have to split in more than 1 strategy and this means the problem i introduced here returns:

the signals are not played in the chronological correct order.
thus it does not simulate the real world.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 30 Jul 2009

ts2mc,
Your issue is clear and your solution is not optimal. Let me explain why it happens what should be done to avoid it.

Although you have equal number of days back for all data series, you have different number of bars on each data series due to data holes or market inactivity. If you print you will see that the number of bars is different.

If you apply your system to daily bars then you will not see the problem since like hood of holes on daily bars significantly less and all bars will be available, but it is not guaranteed too.

So it is not correct to compare bar number and make a conclusion that MultiCharts doesn't generate signals in chronological order.
You must compare a time of bar and you will see that everything is correct.


if currentbar < 50 then
print("pair ", symbolname data1, "-", symbolname data2,
" [", currentbar:5:0, "] ", " date: ", DateToString(eldatetodatetime(date)), " time: ", TimeToString(eltimetodatetime(time)) );

pair EURUSD.FXCM-AUDUSD.FXCM [ 1] date: 19/07/2009 time: 18-24-00
pair AUDUSD.FXCM-EURUSD.FXCM [ 1] date: 19/07/2009 time: 18-24-00
pair EURUSD.FXCM-AUDUSD.FXCM [ 2] date: 19/07/2009 time: 18-25-00
pair AUDUSD.FXCM-EURUSD.FXCM [ 2] date: 19/07/2009 time: 18-25-00
pair EURUSD.FXCM-AUDUSD.FXCM [ 3] date: 19/07/2009 time: 18-26-00
pair AUDUSD.FXCM-EURUSD.FXCM [ 2] date: 19/07/2009 time: 18-25-00
pair EURUSD.FXCM-AUDUSD.FXCM [ 4] date: 19/07/2009 time: 18-27-00
pair AUDUSD.FXCM-EURUSD.FXCM [ 3] date: 19/07/2009 time: 18-27-00
pair EURUSD.FXCM-AUDUSD.FXCM [ 5] date: 19/07/2009 time: 18-28-00
pair AUDUSD.FXCM-EURUSD.FXCM [ 4] date: 19/07/2009 time: 18-28-00
pair EURUSD.FXCM-AUDUSD.FXCM [ 6] date: 19/07/2009 time: 18-29-00
pair AUDUSD.FXCM-EURUSD.FXCM [ 5] date: 19/07/2009 time: 18-29-00
pair EURUSD.FXCM-AUDUSD.FXCM [ 7] date: 19/07/2009 time: 18-30-00
pair AUDUSD.FXCM-EURUSD.FXCM [ 6] date: 19/07/2009 time: 18-30-00
pair EURUSD.FXCM-AUDUSD.FXCM [ 8] date: 19/07/2009 time: 18-31-00
pair AUDUSD.FXCM-EURUSD.FXCM [ 7] date: 19/07/2009 time: 18-31-00
pair EURUSD.FXCM-AUDUSD.FXCM [ 9] date: 19/07/2009 time: 18-32-00
pair AUDUSD.FXCM-EURUSD.FXCM [ 8] date: 19/07/2009 time: 18-32-00
pair EURUSD.FXCM-AUDUSD.FXCM [ 10] date: 19/07/2009 time: 18-33-00
pair AUDUSD.FXCM-EURUSD.FXCM [ 8] date: 19/07/2009 time: 18-32-00

ts2mc
Posts: 122
Joined: 12 Jan 2009
Been thanked: 1 time

Postby ts2mc » 30 Jul 2009

Hello Andrew,
i see your argument with gaps and matching timestamp and
agree of course.

however, please look at a dump of another run below.

this portfolio has 3 strategies each with 2 symbol pairs.
all settings are the same, including max. bars back.
there is in fact only 1 strategy, which makes an init entry to logfile
on barnumber 1.

20090730 14:25:22:167 [1090609 2122][EUR.USD][1] === init ===
20090730 14:25:22:167 [1090609 2122][EUR.USD][1] Param: pip=0.000050 corr=0.000 length=200 TargetPips=240 LossPips=800
20090730 14:25:22:276 [1090609 2122][EUR.USD][1] pair=<AUD.USD-EUR.USD hi=1
20090730 14:25:22:276 [1090609 2122][EUR.USD][1] === init end ===
20090730 14:25:22:276 [1090609 2122][AUD.USD][1] === init ===
20090730 14:25:22:276 [1090609 2122][AUD.USD][1] Param: pip=0.000050 corr=0.000 length=200 TargetPips=240 LossPips=800
20090730 14:25:22:276 [1090609 2122][AUD.USD][1] pair=<AUD.USD-EUR.USD hi=-1
20090730 14:25:22:276 [1090609 2122][AUD.USD][1] === init end ===
20090730 14:25:22:276 [1090609 2123][EUR.USD][2] enter long at 1.408200 cor=0.18
20090730 14:25:22:276 [1090609 2123][AUD.USD][2] enter short at 0.804350 cor=0.18
20090730 14:25:22:354 [1090610 2052][GBP.JPY][1] === init ===
20090730 14:25:22:354 [1090610 2052][GBP.JPY][1] Param: pip=0.005000 corr=0.000 length=200 TargetPips=240 LossPips=800
20090730 14:25:22:354 [1090610 2052][GBP.JPY][1] pair=<AUD.CHF-GBP.JPY hi=1
20090730 14:25:22:354 [1090610 2052][GBP.JPY][1] === init end ===
20090730 14:25:22:354 [1090610 2052][AUD.CHF][1] === init ===
20090730 14:25:22:354 [1090610 2052][AUD.CHF][1] Param: pip=0.000050 corr=0.000 length=200 TargetPips=240 LossPips=800
20090730 14:25:22:354 [1090610 2052][AUD.CHF][1] pair=<AUD.CHF-GBP.JPY hi=-1
20090730 14:25:22:354 [1090610 2052][AUD.CHF][1] === init end ===

as you can see, GBP.JPY barnumber [1] has datetime of 1090610 2052.
so 1 day later in comparison to first 2 symbols.

my screenshot attached shows all symbols contained in this portfolio
starting at the same date (same timeframe) without gaps.
the cursor is above GBP.JPY
at 1090609 2122
and shows Barnumber 202. The setting for Bars back
is 201. so relative, its Barnumber 1.
but the log above shows, that GBP.JPY barnumber [1] has datetime of 1090610 2052. so ~ 1 day later.

can you explain this ?

(timezone Exchange. in bortf.back.test as well as in chart.,
24/5,1 min 50 days back)
Attachments
gbp_jpy_bt_offset.JPG
(270 KiB) Downloaded 1192 times

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 30 Jul 2009

According to the log we presume that data for GBP.JPY and
AUD.CHF symbols start later than for other symbols in the portfolio.
Your attached workspace is an example. See attached.
Attachments
demo_mult_symbols.PNG
(47.6 KiB) Downloaded 1190 times

ts2mc
Posts: 122
Joined: 12 Jan 2009
Been thanked: 1 time

Postby ts2mc » 30 Jul 2009

Hello Andrew, they don't. the screenshot demonstrates this.
this was the sense of the screenshot to show, there are
definitely GBP.JPY data available.
please, check now on your side.
thank you.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 30 Jul 2009

Take a look at the portfolio screenshot I sent. It has DIFFERENT sessions for different symbols. Thus number can be different. You screenshot may have the same sessions and it will bring you identical number of bars.
Please check it.

ts2mc
Posts: 122
Joined: 12 Jan 2009
Been thanked: 1 time

Postby ts2mc » 30 Jul 2009

i checked it and in fact, some of my symbols where set to default
instead of 24/5. my fault, i apologize.
(the trap here is, that always default session is set. so each time, you need
to setup a symbol, you need to switch to your desired session.
or is there a menu, where you can define your own default session ?)

i still have a problem here:
now, the backtest report tells something different from what my
log says.
the log is called within the signal, for instance, when enter a position.
each logentry has a prefix, showing symbol of signal call along with date
and time. the code is:

tcBarStamp = "["+NumToStr(Date,0)+" "+NumToStr(Time,0)+"]["+symbolname+"]["+NumToStr(BarNumber,0)+"] ";


please put a look onto the attachments.
it shows a screenshot of backtest report and contains my logfile.

the logfile shows 6 orders (as expected) placed at
20090610


20090731 00:55:58:829 [1090610 2123][EUR.USD][2] EUR.USDenter long at 1.401350 cor=0.14
20090731 00:55:58:829 [1090610 2123][AUD.USD][2] AUD.USDenter short at 0.805100 cor=0.14
20090731 00:55:58:829 [1090610 2123][GBP.JPY][2] GBP.JPYenter long at 160.600000 cor=0.00
20090731 00:55:58:829 [1090610 2123][AUD.CHF][2] AUD.CHFenter short at 0.868700 cor=0.00
20090731 00:55:58:829 [1090610 2123][USD.JPY][2] USD.JPYenter long at 98.125000 cor=0.52
20090731 00:55:58:829 [1090610 2123][EUR.JPY][2] EUR.JPYenter short at 137.500000 cor=0.52

not so the backtest report.
the four first orders are reported exactly as in my logfile.
but the last 2 orders of the 1. series are missed in this report.

the 2 missing orders are:
20090731 00:55:58:829 [1090610 2123][USD.JPY][2] USD.JPYenter long at 98.125000 cor=0.52
20090731 00:55:58:829 [1090610 2123][EUR.JPY][2] EUR.JPYenter short at 137.500000 cor=0.52

above, my logfile shows various orders for USD.JPY and EUR.JPY are
placed. but none, except 1 USD.JPY, of such orders is listed in the backtest report.
so, the backest report seems to skip some orders.
perhaps, i do something wrong here basically. but where,
and from where my logfile takes those trades not listed in report ?

(the log was produced along with the run from report.)

if you need my signal code and the dll i used, please let me know.
thank you
Attachments
pfbt2.rar
(1.71 MiB) Downloaded 125 times

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 31 Jul 2009

Please your signal and workspace.

ts2mc
Posts: 122
Joined: 12 Jan 2009
Been thanked: 1 time

Postby ts2mc » 31 Jul 2009

the package attached contains the signal and the workspace.

by the way: each time you open portfolio backtester,
you need to explore your favor portfolio. it is not saved.
Attachments
pfbt2.rar
(1.72 MiB) Downloaded 130 times

ts2mc
Posts: 122
Joined: 12 Jan 2009
Been thanked: 1 time

Postby ts2mc » 10 Aug 2009

Hello,
i'd like to ask, if you have any results here ?
thank you

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 10 Aug 2009

Dear ts2mc,
I apologize for the salience. The responsible developer is out now, he will return the next week and continue to work on it.

ts2mc
Posts: 122
Joined: 12 Jan 2009
Been thanked: 1 time

Postby ts2mc » 10 Aug 2009

thank you for information

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 10 Aug 2009

Please remidn the next monday. OK?

ts2mc
Posts: 122
Joined: 12 Jan 2009
Been thanked: 1 time

Postby ts2mc » 10 Aug 2009

yes, will remind next monday

ts2mc
Posts: 122
Joined: 12 Jan 2009
Been thanked: 1 time

Postby ts2mc » 17 Aug 2009

want to remember to this problem

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 17 Aug 2009

W need you to export your signal as PLA.
Plus we need twslink.xml since when we apply the signal we can see it in the logs:
warning;coreInstance;LoadSetup;loading xml setup file <c:\twslinkfiles\twslink.xml> failed,code=-5: failed to open file <c:\twslinkfiles\twslink.xml>

ts2mc
Posts: 122
Joined: 12 Jan 2009
Been thanked: 1 time

Postby ts2mc » 17 Aug 2009

hello Andrew,
i tried 2 times to send the stuff you required for.
its size is of 7.42 MB.


please tell me what to do.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 19 Aug 2009

----
Reason:
Portfolio Equaity is not sufficient to open new positions and the new order is ignored until one of the existing positions will be closed to free some capital.
====

Your settings have 1000000 as an initial capital.

According to your report we sum up all amounts you’ve invested:
20090731 00:55:58:829 [1090610 2123][EUR.USD][2] EUR.USDenter long at 1.401350 cor=0.14
20090731 00:55:58:829 [1090610 2123][AUD.USD][2] AUD.USDenter short at 0.805100 cor=0.14
20090731 00:55:58:829 [1090610 2123][GBP.JPY][2] GBP.JPYenter long at 160.600000 cor=0.00
20090731 00:55:58:829 [1090610 2123][AUD.CHF][2] AUD.CHFenter short at 0.868700 cor=0.00

1.40135*25000 + 0.8051*25000 + 160.6*5882 + 0.8687*137 = 999928.5577

After that your signal generates the order:
[1090610 2123][USD.JPY][2] USD.JPYenter long at 98.125000 cor=0.52


You want to enter long USD.JPY @ 98.125000. You can’t since you have 1000000 - 999928.5577 = 71.4423


Return to “MultiCharts”