Missing bars on IDEALPRO, errorneous BID/ASK based backtest

Questions about MultiCharts and user contributed studies.
TraderJ
Posts: 37
Joined: 14 Jul 2009
Location: Germany

Missing bars on IDEALPRO, errorneous BID/ASK based backtest

Postby TraderJ » 17 Jun 2010

This Problem concerns:
- MultiCharts Version 5.5 Release (Build 2723)
- Interactive Brokers: FOREX on IDEALPRO
- Timeframe 1-min and faster (e.g. 10-sec)
- Workspace with BID and ASK based backtesting

During trading times with low volatility (e.g. asian session), several times per hour no change in BID or ASK occurs for one or more minutes. When this happens, the following problems occur:

1. Multicharts does not show the bar. It shows nothing but empty space in the chart. This is wrong. It should show the current price. There is no point in omitting the price just because neither volume nor price have changed.
Please see the attached screenshot of missing 10-sec bars in EUR/GBP.
This problem lies a lot deeper than just on the visual plane:

2. It seems to me that indicators do not take into account the missing bars.

3. BID and ASK based backtesting does no longer work: I use 1-min ASK (e.g. EUR/GBP) as my Data1 (the primary series in the workspace). When a BID bar is missing, for backtesting purposes MultiCharts seems to assume that BID has the same bar as ASK. It then calculates entries and exits on this assumption. This makes BID and ASK based backtesting pointless.

4. There is another problem: IB's IDEALPRO closes at 21:00 GMT and reopens at 21:15 GMT. I have a workspace with three symbols at 1-min (e.g. EUR/GBP BID, ASK, and EUR/USD ASK). When IDEALPRO reopens, only 1-min ASK of Data1 is shown for 15 minutes. The other symbols remain empty, i.e. no bars are drawn - with all of the consequences stated above. After 21:30 GMT, the other symbols start showing bars.

This problem goes far beyond cosmetics, it is a severe problem when backtesting trading systems on 1-min bars and faster.

I would like to see this problem fixed soon.
Attachments
bug-multicharts-missing-bars.png
Errorneously missing bars in EUR/GBP BID 10-sec on IDEALPRO at asian session.
bug-multicharts-missing-bars.png (103.67 KiB) Viewed 366 times

TraderJ
Posts: 37
Joined: 14 Jul 2009
Location: Germany

Workaround

Postby TraderJ » 19 Jun 2010

In the meantime, I have found an elaborate workaround. It allows me to continue developing and backtesting of trading systems. This is what I did:

1. Export all tick data (BID and ASK) to two text files (one for BID and one for ASK).
2. Write a program which adds the missing ticks to the text files. It repeats the previous tick for every second without a tick. That inflated my 700 MB text files to 1.5 GB each.
3. In QuoteManager, both symbols were reintroduced through an ASCII mapping. This avoids import of the inflated data.

I guess import of the data into a new symbol like EUR.GBP_INFLATED should be possible, but I am not sure how to do it. I do not want to delete my original data. The ASCII mapping allows to specify an arbitrary name for a new symbol.

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 485 times

Postby Dave Masalov » 23 Jun 2010

Dear TraderJ,

Could you please come to our Live Chat (http://www.tssupport.com/support/contacts/) so we can connect to your PC and analyze the problem?

TraderJ
Posts: 37
Joined: 14 Jul 2009
Location: Germany

Inconsistent Data

Postby TraderJ » 23 Jun 2010

Hello Dave,

I guess the problem is a result from inconsistent data. The 10-second bars which are not shown on the chart do not have any ticks in MC's database. I have confirmed this by viewing the data in the QuoteManager and by looking at the exported tick data.

Here is an example:

"Date","Time","Price","Volume"
17.05.2010,03:55:01,0.85645,2000000
17.05.2010,03:57:03,0.85640,5000000
17.05.2010,03:57:04,0.85640,1000000

Between the first line and the second there is a gap of about two minutes where no new ticks occurred. This is okay. But it is not okay for MC's chart to leave out the bars showing the current price which is defined by the first line. When there had not been a change, the old price is still valid.

I guess the problem arises from the fact that MC does not take into account the possibility that there might be a second without a tick. This does not occur when backfilling tick data from Interactive Brokers. IB will always deliver a tick for each second, even if it is an old tick. In contrary, when MC records live data via QuoteManager or worksheet, it does only store new ticks (which is smarter). But MC has a problem working with its own recorded data when it comes to seconds without ticks.

If you need more information, please let me know.

Regards,
TraderJ

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 485 times

Postby Dave Masalov » 24 Jun 2010

Dear TraderJ,

I have discussed this problem with our engineers. They ask you to precize when the symbols stop ticking. Does it happen all the time or periodically (i.e. every hour)?

Normally this does not happen. Which makes us think it can be an IB issue. Otherwize hundreds of users would see it.

TraderJ
Posts: 37
Joined: 14 Jul 2009
Location: Germany

Try EUR/SEK at 1-second

Postby TraderJ » 24 Jun 2010

Dear Dave,

like I said above, the problem nearly always occurs under the following conditions:

1. Low Volume. Example: EUR/SEK during midday or asian session (or even nearly always?).

2. Fast timeframe, e.g. 1-second. It also occurs for 10-second and 1-min, but the larger the timeframe the less obvious the problem. At 10-second, it still makes backtesting futile, but at 1-min backtesting results become quite realistic.

You can easily reproduce the problem by creating a worksheet with 1-second bars on EUR/SEK. You should see the gaps after a few minutes. If not, let it run for a few hours. More popular currency pairs are also affected (like EUR/GBP), but you will have to wait longer. The less volume, the more gaps. I have attached a screenshot for you which shows what you should see after you have created your worksheet: Gaps, gaps, gaps!

Yes, it may be an IB issue. I do not know. But I guess that you do not have so many users looking at 10-second bars or faster during low volume times? But anyway MC should be able to cope with the data delivered by IB.

Best Regards,
TraderJ
Attachments
bug-multicharts-missing-bars2.png
Missing Bars on EUR/SEK at 1-second resolution. Gaps everywhere!
bug-multicharts-missing-bars2.png (18.65 KiB) Viewed 342 times

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 485 times

Postby Dave Masalov » 25 Jun 2010

Dear TraderJ,

We have analyzed the problem. Symbols are not ticking in those moments. You can see it in TWS application.

In the attached video, you can see that there are no ticks in TWS, MC and IB example. They all start and stop ticking at the same time.

If you have problems with .avi file opening, please update your DivX codec.
Attachments
NewFilm_002.zip
(1010.47 KiB) Downloaded 47 times

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 51 times

Re: Try EUR/SEK at 1-second

Postby Tresor » 25 Jun 2010

1. Low Volume. Example: EUR/SEK during midday or asian session (or even nearly always?).


Hi,

USDSEK is known for strange bars on low resolutions during the night time in Europe. This can happen 2 - 3 times during the night, 11.15 p.m. - 5 a.m. German time. Attached are the sreenshots from MT4 (these are from recent days so you can compare the day and hours of occurence with your MC).

Strange bars on USDSEK are a result of the spread getting crazy for 30 minutes - an hour time span. USDSEK is the most dangerous pair to trade during the night. Also USDNOK.
Attachments
USDSEK4.png
USDSEK4.png (14.51 KiB) Viewed 335 times
USDSEK3.png
USDSEK3.png (12.9 KiB) Viewed 344 times
USDSEK2.png
USDSEK2.png (13.06 KiB) Viewed 344 times
USDSEK1.png
USDSEK1.png (13.97 KiB) Viewed 342 times

TraderJ
Posts: 37
Joined: 14 Jul 2009
Location: Germany

Unchanged BID or ASK is not handled correctly

Postby TraderJ » 25 Jun 2010

Hi Tresor,

my concern is purely technical. It is not related to trading. It also has nothing to do with rapid changes in BID and ASK. The problem I described is exactly the opposite. It only occurs when BID or ASK do not change at all. When for example ASK is unchanged for ten minutes, there should be exactly ten one-minute bars on a one minute chart, all with the same value for open, high, low and close. But MC will only draw the first one-minute bar, while leaving out all others. This is a bug.

On a 1-second chart, you will observe this behaviour frequently, even with major currency pairs. EUR/SEK is just an example where you can replicate the bug almost every minute throughout the day.

Anyway, your screenshots do not look like MultiCharts.
It seems you are off topic by far.

Regards,
TraderJ

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 51 times

Postby Tresor » 25 Jun 2010

Yes,

Indeed I was off-track. It rang my bell when you mentioned SEK, low volume and Asian session, which on my end always means strange bars.

Having carefully looked at your pictures it seems to be a bug though. Or IB is not sending their snapshot data as frequent as one would expect.

Assuming IB has its own charting platform, you could rule out IB's being at fault by comparing IB chart to MC chart of a particular pair.

Just a suggestion.

TraderJ
Posts: 37
Joined: 14 Jul 2009
Location: Germany

Unchanged quote does not mean no quote.

Postby TraderJ » 28 Jun 2010

Dear Dave,

We have analyzed the problem. Symbols are not ticking in those moments. You can see it in TWS application.


This is what I already tried to describe to you. There are no CHANGES in BID or ASK quotes. But the old quotes still remain valid. Therefore it is not correct to leave out a bar for a time period where there had not been a change in quote.

This clearly is a bug in MC which needs to be fixed.


Regards,
TraderJ

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 485 times

Postby Dave Masalov » 28 Jun 2010

Dear TraderJ,

It is not a bug. MultiCharts simply does not draw bars, if data provider does not send any. It behaves as it is supposed to.

There is an option "Show empty periods" in Format Window --> X - time scale. If you disaible it, there should be no gaps on your charts.

TraderJ
Posts: 37
Joined: 14 Jul 2009
Location: Germany

Professional Trader's point of view

Postby TraderJ » 02 Jul 2010

Dear Dave,

you wrote:
It is not a bug. MultiCharts simply does not draw bars, if data provider does not send any. It behaves as it is supposed to.


I guess that this opinion is technically motivated - and from a software developer's point of view it may seem logical at first sight. But I doubt that this is a useful behaviour from a professional trader's point of view. Showing no bar suggests that the market is closed. This is not the case. The last BID and ASK are still valid, so there should be a bar. Please ask some PROFESSIONAL traders - I guess that you will have difficulties to find a single one who likes this behaviour. In addition, backtest becomes worthless because it has no data to trade upon anymore. A systematic trader is completely out of business at this point.

For background information, just in case it interests you: I am a full time trader working with automated trading systems for more than a year now. I found backtesting of scalping strategies at 1-min with MultiCharts to be extremely strong. The BID and ASK based backtest is a unique feature I did not find anywhere else. However, I found that automated trade execution with MultiCharts has too many flaws (e.g. untransmitted orders, insufficient handling of partial fills). Therefore I use another trading framework for trade execution (OpenQuant).


There is an option "Show empty periods" in Format Window --> X - time scale. If you disaible it, there should be no gaps on your charts.


I am afraid, but this option seems not to work correctly. I tried it several times, but always there were gaps remaining somewhere. I saw that some gaps were closed, but others were not. I had the impression that gaps were closed in data freshly obtained from the historical data server from IB, but the gaps in already existing data remained. Also the gaps in realtime obtained data remained.

So I still find that MultiCharts has an urgent need to be fixed or improved in this matter.

Best Regards,
TraderJ

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 485 times

Postby Dave Masalov » 15 Jul 2010

Dear TraderJ,

If the "Show empty periods" option is disabled you should not have any gaps if you have only one data series on the chart. If you have several data series, gaps are inevitable, because data series are synchronized with each other. Could you please precise how many data series do you have on the charts with gaps?

If you have gaps on the charts with one data series and the "Show empty periods" option is disabled, please come to our Live Chat so we can connect to your computer remotely and analyze the situation.

TraderJ
Posts: 37
Joined: 14 Jul 2009
Location: Germany

Only one data series

Postby TraderJ » 15 Jul 2010

Dear Dave,

you wrote:
If the "Show empty periods" option is disabled you should not have any gaps


I guess you mean the opposite, "enabled" instead of "disabled".

The gaps are there when there is only one data series in the chart while the "Show empty periods" option is enabled.

To reproduce this, just create a new 1-second chart of EUR/SEK ASK at IB IDEALPRO. The historical bars will show no gaps. The newly created bars in realtime will show gaps.

Regards,
TraderJ

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 485 times

Postby Dave Masalov » 15 Jul 2010

Dear TraderJ,

The "Show empty periods" option should be "off" if you do not want to have gaps. If you have gaps even when it is "off", please come to our Live Chat and show us.

TraderJ
Posts: 37
Joined: 14 Jul 2009
Location: Germany

My mistake

Postby TraderJ » 15 Jul 2010

Dear Dave,

sorry, this was my mistake. Somehow I mixed that up. But it does not matter because I tried both settings and the gaps remain in both settings. This means when "Show empty periods" option is "off", new realtime bars still have gaps.

I already described to you how to reproduce the bug so you should be able to verify this behaviour.

Regards,
TraderJ

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 485 times

Postby Dave Masalov » 16 Jul 2010

Dear TraderJ,

Please send us the screenshot of the gaps on the chart with only one data series and the "Show empty periods" option set as "off".

Nick
Posts: 490
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Postby Nick » 21 Jul 2010


If you have several data series, gaps are inevitable, because data series are synchronized with each other.



This is true for live data however it is not for historical data. MC time stamps data to 1 second resolution so it can not preserve the correct sequencing on historical data. I only mention this as it is a fundamental architectural issue that I hope gets addressed in the very near future :). This is why indicators like the ever more popular 'delta' type indicators only work accurately on live data. It also makes high precision back testing (utilising the prevailing best bid/ask) impossible.

This is a pretty important issue that, to be fair, several other competitors suffer from too. However those competitors are well aware of the need for an accurate tick database, properly sequenced across multiple data streams. I would hate to see MC get left behind when these guys re-architect there products. :)

Edit: I know that was not exactly what you where talking about Dave but this issue needs to be kept at the front of you guys minds!

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 485 times

Postby Dave Masalov » 21 Jul 2010

Dear Nick,

Could you please explain what you mean by
MC time stamps data to 1 second resolution so it can not preserve the correct sequencing on historical data.


I'm not sure I got you right.


Return to “MultiCharts”