MC7.0 beta 4 IB backfill spikes

Questions about MultiCharts and user contributed studies.
bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 23 Dec 2009

I attached a picture showing a strange 1 minute bar that has no matching bar in 10second (or other chart). If you expand it on a 28 inch monitor it shows much clearer. It occured on backfill this morning. I have never seen anything quite like this in MC 2.1.999.999. Missing bars yes but not a bar which does not have another matching bar in a different chart.
Attachments
20091217_161511_ES_.PNG
(89.98 KiB) Downloaded 5699 times
Last edited by bowlesj3 on 15 Sep 2011, edited 9 times in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 23 Dec 2009

Interestingly enough, I had to restart MC for another unrelated reason and this strange bar is gone now. It was a proper shutdown and I did not return the database files.

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 23 Dec 2009

I have seen it before, wish I had captured the screen image.....will be on watch now on.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 31 Dec 2009

Here is another one. This time the 1 minute bars were the only bars that had this price data. When I restarted MC6.0 all the bar sizes were showing it.
Attachments
20091231_084324_ES_01_.PNG
(51.63 KiB) Downloaded 5619 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 31 Dec 2009

I have seen it before, wish I had captured the screen image.....will be on watch now on.
Hi Super,

TSS will probably write something in EL (or maybe already have something) that can do the watching for us.

John.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 31 Dec 2009

Someone reported similar observation here:

How to Remove Spkies in MC
http://www.traderslaboratory.com/forums ... -7311.html

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 31 Dec 2009

I suspect this is a display problem rather than a bad database entry that requires a correction. The reason is when I restart MC it displays it correctly but there is no backfill of this data since it did not abort.

brodnicki steven
Posts: 407
Joined: 01 Jan 2008
Been thanked: 3 times

Postby brodnicki steven » 31 Dec 2009

I've suggested a bad tick filter several times but it gets out voted by other features, like playback etc.
We need to vote for it, the next time a vote is requested.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 01 Jan 2010

Repeating what I said above, I don’t think what I am seeing on my chart is a bad tick. Maybe this time I should give the second reason for my believing it is not a bad tick. Specifically because Marina said that all “IB data source” generated charts below the daily bars have the same data source and logically if it was a supposed bad tick then all charts would show it (in my case charts 10second, 1,5,10,15,30,60 minute). However, my pictures demonstrate that one chart shows one thing and another shows another thing. So logically it leads me to think it has to be a charting logic bug or charting data retrieval bug.

Regarding bad ticks and your Comments Steven about a filter, I agree that TSS writing a bad tick filter would help some traders and maybe make MC more marketable but only if TSS advertise the filter as an option that can be easily turned off with a check box. I am one who would most definitely not have it on and I feel it should default to off. However I only trade the e-mini and it does not have much of a problem here.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 03 Jan 2010

Here is another one (6.0 beta2). This time the difference in display of the data is between the 1-minute bars and the 10-second bars. If you study it closely it appears there are two of them. It is a significant issue since it effects triggering and points out another value in the replay feature in that I noticed it while I was practicing triggering.

You need a fairly large monitor to display the png properly.
Attachments
20091230_160138_ES_01_.PNG
(67.31 KiB) Downloaded 5587 times
Last edited by bowlesj3 on 03 Jan 2010, edited 4 times in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 03 Jan 2010

After creating the above post in this thread, a few hours later I went offsite to do some static trigger practice using MC 2.1.999.999 on the notebook. Just by chance I was starting at exactly the same place as the replay practice I was doing with MC 6.0 beta2.and I discovered that MC 2.1.999.999 was displaying correctly (see attached chart) what MC 6.0 beta2 could not.

I thought that the problem may have been the missing ticks problem but I has occurred to me that this problem was already in MC 2.1.999.999 as well so this problem appears to have occurred after MC 2.1.999.999 somewhere.

I guess I could write a matching records program to compare the charts (open high low close) across all future versions of MC versions against the MC 2.1.999.999 version. However this is a fairly serious problem and maybe TSS would consider doing that. Maybe a variation should match the output of different bar types too within the same version. I don't think anyone would disagree that this should be fixed by the MC 6.0 official release and these types of prerelease tests should be in place by TSS. After all a charting program's most basic feature is the ability to produce charts that very accurately match the data being sent to it. If this is not working the fix really should be the absolute top priority in parallel with the ability to run all day without slowing or aborting. It effects both discretionary and automated traders equally.

You need a fairly large monitor to display the png properly.
Attachments
20091230_200947_ES_01_.PNG
(79.25 KiB) Downloaded 5587 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 05 Jan 2010

I just stopped using MC 2.1.999.999 on the notebook and will stick with MC6.0 as long as things go okay.

However because of this problem with charts not matching each other, I have decided to eventually write a pair of scripts to look for these chart bar relationship errors. You will run these on the latest version of MC that you have (no need to keep an old version for this edit). The first script will write out open/high/low/close bar data to a csv file and be applied to all charts for one symbol. The second script will run on any chart, read in the csv files and look for relationship errors to report them. These will be off line runs. I will post the scripts when they are ready.

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 05 Jan 2010

Here's a screen shot of a strange bar on a new 30 Min chart I tried to plot few minutes back, soon after I created another chart with similar setting and it came out perfect. I have seen this happen few times in past and this the first time I thought of capturing screen. Hope it helps in finding a solution.

Version: Beta 6.0 2
Attachments
Mc-1.1.jpg
(163.2 KiB) Downloaded 5603 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 15 Jan 2010

Another example. One of these days I will get around to writing the code to find these rather than the using the random manual detection of the human eye.

Most traders would agree that priority wise, this problem is right up there with 1/ missing ticks and 2/ DRM. From my position it is ahead of most of the replay completion improvements (just ahead of the speed slider improvements) and ahead of the double data tick chart fix.
Attachments
20100115_083036_ES_01_.PNG
(46.05 KiB) Downloaded 5584 times
Last edited by bowlesj3 on 22 Jan 2010, edited 1 time in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 22 Jan 2010

Here is another one for the collection (this one is special).

For reasons I won't explain I have two one minute bar charts in my workspace. Today this "charts not matching each other" problem occurred again (as the attached png picture shows) but the funny bar was in the first 1 minute bar chart and the second 1 minute bar chart was normal (meaning it matched all the other bar charts which are of different sizes).

Note that the RSI is the same on both charts where that funny bar is. At the same time take note of the number 14 below the bad bar in the first chart. This comes from a study that calculates the number of units that the price has pushed the lower band down. It is correct. So the closing price of the bad bar is correct across charts (explaining why the RSI is the same) but the low price across the charts is different (explaining why the band push number 14 is correct). So one chart is providing a different low price to the studies than the other chart.

Not such a big deal for a discretionary trader but for an auto trader it is a big deal. Yah, except for MC not running due to DRM problems or aborting which it does not seem to do if this problem effects all data feeds then for the system trader (ones who hope to leave the room) it is a very big deal since they can not really trust their system can they (they have to be there doing some discretionary verification that it is okay to let that trade go). So I guess there are two types of auto traders. True auto traders and ones who are actually discretionary traders who use the auto trading to help them find opportunities they might miss (they ultimately write the cheque like many wise small business owners). For the later they are essentially like me (MC is an assistant).

Version is MC 6.0 beta2 (and I have never seen anything like this on MC 2.1.999.999).
Attachments
20100122_083202_ES_01_BadBar.PNG
(64.97 KiB) Downloaded 5600 times

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Postby Nick » 26 Jan 2010

Another example. One of these days I will get around to writing the code to find these rather than the using the random manual detection of the human eye.

Most traders would agree that priority wise, this problem is right up there with 1/ missing ticks and 2/ DRM. From my position it is ahead of most of the replay completion improvements (just ahead of the speed slider improvements) and ahead of the double data tick chart fix.
Absolutely agree. Many times in the past I have said that the accurate and timely reception display and storage of data is absolutely critical. None of the other features matter if you can't rely on those core functions.

Incidentally what happens if you refresh the chart or restart MC? Is it really a display issue rather than a data issue? I guess you know that depending on the period of the the chart that MC will build from either tick, minute, or daily data (the 'core' types MC stores). Edit: I see from your original chart that you compare a tick chart to a 10s chart, both of these will be built from the same tick data.

If it is a display issue then it might be hard writing code to spot it.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 26 Jan 2010

Incidentally what happens if you refresh the chart or restart MC?
Good question. I never thought to refresh the chart. I will give that a try.

Regarding restarting MC I have had a few times when this fixed it.

I mentioned that in the above post that the problem appears to be (at least that one time) submitting the data to the studies running on that chart.

Regarding restarting MC, yesterday (Monday 2010/Jan/25) at 8:03 or 8:04 in the morning (eastern standard time) I had a spike up in the ES e-mini to 1103.00. This one was different in that it appeared on all charts so I did not suspect it was this problem since each example in the posts above is one where the charts do not match each other. In fact this spike matched my trading strategy so well that I took note of it (captured the charts) and had even less reason to suspect it was an MC problem. However later in the day I got a lack of memory error which MC 6.0 seems to be triggering these days and I closed MC and and brought it back up. After that I noticed that this spike was gone from all charts. So I pulled up the TWS chart and the spike was not there either. If this is happening with MC this makes the problem even worse since even the discretionary traders have to pull up a different program to verify strange spikes. It was a head scratcher that I decided not to submit since I can not isolate it to an MC problem. However your post Nick triggered a change of heart.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 03 Feb 2010

Another example of charts different. Like the last one this is an example ofthe pair of 1-minute bar charts being different. I did a reload of the data and it fixed it. However it seems to reload more than one chart (at least if one sticks around to watch their system they can do this much if they happen to notice it that is).


The IB TWS chart matches the one without the downward spike.
Attachments
20100203_083509_ES_01_.PNG
(51.83 KiB) Downloaded 5599 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 05 Feb 2010

Another one effecting only 1 minute bars, 5 minute bars and 10 second bars. The reload while the 1 minute bars was highlighted fixed both the 1 minute bars and the 5 minute bars. However the 10 second bars never reloaded. I highlighted that chart and executed the reload but it did not work. I had to close down MC, return the database and restart it before the charts appeared correct (in this one location at least since I do not have time to go scanning for these problems).
Attachments
20100205_082751_ES_01_.PNG
(69.86 KiB) Downloaded 5591 times

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 53 times

Postby Tresor » 12 Feb 2010

Am I correct to assume that the issue reported happens only with Interactive Brokers?

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 12 Feb 2010

Good question. I know super can use TS data but maybe he can let us know what type of data he is getting the problem with. I only use IB.

I still need to find some time to write that program to find these problems. If I get around to it that will be at least 3 weeks away at a minimum (About to put a new feature into my setup which will take 1 to 3 weeks to complete)

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 12 Feb 2010

It happened again today but I figure I have enough examples.

For some reason I always notice them between 5am and 8am in the morning. I have never seen one after that.

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 12 Feb 2010

John,

The problem I highlighted has been resolved by un-checking "Merge Data Sources into a single chart".

http://forum.tssupport.com/viewtopic.php?t=7052

I will be happy to run a test for you using TS as data source, just let me know what setting you want me to have on the chart.

Regards
Super

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 12 Feb 2010

Hi Super,

My merge settings are unchecked. I guess it is a different problem you were having (I went back and looked at your example and yes it is a lot different, I just get a spike now and then which is resolved with a reload or a complete restart). My settings are exactly as they were with MC 2.1.999.999 and I never saw this problem with that release. Something is different between the releases. Maybe it is the DLL for IB that is different.

Regards,
John.

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 12 Feb 2010

John,

I have setup a workspace with two charts with IB and TS feed on 1 Min resolution (ESHO) and have added bollinger bands....let me run this for next few days and see what I can come up with.

Regards
Super

***Let me know when you see it next so that I can compare my charts straight away.

NiC72
Posts: 111
Joined: 02 Nov 2009
Location: Sweden
Has thanked: 39 times
Been thanked: 14 times

New IB DLL in MC 6.0

Postby NiC72 » 13 Feb 2010

It is true .. IB DLL is a new version of MC 6.0. I had problems with the data feed continue for future so i got a new IB DLL that will be found in MC 6.0. I installed the new DLL in my MC 5.5 and continue data feed worked excellent but I get spikes when I change the time frame. Works well at a day time frame but if I switch to 1 hours can I get spikes.
Attachments
screen.png
(34.95 KiB) Downloaded 5594 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 13 Feb 2010

***Let me know when you see it next so that I can compare my charts straight away.
Hi Super,

I get them every 3 to 5 days (that I notice that is). They do not appear on the TWS chart so yah it does look like it is the IB dll as NiC72 points out. I suspect users will get them at random times.

so far I have seen the spikes in 5 minute bars, 1 minute bars and 10 second bars.

John.

wegi
Posts: 124
Joined: 02 Jun 2009
Has thanked: 5 times
Been thanked: 13 times

Postby wegi » 15 Feb 2010

Is this just a problem of version 6.0 ?
Or does it depend on the symbol you collect the data ?

I collect data from IB for all fx pairs and i plot some
of them with a 5min resolution. (MC 5.5)
I did not have any spikes in my charts.

If it is a new "feature" of version 6.0, i can setup this version
and do some tests with it.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 15 Feb 2010

There only seem to be two reports of this problem (by people following this thread). Nic72 (see the attach above) and my roughly weekly spike. Both of us are using the IB dll. I did a big jump from MC 2.1.999.999 to MC 6.0 beta1/2 because of the replay. I can't speak to any versions in between.

NiC72
Posts: 111
Joined: 02 Nov 2009
Location: Sweden
Has thanked: 39 times
Been thanked: 14 times

Change data

Postby NiC72 » 19 Feb 2010

When I change data in QuoteManager (and clear chache, reload data) nothing happen..
Tryed to:
- Export data (ASCII)
- open it with MS Exel, change data.
- Clear Chache
- Import data (ASCII)
- Reload data
nothing happen.. (spikes still there)

Stange.. When I check the data, its the same data as it was before Import Data.
Does QuoteManager load (know that ASCII import works) the new data and then write over the changes with old data (do not know where it would come from)?

Any suggestions? Can not use data that is wrong..

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 19 Feb 2010

Hi NiC72,

I have never done all the export/import that you have done so I do not have the answers. However if the first reload does not change things the first thing I do is pull up the IB TWS chart to make sure that MC is indeed wrong and it is not bad data at the exchange. With stocks it just might be bad data at the exchange level. Personally I don't trade stocks any more so every spike I have seen in the ones I reported is an MC problem.

John.

NiC72
Posts: 111
Joined: 02 Nov 2009
Location: Sweden
Has thanked: 39 times
Been thanked: 14 times

I C

Postby NiC72 » 19 Feb 2010

Hello John,

Nice to hear your thoughts .. It is actually a futures (OMX Stockholm 30, Sweden index). I took it for granted that the index would work .. my mistake. Now I checked IB TWS chart and it look the same, completely wrong. A bar closes at 933.25 (right) from the lowest at 132.00 (wrong).
There is 4 bar that is wrong tha last 3 weeks. There is always the first nummret is wrong. As above, 132.00 would have been 932.00

I still think I should be able to change the data I have received, tried in two ways: (previous post) but it did not work.
Is the only way, to exchange data feeder (again)?

/NiC
Last edited by NiC72 on 19 Feb 2010, edited 1 time in total.

NiC72
Posts: 111
Joined: 02 Nov 2009
Location: Sweden
Has thanked: 39 times
Been thanked: 14 times

Clarify

Postby NiC72 » 19 Feb 2010

To clarify a bit .. I had another datafeeder before and switched to the IB. It worked well for about a week. Then I updated the new DLL for IB (to get the data continue to feed my index futures).
Then I get the spikes .. I thought it was due to my updating IB DLL but I could be wrong. Now it turns out that data on the IB chart look the same.
But I will not be wiser of what I concluded so far.

/NiC

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 19 Feb 2010

Sounds complex. The emini is generally pretty good datawise. Good luck.
John.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 23 Feb 2010

Here is another one. It may be that this occures at the time that the backfill finishes.

A reload fixed this one. I chose 1 day back.
Attachments
20100223_074455_ES_01_A.PNG
(50.46 KiB) Downloaded 5580 times

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 23 Feb 2010

John,

I did not see it on my end, my chart has been running uninterrupted since sunday night.

Regards
Super
Attachments
j-2.jpg
(215.82 KiB) Downloaded 5587 times
J-1.jpg
(228.31 KiB) Downloaded 5587 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 23 Feb 2010

Hi Super,

I don't think it is as big a problem as first suspected. I always notice it around the area where the backfill ends and the live data starts coming in. Next time I see it I will see if I can determine if this is for sure the case. I just have to get in the habit of looking for it when I start up MC in the morning.

John.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 02 Mar 2010

Today it did it. Again I think it was on backfill and the connection between the backfilled data and the live data after that. Charts effected

10sec - yes
1min #1- yes
1min #2 - yes
5min - yes
10min - no
15min - yes
30min, 60min not sure.
daily - not sure.

This time reload actually did nothing at all. I had to restart MC and that fixed it.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 08 Mar 2010

Another "IB backfill spikes" occurred (the official name now I guess). I won't waste time with a picture. It occured on one chart only. reload fixed it. I will report them as they occur to give a feel for how bad the issue is. Assuming I am correct that they occur only on backfill, they can be fixed on reload or fixed on restart of MC then it is not that big a deal I guess. I have changed the thread name again.

4rd Subject: MC6 chart diffs - IB backfill spikes

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 10 Mar 2010

Another backfill spike occured today on startup of MC at roughly 7:40 am Eastern Time. It is confirmed with the TWS chart. For sure this is where the problem occurs. Today it is in the two 1 minute bar charts and the 10 second bar charts. The high of the spike is exactly the same on all 3 of these charts. The first reload of 1 day back corrected the 1 minute bars but did not correct the 10 second bar chart. I highlighted the 10 second bar chart this time and did the reload of 1 day back but it did not fix the problem. A restart of MC corrected the problem on all charts.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 17 Mar 2010

Another spike appears at the backfill to live feed connection point. This time both one minute bars only. Reload corrected this one.

It seems to be occuring once every 5 days roughly.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 22 Mar 2010

It seems this, "backfill to live feed connection" spike, is a very regular once a week occurance. This one has it in the 1 minute bars and 10 second bars. The reload fixed the 1 minute bars but not the 10 second bars. A restart of MC fixed the 10 second bars. Till next week I guess. Diverting mental focus to more productive things for now.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 23 Mar 2010

Here is a picture of a backfill to live spike. Here is the sequence.

I tried a restart and it did not fix anything.
I tried another restart and it did not fix anything.
I did a reload and the 1 minute bars were fixed.

I highlighted the 10 second bars and did a reload. This showed the word "reloading" on the 10 second bars but there is no fix for a long time. About 5 to 10 minutes later the reload fixed the 10 second bars. The reload I used was 1 day only.

So that is two of these this week.
Attachments
20100323_073628_ES_--_.PNG
(55.7 KiB) Downloaded 5580 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 24 Mar 2010

Here is a row/column grid to report on this.

Code: Select all

Chart Spike Reload Restart
10sec ---
1min #1 yes fixed
1min #2 ---
5min ---
10min yes fixed
15min yes fixed
30min yes fixed
60min ---
daily N/A
Maybe I can condense this somehow and report once a month instead.
3 morning startup backfill spikes this week. Worst week I have seen so far.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 05 Apr 2010

The attached backfill spike in the 10 second bars occured in all charts (10sec, 1min, 1min, 5min, 10min, 30min, 60min). It is an error because the IB TWS chart does not have it.

Restart fixed it on all but the 10 second bars.
Reload after the restart fixed it in the 10 second bars (fairly quickly this time)
Attachments
20100405_091403_ES_--_.PNG
(35.26 KiB) Downloaded 5581 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 12 Apr 2010

I got another one of these spikes. I have noticed that

1/ if I highlight the 1 minute bar chart the reload does 1 minute bars up to 60 minute bars only.

2/ However if I highlight 10 second bars then the reload does 10 second bars only.

3/ Along this line if I highlight the daily bars and do the reload it only reloads the daily bars.

So in conclusion I guess this pretty much lines up with what Bruce was saying that the 10 second bars are a different datastream than the 1 minute through 60 minute bars. In the past I thought there were only two data streams (daily and up) as well as (anything below daily).

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 15 Apr 2010

I think that the reload on the 10 second bars MAY be too slow. I did a reload on the 1 minute bars to fix todays backfill and the reload corrected the problem in maybe 10 seconds. However on the 10 second bars it seems to take 5 to 10 minutes (if I have the patience to wait that long). However if I restart MC after waiting for the 5 minutes it will correct the 10 second bars backfill spike with a new backfill within 1 minute or less. Luckly this restart normally works. I think I have had only 2 times when it did not work and I had to wait for 10 minute before the 10 second bar reload worked.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 28 Apr 2010

an interesting backfill spike. picture attached. It occured in all minute bars and the 10 second bars. Reload fixed minute bars fast. 10 second bars are already over 2 or 3 minutes trying to reload. I may have to restart MC to get that fix completed in reasonable time.

Note, if you view this picture it must be on at least a 20 inch monitor. If it is smaller you can not see the spike on the 5 minute bars due to the png format compression.
Attachments
20100428_072455_ES_--_.PNG
(64.51 KiB) Downloaded 5583 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 28 Apr 2010

I had to restart MC to get the 10 second bars corrected for the above post. Again I think there is an intermittent problem in the reload for second bars because the restart of MC will fix up the 10 second bars in less than 30 seconds (most times - I had one instance where the restart did not fix it either.)

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 30 Apr 2010

Today I found you can keep working while the 10 second bars are waiting to do the reload. The studies keep working and the prices keep coming in. when it finally does the reload the vertical scaling goes completely flat so you have to set it back. It should be interesting to see if it keeps having this spike problem in beta 3 (maybe tonight I will install it).

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 17 May 2010

I have not been reporting these backfill spikes that often lately. However I would like to mention a 100% consistency about it.

I start MC two times during every single trading day. Once at about 5am to get things basically set up before I do a few routines including about 1 hour of research work. Later at about 7:30 to 8:00 am (or sometimes later for family reasons) I restart MC to trade. It is the second startup that it happens and never any other time (not the first time and never during any later restarts during the trading day). It occurs about 80% to 90% of the time. 90% in the 1 minute bars and larger related data series bars and maybe 80% in the 10 second bars.

I am wondering if it has something to do with the fact that the backfill is occuring during a fairly high volume time and MC gets a bit confused. The restarts later in the day are fairly rare. However the fact that I have never seen it occur later in the day either suggests that it could be that other IB traders are starting their software at the 7:30 to 8:30 time period as well and later in the day they are not doing this. In my case I am doing it because I am testing program changes. In other words the 7:30 ti 8:30 time is likely to be a very heavy backfill activity time and at the same time live tick activity is also picking up.

Another theory would be that the fact that there are slightly less spikes in the 10 second bars may indicate that less traders are using second bars (more are stopping at the 1 minute bar level).

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 489 times

Postby Dave Masalov » 25 May 2010

Dear bowlesj3,

We failed to reproduce the problem on our end. If you are still experiencing this problem, please come to our Live Chat so we can connect to your PC remotely and investigate the issue.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 25 May 2010

Thanks Dave,

I will do that. It will be at about 7:45 am eastern standard time when I do.

John.

SUPER
Posts: 646
Joined: 03 Mar 2007
Has thanked: 106 times
Been thanked: 84 times

Postby SUPER » 25 May 2010

John,

Finally you have drawn some attention....good luck.

Regards
Super

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 25 May 2010

Thanks Super,

Actually for me it is not that big an issue (having the MC version of arw_GetActive/Text_GetActive/TL_GetActive available and having the cross hair available while moving arrows or trend lines for me is much more important since I use these constantly where as this problem only occurs once maybe every 2nd or 3rd day). A little extra work at the start of the day I don't mind. All day is another matter. However I think it is important to take the time to point out all issues rather than just put up with them. It helps TSS make their product look better all round so I think they want to know. Admittedly when I first saw it I thought it could occur anywhere at any time and I was very concerned. However I now know it only occurs during the backfill to live connection. Being discretionary it is also not a big deal here.

John.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: 4th Subject: MC7.0 beta 4 IB backfill spikes (history)

Postby bowlesj3 » 15 Sep 2011

Hi Dave,

Regarding the last comments in
https://www.multicharts.com/pm/viewissu ... _no=MC-530
I do not Still do not find these logs. I ran the updater program and get this.
Product: MultiCharts
Path: C:\Program Files\TS Support\MultiCharts\
Image: C:\Program Files\TS Support\MultiCharts\MultiCharts.exe
Version: 7.0.4456.205
Action: Interactive Brokers Datafeed update
File: tsIBDataFeed.dll, Copy: yes, Reg: yes, RegAsm: no, Run: no, OSArch: X86
copy to: C:\Program Files\TS Support\MultiCharts\DataFeeds
registry com...
DONE

I do not know of any Logs Shortcut. (I am running Win XP).
I search the whole C drive for logs and none appear that relate to MC.
I search the whole C drive for TS_DATAFEEDS and nothing shows.
There is no 7.0.4510.400 folder on my computer.
Even if these are hidden files I should find them since I unhide everything.

If I zero in on this dir I get nothing if I search.
C:\Documents and Settings\All Users\Application Data\TS Support

If I zero in on this dir I also get nothing if I search.
C:\Program Files\TS Support

in C:\Program Files\TS Support\MultiCharts\DataFeeds all I see is .dll files.

So something is not right here since obviously your program ran okay as shown above.

So maybe I need the specific exact directory where these mysterious logs are suppose to appear rather than "up one level".

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 489 times

Re: MC7.0 beta 4 IB backfill spikes

Postby Dave Masalov » 15 Sep 2011

Hello John,

There seems to be a visualizaton issue in Project Management adding style and font tags to copy-pasetd text. Sorry for the confusion.

Please paste the following line in Start --> Run: "%USERPROFILE%\Local Settings\Application Data\TS Support\MultiCharts\"

Send us the contents of this folder: 7.0.4510.400 and TS_DATAFEEDS.LOG logs

You don't have to run the Updater more than once. Once the issue occurs, please send us log files and the screenshots. If you are experience any difficulties, please come to our Live Chat Mon-Fri from 6:30 AM to 4 PM EST so we can collect all necessary information.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 15 Sep 2011

Hi Dave,

Thanks for the directory. That is interesting, If I drill down to your directory I find the files. However if I search the whole C drive I do not find them. It turns out the search does not include hidden files and system folders even though the the folders are set up to allow me to see them (all folders). I guess this is the first time I have searched for anything like this.

I will get a snapshot tomorrow of the spikes and send the info. The spikes happen almost every day. After I do that I will try the setting you mentioned. I will bet it may reduce the frequency but no fully eliminate them since I have had them since MC 6.2 beta if I remember correctly.

John.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 23 Sep 2011

Finally a good example. I zipped the 2 logs and the png picture. The zip file is called "SpikeAgainSir - Spike+Logs" because I was trying to link all files in that area what had today's date but MC had one lucked up and the zip program did not like it. So I closed down MC and included the two longs only.

https://www.multicharts.com/pm/viewissue.php

Code: Select all

Product: MultiCharts
Path: C:\Program Files\TS Support\MultiCharts\
Image: C:\Program Files\TS Support\MultiCharts\MultiCharts.exe
Version: 7.0.4456.205

Action: Interactive Brokers Datafeed update
File: tsIBDataFeed.dll, Copy: yes, Reg: yes, RegAsm: no, Run: no, OSArch: X86
copy to: C:\Program Files\TS Support\MultiCharts\DataFeeds
registry com...
DONE

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Re: MC7.0 beta 4 IB backfill spikes

Postby Nick » 17 Oct 2011

Yes the problem persists in V7. I connected to live support a couple of weeks back but had no logs on my machine so wasn't much they could do. I had 2 identical 1 hour charts of EUR one had a spike the other did not. Strangely restarting did not fix it though I guess that means it is a cache issue of some sort?

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 17 Oct 2011

I use to find doing a "view, reload, 1 day back" did not always work. However now it always works. You have to do it on the 1 minute bars and the seconds bars. The one minute bars corrects fairly quick (1 to 5 minutes). The second bars will show "Reloading" for maybe 15 minutes or even 30 minutes then it finally fixes the spike. If you are auto-trading I guess that is a problem. For me I analyse the waves visually and I just visual past it as if it is not there. Considering there are a lot of people wanting to auto-trade I can see MC Staff wanting to fix it.

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Re: MC7.0 beta 4 IB backfill spikes

Postby Nick » 17 Oct 2011

Indeed. 1 minute bars to fix the minute data and <1 minute will refresh tick data. I did actually have one (spike) that was higher than the days high it does no harm to refresh the daily data too.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 23 Oct 2011

(although MC staff are looking at this) If anyone wants to vote for this the link is here
https://www.multicharts.com/pm/viewissu ... _no=MC-530

I am finally getting around to voting for all my submissions and I noticed that no one had voted for this even though 3 are being effected by it. Again it suggests what I suspect that the Bug tracker could use some of the "My Votes" filter features I have been suggesting which would make it a lot easier to search for things to vote for..
Ideas for the bug tracker (Project Management).
viewtopic.php?f=1&t=9252#p43883

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

Hello MC staff,
--
I have some good spike information today.
Unfortunately one of the directories is too large to send even after it has been zipped.
It is the "7.0.4456.205" directory.
--
The files, contents and where they can be found are:
2011Nov10_7.0.4456.205.zip (has all the files in that directory) (Located at home - To large to send)
2011Nov10_TS_DATAFEEDS.LOG.zip (has all the files in that directory) (located in the attached forum thread)
2011Nov10_Spikes.zip (has the png pictures of the pikes) (located in the bug tracker)
--
--
The sequence of events:
** The Registry change was been made a few days ago.
** As usual I started MC at 5am to get an initial backfill.
** I closed MC down before I went for my morning coffee.
** At 7:40 I started MC again the spikes in the png pictures were showing.
** I took the snap shots of the spikes.
** I closed down MC.
** I ran the updater then started MC again.
** Upon the restart the 10 second bars still had the spike.
** However the 1 minute bars did not have the spike.
** I went back in the notes and discovered I have to zip two complete directories (not just files).
** I had to close MC again to zip the "TS_DATAFEEDS.LOG" directory because a few files were being use by MC from what I could tell.
** I sent you the information.
** I will run the reload on the 10 second bars (1 day back) to fix the problem (it always works these days).
--
Have a good day,
John
Attachments
2011Nov10_TS_DATAFEEDS.LOG.zip
(74.46 KiB) Downloaded 254 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

from that directory
Attachments
Installer.zip
(302.48 KiB) Downloaded 236 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

from that directory
Attachments
ATCenterServer.zip
(12.02 KiB) Downloaded 253 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

from the multicharts directory of the logs
Attachments
zipmost.zip
(393.18 KiB) Downloaded 246 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

I could not get the MultiCharts trace file in (even after the zip). Maybe there is an email address I can send this one.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

charting trace file
Attachments
Charting_Trace.zip
(199.61 KiB) Downloaded 242 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

trading server
Attachments
TradingServer.zip
(44.88 KiB) Downloaded 232 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

more files
Attachments
StudyServer.zip
(38.32 KiB) Downloaded 236 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

wsdrunner
Attachments
WzdRunner.zip
(25.47 KiB) Downloaded 232 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

quotemanager
Attachments
QuoteManager.zip
(10.01 KiB) Downloaded 236 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

dataupdater
Attachments
DataUpdater.zip
(2.49 KiB) Downloaded 242 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

sefutility
Attachments
sefutility.zip
(685 Bytes) Downloaded 241 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

Other traces from the multicharts trace directory
Attachments
otherTraces.zip
(396.83 KiB) Downloaded 237 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

1/2 of the multicharts trace. There is overlap between the two multicharts trace files I created.
Attachments
Copy of MultiCharts_Trace2.zip
(1.22 MiB) Downloaded 243 times
Last edited by bowlesj3 on 10 Nov 2011, edited 1 time in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Nov 2011

Another 1/2 of the multicharts trace. There is overlap between the two multicharts trace files I created.
Attachments
Copy of Copy of MultiCharts_Trace2.zip
(1.88 MiB) Downloaded 242 times

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Re: MC7.0 beta 4 IB backfill spikes

Postby Nick » 13 Dec 2011

What is the status of this? I did go to live chat but my computer did not have log files, still at least it demonstrated the issue.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: MC7.0 beta 4 IB backfill spikes

Postby Henry MultiСharts » 13 Dec 2011

What is the status of this? I did go to live chat but my computer did not have log files, still at least it demonstrated the issue.
There is no confirmed issue.
Please, set up your IB data feed in QuoteManager -> Tools -> Data Source like it is on the screen shot below and try to reproduce the issue. If the issue is reproduced, please send us the logs.
To enable the logs-please import the registry key from the attachment.

Here's how to find the logs:
1)You can find log files here %USERPROFILE%\Local Settings\Application
Data\TS Support\MultiCharts\
You should paste this line in Start --> Run and in the window that will pop
up make a zip-folder.
or
2) please click on the Windows Start button-> Programs -> MultiCharts ->
make a right click on Logs-> then click on Properties.
Go to the Shortcut tab and click on "Open file location".
In the appeared window please compress to Zip folder called "Logs".
Send this archive to us and we will analyze it.
Attachments
IB datafeed.png
(27.27 KiB) Downloaded 5575 times
AllTracesON.zip
(1.1 KiB) Downloaded 245 times

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Re: MC7.0 beta 4 IB backfill spikes

Postby Nick » 13 Dec 2011

Hi Henry,

I showed the issue to live support so I am surprised to find it is not confirmed. It was them (live support) that could not find the log files, I seem to remember there is/was a registry entry to turn them off. I seem to recall doing so when they where becoming excessively large.

How does one get an issue confirmed? Live support not only saw the issue they could not fix it. One chart had he spike and another with identical instrument and time frame did not.

Thanks.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: MC7.0 beta 4 IB backfill spikes

Postby Henry MultiСharts » 13 Dec 2011

Hi Henry,

I showed the issue to live support so I am surprised to find it is not confirmed. It was them (live support) that could not find the log files, I seem to remember there is/was a registry entry to turn them off. I seem to recall doing so when they where becoming excessively large.

How does one get an issue confirmed? Live support not only saw the issue they could not fix it. One chart had he spike and another with identical instrument and time frame did not.

Thanks.
Debugging procedure is impossible without the logs.
Please enable the logs and if the issue will be reproduced on the latest version of MultiCharts with the IB datafeed settings specified above-
please come to our live chat Monday-Friday 6:30 am - 4 pm EST.
We will do our best to help you.

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Re: MC7.0 beta 4 IB backfill spikes

Postby Nick » 13 Dec 2011

OK Henry, thanks.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 13 Dec 2011

https://www.multicharts.com/pm/viewissu ... _no=MC-530
The status of this issue at the link above is a bit in limbo on my end.

I changed the check box that is in the quote manager (as described in the link above) but but I still had two startups of MC at about the 7:30 hour Eastern Standard Time where I got spikes. So I too am waiting for another one and going to send the logs. I do not get them every day and have not had one since. So I have the checkbox turned on now and am waiting to see if they occur again.

Super is trying to reproduce it (as far as I know) by starting MC exactly as I do using the same contract E-Mini ES.

I still feel that this problem is similar to the spikes during the day but not exactly the same also. The spikes during the day may well be resolved by the check box but these spike occur a lot more frequently and only at start up. That is a hefty difference.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 10 Feb 2012

I thought I should mention in the original thread on this issue that I was given a fix to test for this problem and it fixed the issue in that I went 3 weeks (21 days) of MC starts without any backfill spikes. The regular very large economic spikes in the e-mini ES were in no way effected so in my mind that was the final confirmation that it worked.

My question is, "when will this fix be included in the next beta or release of MC"?

Thanks again too all the MS staff who worked on this.

John

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: MC7.0 beta 4 IB backfill spikes

Postby Henry MultiСharts » 10 Feb 2012

John, thank you for your cooperation in eliminating this problem and further testing.
The fix will be available for public in MultiCharts 8.0 beta 1.
This version will be released next week.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: MC7.0 beta 4 IB backfill spikes

Postby bowlesj3 » 11 Feb 2012

Thanks Henry.

I will install it either the night it comes out or the weekend after that. I will also post a reminder (for 3 weeks after install) to let you know if any spikes have or have not come through.

John


Return to “MultiCharts”