IB's historical vs real-time volume

Questions about MultiCharts and user contributed studies.
janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

IB's historical vs real-time volume

Postby janus » 30 Mar 2010

Given there can be a huge difference between real-time volume and historical volume from IB due to the different ways they calculate the two, is there a way for a real-time study to retrieve the historical volume for just the current bar once that bar is completed? I suspect there isn't a way so I may have to write my own dll using IB's api to retrieve the historical value.

The better approach would be for IB to fix this issue but I know they have been asked about this for many years and it's highly unlikely they will do anything about it any time soon, if ever.

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 31 Mar 2010

I've been interested in this IB volume issue for some time. I would just like to chime in (as I think you already understand) that it's not a MultiCharts-specific issue - I've seen practically this same extended conversation take place on a number of platforms. IB does some unusual things, and the platform vendors all scramble to try to figure out how best to accomodate. As a large organization, IB may be in a conundrum of not being able to really "fix" certain things short of a new major release because everybody's already built such architectures around the known behaviors even if they're something arbitrary and perhaps not what anyone would have chosen given the opportunity to design a new architecture carefully.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 31 Mar 2010

Another way of looking at it is IB should remain focused on providing a reliable, efficient, and fast order service. The specifics of the volume issue are secondary. Having said that, they should enter the 21st Century and fix it eventually.

BTW, I have tried to discuss it with IB but it appears they are reluctant to do so. They are probably tired of rehashing the issue.

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 31 Mar 2010

I tend to agree with this sentiment, and in general (other than for some specific cases) I'm not entirely a fan of using IB as a data feed. Their focus is not on providing a data feed, and it's something it seems sometimes like they do almost as an after-thought and with considerable restrictions in place that are designed to protect them from excessive use, rather than specifically to help you with what you may need. I've seen truly extraordinary amounts of time be spent trying to deal with various issues in their data feed offering, and often, it makes far more sense to simply pay for a feed from someone like eSignal who's in the data feed business and who therefore has a different big picture architecture in place. There are some special cases where IB's data feed shines, but they're not as common, and often it's the case that individual traders/users start with an assumption that all data feeds are basically the same and they all "should work" in some equivalent manner (ergo that they can blame the software platform if anything at all isn't the same) and since IB's feed is economical they start there, leading to a good deal of pain and lost productivity before they realize it isn't entirely so. I really do agree that software platforms should do everything reasonably possible to make the data feeds work the same from a practical use perspective - after all, that's what a great many people understandable and reasonably want is data feed independence, and that's one of the strong selling points of MC. But, I simply appreciate there are some behind the scenes struggles and difficulties in working with IB's data feed that most people who've never looked at it technically probably don't realize. That having been said, I think this volume issue can probably be fixed - but it's driven by a design issue that's in common among many platforms that use IB's API and not just MC.

I understand re: discussions with IB. I've been working with them for years, both from various platforms and as an individual API developer. They're good people to work with and are knowledgeable and capable, and I have a good working relationship with them. They do have their own sense of priority that is driven from the top down in their organization, and fixing minor issues that have been that way for years tends not to be something they can get immediately motivated about or that will in any way cause them to scramble to immediately fix it (as I think is at least somewhat understandable, even if as an individual we don't always completely agree on individual technical points). They're understandably focused in a top-down way on introducing the "next big" features, and making sure they don't "fix" something that inadvertantly breaks something else (a concern that individual developers don't have to worry about so much, but that for large organizations is one of the principal concerns, even more so in some cases than innovation). For this reason, often, it's better in a certain view not for them to try to quickly "fix" certain kinds of things once their installed API base reaches a certain size and especially if the "bug" has been that way a long time and everybody's coded assuming it's that way, and they have to proceed cautiously with "fixing" something even if it seems obvious to an individual that what they're doing is not the best solution.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 31 Mar 2010

I appreciate your comments about IB's dilemma, and to a certain extent I agree. However, we are just talking about one thing here - volume, not a significant change in there ordering system or interface. I don't know how they calculate volume of contracts but I would have thought exchanges throughout the world provide a reliable tick by tick data feed containing time, bid, ask, last price and volume. I don't see it being rocket science to compute say a 1 minute real-time volume from the tick feed that's consistent with the historical values they compute. I don't understand why the huge difference at all. I can understand how say a package like MC will aggregate the tick volume slightly different due to small skews in the times. But if the data provider has access to the actual source data feed, their real-time and historical values should be virtually in total agreement even if MC can't make them identical but only say within 1% variance. At the moment I can see 40% variances, and I know it's not MC's fault. Anyway, it's not a real show stopper for me at the moment, just bemused.

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 31 Mar 2010

Sometimes I've seen errors as simple as IB returning a "snapshot" of the trading activity with the updated volume for the day, and platforms handling this newly updated volume for the day as the volume of that snapshot period of time, causing the volume for the day as accumulated by all of the snapshots that day to skyrocket, then return to normal as soon as historical data is requested overwriting this. I'm not saying that's what MC is doing, but it's a mistake I've seen other people make, and it's characteristic of the kind of thing that can sometimes go wrong. Of course, if MC were doing this, it would be more wrong than you are describing, but this is the kind of thing that can happen.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 31 Mar 2010

I can understand occasional "simple" errors. No data provider is perfect. What I'm seeing with at least one contract is the volume displayed in MC in real-time on a minute chart around twice on average than that of the historical volume after I re-load the chart. During very quiet times such as as night time the volumes agree. When I compare the volumes to those quoted by another broker, the IB real-time volumes although not exactly the same are more correct. So, they are doing something very odd with the historical ones. I can replicate the problem just using IB's charting system. The volumes there are historical and are about half that displayed in the tws quote bar which is real-time. In my mind it's broken, and broken badly.

Spaceant
Posts: 254
Joined: 30 May 2009
Has thanked: 1 time
Been thanked: 3 times

Postby Spaceant » 31 Mar 2010

I use IB as a data feed and share the feeling of what you both have said. I have tried some strategies which depend on volume; or tried to inculded an entry filter with volume. But, I have never have any confident in putting any of them into real trading. I know that volume is one of the critical indicator, nevertheless this is the one that I try to aviod with IB as a data feed.

Sa

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 31 Mar 2010

Interesting to note that after scanning through the IB SMF forum that this problem had been recognized many times before over the past several years. So, to be blunt, don't trust IB's historical volume, and treat real-time volume with some care, which makes back testing using historical data pointless with IB volume data.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 31 Mar 2010

from my understanding, no data provider has the definitive claim to accuracy...

sad...

:-(

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 31 Mar 2010

from my understanding, no data provider has the definitive claim to accuracy...
:-(
I know. However, a package I use that's provided by a bank displays the exact same volume in real-time and historical, and is the same as reported by the exchange. If they can do it so should so called professional data providers.

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 53 times

Postby Tresor » 01 Apr 2010

The issue of incorrect volume was my nightmare a year ago. I checked Transact first and was not satisfied: http://forum.tssupport.com/viewtopic.ph ... gn&start=0

Then I checked OEC. The same shit.


What I learned was:

1. Each data provider / broker receives 1 data feed with correct volume from the exchange.

2. Then the data feed is then split in broker's API into:

(i) correct volume data feed (the same data feed as the broker / data feed provider receives from the exchange) - this data is available to selected clients or for those clients who ask for it. In this case the real-time volume will always match the historical volume

(ii) incorrect volume data feed / filtered tick data feed that results in volume data being massacred. There are 2 reasons why filtered tick data is being provided by a broker:
- its frontend would crash / would be sluggish if immense unfiltered data is connected to it;
- a broker would need to incurr capex on servers and other infrastructure and increased opex on data transfer.

The reason why a data feed provider may decide to filter ticks is to save money on infrastructure and data transfer costs.

Noone has ever succeeded in filtering real-time data to achieve realistic volume. To filter the data brokers throttle the original exchange data and in result you get volume dicrepancies.

The only way to receive the unfiltered data is (i) ask your broker to tell you which dialogue / communication field in its API transfers the unfiltered data feed or the original data feed from the exchange and then (ii) pass this knowledge onto TSS and request that MC's dll responsible for handling the data from your provider be rewritten

E.g. at the moment MC receives filtered / massacred data feed from OEC. But if you search OEC forum for developers you'll find tips on how your MC can be switched to receive unfiltered / correct data feed for free.

Regards
Last edited by Tresor on 01 Apr 2010, edited 1 time in total.

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 01 Apr 2010

The filtering of tick data (most especially level 1 quote updates such as changes to bid/ask size, but also to a lesser extent other level 1 updates) affects pretty much all data feeds in at least some way. A key issue is that the users of the feed are often expressedly unwilling to pay more when volumes increase (i.e. users say they are strongly against paying by the KB for market data), yet tick volumes increase every year at a pace that far exceeds the rate at which bandwidth and additional hardware/capacity costs go down. With a paid data feed you at least have a business vendor who is answerable to you, and who you can tell you are willing to pay more for a feed of higher quality when they have multiple tiers of services available, but with a typical free broker feed, they are in the position of providing you with a "freebie" and having no incentive whatsoever to incur additional costs just because market volumes are getting greater every year at a dramatic pace. They have to do something, and having costs spiral out of control for more servers, more bandwidth is just not an option. Nobody (almost) wants filtered data. But, you need to be prepared to pay for the increased volumes of data that have to be sent, and this typically means a higher end data feed e.g. bloomberg etc. and not a broker free or discount "best value" type data feed, OR in many cases, you need to use a mid-range feed like eSignal et al and simply understand the nature of the filtration that is being done, so you know what you are looking at and can adjust your expectations and plans accordingly. With this compromise, and with a careful understanding and testing of the issues on your specific trading platform, often you can get what you need or close enough to the ideal for practical purposes.

I'm deeply sympathetic to the problem described here. I've looked at this type of thing in some detail over the course of several years, on several platforms, and with several feeds. Volume inaccuracies can often be resolved and close to ideal outcomes achieved through careful work with the vendor to make sure everything's being handled perfectly. But, the desire to receive ALL level 1 updates e.g. every time the bid/ask size changes is mostly met with data feed vendor denial (or empty statements that "of course we send all of the ticks" that you only later much to your chagrin find out are inaccurate), unless you're willing to pay for a higher end data feed because free broker feeds and discount or value priced paid data feeds simply don't have the incentive to incur this kind of cost, and by and large their users have expressed that they're unwilling to have their own costs go up just because market volumes go up. There are a few exceptions to every rule, of course, but by and large this is a simple problem of business economics, and most of the feeds' users are unwilling to pay more, yet if you simply graph the tick volume over the past few years it's immediately obvious that isn't sustainable even with bandwidth and hardware costs coming down somewhat.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 01 Apr 2010

(ii) incorrect volume data feed / filtered tick data feed that results in volume data being massacred. There are 2 reasons why filtered tick data is being provided by a broker:
- its frontend would crash / would be sluggish if immense unfiltered data is connected to it;
- a broker would need to incurr capex on servers and other infrastructure and increased opex on data transfer.

The reason why a data feed provider may decide to filter ticks is to save money on infrastructure and data transfer costs.
This does not make any sense for any number of reasons. What do you mean by "filtered" data? Do you mean they deliberately drop ticks? I would be shocked to find out that is the case. What sort of saving are they making by doing this anyway? 5%? Big deal! To make a significant saving they would have to drop 50% but that would make the data useless. I an understand perhaps doing this for tick data but what about 1 minute historical aggregated data? Why skimp on that when there are no savings to be made?

Why would they mangle the historical data anyway? Why not just provide historical data "as is" and save themselves the effort of mangling it in the first place? The real-time data appears to be more correct yet what you are implying is they are mangling the real-time more than than historical to save resources. It doesn't add up. If I was going to provide a historical data feed to my customers, I would not go out of my way to "filter" (read mangle) the data. That would require even more resources than providing the raw data in the first place. I might remove spikes but that's it. Is this some sort of trick played by the IT people to get more money under the guise of saving money? Or are we talking about bean counters taking over and not realizing what they are doing? I'm referring to historical data here, in particular 1 minute data, where the data, mangled or not, occupies the same amount of space on a disk drive and requires the same amount of bandwidth over the network. Give me a break please, this is not rocket science :shock:

Perhaps I'm missing soemthing here so I'm prepared listen and learn.
Last edited by janus on 01 Apr 2010, edited 2 times in total.

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 01 Apr 2010

In my view the thing they most omit is real-time or historical updates to bid/ask sizes when the inside price hasn't changed (this is a common flaw when people try to code "pulled quote" type algorithms to attempt to detect fake-outs and manipulation, because they're not seeing every size update typically). The thing that's next most likely to be incorrect is updates to the inside bid/ask price, which are sometimes handled inconsistently from one feed to another or even from one server to another on the same feed (this is a potential issue in attempting to systematize "MD" type algorithms - if you can't rely on the inside bid/ask price being right, you can't categorize the trades 100% reliably, and although some feeds are better than others, they're basically all somewhat limited in this regard). The thing that's least likely to be incorrect (although it does happen sometimes, even with paid data feeds) are the trade ticks themselves, which, usually, except for correction ticks or "bad ticks" that are far from the market (yet, in some debateable cases may actually be real transactions) which may be filtered differently by different feeds or even filtered differently on the same feed real-time vs. historical. However, these correction ticks and bad ticks usually add up to less than 1% of a day's volume, so large discrepancies are not the result of this and are more likely coding issues (assuming you're dealing with a "real" tick data feed, and not just snapshots, which have their own unique problems.)

Tresor, I'm interested in further elaboration on this as well - what you found exactly in this experience with OEC (and any others with which you trod the same path).

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 01 Apr 2010

In my view the thing they most omit is real-time or historical updates to bid/ask sizes when the inside price hasn't changed (this is a common flaw when people try to code "pulled quote" type algorithms to attempt to detect fake-outs and manipulation, because they're not seeing every size update typically).
My observation on my other brokers screen is the real-time bid/ask sizes and the queue depth are updated correctly. I know as when I place a limit order I see it appear immediately in the queue. When I delete it I see it go away. I haven't tried this with IB to see if the same thing appears on the tws screen but I expect it to. As for historical then I don't expect them to provide a complete audit of all bid/ask quotes. That would requires enormous amounts of resources.

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 01 Apr 2010

In fast moving markets, brokers typically combine adjacent updates if the price is the same so that they don't send multiple updates per second, or in some cases, multiple bid/ask updates per last trade price when the bid/ask prices are still the same. If you place a single order when "not a lot" is going on, it will often appear immediately in real-time and even show up in the historical tick feed as a bid/ask size change (and usually so, if you put in a new inside BB/O). But, in fast moving markets and in program trading conditions where there can be dozens or even hundreds of bid/ask size updates per trade, this is not the case and they combine them so they are not overwhelmed. This typically is not a problem for end users, except when they try to code "pulled quote" type algorithms.

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 53 times

Postby Tresor » 01 Apr 2010


This does not make any sense for any number of reasons. What do you mean by "filtered" data? Do you mean they deliberately drop ticks?
This makes perfect sense. Yes, I mean they have been dropping ticks on purpose.

I attach a screenshot that I once taken form OEC developers' forum for the purpose of quarelling with some moron soliciting OEC servces on Traderslaboratory.com

Pay attention to the underlined sentence: ''This way you will receive only ticks, and no data is missing''. SergeK ( OEC moderator) explains to the user how to receive unfiltered data.

Just go to OEC forum for developers and you will find out more.
Attachments
OEC unfiltered.png
(48.13 KiB) Downloaded 2279 times

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 01 Apr 2010

That's a good screenshot and is typical of this type of discussion with a broker or data feed. You'll notice right below what you referenced, the moderator explicitly states that although they have an alternative method for receiving every trade tick, they have NO alternative method for receiving every bid/ask update (just like the problems I'm describing above regarding implementation of "pulled quote" or "MD" type algorithms - imagine trying to classify every trade as a bid or ask trade, then finding out you don't have all of the bid/ask updates...).
Last edited by Bruce DeVault on 01 Apr 2010, edited 2 times in total.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 01 Apr 2010

In fast moving markets, brokers typically combine adjacent updates if the price is the same so that they don't send multiple updates per second, or in some cases, multiple bid/ask updates per last trade price when the bid/ask prices are still the same.
I see. It doesn't explain the huge discrepancies between IB's historical, often widely incorrect, volume data and their apparently more reliable real-time volume data.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 01 Apr 2010

This makes perfect sense. Yes, I mean they have been dropping ticks on purpose.
You are missing the point. See my previous post. Also note that I've already discussed this with TSS support and they are in agreement with me that IB is doing something very wrong. In other words, it's as if they are dropping volume ticks in the historical data sets but not the real-time ones.
Last edited by janus on 01 Apr 2010, edited 1 time in total.

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 01 Apr 2010

I believe also as I stated upthread that IB has a general issue. This same conversation has taken place on a whole host of different trading platforms. Often, they can come up with work-arounds and ways to address it and make it "more right" but it's a frequent type of questioning.

Remember, IB isn't a tick data feed. They have a problem, but it isn't the same as the problem Tresor is describing - IB's problem has to do with how snapshots are handled which is more of a long-standing "snafu", whereas Tresor is describing a capacity/economics problem resulting in the data feed deliberately throttling or filtering the data going out.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 01 Apr 2010

Remember, IB isn't a tick data feed. They have a problem, but it isn't the same as the problem Tresor is describing - IB's problem has to do with how snapshots are handled which is more of a long-standing "snafu", whereas Tresor is describing a capacity/economics problem resulting in the data feed deliberately throttling or filtering the data going out.
Still missing the point. IB's historical data feed is less accurate than the real-time feed. That's not a symptom of trying to save resources by reducing the number of ticks in a real-time feed. On the contrary, they are mangling the historical data sets when they need not do it in the first place since they already have calculated the real-time ones, which happen to be more accurate. I repeat, I'm more focused on the 1 minute data set, not the tick data sets. I understand the issues with real-time tick data feeds but that's a different matter.
Last edited by janus on 01 Apr 2010, edited 1 time in total.

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 01 Apr 2010

I do understand - I believe it to be more of a procedural problem as I said than an economic one in this case. What I suspect, although I don't speak for anyone but myself in this, is that they may possibly be in the predicament of being unable to change it very much short of a major version revision because it's been this way so long.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 01 Apr 2010

I do understand - I believe it to be more of a procedural problem as I said than an economic one in this case. What I suspect, although I don't speak for anyone but myself in this, is that they may possibly be in the predicament of being unable to change it very much short of a major version revision because it's been this way so long.
That's fine. I can live with that, at least until I ever need more accurate 1 minute volume data, which is unlikely (I hope).

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 01 Apr 2010

For a good time, you might want to read some similar threads on other platforms, such as http://www.NT-support2.com/vb/ ... hp?t=11056 (note especially the quote "You should have a warning on your page saying that if you use Interactive Brokers for your data feed that any strategy that uses volume will not be accurate.") and how NT describes having to use a variety of methods to "fix" the known incorrect volume. This isn't regarding the exact problem you're describing (it's a problem with real-time data vs. your problem with historical), but it's conceptually similar, and what's striking is how both sides of the discussion, the user reporting the problem and the platform vendor acknowledge that they and other platforms routinely have to do work-arounds to fix the data coming in because it's known to be inadequate when taken at face value. Google for "interactive brokers volume error OR problem OR bug OR issue" and you'll be busy the rest of the day reading all about these types of things. :)
Last edited by Bruce DeVault on 01 Apr 2010, edited 1 time in total.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 01 Apr 2010

As I understand it those who do require accurate historical volume data (and perhaps this relates to price as well) they download historical data from another source and replace the data in MC's database, and do this on a daily or weekly basis. I can't be bothered so I will put up with the mangled volume data for the moment.

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 53 times

Postby Tresor » 01 Apr 2010

You are missing the point.
Probably because of my poor English.
See my previous post. Also note that I've already discussed this with TSS support and they are in agreement with me that IB is doing something very wrong. In other words, it's as if they are dropping volume ticks in the historical data sets but not the real-time ones.
This is very simple. Brokers and data feed providers filter ticks. I explained reasons on their end for filtering the ticks.

The filtering of ticks results in incorrect volume real-time. The IT guys at brokers' openly admit they are unable to filter ticks to achieve realistic volume real-time. What they typicaly do is to send to MC (the next day) a daily bar data with correct volume for the previous day.

If you want the proper volume (the correct volume by the exchange), you need to make friends with your data provider's IT guys.

If you make friends with them, they will tell you which communications fields in their API you need to connect your MC to receive the unchanged exchange data feed.

If this is done, your historical volume and real-time volume will be in perfect match. And they will be in perfect match with the official volume report by the exchange.

What's more, your data will not be delayed. E.g. a year ago the filtering procedure at OEC was faster than the filtering procedure at Transact by a few seconds, which I documented here http://forum.tssupport.com/download.php?id=2121

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 01 Apr 2010

Regarding OEC specifically, we simply need to check with TSS and see which method they use, and if they need to use the SubscribeTicks method to improve quality specifically for OEC I'm sure they can consider to do this in an upcoming revision. That's separate and apart from the issue with IB, which isn't regarding tick data.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 01 Apr 2010

This is very simple. Brokers and data feed providers filter ticks. I explained reasons on their end for filtering the ticks.
I understand all that. So, what does MC do when it tries to plot a historical volume for a 1 minute chart? Does it request 1 minute snapshot data (if there is such a thing) from IB, or tick data and build the 1 minute data from it?

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 01 Apr 2010

The way IB's historical data works, it doesn't store "minute data" like a lot of data feeds do, but rather you tell IB you want "1 minute" as your interval, and it consults its own database of snapshots and combines them into groupings that make 1 minute groups, then gives you that. You can also request things like 1 second bars, 1 day bars, etc. and they're all handled in a general way, which is by combining the native data they stored to make what you requested. This is in stark contrast with most data feeds like eSignal, IQFeed, etc. which natively store three types of data you can request (rather than giving you the choice of requesting an arbitrary interval): daily, 1 minute, and 1 tick.

They also have extensive rules designed to protect them from you requesting too much historical data, and if you do this, they will cut off your response and not send you everything.

Here's a partial list of their restrictions:
Historical Data Limitations
Historical data requests are subject to the following limitations:

· Historical data requests can go back one full calendar year.

· Each request is restricted to duration and bar size values that return no more than 2000 bars (2000 bars per request).

All of the API technologies support historical data requests. However, requesting the same historical data in a short period of time can cause extra load on the backend and subsequently cause pacing violations. The error code and message that indicates a pacing violation is:

162 - Historical Market Data Service error message: Historical data request pacing violation

The following conditions can cause a pacing violation:

· Making identical historical data requests within 15 seconds;

· Making six or more historical data requests for the same Contract, Exchange and Tick Type within two seconds.

Also, observe the following limitation when requesting historical data:

· Do not make more than 60 historical data requests in any ten-minute period.
It has been my hard earned experience that pacing violations are not always reported as such - in some cases, their API simply stops responding, and you (the developer) never know why you didn't get everything. If you respond to not getting everything by requesting whatever you didn't get, that puts you further in jeopardy because you've now increased the time out you're subject to, and if you do this too often ("hammer their server") your account will be disabled. Consequently, platform developers have to tread an extremely fine line of not requesting too much, and if they don't get all they requested, they have to be careful about re-requesting what they didn't get too quickly (all the while, users complaining that their charts take too long to open, platform X does it faster, etc. - a true balancing act).

All of this is in very marked constrast to the way you're treated when you're on a typical paid data feed. These types of restrictions are indicative of the fact that IB knows you're not paying for this, and therefore that it's an "it is what it is" type situation.

Don't get me wrong - I think IB has a good brokerage interface "in general" and I like working with them. But I would never make the mistake today of assuming their data service is just like a paid one philosophically - it's a whole different type of thing and has to be handled with this understanding.
Last edited by Bruce DeVault on 01 Apr 2010, edited 7 times in total.

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 53 times

Postby Tresor » 01 Apr 2010

That's separate and apart from the issue with IB, which isn't regarding tick data.
Yes, it is separate. I just contributed in hope that it might ring a bell.

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 53 times

Postby Tresor » 01 Apr 2010

I understand all that. So, what does MC do when it tries to plot a historical volume for a 1 minute chart? Does it request 1 minute snapshot data (if there is such a thing) from IB, or tick data and build the 1 minute data from it?
My experiences with OEC and Transact (filtered ticks) surely are separate from the IB issue (snapshot ticks).

When I plotted 1 min / 5 min / 10 min chart real-time based on OEC / Transact data) the bars would show some volume data.

After closing MC and reopening it, the volume data on bars were different than the volume I had observed real-time on the same bars. I do not know the reason for this adjustment. I documented it in lenghth within my limited layman capabilities in the thread that I mentioned in one of my previous posts.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 01 Apr 2010

The way IB's historical data works, it doesn't store "minute data" like a lot of data feeds do, but rather you tell IB you want "1 minute" as your interval, and it consults its own database of snapshots and combines them into group
Now you confused me. MC stores three types of data - tick, 1 minute and daily. So, if I plot a 1 minute chart and add the volume study, how exactly does it construct the volume chart? Does it use the 1 minute data set, an dif not there, will it request 1 minute data from IB, store it and plot it?

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 01 Apr 2010

MC stores three types of data - tick, 1 minute and daily. So, if I plot a 1 minute chart and add the volume study, how exactly does it construct the volume chart? Does it use the 1 minute data set, an dif not there, will it request 1 minute data from IB, store it and plot it?
Here's what happens - you add a 1 minute chart to MC. 1 minute is a native storage type for MC, so it doesn't need to combine anything (if it has the data), therefore MC looks to see if it has the 1 minute data for these dates already. If it does, it uses that. If not, it tells IB "give me the 1 minute data for these dates". IB doesn't store 1 minute data natively, so it looks at its saved snapshots/ticks and builds "1 minute" bars on the fly, then gives those to MC, which stores them as "1 minute native" data in quote manager. Confused? It's confusing. But, in short, IB is a little different than everybody else in that they don't treat 1 minute as a native type and instead will let you request bars of any arbitrary length such as 15 seconds, etc. and whatever you ask for, they combine the data they stored to make for you. It's in this process where the problem likely falls. If you can make it manifest this issue on TWS charting alone (and that's the case) it isn't just something MC is doing incorrectly, but something IB is doing in a less than perfect way.
Last edited by Bruce DeVault on 01 Apr 2010, edited 2 times in total.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 01 Apr 2010

When I plotted 1 min / 5 min / 10 min chart real-time based on OEC / Transact data) the bars would show some volume data.

After closing MC and reopening it, the volume data on bars were different than the volume I had observed real-time on the same bars. I do not know the reason for this adjustment. I documented it in lenghth within my limited layman capabilities in the thread that I mentioned in one of my previous posts.
That's exactly the issue I'm talking about. It's even replicable on TWS alone. If you plot a chart of say 1 minute using TWS, the volume is the historical volume but the volume displayed in the table window in TWS is the real-time volume and is typically much higher when you do the sums for the chart. I've asked IB why the difference and I get back the response that the snapshots of the two data streams are different. So, why not make them the same? As Bruce says, it's probably procedural and they are reluctant to change it.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 01 Apr 2010

But, in short, IB is a little different than everybody else in that they don't treat 1 minute as a native type and instead will let you request bars of any arbitrary length such as 15 seconds, etc. and whatever you ask for, they combine the data they stored to make for you. It's in this process where the problem likely falls.
I see. Yes it makes sense but it as you know it doesn't excuse them for not providing reasonably accurate historical volume data. The real-time data appears more accurate on the tws screen (and on MC while collecting and displaying real-time) but the historical volume data is crook (and on MC too when the chart is re-loaded or MC is restarted).

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 53 times

Postby Tresor » 01 Apr 2010

I've asked IB why the difference and I get back the response that the snapshots of the two data streams are different.
Hahahaha,

When I hear ''two different streams'' or ''two different sources'', it makes me laugh.

There is only one stream / source that the broker / data feed provider is given by and from the exchange!!!

For some reasons the broker decides to split the correct stream into two. The correct stream / source is given to prefered clients and the other (incorrect) stream / source to the remaining 99.99%. As you no doubt have guessed the 99.99% are losers.

It's a bit off-topic but browse any trading forum and you will find a lot of trading gurus supposedly making money on volume based strategies and praising OEC's or Transact's ''high quality data feed'' and you will also find sheeple following them...

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 01 Apr 2010

When I hear ''two different streams'' or ''two different sources'', it makes me laugh.

There is only one stream / source that the broker / data feed provider is given by and from the exchange!!!
Actually, I was referring to the two data streams that are available from IB to all us "common folk". One is the real-time one (they call it Market Data)and the other the historical one (they call it HMDS or Historical Market Data Service).

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 53 times

Postby Tresor » 01 Apr 2010

Actually, I was referring to the two data streams that are available from IB to all us "common folk".
Then ask IB if they by any chance offer a stream for uber-menschen.

Or change your provider in case they don't or in case you do not qualify ;)


Return to “MultiCharts”