Delays receiving real-time data
Delays receiving real-time data
I'm running MC 12 with IQF data and IB API, trading in PT. The changelog for MC12 indicated that this issue might have been fixed, but for me it persists.
I've been logging this issue as follows. I wrote a roughly 10-line signal that, every Data1 bar (which is 1 second), prints the bar time (time_s) and computertime_s. I've been running this signal on 4 instruments: ES, US(ZB), NG, and Soybeans. The behavior is the same in all instruments. Sometimes the two times are equal; usually they differ by 1-2 seconds which is acceptable. Sometimes the difference is 5 seconds. Occasionally it's 10-20 seconds.
As a side effect, when there's a delay of several seconds, sometimes multiple bars are received at the same computer time, and so PT catches up by calculating all of those bars at the same computer time. For instance, maybe there was volume at 10:00:01, then again at 10:00:05, then again at 10:00:07, but no calculation occurs until computer time 10:00:12 (at which point a triple-calculation occurs).
When a several-second delay coincides with an entry (in my trading signal), my entry order gets delayed, which is frustrating. It has not yet coincided with an exit order, but that's probably inevitable.
The IB API log provides no clues. During the delays, the log has the same messages it has during non-delayed times.
Whenever a delay happens in my test signal, it also happens in my trading signal in the same instrument.
When a delay happens in one instrument, I think it happens in all instruments. I'm not yet sure, because usually when I look for a matching delay, the instruments didn't have volume at the same seconds. But from the evidence I've gathered so far, when they do have bars at the same time and a delay occurs in one instrument, it occurs in both. This happened recently, with the same delay happening in ES and US.
I'm trading from a dedicated OVH server with fast internet, and a data bar is a very small download, so I'm not inclined to blame the internet connection, but I also can't rule it out.
There is a way to generate a delay on purpose: if I export one of the symbols in QM, that symbol experiences a delay every time. This made me wonder if running QM or MC while trading in PT caused occasional delays, so I closed everything but PT. However, the delays continue to occur.
Does anyone else experience these delays? Is there anything I can do about it?
I've been logging this issue as follows. I wrote a roughly 10-line signal that, every Data1 bar (which is 1 second), prints the bar time (time_s) and computertime_s. I've been running this signal on 4 instruments: ES, US(ZB), NG, and Soybeans. The behavior is the same in all instruments. Sometimes the two times are equal; usually they differ by 1-2 seconds which is acceptable. Sometimes the difference is 5 seconds. Occasionally it's 10-20 seconds.
As a side effect, when there's a delay of several seconds, sometimes multiple bars are received at the same computer time, and so PT catches up by calculating all of those bars at the same computer time. For instance, maybe there was volume at 10:00:01, then again at 10:00:05, then again at 10:00:07, but no calculation occurs until computer time 10:00:12 (at which point a triple-calculation occurs).
When a several-second delay coincides with an entry (in my trading signal), my entry order gets delayed, which is frustrating. It has not yet coincided with an exit order, but that's probably inevitable.
The IB API log provides no clues. During the delays, the log has the same messages it has during non-delayed times.
Whenever a delay happens in my test signal, it also happens in my trading signal in the same instrument.
When a delay happens in one instrument, I think it happens in all instruments. I'm not yet sure, because usually when I look for a matching delay, the instruments didn't have volume at the same seconds. But from the evidence I've gathered so far, when they do have bars at the same time and a delay occurs in one instrument, it occurs in both. This happened recently, with the same delay happening in ES and US.
I'm trading from a dedicated OVH server with fast internet, and a data bar is a very small download, so I'm not inclined to blame the internet connection, but I also can't rule it out.
There is a way to generate a delay on purpose: if I export one of the symbols in QM, that symbol experiences a delay every time. This made me wonder if running QM or MC while trading in PT caused occasional delays, so I closed everything but PT. However, the delays continue to occur.
Does anyone else experience these delays? Is there anything I can do about it?
-
- Posts: 275
- Joined: 22 Apr 2014
- Has thanked: 70 times
- Been thanked: 72 times
Re: Delays receiving real-time data
Hi Pcrespo,
What operating system are you using? Do you have IQFeed and Multicharts excluded from the antimalware service? What is the first contract in your instrument list in Portfolio trader? It should the the one with the highest tick rate. Thanks to ABC Trading for that invaluable tip.
I have many persistent problems with Portfolio Trader and IB. It gets the job done in the end but not without headaches along the way.
Alex
What operating system are you using? Do you have IQFeed and Multicharts excluded from the antimalware service? What is the first contract in your instrument list in Portfolio trader? It should the the one with the highest tick rate. Thanks to ABC Trading for that invaluable tip.
I have many persistent problems with Portfolio Trader and IB. It gets the job done in the end but not without headaches along the way.
Alex
Re: Delays receiving real-time data
Windows Server 2012; we have no anti-malware installed (Defender is not included by default).Do you have IQFeed and Multicharts excluded from the antimalware service?
In that case, we had the wrong order of instruments. I've just changed it and will see if the problem goes away. Thanks!What is the first contract in your instrument list in Portfolio trader? It should the the one with the highest tick rate.
-
- Posts: 275
- Joined: 22 Apr 2014
- Has thanked: 70 times
- Been thanked: 72 times
Re: Delays receiving real-time data
I am using a Supermicro dedicated server for Portfolio Trader and I have experienced clock drift where my system clock time differs from the actual time. You may want to use a third party tool to contact time servers to sync your time. For what ever reason, Windows 2012r2 was not syncing properly.
Re: Delays receiving real-time data
Changing the order of PT instruments didn't get rid of the delays. I like the time sync suggestion. For now, I've changed the NTP servers to the appropriate pool.ntp.org ones and changed the polling frequency to 1min instead of 1hour. If that doesn't work, I'll investigate whether the server time is inaccurate; if it is, I'll look into 3rd-party ntp/nts clients like you mentioned.
-
- Posts: 275
- Joined: 22 Apr 2014
- Has thanked: 70 times
- Been thanked: 72 times
Re: Delays receiving real-time data
I am using something called Dimension 4 set to sync every five minutes. I also have delays with Portfolio Trader but I have just accepted them. For example yesterday it took 22 seconds to fill five market orders. It is the delay in order generation that is troubling.
For example order four was filled at 9:33.01. Order 5 was generated at 9:33:122 and filled at 9:33:126. The fill was fast but the generation was slow. I am reading orders from a text file and sending all of them at the same time at so I don't know why it took 12 seconds to generate the last order. There is no signal.
The good news is that most of the time I can fill everything in a few tenths of a second but once in a while there is this delay. I am sure there is a valid explanation but it I have not been able to track it down. My strategy is not very time dependent. A delay here or there does not make a difference to the end result.....fortunately.
For example order four was filled at 9:33.01. Order 5 was generated at 9:33:122 and filled at 9:33:126. The fill was fast but the generation was slow. I am reading orders from a text file and sending all of them at the same time at so I don't know why it took 12 seconds to generate the last order. There is no signal.
The good news is that most of the time I can fill everything in a few tenths of a second but once in a while there is this delay. I am sure there is a valid explanation but it I have not been able to track it down. My strategy is not very time dependent. A delay here or there does not make a difference to the end result.....fortunately.
Last edited by hughesfleming on 24 Aug 2018, edited 1 time in total.
- TJ
- Posts: 7744
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Delays receiving real-time data
Have you done a PING test to ibkr?
The instruction is on their website.
The latency could be in your ISP.
The instruction is on their website.
The latency could be in your ISP.
Re: Delays receiving real-time data
Changing the time sync settings hasn't gotten rid of the delays. I think the delays are real, not just a matter of the PC time being inaccurate.
My ping to the IB server is 15ms, but if anything, the delay is with IQF, not IB. The signal's calculation is delayed, suggesting that I'm either receiving the data bar late, or PT's calculation is lagging behind the data stream. When my orders are generated late, it is always because the calculation was late. To be clear, I'm only talking about when volume was present between calculations (since a "delay" without volume is proper behavior).
I grabbed the IQF ip from the connection manager and ran a continuous, timestamped ping to a text file. My connection to IQF doesn't seem to be the issue. My pings range from 40-125 ms. During the most recent delays in my signal, the ping was not higher than normal. For instance, I had a 7-second delay in my signal, but the ping was only 90ms. No seconds-long pings have been recorded so far.
Does this mean the issue is with PT/MC?
My ping to the IB server is 15ms, but if anything, the delay is with IQF, not IB. The signal's calculation is delayed, suggesting that I'm either receiving the data bar late, or PT's calculation is lagging behind the data stream. When my orders are generated late, it is always because the calculation was late. To be clear, I'm only talking about when volume was present between calculations (since a "delay" without volume is proper behavior).
I grabbed the IQF ip from the connection manager and ran a continuous, timestamped ping to a text file. My connection to IQF doesn't seem to be the issue. My pings range from 40-125 ms. During the most recent delays in my signal, the ping was not higher than normal. For instance, I had a 7-second delay in my signal, but the ping was only 90ms. No seconds-long pings have been recorded so far.
Does this mean the issue is with PT/MC?
-
- Posts: 275
- Joined: 22 Apr 2014
- Has thanked: 70 times
- Been thanked: 72 times
Re: Delays receiving real-time data
One last thing to try that seems to have helped me is to turn on fast tracing in the registry. Take a look at this thread. At times it would seems that orders get stuck in the queue. It has to do with empty batches of orders. The thread has all the details.
I have seen the delay you are talking about. You might also want to check your logs to make sure that you are not seeing a disconnect from IQFeed. This was causing me lots of headache with a different data provider but that doesn't mean that it can't happen.
viewtopic.php?f=1&t=47587&p=129250&hili ... ce#p129250
Maybe someone at Multicharts can explain why this is not a default setting.
regards,
Alex
I have seen the delay you are talking about. You might also want to check your logs to make sure that you are not seeing a disconnect from IQFeed. This was causing me lots of headache with a different data provider but that doesn't mean that it can't happen.
viewtopic.php?f=1&t=47587&p=129250&hili ... ce#p129250
Maybe someone at Multicharts can explain why this is not a default setting.
regards,
Alex
Re: Delays receiving real-time data
Several things came to mind:
- "Pings" are not at all representative of real latency under load.
- your network card/TCP stack settings and CPU processing power will somewhat influence your overall "trading system" latency, especially in times of heavy trading/tick count across several instruments
- bear in mind that IQF servers are in Dallas...which means data have to travel:
from Chicago or NY to Dallas
then from Dallas to you
then get processed by MC
then from MC to IB
- so, try trading straight with IB data instead of IQF (to cut 2 legs) and see if you experience the same kind of delays AND EXECUTION.
Since you most probably are not trading tick bars, getting fewer ticks from IB (every 200-250ms) might actually help across the board.
- "Pings" are not at all representative of real latency under load.
- your network card/TCP stack settings and CPU processing power will somewhat influence your overall "trading system" latency, especially in times of heavy trading/tick count across several instruments
- bear in mind that IQF servers are in Dallas...which means data have to travel:
from Chicago or NY to Dallas
then from Dallas to you
then get processed by MC
then from MC to IB
- so, try trading straight with IB data instead of IQF (to cut 2 legs) and see if you experience the same kind of delays AND EXECUTION.
Since you most probably are not trading tick bars, getting fewer ticks from IB (every 200-250ms) might actually help across the board.
Re: Delays receiving real-time data
@Alex - I've turned FastTracing on for good measure, but the problem must be unrelated because the calculation delays happen under all conditions, not just during trades and not just during high volume. They happen when trading just one instrument - any instrument during any hours. Sometimes a delay coincides with when I'm about to make a trade, but I'm pretty sure that when it does, the delay would have happened regardless.
@Zheka - yeah, switching to IB data was going to be my last resort, which seems to be where I'm headed. I just ran a speed test to a server in Dallas. I don't know how indicative it is, but I got 116 MB/s (927 Mb/s) Down and 26 MB/s Up. As for cpu, PT/MC is only using 8-20% cpu and the other processes combined are using <1%. Does every IQF user receive occasional 15-second delays and simply accept them, or does everyone have a faster system than this? I feel like there's something else going on.
@Zheka - yeah, switching to IB data was going to be my last resort, which seems to be where I'm headed. I just ran a speed test to a server in Dallas. I don't know how indicative it is, but I got 116 MB/s (927 Mb/s) Down and 26 MB/s Up. As for cpu, PT/MC is only using 8-20% cpu and the other processes combined are using <1%. Does every IQF user receive occasional 15-second delays and simply accept them, or does everyone have a faster system than this? I feel like there's something else going on.
-
- Posts: 742
- Joined: 09 Apr 2010
- Location: Texas
- Has thanked: 483 times
- Been thanked: 274 times
- Contact:
Re: Delays receiving real-time data
It might be interesting to try the Windows command "tracert" it will trace every step from your pc to the destination and show where the biggest delays are.@Alex - I've turned FastTracing on for good measure, but the problem must be unrelated because the calculation delays happen under all conditions, not just during trades and not just during high volume. They happen when trading just one instrument - any instrument during any hours. Sometimes a delay coincides with when I'm about to make a trade, but I'm pretty sure that when it does, the delay would have happened regardless.
@Zheka - yeah, switching to IB data was going to be my last resort, which seems to be where I'm headed. I just ran a speed test to a server in Dallas. I don't know how indicative it is, but I got 116 MB/s (927 Mb/s) Down and 26 MB/s Up. As for cpu, PT/MC is only using 8-20% cpu and the other processes combined are using <1%. Does every IQF user receive occasional 15-second delays and simply accept them, or does everyone have a faster system than this? I feel like there's something else going on.
Example: in the command prompt type: tracert interactivebrokers.com and hit "enter". It takes a minute to run so be patient.
I'd also make sure that MC's "Bugslayer" is disabled in the registry, it can slow things down too.
-
- Posts: 275
- Joined: 22 Apr 2014
- Has thanked: 70 times
- Been thanked: 72 times
Re: Delays receiving real-time data
Hi pcrespo,
It is very hard to solve these problems sometimes. I had many in the beginning. You might want to send your logs to Multicharts. It does sound like you have another problem. I am using IQFeed for US Equities but my server is in Frankfurt so orders and data still have to cross the Atlantic. I do get a delay once in a while but things have been working well recently. A typical daily re-balance will take 3-5 seconds on average for eight positions. Sometimes it is almost instantaneous. Orders are sent to IB in Zug. If things are working normally, your fills should be similar. My setup is less than ideal as far as latency is concerned so you shouldn't do much worse than that. If you are getting these 15 second delays all the time, then that is not normal.
regards,
Alex
It is very hard to solve these problems sometimes. I had many in the beginning. You might want to send your logs to Multicharts. It does sound like you have another problem. I am using IQFeed for US Equities but my server is in Frankfurt so orders and data still have to cross the Atlantic. I do get a delay once in a while but things have been working well recently. A typical daily re-balance will take 3-5 seconds on average for eight positions. Sometimes it is almost instantaneous. Orders are sent to IB in Zug. If things are working normally, your fills should be similar. My setup is less than ideal as far as latency is concerned so you shouldn't do much worse than that. If you are getting these 15 second delays all the time, then that is not normal.
regards,
Alex
Re: Delays receiving real-time data
Hmm:-) I was talking about latency (and tuning NIC/TCP stack is a big area on its own). And since IQF uses TCP (for completeness and reliability), any "congestion" on the way to/from or on your server might lead to delays.@Zheka - yeah, switching to IB data was going to be my last resort, which seems to be where I'm headed. I just ran a speed test to a server in Dallas. I don't know how indicative it is, but I got 116 MB/s (927 Mb/s) Down and 26 MB/s Up.
PT uses 1 core to process the data (and multithreading it has been requested long ago!). And you cannot really judge by looking in the task mngr (even if looking by core), if CPU is a bottleneck or not.As for cpu, PT/MC is only using 8-20% cpu and the other processes combined are using <1%.
And if CPU does not occasionally manage, then this would exacerbate "network congestion".
So, a CPU with higher frequency/ "turbo" mode is much preferable to a Xeon with many but slow cores. What's your hardware?
Also, change Power options to Performance.
All the above points will hardly cause 15-sec delays.Does every IQF user receive occasional 15-second delays and simply accept them, or does everyone have a faster system than this? I feel like there's something else going on.
But try changing things as people recommend and see if things change.
Re: Delays receiving real-time data
So today, a long delay happened when it was time to exit a position. Trading ES on a 1-second Data1 timeframe (sync mode, IOG disabled), we were supposed to exit at 16:00:01 but instead exited at 16:00:54 even though there was plenty of volume (and thus, opportunities for the signal to calculate) in between.
This time, my colleague happened to be watching the screen as the delay unfolded. He says data was coming in each second on the chart, so apparently the lag has to do with PT/MC, not data.
The signal's log shows that the 25 calculations from 16:00:00 to 16:00:24 all took place at 16:00:54, meaning PT scrambled to play catch-up after the delay.
@Alex - Just now, per Anna's suggestion in the "Execution Delay" thread, I've completely disabled "Trace" logs in the registry. However, the "BugSlayer" directory has several other settings one can disable. Are you saying I should disable all of them?
@Zheka - Based on the new development, it seems CPU is the bottleneck. The server isn't especially fast because we didn't think it needed to be. It has 4 Xeon E3-1231 v3's at 3.4 ghz. If our delays continue, do you think we should rent a different server? One with a higher clock frequency per core, but not necessarily more cores?
Power settings were already set to Performance, but I noticed that the processor allocation was adjusted for background processes rather than programs, so I changed that.
Regarding the TCP/IP stack, do TCP Optimizer's "optimal" settings cover it or do I have to tune it manually?
Thanks guys, I feel like I'm gradually getting to the bottom of this with your help.
This time, my colleague happened to be watching the screen as the delay unfolded. He says data was coming in each second on the chart, so apparently the lag has to do with PT/MC, not data.
The signal's log shows that the 25 calculations from 16:00:00 to 16:00:24 all took place at 16:00:54, meaning PT scrambled to play catch-up after the delay.
@Alex - Just now, per Anna's suggestion in the "Execution Delay" thread, I've completely disabled "Trace" logs in the registry. However, the "BugSlayer" directory has several other settings one can disable. Are you saying I should disable all of them?
@Zheka - Based on the new development, it seems CPU is the bottleneck. The server isn't especially fast because we didn't think it needed to be. It has 4 Xeon E3-1231 v3's at 3.4 ghz. If our delays continue, do you think we should rent a different server? One with a higher clock frequency per core, but not necessarily more cores?
Power settings were already set to Performance, but I noticed that the processor allocation was adjusted for background processes rather than programs, so I changed that.
Regarding the TCP/IP stack, do TCP Optimizer's "optimal" settings cover it or do I have to tune it manually?
Thanks guys, I feel like I'm gradually getting to the bottom of this with your help.
-
- Posts: 275
- Joined: 22 Apr 2014
- Has thanked: 70 times
- Been thanked: 72 times
Re: Delays receiving real-time data
Hi pcrespo, I just have Fast tracing enabled. I have not switched off logging completely. I think it is quite extreme to consider doing that. You should set the OS to prioritize foreground processes. You have enough clock speed. The difference between a v3 and v5 xeon is not large. You would only see a very small difference running optimizations for example. About six months ago I had issues running a new server because of meltdown/spectre bios updates so make sure you are using a current bios. E-mail your service provider and ask them. They would know as they would have had to deal with this.
If I were you, I would rent another server from another provider and see if you have the same problem. That is the only way to eliminate the hardware and network from the problem. Once you can duplicate the issue, look though your code to make sure that everything is in order there. You mentioned that sometimes you are hitting 20% cpu utilization which seems high if your are trading just a few instruments.
The bad news is that I get these delays sometimes as well and I have never been able to track down why. Everything ran this smoothly this week 24/5 without any issues so it must be a random network delay receiving data from IQFeed that can happen. I did have trouble with MC12 and rolled back and did a fresh installation of MC11 including IB Gateway. Things have never worked better!
If I were you, I would rent another server from another provider and see if you have the same problem. That is the only way to eliminate the hardware and network from the problem. Once you can duplicate the issue, look though your code to make sure that everything is in order there. You mentioned that sometimes you are hitting 20% cpu utilization which seems high if your are trading just a few instruments.
The bad news is that I get these delays sometimes as well and I have never been able to track down why. Everything ran this smoothly this week 24/5 without any issues so it must be a random network delay receiving data from IQFeed that can happen. I did have trouble with MC12 and rolled back and did a fresh installation of MC11 including IB Gateway. Things have never worked better!
Re: Delays receiving real-time data
Yes, CPU seems to be "good enough", though you would be better off with 1x i7-7700K (or 6700K).. Why do you need a 4-CPU set-up on a production machine?
TCP Optimizer will mostly cover it, just read the docs/search the web to understand what specific settings mean and how "networking" works.
You can also download your network adapter utility driver and tune it similarly.
BUT, again, experiment with an IB-only setup.
Launch 2 PT instances and feed one with IQF and another with IB data. Judge by absence of delays AND FILLS.
TCP Optimizer will mostly cover it, just read the docs/search the web to understand what specific settings mean and how "networking" works.
You can also download your network adapter utility driver and tune it similarly.
BUT, again, experiment with an IB-only setup.
Launch 2 PT instances and feed one with IQF and another with IB data. Judge by absence of delays AND FILLS.
- TJ
- Posts: 7744
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Delays receiving real-time data
@TJ - dedicated bare metal.
@Zheka - quad core was the lowest available from OVH. I'll shop around for a server with faster (and possibly fewer) cores.
Given that the data was streaming on time when the recent delay happened, I don't see how the problem can be with IQF. Also, switching to IB data would be more complicated than I realized due to pacing violations. For now, I'll see if my recent tweaks and a hardware change do anything. If not, perhaps that will be a good time to experiment with data.
@Zheka - quad core was the lowest available from OVH. I'll shop around for a server with faster (and possibly fewer) cores.
Given that the data was streaming on time when the recent delay happened, I don't see how the problem can be with IQF. Also, switching to IB data would be more complicated than I realized due to pacing violations. For now, I'll see if my recent tweaks and a hardware change do anything. If not, perhaps that will be a good time to experiment with data.
- TJ
- Posts: 7744
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Delays receiving real-time data
@TJ - dedicated bare metal.
@Zheka - quad core was the lowest available from OVH. I'll shop around for a server with faster (and possibly fewer) cores.
Given that the data was streaming on time when the recent delay happened, I don't see how the problem can be with IQF. Also, switching to IB data would be more complicated than I realized due to pacing violations. For now, I'll see if my recent tweaks and a hardware change do anything. If not, perhaps that will be a good time to experiment with data.
The CPU is plenty fast for what you are doing. The extra CPU won't hurt; they will go into idle when not called.
The delay is somewhere, not the CPU.
It could be mismatched memories, it could be out of sync bus, it could be something other than the computer.
I would keep looking.
Re: Delays receiving real-time data
I've read you have a 4xCPU server. Of course, if it's a single 4-core Xeon E3 - then there is no real need. Still, you will do better with a i7-7700K, which is also a quad-core CPU.@Zheka - quad core was the lowest available from OVH. I'll shop around for a server with faster (and possibly fewer) cores.
As TJ said, there might be a problem with the hardware; so just ask your vendor to hot-swap your hard drive to another machine and see if it helps.
IB's "pacing violations" become a concern with historical data retrieval, in volume; rarely, if ever, in real-time.Given that the data was streaming on time when the recent delay happened, I don't see how the problem can be with IQF. Also, switching to IB data would be more complicated than I realized due to pacing violations..
Run both PT instances in parallel: one fed with IQF, another with IB.
One instance of IBG, serving both.
Costs you nothing and can be tried with several mouse clicks.
At the end, you are after GOOD FILLS.
Re: Delays receiving real-time data
Update: I'm setting up a newly rented dedicated server. This one has an E3 1240v6, is located close to NYC, and is cheaper. It's from a different provider, just to eliminate a variable (though I have zero complaints about OVH).
But before I get to the Moment Of Truth, I'm curious about something. ITT I learned that PT only uses one core in auto-trading. Does that mean it only uses one thread and/or that it would be ideal for me to turn of hyper-threading? (I won't be running optimizations on this server.)
But before I get to the Moment Of Truth, I'm curious about something. ITT I learned that PT only uses one core in auto-trading. Does that mean it only uses one thread and/or that it would be ideal for me to turn of hyper-threading? (I won't be running optimizations on this server.)
-
- Posts: 275
- Joined: 22 Apr 2014
- Has thanked: 70 times
- Been thanked: 72 times
Re: Delays receiving real-time data
I would leave it on. There is a benefit in general. Perhaps not with Multicharts but overall. I am using the same CPU with Portfolio trader with second intervals and about 75 instruments. CPU utilization fluctuates between 6% and 12%. Was your last server also near NYC? Where are you connecting from?
- TJ
- Posts: 7744
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Delays receiving real-time data
Update: I'm setting up a newly rented dedicated server. This one has an E3 1240v6, is located close to NYC, and is cheaper. It's from a different provider, just to eliminate a variable (though I have zero complaints about OVH).
But before I get to the Moment Of Truth, I'm curious about something. ITT I learned that PT only uses one core in auto-trading. Does that mean it only uses one thread and/or that it would be ideal for me to turn of hyper-threading? (I won't be running optimizations on this server.)
See post #7
viewtopic.php?t=47045
Re: Delays receiving real-time data
I've been having almost exactly the same issue with v12r2 with cqg data. I used two different servers to test e5 - 2673v4 and e3-1245v6 both seem to have delays in receiving data. i have absolutely no issues on another e5-2673v4 server running v11r9. I'll try another server with a higher clock speed (probably i7-7700) to see if that makes a difference. It was suggested to me that it could be a network issue of my server provider, but the server running v12 and the one running v11r9 have the same provider, so its perplexing.Update: I'm setting up a newly rented dedicated server. This one has an E3 1240v6, is located close to NYC, and is cheaper. It's from a different provider, just to eliminate a variable (though I have zero complaints about OVH).
But before I get to the Moment Of Truth, I'm curious about something. ITT I learned that PT only uses one core in auto-trading. Does that mean it only uses one thread and/or that it would be ideal for me to turn of hyper-threading? (I won't be running optimizations on this server.)
- TJ
- Posts: 7744
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Delays receiving real-time data
Suggestion to MultiCharts:
Add a PING feature to MC. Where the user can press a button and test the MC-to-Dataprovider latency, right from inside of MC.
Add a PING feature to MC. Where the user can press a button and test the MC-to-Dataprovider latency, right from inside of MC.
Re: Delays receiving real-time data
It is misleading to judge the quality of connection by ping (which uses a different protocol for measurement).
One can observe real-time TCP connection latencies under load just by opening a Resource Monitor in a Task Manager and looking under Network/TCP connections.
One can observe real-time TCP connection latencies under load just by opening a Resource Monitor in a Task Manager and looking under Network/TCP connections.
Re: Delays receiving real-time data
That's worrying. Anyone else observing slower performance with MC12r2?I've been having almost exactly the same issue with v12r2 with cqg data. I used two different servers to test e5 - 2673v4 and e3-1245v6 both seem to have delays in receiving data. i have absolutely no issues on another e5-2673v4 server running v11r9. I'll try another server with a higher clock speed (probably i7-7700) to see if that makes a difference. It was suggested to me that it could be a network issue of my server provider, but the server running v12 and the one running v11r9 have the same provider, so its perplexing.Update: I'm setting up a newly rented dedicated server. This one has an E3 1240v6, is located close to NYC, and is cheaper. It's from a different provider, just to eliminate a variable (though I have zero complaints about OVH).
But before I get to the Moment Of Truth, I'm curious about something. ITT I learned that PT only uses one core in auto-trading. Does that mean it only uses one thread and/or that it would be ideal for me to turn of hyper-threading? (I won't be running optimizations on this server.)
I have relatively few issues with MC11r9. Can't upgrade until MC12 performance issues get fixed.
Multicharts: Is this something you guys are aware of and working on?
- Svetlana MultiCharts
- Posts: 645
- Joined: 19 Oct 2017
- Has thanked: 3 times
- Been thanked: 163 times
Re: Delays receiving real-time data
Hello, pcrespo,
Will it be possible for you to come to our live chat MON-FRI from 6:30 AM to 11:00 AM EST and demonstrate the delay of order generation issue to our engineers via remote connection?
Live Chat is accessible from our web site http://www.multicharts.com/ (at the top of the page). We’ll do our best to help you
Will it be possible for you to come to our live chat MON-FRI from 6:30 AM to 11:00 AM EST and demonstrate the delay of order generation issue to our engineers via remote connection?
Live Chat is accessible from our web site http://www.multicharts.com/ (at the top of the page). We’ll do our best to help you
Re: Delays receiving real-time data
Who marked this as Solved? It's not
@Svetlana - I'll enter the livechat Monday or as soon as I can.
I have CloseBarTimeout set to 3. I'm syncing the machine's clock with us.pool.ntp.org every 16 seconds.
So far, my log shows a few several-second delays, for instance there was a 9-second delay despite there being ticks each consecutive second. (All the bars were then calculated at the same PC second, ie the signal "caught up".)
Here is the code for my delay logging signal:It uses a function timeSdiff whose code is below:
TL;DR - the function calculates the #seconds difference between the two input times. The signal uses that function to print out the difference upon each 1-second bar calculation. If the difference is greater than "normalDelay", it prints that there was a delay. If the same PC time has a calculation for two or more bar times, a "catchup" is logged. Catchups are detected by checking whether there were other bar times between the PC time and delayed bar. I think catchups are less likely to be false-positives than a single delayed bar.
@hughesfleming, my previous server was in Montreal and I'm connecting from near Philadelphia.
@zheka - I'll check the latencies in Resource Monitor next week.
@Svetlana - I'll enter the livechat Monday or as soon as I can.
I have CloseBarTimeout set to 3. I'm syncing the machine's clock with us.pool.ntp.org every 16 seconds.
So far, my log shows a few several-second delays, for instance there was a 9-second delay despite there being ticks each consecutive second. (All the bars were then calculated at the same PC second, ie the signal "caught up".)
Here is the code for my delay logging signal:
Code: Select all
Inputs: normalDelay(5), catchup(4);
vars: intrabarpersist doneBT(false), intrabarpersist check(false), intrabarpersist prevTime(0), intrabarpersist prevPCtime(-1), intrabarpersist diff(0), path("");
once path = "C:\Users\...\Desktop\delays_"+symbol+"_filter_"+NumToStr(normalDelay,0)+"_"+NumToStr(catchup,0)+".txt";
doneBT = doneBT or LastBarOnChart_s;
if doneBT then begin
if check and time_s<prevPCtime then print(file(path),"^^^ Catchup: ",diff:1:0," ^^^");
diff = timeSdiff(time_s, currenttime_s);
print(file(path),date:7:0," ",time_s:6:0," ",currenttime_s:6:0," ",diff:1:0);
if diff>normalDelay then print(file(path),"^^^ Delay: ",diff:1:0," ^^^");
if diff>catchup then begin
check = true;
prevTime = time_s;
prevPCtime = currenttime_s;
end else check=false;
end;
Code: Select all
inputs: secTime1(NumericSimple), secTime2(NumericSimple);
vars: mintime1(0), mintime2(0), hours(0), minutes(0), seconds(0), curMinute(0), prevMinute(0), curSec(0), prevSec(0);
mintime1 = time_s2time(sectime1);
mintime2 = time_s2time(sectime2);
hours = IntPortion(mintime2/100) - IntPortion(mintime1/100);
if hours<0 then hours=hours+24;
curMinute = mintime2-(100*IntPortion(mintime2/100));
prevMinute = mintime1-(100*IntPortion(mintime1/100));
minutes = curMinute-prevMinute;
if minutes<0 then begin
minutes = minutes+60;
hours = hours-1;
end;
curSec = sectime2-(100*IntPortion(sectime2/100));
prevSec = secTime1-(100*IntPortion(sectime1/100));
seconds = curSec-prevSec;
if seconds<0 then begin
seconds = seconds+60;
minutes = minutes-1;
if minutes<0 then begin
minutes = minutes+60;
hours = hours-1;
end;
end;
timeSdiff = hours*10000 + minutes*100 + seconds;
@hughesfleming, my previous server was in Montreal and I'm connecting from near Philadelphia.
@zheka - I'll check the latencies in Resource Monitor next week.
- Smoky
- Posts: 519
- Joined: 03 Dec 2010
- Location: Thailand
- Has thanked: 99 times
- Been thanked: 123 times
Re: Delays receiving real-time data
@ sptrader
I'd also make sure that MC's "Bugslayer" is disabled in the registry, it can slow things down too.
good to know , do you have other registry setup to boost MC ?
-
- Posts: 275
- Joined: 22 Apr 2014
- Has thanked: 70 times
- Been thanked: 72 times
Re: Delays receiving real-time data
@pcrespo,
Thanks for posting your code. I will run it as well and let you know if there are any issues. My server is in Frankfurt so about 100ms worse in terms of latency than yours. Have a good weekend.
Alex
Thanks for posting your code. I will run it as well and let you know if there are any issues. My server is in Frankfurt so about 100ms worse in terms of latency than yours. Have a good weekend.
Alex
Re: Delays receiving real-time data
Are any users trading with MC12 r2 and not experiencing delays/lags?
This all sounds like an ongoing MC12 bug, no?
Multicharts; would you consider putting MC12 back into Beta and supporting MC11 again?
This all sounds like an ongoing MC12 bug, no?
Multicharts; would you consider putting MC12 back into Beta and supporting MC11 again?
Re: Delays receiving real-time data
@pcrespo,
I feel you are overdoing it and "observation" influences the results.
I/O operations (like printing to a file) are relatively slow, and you are printing at each tick and bar close!
My advice:
- sync clock once in 15 minutes, not seconds
- use 15-60sec bars
- find time differences via datetime and related functions, rather than your custom function
- log only delays>x, not all cases
You might also find this post useful.
viewtopic.php?t=47499
I feel you are overdoing it and "observation" influences the results.
I/O operations (like printing to a file) are relatively slow, and you are printing at each tick and bar close!
My advice:
- sync clock once in 15 minutes, not seconds
- use 15-60sec bars
- find time differences via datetime and related functions, rather than your custom function
- log only delays>x, not all cases
You might also find this post useful.
viewtopic.php?t=47499
Re: Delays receiving real-time data
@wilkinsw - my delays have been happening since the beginning of the year or maybe late last year, so for me it has nothing to do with version 12 R2.
@Zheka - you make good points. I should only log delays (defined by my inputs), rather than log the delay at every bar close. Syncing every 16s is probably extreme so I'll try 64. The new machine's clock is apparently awful out of the box, because at first, by accident, I wasn't syncing at all and almost every bar was logged as a delay.
I don't know about increasing the bar interval. I wouldn't get the same info. I'd miss many of those moments where the signal has to catch up by calculating several bars at once, which are the least likely to be false-positives and the most likely to cause orders to be delayed. Also, my trading signals use 1-second bars and I'd like the logger signal's conditions to be the same.
That thread you linked looks good but I haven't read it yet and I need to catch some sleep now.
@Zheka - you make good points. I should only log delays (defined by my inputs), rather than log the delay at every bar close. Syncing every 16s is probably extreme so I'll try 64. The new machine's clock is apparently awful out of the box, because at first, by accident, I wasn't syncing at all and almost every bar was logged as a delay.
I don't know about increasing the bar interval. I wouldn't get the same info. I'd miss many of those moments where the signal has to catch up by calculating several bars at once, which are the least likely to be false-positives and the most likely to cause orders to be delayed. Also, my trading signals use 1-second bars and I'd like the logger signal's conditions to be the same.
That thread you linked looks good but I haven't read it yet and I need to catch some sleep now.
- TJ
- Posts: 7744
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Delays receiving real-time data
Sync'ing the clock more than once every day is insane.
Probably it contributed more to your delay than anything.
It takes computer cycles to check the internet,
and it takes computer cycles for the server to reply to you,
and it takes computer cycles to update your computer.
Everything must come to a STOP when you update your computer time.
Unless your computer is unstable, the clock should not drift.
Sync'ing your clock will not improve data streaming.
ps. the Windows default is auto-sync once per week.
Probably it contributed more to your delay than anything.
It takes computer cycles to check the internet,
and it takes computer cycles for the server to reply to you,
and it takes computer cycles to update your computer.
Everything must come to a STOP when you update your computer time.
Unless your computer is unstable, the clock should not drift.
Sync'ing your clock will not improve data streaming.
ps. the Windows default is auto-sync once per week.
Re: Delays receiving real-time data
Ok I've tweaked my code, and in Registry I increased the sync interval to 1024 seconds (I can't get SpecialPollInterval to work despite using the 0x1 flag, so I have to use powers of 2 via the min/max poll interval).
@Zheka - the thread you linked to indirectly provided a big clue. I hadn't thought to check my QM log. It turns out, my 9-second delay (and catchup) from two days ago in the 3am hour was caused by a disconnection.
So now I have to see if all the delays are caused by that, and if so, figure out why IQF disconnects.
(Although, when I had that 50+ second delay mentioned earlier in this thread, my colleague was seeing new data come in on the chart, which would suggest IQF wasn't disconnected.)
@Zheka - the thread you linked to indirectly provided a big clue. I hadn't thought to check my QM log. It turns out, my 9-second delay (and catchup) from two days ago in the 3am hour was caused by a disconnection.
Then, one second before the calculation finally happened, the QM log says "Connection is OK" and "Connecting to real-time data for..."IQFeed Message: S,SERVER DISCONNECTED
So now I have to see if all the delays are caused by that, and if so, figure out why IQF disconnects.
(Although, when I had that 50+ second delay mentioned earlier in this thread, my colleague was seeing new data come in on the chart, which would suggest IQF wasn't disconnected.)
- Mark Brown
- Posts: 184
- Joined: 29 Nov 2016
- Has thanked: 125 times
- Been thanked: 18 times
Re: Delays receiving real-time data
i now think this delay is what caused 100 contract auto order to add 120+ new contracts erroneously to a position i had. i looked up and i had 220+ emini sp's on and a resting order stop for that 100. it got so screwed up i just unwound the whole position and have not tried auto trading again. i have no idea where that order come from except i did notice some huge lag in some orders being processed. i am using TS data mapped to ib symbols for the dom and auto trading.
i have made the registry changes in that ot5her tread and also disabled the logging cause that has to be a drain and won't / hasn't helped anything get fixed to date. my whole program just keeps hanging since build 12, it's slow to optimize, slow for everything.
i am constantly waiting for data that i expect should be in storage already on my drive? i don't think mc is storing data actually as it thinks it is. these constant warnings about objects is something new blowing my mind and the placement of them is bad. oh well maybe they will roll back to version 9 or something. makes me wonder about the future of mc and the umbilical cord it requires to run.
i have made the registry changes in that ot5her tread and also disabled the logging cause that has to be a drain and won't / hasn't helped anything get fixed to date. my whole program just keeps hanging since build 12, it's slow to optimize, slow for everything.
i am constantly waiting for data that i expect should be in storage already on my drive? i don't think mc is storing data actually as it thinks it is. these constant warnings about objects is something new blowing my mind and the placement of them is bad. oh well maybe they will roll back to version 9 or something. makes me wonder about the future of mc and the umbilical cord it requires to run.
Re: Delays receiving real-time data
Sorry for the mess of a code. This new code ought to achieve the same thing (good suggestion by Zheka):@pcrespo,
Thanks for posting your code. I will run it as well and let you know if there are any issues. My server is in Frankfurt so about 100ms worse in terms of latency than yours. Have a good weekend.
Alex
Code: Select all
Inputs: normDel(6), catchup(3);
vars: intrabarpersist doneBT(false), intrabarpersist check(false), intrabarpersist prevTime(0), intrabarpersist prevPCtime(-1), intrabarpersist diff(0), path("");
once path = "C:\Users\...\Desktop\delays_"+symbol+"_filter_"+NumToStr(normDel,0)+"_"+NumToStr(catchup,0)+".txt";
doneBT = doneBT or LastBarOnChart_s;
if doneBT then begin
if check and time_s<prevPCtime then print(file(path),date:7:0," ",time_s:6:0," ",currenttime_s:6:0," There was volume during the delay.");
diff = datetime2eltime_s(computerdatetime-datetime);
if diff>normDel then print(file(path),date:7:0," ",time_s:6:0," ",currenttime_s:6:0," Delay: ",diff:1:0);
if diff>catchup then begin
check = true;
prevPCtime = currenttime_s;
end else check=false;
end;
-
- Posts: 275
- Joined: 22 Apr 2014
- Has thanked: 70 times
- Been thanked: 72 times
Re: Delays receiving real-time data
Hi Mark,
I wouldn't switch off logging until you know for sure what caused your double sized mini order. Unless you are seeing repeated queue delays (the bar at the bottom right that shows up) when you are trying to get orders filled then messing with the registry is probably not necessary. On the rare occasion that something slows down, it is because of a random disconnect to IQFeed.
I wouldn't switch off logging until you know for sure what caused your double sized mini order. Unless you are seeing repeated queue delays (the bar at the bottom right that shows up) when you are trying to get orders filled then messing with the registry is probably not necessary. On the rare occasion that something slows down, it is because of a random disconnect to IQFeed.
Re: Delays receiving real-time data
Sorry I missed this question before. Currently I have two signals trading. One of them trades 4 instruments using 1-hour Data2 bars while the other trades one instrument using 1-min Data2. For both I use 1-sec Data1 bars, which isn't important to their success, so should I not? Typical trade length is 4-5 hours in the 1min signal and several hours to a few days in the 1-hour signal.@pcrespo,
What's the bar interval for your primary trading logic and what would be your typical trade length?
Re: Delays receiving real-time data
The question should be just the opposite: do you really, REALLY need it?For both I use 1-sec Data1 bars, which isn't important to their success, so should I not
You whole production set-up should be scrutinized from this perspective, and optimized for robustness.
Re: Delays receiving real-time data
Earlier today, there was a delay that did not correspond to an IQF disconnection, so unfortunately there remains a mystery cause.
@Zheka - I didn't realize it was hard for the system, since MC is allegedly capable of trading on tick charts. If it can handle ticks then 1-second should be a piece of cake (was my impression). But no big deal, I'll increase the interval. Maybe 15 seconds.
@Zheka - I didn't realize it was hard for the system, since MC is allegedly capable of trading on tick charts. If it can handle ticks then 1-second should be a piece of cake (was my impression). But no big deal, I'll increase the interval. Maybe 15 seconds.
Re: Delays receiving real-time data
Heh I mean, I'm making the same number of trades (paying the same commission & slippage) regardless of my Data1 interval. It's just, when my signal says it's time to enter/exit, I'd rather not wait. Are you saying the Data1 interval should be at least a minute? Either way, if the delays persist, something is wrong. I find it highly doubtful that a 15s interval should cause delays.
Re: Delays receiving real-time data
You don't need a more granular Data1 at all - unless you are really micro-managing entries/exits with an "execution algo".
If you need to trade intrabar, then a stop/limit order submitted at bar end will work just fine.
Getting to a more granular time-frame will not solve the problem of "delays" - it will most probably make it worse/ more frequent.
And - by the way - delays are not "bad" by default. Depending on your system, 40-50% of the time you get a better price...
If you need to trade intrabar, then a stop/limit order submitted at bar end will work just fine.
Getting to a more granular time-frame will not solve the problem of "delays" - it will most probably make it worse/ more frequent.
And - by the way - delays are not "bad" by default. Depending on your system, 40-50% of the time you get a better price...
Re: Delays receiving real-time data
I made it less granular, not more. It was 1s and then I made it 15s after your earlier post.Getting to a more granular time-frame will not solve the problem of "delays" - it will most probably make it worse/ more frequent.
My small intervals are admittedly OCD, but when it comes to delays, it's not the price that concerns me. It's that I don't know their cause, have no control of them, can't predict when they'll happen or how long they'll last. I don't want to have to tell clients, "Sorry, this trade lost X additional money because we had no ability to exit the position when we wanted to. We don't know why and it could happen again at any time." (Of course, that can't happen if someone is at the screen to manually exit, but human intervention should be the last line of defense, not something you expect to rely on.)And - by the way - delays are not "bad" by default. Depending on your system, 40-50% of the time you get a better price...
In other words, set Data1 interval to what Data2 is and then delete Data2? With our 1-minute strategy, fine, but with our 1-hour strategy this would be a drastic change that we'd need to backtest. (Edit - actually, I hadn't considered stop-limit orders. With them, I suppose the change would be negligible, in fact it would be an improvement.)You don't need a more granular Data1 at all
Re: Delays receiving real-time data
You made it more granular compared to a "decision" time frame.I made it less granular, not more.
If your strategy logic relies on a smaller timeframe - then it means it is not really a "1-hour strategy", or did you introduce it to solve a problem with "delays"?With our 1-minute strategy, fine, but with our 1-hour strategy this would be a drastic change that we'd need to backtest.
How are you using a smaller timeframe?
Re: Delays receiving real-time data
I suppose it's two strategies: a one-hour entry strategy and a smaller-timeframe exit strategy. We could achieve basically the same thing with stop-limit orders and "SetTrailingStop_pt", but I just remembered we have a good reason not to do that. I'd share the reason, but I'm not sure if my colleague considers it a trade secret lol.
Re: Delays receiving real-time data
OK, then it makes sense.
15-60sec bars would be a more robust solution then IOG.
All logic besides exit split behind a barstatus(2)=1 check.
Also, if you do not employ a portfolio-level MM logic in real-time, you might be better off trading from MC rather than PT - since MC uses 1 core per chart and PT uses 1 core for everything.
15-60sec bars would be a more robust solution then IOG.
All logic besides exit split behind a barstatus(2)=1 check.
Also, if you do not employ a portfolio-level MM logic in real-time, you might be better off trading from MC rather than PT - since MC uses 1 core per chart and PT uses 1 core for everything.