Delays receiving real-time data

Questions about MultiCharts and user contributed studies.
User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Delays receiving real-time data

Postby pcrespo » 15 Aug 2018

I'm running MC 12 with IQF data and IB API, trading in PT. The changelog for MC12 indicated that this issue might have been fixed, but for me it persists.

I've been logging this issue as follows. I wrote a roughly 10-line signal that, every Data1 bar (which is 1 second), prints the bar time (time_s) and computertime_s. I've been running this signal on 4 instruments: ES, US(ZB), NG, and Soybeans. The behavior is the same in all instruments. Sometimes the two times are equal; usually they differ by 1-2 seconds which is acceptable. Sometimes the difference is 5 seconds. Occasionally it's 10-20 seconds.

As a side effect, when there's a delay of several seconds, sometimes multiple bars are received at the same computer time, and so PT catches up by calculating all of those bars at the same computer time. For instance, maybe there was volume at 10:00:01, then again at 10:00:05, then again at 10:00:07, but no calculation occurs until computer time 10:00:12 (at which point a triple-calculation occurs).

When a several-second delay coincides with an entry (in my trading signal), my entry order gets delayed, which is frustrating. It has not yet coincided with an exit order, but that's probably inevitable.

The IB API log provides no clues. During the delays, the log has the same messages it has during non-delayed times.

Whenever a delay happens in my test signal, it also happens in my trading signal in the same instrument.

When a delay happens in one instrument, I think it happens in all instruments. I'm not yet sure, because usually when I look for a matching delay, the instruments didn't have volume at the same seconds. But from the evidence I've gathered so far, when they do have bars at the same time and a delay occurs in one instrument, it occurs in both. This happened recently, with the same delay happening in ES and US.

I'm trading from a dedicated OVH server with fast internet, and a data bar is a very small download, so I'm not inclined to blame the internet connection, but I also can't rule it out.

There is a way to generate a delay on purpose: if I export one of the symbols in QM, that symbol experiences a delay every time. This made me wonder if running QM or MC while trading in PT caused occasional delays, so I closed everything but PT. However, the delays continue to occur.

Does anyone else experience these delays? Is there anything I can do about it?

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Delays receiving real-time data

Postby hughesfleming » 15 Aug 2018

Hi Pcrespo,

What operating system are you using? Do you have IQFeed and Multicharts excluded from the antimalware service? What is the first contract in your instrument list in Portfolio trader? It should the the one with the highest tick rate. Thanks to ABC Trading for that invaluable tip.

I have many persistent problems with Portfolio Trader and IB. It gets the job done in the end but not without headaches along the way.

Alex

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 15 Aug 2018

Do you have IQFeed and Multicharts excluded from the antimalware service?
Windows Server 2012; we have no anti-malware installed (Defender is not included by default).
What is the first contract in your instrument list in Portfolio trader? It should the the one with the highest tick rate.
In that case, we had the wrong order of instruments. I've just changed it and will see if the problem goes away. Thanks!

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Delays receiving real-time data

Postby hughesfleming » 15 Aug 2018

I am using a Supermicro dedicated server for Portfolio Trader and I have experienced clock drift where my system clock time differs from the actual time. You may want to use a third party tool to contact time servers to sync your time. For what ever reason, Windows 2012r2 was not syncing properly.

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 23 Aug 2018

Changing the order of PT instruments didn't get rid of the delays. I like the time sync suggestion. For now, I've changed the NTP servers to the appropriate pool.ntp.org ones and changed the polling frequency to 1min instead of 1hour. If that doesn't work, I'll investigate whether the server time is inaccurate; if it is, I'll look into 3rd-party ntp/nts clients like you mentioned.

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Delays receiving real-time data

Postby hughesfleming » 24 Aug 2018

I am using something called Dimension 4 set to sync every five minutes. I also have delays with Portfolio Trader but I have just accepted them. For example yesterday it took 22 seconds to fill five market orders. It is the delay in order generation that is troubling.

For example order four was filled at 9:33.01. Order 5 was generated at 9:33:122 and filled at 9:33:126. The fill was fast but the generation was slow. I am reading orders from a text file and sending all of them at the same time at so I don't know why it took 12 seconds to generate the last order. There is no signal.

The good news is that most of the time I can fill everything in a few tenths of a second but once in a while there is this delay. I am sure there is a valid explanation but it I have not been able to track it down. My strategy is not very time dependent. A delay here or there does not make a difference to the end result.....fortunately.
Last edited by hughesfleming on 24 Aug 2018, edited 1 time in total.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Delays receiving real-time data

Postby TJ » 24 Aug 2018

Have you done a PING test to ibkr?

The instruction is on their website.

The latency could be in your ISP.

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 28 Aug 2018

Changing the time sync settings hasn't gotten rid of the delays. I think the delays are real, not just a matter of the PC time being inaccurate.

My ping to the IB server is 15ms, but if anything, the delay is with IQF, not IB. The signal's calculation is delayed, suggesting that I'm either receiving the data bar late, or PT's calculation is lagging behind the data stream. When my orders are generated late, it is always because the calculation was late. To be clear, I'm only talking about when volume was present between calculations (since a "delay" without volume is proper behavior).

I grabbed the IQF ip from the connection manager and ran a continuous, timestamped ping to a text file. My connection to IQF doesn't seem to be the issue. My pings range from 40-125 ms. During the most recent delays in my signal, the ping was not higher than normal. For instance, I had a 7-second delay in my signal, but the ping was only 90ms. No seconds-long pings have been recorded so far.

Does this mean the issue is with PT/MC?

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Delays receiving real-time data

Postby hughesfleming » 29 Aug 2018

One last thing to try that seems to have helped me is to turn on fast tracing in the registry. Take a look at this thread. At times it would seems that orders get stuck in the queue. It has to do with empty batches of orders. The thread has all the details.

I have seen the delay you are talking about. You might also want to check your logs to make sure that you are not seeing a disconnect from IQFeed. This was causing me lots of headache with a different data provider but that doesn't mean that it can't happen.

viewtopic.php?f=1&t=47587&p=129250&hili ... ce#p129250

Maybe someone at Multicharts can explain why this is not a default setting.

regards,

Alex

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Delays receiving real-time data

Postby Zheka » 29 Aug 2018

Several things came to mind:
- "Pings" are not at all representative of real latency under load.
- your network card/TCP stack settings and CPU processing power will somewhat influence your overall "trading system" latency, especially in times of heavy trading/tick count across several instruments

- bear in mind that IQF servers are in Dallas...which means data have to travel:
from Chicago or NY to Dallas
then from Dallas to you
then get processed by MC
then from MC to IB

- so, try trading straight with IB data instead of IQF (to cut 2 legs) and see if you experience the same kind of delays AND EXECUTION.
Since you most probably are not trading tick bars, getting fewer ticks from IB (every 200-250ms) might actually help across the board.

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 31 Aug 2018

@Alex - I've turned FastTracing on for good measure, but the problem must be unrelated because the calculation delays happen under all conditions, not just during trades and not just during high volume. They happen when trading just one instrument - any instrument during any hours. Sometimes a delay coincides with when I'm about to make a trade, but I'm pretty sure that when it does, the delay would have happened regardless.

@Zheka - yeah, switching to IB data was going to be my last resort, which seems to be where I'm headed. I just ran a speed test to a server in Dallas. I don't know how indicative it is, but I got 116 MB/s (927 Mb/s) Down and 26 MB/s Up. As for cpu, PT/MC is only using 8-20% cpu and the other processes combined are using <1%. Does every IQF user receive occasional 15-second delays and simply accept them, or does everyone have a faster system than this? I feel like there's something else going on.

sptrader
Posts: 742
Joined: 09 Apr 2010
Location: Texas
Has thanked: 483 times
Been thanked: 274 times
Contact:

Re: Delays receiving real-time data

Postby sptrader » 31 Aug 2018

@Alex - I've turned FastTracing on for good measure, but the problem must be unrelated because the calculation delays happen under all conditions, not just during trades and not just during high volume. They happen when trading just one instrument - any instrument during any hours. Sometimes a delay coincides with when I'm about to make a trade, but I'm pretty sure that when it does, the delay would have happened regardless.

@Zheka - yeah, switching to IB data was going to be my last resort, which seems to be where I'm headed. I just ran a speed test to a server in Dallas. I don't know how indicative it is, but I got 116 MB/s (927 Mb/s) Down and 26 MB/s Up. As for cpu, PT/MC is only using 8-20% cpu and the other processes combined are using <1%. Does every IQF user receive occasional 15-second delays and simply accept them, or does everyone have a faster system than this? I feel like there's something else going on.
It might be interesting to try the Windows command "tracert" it will trace every step from your pc to the destination and show where the biggest delays are.
Example: in the command prompt type: tracert interactivebrokers.com and hit "enter". It takes a minute to run so be patient.

I'd also make sure that MC's "Bugslayer" is disabled in the registry, it can slow things down too.

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Delays receiving real-time data

Postby hughesfleming » 31 Aug 2018

Hi pcrespo,

It is very hard to solve these problems sometimes. I had many in the beginning. You might want to send your logs to Multicharts. It does sound like you have another problem. I am using IQFeed for US Equities but my server is in Frankfurt so orders and data still have to cross the Atlantic. I do get a delay once in a while but things have been working well recently. A typical daily re-balance will take 3-5 seconds on average for eight positions. Sometimes it is almost instantaneous. Orders are sent to IB in Zug. If things are working normally, your fills should be similar. My setup is less than ideal as far as latency is concerned so you shouldn't do much worse than that. If you are getting these 15 second delays all the time, then that is not normal.

regards,

Alex

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Delays receiving real-time data

Postby Zheka » 31 Aug 2018

@Zheka - yeah, switching to IB data was going to be my last resort, which seems to be where I'm headed. I just ran a speed test to a server in Dallas. I don't know how indicative it is, but I got 116 MB/s (927 Mb/s) Down and 26 MB/s Up.
Hmm:-) I was talking about latency (and tuning NIC/TCP stack is a big area on its own). And since IQF uses TCP (for completeness and reliability), any "congestion" on the way to/from or on your server might lead to delays.
As for cpu, PT/MC is only using 8-20% cpu and the other processes combined are using <1%.
PT uses 1 core to process the data (and multithreading it has been requested long ago!). And you cannot really judge by looking in the task mngr (even if looking by core), if CPU is a bottleneck or not.
And if CPU does not occasionally manage, then this would exacerbate "network congestion".
So, a CPU with higher frequency/ "turbo" mode is much preferable to a Xeon with many but slow cores. What's your hardware?
Also, change Power options to Performance.
Does every IQF user receive occasional 15-second delays and simply accept them, or does everyone have a faster system than this? I feel like there's something else going on.
All the above points will hardly cause 15-sec delays.
But try changing things as people recommend and see if things change.

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 31 Aug 2018

So today, a long delay happened when it was time to exit a position. Trading ES on a 1-second Data1 timeframe (sync mode, IOG disabled), we were supposed to exit at 16:00:01 but instead exited at 16:00:54 even though there was plenty of volume (and thus, opportunities for the signal to calculate) in between.

This time, my colleague happened to be watching the screen as the delay unfolded. He says data was coming in each second on the chart, so apparently the lag has to do with PT/MC, not data.

The signal's log shows that the 25 calculations from 16:00:00 to 16:00:24 all took place at 16:00:54, meaning PT scrambled to play catch-up after the delay.

@Alex - Just now, per Anna's suggestion in the "Execution Delay" thread, I've completely disabled "Trace" logs in the registry. However, the "BugSlayer" directory has several other settings one can disable. Are you saying I should disable all of them?

@Zheka - Based on the new development, it seems CPU is the bottleneck. The server isn't especially fast because we didn't think it needed to be. It has 4 Xeon E3-1231 v3's at 3.4 ghz. If our delays continue, do you think we should rent a different server? One with a higher clock frequency per core, but not necessarily more cores?

Power settings were already set to Performance, but I noticed that the processor allocation was adjusted for background processes rather than programs, so I changed that.

Regarding the TCP/IP stack, do TCP Optimizer's "optimal" settings cover it or do I have to tune it manually?

Thanks guys, I feel like I'm gradually getting to the bottom of this with your help.

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Delays receiving real-time data

Postby hughesfleming » 01 Sep 2018

Hi pcrespo, I just have Fast tracing enabled. I have not switched off logging completely. I think it is quite extreme to consider doing that. You should set the OS to prioritize foreground processes. You have enough clock speed. The difference between a v3 and v5 xeon is not large. You would only see a very small difference running optimizations for example. About six months ago I had issues running a new server because of meltdown/spectre bios updates so make sure you are using a current bios. E-mail your service provider and ask them. They would know as they would have had to deal with this.

If I were you, I would rent another server from another provider and see if you have the same problem. That is the only way to eliminate the hardware and network from the problem. Once you can duplicate the issue, look though your code to make sure that everything is in order there. You mentioned that sometimes you are hitting 20% cpu utilization which seems high if your are trading just a few instruments.

The bad news is that I get these delays sometimes as well and I have never been able to track down why. Everything ran this smoothly this week 24/5 without any issues so it must be a random network delay receiving data from IQFeed that can happen. I did have trouble with MC12 and rolled back and did a fresh installation of MC11 including IB Gateway. Things have never worked better!

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Delays receiving real-time data

Postby Zheka » 02 Sep 2018

Yes, CPU seems to be "good enough", though you would be better off with 1x i7-7700K (or 6700K).. Why do you need a 4-CPU set-up on a production machine?
TCP Optimizer will mostly cover it, just read the docs/search the web to understand what specific settings mean and how "networking" works.
You can also download your network adapter utility driver and tune it similarly.

BUT, again, experiment with an IB-only setup.
Launch 2 PT instances and feed one with IQF and another with IB data. Judge by absence of delays AND FILLS.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Delays receiving real-time data

Postby TJ » 02 Sep 2018

Is this a dedicated server?

or a shared rental?

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 03 Sep 2018

@TJ - dedicated bare metal.

@Zheka - quad core was the lowest available from OVH. I'll shop around for a server with faster (and possibly fewer) cores.

Given that the data was streaming on time when the recent delay happened, I don't see how the problem can be with IQF. Also, switching to IB data would be more complicated than I realized due to pacing violations. For now, I'll see if my recent tweaks and a hardware change do anything. If not, perhaps that will be a good time to experiment with data.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Delays receiving real-time data

Postby TJ » 03 Sep 2018

@TJ - dedicated bare metal.

@Zheka - quad core was the lowest available from OVH. I'll shop around for a server with faster (and possibly fewer) cores.

Given that the data was streaming on time when the recent delay happened, I don't see how the problem can be with IQF. Also, switching to IB data would be more complicated than I realized due to pacing violations. For now, I'll see if my recent tweaks and a hardware change do anything. If not, perhaps that will be a good time to experiment with data.

The CPU is plenty fast for what you are doing. The extra CPU won't hurt; they will go into idle when not called.

The delay is somewhere, not the CPU.
It could be mismatched memories, it could be out of sync bus, it could be something other than the computer.
I would keep looking.

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Delays receiving real-time data

Postby Zheka » 03 Sep 2018

@Zheka - quad core was the lowest available from OVH. I'll shop around for a server with faster (and possibly fewer) cores.
I've read you have a 4xCPU server. Of course, if it's a single 4-core Xeon E3 - then there is no real need. Still, you will do better with a i7-7700K, which is also a quad-core CPU.
As TJ said, there might be a problem with the hardware; so just ask your vendor to hot-swap your hard drive to another machine and see if it helps.
Given that the data was streaming on time when the recent delay happened, I don't see how the problem can be with IQF. Also, switching to IB data would be more complicated than I realized due to pacing violations..
IB's "pacing violations" become a concern with historical data retrieval, in volume; rarely, if ever, in real-time.
Run both PT instances in parallel: one fed with IQF, another with IB.
One instance of IBG, serving both.
Costs you nothing and can be tried with several mouse clicks.
At the end, you are after GOOD FILLS.

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 19 Sep 2018

Update: I'm setting up a newly rented dedicated server. This one has an E3 1240v6, is located close to NYC, and is cheaper. It's from a different provider, just to eliminate a variable (though I have zero complaints about OVH).

But before I get to the Moment Of Truth, I'm curious about something. ITT I learned that PT only uses one core in auto-trading. Does that mean it only uses one thread and/or that it would be ideal for me to turn of hyper-threading? (I won't be running optimizations on this server.)

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Delays receiving real-time data

Postby hughesfleming » 19 Sep 2018

I would leave it on. There is a benefit in general. Perhaps not with Multicharts but overall. I am using the same CPU with Portfolio trader with second intervals and about 75 instruments. CPU utilization fluctuates between 6% and 12%. Was your last server also near NYC? Where are you connecting from?

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Delays receiving real-time data

Postby TJ » 19 Sep 2018

Update: I'm setting up a newly rented dedicated server. This one has an E3 1240v6, is located close to NYC, and is cheaper. It's from a different provider, just to eliminate a variable (though I have zero complaints about OVH).

But before I get to the Moment Of Truth, I'm curious about something. ITT I learned that PT only uses one core in auto-trading. Does that mean it only uses one thread and/or that it would be ideal for me to turn of hyper-threading? (I won't be running optimizations on this server.)

See post #7
viewtopic.php?t=47045

kagein
Posts: 55
Joined: 12 Oct 2017
Has thanked: 16 times
Been thanked: 10 times

Re: Delays receiving real-time data

Postby kagein » 19 Sep 2018

Update: I'm setting up a newly rented dedicated server. This one has an E3 1240v6, is located close to NYC, and is cheaper. It's from a different provider, just to eliminate a variable (though I have zero complaints about OVH).

But before I get to the Moment Of Truth, I'm curious about something. ITT I learned that PT only uses one core in auto-trading. Does that mean it only uses one thread and/or that it would be ideal for me to turn of hyper-threading? (I won't be running optimizations on this server.)
I've been having almost exactly the same issue with v12r2 with cqg data. I used two different servers to test e5 - 2673v4 and e3-1245v6 both seem to have delays in receiving data. i have absolutely no issues on another e5-2673v4 server running v11r9. I'll try another server with a higher clock speed (probably i7-7700) to see if that makes a difference. It was suggested to me that it could be a network issue of my server provider, but the server running v12 and the one running v11r9 have the same provider, so its perplexing.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Delays receiving real-time data

Postby TJ » 19 Sep 2018

Suggestion to MultiCharts:

Add a PING feature to MC. Where the user can press a button and test the MC-to-Dataprovider latency, right from inside of MC.

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Delays receiving real-time data

Postby Zheka » 20 Sep 2018

It is misleading to judge the quality of connection by ping (which uses a different protocol for measurement).

One can observe real-time TCP connection latencies under load just by opening a Resource Monitor in a Task Manager and looking under Network/TCP connections.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Delays receiving real-time data

Postby wilkinsw » 20 Sep 2018

Update: I'm setting up a newly rented dedicated server. This one has an E3 1240v6, is located close to NYC, and is cheaper. It's from a different provider, just to eliminate a variable (though I have zero complaints about OVH).

But before I get to the Moment Of Truth, I'm curious about something. ITT I learned that PT only uses one core in auto-trading. Does that mean it only uses one thread and/or that it would be ideal for me to turn of hyper-threading? (I won't be running optimizations on this server.)
I've been having almost exactly the same issue with v12r2 with cqg data. I used two different servers to test e5 - 2673v4 and e3-1245v6 both seem to have delays in receiving data. i have absolutely no issues on another e5-2673v4 server running v11r9. I'll try another server with a higher clock speed (probably i7-7700) to see if that makes a difference. It was suggested to me that it could be a network issue of my server provider, but the server running v12 and the one running v11r9 have the same provider, so its perplexing.
That's worrying. Anyone else observing slower performance with MC12r2?

I have relatively few issues with MC11r9. Can't upgrade until MC12 performance issues get fixed.

Multicharts: Is this something you guys are aware of and working on?

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Delays receiving real-time data

Postby Svetlana MultiCharts » 21 Sep 2018

Hello, pcrespo,

Will it be possible for you to come to our live chat MON-FRI from 6:30 AM to 11:00 AM EST and demonstrate the delay of order generation issue to our engineers via remote connection?
Live Chat is accessible from our web site http://www.multicharts.com/ (at the top of the page). We’ll do our best to help you

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 28 Sep 2018

Who marked this as Solved? It's not :(

@Svetlana - I'll enter the livechat Monday or as soon as I can.

I have CloseBarTimeout set to 3. I'm syncing the machine's clock with us.pool.ntp.org every 16 seconds.

So far, my log shows a few several-second delays, for instance there was a 9-second delay despite there being ticks each consecutive second. (All the bars were then calculated at the same PC second, ie the signal "caught up".)

Here is the code for my delay logging signal:

Code: Select all

Inputs: normalDelay(5), catchup(4);
vars: intrabarpersist doneBT(false), intrabarpersist check(false), intrabarpersist prevTime(0), intrabarpersist prevPCtime(-1), intrabarpersist diff(0), path("");

once path = "C:\Users\...\Desktop\delays_"+symbol+"_filter_"+NumToStr(normalDelay,0)+"_"+NumToStr(catchup,0)+".txt";

doneBT = doneBT or LastBarOnChart_s;
if doneBT then begin
if check and time_s<prevPCtime then print(file(path),"^^^ Catchup: ",diff:1:0," ^^^");
diff = timeSdiff(time_s, currenttime_s);
print(file(path),date:7:0," ",time_s:6:0," ",currenttime_s:6:0," ",diff:1:0);
if diff>normalDelay then print(file(path),"^^^ Delay: ",diff:1:0," ^^^");
if diff>catchup then begin
check = true;
prevTime = time_s;
prevPCtime = currenttime_s;
end else check=false;
end;
It uses a function timeSdiff whose code is below:

Code: Select all

inputs: secTime1(NumericSimple), secTime2(NumericSimple);
vars: mintime1(0), mintime2(0), hours(0), minutes(0), seconds(0), curMinute(0), prevMinute(0), curSec(0), prevSec(0);

mintime1 = time_s2time(sectime1);
mintime2 = time_s2time(sectime2);
hours = IntPortion(mintime2/100) - IntPortion(mintime1/100);
if hours<0 then hours=hours+24;

curMinute = mintime2-(100*IntPortion(mintime2/100));
prevMinute = mintime1-(100*IntPortion(mintime1/100));
minutes = curMinute-prevMinute;
if minutes<0 then begin
minutes = minutes+60;
hours = hours-1;
end;
curSec = sectime2-(100*IntPortion(sectime2/100));
prevSec = secTime1-(100*IntPortion(sectime1/100));
seconds = curSec-prevSec;
if seconds<0 then begin
seconds = seconds+60;
minutes = minutes-1;
if minutes<0 then begin
minutes = minutes+60;
hours = hours-1;
end;
end;
timeSdiff = hours*10000 + minutes*100 + seconds;
TL;DR - the function calculates the #seconds difference between the two input times. The signal uses that function to print out the difference upon each 1-second bar calculation. If the difference is greater than "normalDelay", it prints that there was a delay. If the same PC time has a calculation for two or more bar times, a "catchup" is logged. Catchups are detected by checking whether there were other bar times between the PC time and delayed bar. I think catchups are less likely to be false-positives than a single delayed bar.

@hughesfleming, my previous server was in Montreal and I'm connecting from near Philadelphia.

@zheka - I'll check the latencies in Resource Monitor next week.

User avatar
Smoky
Posts: 507
Joined: 03 Dec 2010
Location: Thailand
Has thanked: 97 times
Been thanked: 115 times

Re: Delays receiving real-time data

Postby Smoky » 28 Sep 2018


I'd also make sure that MC's "Bugslayer" is disabled in the registry, it can slow things down too.
@ sptrader

good to know , do you have other registry setup to boost MC ?

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Delays receiving real-time data

Postby hughesfleming » 29 Sep 2018

@pcrespo,

Thanks for posting your code. I will run it as well and let you know if there are any issues. My server is in Frankfurt so about 100ms worse in terms of latency than yours. Have a good weekend.

Alex

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Delays receiving real-time data

Postby wilkinsw » 29 Sep 2018

Are any users trading with MC12 r2 and not experiencing delays/lags?

This all sounds like an ongoing MC12 bug, no?

Multicharts; would you consider putting MC12 back into Beta and supporting MC11 again?

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Delays receiving real-time data

Postby Zheka » 29 Sep 2018

@pcrespo,

I feel you are overdoing it and "observation" influences the results.

I/O operations (like printing to a file) are relatively slow, and you are printing at each tick and bar close!

My advice:
- sync clock once in 15 minutes, not seconds
- use 15-60sec bars
- find time differences via datetime and related functions, rather than your custom function
- log only delays>x, not all cases

You might also find this post useful.
viewtopic.php?t=47499

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 30 Sep 2018

@wilkinsw - my delays have been happening since the beginning of the year or maybe late last year, so for me it has nothing to do with version 12 R2.

@Zheka - you make good points. I should only log delays (defined by my inputs), rather than log the delay at every bar close. Syncing every 16s is probably extreme so I'll try 64. The new machine's clock is apparently awful out of the box, because at first, by accident, I wasn't syncing at all and almost every bar was logged as a delay.

I don't know about increasing the bar interval. I wouldn't get the same info. I'd miss many of those moments where the signal has to catch up by calculating several bars at once, which are the least likely to be false-positives and the most likely to cause orders to be delayed. Also, my trading signals use 1-second bars and I'd like the logger signal's conditions to be the same.

That thread you linked looks good but I haven't read it yet and I need to catch some sleep now.

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Delays receiving real-time data

Postby Zheka » 30 Sep 2018

What's the bar interval for your primary trading logic and what would be your typical trade length?

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Delays receiving real-time data

Postby TJ » 30 Sep 2018

Sync'ing the clock more than once every day is insane.

Probably it contributed more to your delay than anything.

It takes computer cycles to check the internet,
and it takes computer cycles for the server to reply to you,
and it takes computer cycles to update your computer.

Everything must come to a STOP when you update your computer time.


Unless your computer is unstable, the clock should not drift.
Sync'ing your clock will not improve data streaming.

ps. the Windows default is auto-sync once per week.

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 30 Sep 2018

Ok I've tweaked my code, and in Registry I increased the sync interval to 1024 seconds (I can't get SpecialPollInterval to work despite using the 0x1 flag, so I have to use powers of 2 via the min/max poll interval).

@Zheka - the thread you linked to indirectly provided a big clue. I hadn't thought to check my QM log. It turns out, my 9-second delay (and catchup) from two days ago in the 3am hour was caused by a disconnection.
IQFeed Message: S,SERVER DISCONNECTED
Then, one second before the calculation finally happened, the QM log says "Connection is OK" and "Connecting to real-time data for..."

So now I have to see if all the delays are caused by that, and if so, figure out why IQF disconnects.

(Although, when I had that 50+ second delay mentioned earlier in this thread, my colleague was seeing new data come in on the chart, which would suggest IQF wasn't disconnected.)

User avatar
Mark Brown
Posts: 181
Joined: 29 Nov 2016
Has thanked: 111 times
Been thanked: 17 times

Re: Delays receiving real-time data

Postby Mark Brown » 30 Sep 2018

i now think this delay is what caused 100 contract auto order to add 120+ new contracts erroneously to a position i had. i looked up and i had 220+ emini sp's on and a resting order stop for that 100. it got so screwed up i just unwound the whole position and have not tried auto trading again. i have no idea where that order come from except i did notice some huge lag in some orders being processed. i am using TS data mapped to ib symbols for the dom and auto trading.

i have made the registry changes in that ot5her tread and also disabled the logging cause that has to be a drain and won't / hasn't helped anything get fixed to date. my whole program just keeps hanging since build 12, it's slow to optimize, slow for everything.

i am constantly waiting for data that i expect should be in storage already on my drive? i don't think mc is storing data actually as it thinks it is. these constant warnings about objects is something new blowing my mind and the placement of them is bad. oh well maybe they will roll back to version 9 or something. makes me wonder about the future of mc and the umbilical cord it requires to run.

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 30 Sep 2018

@pcrespo,

Thanks for posting your code. I will run it as well and let you know if there are any issues. My server is in Frankfurt so about 100ms worse in terms of latency than yours. Have a good weekend.

Alex
Sorry for the mess of a code. This new code ought to achieve the same thing (good suggestion by Zheka):

Code: Select all

Inputs: normDel(6), catchup(3);
vars: intrabarpersist doneBT(false), intrabarpersist check(false), intrabarpersist prevTime(0), intrabarpersist prevPCtime(-1), intrabarpersist diff(0), path("");

once path = "C:\Users\...\Desktop\delays_"+symbol+"_filter_"+NumToStr(normDel,0)+"_"+NumToStr(catchup,0)+".txt";

doneBT = doneBT or LastBarOnChart_s;
if doneBT then begin
if check and time_s<prevPCtime then print(file(path),date:7:0," ",time_s:6:0," ",currenttime_s:6:0," There was volume during the delay.");
diff = datetime2eltime_s(computerdatetime-datetime);
if diff>normDel then print(file(path),date:7:0," ",time_s:6:0," ",currenttime_s:6:0," Delay: ",diff:1:0);
if diff>catchup then begin
check = true;
prevPCtime = currenttime_s;
end else check=false;
end;
And ignore the entries that appear the moment you start the signal.

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Delays receiving real-time data

Postby Zheka » 30 Sep 2018

@pcrespo,
What's the bar interval for your primary trading logic and what would be your typical trade length?

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Delays receiving real-time data

Postby hughesfleming » 01 Oct 2018

Hi Mark,

I wouldn't switch off logging until you know for sure what caused your double sized mini order. Unless you are seeing repeated queue delays (the bar at the bottom right that shows up) when you are trying to get orders filled then messing with the registry is probably not necessary. On the rare occasion that something slows down, it is because of a random disconnect to IQFeed.

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 01 Oct 2018

@pcrespo,
What's the bar interval for your primary trading logic and what would be your typical trade length?
Sorry I missed this question before. Currently I have two signals trading. One of them trades 4 instruments using 1-hour Data2 bars while the other trades one instrument using 1-min Data2. For both I use 1-sec Data1 bars, which isn't important to their success, so should I not? Typical trade length is 4-5 hours in the 1min signal and several hours to a few days in the 1-hour signal.

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Delays receiving real-time data

Postby Zheka » 01 Oct 2018

For both I use 1-sec Data1 bars, which isn't important to their success, so should I not
The question should be just the opposite: do you really, REALLY need it?
You whole production set-up should be scrutinized from this perspective, and optimized for robustness.

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 01 Oct 2018

Earlier today, there was a delay that did not correspond to an IQF disconnection, so unfortunately there remains a mystery cause.

@Zheka - I didn't realize it was hard for the system, since MC is allegedly capable of trading on tick charts. If it can handle ticks then 1-second should be a piece of cake (was my impression). But no big deal, I'll increase the interval. Maybe 15 seconds.

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Delays receiving real-time data

Postby Zheka » 01 Oct 2018

But no big deal, I'll increase the interval. Maybe 15 seconds.
:D You really insist on paying your dues.

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 01 Oct 2018

Heh I mean, I'm making the same number of trades (paying the same commission & slippage) regardless of my Data1 interval. It's just, when my signal says it's time to enter/exit, I'd rather not wait. Are you saying the Data1 interval should be at least a minute? Either way, if the delays persist, something is wrong. I find it highly doubtful that a 15s interval should cause delays.

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Delays receiving real-time data

Postby Zheka » 01 Oct 2018

You don't need a more granular Data1 at all - unless you are really micro-managing entries/exits with an "execution algo".

If you need to trade intrabar, then a stop/limit order submitted at bar end will work just fine.

Getting to a more granular time-frame will not solve the problem of "delays" - it will most probably make it worse/ more frequent.

And - by the way - delays are not "bad" by default. Depending on your system, 40-50% of the time you get a better price...

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 01 Oct 2018

Getting to a more granular time-frame will not solve the problem of "delays" - it will most probably make it worse/ more frequent.
I made it less granular, not more. It was 1s and then I made it 15s after your earlier post.
And - by the way - delays are not "bad" by default. Depending on your system, 40-50% of the time you get a better price...
My small intervals are admittedly OCD, but when it comes to delays, it's not the price that concerns me. It's that I don't know their cause, have no control of them, can't predict when they'll happen or how long they'll last. I don't want to have to tell clients, "Sorry, this trade lost X additional money because we had no ability to exit the position when we wanted to. We don't know why and it could happen again at any time." (Of course, that can't happen if someone is at the screen to manually exit, but human intervention should be the last line of defense, not something you expect to rely on.)
You don't need a more granular Data1 at all
In other words, set Data1 interval to what Data2 is and then delete Data2? With our 1-minute strategy, fine, but with our 1-hour strategy this would be a drastic change that we'd need to backtest. (Edit - actually, I hadn't considered stop-limit orders. With them, I suppose the change would be negligible, in fact it would be an improvement.)

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Delays receiving real-time data

Postby Zheka » 01 Oct 2018

I made it less granular, not more.
You made it more granular compared to a "decision" time frame.
With our 1-minute strategy, fine, but with our 1-hour strategy this would be a drastic change that we'd need to backtest.
If your strategy logic relies on a smaller timeframe - then it means it is not really a "1-hour strategy", or did you introduce it to solve a problem with "delays"?
How are you using a smaller timeframe?

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 01 Oct 2018

I suppose it's two strategies: a one-hour entry strategy and a smaller-timeframe exit strategy. We could achieve basically the same thing with stop-limit orders and "SetTrailingStop_pt", but I just remembered we have a good reason not to do that. I'd share the reason, but I'm not sure if my colleague considers it a trade secret lol.

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Delays receiving real-time data

Postby Zheka » 01 Oct 2018

OK, then it makes sense.
15-60sec bars would be a more robust solution then IOG.
All logic besides exit split behind a barstatus(2)=1 check.
Also, if you do not employ a portfolio-level MM logic in real-time, you might be better off trading from MC rather than PT - since MC uses 1 core per chart and PT uses 1 core for everything.

User avatar
pcrespo
Posts: 49
Joined: 07 Feb 2015
Has thanked: 7 times
Been thanked: 4 times

Re: Delays receiving real-time data

Postby pcrespo » 02 Oct 2018

If I use 4 PT windows, will that use 4 cores?

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Delays receiving real-time data

Postby Zheka » 02 Oct 2018

Yes, experiment..


Return to “MultiCharts”