Order submission lagging

Questions about MultiCharts and user contributed studies.
waveslider
Posts: 222
Joined: 16 Oct 2011
Has thanked: 66 times
Been thanked: 20 times

Order submission lagging

Postby waveslider » 18 Dec 2014

I realize this has been discussed in a now locked topic, but there was no resolution. I have been experiencing this for a long time and have not received a conclusion from Multicharts, although they have examined my logs, etc. numerous times. Since a large portion of users are now realizing that this is common, a solution seems overdue.

Situation - I am executing on 1 minute charts with autotrading, no intrabar execution. Across the board I see 1-13 seconds of lag between when the bar closes and the order is issued. Using Order and Position tracker page, the "Generated" column lists times 1-13 seconds after the close of the bar.

Facts:
1. I only trade super liquid markets and often at the close of the trading session - there is no lack of activity.
2. MC platform runs on windows 2008 server with adequate RAM and CPU. I monitor the usage levels and have not seen them exceed 60% during the lag event.
3. This has nothing to do with connection latency. The order says "Generated" between 1-13 seconds after the bar closes. The generated time should be the precise time of the next bar open.
4. Computer clock auto-synced daily
5. I do not see red numbers in the "unprocessed quotes" box on the bottom during these events.

One possibility that I see is that the new bar is not actually generated for a period and this is delaying the order issuance. It is difficult to confirm this possibility.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Order submission lagging

Postby orion » 18 Dec 2014

One possibility that I see is that the new bar is not actually generated for a period and this is delaying the order issuance. It is difficult to confirm this possibility.
waveslider, I did not get that last sentence you wrote. Doesn't looking at the chart tell you whether ticks for a new bar have arrived and a new bar is starting to form?

If you would like to record the exact time the new bar starts to form, you can do following: have the signal that generates the order set a global variable. Insert an indicator into your chart that updates on each tick. When the global variable is set, the indicator prints the timestamp of the first tick to a file and then resets the global variable.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Order submission lagging

Postby TJ » 18 Dec 2014

I realize this has been discussed in a now locked topic, but there was no resolution. I have been experiencing this for a long time and have not received a conclusion from Multicharts, although they have examined my logs, etc. numerous times. Since a large portion of users are now realizing that this is common, a solution seems overdue.

Situation - I am executing on 1 minute charts with autotrading, no intrabar execution. Across the board I see 1-13 seconds of lag between when the bar closes and the order is issued. Using Order and Position tracker page, the "Generated" column lists times 1-13 seconds after the close of the bar.

Facts:
1. I only trade super liquid markets and often at the close of the trading session - there is no lack of activity.
2. MC platform runs on windows 2008 server with adequate RAM and CPU. I monitor the usage levels and have not seen them exceed 60% during the lag event.
3. This has nothing to do with connection latency. The order says "Generated" between 1-13 seconds after the bar closes. The generated time should be the precise time of the next bar open.
4. Computer clock auto-synced daily
5. I do not see red numbers in the "unprocessed quotes" box on the bottom during these events.

One possibility that I see is that the new bar is not actually generated for a period and this is delaying the order issuance. It is difficult to confirm this possibility.
You can do the following to garner more public support:

1. Include a link to the "closed" thread. It turns people off when you say something but don't tell people anything.

2. Give an example of your "super liquid market". Not that we do not believe you, but if you are going to rally for support, at least get those who trade the same market on your side.

3. 1~13 sec is a long time. Are you talking about the confirmation returned from your broker? or the time the order actually generated by MultiCharts? Please show us how did you come up with this 13 sec statistics. Do you have screen shots or your MC log, broker statement time stamp? Do you have a PRINT statement that logs the time when MC triggers the order?

4. How many indicators do you have on the charts? How many drawing objects? Do you think they are slowing down your strategies? Do you see delays on plotting a new bar on your charts?

5. How big a dataset you are running? If you are on MC32, you will always see plenty of RAM available, because MC32 cannot see/use any RAM beyond 3 GB.

6. Does your trading logic reference to long history in your dataset, or employ multiple regression type analysis? Are you running multiple data stream that require synchronization?

7. Can you make a video of the order delay?

You sound frustrated, I hope you can find a solution soon.

waveslider
Posts: 222
Joined: 16 Oct 2011
Has thanked: 66 times
Been thanked: 20 times

Re: Order submission lagging

Postby waveslider » 18 Dec 2014

Hi TJ - here are some responses
1. regarding lagging order submission viewtopic.php?f=1&t=46208 - now locked due to inappropriate comments.

2. I am trading ES (S&P emini futures) primarily.

3. The one minute bar closes, then the order is generated seconds later. Of course the order should be issued immediately when the bar closes. On the screen capture you will see all the orders in the "Generated" column are lagging by about 5 seconds today, in the past I have seen as much at 13 sec. delay. The bar closed at 1300 and orders were issued at +/- 1300:05

4. I generally do not use indicators or drawing objects, but I do have multiple charts open - about 22 charts. If there was a slowdown due to indicators/objects/charts, wouldn't this be reflected in the CPU usage?

5. I am running MC64. The largest dataset for 1 minute charts is about 2 years.

6. System logic include some arrays that are no more than 200 values. Systems do use one other data stream with the same resolution.

7. I could make a video of the lag, but would rather try other avenues first. The order and position tracker window seem to say it all - specifically that the orders were generated quite a few seconds after the bar closed.
Attachments
ScreenHunter_166 Dec. 18 15.43.jpg
bar closed at 1300
(12.95 KiB) Downloaded 2737 times

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Order submission lagging

Postby MAtricks » 18 Dec 2014

The worst I've seen while MC was operating "correctly" was 3 seconds of lag while submitting an order. I'm not attempting to be an HFT firm, but 3 seconds is a long time in this biz. I'm with you Waveslider.. something needs to be done about this.

What are the specs of the machine you are using? An inadequate machine seems to be what affects MC the most. Age of machine, CPU model, and RAM speed and size.

Other than that..:
-the amount of bars on the charts and the amount of indicators calculated on the charts.
-graphics of many orders seem to slow down MC.
-order and position tracker seems to slow things down.
-the DOM definitely drags down the machine.

With all that said, MC definitely has been bogged down since the original 8.0 release and it has gotten increasingly more noticeable.

waveslider
Posts: 222
Joined: 16 Oct 2011
Has thanked: 66 times
Been thanked: 20 times

Re: Order submission lagging

Postby waveslider » 19 Dec 2014

Machine is dedicated server:
Quad core Xeon 1230 (3.2 gig)
16GB Ram

It's maybe 3 years old, the Sandy Bridge processors are solid.
I had not considered that Order Tracker would create a drain.

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Order submission lagging

Postby MAtricks » 19 Dec 2014

Well, that rules out the machine.

-How many windows do you have open?
-Have you reduced all charts to the minimum amount of bars?
-Have you removed all indicators from charts?
-Have you cleaned up any inefficient code?

I did these and I didn't see much difference, but its what MC will suggest.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Order submission lagging

Postby orion » 19 Dec 2014

waveslider, your statement about new bar not being actually generated and that being difficult to confirm was somewhat surprising to me. What do you think of post #2 on debugging the new bar generation delay issue? There are 4 possibilities:

1. ISP service
2. Datafeed service
3. Machine-workload issue
4. MC software issue

What datafeed and ISP service are you using? Is it possible that either the datafeed provider had a glitch or the ISP routing your datafeed had a glitch? Best way to rule that out is to instrument the datafeed itself. Then you can rule out #1 and #2 and can be sure that the problem lies with either #3 or #4.

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Order submission lagging

Postby MAtricks » 19 Dec 2014

Dial-up ISP in Guam wouldn't create 13 seconds of lag :)

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Order submission lagging

Postby orion » 19 Dec 2014

The worst I've seen while MC was operating "correctly" was 3 seconds of lag while submitting an order. I'm not attempting to be an HFT firm, but 3 seconds is a long time in this biz. I'm with you Waveslider.. something needs to be done about this.
Henry, I read your post https://www.multicharts.com/discussion/viewtopic.php?f=1&t=46208#p102838 carefully. Can you please confirm whether I am understanding you correctly? You are saying that the order is shown "generated" in MC after order status is received from broker? So MC does not show the order as "generated" at the time of sending the placeOrder() message with the broker?

MAtricks, I have instrumented order processing latency with IB via a standalone C++ order engine. 500ms delay is the lowest latency that I measured for the most liquid futures markets. This is the latency measured from the point the placeOrder() message was fired in my code to receipt of executionReport() or orderStatus() report whichever came first. I live in a US metro with a good ISP service. Note that 500ms was the lowest latency measured. The 99th percentile latency, which would correspond to the worst latency one would experience when trading, would be many times this number.

If Henry says that MC shows the order as "generated" as soon as the placeOrder() message is fired to the broker, then I would say that your point about the 3 sec latency is valid. OTOH, if Henry says that MC displays the order as "generated" after receiving the order status from the broker, then the 3 sec worst case latency is not at all surprising.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Order submission lagging

Postby orion » 19 Dec 2014

Dial-up ISP in Guam wouldn't create 13 seconds of lag :)
MAtricks, do you know the answer to the question for Henry; was your 3 sec worst case latency between (a) script calculation and place order fired to broker, or (b) script calculation and order status returned by broker? I am very interested in the answer since I can help you once I know the answer to that question.

By the way, don't be so hasty in ruling out ISP service or datafeed service issue. ISP bandwidth should not to be conflated with latency. TCP works well when you have no packet drops. But latency goes up several fold as soon as you have packet loss. The following paper measured latency to Google services for billions of connections and found that 10% of clients incurred at least one packet loss and flows with even one packet loss took an average of five times longer to complete than those without loss.

Another issue to note is that TCP uses a 3-way handshake to start the connection. Unlike ping which runs over ICMP and measures latency measured on round trip from source to destination and back, TCP makes 3 trips between source and destination to just open the connection. And that is without any actual message having been transferred. The actual message is the 4th trip across the Internet!

Welcome to Guam. :-)
Attachments
TCP-Latency.pdf
(581.08 KiB) Downloaded 229 times

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Order submission lagging

Postby MAtricks » 19 Dec 2014

Thanks for the interest! It would be great if we can nail this down... although, I think the culprit is obviously MC.

Your experience with internet hops is fascinating. I'd love to hear more. I thought it was 2 hops where the message got a across, not 4! Wow!

The lag I've noticed is post calculation and between order generation and order execution.

I'm connected at the exchange with a 1ms connection... its definitely not ISP problems with my situation. I monitor and record the connection between my machine and the order routing server in 1 second intervals. With the exception of a few anomalies in the past few months where I had ~12% packet loss for 1 second, there's no other recorded times where I have any packet loss at all.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Order submission lagging

Postby Henry MultiСharts » 19 Dec 2014

Henry, I read your post https://www.multicharts.com/discussion/viewtopic.php?f=1&t=46208#p102838 carefully. Can you please confirm whether I am understanding you correctly? You are saying that the order is shown "generated" in MC after order status is received from broker? So MC does not show the order as "generated" at the time of sending the placeOrder() message with the broker
orion, the order is shown "generated" in MC after order status is received from broker.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Order submission lagging

Postby Henry MultiСharts » 19 Dec 2014

waveslider, after the order is generated it is (usually) sent only on the next received tick (IOG) or when an opening tick (first tick) of the new bar is received. orion has already suggested how to trace the time of the next tick. That is also possible to use RecalcLastBar + IOG to have the code recalculated and orders sent with the specified timeout, without waiting for a new tick.

I would also add that you need to exclude all MultiCharts processes from the anti virus scanning list, or completely disable the anti virus, turn off all "realtime speed optimizing" tools, have your PC time automatically synchronized often enough (ex. use Dimension 4) - constant 5 sec lag looks like incorrect PC clock.

waveslider
Posts: 222
Joined: 16 Oct 2011
Has thanked: 66 times
Been thanked: 20 times

Re: Order submission lagging

Postby waveslider » 19 Dec 2014

Orion- I appreciate your creative problem solving. Unfortunately I don't have the time to get as deeply involved as you suggest.

The computer is co-located in Chicago, the data connection is fast, the datafeed is CQG which in my experience is good.
In this past example I watched CPU and RAM usage when the bar closed/orders were issued, as I mentioned, the usage did not go over 60%.
I do not have any antivirus aside from Microsoft essentials, it scans once a week on saturdays. PC clock is synced every 10 minutes (maybe a little excessive), there are no speed optimizing tools or other real time tools active.
In most cases at 4pm EST the S&P emini contract is very active and a tick would be forthcoming immediately after a bar closes.
Going to IOG would require retooling the code and bug checking that I would rather not get into. The strategy works fine, the execution is the problem. I can find no reason for a 5 second delay.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Order submission lagging

Postby orion » 19 Dec 2014

Thanks for the interest! It would be great if we can nail this down... although, I think the culprit is obviously MC. Your experience with internet hops is fascinating. I'd love to hear more. I thought it was 2 hops where the message got a across, not 4! Wow! The lag I've noticed is post calculation and between order generation and order execution. I'm connected at the exchange with a 1ms connection... its definitely not ISP problems with my situation. I monitor and record the connection between my machine and the order routing server in 1 second intervals. With the exception of a few anomalies in the past few months where I had ~12% packet loss for 1 second, there's no other recorded times where I have any packet loss at all.
MAtricks, with 1ms distance to the exchange, we have one less thing to focus on. But following need consideration:

1. Henry has responded that the order is shown as "generated" in MC after the order status from the broker. I would suggest putting in a feature request whereby the word "generated" is changed to "acknowledged" (or "acked" for short) since this is time when the order status acknowledgement is received from the broker. There should be a separate column displayed for "sent" which would be the time at which the initial order was sent to the broker by MC.

2. As it now stands, the "generated" delay includes all the queuing delays between MC and the broker, the broker's server for checking order validity, and then the acknowledgement back. The end-to-end system from MC to your broker and back can be considered one big distributed queuing system. So next point is important.

4. Is the 3 sec delay number that you mentioned the worst case or the average case? Note that worst case would be the 99th percentile latency (or 99.9th percentile latency) while average case would be the 50th percentile. We need to consider data on both latency and latency variability. What is your latency variability?

5. There is a big multiplicative factor between 50th percentile and 99th percentile latency in queuing systems. A well known result from queuing theory is delay D (latency) is proportional to 1/(1 - u) where u is utilization. For u = 0.2, D = 1.25 but for u = 0.9, D = 10. The delay can increase by a factor of 10X or more depending on infrastructure utilization conditions. It is for this reason that commuting a 5 mile distance in city rush hour can take 10X longer than it takes when there is no or very light traffic.

6. If during active market conditions any part of the infrastructure peaks at 90% utilization then it is going to service you with 10X the latency it would have when it was running at 10% utilization. The word "infrastructure" here includes the brokers servers, the exchange servers, the network pipes between you and the broker, and the network pipes between the broker and the exchange. Your broker's server is involved twice, once to process the order before sending to exchange and then to process the order status from the exchange. All of this happens before the order shows up in MC as "generated". "Generated" is a big misnomer and needs to be changed as discussed above.

7. As an example, I did the math on the infrastructure latency of a popular datafeed provider. Unlike the free datafeed one can get from one's broker, this datafeed provider is not free. What I found was that their infrastructure was designed for average case load and during active market conditions their datafeed would incur some significant delay. Same analysis applies to a broker's server or network infrastructure. How has your broker provisioned their infrastructure? Do they target average case workload or worst case workload in their infrastructure provisioning?

8. It is good that you are only 1 ms away from your broker but make sure you do not have a TCP pathology due to TCP timer setting. Windows TCP has a default timer setting of 3 seconds which means that if a packet did get lost then it will wait for 3 seconds to retransmit. This timer is auto-tuned by TCP during operation but it is best to set to a 50 millisecond value in your case since you are 1ms away from your broker. [Edit note: I had originally recommended setting TCP timer to 5 milisecond. It would be best to set it to 100 millisecond to account for not just the network flight time between you and your broker but also the operating system latencies incurred due to any context switching on both your machine and the broker's machine. It needs to be much larger than 5 ms unless both you and your broker are running a customized operating system.]

You can read more about Windows TCP timer setting here:
http://technet.microsoft.com/en-us/libr ... 38207.aspx
Last edited by orion on 19 Dec 2014, edited 1 time in total.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Order submission lagging

Postby orion » 19 Dec 2014

Orion- I appreciate your creative problem solving. Unfortunately I don't have the time to get as deeply involved as you suggest. The computer is co-located in Chicago, the data connection is fast, the datafeed is CQG which in my experience is good. In this past example I watched CPU and RAM usage when the bar closed/orders were issued, as I mentioned, the usage did not go over 60%. I do not have any antivirus aside from Microsoft essentials, it scans once a week on saturdays. PC clock is synced every 10 minutes (maybe a little excessive), there are no speed optimizing tools or other real time tools active. In most cases at 4pm EST the S&P emini contract is very active and a tick would be forthcoming immediately after a bar closes. Going to IOG would require retooling the code and bug checking that I would rather not get into. The strategy works fine, the execution is the problem. I can find no reason for a 5 second delay.
waveslider, please read my post #16 and make sure you understand all the points. They explain the 5 sec delay. This is not an anti-virus issue and not a PC clock sync issue.

waveslider
Posts: 222
Joined: 16 Oct 2011
Has thanked: 66 times
Been thanked: 20 times

Re: Order submission lagging

Postby waveslider » 19 Dec 2014

Thanks Orion -
It is apparent that you know significantly more about this stuff than I do. I agree that "Generated" is a misnomer and think you suggestion is valuable to break out when the order was issued and when the broker confirmed.
Our objective here is to find the source of the delay and (if possible) eliminate it. From what you are saying, my converting strategies to IOG is not necessarily going to help.
Setting the TCPInitialRTT seems like worth a try, but I am hesitant to mess with the registry without knowing exactly how to do it. The link you sent is appreciated, but not specific enough.

If the latency was consistent, and could be anticipated, the orders could be issued early by this amount and the result would be more accurate.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Order submission lagging

Postby orion » 19 Dec 2014

Setting the TCPInitialRTT seems like worth a try, but I am hesitant to mess with the registry without knowing exactly how to do it. The link you sent is appreciated, but not specific enough. If the latency was consistent, and could be anticipated, the orders could be issued early by this amount and the result would be more accurate.
waveslider, here is another link on how to set the TCP timer. Your setting it to 100ms is the right thing but it will only help a small part of the problem. Your broker's server, and for all you know, the exchange server talking to the broker's server may still be operating with a TCP timer setting of 3 seconds. The biggest issue, which is unsolvable unless you throw a lot of money at it, is queuing effects across all of the end-to-end infrastructure. Latency is "consistent" only under light load. Plot the graph of D = 1/(1-u) and see how it behaves.

http://support.microsoft.com/KB/170359

Jad
Posts: 92
Joined: 15 Jun 2014
Has thanked: 13 times
Been thanked: 21 times

Re: Order submission lagging

Postby Jad » 19 Dec 2014

That is also possible to use RecalcLastBar + IOG to have the code recalculated and orders sent with the specified timeout, without waiting for a new tick.
Henry, that's what I used to think but my faith in that is shaken. I have always used ReCalcLastBarAfter and even though I set it to its maximum refresh rate of 0.1 (FWIW, 0.001 made no difference), the delay in order submission is not limited to 100ms. Just printing out the clock time often shows a >200ms difference. I haven't experienced anywhere near the 13 seconds that waveslider is reporting. So, that's probably the result of another more serious issue - but perhaps exacerbated by this

Also, RecalcLastBarAfter often only seems to have a bearing on the chart's elements being updated and doesn't necessarily submit the order that was generated on the previous tick or several ticks ago. That lag can often exceed a second as I reported on the other thread and as others have also reported here. It's difficult to know for sure why sometimes, even another tick does not guarantee a submission but it does often happen.

Question: I found the following on http://www.multicharts.com/traders-blog/?cat=12" PlaceMarketOrder can now send orders when the bar is closed but is recalculated using RecalcLastBarAfter"

I realise that this was related to MC.Net but it made me wonder: Are we sure that all order types are affected by RecalcLastBar and under all conditions?

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Order submission lagging

Postby TJ » 19 Dec 2014

Thanks Orion -
It is apparent that you know significantly more about this stuff than I do. I agree that "Generated" is a misnomer and think you suggestion is valuable to break out when the order was issued and when the broker confirmed.
Our objective here is to find the source of the delay and (if possible) eliminate it. From what you are saying, my converting strategies to IOG is not necessarily going to help.
Setting the TCPInitialRTT seems like worth a try, but I am hesitant to mess with the registry without knowing exactly how to do it. The link you sent is appreciated, but not specific enough.

If the latency was consistent, and could be anticipated, the orders could be issued early by this amount and the result would be more accurate.
That's why I asked for a video... It is possible that the "delay" is a perception -- The order was actually sent on time, it is only the reporting that was delayed.

paulc
Posts: 59
Joined: 26 Sep 2012
Has thanked: 13 times
Been thanked: 2 times

Re: Order submission lagging

Postby paulc » 19 Dec 2014

Can MC staff comment on their product testing experience here. I'm sure MC will have tested orders with all the brokers they integrate with and hence have the widest experience amongst us all.

What are the expected range of times for orders "generated". If MC can provide figures per broker, CPU and RAM, etc that would be a useful benchmark and provide some light on what appears right now to be a somewhat dark art. Clearly times will vary depending on so many influencing factors but at least if MC can provide the figures we can see how our setups measure up.

waveslider
Posts: 222
Joined: 16 Oct 2011
Has thanked: 66 times
Been thanked: 20 times

Re: Order submission lagging

Postby waveslider » 19 Dec 2014

I had 35 second delays on the close today, so, yes I have some serious issues.

During that period I was watching the CPU and RAM usage, nothing was overworked.

I noticed that MC was acting super sluggish during the period when the orders "should" have been submitted. CQG confirms that they received the order 37 seconds after the bar closed - about 0.5 seconds after the order was listed as "Generated".

How can I find out exactly when the order left the platform? That would help me begin to trace the latency

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Order submission lagging

Postby orion » 19 Dec 2014

I had 35 second delays on the close today, so, yes I have some serious issues. How can I find out exactly when the order left the platform? That would help me begin to trace the latency
MC staff (Henry or Andrew) can comment on what sort of logs MC has for analysis. In addition, I would recommend using Wireshark. Wireshark will allow you to peek inside the network packets to see what is inside. But since all MC communication with the broker is encrypted, you won't be able to peek inside the packets. However, you can still instrument the exact times when packets entered or exited your machine and from which source and to which destination. There is a lot that can be deciphered with such a packet trace. Keep us posted on what you find out.

http://en.wikipedia.org/wiki/Wireshark

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Order submission lagging

Postby MAtricks » 20 Dec 2014

I love all the information pouring out in this topic :) But because of that, I'm being a little indirect. So before we get too far derailed I must state that the lag problem is the MC software without any question. I've been working on this for a long time... and tried my hardest to blame it on everything under the sun, but after studying every aspect with several machines, locations, and every piece of monitoring software that we could get/build--the conclusion is always the same.

We just learn how to deal with it and wait for a fix that hopefully will come.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Order submission lagging

Postby orion » 20 Dec 2014

MAtricks, I am new to MC and do not have the experience of trading MC like you do; so I value your input highly. Trading latency being an MC problem is certainly a valid hypothesis. Hence my suggestion to waveslider to instrument his network packet traffic.

I know for a fact that MC has latency issues in other areas. For example, I have witnessed that it takes a long time to write out a strategy report in MC and even a null strategy report that does not contain any trades in the backtest takes a long time. This is a significant latency issue that needs to be fixed. However, unlike the trading latency issue, the cause for this latency is easily isolated and does not require the weight of statistical evidence.

Since trading latency is not a single number but a statistical distribution with a tail, it will help if we had statistical evidence on this topic instead of isolated trades. By statistical evidence, we are talking about a latency histogram with 100+ trades in liquid instruments at various times in the market. I would be interested in comparing notes on realtime trading latency measurement statistics with interested parties in a few months when I have collected enough evidence to either accept or reject the hypothesis that MC has an issue.

Jad
Posts: 92
Joined: 15 Jun 2014
Has thanked: 13 times
Been thanked: 21 times

Re: Order submission lagging

Postby Jad » 21 Dec 2014

Readers: please be extremely wary of people like orion. He has sowed enough confusion with his continued flip-flopping. This is my last post on this topic and its purpose is to counter the confusion. MC is undoubtedly, at the very least, partly responsible for the latency issues that many of us have experienced and which we've been trying to report here.

By his own admission, he is new to MC and does not have the experience of trading MC like some of us do. This did not stop him saying, with absolute certainty just a few days ago that "MC is not the issue" and irrelevantly, "The latency you measure via ping tests to your datafeed or your broker is not the latency that you will experience when trading" No one ever said it was.
The latency being reported was and is one (of possibly a few) caused by MC's design. Now he says "Trading latency being an MC problem is certainly a valid hypothesis." Sheesh!

Sarcasm aside, it's important that no one should be allowed to demand that anyone reporting a problem should be ignored and the moderators should be ashamed of themselves for not stamping on that immediately rather than my response to it.

Don't allow anyone like this to bully or frighten you into not reporting any problems that you may experience just because 'they' don't think it's important. Such reports help all of us and will help to ensure that Multicharts leads the fierce competition with a comparatively bug-free product. Self-enamoured egoists do not.

TA100
Posts: 54
Joined: 10 Feb 2014
Has thanked: 39 times
Been thanked: 9 times

Re: Order submission lagging

Postby TA100 » 21 Dec 2014

I love all the information pouring out in this topic :) But because of that, I'm being a little indirect. So before we get too far derailed I must state that the lag problem is the MC software without any question. I've been working on this for a long time... and tried my hardest to blame it on everything under the sun, but after studying every aspect with several machines, locations, and every piece of monitoring software that we could get/build--the conclusion is always the same.

We just learn how to deal with it and wait for a fix that hopefully will come.
Thanks WaveSlider, Orion, MATricks & TJ

This is a very helpful thread and like many I've been experiencing delays for months - MC reckoned it was RHM (realtime history matching) which for some bizarre reason is auto enabled in the Backtesting box of Strategy Testing - disabling this seems to make no difference. Worse I've had market orders painting (appearing) on my charts when price left 30 points ago and over the course of a few months managed to minimise that issue through finer granularity - now the order delay has surfaced in the past 3 months. There is definitely a problem with MC and so much so I'm trying to compare by using alternative data providers and coding same strat with workarounds in TS which is not as easy as it sounds due to bid/ask not being available as separate data streams in TS.

Using a server in UK & Chicago with roughly the same setup and using recalclastbarafter(1) I've noticed debilitating cpu spikes from the UK and not experienced half as bad from Chicago server which by comparison is a much slower (older) cpu - so perhaps the TCP issue which Orion mentioned could be the cause in order delay - I've looked at both links and tried to figure out configuring a new registry DWord entry as per links but that is beyond me right now - perhaps its because I'm using a 64 bit OS..?

- Equally as disconcerting I've found is that the longer the delay the greater the move on the bar - double whammy. Last week I got so fed up with waiting for the order to fire I entered manually at market and received the fill 20 seconds later - great..

Seasons Greetings to all.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Order submission lagging

Postby orion » 21 Dec 2014

This thread has been poisoned like the last one. So I will end my posting here with one final note:

1. There has been no flip-flopping since my very first post in this thread posited MC software as potential hypothesis. See post #8 (excerpted below):
There are 4 possibilities:
1. ISP service
2. Datafeed service
3. Machine-workload issue
4. MC software issue
2. MC software is always a potential hypothesis given that I have not yet collected statistical data with my own trading. The intent behind my posts in this thread was to help with collective problem solving by elevating the level of awareness on other potential causes of latency due to end-to-end infrastructure given my two decades of experience building infrastructure hardware and software.

3. I was cautioning that while ping may give you a latency of 1ms because you are colocated in your broker's datacenter, your TCP latency within that same datacenter may actually be 3 seconds due to the default TCP timer setting. You won't know until you instrument all your TCP traffic. Just one packet loss is required for your application to experience that kind of latency even within a datacenter where ping gives you a latency measurement of 1ms!

4. There are quite a few other subtle issues that would have been worthy of discussion (for example, google "datacenter tcp incast congestion"). There is also another insidious issue (google "cisco switch qos acl") which can make one user in the datacenter appear like he is on the moon compared to another user even though they are both 1000 feet away from the broker's server.

The flash crash of May 2010 showed that even institutions can hit latencies of 30 seconds due to the nature of the latency tail. The question of where the issue lies requires both deep understanding of how infrastructure operates and the willingness to discuss and brainstorm issues with objectivity and civility.

waveslider
Posts: 222
Joined: 16 Oct 2011
Has thanked: 66 times
Been thanked: 20 times

Re: Order submission lagging

Postby waveslider » 21 Dec 2014

Readers: please be extremely wary of people like orion.
Jad -
I believe others would agree that criticism of other contributors is not appreciated. You have offered wisdom here and if you disagree with someone else's suggestion, that's fine, but please keep it to yourself. This is not a popularity contest and the last thread was closed due to these personal attacks.

Let's keep on topic - we have some very smart people here and experienced MC users, we can get a lot done. I would hope to persuade Orion to continue his (IMHO) valuable suggestions, and I would be interested to hear Henry's (or other's) views on their possibilities as contributing factors to the order latency issue.

I plan to install wireshark as suggested by Orion to determine when packet left the server and arrived at their destination. By simple deduction we should be able to narrow down the possibilities.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Order submission lagging

Postby Henry MultiСharts » 22 Dec 2014

Dear users,

To address concerns regarding the subject please make sure the logging is enabled for MultiCharts (it is often disabled by anti-virus/UAC). Save the attached file, unzip the reg file corresponding to your version of MultiCharts. Double-click on it and confirm registry modifications. Upon next start of MultiCharts - regular application logging will be enabled.

When the described behavior is replicated please do not restart MultiCharts and provide the following information for further analysis (send it so support@multicharts.com):
1) Specify exact version and build number of MultiCharts are you running (in MultiCharts go to Help tab-> About);
2) Specify the broker you are using;
3) MultiCharts logs (keep in mind that the logs from the previous run are erased when you start MC):
In MultiCharts go to Help->Feedback->Send logs. Please let me know that you have uploaded the logs. If you want to send the logs manually please follow this guide: https://www.multicharts.com/trading-sof ... harts_Logs
4) In MultiCharts go to File->New->Open Order and position tracker window-> Orders tab->make sure you are not filtering the information in columns, then go to File->Export to excel.
Please also export the information from the Logs tab;
5) Attach the workspace you are using (it has not personal information/no studies source code);
6) Attach detailed description of the situation, highlight the orders in question, attach your own logs/analysis results (if you have any).

If the provided information is incomplete (any of the steps are skipped) - we cannot guarantee complete analysis of the situation. If that is required - we will request enabling additional logging for further analysis of the situation and will provide further guidance.
Attachments
all_traces_on.zip
(3.4 KiB) Downloaded 208 times


Return to “MultiCharts”