Latency. Please share experiences.

Questions about MultiCharts and user contributed studies.
wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Latency. Please share experiences.

Postby wilkinsw » 03 Mar 2014

Hi,

I'm noticing that it is often taking at least a second and on occassions nearer two seconds to get an order to market from the time the calculation is initiated.

I use MC64 with Patsystems as my broker and IQ feed as my data vendor.

I run about 17 live charts which on average run with about 5k bars history, 1 signal and about 3 indicators (all simple code) each.

My hardware: AMD phenom II x4 850 3.3ghz, 4gb Ram, windows 7 64bit.

Please share you experiences as I have no idea if this is normal for Multicharts64/my setup.

What has specifically made me submit this post is the following:

Last night for the open of Nymex's CL, the first tick triggered my signal to calculate and submit a buy limit order above market (as intended). Unfortunately, I had a lag of at least 1.015 seconds meaning I missed the fill. Sod's law, this missed trade went straight to my target 44 minutes later in pretty much a straight line. The order tracker states that the order was submitted 2 seconds after the opening time of 2300gmt. The time betwen the first tick following the opening time and the tick where my limit order price was exceeded (and never touched again) was 1.015 seconds.

Please let me know your thoughts.

Specifically, is it:

A) Multicharts and/or powerlanguage are not capable of lower latency times
B) My computer is too slow.
C) Its Pats API/Plugin related.

Many thanks,

Will
Attachments
CL tick chart annotated.PNG
CL tick chart annotated
(47 KiB) Downloaded 6913 times

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Latency / slow order submission. Please share experience

Postby TJ » 03 Mar 2014

My first thought is...
4GB is not a lot of memory these days, especially for a trading computer with 17 charts.
Please make a screenshot of your TaskManager Performance tab. It should tell you if your memory utilization is ok.

The CPU, though not a fast one, should be adequate. Again, the TaskManager should give you the indication if you are overloaded.

This chart will show you where your CPU stands in relation to the current offerings:
http://www.cpubenchmark.net/cpu_list.php

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency / slow order submission. Please share experience

Postby wilkinsw » 03 Mar 2014

Hi TJ,

Please find attached my task manager.

So from your response, you think that the latency I'm experiencing is indeed excessive (calculation to order hitting exchange)?

I suspect I'm near the limit to whats advisable, performance wise, and am awaiting a move onto a server. But when I say near the limit would you agree that the spare capacity that you can see on my task manager should be okay (or not)?

Thanks.
Attachments
TM snip.PNG
Task manager image
(43.79 KiB) Downloaded 6925 times

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Latency / slow order submission. Please share experience

Postby TJ » 03 Mar 2014

Hi TJ,
Please find attached my task manager.
So from your response, you think that the latency I'm experiencing is indeed excessive (calculation to order hitting exchange)?
I suspect I'm near the limit to whats advisable, performance wise, and am awaiting a move onto a server. But when I say near the limit would you agree that the spare capacity that you can see on my task manager should be okay (or not)?
Thanks.
My response was my first thoughts only... the computer spec caught my eyes, I did not have time to study your other data.

The opening is always a kaotic time. One second at the opening is not a lot of time, and I do not know what kind of internet latency you have, and how accurate is your computer clock compared with the exchange time.

It is difficult to tell the exact cause of events from your chart; I would gather the logs (both from MultiCharts and your broker) and use Live Chat to let tech support check your system out.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency / slow order submission. Please share experience

Postby wilkinsw » 04 Mar 2014

Thanks TJ.

I trade from a prop firm in London so our internet bandwidth (especially on a Sunday evening) is good.

Unfortunately I had logs disabled as I was troubleshooting another issue I was having with MC.

As you say, I suspect the high load of the market opening might be the cause of a small lag. I'll start monitoring the latency for intraday order submissions.

If anyone wants to share their experience/knowledge of latency with what they think are typical/acceptable times taken from calculation till order hitting the exchange, I would be very interested to hear.

escamillo
Posts: 203
Joined: 25 Mar 2011
Has thanked: 23 times
Been thanked: 56 times

Re: Latency / slow order submission. Please share experience

Postby escamillo » 04 Mar 2014

it is your computer.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency / slow order submission. Please share experience

Postby wilkinsw » 04 Mar 2014

escamillo, I hope that is the case as it's easily rectified.

Would you care to elaborate. What are you running and with what latency or with what spare capacity? Or how was such a conclusion so obvious to you?

Thanks.

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Latency / slow order submission. Please share experience

Postby MAtricks » 04 Mar 2014

1 full second is extremely bad latency. MC should process your orders within a few microseconds on your side of things. Although its the easiest to blame, I doubt its your machine. Your connection is almost always the issue.

1. Download PingPlotter to your trading machine
2. Find the relevant IPs for IQ and Pats
3. Do a round trip ping test to IQ feeds servers (my average is 2ms)
4. Do a round trip ping test to Pats Order servers (my average is 2ms)
5. Add that up and double it for good measure


I'd love to hear from the MC team on this as well. I'm curious how "fast" this software is.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency / slow order submission. Please share experience

Postby wilkinsw » 04 Mar 2014

Great thanks MAtricks, very useful. I'll start investigating that now and post the results here.

Additionally:

My computer clock is 4 secs fast.....could that be an issue?

I use IQfeed for the calculation of my signals which are submitted to Pats. Could not using Pats as the basis for calculation be an issue and a possible cause of latency? I use the former setup and I find IQ the smoother and more reliable of the two.

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Latency / slow order submission. Please share experience

Postby MAtricks » 04 Mar 2014

3rd party anything adds another trip to the process, but it won't be more than a few milliseconds.. Unless you're attempting low latency projects which should not be attempted with MC, this won't be an issue.

Your pc clock shouldn't matter unless your signals are triggered on a time of day event.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Latency / slow order submission. Please share experience

Postby TJ » 04 Mar 2014

::
Your pc clock shouldn't matter unless your signals are triggered on a time of day event.
It matters in the first second of the trading session.

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Latency / slow order submission. Please share experience

Postby MAtricks » 04 Mar 2014

How so? Actual trading is based off of the order server, not the pc clock. Unless the trade is based on a time of day event..

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency / slow order submission. Please share experience

Postby wilkinsw » 05 Mar 2014

I'd love some help from MC.

Maybe some hyperlinks to other threads.

My computer clock is now in synch with exchange time. I'm noticing that "generated" order column in the tracker is showing 1-2 seconds post the turn of time intervals for the submission of orders.

Would anyone else out there care to check their order and position tracker and report back with the seconds passed a given time interval that your orders are submitted. Would be very helpful indeed.

Also, in TS you can use to measure code calc speed:

Code: Select all

inputs:
TimerON(true);

{ Components declaration }
var:
elsystem.StopWatch StopWatch1( NULL ),
intrabarpersist double StrategyTime(0);

{ This method gets called by EasyLanguage one time at the beginning to create and initialize the components }
Method override void InitializeComponent()
begin
StopWatch1 = new elsystem.StopWatch;
StopWatch1.Start();
end;


if LastBarOnChart then begin
If TimerON then begin
StrategyTime = StopWatch1.ElapsedMilliseconds/1000;
print("Strategy Time = ", StrategyTime:14:3, " Sec Bars Analyzed = ", BarNumber:11:0);
end;

end;
What might be the equivalent in Multicharts (down to milliseconds)? Is there a stopwatch / milliseconds DLL timer type thing I can use?

Thanks!

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Latency / slow order submission. Please share experience

Postby Henry MultiСharts » 05 Mar 2014

Dear users,

In general case:
Once the order is generated in MultiCharts with certain parameters – it is sent to the broker. The broker processes the order and fills it or rejects it. The broker is in charge of order execution price, time and order status report. MultiCharts waits for the order status from the broker to show it in MultiCharts. If any modification or cancel command is generated by the code in MultiCharts – it is sent to the broker.
Usually such commands are generated and processed in MultiCharts within milliseconds.

From MultiCharts logs we can measure the time from generating the command and sending it outside of MultiCharts (broker API), and the time from receiving the reply command outside of MultiCharts (broker API) until processing it and displaying the status in MultiCharts interface. MultiCharts cannot affect the possible Internet latency and/or the broker order processing speed, as well as the broker sending the order status.

Order and Position tracker shows local PC time. Some millisecond delay is possible between MultiCharts receiving the order status and processing it, showing it on the OPT window.

Wilkinsw, as for the your particular case, as we have already discussed in the email correspondence, please send me the MultiCharts logs and OPT tabs export so that we can study them and provide the analysis: how long did it take to generate the order in MultiCharts, process it and send the order command to the broker, when did the broker provide the order status reply.

I will ask our developers if that is possible to create a code for measuring the code calculation speed.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency / slow order submission. Please share experience

Postby wilkinsw » 05 Mar 2014

Thanks for the reply Henry. It's made a few things clearer.

So what might look like latency is in fact a whole raft of confirmations going back and forth which make it appear to delay the order submission speed but, from what you say, it is probably taking milliseconds on Multicharts' end to calculate and submit.

I haven't sent you logs re the Sunday night missed fill because, as mentioned previously, logging was off due to another issue I've been struggling with.

However there is definitely a lag somewhere otherwise I would have got that fill on Sunday night (>1second window). My colleague (as you know he also uses MC) also missed the fill.

Therefore, now I have logging back on, I will send you my logs for analysis to see if there are any obvious signs of latency since I last restarted MC (yesterday evening).

I still need to carry out the ping tests that MAtricks kindly advised me to do.

Some way of calculating code calc speed will be great and I'd imagine would be a useful tool to many system developers using MC.

Thanks!

gpw797
Posts: 216
Joined: 04 Mar 2006
Has thanked: 3 times
Been thanked: 7 times

Re: Latency / slow order submission. Please share experience

Postby gpw797 » 05 Mar 2014

I agree about it being slow... I upgraded my computer because I thought that was the problem and now have a pretty nice machine (8 core AMD, 16 gig RAM, SSD hard drives) and still see some latency. I tried SC (non .net version) and it executes my orders substantially faster. Sometimes by the time my order shows up price has moved 2-3 ticks

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency / slow order submission. Please share experience

Postby wilkinsw » 05 Mar 2014

gpw797, thanks for sharing.

Henry, if you could just confirm (or correct me) that the reason gpw797 might see a lag in their order showing up is due to the confirmation process needing to take place first between MC and the broker before that order shows up.

But, critically, the order has already been submitted to the broker before the order shows up on his ront end.

I hope I got it right.

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Latency / slow order submission. Please share experience

Postby MAtricks » 05 Mar 2014

Another thing to consider would be the order type and where price was as the order was being placed. If two people received the same nonfill with the same strategy, I would look at the how the execution is coded.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency / slow order submission. Please share experience

Postby wilkinsw » 05 Mar 2014

I don't know the exact details of my colleague's strategy.

My strategy literally submitted a buy limit order for CL at 103.05. Submitted following the calculation of my code after the first tick after 1800 est / 2300 gmt on 02/03/2014.

The time between the first tick and the offer exceeding 103.05 was 1.015 seconds. So don't understand why I wasn't filled.

User avatar
JoshM
Posts: 2195
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1544 times
Been thanked: 1565 times
Contact:

Re: Latency / slow order submission. Please share experience

Postby JoshM » 08 Mar 2014

What might be the equivalent in Multicharts (down to milliseconds)? Is there a stopwatch / milliseconds DLL timer type thing I can use?
I thought like this, but feedback is welcomed since it seems a little quick to me:

Code: Select all

Variables:
dtBegin(0), dtEnd(0), oneMS(1/86400000);

if (BarStatus(1) = 2) then begin

dtBegin = ComputerDateTime;

for value20 = 0 to 100 begin

for value1 = 0 to 100 begin

value2 = Average(Close[value1], 20);
value3 = Average(High[value1], 20);
value4 = Average(Low[value1], 20);
value5 = Average(Open[value1], 20);

end; //: Inner loop

end; //: Outmost loop

dtEnd = ComputerDateTime;

Print("Execution time (ms): ",
(dtEnd - dtBegin) / oneMS);

end; // OnBarClose
Output:

Code: Select all

Execution time (ms): 78.00
Execution time (ms): 63.00
Execution time (ms): 78.00
Execution time (ms): 63.00
Execution time (ms): 62.00
Execution time (ms): 63.00
Execution time (ms): 78.00
Execution time (ms): 62.00
Execution time (ms): 78.00
Execution time (ms): 63.00
Execution time (ms): 62.00
Execution time (ms): 63.00
Execution time (ms): 62.00
Execution time (ms): 78.00
Execution time (ms): 63.00
Execution time (ms): 62.00
Execution time (ms): 79.00
Execution time (ms): 62.00
Execution time (ms): 63.00
Execution time (ms): 62.00
Execution time (ms): 78.00
Execution time (ms): 63.00
Execution time (ms): 78.00
Execution time (ms): 62.00
Execution time (ms): 94.00
Execution time (ms): 78.00
Execution time (ms): 63.00
Execution time (ms): 62.00
Execution time (ms): 63.00
Execution time (ms): 78.00
Execution time (ms): 62.00
Execution time (ms): 63.00
Execution time (ms): 78.00
Execution time (ms): 62.00
Execution time (ms): 63.00
Execution time (ms): 78.00
Execution time (ms): 63.00
Execution time (ms): 62.00
Execution time (ms): 63.00
Execution time (ms): 62.00
Execution time (ms): 78.00
Execution time (ms): 63.00
Execution time (ms): 62.00
Execution time (ms): 78.00
Execution time (ms): 63.00
Execution time (ms): 78.00
Execution time (ms): 62.00
Execution time (ms): 63.00
Execution time (ms): 78.00
Execution time (ms): 63.00

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Latency. Please share experiences.

Postby Henry MultiСharts » 20 Mar 2014

Please find our sample code attached.
Attachments
calculate_time_ms.pla
(2.96 KiB) Downloaded 614 times

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby wilkinsw » 20 Mar 2014

Hi,

Thanks for that. I know I must be doing something wrong:

When I add your indicator using the ADX example in the code......It takes a while to load (that'll be the loop) but then all I get in the output is:

"execTime = 0.00000"

What have I done wrong?

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Latency. Please share experiences.

Postby Henry MultiСharts » 24 Mar 2014

"execTime = 0.00000" means the code was calculated very fast, there is nothing wrong with it.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency / slow order submission. Please share experience

Postby wilkinsw » 27 Mar 2014

1 full second is extremely bad latency. MC should process your orders within a few microseconds on your side of things. Although its the easiest to blame, I doubt its your machine. Your connection is almost always the issue.

1. Download PingPlotter to your trading machine
2. Find the relevant IPs for IQ and Pats
3. Do a round trip ping test to IQ feeds servers (my average is 2ms)
4. Do a round trip ping test to Pats Order servers (my average is 2ms)
5. Add that up and double it for good measure


I'd love to hear from the MC team on this as well. I'm curious how "fast" this software is.
Right. For those following this thread, I still haven't found a comprehensive answer as to why it took >1sec to recognise the opening first trade of the session and get my order to market.

However I've found some answers and have made some progress re solutions.

Firstly:

Ping tests: IQfeed servers (from London) =140ms London Patsystems servers = 3ms Totalx2 (see MAtricks) = 286ms ......a whopping 1/3 of a second.

Secondly:

Multicharts script calculation speed: Not an issue ..... a few microseconds (forget milli!).

Thirdly:

API plugin / API limitations. Please contact Henry for more information regarding this. In a nutshell when multiple orders are submitted to the same API account, in an instance (such as market open) they are queued and submitted one at a time. This can be improved by having multiple API accounts (and multiple cores using each account). I haven't been able to nail down a specific delay to this particular bottle neck.

Fourthly:

Computer limitations. I think my computer may have not been able to efficiently handle the acute load with 17 charts all submitting orders at virtually the same exact millisecond event (the open).

Conclusion and Solutions:

Firstly. Most of the third of a second datefeed/trade server latency can be saved with a better selected remote server location. If you can get colocated then even better. For example, if I switched to the much more expensive TT API that Marex offer then I'll be colocated at the Cermak datacentre in Chicago. However, I think I'll be looking at about £1000pm for that vs. Pats's £40pm (not inc Jtrader). Can't afford this at the moment. Unfortunately my nearest Pats trade server to Chicago is London.

Secondly. I have just upgraded to a server with 8GB Ram and an Intel Xeon E5-2620v2 2.10Ghz processor.

Thirdly. To help avoid API plugin bottlenecks.
1) Use GTC/D orders. Most ideally exchange supported GTCs. Your orders are then already part of the premarket order book building process, bypassing the acutely demanding event of the open on Multicharts and your computer. If not exhange supported GTCs and are instead synthetic then: Your broker will be responsible for getting your orders to market AT the open. Question is; who is faster. You and your computer or the broker? Do you trust your broker? etc etc.

Finally, to help avoid event driven overload, that also occurs when multiple bars on different charts all close at the same interval*, it might be worth considering having bars close on unusual minutes. E.g. GC=59past...HG=58past....EU=57 past etc etc instead of 00 on the hour when using 60min timeframe. Or alternatively acheive the same effect but with IOG=ON and creating the calculation event on an unconventional interval.

I hope my findings help. Basically, ping tests, computer power, bottleneck workarounds and emails to your broker!

*but note that the variance in milliseconds of when the first tick of the new bar arrives for all your instruments is far larger and hence less of an issue than the first tick that arrives at the start of a new session (which for my CME group products will all arrive well within the same millisecond!). Hence the open is still, by far, a more demanding issue and the bigger driver of latency.
Last edited by wilkinsw on 27 Mar 2014, edited 2 times in total.

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Latency. Please share experiences.

Postby MAtricks » 27 Mar 2014

Good to hear that you are nailing down the issue. 1/3 of a second is hideous :) and most definitely a lot of the issue.

If you can lower that latency, that'd be great. If not, I'd suggest looking into switching things to AA mode. This will cut your latency issues in half seeing as MC will not have to confirm orders on its end.

I'm a little unsure of the process of exact process of things. I think this is how it works-

SA mode: 1. Data to MC, 2. MC to Order Server, 3. Order Server to MC.

AA mode: 1. Data to MC 2. MC to Order Server.


Henry or Andrew, could you confirm?

User avatar
JoshM
Posts: 2195
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1544 times
Been thanked: 1565 times
Contact:

Re: Latency / slow order submission. Please share experience

Postby JoshM » 28 Mar 2014

Ping tests: IQfeed servers (from London) =140ms London Patsystems servers = 3ms Totalx2 (see MAtricks) = 286ms ......a whopping 1/3 of a second.
I would not put a terrible amount of faith in ping numbers. For example:
Monitoring Internet performance and measuring user quality of experience are drawing increased attention from both research and industry. To match this interest, large-scale measurement infrastructures have been constructed. We believe that this effort must be combined with a critical review and calibrarion of the tools being used to measure performance.

In this paper, we analyze the suitability of ping for delay measurement. By performing several experiments on different source and destination pairs, we found cases in which ping gave very poor estimates of delay and jitter as they might be experienced by an application. In those cases, delay was heavily dependent on the flow identifier, even if only one IP path was used. For accurate delay measurement we propose to replace the ping tool with an adaptation of paris-traceroute which supports delay and jitter estimation, without being biased by per-flow network load balancing.
From Paris to Tokyo: On the Suitability of ping to Measure Latency (pdf)

Also interesting: Don’t use ping for accurate delay measurements.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby wilkinsw » 28 Mar 2014

Yes that is interesting.

But still safe to say that the London to chicago distance makes up a good chunk of that 1+second latency I experienced in my case study.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Latency. Please share experiences.

Postby Henry MultiСharts » 28 Mar 2014

If not, I'd suggest looking into switching things to AA mode. This will cut your latency issues in half seeing as MC will not have to confirm orders on its end.
I'm a little unsure of the process of exact process of things. I think this is how it works-
SA mode: 1. Data to MC, 2. MC to Order Server, 3. Order Server to MC.
AA mode: 1. Data to MC 2. MC to Order Server.
Henry or Andrew, could you confirm?
That is not corect. The speed and procedure of communication with the broker is the same when using SA and AA mode.

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Latency. Please share experiences.

Postby MAtricks » 28 Mar 2014

Thanks Henry.

So MC does confirm a fill on the orders in AA mode? It sure doesn't seem like it.. but I stand corrected.

I'd really like to understand the paths that MC takes. Is there a flow chart or a quick description of them somewhere?

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby wilkinsw » 28 Mar 2014

Thanks Henry.

So MC does confirm a fill on the orders in AA mode? It sure doesn't seem like it.. but I stand corrected.

I'd really like to understand the paths that MC takes. Is there a flow chart or a quick description of them somewhere?
I second that. Would absolutely love a schematic or just a better description of the communications that take place.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Latency. Please share experiences.

Postby Henry MultiСharts » 04 Apr 2014

I second that. Would absolutely love a schematic or just a better description of the communications that take place.
You can share the log analysis I have sent you on 3/6/2014. It contains detailed description of each event that took place.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby wilkinsw » 04 Apr 2014

As requested:
Hello William,


Our engineers have studied the logs from your PC for the orders in question.
Here are the details of order generation and API replies with the time stamps with millisecond precision.
All timestamps are Local time = correspond to your computer clock.

PATS API replies (as far as we know they mean the following):

Queued - Order accepted by the API (API dll on your local PC)
Sent - Order sent to the server
Working - Order accepted by the exchange


Order #6012496
23:00:00.709 (inside MultiCharts) - order is generated and sent to the trading plugin
23:00:00.726 (MultiCharts to API): command to send the order to the API. Place Order (Buy stop-limit order Quantity = 1, StopPrice = 1354.700000, LimitPrice = 1355.500000)
23:00:00.738 (from API to MultiCharts): Queued
23:00:00.747 (from API): Sent
23:00:01.206 (from API): Working

No further status updates for this order in the logs.


Order #6012498
23:00:01.495 (MC) - order is generated and sent to the trading plugin
23:00:01.507 (to API): Place Order (Buy stop order Lots: 1 ; Stop Price: 1340.9)
23:00:01.511 (from API): Queued
23:00:01.520 (from API): Sent
23:00:02.004 (from API): Working (Lots: 1 ; Stop Price: 1340.9)

08:45:12.752 (to API): Order modify
08:45:12.783 (from API): AmendPending (Lots: 1 ; Stop Price: 1339.3)
08:45:13.080 (from API): Working (Lots: 1 ; Stop Price: 1339.3)

09:15:03.297 (to API): Order modify
09:15:03.328 (from API): AmendPending (Lots: 1 ; Stop Price: 1338.3)
09:15:03.593 (from API): Working (Lots: 1 ; Stop Price: 1338.3)

Order has become a Limit order (no commands from MultiCharts to change the order type, possibly order type changed by broker or via broker platform) and got executed at once.

13:16:07.403 (from API): Working (Lots: 1 ; Limit Price: 1343.3)
13:16:07.403 (from API): Filled (Lots: 1 ; Limit Price: 1343.3 ; FillPrice: 1338.4)


Order #6012499
23:00:01.495 (MC) - order is generated and sent to the trading plugin
23:00:01.510 (to API): Place Order (Buy Limit order Lots: 1 ; Price: 1310.4)
23:00:01.515 (from API): Queued
23:00:01.525 (from API): Sent
23:00:02.012 (from API): Working

13:16:07.497 (to API): Cancel order
13:16:07.497 (from API): CancelPending
13:16:07.778 (from API): Cancelled


As you can see it takes milliseconds to get the order generated in MultiCharts and sent to the broker API.

Then it takes about half a second for the broker API to get the order active at broker/exchange.

After the order is active at broker/exchange – the broker is in charge of order execution price and time.

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Latency. Please share experiences.

Postby MAtricks » 04 Apr 2014

Why does it only take micro seconds to generate and send the command to cancel the order but it takes several milliseconds to generate an order?

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby wilkinsw » 04 Apr 2014

A schematic of some kind would still be of value, at least to me.

order #6012496 and order #6012498 in the above logs were for the same instrument and generated from two different charts. So why 0.7 seconds difference in sending the order to the plugin? Both had the same event trigger their calculations (first tick of the open) and both were identical strategies (bar small differences in a few variables).

Is this the API plugin order queuing that you described to me before?

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Latency. Please share experiences.

Postby MAtricks » 04 Apr 2014

A schematic of some kind would still be of value
+1

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby wilkinsw » 04 Apr 2014

Why does it only take micro seconds to generate and send the command to cancel the order but it takes several milliseconds to generate an order?
Interesting point, that I missed.

tony
Posts: 420
Joined: 14 Jun 2013
Has thanked: 30 times
Been thanked: 81 times
Contact:

Re: Latency. Please share experiences.

Postby tony » 04 Apr 2014

I had been running a Mac (fairly new model) with VMWare to run MC. My machine slowed with time and I had one WS open with 3 data series, that's it. A few months ago I purchased a PC (old school, CPU on the floor) with 12 GB RAM and a pretty fast processor. To say the PC is faster is an understatement. Night and day. If you are serious about your trading and the time you likely spending developing on MC then invest in a dedicated and fast machine.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby wilkinsw » 05 Apr 2014

If you are serious about your trading and the time you likely spending developing on MC then invest in a dedicated and fast machine.
Agreed! MC does require a boat load of performance to run smoothly for the heavy user. As I've noticed since moving onto a server in our office.

The company I work for spent £6000 on this brand new monster of a server. The IT guy has gradually ramped up my allocation of cores and the handling of MC significantly changed with each core added. Having said that I still am having some issues with the Pats plugin (but I think that's another story).

My setup now:

Intel Xeon CPU E5-2620 v2 2.1Ghz.... 8 cores on my VM.
8GB Ram
Windows Server 2012 R2.

tony
Posts: 420
Joined: 14 Jun 2013
Has thanked: 30 times
Been thanked: 81 times
Contact:

Re: Latency. Please share experiences.

Postby tony » 05 Apr 2014

I have not used Pats, only IB in testing (and will never touch them for anything in the future). My client is using TT and I must say that it is working beautifully. Getting off subject but after spending nearly nine months in development and then going "live" three surprises popped out at me that you never see in development, (1) clock syncing, (2) broker connection and (3) data providers. If you are testing and in development, I strongly suggest you do (1) daily (either manually or edit the registry to do daily, currently weekly), (2) and (3) research now before going live whether your own trading or for a client.

I run IQ at my home and my client runs TT. Clocks are synced daily for each of us and yet the results still vary at times. Sometimes pretty significantly. If I had known this prior, I would have pushed the client to run IQ and not TT. There are a lot of components to MC beyond the machine. I never realized just how much in development.

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Latency. Please share experiences.

Postby MAtricks » 05 Apr 2014

Lots more to come as well :) This industry has monthly surprises regardless how long we've been in it.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby wilkinsw » 05 Apr 2014

Clocks are synced daily
Will look at this.
My client is using TT and I must say that it is working beautifully.
That's good to know. Thanks.
Clocks are synced daily for each of us and yet the results still vary at times. Sometimes pretty significantly.
What results vary?
There are a lot of components to MC beyond the machine.
Would love some more info around this from you, Henry or anyone who cares to share. I've still got very significant latency beyond the hop across the Atlantic from London. Want to know precisely why. Not with the aim of slinging mud or trying to show anyone up, just so I know what I'm dealing with. For the record I think MC is a great value for money product.

tony
Posts: 420
Joined: 14 Jun 2013
Has thanked: 30 times
Been thanked: 81 times
Contact:

Re: Latency. Please share experiences.

Postby tony » 05 Apr 2014

I'll finish my two cents by saying I love MC as a product (and their support group). They have done and continue to do a fantastic job. Many of these elements I speak of are in a way beyond their control. Data providers for example. There are how many exchanges these days? Lot of data that needs to be fed to our machine and who you pick can vary by speed and filtering. I tip my hat to the MC crew. I've been running the software for about a year now and it is very stable. And they work with so many data providers, brokers, etc. Glad I chose them versus NT or TS for example. Including being able to send a file in .sef format and not be concerned about my code be decompiled. They really thought pretty much everything through. Now I'm way off topic and will stop yapping.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Latency. Please share experiences.

Postby Henry MultiСharts » 07 Apr 2014

Why does it only take micro seconds to generate and send the command to cancel the order but it takes several milliseconds to generate an order?
Please specify the particular lines you are referring to.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Latency. Please share experiences.

Postby Henry MultiСharts » 07 Apr 2014

A schematic of some kind would still be of value, at least to me.

order #6012496 and order #6012498 in the above logs were for the same instrument and generated from two different charts. So why 0.7 seconds difference in sending the order to the plugin? Both had the same event trigger their calculations (first tick of the open) and both were identical strategies (bar small differences in a few variables).

Is this the API plugin order queuing that you described to me before?
That is correct.
One trading plugin can process 1 order per time. It means all orders that are generated by the strategy are put into a queue and sent one by one to the API. Once one order is sent, the trading engine starts sending the next order. That is not technically possible to send two orders simultaneously to one API.
If there are two APIs (two different brokers) – then two orders can be sent almost at the same time.
In order to perform such high speed calculations you need a CPU with 4 cores or more.
Each chart with a strategy will get one flow.
Each trading instance will get one flow.
4 cores can get 1 flow each and supply parallel calculation.
But if you have a single broker profile then the amount of cores you have will not help to process the orders faster.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby wilkinsw » 10 Apr 2014

MAtricks wrote:
Why does it only take micro seconds to generate and send the command to cancel the order but it takes several milliseconds to generate an order?
Please specify the particular lines you are referring to.
I don't fully understand what MAtricks is referring to (I don't see the difference to the extent he's stating) but it is still interesting that there is a difference of a few milliseconds:
Order #6012499
23:00:01.495 (MC) - order is generated and sent to the trading plugin
23:00:01.510 (to API): Place Order (Buy Limit order Lots: 1 ; Price: 1310.4)
23:00:01.515 (from API): Queued
23:00:01.525 (from API): Sent
23:00:02.012 (from API): Working

13:16:07.497 (to API): Cancel order
13:16:07.497 (from API): CancelPending
13:16:07.778 (from API): Cancelled
To send cancel and confirm cancelled = 13:16:07.778 - 13:16:07.497 = 281 milliseconds
To place order and confirm working = 23:00:02.012 - 23:00:01.510 = 502 milliseconds....twice as long
To send modify and confirm modified is also about 300 milliseconds.

Therefore, correct me if I'm wrong:

As the place orders were all sent at the open and as they took significantly longer than the modify/cancel orders (which took place at varying times of non significance)....one can isolate that submitting orders at demanding times will cost me (using Patsystems) an extra 2/10s of a second. Just because of the extra load on Patsystems at this time.

Wow that's poor latency from Pats (unless I've missed something)! Even low load times when instruction were submitted still take 2/10s of a second anyway just because of that annoying distance between the UK and the US!

But what is particularly scary is that the Pats broker plugin queue costed me (in the logs above) approx 7/10s of a second!! In the case study where I missed the CL fill I had almost all my charts and instruments firing orders at the opening tick. Therefore I think the plugin queue cost me far more than 7/10 of a second as in the lower load open in the logs above. Again, the high load of the open on my PC probably exacerbated this. What I need to do is see what this figure is like for when two modify/cancel orders are submitted at the same time intraday.

Ok, so I feel like I'm much closer to fully understanding where latency lies.

Imagining I'm colocated: Latency, using one PATS API account is:

high load (session opening tick): Patsystems API: approx 0.3 seconds.....MC: approx 0.7+ seconds (virtually all of which is from the broker plugin queue). = approx 0.9+ seconds!

low load: Patsystems API: approx 0.1 seconds or less......MC: approx a few millionths of a seconds AS LONG AS no simultaneous orders are sent..in which case can be up to 0.7 seconds.
=0.1 seconds or less

Conclusions: See previous conclusions (GTC/D orders, computer power, server location etc), but also:

Use multiple broker accounts. If my assessment is correct and the high load example cost an extra 0.2 seconds just from the Pats API then maybe look at other brokers (but Pats is very cheap)?
Last edited by wilkinsw on 10 Apr 2014, edited 2 times in total.

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Latency. Please share experiences.

Postby MAtricks » 10 Apr 2014

Code: Select all

13:16:07.497 (to API): Cancel order
13:16:07.497 (from API): CancelPending
There's no millisecond change in the time stamp. That means that this was faster than a millisecond.

This cancel order is unique to that according to the few examples given.


Location is everything.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby wilkinsw » 10 Apr 2014

That is not technically possible to send two orders simultaneously to one API.
If there are two APIs (two different brokers) – then two orders can be sent almost at the same time.
In order to perform such high speed calculations you need a CPU with 4 cores or more.
Each chart with a strategy will get one flow.
Each trading instance will get one flow.
4 cores can get 1 flow each and supply parallel calculation.
But if you have a single broker profile then the amount of cores you have will not help to process the orders faster.
Can I use two different Pats API accounts? Or does it have to be two different API target servers too?

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Latency. Please share experiences.

Postby Henry MultiСharts » 17 Apr 2014

That is not technically possible to send two orders simultaneously to one API.
If there are two APIs (two different brokers) – then two orders can be sent almost at the same time.
In order to perform such high speed calculations you need a CPU with 4 cores or more.
Each chart with a strategy will get one flow.
Each trading instance will get one flow.
4 cores can get 1 flow each and supply parallel calculation.
But if you have a single broker profile then the amount of cores you have will not help to process the orders faster.
Can I use two different Pats API accounts? Or does it have to be two different API target servers too?
You need to have two PATS logins/two separate PATS broker profiles - this should bring up two API instances.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby wilkinsw » 17 Apr 2014

Ok great. I'll test this solution with another Pats account soon. Based on the findings above it should have a massive effect on reducing latency. If I you use one broker account profile (Api instance) per core I'll be getting close to halving the latency I currently experience.

This is something that anyone trading from a large number of charts should consider IMHO.

User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 30 times

Re: Latency. Please share experiences.

Postby PatrickSocal » 17 Oct 2014

Hi,

Thanks for that. I know I must be doing something wrong:

When I add your indicator using the ADX example in the code......It takes a while to load (that'll be the loop) but then all I get in the output is:

"execTime = 0.00000"

What have I done wrong?
Hi Wil,

I just checked that calculate_ms indicator, and it does work, but I modified it a bit to be more instructive. Here's a screenshot:

Image

The system call they've used has a resolution of about 15-16 ms. So it increments gradually over the chart, as you can see in the cyan line. I'm attaching the code for this modified version.

Hope that helps...
Attachments
calculate_time_ms_2.pla
(1.46 KiB) Downloaded 410 times
calculate_time_ms.png
(76.01 KiB) Downloaded 7460 times

User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 30 times

Re: Latency. Please share experiences.

Postby PatrickSocal » 19 Oct 2014

Hi again... Just for fun, I also did a test using the "computerdatetime" function. It has much finer resolution on my machine (around 1 ms), compared to the "calculate_time_ms" function (around 15 ms). Compare the following chart to the one in my last post:

Image

I'm including the code for the modified indicator as calculate_time_ms_3.pla

You can change the input argument from 1 to 2 to see how the different timers perform on your machine. Happy times...
Attachments
computerdatetime_vs_calculate_time_ms.png
(75.95 KiB) Downloaded 7504 times
calculate_time_ms_3.pla
(4.45 KiB) Downloaded 413 times

hilbert
Posts: 224
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: Latency. Please share experiences.

Postby hilbert » 03 Dec 2014

Hi,

Thanks for that. I know I must be doing something wrong:

When I add your indicator using the ADX example in the code......It takes a while to load (that'll be the loop) but then all I get in the output is:

"execTime = 0.00000"

What have I done wrong?
Agree, its a little confusing.
It gives the result as 0.00000 for calculation over a single bar. Resolution of timer is around 15 ms, so it will show 0.00000 if time taken to calculate ADX (or any other indicator or your strategy) on one bar is < 15ms. This will be true for almost all the cases.

What you should do to get a useful value is to modify the first line as follows:

Code: Select all

if barnumber = 1 then value1 = calculate_ms;
This will give you the execution time of ADX for the whole chart, instead of just over a single bar. Also, suggest you to make number of iterations 1000 instead of 10000. Otherwise, it will take a very long time to compute.

hilbert
Posts: 224
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: Latency. Please share experiences.

Postby hilbert » 03 Dec 2014

Hi again... Just for fun, I also did a test using the "computerdatetime" function. It has much finer resolution on my machine (around 1 ms), compared to the "calculate_time_ms" function (around 15 ms).
Hello PatrickSocial, I think you should modify your code as follows:

Code: Select all

//start_ms = MillisecondsFromDateTime(computerdatetime)
start_ms = MillisecondsFromDateTime(computerdatetime)+secondsFromDateTime(computerdatetime)*1000
and

Code: Select all

//value2 = MillisecondsFromDateTime(computerdatetime)
value2 = MillisecondsFromDateTime(computerdatetime)+secondsFromDateTime(computerdatetime)*1000
Also, when I used computerdatetime instead of GetTickCount (which is used in calculate_ms function), I also get 1ms resolution in the beginning. However, as I scrolled my chart forward, I saw that it suddenly changed to 10ms resolution after around 300 barnumber and then it changed to 15-16ms after around 2000 barnumber. Resolution stayed constant around 15-16ms till the last bar on chart ( I had loaded 95,000 bars on chart). Can you please confirm if computerdatetime also shows similar irregular resolution behavior on your MC also? If yes, then I think using GetTickCount would be preferable rather than using computerdatetime.

I am not sure what underlying windows call has MC used while implementing computerdatetime.

User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 30 times

Re: Latency. Please share experiences.

Postby PatrickSocal » 04 Dec 2014

Hi Hilbert,

Yes, if you want to handle rollover effects, the code should be modified to handle seconds, minutes, hours, days, etc. I was just keeping it simple. Code that handles second, minute and hour rollovers (but not days or longer) would look like this:

Code: Select all

start_ms = MillisecondsFromDateTime(computerdatetime) + 1000*SecondsFromDateTime(computerdatetime) + 60000*MinutesFromDateTime(computerdatetime) + 3600000*HoursFromDateTime(computerdatetime)
So if you start your test at 3:59:59 and it takes more than a second, it should work. But if you start it at 23:59:59 it won't. And if the test takes more than one day, it also won't... But your computer is probably faster than that :)

Actually, a better way to do it is to calculate the differences in terms of DateTime, and then compute the milliseconds from that difference. That way, you handle all rollover effects, and you can measure multi-day time differences too (in the unlikely event that you need to). My modified code looks like this:

Code: Select all

Variables:
ii(0),
start_DateTime(0), end_DateTime(0),
deltaDateTimeBar(0), deltaDateTimeChart(0),
elapsedThisBar(0), elapsedThisChart(0),
maxElapsedPerBar(0), myCount(0), mySum(0);

Once start_DateTime = computerdatetime;

For ii=0 To 10000 Begin // Do something slow
value3=ADX(140);
End;

end_DateTime = computerdatetime;

if end_DateTime[1] > 0 then begin
deltaDateTimeBar = end_DateTime - end_DateTime[1];
deltaDateTimeChart = end_DateTime - start_DateTime;
elapsedThisBar = MillisecondsFromDateTime(deltaDateTimeBar) +
1000*SecondsFromDateTime(deltaDateTimeBar) +
60000*MinutesFromDateTime(deltaDateTimeBar) +
3600000*HoursFromDateTime(deltaDateTimeBar) +
86400000*floor(deltaDateTimeBar);

elapsedThisChart = MillisecondsFromDateTime(deltaDateTimeChart) +
1000*SecondsFromDateTime(deltaDateTimeChart) +
60000*MinutesFromDateTime(deltaDateTimeChart) +
3600000*HoursFromDateTime(deltaDateTimeChart) +
86400000*floor(deltaDateTimeChart);

maxElapsedPerBar = maxlist(maxElapsedPerBar, elapsedThisBar);
myCount += 1;
mySum += elapsedThisBar;

plot1( value3, "ADX");
Plot2(elapsedThisBar, "Elapsed This Bar");
Plot3(elapsedThisChart, "Elapsed This Chart");
Plot4(maxElapsedPerBar, "Max Elapsed Per Bar");
Plot5(mySum / myCount, "Average Elapsed Per Bar");
end;
Regarding the more tricky question of irregular resolution behavior... I haven't seen it on my machines. As you see in the above code, I computed a max-value on the time elapsed per bar. I tried it on a chart with about 65,000 bars, and it was never more than 3 ms on my machine. Average time per bar was about 0.6 ms. This was using a Lenovo W530 with a 4-core i7-3820QM processor with hyperthreading turned off, running Win7. In fact, I just tried again on a chart with 350k bars, and again, the resolution was consistent. Now it got a max of 5 ms, but still an average of 0.6 ms. Here's a screenshot:

Image

What kind of hardware are you using to get your results? If you have access to another machine, perhaps it would be worth trying it there? Also, I'm attaching my indicator in case you want to try it.

Hope this helps...
Attachments
calculate_time_ms_4.pla
(6.41 KiB) Downloaded 400 times
calculate_time_ms_4.png
(44.67 KiB) Downloaded 7359 times

User avatar
virginiatrader
Posts: 79
Joined: 05 May 2007
Location: Virginia
Has thanked: 5 times
Been thanked: 5 times

Re: Latency. Please share experiences.

Postby virginiatrader » 05 Dec 2014

wilkinsw, and to all who have left comments and guidance...

I read the entire string from beginning to end, and I am always thankful that I am an MC user and member of the forum. I never fail to learn something new each time I am here.

While I have traded in real-time off MC 15 minute charts of the E-Mini S&P500, I didn't do it with MC as an automated platform. I am primarily an end-of-day futures trader, using MC to chart and analyze my markets, and have coded very simple trend-following systems using PowerLanguage.

I monitor my markets and charts daily, and should I miss entering or exiting a trade, or should MC fail to "fire" a signal when I expect it to (yes, it has happened), my trading plan allows me to enter or exit on a discretionary basis, so while I may be late to the trade, i do not miss it entirely.

wilkinsw, and everyone, I am curious: if you miss your fill, why not enter the order manually? While "going retail" with a market order is not always preferable, it does get you into the trade. Or, as an alternative, use a "marketable limit" order, i.e., hit the bid or lift the offer and then follow the trade according to your template.

Just wondering...

virginiatrader

Jad
Posts: 92
Joined: 15 Jun 2014
Has thanked: 13 times
Been thanked: 21 times

Re: Latency. Please share experiences.

Postby Jad » 07 Dec 2014

I don't know if this is the 'main' cause of the lag being reported but you should be aware that a 1-tick lag exists, by design, when submitting 'Next Bar' orders with IOG-ON. The resultant lag is difficult to spot during RTH as there are so many ticks but the potential for adverse consequences exists regardless of time-of-day.

e.g.
The following example assumes a downward moving market and the placing of a trailing Buy Stop. An order has already been sent to Buy Next Bar @ 100 on a Stop and is visible on the book.

So, a Buy Stop is @ 100

Tick 1 - Market moves down a tick so an order is generated in the MC script to move the Stop to 99.
However, the order at the broker remains @ 100 even though the script requested it to be moved to 99.

Tick 2 - The order at the broker changes to 99 - Correct but one tick after the order was first generated.

Tick 3 - Market moves down again so an order is generated in the MC script to move the Stop to 98 ...
The order at the broker remains @ 99 even though the script requested it to be moved to 98.

Tick 4 - The order at the broker changes to 98 - Correct but one tick after the order was first generated.

etc. etc.

Any delays risk fills being missed but it also runs the risk of orders being filled when they shouldn't be. Both can result in significant losses either due to missing a good move or being stopped out too early while in a move.

Of course, there is always going to be some lag between issuing an order and that order reaching the exchange but this is a case where it is MC itself that is adding to that lag by also sending these orders one tick after the fact. I consider this to be a bug/flaw – not in the code (that’s doing what it was designed to do) but in the design itself. I can't think of a single reason why such a design would be anything but a liability and I see no potential benefit. Other ATM platforms don't have this lag (so it's avoidable) and I can't think any reason why someone would introduce it into their design if say, they were writing directly to the broker's API. If I've missed a really good reason, I'd love to hear it.

I think it's great that MC is continually seeking ways to improve performance but it seems strange to continue with a design that is deliberately (and unnecessarily IMO) inserting lag.

gerler
Posts: 7
Joined: 11 Jun 2014
Has thanked: 1 time
Been thanked: 2 times

Re: Latency. Please share experiences.

Postby gerler » 08 Dec 2014

I've written a java program that places an order with IB via TWS-API.
I've used the same logic in an MC-indicator and compared the execution times

Multicharts was always approx. 100ms slower to place the same order than my java program. C++ would probably be even faster

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby orion » 08 Dec 2014

gerler, can you give some more color on what was measured. Specifically, the time being 100ms longer: what events bracket the time interval in both the MC code and your Java code?

TA100
Posts: 54
Joined: 10 Feb 2014
Has thanked: 39 times
Been thanked: 9 times

Re: Latency. Please share experiences.

Postby TA100 » 08 Dec 2014

I've written a java program that places an order with IB via TWS-API.
I've used the same logic in an MC-indicator and compared the execution times

Multicharts was always approx. 100ms slower to place the same order than my java program. C++ would probably be even faster
Hi Gerler that is pretty amazing time saving - any chance you can share your methodology with either MC so they could develop into next release or alternatively could you offer some configurable add on for broker API's too?

Thanks.

Jad
Posts: 92
Joined: 15 Jun 2014
Has thanked: 13 times
Been thanked: 21 times

Re: Latency. Please share experiences.

Postby Jad » 08 Dec 2014

Hi Gerler that is pretty amazing time saving - any chance you can share your methodology Thanks.
I think that anyone choosing to write directly to any broker's API is going to send their order on the tick that satisfies their chosen set of conditions rather than wait to send it on the following tick - as Multicharts does with Buy/Sell Next bar orders and IOG set.

How much of a time difference is there between the two? It depends when the next tick arrives. Until it arrives, your order isn't sent. So, anywhere from sub microsecond to ????

During slow periods, I've seen orders not appear on the book for well over a second. For FIFO markets, that is always going to mean that your order will be much further down the queue than it would have been if the order had been sent immediately.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Latency. Please share experiences.

Postby TJ » 08 Dec 2014

Hi Gerler that is pretty amazing time saving - any chance you can share your methodology Thanks.
I think that anyone choosing to write directly to any broker's API is going to send their order on the tick that satisfies their chosen set of conditions rather than wait to send it on the following tick - as Multicharts does with Buy/Sell Next bar orders and IOG set.

How much of a time difference is there between the two? It depends when the next tick arrives. Until it arrives, your order isn't sent. So, anywhere from sub microsecond to ????

During slow periods, I've seen orders not appear on the book for well over a second. For FIFO markets, that is always going to mean that your order will be much further down the queue than it would have been if the order had been sent immediately.
For slow markets, you should always deploy RecalcLastBarAfter.

As a matter of fact, for prudent trade management,
one should always deploy RecalcLastBarAfter.

Jad
Posts: 92
Joined: 15 Jun 2014
Has thanked: 13 times
Been thanked: 21 times

Re: Latency. Please share experiences.

Postby Jad » 08 Dec 2014

For slow markets, you should always deploy RecalcLastBarAfter.

As a matter of fact, for prudent trade management,
one should always deploy RecalcLastBarAfter.
All that does is initialize the calculation after expiration of the timeout and force recalculation. It doesn't ensure that the order will be sent on the tick it was generated.

For 'really' prudent trade management, all orders should be sent to the broker immediately on the tick that they were generated.

User avatar
JoshM
Posts: 2195
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1544 times
Been thanked: 1565 times
Contact:

Re: Latency. Please share experiences.

Postby JoshM » 10 Dec 2014

All that does is initialize the calculation after expiration of the timeout and force recalculation. It doesn't ensure that the order will be sent on the tick it was generated.

For 'really' prudent trade management, all orders should be sent to the broker immediately on the tick that they were generated.
I don't think you can submit an order on the tick it was generated, since that tick will trigger the script calculation process and so will always be a 'historical' tick by the time the order is generated by the code.

With IOG you can send orders for the next tick of which they are triggered on. And for slow markets you can indeed add `RecalcLastBarAfter()`, which will send the order on the next script calculation (so no need to wait for a new tick).

Jad
Posts: 92
Joined: 15 Jun 2014
Has thanked: 13 times
Been thanked: 21 times

Re: Latency. Please share experiences.

Postby Jad » 10 Dec 2014

I don't think you can submit an order on the tick it was generated, since that tick will trigger the script calculation process and so will always be a 'historical' tick by the time the order is generated by the code.
Thanks JoshM, I think you are correct but that restriction is ONLY due to MC's built-in design of 'Buy Next Bar' which in effect becomes 'Buy Next Tick' when IOG is on.

Regarding the historical nature of ticks: The second tick also has to be received before the order generated by the first tick is sent. So in that sense, it too is 'historical' by the time it is received.

Receiving a tick is just like any other 'event' that triggers an action (e.g. Print, Write to a File, Send an SMS etc. Those processes don't have to wait for the next tick to be actioned).

I see no reason why the same tick satisfying a set of trade condition(s) can't also call the broker's API to place a trade without having to wait for another tick. Why design a process flow that has to wait for the next tick when it isn't necessary? It kinda violates the K.I.S.S. principle that is often preached.

e.g. If I create a dll that places trades via the broker's API, and call that instead of MC's 'Buy Next bar', that dll will immediately place the order on the same tick - No?

If I can do that, then Multicharts could do that. It may require the addition of a new instruction e.g. something like: 'Buy/Sell This Bar @ 'xxxx.xx Price' with IOG on - to fit in with the Multicharts process but it isn't impossible.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby orion » 10 Dec 2014

I see no reason why the same tick satisfying a set of trade condition(s) can't also call the broker's API to place a trade without having to wait for another tick. Why design a process flow that has to wait for the next tick when it isn't necessary? It kinda violates the K.I.S.S. principle that is often preached.

e.g. If I create a dll that places trades via the broker's API, and call that instead of MC's 'Buy Next bar', that dll will immediately place the order on the same tick - No?
I am not privy to the internals of MC event handling architecture but I suspect that compatibility with large legacy base of EL was perhaps a consideration. I find MC to be fairly well thought out and well designed. In any case, as you say, it is quite easy to insert your own DLL and call the broker's API if this is a high priority for your strategy. Note that not all strategies care about acting on each tick.

Jad
Posts: 92
Joined: 15 Jun 2014
Has thanked: 13 times
Been thanked: 21 times

Re: Latency. Please share experiences.

Postby Jad » 10 Dec 2014

I am not privy to the internals of MC event handling architecture but I suspect that compatibility with large legacy base of EL was perhaps a consideration.
I think you're correct in saying that this results from an old architecture that was originally conceived out of necessity and to overcome limitations a long time ago.
e.g. EL pre-IntrabarPersist comes to mind as a possible reason for a now, very inefficient design that requires repeated sending of the same order on every tick but there may be other, legacy related reasons.
I find MC to be fairly well thought out and well designed.
To be fair to MC, I don't think the design of the (at least) one tick lag and order 'loop' where it is mandatory to resend the same order(s) on every single tick is MC's. It was Omega Research's and MC would have had to follow suit if it wanted to be compatible with TS. It may well have been well thought out when it was first conceived, as a workaround to various constraints - but it isn't necessary now and it is an incredibly inefficient and dated process design. So to still describe such an inefficient process flow as 'well designed' is extremely generous IMO.
In any case, as you say, it is quite easy to insert your own DLL and call the broker's API if this is a high priority for your strategy. Note that not all strategies care about acting on each tick.
Yes, it should be relatively easy to insert a dll that only places an order once and then only resends an order to modify if/when conditions change - even more so for MC as position monitoring is already integrated - which is why it is strange that they haven't done it.
Note that not all strategies care about acting on each tick.
That is correct but every strategy will be impacted to some extent regarding the queue position of its orders that result from them being held back by (at least) one tick.
As I'm sure you know, a lot of orders can be sent in one tick and that could represent 100's of positions lower in the queue - even if your datafeed provides your script with every tick.
Plus, it has knock-on effects which result in other problems which would otherwise not occur - but that's a whole other story.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby orion » 10 Dec 2014

It may well have been well thought out when it was first conceived, as a workaround to various constraints - but it isn't necessary now and it is an incredibly inefficient and dated process design. So to still describe such an inefficient process flow as 'well designed' is extremely generous IMO.
The flow is what it is perhaps because of legacy compatibility. I am not referring to the flow and semantics of EL/PL as well designed. I am referring to MC platform design which gives you a broker neutral and datafeed neutral platform that allows portfolio trading for a large symbol universe. MC does all that while maintaining compatibility with EL. My benchmarks on MC and TS show that MC executes PL code 5-10X faster than TS executes identical code. That says something about the quality of the design.
Every strategy will be impacted to some extent regarding the queue position of its orders that result from them being held back by (at least) one tick. As I'm sure you know, a lot of orders can be sent in one tick and that could represent 100's of positions lower in the queue - even if your datafeed provides your script with every tick. Plus, it has knock-on effects which result in other problems which would otherwise not occur - but that's a whole other story.
I don't agree with this. I know of plenty of strategies where even 500ms latency does not matter; leave alone a single tick. That is not to say that there aren't strategies where latency is critical. By the way, I implemented a standalone IB client in C++ to measure latency and it was over 500ms between dispatch of the order in my code and receipt of the execution report from IB. That is not including the latency in receiving the data for placing the order.

Jad
Posts: 92
Joined: 15 Jun 2014
Has thanked: 13 times
Been thanked: 21 times

Re: Latency. Please share experiences.

Postby Jad » 11 Dec 2014

The flow is what it is perhaps because of legacy compatibility.
Yes, I think we already agreed on that but we no longer live in the world that made those legacy inefficiencies necessary.
I am not referring to the flow and semantics of EL/PL as well designed.
You may not be but I am. The topic of the thread is 'Latency' - not broker and datafeed compatibility (which, BTW I also think is a fantastic feature in MC). I'm just addressing the original question and pointing out that there is a latency that has been deliberately built-in by design.
My benchmarks on MC and TS show that MC executes PL code 5-10X faster than TS executes identical code. That says something about the quality of the design.
I've never conducted such a test so I'll have to take your word for it - but 5-10X faster?? Even if that's correct, maybe it just says more about the even worse quality of the TS design.
That kind of benchmarking, which focuses on speed of code execution, only tests the ability of your machine and the software on it to crunch that code. It does not address 'latency' of order submission which is what this thread was discussing.
I know of plenty of strategies where even 500ms latency does not matter; leave alone a single tick.
Yes, so do I. There are many strategies that are profitable regardless of latency - so I agree if you mean: 'does not matter' in the sense of: 'still profitable'. However, unless you've never missed a move because your Limit order wasn't filled, how could you possibly know how many orders would have been filled if you had joined the queue one (or more) ticks ago - and then quantify that impact on the strategy's profitability over the course of a year?

I wouldn't underestimate the value of a tick. e.g. If a one tick delay was universally enforced, the entire HFT industry would shut down overnight.
But you don't have to be involved with HFT to be impacted by inefficient design. All large moves begin with a single tick. You're either on those moves or not. I see no reason to increase the likelihoood of missing those moves (and/or optimal exits) due to a legacy design when it is avoidable.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby orion » 11 Dec 2014

I wouldn't underestimate the value of a tick. e.g. If a one tick delay was universally enforced, the entire HFT industry would shut down overnight. But you don't have to be involved with HFT to be impacted by inefficient design. All large moves begin with a single tick.
In my assessment, there are too many other things missing with my platform (MC), my datafeed, and my broker for this this single tick delay to matter. But since you seem to believe otherwise, perhaps you can do a forward test of your strategies running inside MC. You will need two MC licenses running side by side with one placing orders with MC and thus incurring the tick delay and the other bypassing the MC tick delay and placing orders via your DLL. You will need a minimum of three months of such forward testing to give you a sound statistical basis for such a belief. When done with the experiment, please post your results on this forum.

gerler
Posts: 7
Joined: 11 Jun 2014
Has thanked: 1 time
Been thanked: 2 times

Re: Latency. Please share experiences.

Postby gerler » 11 Dec 2014

I've placed the order from MC the exact moment when a tick came in that satisfied my conditions. I've achieved this with 2 different methods:
1. by using a 1 tick chart
2. by placing the order via a custom dll
Hi Gerler that is pretty amazing time saving - any chance you can share your methodology Thanks.
I think that anyone choosing to write directly to any broker's API is going to send their order on the tick that satisfies their chosen set of conditions rather than wait to send it on the following tick - as Multicharts does with Buy/Sell Next bar orders and IOG set.

How much of a time difference is there between the two? It depends when the next tick arrives. Until it arrives, your order isn't sent. So, anywhere from sub microsecond to ????

During slow periods, I've seen orders not appear on the book for well over a second. For FIFO markets, that is always going to mean that your order will be much further down the queue than it would have been if the order had been sent immediately.
For slow markets, you should always deploy RecalcLastBarAfter.

As a matter of fact, for prudent trade management,
one should always deploy RecalcLastBarAfter.

gerler
Posts: 7
Joined: 11 Jun 2014
Has thanked: 1 time
Been thanked: 2 times

Re: Latency. Please share experiences.

Postby gerler » 11 Dec 2014

I think there is nothing MC can do to speed up things because it is a complex software whereas my java program only had to deal with the trading logic and order execution.
I believe powerlanguage is interpreted so that alone slows down the execution. Then there is the separate IB module and quotemanager which require interprocess communication and probably a lot of other features I'm not aware of.
Hi Gerler that is pretty amazing time saving - any chance you can share your methodology Thanks.
I think that anyone choosing to write directly to any broker's API is going to send their order on the tick that satisfies their chosen set of conditions rather than wait to send it on the following tick - as Multicharts does with Buy/Sell Next bar orders and IOG set.

How much of a time difference is there between the two? It depends when the next tick arrives. Until it arrives, your order isn't sent. So, anywhere from sub microsecond to ????

During slow periods, I've seen orders not appear on the book for well over a second. For FIFO markets, that is always going to mean that your order will be much further down the queue than it would have been if the order had been sent immediately.

Jad
Posts: 92
Joined: 15 Jun 2014
Has thanked: 13 times
Been thanked: 21 times

Re: Latency. Please share experiences.

Postby Jad » 12 Dec 2014

In my assessment, there are too many other things missing with my platform (MC), my datafeed, and my broker for this this single tick delay to matter.
If such small, millisecond delays don't matter to you, then why bother developing software to measure millisecond latency?
"I implemented a standalone IB client in C++ to measure latency and it was over 500ms"

FYI, the latency caused by that single tick is variable but can be more than 500ms. But this isn't about 'you' or your set up. The original poster was asking about latency. As I've said repeatedly, I'm simply pointing out that there is a latency built-in to the design that no longer 'has' to be there.

Obviously he/she thought this topic was important enough to have requested feedback. He/she is not alone. Such matters seem to be very important to many - and (when it suits your argument) that includes you.

The forum is full of posts where people discuss multi-core rigs, benchmarking, tweaking and testing to improve perfomance - apparently all trying to improve performance measured in milliseconds. If it "doesn't matter" to you, I'm very happy for you but then I don't why you're even bothering to comment (or develop measurement software to measure millisecond latency??)
But since you seem to believe otherwise, perhaps you can do a forward test of your strategies running inside MC. You will need two MC licenses running side by side with one placing orders with MC and thus incurring the tick delay and the other bypassing the MC tick delay and placing orders via your DLL. You will need a minimum of three months of such forward testing to give you a sound statistical basis for such a belief. When done with the experiment, please post your results on this forum.
If you think such tests are necesary to prove that there is a cost in sending an order later than it could be sent, then you haven't understood the problem. MC acknowledges that the one tick delay exists. So, 'whether' there is an impact should not be in question. Determining the exact $ impact is not possible because you can't know how many times it results in a queue position that missed an entry/exit vs. the times it would have been filled if sent sooner.
But there is no doubt that those missed fills will happen. Only the $ impact will be variable by individual.

e.g. Someone placing 100 trades a day is going to be affected more than someone trading once a year. The delay (and queue positions) will be even worse for those who have a datafeed that doesn't transmit every tick etc.
Last edited by Jad on 12 Dec 2014, edited 1 time in total.

Jad
Posts: 92
Joined: 15 Jun 2014
Has thanked: 13 times
Been thanked: 21 times

Re: Latency. Please share experiences.

Postby Jad » 12 Dec 2014

I believe powerlanguage is interpreted so that alone slows down the execution.
The latency caused by sending orders 1 tick later than they could be is not due to whether PL is a interpreted language.
I've placed the order from MC the exact moment when a tick came in that satisfied my conditions. I've achieved this with 2 different methods: 1. by using a 1 tick chart 2. by placing the order via a custom dll
1. I don't trade from the chart so I don't know if an order sent from an MC chart is sent immediately but I think(hope) it would be like any other manual method (DOM etc.) and send the order immediately.

2. Very good, That's definitely one way around the issue but managing the log in, position status, order rejections etc. won't be automatic when MC is bypassed.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Latency. Please share experiences.

Postby TJ » 12 Dec 2014

I believe powerlanguage is interpreted so that alone slows down the execution.
The latency caused by sending orders 1 tick later than they could be is not due to whether PL is a interpreted language.
::
The PowerLanguage indicators/strategies in MultiCharts are compiled. You can see the difference in execution speed when compared to other platforms.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby orion » 12 Dec 2014

If you think such tests are necesary to prove that there is a cost in sending an order later than it could be sent, then you haven't understood the problem. MC acknowledges that the one tick delay exists. So, 'whether' there is an impact should not be in question. Determining the exact $ impact is not possible because you can't know how many times it results in a queue position that missed an entry/exit vs. the times it would have been filled if sent sooner.
I understand the problem a lot better than you think I do. If you are up to the task, it is also quite easy to do a statistical validation of the material impact of the one tick delay matters as per the controlled experiment I outlined. You should the statistical validation instead of relying on folklore since that one tick delay is such a big deal for you. The results will be illuminating.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Latency. Please share experiences.

Postby TJ » 12 Dec 2014

You can make an improvement proposal to MultiCharts:

https://www.multicharts.com/pm/

Jad
Posts: 92
Joined: 15 Jun 2014
Has thanked: 13 times
Been thanked: 21 times

Re: Latency. Please share experiences.

Postby Jad » 12 Dec 2014

I understand the problem a lot better than you think I do.
Obviously not. I doubt even you know what you think about this subject. You seem to find it very difficult to maintain any consistency of thought about it.

On the one hand you accept that latency is important to some people and to some strategies. Then, in the very same thread (started by someone for whom the subject obviously is important), you feel compelled to announce that a latency, built-in to MC, "doesn't matter" to you.

How does its unimportance to you have any bearing on other strategies.......

....... that you've already acknowledged might be affected by it?
e.g.
That is not to say that there aren't strategies where latency is critical.
Your flip-flopping continues as you say that 500ms+ latency "doesn't matter" to you yet you feel compelled to regale us with unsolicited boasts about writing software specifically designed to expose sub-second latency. Why did you bother if such things don't matter to you?

Dude! You're all over the place!

I have to say that I am puzzled by your aggressive response to my mere reporting of a fact. It doesn't matter to you. I get it. Honestly, I really do - and thank you for drumming into us all exactly how much it doesn't matter to you and how all-knowing you are about how it should also be unimportant to everyone else - but you seem to be taking this very personally.

To recap and drag this back on point. My original post was only to pass on information about which I have only recently been made aware: i.e. that MC holds on to orders for (at least) a tick. Maybe everyone else was already aware of it and I am the only one who wasn't. If so, then I apologize for being so ignorant of its existence but, if it doesn't affect you, then why are you so keen to repeatedly inform us of its unimportance to you? It's a very strange OCD to have.

There are far more MC bugs and features reported on this forum that do not affect me personally than those which do. I am very grateful to everyone who reports them. Unfortunately, I only have time to read the ones that catch my eye in case they 'might' affect me.

In this instance, I thought I'd point out something that may not be universally known. But I don't see any point in adding comments to threads which amount to little more than expecting (almost 'demanding' in your case) that what is being reported should be ignored because "I like MC's design" and because the issue "doesn't matter" to me.

How self-absorbed and delusional about my omnipotence would I have to be to think anyone would care about my irrelevant, shill-like, love affair with MC's overall design or that some of the flaws in that design should be ignored just because they don't affect me personally?

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Latency. Please share experiences.

Postby orion » 13 Dec 2014

Readers: please ignore Jad. He has sowed enough confusion. This is my last post on this topic and its purpose is to counter the confusion. MC is not the issue given the many other issues that stand in the way of a retail trader. The latency you measure via ping tests to your datafeed or your broker is not the latency that you will experience when trading. If you want more insight into this, you can read up on queuing theory but you can perhaps relate to the experience of city rush hour traffic. Just because your destination is 5 miles away does not mean you are going to get there in 5 minutes during the rush hour. Queuing delays manifest in all areas of the infrastructure and not just in the order book. Be forewarned that chasing latency is a fool's errand for a retail trader.


Return to “MultiCharts”