Sync didn't sync lost me $26,000 USD!!!

Questions about MultiCharts and user contributed studies.
User avatar
t-rader
Posts: 139
Joined: 02 Feb 2011
Location: Australia
Has thanked: 11 times
Been thanked: 27 times

Sync didn't sync lost me $26,000 USD!!!

Postby t-rader » 26 Jan 2015

Multicharts please explain and help me!

I have Multicharts connected to lmax and set to sync but it never synced! It also doubled up my orders! On top of this I always send a stop order using SetStopLoss to be held at the broker in case something happens to my server but this was never sent!

So because Multicharts didnt sync, doubled up my orders and never sent my hard stop to the broker the trade never exited which in turn resulted in a margin call and me losing $26,000 USD!!! Now I know my server didn't lose connection to the internet because I monitor my server using remote monitoring service pulseway.com so to me this seems like a Multicharts error!

I don't want to jump to conclusions and blame Multicharts but why didn't Multicharts sync!?
Why did it double my orders!?
Why did it not send my hard stop loss to the broker!?
Why weren't my orders cancelled within 4 seconds as per the strategy properties settings?

I've been trading with Multicharts for approx 3 years now and with Lmax as my broker for the last 1.6 years. I haven't changed my server, Multicharts configuration or my code and all I do is reboot my server each week and turn autotrading on.

I really need a please explain and some help here!
Attachments
Orders-BrokerLmax.PNG
Orders-BrokerLmax
(119.84 KiB) Downloaded 3045 times
StrategyProperties.PNG
StrategyProperties
(166.87 KiB) Downloaded 3028 times
PosHistory-WhatHappened.PNG
PosHistory-WhatHappened
(443.72 KiB) Downloaded 2988 times
Orders-WhatHappened.PNG
(635.67 KiB) Downloaded 2958 times

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby orion » 26 Jan 2015

First, let me acknowledge the pain of your loss. As someone said, the joy of winning and the pain of losing are right up there with the pain of winning and the joy of losing. Now, some points that may be relevant to your case:

1. Looking at your OPT, it seems that the order status never changed from 'Sent' to 'Filled' in MC even though the position changed which can happen if the execution fills messages never came through. This thread is about how there are two channels of information from the broker to MC and the two channels can get out of sync. Essentially, MarketPosition_at_broker and MarketPosition_at_broker_for_the_strategy are updated by different channels. This is also mentioned in MC documentation under the 'unexpected mismatch' case.

2. If the above happened, that explains why your hard stop wasn't placed and so you had an exposed trade. It also explains why your strategy logic which perhaps worked off the MarketPosition or MarketPosition_at_broker_for_the_strategy fired off twice to get you into the same trade twice.

3. Why did the execution fill messages not come through even though your Internet connection was working. Internet connection working is a necessary but not sufficient condition for the messages to come through. Is it an MC problem? I would estimate MC has a few million lines of code and has it share of bugs but it is a fantastic platform nonetheless. I would first suspect the message loss to be a problem with either your broker's software or infrastructure. Having spent two decades building infrastructure, I have an appreciation for the fallibility of the underlying infrastructure and a great respect for anything that is able to work 99.9% of the time given the underlying imperfections.

4. Finally, auto trading is not the same as unattended trading. Simple alerts about Internet connection loss are not sufficient. We need to always keep an eye on the positions in both MC OPT and the broker GUI since we can all be victims of Murphy's law at any time. If something can go wrong it will.

Andrew or Henry, please feel free to correct any of my points above especially points #1 and #2.

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby MAtricks » 26 Jan 2015

Very sorry to hear this :(

While I feel terrible for you, we must get to the bottom of this so it doesn't happen again:

-How long did you leave MC unattended? Any longer than the time it takes you to sleep is a giant nono.

-When was the last time you restarted MC? Bugs happen.. restarting the platform every day is a pain in the neck, but I haven't had any bugs since I started doing this years ago

-How many contracts do you have in the Maximum Shares input?

-Do you have the MC Logs for the event?


I've run into instances where the Broker disconnects from MC which disables SA/AA with certain APIs. This has always been a fear of mine..

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby Henry MultiСharts » 27 Jan 2015

Hello t-rader,

We are very sorry to hear about your issue and we will do our best to investigate it. In case of trading such high volumes the loss in case of an issue like this can be significant. Unfortunately, now there is no way to recover the loss but I would like to reassure you that we will do all possible to find out the reason of the problem so it can be avoided in future.

MultiCharts has always recommended all users to monitor auto-trading when trading live. I understand that it may seem unnecessary, especially since you have traded successfully for more than a year with the same setup, but still it is something that you may want to consider.

We will need the following for investigation:

1) MultiCharts logs (if you have not restarted the platform. By default the logs from the previous run are erased when you restart MC).
In MultiCharts go to Help->Feedback->Send logs. Please let me know that you have uploaded the logs.
If you want to send the logs manually please follow this guide: https://www.multicharts.com/trading-sof ... harts_Logs

2) In MultiCharts go to File->New->Open Order and position tracker window-> Orders tab->make sure you are not filtering the information in columns and the dates when the orders were placed are covered, then go to File->Export to excel.
Please also export the information from the Logs tab and send it to me (support@multicharts.com) for analysis.

Again, I am very sad to hear about this issue and will provide you with any assistance possible to make sure it won’t happen in future.

User avatar
t-rader
Posts: 139
Joined: 02 Feb 2011
Location: Australia
Has thanked: 11 times
Been thanked: 27 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby t-rader » 29 Jan 2015

Thanks for the support guys, its a bit rough taking such a hit, more so when it wasn't my trading system which was the reason for the loss. It just sucks I never encountered this sync problem earlier in my trading.
Very sorry to hear this :(

While I feel terrible for you, we must get to the bottom of this so it doesn't happen again:

-How long did you leave MC unattended? Any longer than the time it takes you to sleep is a giant nono.

-When was the last time you restarted MC? Bugs happen.. restarting the platform every day is a pain in the neck, but I haven't had any bugs since I started doing this years ago

-How many contracts do you have in the Maximum Shares input?

-Do you have the MC Logs for the event?

- Maximum shares/contracts I have set to 10,000. I can trade up 2,400 on a usual trade so I can see this is a safety measure I should definitely use!

I've run into instances where the Broker disconnects from MC which disables SA/AA with certain APIs. This has always been a fear of mine..
Hello MAtricks

- Multicharts was left unattended for approx 12 hours which was evening my time.

- Server was running for 9-10 days before its last restart.

- Multicharts logs sent to Multicharts support.
Hello t-rader,
We will need the following for investigation:

1) MultiCharts logs (if you have not restarted the platform. By default the logs from the previous run are erased when you restart MC).
In MultiCharts go to Help->Feedback->Send logs. Please let me know that you have uploaded the logs.
If you want to send the logs manually please follow this guide: https://www.multicharts.com/trading-sof ... harts_Logs

2) In MultiCharts go to File->New->Open Order and position tracker window-> Orders tab->make sure you are not filtering the information in columns and the dates when the orders were placed are covered, then go to File->Export to excel.
Please also export the information from the Logs tab and send it to me (support@multicharts.com) for analysis.
Hi Henry,

I've sent the logs via Help > Feedback > Send Logs.

I've also emailed you the exported orders and logs file from the OPT.

I await your response.

Cheers,

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby Henry MultiСharts » 29 Jan 2015

Hello t-rader,

Our developers have studied the logs from your PC. From the logs we can see that there were issues with connection to LMAX servers for multiple days before the orders were placed (constant reconnects starting 21-01-2015 14:37:03 until 23-01-2015 22:07) – you can see the corresponding messages in Order and Position Tracker Logs tab: Reconnecting ...Connection established. Reconnecting ....
The orders were placed between the reconnects and no order confirmation has been received from the broker (you can see the corresponding messages in Order and Position Tracker Logs tab: Didn't receive final status for order … ). In such case MultiCharts considers the orders it has sent to be cancelled or not existing. Once order generation conditions are met in the strategy – the orders will be sent to broker.

From the logs we can also see that you are running an outdated version of MultiCharts64 8.8 Release 1 build 8363 from 12-3-2013 that had known LMAX connection loss issues. The current version of MultiCharts64 is 9.0 Release 4 build 10360 from 12-01-2014. This version has critical improvements addressing this issue and other bug fixes.

As I can see on the screenshots – you are using two LMAX broker profiles. Per LMAX Support: LMAX API can handle connection to a single LMAX login (same for data provider and broker with identical connection settings). Connecting multiple LMAX broker profiles with different accounts is claimed to be unstable and unsupported by LMAX.

In order to avoid such issue in the future I would recommend the following:

1. Update MultiCharts to the latest Release version (9.0 build 10360);
2. Configure LMAX data provider and broker with identical connection settings using the same login;
3. Avoid using more than one LMAX broker profile (may work seamless to some users);
4. Add market_position_at_broker check to your strategy to avoid generation of unnecessary orders;
5. Monitor Order and Position Tracker Logs tab for errors;
6. Do not start auto trading when you have issues with connection to the broker.

User avatar
t-rader
Posts: 139
Joined: 02 Feb 2011
Location: Australia
Has thanked: 11 times
Been thanked: 27 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby t-rader » 30 Jan 2015

Hello t-rader,

Our developers have studied the logs from your PC. From the logs we can see that there were issues with connection to LMAX servers for multiple days before the orders were placed (constant reconnects starting 21-01-2015 14:37:03 until 23-01-2015 22:07) – you can see the corresponding messages in Order and Position Tracker Logs tab: Reconnecting ...Connection established. Reconnecting ....
The orders were placed between the reconnects and no order confirmation has been received from the broker (you can see the corresponding messages in Order and Position Tracker Logs tab: Didn't receive final status for order … ). In such case MultiCharts considers the orders it has sent to be cancelled or not existing. Once order generation conditions are met in the strategy – the orders will be sent to broker.

From the logs we can also see that you are running an outdated version of MultiCharts64 8.8 Release 1 build 8363 from 12-3-2013 that had known LMAX connection loss issues. The current version of MultiCharts64 is 9.0 Release 4 build 10360 from 12-01-2014. This version has critical improvements addressing this issue and other bug fixes.

As I can see on the screenshots – you are using two LMAX broker profiles. Per LMAX Support: LMAX API can handle connection to a single LMAX login (same for data provider and broker with identical connection settings). Connecting multiple LMAX broker profiles with different accounts is claimed to be unstable and unsupported by LMAX.
Hi Henry,

Firstly thank you for studying the logs. I have a few questions to seek clarification and to prevent this from happening in the future.

- Why was Multcharts constantly reconnecting to lmax?
Using pulseway.com to monitor my server I received no notification of a lost internet connection.

- Could the constant reconnects be caused by the older Multicharts version?

- The reason I use Multicharts64 8.8 Release 1 Build 8363 is because I tested this version for a long time with no issues. You can even see in this thread that I'm defending Multicharts and Lmax. I mention this exact version and that thread was from last year back in May I firsted commented in this thread http://www.multicharts.com/discussion/v ... ax#p107297

- ** I just remembered a couple of weeks ago I had lmax increase their quotes per second. By default Lmax issues 10 quotes per second and they increased this to 50 quotes per second to increase pricing accuracy for me. The reason I did this is because I run 2 servers, a test server in the same data center with the same configuration but connected to a smaller live account. For 1.5 years these 2 servers were executing trades exactly the same with maybe 0 - 0.4 of a pip difference. Lately 1 of my servers wouldn't execute a trade because the low or high of the price wasn't the same due to pricing inaccuracy. I contacted Lmax about the issue and they advised that yes this could happen and to increase accuracy they can increase the number of quotes per second. They said I could go as high as 100 quotes per second but I only went to 50. Could this be a factor? If yes why could this be a factor can Multicharts not handle this many quotes per second?

- I was never aware that Lmax doesn't support multiple broker profiles, where is this stated?

- More on the multiple broker profiles with Lmax. I have 1 server running 2 workspaces. 1 workspace for long trades connected to one account and another workspace for short trades connected to another account. I HAVE to do this because Lmax doesn't support hedging. Can you please clarify that the only way I can continue to trade with Multicharts and Lmax is to have a new server for each account? I.e. 1 server 1 workspace 1 account for long trades and then another server exactly the same but for short trades?

You say to avoid further issues I need to do the following:
In order to avoid such issue in the future I would recommend the following:

1. Update MultiCharts to the latest Release version (9.0 build 10360);
2. Configure LMAX data provider and broker with identical connection settings using the same login;
3. Avoid using more than one LMAX broker profile (may work seamless to some users);
4. Add market_position_at_broker check to your strategy to avoid generation of unnecessary orders;
5. Monitor Order and Position Tracker Logs tab for errors;
6. Do not start auto trading when you have issues with connection to the broker.
1. Will do.
2. Please clarify as per my questions above.
3. Please clarify as per my questions above.
4. In my code do I just replace Marketposition with Marketposition_at_broker? Or do I need to add some more smarts like after a trade has been sent then do a check to make sure the position has reached the broker? Any chance you can provide some reliable code for this as I don't want this issue to happen again.
5. I don't really know how I can do this, I can check the logs tab before turning on auto trading but this wouldn't stop this problem happening mid way through trading. Is there a way I can get an email alert for connection issues?

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby Henry MultiСharts » 30 Jan 2015

- Why was Multcharts constantly reconnecting to lmax?
Using pulseway.com to monitor my server I received no notification of a lost internet connection.
There has been no reply to heartbeats sent by MultiCharts to LMAX servers. Heartbeats were introduced by LMAX for detection of connection loss, as LMAX API does not have direct "connection is lost" method. If there is no turnaround for MultiCharts requests - we consider the connection to be down and try to reconnect.
- Could the constant reconnects be caused by the older Multicharts version?

- The reason I use Multicharts64 8.8 Release 1 Build 8363 is because I tested this version for a long time with no issues. You can even see in this thread that I'm defending Multicharts and Lmax. I mention this exact version and that thread was from last year back in May I firsted commented in this thread http://www.multicharts.com/discussion/v ... ax#p107297
Since the version you are using we have provided multiple attempts together with LMAX to have this issue resolved. The latest improvements can be found in the current version of MultiCharts. We have also added an option for enabling/disabling heartbeats as LMAX connection works differently for different users - some are ok with it enabled, some need to have it disabled.
- ** I just remembered a couple of weeks ago I had lmax increase their quotes per second. By default Lmax issues 10 quotes per second and they increased this to 50 quotes per second to increase pricing accuracy for me. The reason I did this is because I run 2 servers, a test server in the same data center with the same configuration but connected to a smaller live account. For 1.5 years these 2 servers were executing trades exactly the same with maybe 0 - 0.4 of a pip difference. Lately 1 of my servers wouldn't execute a trade because the low or high of the price wasn't the same due to pricing inaccuracy. I contacted Lmax about the issue and they advised that yes this could happen and to increase accuracy they can increase the number of quotes per second. They said I could go as high as 100 quotes per second but I only went to 50. Could this be a factor? If yes why could this be a factor can Multicharts not handle this many quotes per second?
This should not be an issue until you have any realtime quotes in the data queue.
- I was never aware that Lmax doesn't support multiple broker profiles, where is this stated?
This was stated by LMAX representatives while we were analyzing the heartbeats issue.
- More on the multiple broker profiles with Lmax. I have 1 server running 2 workspaces. 1 workspace for long trades connected to one account and another workspace for short trades connected to another account. I HAVE to do this because Lmax doesn't support hedging. Can you please clarify that the only way I can continue to trade with Multicharts and Lmax is to have a new server for each account? I.e. 1 server 1 workspace 1 account for long trades and then another server exactly the same but for short trades?
It works ok for some users, for some users it can cause issues. As it was stated by LMAX representatives -
the behaviour can be unstable, unpredictable and unsupported. So you can use it at your own risk.
4. In my code do I just replace Marketposition with Marketposition_at_broker? Or do I need to add some more smarts like after a trade has been sent then do a check to make sure the position has reached the broker? Any chance you can provide some reliable code for this as I don't want this issue to happen again.
If you are trading this instrument using this broker profile from one chart in SA mode without placing orders manually - you can check for mismatch between MarketPosition_at_Broker and MarketPosition_at_Broker_for_the_Strategy. But this won't be a simple copy/paste replacement as MarketPosition returns position direction for a given chart, while MarketPosition_at_Broker/_for_The_Strategy return number of contracts and the type of position at the broker for the symbol. So you need to incorporate this reserved words into your strategy logic.
5. I don't really know how I can do this, I can check the logs tab before turning on auto trading but this wouldn't stop this problem happening mid way through trading. Is there a way I can get an email alert for connection issues?
Please see the links from this post.

User avatar
t-rader
Posts: 139
Joined: 02 Feb 2011
Location: Australia
Has thanked: 11 times
Been thanked: 27 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby t-rader » 04 Feb 2015

- Could the constant reconnects be caused by the older Multicharts version?

- The reason I use Multicharts64 8.8 Release 1 Build 8363 is because I tested this version for a long time with no issues. You can even see in this thread that I'm defending Multicharts and Lmax. I mention this exact version and that thread was from last year back in May I firsted commented in this thread http://www.multicharts.com/discussion/v ... ax#p107297
Since the version you are using we have provided multiple attempts together with LMAX to have this issue resolved. The latest improvements can be found in the current version of MultiCharts. We have also added an option for enabling/disabling heartbeats as LMAX connection works differently for different users - some are ok with it enabled, some need to have it disabled.
The whole connection setup with Lmax seems so flakey, how stable is this new version of Multicharts with Lmax?

- ** I just remembered a couple of weeks ago I had lmax increase their quotes per second. By default Lmax issues 10 quotes per second and they increased this to 50 quotes per second to increase pricing accuracy for me. The reason I did this is because I run 2 servers, a test server in the same data center with the same configuration but connected to a smaller live account. For 1.5 years these 2 servers were executing trades exactly the same with maybe 0 - 0.4 of a pip difference. Lately 1 of my servers wouldn't execute a trade because the low or high of the price wasn't the same due to pricing inaccuracy. I contacted Lmax about the issue and they advised that yes this could happen and to increase accuracy they can increase the number of quotes per second. They said I could go as high as 100 quotes per second but I only went to 50. Could this be a factor? If yes why could this be a factor can Multicharts not handle this many quotes per second?
This should not be an issue until you have any realtime quotes in the data queue.
Can you please explain what you mean here a little more? It sounds like it could be an issue?

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby Henry MultiСharts » 05 Feb 2015

The whole connection setup with Lmax seems so flakey, how stable is this new version of Multicharts with Lmax?
We have not received any complaints regarding this functionality lately.
Can you please explain what you mean here a little more? It sounds like it could be an issue?
In MultiCharts 8.8 there is "Nq/Ns" indication of quotes' queue status in the bottom right corner of the main MultiCharts window. I.e. the number of quotes in the queue to be processed (in case the quotes number is too large and they are coming too fast and MC can not handle all these these quotes in realtime). If you have "0q/0s" it means everything okay, you have no queue and all the quotes are being processed in time. These numbers will be displayed in red as soon as a queue is formed.

"Nq" stands for the number of quotes in the queue.
"Ns" stands for the number of seconds for which the first quote in the queue is already delayed.

In MultiCharts 9.0 the indicator is not shown while there is no queue (0q/0s), once there is a queue it shows up and is being painted red.

Zoli
Posts: 90
Joined: 12 Sep 2012
Has thanked: 24 times
Been thanked: 38 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby Zoli » 05 Feb 2015

Hello,

This is an unfortunate situation and am sorry for your loss.
I think as well there are some reliability issues with LMAX. Today I had a Stop Order triggered 3x and I cannot figure out why and how this happened. I do have now 2 positions opened in the contrarian direction of my initial trade. I add a take profit and stop order to each of my trades using GenerateStopLoss and GenerateProfitTarget and my code is as simple as possible. I have the signal running on 32 LMAX symbols and this is barely the 2nd time I witnessed something wrong out of tens of trades (probably random issue that cannot be replicated easily).
This particular time it didn't even cancel the TakeProfit (sell limit is still there).
Please see attached screenshot.
Attachments
lmax.png
(60.52 KiB) Downloaded 2894 times

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby Henry MultiСharts » 05 Feb 2015

Hello,

This is an unfortunate situation and am sorry for your loss.
I think as well there are some reliability issues with LMAX. Today I had a Stop Order triggered 3x and I cannot figure out why and how this happened. I do have now 2 positions opened in the contrarian direction of my initial trade. I add a take profit and stop order to each of my trades using GenerateStopLoss and GenerateProfitTarget and my code is as simple as possible. I have the signal running on 32 LMAX symbols and this is barely the 2nd time I witnessed something wrong out of tens of trades (probably random issue that cannot be replicated easily).
This particular time it didn't even cancel the TakeProfit (sell limit is still there).
Please see attached screenshot.
Hello Zoli,

Please send us the information requested in post #4 above + specify IDs of the orders in question.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby orion » 05 Feb 2015

In MultiCharts 9.0 the indicator is not shown while there is no queue (0q/0s), once there is a queue it shows up and is being painted red.
Henry, it would be great if you could provide us programmatic access to this in a future release!

champski
Posts: 71
Joined: 28 Jun 2011
Has thanked: 20 times
Been thanked: 1 time

Re: Sync didn't sync lost me $26,000 USD!!!

Postby champski » 16 Feb 2015

LMAX is also my broker and I experienced connection issues last Friday for the first time whilst auto-trading.

I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts. In autotrade properties, orders not filled are set to expire after 5 seconds.
Also, my email alerts went berzerk. Most ticks sent multiple email alerts and one actually sent 24!

I wouldn't expect this to happen considering they are just market orders. I have been auto-trading with LMAX for over a year and never had this issue.

My server is running the latest version of MC (v9) on Windows 2008 server.

I run 2 signals through one broker profile.

Anyone have any ideas?

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby MAtricks » 16 Feb 2015

Sounds like slippage during a fast market move to me. What time was it? Do you measure and record the bid/ask spread?

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby orion » 16 Feb 2015

I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts. In autotrade properties, orders not filled are set to expire after 5 seconds.
What is your bar resolution and how many times were they cancelled?

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby TJ » 16 Feb 2015

LMAX is also my broker and I experienced connection issues last Friday for the first time whilst auto-trading.
I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts. In autotrade properties, orders not filled are set to expire after 5 seconds.
Also, my email alerts went berzerk. Most ticks sent multiple email alerts and one actually sent 24!
I wouldn't expect this to happen considering they are just market orders. I have been auto-trading with LMAX for over a year and never had this issue.
My server is running the latest version of MC (v9) on Windows 2008 server.
I run 2 signals through one broker profile.
Anyone have any ideas?
Don't exit your MultiCharts.
Don't reboot your machine.
Contact tech support so that they can connect to your computer to retrieve the logs and do online diagnostics.

champski
Posts: 71
Joined: 28 Jun 2011
Has thanked: 20 times
Been thanked: 1 time

Re: Sync didn't sync lost me $26,000 USD!!!

Postby champski » 16 Feb 2015

LMAX is also my broker and I experienced connection issues last Friday for the first time whilst auto-trading.
I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts. In autotrade properties, orders not filled are set to expire after 5 seconds.
Also, my email alerts went berzerk. Most ticks sent multiple email alerts and one actually sent 24!
I wouldn't expect this to happen considering they are just market orders. I have been auto-trading with LMAX for over a year and never had this issue.
My server is running the latest version of MC (v9) on Windows 2008 server.
I run 2 signals through one broker profile.
Anyone have any ideas?
Don't exit your MultiCharts.
Don't reboot your machine.
Contact tech support so that they can connect to your computer to retrieve the logs and do online diagnostics.
Before I read this I renamed my log file in an attempt to back it up and rebooted my server but unfortunatey it was deleted in the process.

champski
Posts: 71
Joined: 28 Jun 2011
Has thanked: 20 times
Been thanked: 1 time

Re: Sync didn't sync lost me $26,000 USD!!!

Postby champski » 16 Feb 2015

I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts. In autotrade properties, orders not filled are set to expire after 5 seconds.
What is your bar resolution and how many times were they cancelled?
MAtricks and Orion,

Bar resolution was hourly and one particular trade tried to exit 3-4 times during the 11am (EDT) bar. Therefore the exit orders were trying to send (or were ignored by LMAX) for 15 seconds.
**Remember "cancel obsolete orders" is set to 5 seconds.
NB. The issue occured on my test server and the trade size was only 1 std lot.

Question: From a 'communication route' perspective, when orders are sent from the MC client to the broker are they routed through any Multicharts hosted middleware or does the MC client communicate orders solely with the broker?

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby orion » 16 Feb 2015

Given your market timer is set to 5 secs, a 15-20 sec broker connection loss will result in 3-4 retries. We need a new PL keyword broker.connectionStatus to give user code visibility into connection status so that user code may take appropriate action in such cases.

User avatar
JoshM
Posts: 2195
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1544 times
Been thanked: 1565 times
Contact:

Re: Sync didn't sync lost me $26,000 USD!!!

Postby JoshM » 17 Feb 2015

We need a new PL keyword broker.connectionStatus to give user code visibility into connection status so that user code may take appropriate action in such cases.
I fully agree with this, it's risk management 101 to have programmatic access to things like this. But I'm sceptical about it being implemented any time soon, since...
Unfortunately there is no technical possibility for implementing this functionality at the moment.
Source: broker and datafeed connectivity status event.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby orion » 17 Feb 2015

one particular trade tried to exit 3-4 times during the 11am (EDT) bar. Therefore the exit orders were trying to send (or were ignored by LMAX) for 15 seconds.
champski, I read your post more carefully. Were all your problems with exits or did you have problems with entries too?

champski
Posts: 71
Joined: 28 Jun 2011
Has thanked: 20 times
Been thanked: 1 time

Re: Sync didn't sync lost me $26,000 USD!!!

Postby champski » 17 Feb 2015

one particular trade tried to exit 3-4 times during the 11am (EDT) bar. Therefore the exit orders were trying to send (or were ignored by LMAX) for 15 seconds.
champski, I read your post more carefully. Were all your problems with exits or did you have problems with entries too?
Orion, I only had issues with the exits.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby Henry MultiСharts » 18 Feb 2015

I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts.
What is the State of the orders in File -> New -> Order and position tracker window->Orders tab? Are there any messages corresponding to these orders on the Logs tab?
In autotrade properties, orders not filled are set to expire after 5 seconds.
If you are referring to "Cancel Market Orders as Obsolete If not Filled Within X Seconds" option then it will cancel the order that was accepted by broker within specified timeout. If MultiCharts was unable to place an order to the broker then there is nothing to cancel.
Question: From a 'communication route' perspective, when orders are sent from the MC client to the broker are they routed through any Multicharts hosted middleware or does the MC client communicate orders solely with the broker?
MultiCharts is a standalone application. All modules, configuration parameters and user data (workspaces, studies, data) are kept on your computer. The data/broker connection is performed from your computer directly to your data/broker servers.
There are no MultiCharts servers that keep your information or that work as a middleware.
You can consult with your broker directly regarding order processing mechanisms.

champski
Posts: 71
Joined: 28 Jun 2011
Has thanked: 20 times
Been thanked: 1 time

Re: Sync didn't sync lost me $26,000 USD!!!

Postby champski » 19 Feb 2015

I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts.
What is the State of the orders in File -> New -> Order and position tracker window->Orders tab? Are there any messages corresponding to these orders on the Logs tab?
In autotrade properties, orders not filled are set to expire after 5 seconds.
If you are referring to "Cancel Market Orders as Obsolete If not Filled Within X Seconds" option then it will cancel the order that was accepted by broker within specified timeout. If MultiCharts was unable to place an order to the broker then there is nothing to cancel.
Question: From a 'communication route' perspective, when orders are sent from the MC client to the broker are they routed through any Multicharts hosted middleware or does the MC client communicate orders solely with the broker?
MultiCharts is a standalone application. All modules, configuration parameters and user data (workspaces, studies, data) are kept on your computer. The data/broker connection is performed from your computer directly to your data/broker servers.
There are no MultiCharts servers that keep your information or that work as a middleware.
You can consult with your broker directly regarding order processing mechanisms.
The orders tab shows the "expected" trade entries and exits. Therefore, no signs of issues. The log tab is interesting though. Logging information for the day in which the problem occured (13/02/2015) is completely missing (See attachment). Logging info for the dates post and prior is there however.

Where could it have gone?

NB. I renamed my log file in an attempt to back it up from (%appdata%\TS Support\MultiCharts64\9.0.10360.400\Logs\LMAXServerHost) and unfortunatey it was gone after I restarted the server.
Attachments
Missing13thFeb.jpg
(50.55 KiB) Downloaded 2876 times

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby Henry MultiСharts » 19 Feb 2015

The orders tab shows the "expected" trade entries and exits. Therefore, no signs of issues. The log tab is interesting though. Logging information for the day in which the problem occured (13/02/2015) is completely missing (See attachment). Logging info for the dates post and prior is there however.

Where could it have gone?
champski,

Please make sure there are no filters applied to your Order and Position Tracker Logs tab.
In case the issue reappears - please send us the information requested in this post.

champski
Posts: 71
Joined: 28 Jun 2011
Has thanked: 20 times
Been thanked: 1 time

Re: Sync didn't sync lost me $26,000 USD!!!

Postby champski » 20 Feb 2015

The orders tab shows the "expected" trade entries and exits. Therefore, no signs of issues. The log tab is interesting though. Logging information for the day in which the problem occured (13/02/2015) is completely missing (See attachment). Logging info for the dates post and prior is there however.

Where could it have gone?
champski,

Please make sure there are no filters applied to your Order and Position Tracker Logs tab.
In case the issue reappears - please send us the information requested in this post.
Henry, I checked the Order and Position Tracker Logs tab and made sure there are no filters applied. All options are set to "All" except for the dates.

champski
Posts: 71
Joined: 28 Jun 2011
Has thanked: 20 times
Been thanked: 1 time

Re: Sync didn't sync lost me $26,000 USD!!!

Postby champski » 23 Feb 2015

Hi,

I just logged onto my server to give it it's weekly reboot and discovered my account was not showing under accounts in the Order and Position tracker and one of the two workspaces was disconnected from autotrading. I would send you the log file but unfortunately it was 60gb - so I had to delete it.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby Henry MultiСharts » 23 Feb 2015

Hi,

I just logged onto my server to give it it's weekly reboot and discovered my account was not showing under accounts in the Order and Position tracker and one of the two workspaces was disconnected from autotrading. I would send you the log file but unfortunately it was 60gb - so I had to delete it.
Hello champski,

Can you see any messages under File -> New -> Order and position tracker window->Logs tab corresponding to this situation?

GTC
Posts: 1
Joined: 25 Feb 2015

Re: Sync didn't sync lost me $26,000 USD!!!

Postby GTC » 25 Feb 2015

Hi Sorry to hear your problems. I too have experienced this bug frequently & in fact it happened again today. I you refer to the enclosed pic, you can see my position is - 2 lots, yet the Master Strategy has put in orders for a total of 4.

The interesting thing is that I trade from 2 separate environments - home & office, using different PC's and installations of MultiCharts and I get this bug occurring from both locations. I re-start my PC every day & MultiCharts through out the session.

This leads me to believe that the issue is a bug with MultiCharts, rather than at my end. Its a pity
because the MasterStrategy idea is good, but its too unreliable so now I just enter my orders through I.B. (the example I enclosed in the pic was during Sim trading).
Attachments
MC orders.JPG
(190.59 KiB) Downloaded 2881 times

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby Henry MultiСharts » 26 Feb 2015

Hi Sorry to hear your problems. I too have experienced this bug frequently & in fact it happened again today. I you refer to the enclosed pic, you can see my position is - 2 lots, yet the Master Strategy has put in orders for a total of 4.

The interesting thing is that I trade from 2 separate environments - home & office, using different PC's and installations of MultiCharts and I get this bug occurring from both locations. I re-start my PC every day & MultiCharts through out the session.

This leads me to believe that the issue is a bug with MultiCharts, rather than at my end. Its a pity
because the MasterStrategy idea is good, but its too unreliable so now I just enter my orders through I.B. (the example I enclosed in the pic was during Sim trading).
Hello GTC,

Our engineers are analyzing your case.

User avatar
t-rader
Posts: 139
Joined: 02 Feb 2011
Location: Australia
Has thanked: 11 times
Been thanked: 27 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby t-rader » 04 Apr 2015

After taking a break since getting smashed by a Multicharts/Lmax issue! I'm getting back into trading and need the help of the community and MC moderators to ensure this issue doesn't happen again.

I've updated to the latest MC version 9.0 Build 10759.

With all the latest updates to this version of Multicharts, how do I monitor the broker connection status as I was advised to do:
5. Monitor Order and Position Tracker Logs tab for errors;
As Henry advised that the issue was caused by:
Our developers have studied the logs from your PC. From the logs we can see that there were issues with connection to LMAX servers for multiple days before the orders were placed (constant reconnects starting 21-01-2015 14:37:03 until 23-01-2015 22:07) – you can see the corresponding messages in Order and Position Tracker Logs tab: Reconnecting ...Connection established. Reconnecting ....
The orders were placed between the reconnects and no order confirmation has been received from the broker (you can see the corresponding messages in Order and Position Tracker Logs tab: Didn't receive final status for order … ).
so specifically how do I send an email alert when the Order and Position Tracker are reporting connection loss, reconnecting, connection established?

In response to this question previously Henry linked me to the follow thread:
5. I don't really know how I can do this, I can check the logs tab before turning on auto trading but this wouldn't stop this problem happening mid way through trading. Is there a way I can get an email alert for connection issues?
Please see the links from this post.
This thread then has a link to this thread http://www.multicharts.com/discussion/v ... f=5&t=9890
which firstly seems old as it states that Autotrading turns off when broker connection is lost, which never happened for me and the solution provided is to monitor if Autotrading is turned off. So this solution seems outdated and of no use now, unless I'm reading this wrong?

I would also like to confirm as MATricks mentioned:
-How many contracts do you have in the Maximum Shares input?
Would setting my trade size and the setting 'maximum shares/contracts per position' via the Strategy properties as opposed to via my code prevent the issue of my trade doubling up?

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Sync didn't sync lost me $26,000 USD!!!

Postby Henry MultiСharts » 07 Apr 2015

Hello t-rader,

There is no prebuilt alert functionality for Order and Position Tracker window messages. You can create your own script that will monitor MultiCharts logs (LMAXServerHost_NET_Trace.txt) for the following message (or part of it): LMAXServerHost.exe Information: 0 : Connection changed: from Connected to ReConnecting similar to what Faraz has described in this post.
I would also like to confirm as MATricks mentioned:
MAtricks wrote:
-How many contracts do you have in the Maximum Shares input?

Would setting my trade size and the setting 'maximum shares/contracts per position' via the Strategy properties as opposed to via my code prevent the issue of my trade doubling up?
This option can be useful both in backtesting and live trading. I would also recommend adding market_position_at_broker check to your strategy to avoid generation of unnecessary orders as I have mentioned in one of my previous posts. Please keep in mind that if there are multiple orders too close to market and market moves fast then there is a chance that both orders are filled before MultiCharts sends cancel command.


Return to “MultiCharts”