Sync didn't sync lost me $26,000 USD!!!
- t-rader
- Posts: 139
- Joined: 02 Feb 2011
- Location: Australia
- Has thanked: 11 times
- Been thanked: 27 times
Sync didn't sync lost me $26,000 USD!!!
Multicharts please explain and help me!
I have Multicharts connected to lmax and set to sync but it never synced! It also doubled up my orders! On top of this I always send a stop order using SetStopLoss to be held at the broker in case something happens to my server but this was never sent!
So because Multicharts didnt sync, doubled up my orders and never sent my hard stop to the broker the trade never exited which in turn resulted in a margin call and me losing $26,000 USD!!! Now I know my server didn't lose connection to the internet because I monitor my server using remote monitoring service pulseway.com so to me this seems like a Multicharts error!
I don't want to jump to conclusions and blame Multicharts but why didn't Multicharts sync!?
Why did it double my orders!?
Why did it not send my hard stop loss to the broker!?
Why weren't my orders cancelled within 4 seconds as per the strategy properties settings?
I've been trading with Multicharts for approx 3 years now and with Lmax as my broker for the last 1.6 years. I haven't changed my server, Multicharts configuration or my code and all I do is reboot my server each week and turn autotrading on.
I really need a please explain and some help here!
I have Multicharts connected to lmax and set to sync but it never synced! It also doubled up my orders! On top of this I always send a stop order using SetStopLoss to be held at the broker in case something happens to my server but this was never sent!
So because Multicharts didnt sync, doubled up my orders and never sent my hard stop to the broker the trade never exited which in turn resulted in a margin call and me losing $26,000 USD!!! Now I know my server didn't lose connection to the internet because I monitor my server using remote monitoring service pulseway.com so to me this seems like a Multicharts error!
I don't want to jump to conclusions and blame Multicharts but why didn't Multicharts sync!?
Why did it double my orders!?
Why did it not send my hard stop loss to the broker!?
Why weren't my orders cancelled within 4 seconds as per the strategy properties settings?
I've been trading with Multicharts for approx 3 years now and with Lmax as my broker for the last 1.6 years. I haven't changed my server, Multicharts configuration or my code and all I do is reboot my server each week and turn autotrading on.
I really need a please explain and some help here!
- Attachments
-
- Orders-BrokerLmax.PNG
- Orders-BrokerLmax
- (119.84 KiB) Downloaded 3045 times
-
- StrategyProperties.PNG
- StrategyProperties
- (166.87 KiB) Downloaded 3028 times
-
- PosHistory-WhatHappened.PNG
- PosHistory-WhatHappened
- (443.72 KiB) Downloaded 2988 times
-
- Orders-WhatHappened.PNG
- (635.67 KiB) Downloaded 2958 times
Re: Sync didn't sync lost me $26,000 USD!!!
First, let me acknowledge the pain of your loss. As someone said, the joy of winning and the pain of losing are right up there with the pain of winning and the joy of losing. Now, some points that may be relevant to your case:
1. Looking at your OPT, it seems that the order status never changed from 'Sent' to 'Filled' in MC even though the position changed which can happen if the execution fills messages never came through. This thread is about how there are two channels of information from the broker to MC and the two channels can get out of sync. Essentially, MarketPosition_at_broker and MarketPosition_at_broker_for_the_strategy are updated by different channels. This is also mentioned in MC documentation under the 'unexpected mismatch' case.
2. If the above happened, that explains why your hard stop wasn't placed and so you had an exposed trade. It also explains why your strategy logic which perhaps worked off the MarketPosition or MarketPosition_at_broker_for_the_strategy fired off twice to get you into the same trade twice.
3. Why did the execution fill messages not come through even though your Internet connection was working. Internet connection working is a necessary but not sufficient condition for the messages to come through. Is it an MC problem? I would estimate MC has a few million lines of code and has it share of bugs but it is a fantastic platform nonetheless. I would first suspect the message loss to be a problem with either your broker's software or infrastructure. Having spent two decades building infrastructure, I have an appreciation for the fallibility of the underlying infrastructure and a great respect for anything that is able to work 99.9% of the time given the underlying imperfections.
4. Finally, auto trading is not the same as unattended trading. Simple alerts about Internet connection loss are not sufficient. We need to always keep an eye on the positions in both MC OPT and the broker GUI since we can all be victims of Murphy's law at any time. If something can go wrong it will.
Andrew or Henry, please feel free to correct any of my points above especially points #1 and #2.
1. Looking at your OPT, it seems that the order status never changed from 'Sent' to 'Filled' in MC even though the position changed which can happen if the execution fills messages never came through. This thread is about how there are two channels of information from the broker to MC and the two channels can get out of sync. Essentially, MarketPosition_at_broker and MarketPosition_at_broker_for_the_strategy are updated by different channels. This is also mentioned in MC documentation under the 'unexpected mismatch' case.
2. If the above happened, that explains why your hard stop wasn't placed and so you had an exposed trade. It also explains why your strategy logic which perhaps worked off the MarketPosition or MarketPosition_at_broker_for_the_strategy fired off twice to get you into the same trade twice.
3. Why did the execution fill messages not come through even though your Internet connection was working. Internet connection working is a necessary but not sufficient condition for the messages to come through. Is it an MC problem? I would estimate MC has a few million lines of code and has it share of bugs but it is a fantastic platform nonetheless. I would first suspect the message loss to be a problem with either your broker's software or infrastructure. Having spent two decades building infrastructure, I have an appreciation for the fallibility of the underlying infrastructure and a great respect for anything that is able to work 99.9% of the time given the underlying imperfections.
4. Finally, auto trading is not the same as unattended trading. Simple alerts about Internet connection loss are not sufficient. We need to always keep an eye on the positions in both MC OPT and the broker GUI since we can all be victims of Murphy's law at any time. If something can go wrong it will.
Andrew or Henry, please feel free to correct any of my points above especially points #1 and #2.
Re: Sync didn't sync lost me $26,000 USD!!!
Very sorry to hear this
While I feel terrible for you, we must get to the bottom of this so it doesn't happen again:
-How long did you leave MC unattended? Any longer than the time it takes you to sleep is a giant nono.
-When was the last time you restarted MC? Bugs happen.. restarting the platform every day is a pain in the neck, but I haven't had any bugs since I started doing this years ago
-How many contracts do you have in the Maximum Shares input?
-Do you have the MC Logs for the event?
I've run into instances where the Broker disconnects from MC which disables SA/AA with certain APIs. This has always been a fear of mine..
While I feel terrible for you, we must get to the bottom of this so it doesn't happen again:
-How long did you leave MC unattended? Any longer than the time it takes you to sleep is a giant nono.
-When was the last time you restarted MC? Bugs happen.. restarting the platform every day is a pain in the neck, but I haven't had any bugs since I started doing this years ago
-How many contracts do you have in the Maximum Shares input?
-Do you have the MC Logs for the event?
I've run into instances where the Broker disconnects from MC which disables SA/AA with certain APIs. This has always been a fear of mine..
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Sync didn't sync lost me $26,000 USD!!!
Hello t-rader,
We are very sorry to hear about your issue and we will do our best to investigate it. In case of trading such high volumes the loss in case of an issue like this can be significant. Unfortunately, now there is no way to recover the loss but I would like to reassure you that we will do all possible to find out the reason of the problem so it can be avoided in future.
MultiCharts has always recommended all users to monitor auto-trading when trading live. I understand that it may seem unnecessary, especially since you have traded successfully for more than a year with the same setup, but still it is something that you may want to consider.
We will need the following for investigation:
1) MultiCharts logs (if you have not restarted the platform. By default the logs from the previous run are erased when you restart MC).
In MultiCharts go to Help->Feedback->Send logs. Please let me know that you have uploaded the logs.
If you want to send the logs manually please follow this guide: https://www.multicharts.com/trading-sof ... harts_Logs
2) In MultiCharts go to File->New->Open Order and position tracker window-> Orders tab->make sure you are not filtering the information in columns and the dates when the orders were placed are covered, then go to File->Export to excel.
Please also export the information from the Logs tab and send it to me (support@multicharts.com) for analysis.
Again, I am very sad to hear about this issue and will provide you with any assistance possible to make sure it won’t happen in future.
We are very sorry to hear about your issue and we will do our best to investigate it. In case of trading such high volumes the loss in case of an issue like this can be significant. Unfortunately, now there is no way to recover the loss but I would like to reassure you that we will do all possible to find out the reason of the problem so it can be avoided in future.
MultiCharts has always recommended all users to monitor auto-trading when trading live. I understand that it may seem unnecessary, especially since you have traded successfully for more than a year with the same setup, but still it is something that you may want to consider.
We will need the following for investigation:
1) MultiCharts logs (if you have not restarted the platform. By default the logs from the previous run are erased when you restart MC).
In MultiCharts go to Help->Feedback->Send logs. Please let me know that you have uploaded the logs.
If you want to send the logs manually please follow this guide: https://www.multicharts.com/trading-sof ... harts_Logs
2) In MultiCharts go to File->New->Open Order and position tracker window-> Orders tab->make sure you are not filtering the information in columns and the dates when the orders were placed are covered, then go to File->Export to excel.
Please also export the information from the Logs tab and send it to me (support@multicharts.com) for analysis.
Again, I am very sad to hear about this issue and will provide you with any assistance possible to make sure it won’t happen in future.
- t-rader
- Posts: 139
- Joined: 02 Feb 2011
- Location: Australia
- Has thanked: 11 times
- Been thanked: 27 times
Re: Sync didn't sync lost me $26,000 USD!!!
Thanks for the support guys, its a bit rough taking such a hit, more so when it wasn't my trading system which was the reason for the loss. It just sucks I never encountered this sync problem earlier in my trading.
- Multicharts was left unattended for approx 12 hours which was evening my time.
- Server was running for 9-10 days before its last restart.
- Multicharts logs sent to Multicharts support.
I've sent the logs via Help > Feedback > Send Logs.
I've also emailed you the exported orders and logs file from the OPT.
I await your response.
Cheers,
Hello MAtricksVery sorry to hear this
While I feel terrible for you, we must get to the bottom of this so it doesn't happen again:
-How long did you leave MC unattended? Any longer than the time it takes you to sleep is a giant nono.
-When was the last time you restarted MC? Bugs happen.. restarting the platform every day is a pain in the neck, but I haven't had any bugs since I started doing this years ago
-How many contracts do you have in the Maximum Shares input?
-Do you have the MC Logs for the event?
- Maximum shares/contracts I have set to 10,000. I can trade up 2,400 on a usual trade so I can see this is a safety measure I should definitely use!
I've run into instances where the Broker disconnects from MC which disables SA/AA with certain APIs. This has always been a fear of mine..
- Multicharts was left unattended for approx 12 hours which was evening my time.
- Server was running for 9-10 days before its last restart.
- Multicharts logs sent to Multicharts support.
Hi Henry,Hello t-rader,
We will need the following for investigation:
1) MultiCharts logs (if you have not restarted the platform. By default the logs from the previous run are erased when you restart MC).
In MultiCharts go to Help->Feedback->Send logs. Please let me know that you have uploaded the logs.
If you want to send the logs manually please follow this guide: https://www.multicharts.com/trading-sof ... harts_Logs
2) In MultiCharts go to File->New->Open Order and position tracker window-> Orders tab->make sure you are not filtering the information in columns and the dates when the orders were placed are covered, then go to File->Export to excel.
Please also export the information from the Logs tab and send it to me (support@multicharts.com) for analysis.
I've sent the logs via Help > Feedback > Send Logs.
I've also emailed you the exported orders and logs file from the OPT.
I await your response.
Cheers,
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Sync didn't sync lost me $26,000 USD!!!
Hello t-rader,
Our developers have studied the logs from your PC. From the logs we can see that there were issues with connection to LMAX servers for multiple days before the orders were placed (constant reconnects starting 21-01-2015 14:37:03 until 23-01-2015 22:07) – you can see the corresponding messages in Order and Position Tracker Logs tab: Reconnecting ...Connection established. Reconnecting ....
The orders were placed between the reconnects and no order confirmation has been received from the broker (you can see the corresponding messages in Order and Position Tracker Logs tab: Didn't receive final status for order … ). In such case MultiCharts considers the orders it has sent to be cancelled or not existing. Once order generation conditions are met in the strategy – the orders will be sent to broker.
From the logs we can also see that you are running an outdated version of MultiCharts64 8.8 Release 1 build 8363 from 12-3-2013 that had known LMAX connection loss issues. The current version of MultiCharts64 is 9.0 Release 4 build 10360 from 12-01-2014. This version has critical improvements addressing this issue and other bug fixes.
As I can see on the screenshots – you are using two LMAX broker profiles. Per LMAX Support: LMAX API can handle connection to a single LMAX login (same for data provider and broker with identical connection settings). Connecting multiple LMAX broker profiles with different accounts is claimed to be unstable and unsupported by LMAX.
In order to avoid such issue in the future I would recommend the following:
1. Update MultiCharts to the latest Release version (9.0 build 10360);
2. Configure LMAX data provider and broker with identical connection settings using the same login;
3. Avoid using more than one LMAX broker profile (may work seamless to some users);
4. Add market_position_at_broker check to your strategy to avoid generation of unnecessary orders;
5. Monitor Order and Position Tracker Logs tab for errors;
6. Do not start auto trading when you have issues with connection to the broker.
Our developers have studied the logs from your PC. From the logs we can see that there were issues with connection to LMAX servers for multiple days before the orders were placed (constant reconnects starting 21-01-2015 14:37:03 until 23-01-2015 22:07) – you can see the corresponding messages in Order and Position Tracker Logs tab: Reconnecting ...Connection established. Reconnecting ....
The orders were placed between the reconnects and no order confirmation has been received from the broker (you can see the corresponding messages in Order and Position Tracker Logs tab: Didn't receive final status for order … ). In such case MultiCharts considers the orders it has sent to be cancelled or not existing. Once order generation conditions are met in the strategy – the orders will be sent to broker.
From the logs we can also see that you are running an outdated version of MultiCharts64 8.8 Release 1 build 8363 from 12-3-2013 that had known LMAX connection loss issues. The current version of MultiCharts64 is 9.0 Release 4 build 10360 from 12-01-2014. This version has critical improvements addressing this issue and other bug fixes.
As I can see on the screenshots – you are using two LMAX broker profiles. Per LMAX Support: LMAX API can handle connection to a single LMAX login (same for data provider and broker with identical connection settings). Connecting multiple LMAX broker profiles with different accounts is claimed to be unstable and unsupported by LMAX.
In order to avoid such issue in the future I would recommend the following:
1. Update MultiCharts to the latest Release version (9.0 build 10360);
2. Configure LMAX data provider and broker with identical connection settings using the same login;
3. Avoid using more than one LMAX broker profile (may work seamless to some users);
4. Add market_position_at_broker check to your strategy to avoid generation of unnecessary orders;
5. Monitor Order and Position Tracker Logs tab for errors;
6. Do not start auto trading when you have issues with connection to the broker.
- t-rader
- Posts: 139
- Joined: 02 Feb 2011
- Location: Australia
- Has thanked: 11 times
- Been thanked: 27 times
Re: Sync didn't sync lost me $26,000 USD!!!
Hi Henry,Hello t-rader,
Our developers have studied the logs from your PC. From the logs we can see that there were issues with connection to LMAX servers for multiple days before the orders were placed (constant reconnects starting 21-01-2015 14:37:03 until 23-01-2015 22:07) – you can see the corresponding messages in Order and Position Tracker Logs tab: Reconnecting ...Connection established. Reconnecting ....
The orders were placed between the reconnects and no order confirmation has been received from the broker (you can see the corresponding messages in Order and Position Tracker Logs tab: Didn't receive final status for order … ). In such case MultiCharts considers the orders it has sent to be cancelled or not existing. Once order generation conditions are met in the strategy – the orders will be sent to broker.
From the logs we can also see that you are running an outdated version of MultiCharts64 8.8 Release 1 build 8363 from 12-3-2013 that had known LMAX connection loss issues. The current version of MultiCharts64 is 9.0 Release 4 build 10360 from 12-01-2014. This version has critical improvements addressing this issue and other bug fixes.
As I can see on the screenshots – you are using two LMAX broker profiles. Per LMAX Support: LMAX API can handle connection to a single LMAX login (same for data provider and broker with identical connection settings). Connecting multiple LMAX broker profiles with different accounts is claimed to be unstable and unsupported by LMAX.
Firstly thank you for studying the logs. I have a few questions to seek clarification and to prevent this from happening in the future.
- Why was Multcharts constantly reconnecting to lmax?
Using pulseway.com to monitor my server I received no notification of a lost internet connection.
- Could the constant reconnects be caused by the older Multicharts version?
- The reason I use Multicharts64 8.8 Release 1 Build 8363 is because I tested this version for a long time with no issues. You can even see in this thread that I'm defending Multicharts and Lmax. I mention this exact version and that thread was from last year back in May I firsted commented in this thread http://www.multicharts.com/discussion/v ... ax#p107297
- ** I just remembered a couple of weeks ago I had lmax increase their quotes per second. By default Lmax issues 10 quotes per second and they increased this to 50 quotes per second to increase pricing accuracy for me. The reason I did this is because I run 2 servers, a test server in the same data center with the same configuration but connected to a smaller live account. For 1.5 years these 2 servers were executing trades exactly the same with maybe 0 - 0.4 of a pip difference. Lately 1 of my servers wouldn't execute a trade because the low or high of the price wasn't the same due to pricing inaccuracy. I contacted Lmax about the issue and they advised that yes this could happen and to increase accuracy they can increase the number of quotes per second. They said I could go as high as 100 quotes per second but I only went to 50. Could this be a factor? If yes why could this be a factor can Multicharts not handle this many quotes per second?
- I was never aware that Lmax doesn't support multiple broker profiles, where is this stated?
- More on the multiple broker profiles with Lmax. I have 1 server running 2 workspaces. 1 workspace for long trades connected to one account and another workspace for short trades connected to another account. I HAVE to do this because Lmax doesn't support hedging. Can you please clarify that the only way I can continue to trade with Multicharts and Lmax is to have a new server for each account? I.e. 1 server 1 workspace 1 account for long trades and then another server exactly the same but for short trades?
You say to avoid further issues I need to do the following:
1. Will do.In order to avoid such issue in the future I would recommend the following:
1. Update MultiCharts to the latest Release version (9.0 build 10360);
2. Configure LMAX data provider and broker with identical connection settings using the same login;
3. Avoid using more than one LMAX broker profile (may work seamless to some users);
4. Add market_position_at_broker check to your strategy to avoid generation of unnecessary orders;
5. Monitor Order and Position Tracker Logs tab for errors;
6. Do not start auto trading when you have issues with connection to the broker.
2. Please clarify as per my questions above.
3. Please clarify as per my questions above.
4. In my code do I just replace Marketposition with Marketposition_at_broker? Or do I need to add some more smarts like after a trade has been sent then do a check to make sure the position has reached the broker? Any chance you can provide some reliable code for this as I don't want this issue to happen again.
5. I don't really know how I can do this, I can check the logs tab before turning on auto trading but this wouldn't stop this problem happening mid way through trading. Is there a way I can get an email alert for connection issues?
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Sync didn't sync lost me $26,000 USD!!!
There has been no reply to heartbeats sent by MultiCharts to LMAX servers. Heartbeats were introduced by LMAX for detection of connection loss, as LMAX API does not have direct "connection is lost" method. If there is no turnaround for MultiCharts requests - we consider the connection to be down and try to reconnect.- Why was Multcharts constantly reconnecting to lmax?
Using pulseway.com to monitor my server I received no notification of a lost internet connection.
Since the version you are using we have provided multiple attempts together with LMAX to have this issue resolved. The latest improvements can be found in the current version of MultiCharts. We have also added an option for enabling/disabling heartbeats as LMAX connection works differently for different users - some are ok with it enabled, some need to have it disabled.- Could the constant reconnects be caused by the older Multicharts version?
- The reason I use Multicharts64 8.8 Release 1 Build 8363 is because I tested this version for a long time with no issues. You can even see in this thread that I'm defending Multicharts and Lmax. I mention this exact version and that thread was from last year back in May I firsted commented in this thread http://www.multicharts.com/discussion/v ... ax#p107297
This should not be an issue until you have any realtime quotes in the data queue.- ** I just remembered a couple of weeks ago I had lmax increase their quotes per second. By default Lmax issues 10 quotes per second and they increased this to 50 quotes per second to increase pricing accuracy for me. The reason I did this is because I run 2 servers, a test server in the same data center with the same configuration but connected to a smaller live account. For 1.5 years these 2 servers were executing trades exactly the same with maybe 0 - 0.4 of a pip difference. Lately 1 of my servers wouldn't execute a trade because the low or high of the price wasn't the same due to pricing inaccuracy. I contacted Lmax about the issue and they advised that yes this could happen and to increase accuracy they can increase the number of quotes per second. They said I could go as high as 100 quotes per second but I only went to 50. Could this be a factor? If yes why could this be a factor can Multicharts not handle this many quotes per second?
This was stated by LMAX representatives while we were analyzing the heartbeats issue.- I was never aware that Lmax doesn't support multiple broker profiles, where is this stated?
It works ok for some users, for some users it can cause issues. As it was stated by LMAX representatives -- More on the multiple broker profiles with Lmax. I have 1 server running 2 workspaces. 1 workspace for long trades connected to one account and another workspace for short trades connected to another account. I HAVE to do this because Lmax doesn't support hedging. Can you please clarify that the only way I can continue to trade with Multicharts and Lmax is to have a new server for each account? I.e. 1 server 1 workspace 1 account for long trades and then another server exactly the same but for short trades?
the behaviour can be unstable, unpredictable and unsupported. So you can use it at your own risk.
If you are trading this instrument using this broker profile from one chart in SA mode without placing orders manually - you can check for mismatch between MarketPosition_at_Broker and MarketPosition_at_Broker_for_the_Strategy. But this won't be a simple copy/paste replacement as MarketPosition returns position direction for a given chart, while MarketPosition_at_Broker/_for_The_Strategy return number of contracts and the type of position at the broker for the symbol. So you need to incorporate this reserved words into your strategy logic.4. In my code do I just replace Marketposition with Marketposition_at_broker? Or do I need to add some more smarts like after a trade has been sent then do a check to make sure the position has reached the broker? Any chance you can provide some reliable code for this as I don't want this issue to happen again.
Please see the links from this post.5. I don't really know how I can do this, I can check the logs tab before turning on auto trading but this wouldn't stop this problem happening mid way through trading. Is there a way I can get an email alert for connection issues?
- t-rader
- Posts: 139
- Joined: 02 Feb 2011
- Location: Australia
- Has thanked: 11 times
- Been thanked: 27 times
Re: Sync didn't sync lost me $26,000 USD!!!
The whole connection setup with Lmax seems so flakey, how stable is this new version of Multicharts with Lmax?Since the version you are using we have provided multiple attempts together with LMAX to have this issue resolved. The latest improvements can be found in the current version of MultiCharts. We have also added an option for enabling/disabling heartbeats as LMAX connection works differently for different users - some are ok with it enabled, some need to have it disabled.- Could the constant reconnects be caused by the older Multicharts version?
- The reason I use Multicharts64 8.8 Release 1 Build 8363 is because I tested this version for a long time with no issues. You can even see in this thread that I'm defending Multicharts and Lmax. I mention this exact version and that thread was from last year back in May I firsted commented in this thread http://www.multicharts.com/discussion/v ... ax#p107297
Can you please explain what you mean here a little more? It sounds like it could be an issue?This should not be an issue until you have any realtime quotes in the data queue.- ** I just remembered a couple of weeks ago I had lmax increase their quotes per second. By default Lmax issues 10 quotes per second and they increased this to 50 quotes per second to increase pricing accuracy for me. The reason I did this is because I run 2 servers, a test server in the same data center with the same configuration but connected to a smaller live account. For 1.5 years these 2 servers were executing trades exactly the same with maybe 0 - 0.4 of a pip difference. Lately 1 of my servers wouldn't execute a trade because the low or high of the price wasn't the same due to pricing inaccuracy. I contacted Lmax about the issue and they advised that yes this could happen and to increase accuracy they can increase the number of quotes per second. They said I could go as high as 100 quotes per second but I only went to 50. Could this be a factor? If yes why could this be a factor can Multicharts not handle this many quotes per second?
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Sync didn't sync lost me $26,000 USD!!!
We have not received any complaints regarding this functionality lately.The whole connection setup with Lmax seems so flakey, how stable is this new version of Multicharts with Lmax?
In MultiCharts 8.8 there is "Nq/Ns" indication of quotes' queue status in the bottom right corner of the main MultiCharts window. I.e. the number of quotes in the queue to be processed (in case the quotes number is too large and they are coming too fast and MC can not handle all these these quotes in realtime). If you have "0q/0s" it means everything okay, you have no queue and all the quotes are being processed in time. These numbers will be displayed in red as soon as a queue is formed.Can you please explain what you mean here a little more? It sounds like it could be an issue?
"Nq" stands for the number of quotes in the queue.
"Ns" stands for the number of seconds for which the first quote in the queue is already delayed.
In MultiCharts 9.0 the indicator is not shown while there is no queue (0q/0s), once there is a queue it shows up and is being painted red.
Re: Sync didn't sync lost me $26,000 USD!!!
Hello,
This is an unfortunate situation and am sorry for your loss.
I think as well there are some reliability issues with LMAX. Today I had a Stop Order triggered 3x and I cannot figure out why and how this happened. I do have now 2 positions opened in the contrarian direction of my initial trade. I add a take profit and stop order to each of my trades using GenerateStopLoss and GenerateProfitTarget and my code is as simple as possible. I have the signal running on 32 LMAX symbols and this is barely the 2nd time I witnessed something wrong out of tens of trades (probably random issue that cannot be replicated easily).
This particular time it didn't even cancel the TakeProfit (sell limit is still there).
Please see attached screenshot.
This is an unfortunate situation and am sorry for your loss.
I think as well there are some reliability issues with LMAX. Today I had a Stop Order triggered 3x and I cannot figure out why and how this happened. I do have now 2 positions opened in the contrarian direction of my initial trade. I add a take profit and stop order to each of my trades using GenerateStopLoss and GenerateProfitTarget and my code is as simple as possible. I have the signal running on 32 LMAX symbols and this is barely the 2nd time I witnessed something wrong out of tens of trades (probably random issue that cannot be replicated easily).
This particular time it didn't even cancel the TakeProfit (sell limit is still there).
Please see attached screenshot.
- Attachments
-
- lmax.png
- (60.52 KiB) Downloaded 2894 times
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Sync didn't sync lost me $26,000 USD!!!
Hello Zoli,Hello,
This is an unfortunate situation and am sorry for your loss.
I think as well there are some reliability issues with LMAX. Today I had a Stop Order triggered 3x and I cannot figure out why and how this happened. I do have now 2 positions opened in the contrarian direction of my initial trade. I add a take profit and stop order to each of my trades using GenerateStopLoss and GenerateProfitTarget and my code is as simple as possible. I have the signal running on 32 LMAX symbols and this is barely the 2nd time I witnessed something wrong out of tens of trades (probably random issue that cannot be replicated easily).
This particular time it didn't even cancel the TakeProfit (sell limit is still there).
Please see attached screenshot.
Please send us the information requested in post #4 above + specify IDs of the orders in question.
Re: Sync didn't sync lost me $26,000 USD!!!
Henry, it would be great if you could provide us programmatic access to this in a future release!In MultiCharts 9.0 the indicator is not shown while there is no queue (0q/0s), once there is a queue it shows up and is being painted red.
Re: Sync didn't sync lost me $26,000 USD!!!
LMAX is also my broker and I experienced connection issues last Friday for the first time whilst auto-trading.
I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts. In autotrade properties, orders not filled are set to expire after 5 seconds.
Also, my email alerts went berzerk. Most ticks sent multiple email alerts and one actually sent 24!
I wouldn't expect this to happen considering they are just market orders. I have been auto-trading with LMAX for over a year and never had this issue.
My server is running the latest version of MC (v9) on Windows 2008 server.
I run 2 signals through one broker profile.
Anyone have any ideas?
I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts. In autotrade properties, orders not filled are set to expire after 5 seconds.
Also, my email alerts went berzerk. Most ticks sent multiple email alerts and one actually sent 24!
I wouldn't expect this to happen considering they are just market orders. I have been auto-trading with LMAX for over a year and never had this issue.
My server is running the latest version of MC (v9) on Windows 2008 server.
I run 2 signals through one broker profile.
Anyone have any ideas?
Re: Sync didn't sync lost me $26,000 USD!!!
What is your bar resolution and how many times were they cancelled?I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts. In autotrade properties, orders not filled are set to expire after 5 seconds.
- TJ
- Posts: 7742
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2222 times
Re: Sync didn't sync lost me $26,000 USD!!!
Don't exit your MultiCharts.LMAX is also my broker and I experienced connection issues last Friday for the first time whilst auto-trading.
I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts. In autotrade properties, orders not filled are set to expire after 5 seconds.
Also, my email alerts went berzerk. Most ticks sent multiple email alerts and one actually sent 24!
I wouldn't expect this to happen considering they are just market orders. I have been auto-trading with LMAX for over a year and never had this issue.
My server is running the latest version of MC (v9) on Windows 2008 server.
I run 2 signals through one broker profile.
Anyone have any ideas?
Don't reboot your machine.
Contact tech support so that they can connect to your computer to retrieve the logs and do online diagnostics.
Re: Sync didn't sync lost me $26,000 USD!!!
Before I read this I renamed my log file in an attempt to back it up and rebooted my server but unfortunatey it was deleted in the process.Don't exit your MultiCharts.LMAX is also my broker and I experienced connection issues last Friday for the first time whilst auto-trading.
I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts. In autotrade properties, orders not filled are set to expire after 5 seconds.
Also, my email alerts went berzerk. Most ticks sent multiple email alerts and one actually sent 24!
I wouldn't expect this to happen considering they are just market orders. I have been auto-trading with LMAX for over a year and never had this issue.
My server is running the latest version of MC (v9) on Windows 2008 server.
I run 2 signals through one broker profile.
Anyone have any ideas?
Don't reboot your machine.
Contact tech support so that they can connect to your computer to retrieve the logs and do online diagnostics.
Re: Sync didn't sync lost me $26,000 USD!!!
MAtricks and Orion,What is your bar resolution and how many times were they cancelled?I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts. In autotrade properties, orders not filled are set to expire after 5 seconds.
Bar resolution was hourly and one particular trade tried to exit 3-4 times during the 11am (EDT) bar. Therefore the exit orders were trying to send (or were ignored by LMAX) for 15 seconds.
**Remember "cancel obsolete orders" is set to 5 seconds.
NB. The issue occured on my test server and the trade size was only 1 std lot.
Question: From a 'communication route' perspective, when orders are sent from the MC client to the broker are they routed through any Multicharts hosted middleware or does the MC client communicate orders solely with the broker?
Re: Sync didn't sync lost me $26,000 USD!!!
Given your market timer is set to 5 secs, a 15-20 sec broker connection loss will result in 3-4 retries. We need a new PL keyword broker.connectionStatus to give user code visibility into connection status so that user code may take appropriate action in such cases.
- JoshM
- Posts: 2195
- Joined: 20 May 2011
- Location: The Netherlands
- Has thanked: 1544 times
- Been thanked: 1565 times
- Contact:
Re: Sync didn't sync lost me $26,000 USD!!!
I fully agree with this, it's risk management 101 to have programmatic access to things like this. But I'm sceptical about it being implemented any time soon, since...We need a new PL keyword broker.connectionStatus to give user code visibility into connection status so that user code may take appropriate action in such cases.
Source: broker and datafeed connectivity status event.Unfortunately there is no technical possibility for implementing this functionality at the moment.
Re: Sync didn't sync lost me $26,000 USD!!!
champski, I read your post more carefully. Were all your problems with exits or did you have problems with entries too?one particular trade tried to exit 3-4 times during the 11am (EDT) bar. Therefore the exit orders were trying to send (or were ignored by LMAX) for 15 seconds.
Re: Sync didn't sync lost me $26,000 USD!!!
Orion, I only had issues with the exits.champski, I read your post more carefully. Were all your problems with exits or did you have problems with entries too?one particular trade tried to exit 3-4 times during the 11am (EDT) bar. Therefore the exit orders were trying to send (or were ignored by LMAX) for 15 seconds.
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Sync didn't sync lost me $26,000 USD!!!
What is the State of the orders in File -> New -> Order and position tracker window->Orders tab? Are there any messages corresponding to these orders on the Logs tab?I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts.
If you are referring to "Cancel Market Orders as Obsolete If not Filled Within X Seconds" option then it will cancel the order that was accepted by broker within specified timeout. If MultiCharts was unable to place an order to the broker then there is nothing to cancel.In autotrade properties, orders not filled are set to expire after 5 seconds.
MultiCharts is a standalone application. All modules, configuration parameters and user data (workspaces, studies, data) are kept on your computer. The data/broker connection is performed from your computer directly to your data/broker servers.Question: From a 'communication route' perspective, when orders are sent from the MC client to the broker are they routed through any Multicharts hosted middleware or does the MC client communicate orders solely with the broker?
There are no MultiCharts servers that keep your information or that work as a middleware.
You can consult with your broker directly regarding order processing mechanisms.
Re: Sync didn't sync lost me $26,000 USD!!!
The orders tab shows the "expected" trade entries and exits. Therefore, no signs of issues. The log tab is interesting though. Logging information for the day in which the problem occured (13/02/2015) is completely missing (See attachment). Logging info for the dates post and prior is there however.What is the State of the orders in File -> New -> Order and position tracker window->Orders tab? Are there any messages corresponding to these orders on the Logs tab?I had 2 trades at different times and noticed my entry and exit orders weren't filled on the first few attempts.
If you are referring to "Cancel Market Orders as Obsolete If not Filled Within X Seconds" option then it will cancel the order that was accepted by broker within specified timeout. If MultiCharts was unable to place an order to the broker then there is nothing to cancel.In autotrade properties, orders not filled are set to expire after 5 seconds.
MultiCharts is a standalone application. All modules, configuration parameters and user data (workspaces, studies, data) are kept on your computer. The data/broker connection is performed from your computer directly to your data/broker servers.Question: From a 'communication route' perspective, when orders are sent from the MC client to the broker are they routed through any Multicharts hosted middleware or does the MC client communicate orders solely with the broker?
There are no MultiCharts servers that keep your information or that work as a middleware.
You can consult with your broker directly regarding order processing mechanisms.
Where could it have gone?
NB. I renamed my log file in an attempt to back it up from (%appdata%\TS Support\MultiCharts64\9.0.10360.400\Logs\LMAXServerHost) and unfortunatey it was gone after I restarted the server.
- Attachments
-
- Missing13thFeb.jpg
- (50.55 KiB) Downloaded 2876 times
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Sync didn't sync lost me $26,000 USD!!!
champski,The orders tab shows the "expected" trade entries and exits. Therefore, no signs of issues. The log tab is interesting though. Logging information for the day in which the problem occured (13/02/2015) is completely missing (See attachment). Logging info for the dates post and prior is there however.
Where could it have gone?
Please make sure there are no filters applied to your Order and Position Tracker Logs tab.
In case the issue reappears - please send us the information requested in this post.
Re: Sync didn't sync lost me $26,000 USD!!!
Henry, I checked the Order and Position Tracker Logs tab and made sure there are no filters applied. All options are set to "All" except for the dates.champski,The orders tab shows the "expected" trade entries and exits. Therefore, no signs of issues. The log tab is interesting though. Logging information for the day in which the problem occured (13/02/2015) is completely missing (See attachment). Logging info for the dates post and prior is there however.
Where could it have gone?
Please make sure there are no filters applied to your Order and Position Tracker Logs tab.
In case the issue reappears - please send us the information requested in this post.
Re: Sync didn't sync lost me $26,000 USD!!!
Hi,
I just logged onto my server to give it it's weekly reboot and discovered my account was not showing under accounts in the Order and Position tracker and one of the two workspaces was disconnected from autotrading. I would send you the log file but unfortunately it was 60gb - so I had to delete it.
I just logged onto my server to give it it's weekly reboot and discovered my account was not showing under accounts in the Order and Position tracker and one of the two workspaces was disconnected from autotrading. I would send you the log file but unfortunately it was 60gb - so I had to delete it.
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Sync didn't sync lost me $26,000 USD!!!
Hello champski,Hi,
I just logged onto my server to give it it's weekly reboot and discovered my account was not showing under accounts in the Order and Position tracker and one of the two workspaces was disconnected from autotrading. I would send you the log file but unfortunately it was 60gb - so I had to delete it.
Can you see any messages under File -> New -> Order and position tracker window->Logs tab corresponding to this situation?
Re: Sync didn't sync lost me $26,000 USD!!!
Hi Sorry to hear your problems. I too have experienced this bug frequently & in fact it happened again today. I you refer to the enclosed pic, you can see my position is - 2 lots, yet the Master Strategy has put in orders for a total of 4.
The interesting thing is that I trade from 2 separate environments - home & office, using different PC's and installations of MultiCharts and I get this bug occurring from both locations. I re-start my PC every day & MultiCharts through out the session.
This leads me to believe that the issue is a bug with MultiCharts, rather than at my end. Its a pity
because the MasterStrategy idea is good, but its too unreliable so now I just enter my orders through I.B. (the example I enclosed in the pic was during Sim trading).
The interesting thing is that I trade from 2 separate environments - home & office, using different PC's and installations of MultiCharts and I get this bug occurring from both locations. I re-start my PC every day & MultiCharts through out the session.
This leads me to believe that the issue is a bug with MultiCharts, rather than at my end. Its a pity
because the MasterStrategy idea is good, but its too unreliable so now I just enter my orders through I.B. (the example I enclosed in the pic was during Sim trading).
- Attachments
-
- MC orders.JPG
- (190.59 KiB) Downloaded 2881 times
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Sync didn't sync lost me $26,000 USD!!!
Hello GTC,Hi Sorry to hear your problems. I too have experienced this bug frequently & in fact it happened again today. I you refer to the enclosed pic, you can see my position is - 2 lots, yet the Master Strategy has put in orders for a total of 4.
The interesting thing is that I trade from 2 separate environments - home & office, using different PC's and installations of MultiCharts and I get this bug occurring from both locations. I re-start my PC every day & MultiCharts through out the session.
This leads me to believe that the issue is a bug with MultiCharts, rather than at my end. Its a pity
because the MasterStrategy idea is good, but its too unreliable so now I just enter my orders through I.B. (the example I enclosed in the pic was during Sim trading).
Our engineers are analyzing your case.
- t-rader
- Posts: 139
- Joined: 02 Feb 2011
- Location: Australia
- Has thanked: 11 times
- Been thanked: 27 times
Re: Sync didn't sync lost me $26,000 USD!!!
After taking a break since getting smashed by a Multicharts/Lmax issue! I'm getting back into trading and need the help of the community and MC moderators to ensure this issue doesn't happen again.
I've updated to the latest MC version 9.0 Build 10759.
With all the latest updates to this version of Multicharts, how do I monitor the broker connection status as I was advised to do:
In response to this question previously Henry linked me to the follow thread:
which firstly seems old as it states that Autotrading turns off when broker connection is lost, which never happened for me and the solution provided is to monitor if Autotrading is turned off. So this solution seems outdated and of no use now, unless I'm reading this wrong?
I would also like to confirm as MATricks mentioned:
I've updated to the latest MC version 9.0 Build 10759.
With all the latest updates to this version of Multicharts, how do I monitor the broker connection status as I was advised to do:
As Henry advised that the issue was caused by:5. Monitor Order and Position Tracker Logs tab for errors;
so specifically how do I send an email alert when the Order and Position Tracker are reporting connection loss, reconnecting, connection established?Our developers have studied the logs from your PC. From the logs we can see that there were issues with connection to LMAX servers for multiple days before the orders were placed (constant reconnects starting 21-01-2015 14:37:03 until 23-01-2015 22:07) – you can see the corresponding messages in Order and Position Tracker Logs tab: Reconnecting ...Connection established. Reconnecting ....
The orders were placed between the reconnects and no order confirmation has been received from the broker (you can see the corresponding messages in Order and Position Tracker Logs tab: Didn't receive final status for order … ).
In response to this question previously Henry linked me to the follow thread:
This thread then has a link to this thread http://www.multicharts.com/discussion/v ... f=5&t=9890Please see the links from this post.5. I don't really know how I can do this, I can check the logs tab before turning on auto trading but this wouldn't stop this problem happening mid way through trading. Is there a way I can get an email alert for connection issues?
which firstly seems old as it states that Autotrading turns off when broker connection is lost, which never happened for me and the solution provided is to monitor if Autotrading is turned off. So this solution seems outdated and of no use now, unless I'm reading this wrong?
I would also like to confirm as MATricks mentioned:
Would setting my trade size and the setting 'maximum shares/contracts per position' via the Strategy properties as opposed to via my code prevent the issue of my trade doubling up?-How many contracts do you have in the Maximum Shares input?
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Sync didn't sync lost me $26,000 USD!!!
Hello t-rader,
There is no prebuilt alert functionality for Order and Position Tracker window messages. You can create your own script that will monitor MultiCharts logs (LMAXServerHost_NET_Trace.txt) for the following message (or part of it): LMAXServerHost.exe Information: 0 : Connection changed: from Connected to ReConnecting similar to what Faraz has described in this post.
There is no prebuilt alert functionality for Order and Position Tracker window messages. You can create your own script that will monitor MultiCharts logs (LMAXServerHost_NET_Trace.txt) for the following message (or part of it): LMAXServerHost.exe Information: 0 : Connection changed: from Connected to ReConnecting similar to what Faraz has described in this post.
This option can be useful both in backtesting and live trading. I would also recommend adding market_position_at_broker check to your strategy to avoid generation of unnecessary orders as I have mentioned in one of my previous posts. Please keep in mind that if there are multiple orders too close to market and market moves fast then there is a chance that both orders are filled before MultiCharts sends cancel command.I would also like to confirm as MATricks mentioned:
MAtricks wrote:
-How many contracts do you have in the Maximum Shares input?
Would setting my trade size and the setting 'maximum shares/contracts per position' via the Strategy properties as opposed to via my code prevent the issue of my trade doubling up?