real-time failure alerts

Questions about MultiCharts and user contributed studies.
User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 30 times

real-time failure alerts

Postby PatrickSocal » 16 Oct 2014

For those interested in receiving real-time failure alerts when problems occur with autotrading, I just created this topic:
viewtopic.php?f=1&t=47391
Please have a look and contribute if you can :)

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Getting MC to send an email alert

Postby TJ » 16 Oct 2014

For those interested in receiving real-time failure alerts when problems occur with autotrading, I just created this topic:
viewtopic.php?f=1&t=47391
Please have a look and contribute if you can :)
Most of the failure is due to local internet disconnect... when the internet is down you can't send email alerts. No amount of MultiCharts functions can save your bacon if you are not there.


(unless you have a cell phone built into your computer and you can get MultiCharts to call you.)

User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 30 times

Re: Getting MC to send an email alert

Postby PatrickSocal » 16 Oct 2014

Most of the failure is due to local internet disconnect... when the internet is down you can't send email alerts. No amount of MultiCharts functions can save your bacon if you are not there.
Yes, this definitely happens. (And so do lots of other problems.) Have a look at the topic I referenced just above... it attempts to deal with this problem and others. A remote (highly-available) service hosted in the cloud will do the alerting, if/when it hasn't heard from MultiCharts or the TradingServer in a configurable period of time.

If you get such an alert, and find your TradingServer is down or has lost its net connection, at least you can connect to your broker from your laptop machine and manage your positions. You can also call your sysadmin and have him debug or restart your TradingServer. Not a fun scenario, but better than blissful ignorance.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Getting MC to send an email alert

Postby TJ » 16 Oct 2014

Most of the failure is due to local internet disconnect... when the internet is down you can't send email alerts. No amount of MultiCharts functions can save your bacon if you are not there.
Yes, this definitely happens. (And so do lots of other problems.) Have a look at the topic I referenced just above... it attempts to deal with this problem and others. A remote (highly-available) service hosted in the cloud will do the alerting, if/when it hasn't heard from MultiCharts or the TradingServer in a configurable period of time.

If you get such an alert, and find your TradingServer is down or has lost its net connection, at least you can connect to your broker from your laptop machine and manage your positions. You can also call your sysadmin and have him debug or restart your TradingServer. Not a fun scenario, but better than blissful ignorance.
If you are looking for a heartbeat monitor... there are lots of solutions already available on the market, both free and paid. Why bog down MultiCharts to do something that is not trading related?

User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 30 times

Re: Getting MC to send an email alert

Postby PatrickSocal » 16 Oct 2014

If you are looking for a heartbeat monitor... there are lots of solutions already available on the market, both free and paid. Why bog down MultiCharts to do something that is not trading related?
Lots of reasons. For example, last week MC entered 2 positions that it shouldn't have. There was a 1-day gap in the backfill data on a few of our charts (we have 12 up at the moment) so the indicators were wrong. Our strats therefore got spurious entry signals, and we ended up with 2 losing trades. That's failure mode A3 in my list. A heartbeat monitor would not have caught that. It requires a signal or indicator running in MC to detect the missing bars.

And a few weeks before that, we were running a strat on tick-based bars when the market was active. MC got bogged down. Like really bogged down. The chart was 20-30 minutes behind the market. Not milliseconds. Minutes. It placed some orders it had no business placing, and we had to clean up the mess. That's failure mode A2. A strat or an indicator that compared computer clock to data clock could have caught the error. Nothing outside MC could have.

And on several other occasions, we had orders wrongly rejected. Our broker uses position limits and counts working orders plus open positions against them. We placed OCO exit orders, the broker (improperly) counted the exit orders against our position limits, and rejected our exit orders. This is a bug/limitation in the MC/Rithmic interface, which has been reported and acknowledged but not yet fixed. This is failure mode A4, and it could have been caught immediately with the framework I propose.

We've also had our server lose its connectivity when our provider was hit by a DDoS attack. And we've had it fail to restart due to an anti-malware conflict. And we've had MultiCharts use up all the memory on a machine and require a forced reboot. And we've had latency shoot up when our provider installed a new firewall. All these things are real and have happened to us. And that's what this framework is designed to detect and alert.

No system can be totally foolproof, but we are tired of all these system failures and we feel compelled to get on top of them.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Getting MC to send an email alert

Postby TJ » 16 Oct 2014

If you are looking for a heartbeat monitor... there are lots of solutions already available on the market, both free and paid. Why bog down MultiCharts to do something that is not trading related?
Lots of reasons. For example, last week MC entered 2 positions that it shouldn't have. There was a 1-day gap in the backfill data on a few of our charts (we have 12 up at the moment) so the indicators were wrong. Our strats therefore got spurious entry signals, and we ended up with 2 losing trades. That's failure mode A3 in my list. A heartbeat monitor would not have caught that. It requires a signal or indicator running in MC to detect the missing bars.
And a few weeks before that, we were running a strat on tick-based bars when the market was active. MC got bogged down. Like really bogged down. The chart was 20-30 minutes behind the market. Not milliseconds. Minutes. It placed some orders it had no business placing, and we had to clean up the mess. That's failure mode A2. A strat or an indicator that compared computer clock to data clock could have caught the error. Nothing outside MC could have.
And on several other occasions, we had orders wrongly rejected. Our broker uses position limits and counts working orders plus open positions against them. We placed OCO exit orders, the broker (improperly) counted the exit orders against our position limits, and rejected our exit orders. This is a bug/limitation in the MC/Rithmic interface, which has been reported and acknowledged but not yet fixed. This is failure mode A4, and it could have been caught immediately with the framework I propose.
We've also had our server lose its connectivity when our provider was hit by a DDoS attack. And we've had it fail to restart due to an anti-malware conflict. And we've had MultiCharts use up all the memory on a machine and require a forced reboot. And we've had latency shoot up when our provider installed a new firewall. All these things are real and have happened to us. And that's what this framework is designed to detect and alert.
No system can be totally foolproof, but we are tired of all these system failures and we feel compelled to get on top of them.
What you are talking about are your trading management issues, your data integrity issues, your computer hardware issues, your virus problem, and your 3rd party issues, that are remotely related to MultiCharts.

You can dream up a wish list of all kinds that can auto-fix everything. But at the end of the day, trading is a business that is just like another business -- you are required to pull up your shirt sleeves and monitor it like a hawk. The devil is in the details, and it is up you to make it happen. I wish you in your endeavor.

User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 30 times

Re: Getting MC to send an email alert

Postby PatrickSocal » 16 Oct 2014

Hi TJ,

I'm trying to improve the platform for people who actually use it for autotrading. Our sleeves are pulled up and we are watching it like a hawk. But even hawks need to sleep. (Whether they wear shirt sleeves or not!)

Anyway, this is not a wishlist. It is a development plan. It involves no auto-fix code, but it does involve auto-alert code, so we can jump on critical issues right when they happen.

If you or others want to participate or find it valuable, I'm open to sharing my work. Otherwise, feel free to ignore it.

User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 30 times

Re: Getting MC to send an email alert

Postby PatrickSocal » 16 Oct 2014

I intend to contribute code, related to this topic, for those who are interested. If you see that as a hijack, then I apologize.


Return to “MultiCharts”