Failure Alerts During Real-Time Automated Trading

Questions about MultiCharts and user contributed studies.
User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 29 times

Failure Alerts During Real-Time Automated Trading

Postby PatrickSocal » 16 Oct 2014

I believe MultiCharts needs a robust health monitoring system to detect and report problems during automated trading. There are many ways for an ATS platform to fail, and there are many people who rightly demand that an ATS be monitored around the clock. (For example: institutional customers, risk departments, new regulations, and if it's your money, you should want it too!)

I propose to help create such a system for MultiCharts, perhaps with the help of the community here, for my own use and for the use of those who contribute. I've spent many hours researching options, defining requirements, and sketching out an initial design. I'll start out by describing the system I have in mind.

PROBLEM DEFINITION =================================
A__ Trading strategies and platforms can fail for many reasons:
(1) Real-Time Data interruption
(2) MC bogs down, and stops recalculating the signal often enough
(3) On startup, large gaps exist in backfill data resulting in wrong indicators and signals
(4) Orders are rejected by the broker
(5) MC process hangs and stops responding altogether
(6) Host machine loses all internet connectivity, or has intermittent outage (e.g. DDoS attack)
(7) Machine crashes

SOME REQUIREMENTS =================================
B__ When a critical failure happens, a responsible person should be alerted immediately, no matter the time of day or night.

C__ The trading server and the monitoring server must be different machines
1) If they are not, there is no way to detect things like A6 & A7
2) Monitoring machine should be high-availability with guaranteed uptime
3) Monitoring machine should be responsible for sending alerts via email, text, phone

D__ Monitoring must occur within the MultiCharts process, and also external to it
1) Internal can catch A1, A2, A3, A4
2) External can catch A5 and A2
3) Remote monitoring framework *must* have useable plugin architecture to enable it to receive updates from MultiCharts and to apply decision logic to those updates.

DESIGN DISCUSSION - REMOTE MONITORING SERVICE ==========================
I looked into a number of remote monitoring solutions for servers and networks and applications.
Leading contenders in the self-hosted open-source space were Zabbix, Munin, Nagios, with Zabbix being my favorite. However, any self-hosted solution will take time and effort to administer (they are Linux based), and unless it's deployed on a redundant cloud server like AWS or Google it won't satisfy requirement C2.
So I turned to hosted solutions. In that category, I found two good options: LogicMonitor and ServerDensity. Both are cloud based, with multiple replicated server locations, and both appear to offer the necessary features, for an affordable price. I installed them both on my Windows server in a few minutes time, and started evaluating them. After trying their features, reading their API docs, and stumbling through their bugs & limitations, I chose ServerDensity.

DESIGN DISCUSSION - TRADING SERVER =================================
I've thought of a modular design consisting of a Probe, a Collector, and a Plugin.
E__ ChartProbe
1) Implemented as a PowerLanguage indicator, applied to each chart on which an ATS runs.
_i) Fires using RecalcLastBarAfter() once per minute, to match the frequency of ServerDensity
2) Checks for A1-A3
3) Takes 2 inputs: StrategyName, GV_Address (an address in global variables to store the status)
4) Written to be as fast and efficient as reasonably possible, b/c it will run in all the chart windows

F__ ChartCollector
1) Implemented as a PowerLanguage indicator, applied to one low-frequency chart
_i) Fires using RecalcLastBarAfter() once per minute, to match the frequency of ServerDensity
2) Checks Order & Position Tracker for failure mode A4 (possible in upcoming MC release 9.1 I think)
3) Collects all the strat names and status values from the individual ChartProbes, by reading the values in their GV_Addresses
4) Takes inputs: GV_Address list for strategies it will monitor
5) Writes a text file with the names and status of all the strats, together with a timestamp

G__ Plugin
1) Implemented as a C# DLL according to the ServerDensity API
_i) Called once per minute by the ServerDensity agent, running on the TradingServer
2) Reads the text file F5 written by ChartCollector
3) Decides if any strats are reporting a critical error
4) Checks if Timestamp of text file is too old, indicating that MC is not updating fast enough
5) Reports critical errors to ServerDensity web service hosted in the cloud
_i) Includes descriptive text message and strategy name if possible

H__ ServerDensity Web Service
1) Receives updates from Plugin
2) Receives performance metrics from Windows system counters, etc.
_i) CPU load, Memory usage, Network throughput, Ping time
3) User uses GUI to set conditions for alerts
_i) CPU too high, critical error in Plugin, Server not responding, etc.
4) Alert conditions trigger messages to responsible persons via email, text, phone

LIMITATIONS ========================================
I don't think this design works for strategies run on the Portfolio Trader, because I don't think the PortfolioTrader can run indicators. Perhaps a PortProbe and PortCollector can be designed and implemented as signals instead.


SUMMARY ==========================================
I've just described a preliminary design for a remote health-monitoring system for MultiCharts, to detect problems during automated trading. I believe such a function is necessary. The failure modes I've listed in A above are all *real* things that have happened to us since we went live. They have cost us real money. We have been fortunate up to now, that none of these things have resulted in total disasters. But I don't want to push my luck!

I would love it if the team at MultiCharts would support this effort, and maybe even build some of the features into MC. If not, I'm prepared to do it myself. I've written some of this stuff already, and I'm gradually assembling the code I need to get the rest of it done. If you see any flaws or have any suggested improvements, I'd love to hear from you. And if anyone thinks this is interesting and/or wants to contribute, please let me know.

Thanks!
These users thanked the author PatrickSocal for the post (total 8):
MAtricksmosesMeecchilbertthomashckwavesliderfurytraderkiasom

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 279 times

Re: Failure Alerts During Real-Time Automated Trading

Postby MAtricks » 16 Oct 2014

THIS is a great idea!

User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 29 times

Re: Failure Alerts During Real-Time Automated Trading

Postby PatrickSocal » 16 Oct 2014

BTW, Here are some prior posts that are relevant to this topic:

viewtopic.php?f=5&t=9890

viewtopic.php?f=5&t=11715

viewtopic.php?f=5&t=46551

viewtopic.php?f=5&t=46553

http://www.forexfactory.com/showthread.php?t=279654

I've read them and tried to incorporate their lessons into my design above. If anyone knows of some other good posts, either here or on an external forum like BMT, EliteTrader, StackXXX, or whatever, please feel free to post them here. Thanks.

User avatar
JoshM
Posts: 2099
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1527 times
Been thanked: 1502 times
Contact:

Re: Failure Alerts During Real-Time Automated Trading

Postby JoshM » 16 Oct 2014

Interesting idea/suggestion that certainly has value. It seems that you have thought it out very well, so I don't see any flaws in your reasoning. Thanks for your layout of the problem and potential solutions.

For the moment I'm personally not interested in contributing from the programming side of things for a few reasons, but am going to keep a close eye on it. :)

---
Edit:
I'm wondering what the additional benefit is of something like ServerDensity given the disadvantages of additional complexity (i.e., maintenance, development time, the temporary failure/unavailability of the ServerDensity service) and direct costs of ServerDensity.

Azure, for example, already offers alerts and monitoring (see here), so hosting an ATS on an Azure instance reduces what, from my standpoint, is additional complexity introduced by a service like ServerDensity.
These users thanked the author JoshM for the post:
PatrickSocal

User avatar
bensat
Posts: 331
Joined: 04 Oct 2014
Has thanked: 46 times
Been thanked: 104 times

Re: Failure Alerts During Real-Time Automated Trading

Postby bensat » 16 Oct 2014

@PatrickSocal

If you have all these requirements you are asking for implementation, you are using the wrong environment.
These users thanked the author bensat for the post:
TJ

User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 29 times

Re: Failure Alerts During Real-Time Automated Trading

Postby PatrickSocal » 16 Oct 2014

JoshM wrote:I'm wondering what the additional benefit is of something like ServerDensity given the disadvantages of additional complexity (i.e., maintenance, development time, the temporary failure/unavailability of the ServerDensity service) and direct costs of ServerDensity.

Azure, for example, already offers alerts and monitoring (see here), so hosting an ATS on an Azure instance reduces what, from my standpoint, is additional complexity introduced by a service like ServerDensity.

Interesting point... but we prefer not to host our ATS in the cloud for reasons of hardware latency and proximity to the exchange. We have a dedicated server that's in a good physical location and even so, we sometimes see latency issues.

Also, direct cost of ServerDensity to monitor only 1 machine is just $10 per month, and they are even offering a promotion (together with google cloud) to make that free. Plus I think their uptime is excellent, in part because they host their service at google, softlayer, AWS, and others to get cross-cloud redundancy. Finally, development time for me is 0 to monitor basic "server health" type metrics... the Plugin development is necessary because we want to monitor the insides of MultiCharts (see failure modes A1-A4), and we think that's an essential piece.

bensat wrote:If you have all these requirements you are asking for implementation, you are using the wrong environment.

Also an interesting point... If you can suggest an ATS platform of comparable power in a similar price range, I'll consider it. But please PM me so this thread can stay on topic.

User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 29 times

ServerDensity Screenshots

Postby PatrickSocal » 16 Oct 2014

I thought I'd provide a glimpse of what I'm monitoring on our TradingServer using this remote cloud-hosted monitoring service. This shows some metrics over the last 12 hours, at a reporting interval of once per minute:

Image

We had no major problems in the last 12 hours... just a busy time in the morning when we were getting lots of ticks, and MC was using lots of CPU. (And probably lagging, but I haven't written the probe to measure that yet.) Later in the day after the Globex market close, we do our daily MC restart (it will eventually crash if we don't do this), and you can see that too.

Here is a screen where we can configure alerts:

Image

There are many more we can add, but we are just testing things now. In particular, I'll write a C# plugin that aggregates all the probe-status info from the individual strategies. It will report the metrics and failures to ServerDensity, which will trigger alerts if things are out of range. Also, the ServerDensity agent will "latch" alert conditions, so even if they last only X seconds within the 1 minute reporting interval, they will be reported. (And X is configurable.)
Attachments
SD_Device_Alerts.png
SD_Device_Alerts.png (91.91 KiB) Viewed 5300 times
SD_Device_Metrics.png
SD_Device_Metrics.png (150.1 KiB) Viewed 5293 times

User avatar
bensat
Posts: 331
Joined: 04 Oct 2014
Has thanked: 46 times
Been thanked: 104 times

Re: Failure Alerts During Real-Time Automated Trading

Postby bensat » 17 Oct 2014

@PatrickSocal

I may confused you and your intro post may confused me a little bit. I understood your intro post you are demanding Multicharts to implement all the the monitoring features you are asking for.

I took it from this comment, but may I'm wrong :

I would love it if the team at MultiCharts would support this effort, and maybe even build some of the features into MC.


In my opinion, implementing a full reliable and stress-resistant ATS monitoring system for Multicharts (even Multicharts on it's own) for the price it is asking for and the current support MC is giving would be a joke. I'm more asking fixing basic buggy stuff. This should be worth the price and service I was asked and I'm asking for.

There must be a reason other seriously (maybe pro) reliable ATS's are at that higher prices. They are worth the price because of reliability, feature rich & support or Multicharts is way toooo cheap.

Your choice.

I like you intention to build up a monitoring system by yourself and I support the way you are thinking and doing it. Please don't get me wrong on that point. Thank you.

Regards.

Ben
These users thanked the author bensat for the post:
TJ

faraz
Posts: 140
Joined: 25 Feb 2011
Has thanked: 26 times
Been thanked: 56 times

Re: Failure Alerts During Real-Time Automated Trading

Postby faraz » 17 Oct 2014

Dear Traders,

If you are interested in solutions of these issues then I believe you have already completed designing your Strategies and would like to go live trading with one more automated step. First Congratulations on your hard work and welcome to the dreamed level. Based on my monitoring setup I recommend the following;

a) I use windows build in dos language for monitoring. Command line is very powerful and provides ever thing you need as a professional. I have one batch file which I run and it do all the these below things for me. I mentioned different portions of that file so you can write your own as per your requirement, need, situation and windows setup. You can use this dos form to get help if require.
http://www.dostips.com/forum

b) End of Day and Low Frequency strategies require low monitoring as compare to High Frequency strategies. You can set monitoring check level from few seconds to once a day whatever you desire.

c) Solid, professional hardware setup, 24hr support, alternative internet and power backups and co-location is first priority. I recommend Dedicated server only (VPNs are not reliable) as they gave highest uptime and rarely go down, near your exchange to keep the latency low. Rithmic R|Trader provide a feature to check latency level. Based your requirement you could choose rithmic data center dedicated server as well. I recommend Solid State hard drive (to remove the hard drive issues) with good ram and processor as per your strategy requirements. If one internet connection is down then you will be directed to 2nd backup internet connection automatically within seconds so your trading would get minimum effect. If main power is down then backup power is up. Further more professional networking engineers monitor all data center things on 60 second interval. If your hardware is faulty then you get replacement in minutes. Also you get 24 hrs support 365 days for any issues and technical help. Check these;

Dedicated servers
1. Specially for Trading purposes
a. http://mytradehost.com/ (Chicago)
b. http://steadfast.net/solutions/trading-financial.php (Chicago)
c. http://old.steadfast.net/services/tradi ... ervers.php
d. http://www.turnkeyinternet.net/high-spe ... r-desktop/
e. http://www.colocationamerica.com/dedica ... icated.htm

2. CME Aurora
a. http://www.eotpro.com/products-colocation.php
b. http://agnfutures.com/products-services ... solutions/
c. http://www.bluedotservers.com/Financial ... rvices.php
d. http://www.spalgo.com/
e. http://www.vdiware.com/datacenter-locations
f. http://www.dormantrading.com/CME-Data-Center/
g. http://www.algoadvantage.com/

3. Chicago Dedicated servers
a. http://www.singlehop.com/
b. http://fdcservers.net/dedicated_servers.php
c. https://www.ubiquityservers.com/dedicated-servers
d. http://www.colocationamerica.com/dedica ... icated.htm
e. http://www.ayksolutions.com/datacenters.htm
f. http://chicagodedicatedservers.com/dedicated.html
g. http://chromobyte.com/bds
h. http://www.pingpipe.com/chicago-dedicated-server.php
i. http://www.softsyshosting.com/Dedicated.aspx


d) I always make sure when Multicharts is re-started it kill all its older instances for a proper clean fresh start. So this code which you can found on this page
http://www.multicharts.com/discussion/viewtopic.php?f=5&t=46468

e) Multicharts uses windows time clock so our clock must be accurate so I use code on this link
http://www.multicharts.com/discussion/viewtopic.php?f=5&t=46553

f) For multicharts order “Rejected” status alert generation I use dos code on this link
http://www.multicharts.com/discussion/viewtopic.php?f=5&t=47400&p=109476#p109476

g) for multicharts and all required softwares “NOT RESPONDING” status check, I use code on this link
http://www.multicharts.com/discussion/viewtopic.php?f=5&t=46551

h) For Dedicated Server failure monitoring I use this 3rd party website which check my server from 3 different locations
http://www.binarycanary.com/en/feature-tour/high-speed-website-monitoring.cfm

i) For issuing SMS alerts I use this web site;
https://bulksms.vsms.net/login.mc

J) I use this webserver to upload live still picture screen shots of my desktop and log files, history pic update etc which then I monitor from my mobile and PC without getting logged in to Dedicated Server. This increase quick and easy monitoring level.
http://www.aprelium.com/

I use this dos code to keep my windows desktop active while I am not logged in so I don’t get Black pictures

Code: Select all

 tscon 2 /dest:console


For screen shots generation I use this command line software
[url] http://www.ducklink.com/free-command-li ... apture.php
[/url]

k) For emergency remote control I use this software on my mobile
http://www.splashtop.com/personal

l) Then for some special repeated tasks like exiting not filled limit orders after market close or checking windows clock “Successful” update status, updating OPT dates etc or error and bugs handling of multicharts this simple mouse recorder is the real quick solution, which can read images and do different tasks as per different situations. It can issue alerts for almost all Multicharts error, issues, etc. It can do anything which you want. This is the real quick solution solver on your figure tips.
https://www.jitbit.com/macro-recorder/

m) If you are using Limit/Stop/EOD/SetExitOnClose then you should know that all Multicharts order stay active for 5 minutes after bar close or market close or custom session close and you can change this 5 minute from this path
Registry value: HKEY_CURRENT_USER\Software\TS Support\MultiCharts64\Shaper\CloseBarTimeout
This means if you want to cancel you unfilled limit/stop orders after market close then it should after this 5 minute time interval. Or you need to do some thing in your strategy code to cancel these limit / stop orders before close time. Other wise it would he likely that you get unusual fills.

n) To avoid possible Rejected order for market orders, I suggest enable “cancel market orders as obsolete if not filled within 10 seconds” in strategy properties > auto trading

o) To avoid possible Rejected order generation Sequence, I recommend to always enable “Stop auto-trading after x rejected orders” in Strategy properties > AutoTrading

p) Then at the end of my batch file I use Multicharts copy log files backup command if alert is issued. To later send it to support time for viewing for tracing and fixing the issue.

q) I recommend to refresh all data at the start of workspace once either full or for last 7 days etc as per your data settings and requirements.

r) I recommend a re-start of your Dedicated server once a week. I found system gets slow if it runs more than 2 weeks. A re-start release the memory and fix the issue.

s) Small maintains for server like updating windows, defragmentation, cleaning extra files etc once 2 in a month would be fine.

t) Do not put over load on your trading server with extra internet usage, software installation, browsing etc. Only install trading related required software and windows features only.

u) Above all that I recommend 1 computer at your home/office to be available as backup plan. This computer should only be used for emergency purposes.

v) I recommend close monitoring your strategy for 3-6 months with 1 contract only as I found a lot of differences between Back Testing and Real Trading.

w) Furthermore be very careful ever time you update Multicharts. Multicharts new updates always create issues for complex strategies and some time require adjustments in strategy code and settings.

x) I recommend that you Understanding your strategy and increase your personal monitoring at the times when strategy most likely to place orders.

y) To keep the things understandable and for tracing faults, keep Your Trading server allowed access limited to 1 or 2 persons. Your monitoring webserver address should also be limited persons. Once I overloaded my webserver and it created issues.

z) Regularly checking multicharts OPT remarks, your monitoring file log file and windows error logs in eventviewer will help you fix issue before they happen.

I did not liked 3rd party monitoring software. I wanted EXACT information that I needed so I written my own software as per my desire. It would take some time but once it is set, you are fully automated the dream will come true and then you will just sit, relax and wait for the fish.

I hope this info will save all traders time and efforts.
Successful Trading!!!

Thanks
These users thanked the author faraz for the post (total 9):
PatrickSocalMAtricksMeecchilbertAndrew MultiChartsWally_ADzfsamzfsamfurytraderkiasom

User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 29 times

Re: Failure Alerts During Real-Time Automated Trading

Postby PatrickSocal » 17 Oct 2014

Thank you very very much faraz! There are so many good ideas in here! It's great to hear from someone who is facing the same issues that we are, and has developed so many good solutions. Congratulations!

I'll need some time to digest all of the ideas you've posted here... I'll be checking the links you provided, and looking at some of the service offerings. It sounds like you already have your systems functioning fine... but if you can benefit from any of the code I develop I'm happy to share with you.

I've already developed indicator code to catch a few of the failure modes in my original post, and I expect I'll soon be able to detect A1-A3 on time bars, tick bars, renko bars, and range bars. Let me know if you need something like that?

Again, thanks for the great post, and the spirit of collaboration.

faraz
Posts: 140
Joined: 25 Feb 2011
Has thanked: 26 times
Been thanked: 56 times

Re: Failure Alerts During Real-Time Automated Trading

Postby faraz » 18 Oct 2014

PatrickSocal wrote:Thank you very very much faraz! There are so many good ideas in here! It's great to hear from someone who is facing the same issues that we are, and has developed so many good solutions. Congratulations!

I'll need some time to digest all of the ideas you've posted here... I'll be checking the links you provided, and looking at some of the service offerings. It sounds like you already have your systems functioning fine... but if you can benefit from any of the code I develop I'm happy to share with you.

I've already developed indicator code to catch a few of the failure modes in my original post, and I expect I'll soon be able to detect A1-A3 on time bars, tick bars, renko bars, and range bars. Let me know if you need something like that?

Again, thanks for the great post, and the spirit of collaboration.



You are all most well come.

I suggest not to use MC indicators to check failure modes of multicharts unless you can not do it from out side of MC. Because;
a) Priority!!!! Must keep MC as light as possible for fast execution of trades. Adding additional tasks will create issues and execution Clock speed will eventual gets effected. When you run over few days Multicharts continuously then you will see the real difference increasing in milliseconds v9.0 now show us clearly how we effect clock speed with our extra tasks and then eventually Profit percentage gets effected.

I already submitted a strategy to "Check your Strategy calculation time in MilliSeconds"
viewtopic.php?f=5&t=46604&p=104988#p104988
this test will give you an idea that we must keep MC as light as we can for faster executions. MC is not designed for faster execution and professional High frequency traders are now working on low latency, faster, micro second execution platforms, for an idea Search Internet for "Low latency trading software or solutions".

b) When we are monitoring a software health, I like to do it from a 3rd party software which should be very light, low memory requirement, bug free, reliable, trust able, no crashing, no hanging issues and so far dos command line is providing all these features to me.

c) keeping the MC at minimum will help you and MC support team to figure out the error issues from log files. When you are in Live trading it is very important to get things solved at minimum duration.

If you INCLUDE all codes I mentioned then You will see that you need nothing more and it is so power full that it can cover every single failure error of MC from OUTSIDE of MC as a 3rd party monitoring. I would like to hear if you can tell me any single ERROR/Issue which my above points does not provide a solution in it.

Thanks
These users thanked the author faraz for the post (total 2):
TJfurytrader

User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 29 times

Re: Failure Alerts During Real-Time Automated Trading

Postby PatrickSocal » 19 Oct 2014

faraz wrote:I would like to hear if you can tell me any single ERROR/Issue which my above points does not provide a solution in it.

Hi Faraz,

One example of a fault that I don't know how to detect outside MultiCharts is incomplete data loading. For example, recently when we started MC on a Sunday night, some of the charts failed to load the backfill data from Friday. So there was a 1 day gap. The signals on those charts fired spurious entry orders and we got into two bad trades. It was several hours before we found the error, because we have lots of charts up. Oops.

So I'd like to detect unexpected data gaps on the chart in the most computationally efficient way I can. This really only needs to be done at strategy startup. I'm working on code for it now, and I have it partially solved.

Do you know of a way to detect this type of condition outside of MultiCharts?

faraz
Posts: 140
Joined: 25 Feb 2011
Has thanked: 26 times
Been thanked: 56 times

Re: Failure Alerts During Real-Time Automated Trading

Postby faraz » 20 Oct 2014

PatrickSocal wrote:
faraz wrote:I would like to hear if you can tell me any single ERROR/Issue which my above points does not provide a solution in it.

Hi Faraz,

One example of a fault that I don't know how to detect outside MultiCharts is incomplete data loading. For example, recently when we started MC on a Sunday night, some of the charts failed to load the backfill data from Friday. So there was a 1 day gap. The signals on those charts fired spurious entry orders and we got into two bad trades. It was several hours before we found the error, because we have lots of charts up. Oops.

So I'd like to detect unexpected data gaps on the chart in the most computationally efficient way I can. This really only needs to be done at strategy startup. I'm working on code for it now, and I have it partially solved.

Do you know of a way to detect this type of condition outside of MultiCharts?



PatrickSocal

You should consider the above point number (c) for a clean solid hardware setup. This issue will cover 90% in that point.

I have already mentioned the solution for this issue in the above list. Check (q) point. You do not need any coding. Also I do not suggest to identify data issues through MC code. Because to identify data issues you must have a clean data first from which you can match your data to see if there is any issue in your data or not. With out matching data you can not be sure if your data got issues or not. So quick reliable and easy way is that you refresh your all data or The days which are identical data on your and data feed provider if you got some back adjusted data running on your data file. You need to do this once at the start of your trading day. This will wipe out all your older data and put all the latest data so if you have any gaps or unusual price on bar it will fix them all. After that you will be 100% confident that all your data is clean now. Reduce the number of bars as minimum as your strategy requires for a quick data Refreshing. My Dedicated Server provide me 100 Mbps speed so its just a second to refresh my charts data.

Now you need a alert mechanism out side MC to watch Order and Position Tracker (OPT) all the time and if your connection to your data feed is broken and OPT issues error then you need to personally see if you have a Internet issue or its a minor Internet switching between 2 Internet backup lines etc and then you will ignore it if its just for few seconds. But if this broken error duration is huge then you need to fix your connection and reload your data. You can write a alert mechanism as per my above mentioned instructions list.

Thanks

User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 29 times

Re: Failure Alerts During Real-Time Automated Trading

Postby PatrickSocal » 20 Oct 2014

faraz wrote:You should consider the above point number (c) for a clean solid hardware setup.

Yes, I agree. We have a dedicated server hosted close to the exchange with backup power and 1Gpbs data through multiple transit paths.

Regarding data gaps, you say:
faraz wrote:I have already mentioned the solution for this issue in the above list. Check (q) point. You do not need any coding. Also I do not suggest to identify data issues through MC code. Because to identify data issues you must have a clean data first from which you can match your data to see if there is any issue in your data or not. With out matching data you can not be sure if your data got issues or not. So quick reliable and easy way is that you refresh your all data or The days which are identical data on your and data feed provider if you got some back adjusted data running on your data file...

I think we have different points of view on this topic, and that's OK. I agree that reloading data is a good idea. But sometimes we forget to do it. And sometimes even if we do it, there is a gap in the feed's historical data. I didn't know this until I started doing tests. The way I detect gaps without needing a "ground truth" dataset is simple... On time bars, I use the interval and the session open/close times to compute every time when there should be a bar. If one isn't there, it's a sign that a bar is missing. This is very reliable. On tick-based bars, I look for cases when the time gap between bars is outside a "typical" range. This is less reliable, but it can flag situations that require a second look. Then I use keywords like aiRealTimeCalc and LastBarOnChart and aiStrategyAuto to detect if we are live trading. If so, to save CPU cycles, I usually call #return. (But I'm experimenting with different call frequencies to figure out just how big a load my code imposes, so maybe I can use it in live trading too.)

I do like your idea of scanning the OPT log file outside MultiCharts to detect errors. We'll definitely look into this. Thanks for the suggestion.

faraz
Posts: 140
Joined: 25 Feb 2011
Has thanked: 26 times
Been thanked: 56 times

Re: Failure Alerts During Real-Time Automated Trading

Postby faraz » 20 Oct 2014

PatrickSocal wrote:I agree that reloading data is a good idea. But sometimes we forget to do it. And sometimes even if we do it, there is a gap in the feed's historical data.


a) Forgetting some important Live Trading Execution Task is not an option.

This below One command line tool bar code will Reload all your charts data.

Code: Select all

.rld glob

https://www.multicharts.com/trading-soft ... ing_Charts



b) Gaps in Feed Historical data are normal. Mostly after holidays or week end or Different custom sessions get Gaps. Your strategy should understand that Gaps are normal and it should deal with them in a normal way.
These users thanked the author faraz for the post (total 3):
TJPatrickSocalhughesfleming

User avatar
PatrickSocal
Posts: 58
Joined: 27 Apr 2013
Location: San Diego, CA
Has thanked: 23 times
Been thanked: 29 times

Re: Failure Alerts During Real-Time Automated Trading

Postby PatrickSocal » 21 Oct 2014

Thanks for the global reload command. That's helpful.

On the topic of data gaps, we expect them during holidays and weekends. But I've seen gaps unexplained by either... which the data vendor has fixed after I sent an error report. So I still see value in checking for these.

Thanks again for all the suggestions.

User avatar
Smoky
Posts: 388
Joined: 03 Dec 2010
Location: Thailand
Has thanked: 65 times
Been thanked: 104 times

Re: Failure Alerts During Real-Time Automated Trading

Postby Smoky » 26 Oct 2014

hi Patrick, I already made all this stuff in 2 third party applications, one runing on the VPS and a second runing on a smartphone.

you forget that with all this tools we know what append (before MC LoL) but you can't do anything ! !
because easylangage don't manage corectly orders !(we have no status in autotrading)

for exemple my third party software detect non reponding ping trading server, my easylangage code can't manage reconnection ! it's not the same thing for 10 seconds fealure or 1 hour ...

my easylangage code can't bakfill or not chart datas, clear no sending orders ! stop autotrading for 15 minutes to make my indicators more stable and then make auto-trading on again.
Easylangage manage nothing in real life ! no status connections, no status broker / status datas .
we don't have MC variables, desktop variables, worksapces variables, charts variables, portfolio variables we do use global variables with his latency ....

All is done with chexbox choice (in propriety folder), but never with easylangage management !
all is done with popup and no informations to easylangage ! the worst, is orders management without any reject status/code ID to easylangage !

If you want my stuff PM me, I stop to work with MC (spend to many time to make all tasks than MC don't make
like 1 : a true backtest in tick/tick on ALL bar chart !
2: My portfolio tools use IOG and exotic bar study and all trades are sending to mobil-phone with alerts on link fealure, sms messages, master swith ON/OFF ...
These users thanked the author Smoky for the post (total 2):
PatrickSocalMAtricks

User avatar
Henry MultiСharts
Posts: 9047
Joined: 25 Aug 2011
Has thanked: 1250 times
Been thanked: 2905 times

Re: Failure Alerts During Real-Time Automated Trading

Postby Henry MultiСharts » 27 Oct 2014

Smoky wrote:hi Patrick, I already made all this stuff in 2 third party applications, one runing on the VPS and a second runing on a smartphone.

you forget that with all this tools we know what append (before MC LoL) but you can't do anything ! !
because easylangage don't manage corectly orders !(we have no status in autotrading)
As we have already discussed - new backtesting/auto trading engine with direct order control is coming in MultiCharts 9.1.

Smoky wrote:for exemple my third party software detect non reponding ping trading server, my easylangage code can't manage reconnection ! it's not the same thing for 10 seconds fealure or 1 hour ...

my easylangage code can't bakfill or not chart datas, clear no sending orders ! stop autotrading for 15 minutes to make my indicators more stable and then make auto-trading on again.
Easylangage manage nothing in real life ! no status connections, no status broker / status datas .
we don't have MC variables, desktop variables, worksapces variables, charts variables, portfolio variables we do use global variables with his latency ....

All is done with chexbox choice (in propriety folder), but never with easylangage management !
all is done with popup and no informations to easylangage ! the worst, is orders management without any reject status/code ID to easylangage !

If you want my stuff PM me, I stop to work with MC (spend to many time to make all tasks than MC don't make
like 1 : a true backtest in tick/tick on ALL bar chart !
2: My portfolio tools use IOG and exotic bar study and all trades are sending to mobil-phone with alerts on link fealure, sms messages, master swith ON/OFF ...
Datafeed/broker reconnects are managed by MultiCharts. Such things cannot be directly controlled from the PowerLanguage code. Links to the studies for monitoring these events were already provided above.

faraz
Posts: 140
Joined: 25 Feb 2011
Has thanked: 26 times
Been thanked: 56 times

Re: Failure Alerts During Real-Time Automated Trading

Postby faraz » 31 Dec 2014

Dear Traders !!!

I have added a new topic which can take you one more step to provide more stability to your strategy for Real Trading. I recommend you to please review it and take necessary steps.

viewtopic.php?f=5&t=47845

Successful Trading!!!

Thanks
These users thanked the author faraz for the post (total 2):
sptraderPatrickSocal

faraz
Posts: 140
Joined: 25 Feb 2011
Has thanked: 26 times
Been thanked: 56 times

Re: Failure Alerts During Real-Time Automated Trading

Postby faraz » 06 May 2015

Dear Traders !!!

Low byte viewing of your remote server desktop via mobile/Pc gives a great freedom. If you are using my point number (J) of my posted 17 Oct 2014. Then this additional feature will improve your monitoring frequency speed specially for Cell phone where we face low signal issues;

Lossy compression of PNG images
https://pngquant.org/

Batch file Command Line Code;

Code: Select all

Start /wait /min C:\pngquant2.exe --quality=65-80 C:\im.png --ext g.png
Start /wait /min C:\pngquant3.exe --quality=0-0 C:\im.png --ext 2.png


Now my 110 kb image size is;
i) 40kb with loss less quality
ii) 14kb with blank and white quality for low INTERNET connection times.

Successful Trading!!!

Thanks


Return to “MultiCharts”