What's the most stable autotrading setup?

Questions about MultiCharts and user contributed studies.
User avatar
c0ntango
Posts: 70
Joined: 13 Sep 2018
Has thanked: 3 times
Been thanked: 16 times
Contact:

What's the most stable autotrading setup?

Postby c0ntango » 05 Jan 2021

Hey everyone,

I wanted to poll the senior user community of the forum: how do you automate your trading... and would like to ask for some advice in this regard as well.

My experiences (throughout roughly 3-4 years with the software):

- So far it seems to me, that Portfolio Trader up to v12 eventually would always run into memory errors if running a certain amount of charts - 60-80, sometimes more, sometimes less. Autotrading would shut down without notification, sometimes the software itself becomes unstable. Also, it barely matters how much RAM you have, it just seems to eat it all up eventually. Is this congruent with your experience?

- With MCv14 according to my testing, the problem got even worse. At least now the memory exception is handled - I was running 28 charts (about 15 of them with 750 lookback on 5 min), and within a few days I would get an "STD error: out of memory" red message, and autotrading would stop with unmanaged trades open on the broker side. There would also be a lingering MC Portfolio.exe process claiming all the memory of the machine I'm using it on (which is not much but is still 32 GB - and the history was 50 days, so there's no reason for it, really). I suspect there's a new memory leak somewhere. In any case, I'm starting to feel like PT is not an appropriate for live trading when it comes to a sizable account.

- Some competing products also use "in case of disconnect" and "in case of reconnect" functionality which lets you take action in case of certain events. Here, it seems like you have to handle everything manually, and you can't even auto-start the software after a reboot or something, in case it stops because of the legal notice you have to approve every time. It is a spectacularly annoying limitation of the software - I just don't understand what am I supposed to do. Hire people from India for 24/7 monitoring so there is always a human ready to click the ***** legal notice? :D Did anyone find a way around this?

- I have some friends who are saying - just open everything on its own chart and it's going to be SUPER stable. You put the instrument, you put the strategy on it, you turn autotrading on, and it runs, on 50-100 charts, with 4 GB of RAM (!!) for literal years without ever having to touch it. Is this congruent with you guys experience? The problem here is if somebody is running a portfolio of 600 or 1000 strategies. I guess at some point you would say MC is just not the right software for it... But still I'd really love to hear about your experience on this.

- I know some people including some fund managers who use third party software to submit trades. They just open charts, don't even turn autotrading on, just let it run real time, and submit orders from the signals. I guess it's a very stable option - but again, there's a limit to the amount of strategies you can reasonably manage like that..

Anyway - I'm a little bit frustrated with this situation. Your advice on this would be very much appreciated.

Thank you,

-Ben

User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Re: What's the most stable autotrading setup?

Postby syswizard » 06 Jan 2021

- I have some friends who are saying - just open everything on its own chart and it's going to be SUPER stable. You put the instrument, you put the strategy on it, you turn autotrading on, and it runs, on 50-100 charts, with 4 GB of RAM (!!) for literal years without ever having to touch it. Is this congruent with you guys experience? The problem here is if somebody is running a portfolio of 600 or 1000 strategies. I guess at some point you would say MC is just not the right software for it... But still I'd really love to hear about your experience on this.
MC (without Portfolio Trader) is super stable....never crashes, I never have to restart it....even after a week of operation.
The problem here is if somebody is running a portfolio of 600 or 1000 strategies. I guess at some point you would say MC is just not the right software for it.
You would be right on that.....in fact I'm not sure that <any> off-the-shelf could do that...including NT.
Best to create your own platform in optimized C,C++. Start now and with the need for about 500k lines of code, you should be done sometime in 2024.
Good luck.

User avatar
c0ntango
Posts: 70
Joined: 13 Sep 2018
Has thanked: 3 times
Been thanked: 16 times
Contact:

Re: What's the most stable autotrading setup?

Postby c0ntango » 06 Jan 2021

Umm... "Even after a week of operation". The comment really kind of worries me. How often DO you have to restart your platform?

I mean, there's no need for c/c++ here... One could research strategies in MC which I would argue is still the best possible software in the world to do that in... And also code up an execution platform on Linux with python in a few weeks, and execute in that. In actual fact, in one of my businesses I have already done this. And it would never need to be restarted. Literally, ever, there's no reason for it. And if it's restarted, it automatically resumes trading like nothing happens, all that's missing is 1 minute of data from the archives because that's the boot-up time of Linux. There's no need for any human interaction or anything else at all - market position for strategy at broker is detected automatically, calculated locally, synced up and trading is resumed.

Also, this is not a complicated exercise. When we did it, it took about a month and a half to get it up live, and it's been running ever since.

I'm just asking... MC is sooo good.. But is there, like, seriously no good and stable solution to this problem? And why am I, like, the first person to mention it? :)

User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Re: What's the most stable autotrading setup?

Postby syswizard » 06 Jan 2021

Also, this is not a complicated exercise. When we did it, it took about a month and a half to get it up live, and it's been running ever since.

I'm just asking... MC is sooo good.. But is there, like, seriously no good and stable solution to this problem? And why am I, like, the first person to mention it? :)
Yeah, well you think you are a code jockey....good for you. You are much better than me. But trust me, if your solution had one-tenth of the functionality of MC...with the data adapters, the execution adapters, the interpreter/compiler, etc....it would take a LONG, LONG time.
What I am saying is that your current solution is likely bare-bones....at best. But you know what ?....that's probably what you need. Just keep plugging away.
Python eh ? Why would you go with an interpreted solution when you need the performance of native compiled code ? Oh...because you can code much faster, right ?
MC is built on optimized C/C++ and it runs on a 4 year old quad core processor and the CPU utilization rarely goes above 5 percent. That's the power of native code.

User avatar
c0ntango
Posts: 70
Joined: 13 Sep 2018
Has thanked: 3 times
Been thanked: 16 times
Contact:

Re: What's the most stable autotrading setup?

Postby c0ntango » 06 Jan 2021

I don't "think" I am anything, really, and certainly not a "code jockey".

Also, respectfully... I don't know who you are or what you do, and judging by the age of your user name you're probably a much more senior user of this software that I am. However, I would beg to differ on your assessment and judgment on this, and not just because it's far from bare bones (we have 3 data adapters in there, it's a FIX API / other broker-specific historical API connector), but because MC is not native c/c++, it runs on .NET and loads up all the trash with it up to memory.

Obviously there's no interpreter/compiler, you have to do your signal processing in python for it, but it's not that big of a deal. I mean, you get an OHLC data frame natively from pandas and there's also talib. How complicated do you imagine it is to calculate whatever math you're doing on your bars? That's the only thing that needs to be ported, and it's actually the simplest part of the whole thing, so there's really no need to re-code an interpreter.

Also, with this you can reliably process almost any amount of tick data streams (tried with 80 simultaneous instruments but it could've been much more too) into "charts" with a 12€/month VM on Linux (quad core, 8 GB RAM but I think it would probably run all the same on a fraction of this). This includes saving to database and running some other services as well. I am confident that I could load up 10,000 different strategies on this setup, and run it with equal to less latency than I'd run it with MC even on this server. I don't know how many I'd have to do before it would require an upgrade, but I assure you, it's fast. And obviously, the robustness and stability factor that I described above about this would outweigh the latency question even if it would be an issue, which it is not.

Also, just to note. I would like to stress, again, that I firmly believe that MC is the best possible tool on the planet for quantitative research when you're not a software engineer (and I am certainly not). This is not me arguing against it. I am just saying - what is going on with this automated trading thing and asking the community whether there exists a solution to this issue.

User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Re: What's the most stable autotrading setup?

Postby syswizard » 06 Jan 2021

Dude: re: because MC is not native c/c++, it runs on .NET and loads up all the trash with it up to memory.
this is from the horses mouth...
MC14-architecture.PNG
(19.38 KiB) Not downloaded yet
I'm not a software engineer, but I am a data analyst which comes in real handy for dealing large volumes of market data and stats.
Also, you relying on open source components for live trading ? I would never do that....too many bad experiences with open source solutions.

User avatar
c0ntango
Posts: 70
Joined: 13 Sep 2018
Has thanked: 3 times
Been thanked: 16 times
Contact:

Re: What's the most stable autotrading setup?

Postby c0ntango » 06 Jan 2021

I understand that support is going to go ahead and simplify that answer for you to the point where the horse is going to say it's written in C++. But when you're running strategies live, in the end you're not running what you imagine as "native C++ code". I hate to be the bearer of bad news but that statement is simply not true. There are components which require .NET and other stuff as well.

The open source discussion is irrelevant here and I'm not too keen on going down this route with you to avoid flame wars that will divert this topic.

Let's stay on topic here. Good people of the MC community. What is, in your experience, the most stable auto-trading setup with MC? :)

User avatar
joebone
Posts: 175
Joined: 05 Sep 2018
Has thanked: 53 times
Been thanked: 4 times

Re: What's the most stable autotrading setup?

Postby joebone » 07 Jan 2021

Interested in any comments or suggestions on this subject. I use chart trading in AA mode on several charts

I stack my live trading charts in 1 or 2 workspaces using additional windows and just flip through the windows in those workspaces. Then I use other workspaces to stack windows to monitor charts and signals.

I am on MC12 though. Im not jumping my live trading up to 14 for a little while. I use several licenses so experimenting with the new updates isn't a big deal to me.

User avatar
c0ntango
Posts: 70
Joined: 13 Sep 2018
Has thanked: 3 times
Been thanked: 16 times
Contact:

Re: What's the most stable autotrading setup?

Postby c0ntango » 07 Jan 2021

Hi Joe,

Thanks for sharing.

How stable is that setup for you, in your experience? How often do you need to restart MC?

-Ben

User avatar
joebone
Posts: 175
Joined: 05 Sep 2018
Has thanked: 53 times
Been thanked: 4 times

Re: What's the most stable autotrading setup?

Postby joebone » 08 Jan 2021

Hi Joe,

Thanks for sharing.

How stable is that setup for you, in your experience? How often do you need to restart MC?

-Ben
I have never had a crash on 12. I have really never had a software end meltdown of any kind. One time I accidentally changed a large chart to a 1 tic resolution and that ended with me having to Ctl-Alt-Dlt the program but I have actually never had a problem with 12 that I would consider a software meltdown.

However I do restart my software every weekend as well as my entire computer. (most weekends at least, sometimes I forget)

On MC12 I have never found a problem that I would consider a deal breaker for live trading..... on 14 however.... lets just say I am waiting a bit longer, and I am excited about the new features but haven't made the live trading jump yet. But people said that about the 12 jump as well. 14 will be fine eventually, just gotta let the devs hammer out the problems. MC is cool with people running 12 for a while until they get it all sorted.

As someone trading professionally with MC. I would say owning several licenses is the way.

brakkar
Posts: 9
Joined: 07 Feb 2014
Has thanked: 3 times

Re: What's the most stable autotrading setup?

Postby brakkar » 23 Jan 2021

A quick question: when it auto trade with portfolio, do you need to have charts opened?
Or can I just run 1 strategy on 1000 tickers without having to load each chart?

User avatar
c0ntango
Posts: 70
Joined: 13 Sep 2018
Has thanked: 3 times
Been thanked: 16 times
Contact:

Re: What's the most stable autotrading setup?

Postby c0ntango » 23 Jan 2021

Well there's no way for you to "open a chart" in portfolio trader, so the question is a bit unclear. And in MC you have to have charts to put the strategy on.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: What's the most stable autotrading setup?

Postby wilkinsw » 28 Jan 2021

I dare say trial and error goes a long way.

Rules of thumb for stability (just my opinion):

-Use MC vs PT
-Don't exceed 15 workspaces per machine (split them across multiple MC instances)
-Note that some symbols eat up resource more than others (emini S&P (ES) the main culprit: 1 ES chart is like 10 Gold (GC) charts).
-Your code efficiency goes a long way
-Have a bucket load of spare CPU and Ram available.

I've had live sessions stay stable for at least 2 months at a time. Factors that have worked against me are IOG code and when ES gets particularly busy.

I personally would love MC to build a stripped down version of MC that is designed to be as fast as possible to the sacrifice of all else. Then a separate, fully loaded version to cater for the visual and manual orientated trader.

User avatar
c0ntango
Posts: 70
Joined: 13 Sep 2018
Has thanked: 3 times
Been thanked: 16 times
Contact:

Re: What's the most stable autotrading setup?

Postby c0ntango » 28 Jan 2021

Those are good tips, congruent with my findings as well so far. Thanks for sharing them, Wilkins!

Mydesign
Posts: 177
Joined: 15 Feb 2017
Has thanked: 32 times
Been thanked: 39 times

Re: What's the most stable autotrading setup?

Postby Mydesign » 28 Jan 2021

Rules of thumb for stability (just my opinion):
-Use MC vs PT
Hi,

Could you elaborate on that one ? Isn't PT supposedly a "stripped down version of MC" with just what is necessary for automated tranding ?

User avatar
c0ntango
Posts: 70
Joined: 13 Sep 2018
Has thanked: 3 times
Been thanked: 16 times
Contact:

Re: What's the most stable autotrading setup?

Postby c0ntango » 28 Jan 2021

Well... It is. But it doesn't work that way. There seems to be a memory leak in almost every version of PT, that just eats up all the RAM in your machine eventually. Also the "currency conversion" nightmare, with the conversion server being periodically down. And no granular control on charts - you can't see what's going on. Tick data also stops, without notice, and it's quite tricky to figure out a way to monitor it. Tons of issues there, charts are way superior. But this was the main question I raised in the thread.

MC and PT are the best research tools out there I can think of. But live trading can get tricky.

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: What's the most stable autotrading setup?

Postby wilkinsw » 28 Jan 2021

Rules of thumb for stability (just my opinion):
-Use MC vs PT
Hi,

Could you elaborate on that one ? Isn't PT supposedly a "stripped down version of MC" with just what is necessary for automated tranding ?
I have to be honest and admit that my PT experience in live trading is minimal.

I opt for MC over PT purely because it is faster/easier for me to debug trade breakages... which are inevitable in any environment. In MC, even with the most basic of charts running auto trading; you can instantly see when something doesn't look right.

If users of PT disagree with my views, I'm all ears. Would be willing to give PT a proper try live if someone can make a compelling case.

Salzburg
Posts: 140
Joined: 19 Sep 2018
Has thanked: 9 times
Been thanked: 35 times

Re: What's the most stable autotrading setup?

Postby Salzburg » 28 Jan 2021

My few cents on the matter... been live trading system with MC for several years.. always waiting until new version is stable before updating, so i am lagging 2-3 versions back in order to make sure things are as stable as possible. Still using MC12.

Live trading intraday systems on 7 future markets, 30 minute bars.. never crashed on me
Portfolio trader using Daily bars, 50 symbols, never crashed...

Attempted portfolio with intraday minute bars, very often crashed, so i had to abandon this.

I had huge issues in the start when i used IB for both chart data and as broker.. did not work, almost everything crashed and hung on me.
Changed to IQ as data provider and using IB as broker, works perfect.

However, i am slowly migrating myself away from MC, for equity systems that require proper database handling and needs a structure that is better fit for dealing with thousands of symbols, mapping, listed, delisted, point in time index constituents etc... MC is not this tool, however in my book MC works amazing for stable live trading futures and research.
For Equities, MC its just not the tool for the job.

User avatar
c0ntango
Posts: 70
Joined: 13 Sep 2018
Has thanked: 3 times
Been thanked: 16 times
Contact:

Re: What's the most stable autotrading setup?

Postby c0ntango » 29 Jan 2021

Cool, Salzburg. Can I ask how often you restart your systems? Do you have a "weekend restart" or anything like that? What's the longest time you've been running your algos without touching them in MC/PT?

Salzburg
Posts: 140
Joined: 19 Sep 2018
Has thanked: 9 times
Been thanked: 35 times

Re: What's the most stable autotrading setup?

Postby Salzburg » 29 Jan 2021

Cool, Salzburg. Can I ask how often you restart your systems? Do you have a "weekend restart" or anything like that? What's the longest time you've been running your algos without touching them in MC/PT?
c0ontango, I restart MC every weekend. I restart Server every weekend, I restart IQ feed every day. I flush cash 1-2 times per week.
My server is a maintained dedicated server only running live trading MC, 8 core, 32gb ram at Chicago host near the exchange servers.
Not a glitch in operation from MC :)

longest time it been up an running without anything else than IB login, was 3 weeks, worked fine.

(one thing that seem to be able to cause errors, IB gateway for some reasons seem to add a "client111 etc" normyl it should be 1 client only.
If i ever see more than one, i restart IB.
One time during all the years IB gateway added 4 clients for some strange reason, and that cause execution error, but i am not sure if this is the reason due to my not updating IB gateway in my trading server more than perhaps 1 time per year or less.

Cheers

User avatar
c0ntango
Posts: 70
Joined: 13 Sep 2018
Has thanked: 3 times
Been thanked: 16 times
Contact:

Re: What's the most stable autotrading setup?

Postby c0ntango » 29 Jan 2021

Yeah, well. I guess it's not a problem for most traders, but that "manual restarting" thing just doesn't scale very well unfortunately :)

Salzburg
Posts: 140
Joined: 19 Sep 2018
Has thanked: 9 times
Been thanked: 35 times

Re: What's the most stable autotrading setup?

Postby Salzburg » 29 Jan 2021

Yeah, well. I guess it's not a problem for most traders, but that "manual restarting" thing just doesn't scale very well unfortunately :)
Yes, i agree, for lots of reasons the manual "restart" and manual precautionary measures works ok for a small medium trader not doing overly complex portfolio stuff, BUT to scale the operation in a more professional automatic way with as little manual hand on as possible, i believe there are other better tools for the job. However for research and algo future chart trading think MC is awesome.

User avatar
c0ntango
Posts: 70
Joined: 13 Sep 2018
Has thanked: 3 times
Been thanked: 16 times
Contact:

Re: What's the most stable autotrading setup?

Postby c0ntango » 29 Jan 2021

I agree, I think it's the best in class if you're a researcher and know how to use it properly :)


Return to “MultiCharts”