Can MultiCharts handle hundreds or thousands of strategies?

Questions about MultiCharts and user contributed studies.
MarkJ
Posts: 10
Joined: 03 Oct 2022
Has thanked: 4 times
Been thanked: 2 times

Can MultiCharts handle hundreds or thousands of strategies?

Postby MarkJ » 04 Oct 2022

Hi everyone,

Apologies in advance if this has been covered before; I did do a search and didn't find anything definitive. I'm migrating from TS to MultiCharts right now and am currently trading 150 strategies, with substantial issues in TS; consistent failures to submit orders and multiple position mismatches each day. I was also told by support that TS will not work in a cloud environment, which was the answer to insane data disconnections which were happening constantly. It turns out TS requires the gui/monitor to be open/attached constantly to work correctly.

Can MultiCharts handle this many strategies and operate in a cloud environment without a constant gui/remote connection? Specifically I trade many strategies/charts on a single symbol, and about 40 symbols total. I also expect to scale to 300+ strategies by the end of next year. I just don't want to go down another rabbit hole to find a technological dead end.

Also what data providers are advised for CME/NYMEX/CBOE strategies? Rithmic seems great at first glance, but definitely open to whatever works best.

Thank you so much for your help!
Mark

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby TJ » 04 Oct 2022

There are so many bottlenecks in trading, you have to find out where are your bottlenecks.

1. How far are you from your broker?
2. What is your latency?

Today's computers might seem fast, but they are still operating in a very archaic 86 architecture. ie there are lots and lots of bottlenecks inside the computer.

You have to sort out those limitations before starting trading 1 strategy.

If you don't even know what data providers to use, you are biting off more than you can chew.

MarkJ
Posts: 10
Joined: 03 Oct 2022
Has thanked: 4 times
Been thanked: 2 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby MarkJ » 04 Oct 2022

There are so many bottlenecks in trading, you have to find out where are your bottlenecks.

1. How far are you from your broker?
2. What is your latency?

Today's computers might seem fast, but they are still operating in a very archaic 86 architecture. ie there are lots and lots of bottlenecks inside the computer.

You have to sort out those limitations before starting trading 1 strategy.

If you don't even know what data providers to use, you are biting off more than you can chew.
Hi TJ,

I believe you may have misunderstood my issue. Latency is currently not my concern, since I'm using TS anyways, which has a fair amount of latency. I also trade all swing strategies, although for some my breakout strategies I do send the stop orders to the market.

My question is, for anyone who has traded hundreds of strategies before in MultiCharts, how did it go, and if you could provide tips to get me there faster it would be very appreciated.

On the other hand, I must tell you that since April 2021 I've turned a $400k Roth IRA into over $1m and have over 10 years experience as a technologist and data scientist so I am absolutely not biting off more than I can chew.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby TJ » 04 Oct 2022

I would buy a new computer and an extra MultiCharts license for every 20 strategies.
YMMV

MarkJ
Posts: 10
Joined: 03 Oct 2022
Has thanked: 4 times
Been thanked: 2 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby MarkJ » 04 Oct 2022

Thanks for your response. Not gonna lie that's pretty disappointing to hear though, as that would not just require multiple MC licenses, but also broker data connection licenses and a lot of remote instances to manage.

Maybe in the end I should just move to Columbia and develop a properly scalable distributed system like I did for Deutsche... lotta work though.

Still, thanks for your opinion, I hope to get a few more as well :)

User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby syswizard » 07 Oct 2022

I'll chime in here....
first of all, congrats on the great performance...$400k to $1 million....wow !
Secondly, how are you getting all of those signals coordinated for a single instrument ?
For instance, one signal has you going long, but another signal wants to place a short....or place another long.
To do this, you must coordinate the strategies/signals. Perhaps the portfolio mgmt system can help in this regard...see PMMS functions.
Also, why use a cloud connection for the data ? I refuse to even use a wireless connection and instead use a wired CAT-5 connection instead....with blindingly fast speed using Rithmic datafeed.
I would say that operating with hundreds of signals active is quite possible. Thousands ? Maybe.
Critical factors: Is your chart setting indicating tick-level data ?
Is IOG turned on ? (IntrabarOrderGeneration)
Are you using SET statements....SETTDOLLARTRAILING, SETPERCENTTRAILING, etc.
These operate intrabar and there is no way to make them work on the end of the bar. Intrabar order executions consume a lot of CPU !
Intel has come out with a new CPU: Intel's 13th-Gen Raptor Lake processors will bring more cores, more connectivity, a revamped core architecture, support for PCIe 5.0 SSDs, and even an officially-verified 6.0 GHz peak boost clock to bear.
Now the big issue is: will Multicharts take advantage of all of the cores ?
It certainly does in optimization runs, but for regular order generation ? I don't know.

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby Svetlana MultiCharts » 11 Oct 2022

will Multicharts take advantage of all of the cores ?
Hi,

Multi-core CPU will increase the optimization speed as this calculation can be divided into multiple flows using multi-core CPU. Multi-core CPU will not increase the backtesting speed as it is a linear calculation procedure and it cannot be divided into parts for calculation.
We recommend to give preference to the machines with higher CPU frequency, than to the number of cores: e.g. 8 cores with 3600 base frequency is better than 16 cores with 2300 base frequency.

MarkJ
Posts: 10
Joined: 03 Oct 2022
Has thanked: 4 times
Been thanked: 2 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby MarkJ » 11 Oct 2022

Thank you for your response Svetlana! Can you verify that calculation is parallelized across charts? I have a separate, custom algorithm generator ("factory") which runs walkforward and chooses parameters, so I won't be needing optimization in MultiCharts... simply looking for it to run a large number of charts, 100+, in parallel, in real time.

Totally get the higher clock speed and I absolutely will be using the fastest processors possible on my prod rig... and I use water coolers and overclocking as well.

User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby syswizard » 12 Oct 2022

I think it's possible to place separate MC processes into separate cores. It's called affinity, but I think it can only be done in Windows 10 or 11.
For instance, place tsserver.exe into it's own core and RTPServerProxy.exe into it's own core.
Not sure about the parallelization issue.

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby Svetlana MultiCharts » 12 Oct 2022

Thank you for your response Svetlana! Can you verify that calculation is parallelized across charts? I have a separate, custom algorithm generator ("factory") which runs walkforward and chooses parameters, so I won't be needing optimization in MultiCharts... simply looking for it to run a large number of charts, 100+, in parallel, in real time.

Totally get the higher clock speed and I absolutely will be using the fastest processors possible on my prod rig... and I use water coolers and overclocking as well.
Here’re general recommendations on how to improve the performance:
1. disable your anti-virus and all excessive applications running in background;
2. make sure none of the applications are doing any updates/scans while you are trading;
3. make sure Windows is not downloading and installing any updates while you are trading;
4. create several desktops and split the number of charts you have between MC desktops*.

*This approach does not split all MultiCharts processes between the cores. It mostly improves graphical rendering: For example, if you have split a workspace with 100 charts to say 10 desktops with 10 charts, the computer will have to process 10 threads with a queue of objects from 10 charts, instead of 1 thread with a queue from 100 charts.
If you have multiple objects to draw in one desktop it may be reasonable to split them between multiple desktops.
This approach does not reduce the load, but helps to distribute it more evenly. This solution requires testing in your specific setup in your environment.

In order to determine what specific resources are required, you can start your usual workspace and watch the load in Task Manager. Adding RAM is reasonable only if the computer lacks RAM.

MarkJ
Posts: 10
Joined: 03 Oct 2022
Has thanked: 4 times
Been thanked: 2 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby MarkJ » 12 Oct 2022

If you have multiple objects to draw in one desktop it may be reasonable to split them between multiple desktops.
Awesome, this is consistent with the advice a new friend/fund manager out of France told me this morning, he has production all on one machine but splits his strategies across MC desktops (20 charts each). Also uses TS data and IB as the broker... I'll be trying this setup asap.

MarkJ
Posts: 10
Joined: 03 Oct 2022
Has thanked: 4 times
Been thanked: 2 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby MarkJ » 12 Oct 2022

I think it's possible to place separate MC processes into separate cores. It's called affinity, but I think it can only be done in Windows 10 or 11.
For instance, place tsserver.exe into it's own core and RTPServerProxy.exe into it's own core.
Not sure about the parallelization issue.
I will look into affinity for prioritizing the TS data stream (data access layer), thank you for the tip. And is RTPServerProxy.exe the MC broker access layer?

MarkJ
Posts: 10
Joined: 03 Oct 2022
Has thanked: 4 times
Been thanked: 2 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby MarkJ » 12 Oct 2022

Secondly, how are you getting all of those signals coordinated for a single instrument ?
Just saw this, right now I'm running them all as separate charts in TS with TradeManager checking off. So, any position mismatch fix is manual and I have to check very frequently and do full platform restarts ~5x per day to identify these mismatches and then fix them manually. Happily I trade all swing strategies so the approximately 10 minute full restart rarely affects me if I do it at the right times.

Also the strategies are the standard recipe: EasyLanguage applied to a continuous tradable contract on some minute bar size, all execution next bar, no IBOG or tick level things. Lowest timeframe is 15 min. I do use data2, 3, and 4 often, with either correlated markets, higher timeframes, or both.

User avatar
Mark Brown
Posts: 181
Joined: 29 Nov 2016
Has thanked: 111 times
Been thanked: 17 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby Mark Brown » 12 Oct 2022

lol it's not a unix program.

User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby syswizard » 12 Oct 2022


4. create several desktops and split the number of charts you have between MC desktops*.

*This approach does not split all MultiCharts processes between the cores.
Why doesn't MC split the processes ?

tonyng
Posts: 51
Joined: 04 Oct 2013
Has thanked: 2 times
Been thanked: 5 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby tonyng » 16 Oct 2022

Use MC12 instead of 14. We 've done the testing and only MC12 is able to run many charts at the same time with 10 years data with APPL. we were told that this will be fixed in the next up coming version.

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby Svetlana MultiCharts » 17 Oct 2022


4. create several desktops and split the number of charts you have between MC desktops*.

*This approach does not split all MultiCharts processes between the cores.
Why doesn't MC split the processes ?
This approach creates several MultiCharts.exe, but all of them share the same TradingServer.exe and TsServer.exe.

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby Svetlana MultiCharts » 17 Oct 2022

By the way, the Fail-Watch system is implemented since MultiCharts 14. It means one will be notified if the limits of the OS, hardware, bandwidth, software and/or broker processing speed are reached.
MultiCharts 12 does not feature this system, i.e. one risks to be unaware of such situations.

User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby syswizard » 17 Oct 2022


This approach creates several MultiCharts.exe, but all of them share the same TradingServer.exe and TsServer.exe.
Why doesn't TradingServer.exe and TsServer.exe execute in separate cores from the desktop instance ?

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby Svetlana MultiCharts » 18 Oct 2022


This approach creates several MultiCharts.exe, but all of them share the same TradingServer.exe and TsServer.exe.
Why doesn't TradingServer.exe and TsServer.exe execute in separate cores from the desktop instance ?
TradingServer.exe and TsServer.exe act as databases in the memory, therefore they cannot be split. Moreover, MultiCharts is a software with multiple brokers/data feeds, and we have to refer to a broker/data feed from a single process (few brokers/feeds allow multiple logins in the frames of a single account).

MarkJ
Posts: 10
Joined: 03 Oct 2022
Has thanked: 4 times
Been thanked: 2 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby MarkJ » 10 Aug 2023

Thanks again for all of your responses; I'm now running 472 strategies (across 30 desktops) with what seems to be fairly successful execution. Unfortunately, the GUI is frozen for most of the open time (until about 11am est) and I get a lot of popups saying there's a high CPU load when there isn't (I guess this is a single thread process getting bogged down?). I am using a cloud computer with Intel Xenon which maxes at 5Ghz so I don't know how to get a faster thread.

A few questions:
  • Would a GPU help the GUI remain stable? When I say the GUI I mean Windows itself... all windows freeze including things like task manager
  • Is there any way to run the strategy code and submit orders without actually printing the candles? Like if I minimize the chart windows would that help the Windows GUI process at all, and if not is there anything else I can do?
  • Is there a way to turn off the warning/error message popups and just have them in the log? They seem to use a massive amount of graphics and the entire system freezes horribly when there's a lot of popups coming in
Thanks again,
Mark

HellGhostEvocatorX
Posts: 77
Joined: 10 Feb 2022
Has thanked: 47 times
Been thanked: 9 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby HellGhostEvocatorX » 17 Aug 2023

to your statement: what does the performance monitor in the task manager say which cores are busy? All? evenly?
There are now CPUs with a large number of cores that are designed for parallel work. (threadripper e.g.) maybe that would be a solution.

Basically, I'm wondering what's the point of trading over 150 (different?) strategies? It shouldn't bring much benefit. I would always only trade the most successful ones...

HellGhostEvocatorX
Posts: 77
Joined: 10 Feb 2022
Has thanked: 47 times
Been thanked: 9 times

Re: Can MultiCharts handle hundreds or thousands of strategies?

Postby HellGhostEvocatorX » 17 Aug 2023

*
resource monitor is the right name :)


Return to “MultiCharts”