Can MultiCharts handle hundreds or thousands of strategies?
Can MultiCharts handle hundreds or thousands of strategies?
Hi everyone,
Apologies in advance if this has been covered before; I did do a search and didn't find anything definitive. I'm migrating from TS to MultiCharts right now and am currently trading 150 strategies, with substantial issues in TS; consistent failures to submit orders and multiple position mismatches each day. I was also told by support that TS will not work in a cloud environment, which was the answer to insane data disconnections which were happening constantly. It turns out TS requires the gui/monitor to be open/attached constantly to work correctly.
Can MultiCharts handle this many strategies and operate in a cloud environment without a constant gui/remote connection? Specifically I trade many strategies/charts on a single symbol, and about 40 symbols total. I also expect to scale to 300+ strategies by the end of next year. I just don't want to go down another rabbit hole to find a technological dead end.
Also what data providers are advised for CME/NYMEX/CBOE strategies? Rithmic seems great at first glance, but definitely open to whatever works best.
Thank you so much for your help!
Mark
Apologies in advance if this has been covered before; I did do a search and didn't find anything definitive. I'm migrating from TS to MultiCharts right now and am currently trading 150 strategies, with substantial issues in TS; consistent failures to submit orders and multiple position mismatches each day. I was also told by support that TS will not work in a cloud environment, which was the answer to insane data disconnections which were happening constantly. It turns out TS requires the gui/monitor to be open/attached constantly to work correctly.
Can MultiCharts handle this many strategies and operate in a cloud environment without a constant gui/remote connection? Specifically I trade many strategies/charts on a single symbol, and about 40 symbols total. I also expect to scale to 300+ strategies by the end of next year. I just don't want to go down another rabbit hole to find a technological dead end.
Also what data providers are advised for CME/NYMEX/CBOE strategies? Rithmic seems great at first glance, but definitely open to whatever works best.
Thank you so much for your help!
Mark
- TJ
- Posts: 7745
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2224 times
Re: Can MultiCharts handle hundreds or thousands of strategies?
There are so many bottlenecks in trading, you have to find out where are your bottlenecks.
1. How far are you from your broker?
2. What is your latency?
Today's computers might seem fast, but they are still operating in a very archaic 86 architecture. ie there are lots and lots of bottlenecks inside the computer.
You have to sort out those limitations before starting trading 1 strategy.
If you don't even know what data providers to use, you are biting off more than you can chew.
1. How far are you from your broker?
2. What is your latency?
Today's computers might seem fast, but they are still operating in a very archaic 86 architecture. ie there are lots and lots of bottlenecks inside the computer.
You have to sort out those limitations before starting trading 1 strategy.
If you don't even know what data providers to use, you are biting off more than you can chew.
Re: Can MultiCharts handle hundreds or thousands of strategies?
Hi TJ,There are so many bottlenecks in trading, you have to find out where are your bottlenecks.
1. How far are you from your broker?
2. What is your latency?
Today's computers might seem fast, but they are still operating in a very archaic 86 architecture. ie there are lots and lots of bottlenecks inside the computer.
You have to sort out those limitations before starting trading 1 strategy.
If you don't even know what data providers to use, you are biting off more than you can chew.
I believe you may have misunderstood my issue. Latency is currently not my concern, since I'm using TS anyways, which has a fair amount of latency. I also trade all swing strategies, although for some my breakout strategies I do send the stop orders to the market.
My question is, for anyone who has traded hundreds of strategies before in MultiCharts, how did it go, and if you could provide tips to get me there faster it would be very appreciated.
On the other hand, I must tell you that since April 2021 I've turned a $400k Roth IRA into over $1m and have over 10 years experience as a technologist and data scientist so I am absolutely not biting off more than I can chew.
- TJ
- Posts: 7745
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2224 times
Re: Can MultiCharts handle hundreds or thousands of strategies?
I would buy a new computer and an extra MultiCharts license for every 20 strategies.
YMMV
YMMV
Re: Can MultiCharts handle hundreds or thousands of strategies?
Thanks for your response. Not gonna lie that's pretty disappointing to hear though, as that would not just require multiple MC licenses, but also broker data connection licenses and a lot of remote instances to manage.
Maybe in the end I should just move to Columbia and develop a properly scalable distributed system like I did for Deutsche... lotta work though.
Still, thanks for your opinion, I hope to get a few more as well
Maybe in the end I should just move to Columbia and develop a properly scalable distributed system like I did for Deutsche... lotta work though.
Still, thanks for your opinion, I hope to get a few more as well
Re: Can MultiCharts handle hundreds or thousands of strategies?
I'll chime in here....
first of all, congrats on the great performance...$400k to $1 million....wow !
Secondly, how are you getting all of those signals coordinated for a single instrument ?
For instance, one signal has you going long, but another signal wants to place a short....or place another long.
To do this, you must coordinate the strategies/signals. Perhaps the portfolio mgmt system can help in this regard...see PMMS functions.
Also, why use a cloud connection for the data ? I refuse to even use a wireless connection and instead use a wired CAT-5 connection instead....with blindingly fast speed using Rithmic datafeed.
I would say that operating with hundreds of signals active is quite possible. Thousands ? Maybe.
Critical factors: Is your chart setting indicating tick-level data ?
Is IOG turned on ? (IntrabarOrderGeneration)
Are you using SET statements....SETTDOLLARTRAILING, SETPERCENTTRAILING, etc.
These operate intrabar and there is no way to make them work on the end of the bar. Intrabar order executions consume a lot of CPU !
Intel has come out with a new CPU: Intel's 13th-Gen Raptor Lake processors will bring more cores, more connectivity, a revamped core architecture, support for PCIe 5.0 SSDs, and even an officially-verified 6.0 GHz peak boost clock to bear.
Now the big issue is: will Multicharts take advantage of all of the cores ?
It certainly does in optimization runs, but for regular order generation ? I don't know.
first of all, congrats on the great performance...$400k to $1 million....wow !
Secondly, how are you getting all of those signals coordinated for a single instrument ?
For instance, one signal has you going long, but another signal wants to place a short....or place another long.
To do this, you must coordinate the strategies/signals. Perhaps the portfolio mgmt system can help in this regard...see PMMS functions.
Also, why use a cloud connection for the data ? I refuse to even use a wireless connection and instead use a wired CAT-5 connection instead....with blindingly fast speed using Rithmic datafeed.
I would say that operating with hundreds of signals active is quite possible. Thousands ? Maybe.
Critical factors: Is your chart setting indicating tick-level data ?
Is IOG turned on ? (IntrabarOrderGeneration)
Are you using SET statements....SETTDOLLARTRAILING, SETPERCENTTRAILING, etc.
These operate intrabar and there is no way to make them work on the end of the bar. Intrabar order executions consume a lot of CPU !
Intel has come out with a new CPU: Intel's 13th-Gen Raptor Lake processors will bring more cores, more connectivity, a revamped core architecture, support for PCIe 5.0 SSDs, and even an officially-verified 6.0 GHz peak boost clock to bear.
Now the big issue is: will Multicharts take advantage of all of the cores ?
It certainly does in optimization runs, but for regular order generation ? I don't know.
- Svetlana MultiCharts
- Posts: 645
- Joined: 19 Oct 2017
- Has thanked: 3 times
- Been thanked: 163 times
Re: Can MultiCharts handle hundreds or thousands of strategies?
Hi,will Multicharts take advantage of all of the cores ?
Multi-core CPU will increase the optimization speed as this calculation can be divided into multiple flows using multi-core CPU. Multi-core CPU will not increase the backtesting speed as it is a linear calculation procedure and it cannot be divided into parts for calculation.
We recommend to give preference to the machines with higher CPU frequency, than to the number of cores: e.g. 8 cores with 3600 base frequency is better than 16 cores with 2300 base frequency.
Re: Can MultiCharts handle hundreds or thousands of strategies?
Thank you for your response Svetlana! Can you verify that calculation is parallelized across charts? I have a separate, custom algorithm generator ("factory") which runs walkforward and chooses parameters, so I won't be needing optimization in MultiCharts... simply looking for it to run a large number of charts, 100+, in parallel, in real time.
Totally get the higher clock speed and I absolutely will be using the fastest processors possible on my prod rig... and I use water coolers and overclocking as well.
Totally get the higher clock speed and I absolutely will be using the fastest processors possible on my prod rig... and I use water coolers and overclocking as well.
Re: Can MultiCharts handle hundreds or thousands of strategies?
I think it's possible to place separate MC processes into separate cores. It's called affinity, but I think it can only be done in Windows 10 or 11.
For instance, place tsserver.exe into it's own core and RTPServerProxy.exe into it's own core.
Not sure about the parallelization issue.
For instance, place tsserver.exe into it's own core and RTPServerProxy.exe into it's own core.
Not sure about the parallelization issue.
- Svetlana MultiCharts
- Posts: 645
- Joined: 19 Oct 2017
- Has thanked: 3 times
- Been thanked: 163 times
Re: Can MultiCharts handle hundreds or thousands of strategies?
Here’re general recommendations on how to improve the performance:Thank you for your response Svetlana! Can you verify that calculation is parallelized across charts? I have a separate, custom algorithm generator ("factory") which runs walkforward and chooses parameters, so I won't be needing optimization in MultiCharts... simply looking for it to run a large number of charts, 100+, in parallel, in real time.
Totally get the higher clock speed and I absolutely will be using the fastest processors possible on my prod rig... and I use water coolers and overclocking as well.
1. disable your anti-virus and all excessive applications running in background;
2. make sure none of the applications are doing any updates/scans while you are trading;
3. make sure Windows is not downloading and installing any updates while you are trading;
4. create several desktops and split the number of charts you have between MC desktops*.
*This approach does not split all MultiCharts processes between the cores. It mostly improves graphical rendering: For example, if you have split a workspace with 100 charts to say 10 desktops with 10 charts, the computer will have to process 10 threads with a queue of objects from 10 charts, instead of 1 thread with a queue from 100 charts.
If you have multiple objects to draw in one desktop it may be reasonable to split them between multiple desktops.
This approach does not reduce the load, but helps to distribute it more evenly. This solution requires testing in your specific setup in your environment.
In order to determine what specific resources are required, you can start your usual workspace and watch the load in Task Manager. Adding RAM is reasonable only if the computer lacks RAM.
Re: Can MultiCharts handle hundreds or thousands of strategies?
Awesome, this is consistent with the advice a new friend/fund manager out of France told me this morning, he has production all on one machine but splits his strategies across MC desktops (20 charts each). Also uses TS data and IB as the broker... I'll be trying this setup asap.If you have multiple objects to draw in one desktop it may be reasonable to split them between multiple desktops.
Re: Can MultiCharts handle hundreds or thousands of strategies?
I will look into affinity for prioritizing the TS data stream (data access layer), thank you for the tip. And is RTPServerProxy.exe the MC broker access layer?I think it's possible to place separate MC processes into separate cores. It's called affinity, but I think it can only be done in Windows 10 or 11.
For instance, place tsserver.exe into it's own core and RTPServerProxy.exe into it's own core.
Not sure about the parallelization issue.
Re: Can MultiCharts handle hundreds or thousands of strategies?
Just saw this, right now I'm running them all as separate charts in TS with TradeManager checking off. So, any position mismatch fix is manual and I have to check very frequently and do full platform restarts ~5x per day to identify these mismatches and then fix them manually. Happily I trade all swing strategies so the approximately 10 minute full restart rarely affects me if I do it at the right times.Secondly, how are you getting all of those signals coordinated for a single instrument ?
Also the strategies are the standard recipe: EasyLanguage applied to a continuous tradable contract on some minute bar size, all execution next bar, no IBOG or tick level things. Lowest timeframe is 15 min. I do use data2, 3, and 4 often, with either correlated markets, higher timeframes, or both.
- Mark Brown
- Posts: 187
- Joined: 29 Nov 2016
- Has thanked: 130 times
- Been thanked: 18 times
Re: Can MultiCharts handle hundreds or thousands of strategies?
Why doesn't MC split the processes ?
4. create several desktops and split the number of charts you have between MC desktops*.
*This approach does not split all MultiCharts processes between the cores.
Re: Can MultiCharts handle hundreds or thousands of strategies?
Use MC12 instead of 14. We 've done the testing and only MC12 is able to run many charts at the same time with 10 years data with APPL. we were told that this will be fixed in the next up coming version.
- Svetlana MultiCharts
- Posts: 645
- Joined: 19 Oct 2017
- Has thanked: 3 times
- Been thanked: 163 times
Re: Can MultiCharts handle hundreds or thousands of strategies?
This approach creates several MultiCharts.exe, but all of them share the same TradingServer.exe and TsServer.exe.Why doesn't MC split the processes ?
4. create several desktops and split the number of charts you have between MC desktops*.
*This approach does not split all MultiCharts processes between the cores.
- Svetlana MultiCharts
- Posts: 645
- Joined: 19 Oct 2017
- Has thanked: 3 times
- Been thanked: 163 times
Re: Can MultiCharts handle hundreds or thousands of strategies?
By the way, the Fail-Watch system is implemented since MultiCharts 14. It means one will be notified if the limits of the OS, hardware, bandwidth, software and/or broker processing speed are reached.
MultiCharts 12 does not feature this system, i.e. one risks to be unaware of such situations.
MultiCharts 12 does not feature this system, i.e. one risks to be unaware of such situations.
Re: Can MultiCharts handle hundreds or thousands of strategies?
Why doesn't TradingServer.exe and TsServer.exe execute in separate cores from the desktop instance ?
This approach creates several MultiCharts.exe, but all of them share the same TradingServer.exe and TsServer.exe.
- Svetlana MultiCharts
- Posts: 645
- Joined: 19 Oct 2017
- Has thanked: 3 times
- Been thanked: 163 times
Re: Can MultiCharts handle hundreds or thousands of strategies?
TradingServer.exe and TsServer.exe act as databases in the memory, therefore they cannot be split. Moreover, MultiCharts is a software with multiple brokers/data feeds, and we have to refer to a broker/data feed from a single process (few brokers/feeds allow multiple logins in the frames of a single account).Why doesn't TradingServer.exe and TsServer.exe execute in separate cores from the desktop instance ?
This approach creates several MultiCharts.exe, but all of them share the same TradingServer.exe and TsServer.exe.
Re: Can MultiCharts handle hundreds or thousands of strategies?
Thanks again for all of your responses; I'm now running 472 strategies (across 30 desktops) with what seems to be fairly successful execution. Unfortunately, the GUI is frozen for most of the open time (until about 11am est) and I get a lot of popups saying there's a high CPU load when there isn't (I guess this is a single thread process getting bogged down?). I am using a cloud computer with Intel Xenon which maxes at 5Ghz so I don't know how to get a faster thread.
A few questions:
Mark
A few questions:
- Would a GPU help the GUI remain stable? When I say the GUI I mean Windows itself... all windows freeze including things like task manager
- Is there any way to run the strategy code and submit orders without actually printing the candles? Like if I minimize the chart windows would that help the Windows GUI process at all, and if not is there anything else I can do?
- Is there a way to turn off the warning/error message popups and just have them in the log? They seem to use a massive amount of graphics and the entire system freezes horribly when there's a lot of popups coming in
Mark
-
- Posts: 87
- Joined: 10 Feb 2022
- Has thanked: 53 times
- Been thanked: 11 times
Re: Can MultiCharts handle hundreds or thousands of strategies?
to your statement: what does the performance monitor in the task manager say which cores are busy? All? evenly?
There are now CPUs with a large number of cores that are designed for parallel work. (threadripper e.g.) maybe that would be a solution.
Basically, I'm wondering what's the point of trading over 150 (different?) strategies? It shouldn't bring much benefit. I would always only trade the most successful ones...
There are now CPUs with a large number of cores that are designed for parallel work. (threadripper e.g.) maybe that would be a solution.
Basically, I'm wondering what's the point of trading over 150 (different?) strategies? It shouldn't bring much benefit. I would always only trade the most successful ones...
-
- Posts: 87
- Joined: 10 Feb 2022
- Has thanked: 53 times
- Been thanked: 11 times
Re: Can MultiCharts handle hundreds or thousands of strategies?
*
resource monitor is the right name
resource monitor is the right name