Massive memory usage on MC.net

Questions about MultiCharts .NET and user contributed studies.
felixtjung
Posts: 24
Joined: 06 Apr 2011
Has thanked: 5 times
Been thanked: 2 times

Massive memory usage on MC.net

Postby felixtjung » 24 Sep 2012

Hi,

I've been using multicharts.net overnight with 5 workspaces (each workspace roughly have 4 charts) monitoring about 14 different instruments using auto trading. I write my own custom signals and functions.

After leaving them on for a night, the memory usage of Multicharts.exe now is 850 MB! and it keep growing. I'm just curious if there's people having same problem. I'm just trying to figure out if this is a known issue of multicharts.net or this is caused by my own custom stuff.

About my own custom stuff, I don't use any custom code for storage or anything like that (I don't even use any custom library nor do I use any array). I just use standard classes provided by MC.net (e.g. VariableSeries, RSI, AvgTrueRange, etc).

Or, is there any trick to reduce memory usage that I should know?

Thanks,
Felix

MidKnight
Posts: 343
Joined: 12 Aug 2012
Has thanked: 123 times
Been thanked: 56 times

Re: Massive memory usage on MC.net

Postby MidKnight » 24 Sep 2012

Hi there,

I'm no expert in MC.NET by any stretch, but as a start I'd make sure your number of days loaded for each chart is kept low. See if you can get away without using a variableseries type and instead use more simple and typed storage. If you don't need things calculated every tick then disable that, just in case you are making too many new objects too frequently.

That's about all I can suggest at this stage and hope it helps somewhat.

With kind regards,
MK
These users thanked the author MidKnight for the post:
felixtjung

bluejack
Posts: 42
Joined: 02 Aug 2012
Has thanked: 25 times
Been thanked: 27 times

Re: Massive memory usage on MC.net

Postby bluejack » 25 Sep 2012

Latest beta had some fix about memory usage. Try it.
These users thanked the author bluejack for the post:
felixtjung

User avatar
Henry MultiСharts
Posts: 8513
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2733 times

Re: Massive memory usage on MC.net

Postby Henry MultiСharts » 25 Sep 2012

The amount of memory in use is highly dependent on your setup. Realtime data is being received, indicators are being calculated and plotted, signals are being generated and orders are being sent to the broker. All of this takes memory and that is natural. The longer you run the platform-the more memory it utilizes.

felixtjung, what was the memory usage once you started MultiCharts and all of the charts were backfilled, indicators plotted, signals calculated? What data resolution do you use (ticks, minutes, days)?
These users thanked the author Henry MultiСharts for the post:
felixtjung

felixtjung
Posts: 24
Joined: 06 Apr 2011
Has thanked: 5 times
Been thanked: 2 times

Re: Massive memory usage on MC.net

Postby felixtjung » 25 Sep 2012

Hi Henry,

each chart has following setup:
* 1-hour data resolution
* display 500 bars
* There's no indicator plotted
* only has one signal object

Once MC is ready with auto-trading turned on, the memory usages 207 MB which is still acceptable. The problem is, it grow larger and larger indefinitely. (After I restarted it last night, now it reaches 1 GB memory usage).

As bluejack, pointed out, the beta has memory usage fix. I'll give it a shot and let you guys know how it goes.

Thanks,
Felix

holya
Posts: 40
Joined: 28 Aug 2012
Location: Vancouver, Canada
Has thanked: 8 times
Been thanked: 1 time

Re: Massive memory usage on MC.net

Postby holya » 26 Sep 2012

I just realized that my MC.net is taking up 2GB! No wonder it's slow, it's %98 full!

felixtjung
Posts: 24
Joined: 06 Apr 2011
Has thanked: 5 times
Been thanked: 2 times

Re: Massive memory usage on MC.net

Postby felixtjung » 26 Sep 2012

Ok,

I've been using MC 8 beta 1. Here's the result:
1. After updating all of my signals class are broken due to change in API. It's fine because the API looks cleaner. Good job guys!
2. Starting MC with all charts with autotrading turned on starts with 147 MB. Perhaps due to the change of the code and the MC improvement that you guys made.
3. I keep it running for 8 hours, and after I checked it, Multicharts.exe uses 438 MB! It still keeps growing.

So, whatever the problem is/are, still present in MC8. It might have something to do with my code or with MC, perhaps the combination of both. I've re-checked my signal code, there's no custom dynamic storage (e.g. List, array or growable data structure) exists in any of my signal code except VariableSeries<double> / standard indicators (e.g. RSI, Lowest, Highest).

I'm not sure if you guys have consider of doing this, perhaps, if you limit the data kept inside those objects to the max number of bar available on chart / max number look back bar of the strategy / whatever that can be used to limit the number of data kept in each ISeries, might solves the problem if that is the problem.

Cheer,
Felix

Zoli
Posts: 89
Joined: 12 Sep 2012
Has thanked: 24 times
Been thanked: 37 times

Re: Massive memory usage on MC.net

Postby Zoli » 26 Sep 2012

Hi,

I left MC.net Beta 8.1 running for like 5 hours now and reached 440.8 MB usage. I have only a DOM window, a 5 min chart going back 20 days and one simple indicator that plots the chart. On regular MC I have m5 chart going back 2 years, m15, m30, h1, h4 and daily charts going back as much as the broker allows (LMAX), all of them with a complex indicator using multiple arrays and drawing but the usage is only 367.7 MB in the same 5 hours.
I have never watched this as I am still learning to code for .NET but I can definitely confirm the memory usage problem.
I forgot to mention that on regular MC on H4 and Daily I am using MB Trading for historical data combined with LMAX realtime.
Both MC's are 64 bit latest release.

Hope this helps.

holya
Posts: 40
Joined: 28 Aug 2012
Location: Vancouver, Canada
Has thanked: 8 times
Been thanked: 1 time

Re: Massive memory usage on MC.net

Postby holya » 26 Sep 2012

I've been monitoring my MC and noticed that it reaches 2 GB in no time! There's a heavy leak somewhere. Hope they fix this or this is going to be a huge problem. :(

Zoli
Posts: 89
Joined: 12 Sep 2012
Has thanked: 24 times
Been thanked: 37 times

Re: Massive memory usage on MC.net

Postby Zoli » 26 Sep 2012

I must say that the debugger was "hanging" in VS2010 according to what I understood and for some reason MC.net was taking up more memory. After Henry fixed the issue it doesn't seem to use up more than 50 MB compared to 400 this morning with the same workspace (had to compile from PL Editor and reload in VS for the debugger to work).
I'm not sure if this is of any help to you, I thought I should share the experience.

DRCM
Posts: 164
Joined: 17 Apr 2011
Location: England
Has thanked: 65 times
Been thanked: 8 times

Re: Massive memory usage on MC.net

Postby DRCM » 26 Sep 2012

Hello,

I run two workspaces on MC.Net x64 : one of them with 16 one hour charts connected to IB, the other has ~20 charts connected to IQFeeD, no indicators except couple moving averages. Started early in the morning with 147MB running all day without restart, at the end of the day still the same 147MB usage. Windows 7 Ultimate SP1, MC.Net 8.1 beta.
Tested for two days.

bluejack
Posts: 42
Joined: 02 Aug 2012
Has thanked: 25 times
Been thanked: 27 times

Re: Massive memory usage on MC.net

Postby bluejack » 26 Sep 2012

I suggest if the devs can't reproduce it to send a memory dump to them. They should be able to analyze this. 2GB mem usage is not normal.

felixtjung
Posts: 24
Joined: 06 Apr 2011
Has thanked: 5 times
Been thanked: 2 times

Re: Massive memory usage on MC.net

Postby felixtjung » 26 Sep 2012

I've tried to remove "most" VariableSeries and all stateful indicators (e.g. AverageFC, LowestFC, HighestFC, etc) as much as possible.

I run it after 8 hours until this point. It started off with 133 MB memory usage, now (after 8 hours running) the memory usage is 151 MB. It's still acceptable for "me", so I won't make a big fuss about it.

But, it might still be a big problem for other users. My hunch is, that's something to do with "ISeries data structures", keeping track of enormous amount of data without pruning oldest data. Another guess is it's something to do with keeping track unnecessary data (e.g. series of tick data even though it's no longer needed by higher time resolution such as 5 min chart).

It seems the only memory usage problematic program is Multicharts.exe only. Perhaps, if you guys could memory profile this program, you might be able to see clearly what happens.

I hope that helps.

holya
Posts: 40
Joined: 28 Aug 2012
Location: Vancouver, Canada
Has thanked: 8 times
Been thanked: 1 time

Re: Massive memory usage on MC.net

Postby holya » 26 Sep 2012

Yes, it is the MultiCharts64.exe that's memory hungry. On the same note, I'm sure MC developers will come up with a solution so we don't have to keep speculating as to what's causing the problem.

This application seems to be capable of handling complex codes and hopefully we won't need to sacrifice our ideas.

User avatar
Henry MultiСharts
Posts: 8513
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2733 times

Re: Massive memory usage on MC.net

Postby Henry MultiСharts » 27 Sep 2012

We are running some tests in order to reproduce this behavior in our environment.

DRCM
Posts: 164
Joined: 17 Apr 2011
Location: England
Has thanked: 65 times
Been thanked: 8 times

Re: Massive memory usage on MC.net

Postby DRCM » 27 Sep 2012

What does tsServer process do actually? Can anyone give more details about it, looks like this process incresed in memory after some time, but I have no time to watch it closely, just noticed that is doubled after about 5 hours of running. Multicharts64.exe has no problems at all on my system, its exatly the same size as it was after all day/ half day of using MC.

User avatar
Henry MultiСharts
Posts: 8513
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2733 times

Re: Massive memory usage on MC.net

Postby Henry MultiСharts » 27 Sep 2012

holya wrote:I've been monitoring my MC and noticed that it reaches 2 GB in no time! There's a heavy leak somewhere. Hope they fix this or this is going to be a huge problem. :(

Which data feed and broker do you use?
How many workspaces, charts, symbols, bars, indicators, signals do you have? Is auto trading turned on? Are you using prebuilt studies or custom ones? Please attach a screenshot of your system properties:
For Windows Vista/7 you can find this information here: Control Panel\System and Security\System or right click on the Computer icon and select properties.
For Windows XP you can find this information here: Control Panel\System\General or right click on the Computer icon and select properties.
Please make a screenshot of this window.

You can also come to our live chat Monday-Friday 6:30 am - 4 pm EST to demonstrate the issue remotely and we will collect all required for analysis information.

User avatar
Henry MultiСharts
Posts: 8513
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2733 times

Re: Massive memory usage on MC.net

Postby Henry MultiСharts » 27 Sep 2012

DRCM wrote:What does tsServer process do actually? Can anyone give more details about it, looks like this process incresed in memory after some time, but I have no time to watch it closely, just noticed that is doubled after about 5 hours of running. Multicharts64.exe has no problems at all on my system, its exatly the same size as it was after all day/ half day of using MC.

tsServer.exe is in charge of loading data and building bars from lower timeframes (ex. points built from ticks). It loads data, processes it, sends to MC or Portfolio and then releases the memory.
These users thanked the author Henry MultiСharts for the post (total 3):
DRCMbluejackZoli

felixtjung
Posts: 24
Joined: 06 Apr 2011
Has thanked: 5 times
Been thanked: 2 times

Re: Massive memory usage on MC.net

Postby felixtjung » 03 Oct 2012

Any news on this?

User avatar
Henry MultiСharts
Posts: 8513
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2733 times

Re: Massive memory usage on MC.net

Postby Henry MultiСharts » 03 Oct 2012

We did some testing for 4 days in our environment and memory usage has not been increased.
We have information from Holya's PC and will be running more tests in order to replicate the behaviour with his setup.
These users thanked the author Henry MultiСharts for the post (total 2):
felixtjungbluejack

User avatar
Henry MultiСharts
Posts: 8513
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2733 times

Re: Massive memory usage on MC.net

Postby Henry MultiСharts » 26 Oct 2012

No performance issues have been reported after update to MC .Net 8.1 beta 1

felixtjung
Posts: 24
Joined: 06 Apr 2011
Has thanked: 5 times
Been thanked: 2 times

Re: Massive memory usage on MC.net

Postby felixtjung » 30 Oct 2012

Henry MultiСharts wrote:No performance issues have been reported after update to MC .Net 8.1 beta 1


How does this problem is marked as "[SOLVED]" when I constantly get this problem all the time!? I've been constantly restarting MC.net every few hours to get around this massive memory usage. I'm using latest MC 8.1 Beta 1 (Build 6048). The problem still persists.

User avatar
Henry MultiСharts
Posts: 8513
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2733 times

Re: Massive memory usage on MC.net

Postby Henry MultiСharts » 31 Oct 2012

felixtjung wrote:How does this problem is marked as "[SOLVED]" when I constantly get this problem all the time!? I've been constantly restarting MC.net every few hours to get around this massive memory usage. I'm using latest MC 8.1 Beta 1 (Build 6048). The problem still persists.

You need to trace Virtual memory usage of Multicharts components on your PC.
Please compare virtual memory once you start MultiCharts and after running it a certain period of time.

Install Process Explorer http://technet.microsoft.com/en-us/sysi ... 96653.aspx
Go to: View->Select columns->Process memory->check "Virtual Size".
Then you need to sort the processes by the Virtual memory column.

Please send me your results for comparison (support@multicharts.com).

If you are experiencing crashes then you need to send us the logs and dumps for analysis.
After the crash appears, without restarting MultiCharts, please collect the logs manually and send to me for analysis.
Here is how to do that:
https://www.multicharts.com/trading-soft ... harts_Logs
These users thanked the author Henry MultiСharts for the post:
Laurentius


Return to “MultiCharts .NET”