MC slows down during announcements

Questions about MultiCharts and user contributed studies.
kecha1
Posts: 11
Joined: 01 Jul 2014
Has thanked: 1 time
Been thanked: 2 times

MC slows down during announcements

Postby kecha1 » 11 May 2015

Hello,
we are using simultaneously 40 robots connected to 5 Interactive Broker accounts.
So we have 40 charts open at the same time. We trade FOREX and Gold.
Our connection with IB is through the gateway.
During rush trading hours (US opening, announcements) we notice that the MC is slowing down.
More specifically the clock on the bottom right is running erratically, for example it freezes at 00 secs and then jumps to 03 secs.
Also charting sometimes freezes and trailing orders may not be executed.
We are not sure if execution of strategy created orders is missed and we have to run a backtesting afterwards to figure out.
Memory usage on our server during this period is increasing but no more that 50 percent.
Also CPU usage is increasing but also below 50 percent.
That means that other applications on the server are not delayed, just Multicharts.
MC is back to normal after some hours.
Assuming that our code is not creating the issue, what other measures could we take to resolve it?
Should we run more MC instances and divide the workload, ie have 20 charts in each MC instance?
It does not seem to be a hardware issue with our server.
Has anybody else experience this situation?

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: MC slows down during announcements

Postby TJ » 11 May 2015

Hello,
we are using simultaneously 40 robots connected to 5 Interactive Broker accounts.
So we have 40 charts open at the same time. We trade FOREX and Gold.
Our connection with IB is through the gateway.
During rush trading hours (US opening, announcements) we notice that the MC is slowing down.
More specifically the clock on the bottom right is running erratically, for example it freezes at 00 secs and then jumps to 03 secs.
Also charting sometimes freezes and trailing orders may not be executed.
We are not sure if execution of strategy created orders is missed and we have to run a backtesting afterwards to figure out.
Memory usage on our server during this period is increasing but no more that 50 percent.
Also CPU usage is increasing but also below 50 percent.
That means that other applications on the server are not delayed, just Multicharts.
MC is back to normal after some hours.
Assuming that our code is not creating the issue, what other measures could we take to resolve it?
Should we run more MC instances and divide the workload, ie have 20 charts in each MC instance?
It does not seem to be a hardware issue with our server.
Has anybody else experience this situation?
What is your CPU?

40 robots on 5 accounts is a lot, but doable; it depends on how efficient is your code.

The TaskManager CPU utilization does not tell the whole story. If you have wait states in your program, it is not going to show up on the graph.

At the bottom right corner of your MultiCharts, you should see the data queue. Usually it reads 0q/0s. If you see numbers other than zero, or if the numbers turn red, that means there is a fast market, and MultiCharts is behind in processing the quotes.

kecha1
Posts: 11
Joined: 01 Jul 2014
Has thanked: 1 time
Been thanked: 2 times

Re: MC slows down during announcements

Postby kecha1 » 11 May 2015

using a server from https://www.hosteurope.de/en/Server/Root-Server/XXL/ with the following characteristics:
CPU 12 vCores
RAM 60 GB
RAID Hardware RAID10,
SSD HD

We were red on the data queue due to logging and then disabled the TradingServer logging by importing a registry key, a fix to be applied to future MC versions.

I think (have to confirm) no numbers are shown there, certainly no red colours.

No wait states in the program.
Intrabargeneration set to TRUE, but most code is run on condition barstatus=2.
Large TF varying from (1H to 1D, mostly on 4H)

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: MC slows down during announcements

Postby Henry MultiСharts » 12 May 2015

Hello kecha1,

Do you have an antivirus/firewall running on your PC?
Which exact process is using most of the CPU when the slow down happens? Please attach a video or screenshots showing MultiCharts processes resource usage and overall system load.
Which data provider do you use?
Does the time in the status line of the chart lag behind the current time?
Have you tried running plain charts with no studies? Do they slow down as well?
Per TJ's post - please keep an eye on the time/quote/order delay indicator at the bottom right corner of the MultiCharts window. If it shows up with some numbers in red - there is a delay in processing realtime data/orders.

You can also come to our live chat Monday-Friday 6:30 am - 1 pm EST to demonstrate this behavior remotely. We will do our best to help you.

kecha1
Posts: 11
Joined: 01 Jul 2014
Has thanked: 1 time
Been thanked: 2 times

Re: MC slows down during announcements

Postby kecha1 » 12 May 2015

Hello Henry,

No anivirus or Firewall.
The process using most of the CPU is Multicharts_64 exe.
Data provider is IB
Yes, the time in the status line of the chart lags behind the current time. It catches on and lags again many times until resuming to normal.
No, we did not try running plain charts without studies. Setup is for live trading.
No red in the bottom right corner, or numbers in there.

I will provide video or screenshots when available.
Thank you for your support

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: MC slows down during announcements

Postby Henry MultiСharts » 13 May 2015

Hello Henry,

No anivirus or Firewall.
The process using most of the CPU is Multicharts_64 exe.
Data provider is IB
Yes, the time in the status line of the chart lags behind the current time. It catches on and lags again many times until resuming to normal.
No, we did not try running plain charts without studies. Setup is for live trading.
No red in the bottom right corner, or numbers in there.

I will provide video or screenshots when available.
Thank you for your support
kecha1,

As the process using most of the CPU is Multicharts64.exe - you can start with splitting your charts into multiple instances.

kecha1
Posts: 11
Joined: 01 Jul 2014
Has thanked: 1 time
Been thanked: 2 times

Re: MC slows down during announcements

Postby kecha1 » 14 May 2015

Henry, thank you for the suggestion.

I have already initiated a second instance of MC, so now the first one runs 35 charts and the second 15. To fully implement the strategy I need to add 5 more.
I do no have yet a feeling on how the split solution is performing.
For my hardware and software setup, I think 40 charts are the limits for one MC instance.
Now, since both instances are run on the same computer and are connected via the same gateway to the broker, i do not know what to expect.
I could assess that by splitting, graphics handling is simpler because there are less charts to be included in each MC window. What about the rest of the functions?

In live trading, I understand that the MC instances in a computer have to be also connected to MC servers. Does this connection induce any latency?

escamillo
Posts: 203
Joined: 25 Mar 2011
Has thanked: 23 times
Been thanked: 56 times

Re: MC slows down during announcements

Postby escamillo » 14 May 2015

How many other users are on the serving you are using?
How many indicators and strategies do you have running on each open chart?
How many days back are they calculating?
Tick data or minute data for some or all?

My experience with MC is that any CPU can be buried during those times of peak incoming ticks and price changes if too much is asked of a computer.

My experience with MC is also that a high end CPU with only currently used strategies turned On, only a few days of tick data and a limited number of symbols will keep up with anything I have yet to see thrown at it (post flash crash).

Perhaps you are asking a machine to do more than it can handle if too much is being asked of it in too short a time?

I have not used IB data for years, however, I have not heard anything particularly good about it in recent years. Perhaps it is at least partially an IB data issue?

kecha1
Posts: 11
Joined: 01 Jul 2014
Has thanked: 1 time
Been thanked: 2 times

Re: MC slows down during announcements

Postby kecha1 » 14 May 2015

Escamillo, thank you for your post.
Following are the replies to your questions:

How many other users are on the serving you are using?
I have to confirm with the data center, but i think we are the sole users.
How many indicators and strategies do you have running on each open chart?
One strategy per chart, 0-2 indicators (mostly 0)
How many days back are they calculating?
Pending on the chart, from 3 to 10 Y
Tick data or minute data for some or all?
Minute data for all
Am I asking too much from the machine?
I do not know, because CPU and Memory never reach above 50%. Obviously there is a bottleneck during live trading, but haven't figured out where (CPU, memory, I/O, Graphics, MC or IB). In backtesting runs, CPU reaches above 90%.

If the hardware in my side is the limitation, i can adjust, but I am not certain this is the case.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: MC slows down during announcements

Postby TJ » 14 May 2015

::
How many days back are they calculating?
Pending on the chart, from 3 to 10 Y
Tick data or minute data for some or all?
Minute data for all
::
Here is a possible problem... for minute charts, most people use no more than 1 month's history.

You can try to use less history to see if the situation improves.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: MC slows down during announcements

Postby TJ » 14 May 2015

::
Am I asking too much from the machine?
I do not know, because CPU and Memory never reach above 50%. Obviously there is a bottleneck during live trading, but haven't figured out where (CPU, memory, I/O, Graphics, MC or IB). In backtesting runs, CPU reaches above 90%.

If the hardware in my side is the limitation, i can adjust, but I am not certain this is the case.
You will almost never see CPU reach above 50%...

because you use Barstatus=2... everything is on hold until EOB.


let me give you an extreme example...
say, your strategy has 1,000 lines of code.
990 lines of those codes are inside "If Barstatus=2 then begin..."
then you will always see the CPU at idle, except the brief moment at the EOB.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: MC slows down during announcements

Postby TJ » 14 May 2015

A lots of factors influence the processing time...

Check to see if your logic has a lot of look backs,
or array look ups, 3rd party dll accesses, etc.,
or recursive calculations,
or drawing objects,
or text/number conversions.
Those type of calculations usually require more CPU cycles to accomplish.

kecha1
Posts: 11
Joined: 01 Jul 2014
Has thanked: 1 time
Been thanked: 2 times

Re: MC slows down during announcements

Postby kecha1 » 14 May 2015

TJ thank you for your comments and suggestions.

I do not want to compromise the setup, in terms of history on the chart, because it may affect the performance of the strategy.
I will do that only if I am convinced that it is too heavy even for the best machine I can find.

I can and will try to improve the efficiency of the code in order to minimize unessecary calculations.
However, no more that 50-100 lines per strategy.
You will almost never see CPU reach above 50%...

because you use Barstatus=2... everything is on hold until EOB.
When clock is lagging, if my CPU was causing the delay, i should see large numbers there.
If proccess is waiting IB or MC servers to respond, then I wouldn't see large numbers in my CPU albeit the system is slow.
The Barstatus=2 part of the code is not the source of the problem, cause delays are appearing during large traffic only.
Chart updating, limit orders and who knows what other MC proccesses that run on every tick could degrade performance when many ticks are produced.

I have taken a video on the slow clock, but cannot upload videos in here?

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: MC slows down during announcements

Postby TJ » 14 May 2015

One thing others have tried, and you can try too -- that is to disable hyper-threading. Some people have found that under certain circumstances, using virtual cores actually slowed down their processes.

Just a thought... YMMV

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: MC slows down during announcements

Postby Henry MultiСharts » 15 May 2015

kecha1,

MultiCharts authorization does not induce any data latency.
Please send your video to support@multicharts.com

kecha1
Posts: 11
Joined: 01 Jul 2014
Has thanked: 1 time
Been thanked: 2 times

Re: MC slows down during announcements

Postby kecha1 » 28 May 2015

Hello,

we came to a conclusion why MC was slowing down and how to resolve the situation.
Our setup for backtesting (using bid and ask + bar magnifier) together with having many years of historical data on every chart, required a lot of physical memory.
To give some numbers, a year of historical data (bid + ask + 1M bar magnifier) is noticed to increase for about 120MB the memory used by MC exe, per chart of an instrument.
By just reducing the size of historical data we have on the charts, we have a much smoother MC operation, without the previously reported delays of execution during rush trading hours.

Many thanks all forum members for their ideas and suggestions.


Return to “MultiCharts”