I have a problem with running big backtests, my RAM runs out. (16GB)
My question is if i can run the backtest over 3000 symbols and 1 year of backtesting at once without it running into memory problems?
Big Backtest Memory Problems
Re: Big Backtest Memory Problems
I see now that if i have multiple strategies (that i don't use at the same time) that the RAM usage is already higher of the Portfolio Trader. This seems like a bug.
If i run the Portfolio Trader with only one strategy then the RAM usage is lower.
Please fix this
If i run the Portfolio Trader with only one strategy then the RAM usage is lower.
Please fix this
- fbertram
- Posts: 166
- Joined: 16 Oct 2014
- Location: Seattle, USA
- Has thanked: 36 times
- Been thanked: 76 times
- Contact:
Re: Big Backtest Memory Problems
Hi Rien,
please provide a little more detail about the time resolution you are using. In the absence of that information, I can only guess.
But let's do the math here real quick:
* assume end-of-day bars
* 3,000 instruments x 1 bar / day = 3,000 bars / day
* 252 days / year x 3,000 bars / day = 756,000 bars / year
* 6 x 64 bit / bar x 756,000 bars / year = 18,144,000 bytes / year = 18 MB / year
So, end-of-day bars should fit into memory quite easily. If you are using 1 minute bars instead of daily bars, this will be 390 times as much, or about 7GB. Depending on how complex your strategy is, how many indicators you use, and what else is running on that machine, 16GB might be a little tight.
Hope this helps,
cheers, Felix
please provide a little more detail about the time resolution you are using. In the absence of that information, I can only guess.
But let's do the math here real quick:
* assume end-of-day bars
* 3,000 instruments x 1 bar / day = 3,000 bars / day
* 252 days / year x 3,000 bars / day = 756,000 bars / year
* 6 x 64 bit / bar x 756,000 bars / year = 18,144,000 bytes / year = 18 MB / year
So, end-of-day bars should fit into memory quite easily. If you are using 1 minute bars instead of daily bars, this will be 390 times as much, or about 7GB. Depending on how complex your strategy is, how many indicators you use, and what else is running on that machine, 16GB might be a little tight.
Hope this helps,
cheers, Felix
- Anna MultiCharts
- Posts: 560
- Joined: 14 Jul 2017
- Has thanked: 42 times
- Been thanked: 141 times
Re: Big Backtest Memory Problems
Hello, rien!
As we discussed this in Live Chat there are several things you can do:
- decrease the data range,
- decrease the number of symbols,
- distribute your strategies/symbols between several portfolio workspaces.
As for this:
As we discussed this in Live Chat there are several things you can do:
- decrease the data range,
- decrease the number of symbols,
- distribute your strategies/symbols between several portfolio workspaces.
As for this:
Do you mean that you have 2 workspaces: one that contains only 1 strategy/symbol and another that has several strategies, only one of which (the same strategy/symbol combination as on workspace #1) is active and backtested, and these workspaces are giving you different RAM load? Please verify that all strategy settings are identical in both cases.I see now that if i have multiple strategies (that i don't use at the same time) that the RAM usage is already higher of the Portfolio Trader. This seems like a bug.
If i run the Portfolio Trader with only one strategy then the RAM usage is lower.