Backtesting - bar resolution

Questions about MultiCharts and user contributed studies.
User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Backtesting - bar resolution

Postby syswizard » 03 Jan 2020

I need to do some backtesting and optimization utilizing tick data. Is there any performance advantage towards compressing the tick data down to "x" number of ticks.....say 200 ? Or does MC save the compressed tick data automatically in the cache ?
For 2 million ticks, I'd be reducing the number of data points considerably.
Also, when I have the tick bars formed, how do I tell MC "here are 200 tick bars....do not compress these".

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Backtesting - bar resolution

Postby Svetlana MultiCharts » 08 Jan 2020

Hello, syswizard,

MultiCharts compresses data only when it is being saved to the database. It happens when one closes all MultiCharts processes or uses “Flush Cached Data to Database” option manually (MultiCharts->File tab, QuoteManager->Edit tab).
Data in the cache is not compressed.

User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Re: Backtesting - bar resolution

Postby syswizard » 08 Jan 2020

I read this: https://www.multicharts.com/trading-sof ... Data_Cache
I am still confused. This all seems to work like magic with the Database. When opening a chart, how does MC know to use the database vs. the raw data from the data source ? In other words if I open the same chart in another workspace will MC know to use the database ?
When performing optimizations, is there any advantage to using the Database instead of the cache ?
Is there any way to specify to MC "Use the database" vs. "Use the data source" ?

User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Re: Backtesting - bar resolution

Postby syswizard » 09 Jan 2020

You have mistaken the computer operation.
Nothing is "Compressed".
When you create a 200-tick bar, the process is not called "Compression".
Sorry to disagree. Ticks are converted to OHLC bars and that "compresses" the data from milllions of ticks to thousands of bars.
Why on earth would there be a "Flush Cached Data to Database" operation ?
Especially if it doesn't store compressed OHLC data ? Otherwise, what would be the purpose of the Database ?
So many other questions arise:
1) What is the "Database" ?
2) What is it's purpose ?
3) How does it store data ?
4) At what resolution ?
5) What benefits do I get from using the Database ?
6) How do I insure access to the Database ?
There is no such thing as compression.
ps. Do this experiment:
First, create a 200-tick bar chart.
Then, change the chart resolution to 201-tick bar chart.
Watch how MultiCharts can instantly re-draw the chart to the new resolution.
That is because nothing is "compressed". All your data is still there waiting for you.
I don't want all of those millions of ticks to "be there waiting for me". I want MC to operate on my 200 tick resolution and do the optimization on the same resolution.
That's why I asked if it was desirable to purchase tick data and convert it to my own tick resolution for purposes of backtesting. Or does MC handle this automagically ?

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Backtesting - bar resolution

Postby Svetlana MultiCharts » 10 Jan 2020

Syswizard,

Please refer to the following general information about data saving in MultiCharts.

Once the historical data is downloaded from the data feed server and plotted on the chart - it is stored in the cache. In 64-bit MultiCharts by default the cache is in PC's RAM. You can see the process TsServer.exe in the processes list for that.
Next time you plot a chart the required historical data is loaded from the cache or data base first, then the missing historical data and realtime data are requested from the data provider.
When you close MultiCharts, QuoteManager and Portfolio Trader, the cached data is compressed and saved to the local database. In MultiCharts 12 we have also added an option to “Flush Cached Data to Database” manually (MultiCharts->File tab, QuoteManager->Edit tab).
When data is loaded from the database it is being decompressed, that is why it might take some time to load it if a lot of data is utilized.
All requests for missing data that are sent to the data feed are logged in the Event Log of QuoteManager. You can see the instrument, resolution, interval that is being requested to fill the gaps.

For more information please see this page:
http://www.multicharts.com/trading-soft ... s_Database

That's why I asked if it was desirable to purchase tick data and convert it to my own tick resolution for purposes of backtesting. Or does MC handle this automagically ?
You do not need to convert tick data to your resolution by yourself, because MultiCharts does it automatically.

If after reading the above info you still have any questions — please let us know.

User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Re: Backtesting - bar resolution

Postby syswizard » 12 Jan 2020

When you close MultiCharts, QuoteManager and Portfolio Trader, the cached data is compressed and saved to the local database. In MultiCharts 12 we have also added an option to “Flush Cached Data to Database” manually (MultiCharts->File tab, QuoteManager->Edit tab).
I just want to make sure we are on the same page re: "compressed". From my perspective, "compressed" is taking a tick data stream and converting it to xxx ticks or minutes or hours as specified when setting up the chart. That would mean that 1 million ticks coming from the data source (no bid/ask) would translate into roughly 5,000 records in the database for a 200 tick resolution of Open, High, Low, Close, Volume (OHLCV) data.
When data is loaded from the database it is being decompressed, that is why it might take some time to load it if a lot of data is utilized.
This I don't understand this. Once the tick data is compressed, how can it be decompressed ? Also, since the chart is using a 200 tick resolution, it no longer needs tick-level data....only the OHLCV for it's resolution. I don't quite see why there is decompression happening here......seems like a lot of extra processing.

Zheka
Posts: 223
Joined: 13 Jan 2016
Has thanked: 8 times
Been thanked: 53 times

Re: Backtesting - bar resolution

Postby Zheka » 13 Jan 2020

'Compressing' here means 'compressing to an internal database format for storage'.

Why not just open QuoteManager and have a look at the database data for your symbol?

User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Re: Backtesting - bar resolution

Postby syswizard » 14 Jan 2020

'Compressing' here means 'compressing to an internal database format for storage'.
Why not just open QuoteManager and have a look at the database data for your symbol?
Thanks for that suggestion. I had a funny feeling that our definitions of "compression" were different.
It does appear that MC's compression algo may or may not be equivalent to building and storing the "xxx" ticks bars I want to use.
The record compression would be equivalent to the # of ticks in the bar: 200 ticks would be equivalent to 1/200 or 0.005 compression ratio.
Can their compression method exceed that ? I think not.

User avatar
Svetlana MultiCharts
Posts: 645
Joined: 19 Oct 2017
Has thanked: 3 times
Been thanked: 163 times

Re: Backtesting - bar resolution

Postby Svetlana MultiCharts » 14 Jan 2020

I just want to make sure we are on the same page re: "compressed".
By compressed we mean that the file with data will take less place on the disk compared to uncompressed data. Data is not compressed by converting it from tick to xxx ticks, or minutes, or hours, it is stored in the same tick format.

User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Re: Backtesting - bar resolution

Postby syswizard » 14 Jan 2020

Thanks for that Svetlana......now I am aware of the overhead involved with retrieving tick data....even from the "Database".
Going with my definition of "compressed" data would really save a ton of processing time IMHO.


Return to “MultiCharts”