Retaining historic tick data

Questions about MultiCharts and user contributed studies.
User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Retaining historic tick data

Postby syswizard » 16 Nov 2018

IQFeed employees a circular buffer for it's tick-level data: 6 months (180 days) rolling based on calendar days (not trading days).
I want to start to capture the oldest tick data each week (before it rolls off) for purposes of backtesting in the future with a dataset greater than 6 months.

Is this possible to pull off with Quote Manager and using qmd export files ? Or do I need to roll my own solution via scripting ?
Eventually, I would like to put this data into an SQL Server database table.

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Retaining historic tick data

Postby hughesfleming » 17 Nov 2018

I am actually starting to look at the same thing but for a slightly different reason. I have no problem with IQFeed's live data but their historical data is not clean at all at least with equities at the tick level or minute level.

I will post details when I have them. At the moment it consists of mongo db, parquet files and a pit of python. At this point it is just simple experiments.

https://github.com/manahl/arctic

User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Re: Retaining historic tick data

Postby syswizard » 17 Nov 2018

Thanks Huges -
So what is your data source ? Can you read the MC database files or qmd files ?

BTW: I wonder how their data quality is with futures tick data ?

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Retaining historic tick data

Postby hughesfleming » 18 Nov 2018

Hi syswizard,

I am either exporting using csv files or bringing in the data directly into R from IQFeed. Most of my signals are generated outside of Multicharts so for now it is about cleaning and storing that cleaned data. I am not using tick data.

You could take a look at R and Quanttools. It might do what you want. https://quanttools.bitbucket.io/_site/index.html

regards,

Alex

User avatar
syswizard
Posts: 295
Joined: 15 Dec 2012
Has thanked: 16 times
Been thanked: 28 times

Re: Retaining historic tick data

Postby syswizard » 21 Nov 2018

Thanks for that Huges....and indeed it appears Quanttools can read the IQ Feed servers directly using their API.
The only thing that confused me was the reference to "local storage".
Where exactly is the data stored ? I saw no references to file names or paths.

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Retaining historic tick data

Postby hughesfleming » 21 Nov 2018

Quanttools is straight forward. Better than some of the python options that I looked at that required a developer license from DTN. Quanttools has its own database which I have not tested. Instead I am experimenting with Parquet files and MongoDB but MySQL or MariaDB would work just as well for historical data. I became curious after reading this post on the Blackarbs blog about bringing in data directly from IEX.

http://www.blackarbs.com/blog/download- ... nd-parquet


Return to “MultiCharts”