IQFeed employees a circular buffer for it's tick-level data: 6 months (180 days) rolling based on calendar days (not trading days).
I want to start to capture the oldest tick data each week (before it rolls off) for purposes of backtesting in the future with a dataset greater than 6 months.
Is this possible to pull off with Quote Manager and using qmd export files ? Or do I need to roll my own solution via scripting ?
Eventually, I would like to put this data into an SQL Server database table.
Retaining historic tick data
-
- Posts: 275
- Joined: 22 Apr 2014
- Has thanked: 70 times
- Been thanked: 72 times
Re: Retaining historic tick data
I am actually starting to look at the same thing but for a slightly different reason. I have no problem with IQFeed's live data but their historical data is not clean at all at least with equities at the tick level or minute level.
I will post details when I have them. At the moment it consists of mongo db, parquet files and a pit of python. At this point it is just simple experiments.
https://github.com/manahl/arctic
I will post details when I have them. At the moment it consists of mongo db, parquet files and a pit of python. At this point it is just simple experiments.
https://github.com/manahl/arctic
-
- Posts: 275
- Joined: 22 Apr 2014
- Has thanked: 70 times
- Been thanked: 72 times
Re: Retaining historic tick data
Hi syswizard,
I am either exporting using csv files or bringing in the data directly into R from IQFeed. Most of my signals are generated outside of Multicharts so for now it is about cleaning and storing that cleaned data. I am not using tick data.
You could take a look at R and Quanttools. It might do what you want. https://quanttools.bitbucket.io/_site/index.html
regards,
Alex
I am either exporting using csv files or bringing in the data directly into R from IQFeed. Most of my signals are generated outside of Multicharts so for now it is about cleaning and storing that cleaned data. I am not using tick data.
You could take a look at R and Quanttools. It might do what you want. https://quanttools.bitbucket.io/_site/index.html
regards,
Alex
Re: Retaining historic tick data
Thanks for that Huges....and indeed it appears Quanttools can read the IQ Feed servers directly using their API.
The only thing that confused me was the reference to "local storage".
Where exactly is the data stored ? I saw no references to file names or paths.
The only thing that confused me was the reference to "local storage".
Where exactly is the data stored ? I saw no references to file names or paths.
-
- Posts: 275
- Joined: 22 Apr 2014
- Has thanked: 70 times
- Been thanked: 72 times
Re: Retaining historic tick data
Quanttools is straight forward. Better than some of the python options that I looked at that required a developer license from DTN. Quanttools has its own database which I have not tested. Instead I am experimenting with Parquet files and MongoDB but MySQL or MariaDB would work just as well for historical data. I became curious after reading this post on the Blackarbs blog about bringing in data directly from IEX.
http://www.blackarbs.com/blog/download- ... nd-parquet
http://www.blackarbs.com/blog/download- ... nd-parquet