Practical question about data and computer setups

Hello,

I have been trying to break into this industry for years and within the last year It has started to come together. I come for the finance side and not the computer engineering side. So my practical knowledge about coding and hardware is still a big part of my learning curve.

I wanted to pose a question with hopefully some inciteful and helpful conversation about practical storage/backup/processing.

Just as a starting point for my setup.
I have 3 dedicated computers now. All of them are high end consumer level. Each has their own MC license but I only have one subscription to IQFeed and thats a little annoying. Planning on uping that to at least 2 soon.

I run my signals locally. I have gig speed internet here at the house and run the signals from that. This is not my end goal though. Obviously, getting them into the cloud would be a much safer way to run them. Just haven't gotten that far yet.

I trade statistical arbitrage signals mostly on SP500 E-minis. Mostly structural market fluctuations with trades lasting a few minutes to an hour.

The main questions I have for the more experienced members of the forum is what does your setup look like. I have just started expanding to all the other E-Mini contracts and with that I have started to download a LOT of additional data. Especially when you are testing on Bid/Ask spreads! These files are getting massive!

What is your procedure for data management? Do you capture data into a home server? Do you grab the data directly from the data provider or do you DL it through MC while backfilling a chart? Do you bother with capturing Live and Historical data to measure the difference?

Also, if you are trading the Emini contracts... Do you collect data on all of them? Or are some of them illiquid enough not to bother with?

I have never had the chance to work in an office or even have a regular conversation with another systematic trader with experience. I am entirely self taught. So any non Alpha related practical help would be greatly appreciated.

Are you using the IQFEED data for bid/ask data or collecting live? Do you use bid/ask spread for more accurate fill information only or in strategy logic? Because, if it is fill information related you could simply apply slippage model in many liquid products. I figure very few systematic traders are doing what you are doing: most systematic traders are building strategies off simple time data.

Are you using the IQFEED data for bid/ask data or collecting live? Do you use bid/ask spread for more accurate fill information only or in strategy logic? Because, if it is fill information related you could simply apply slippage model in many liquid products. I figure very few systematic traders are doing what you are doing: most systematic traders are building strategies off simple time data.

Thanks for the response.

I am using IQFeed for all data. I use IB as my broker

I have found that there can be a pretty measurable difference between 1 tick past close for fill on limit orders and using BId Ask for extending testing. Can you elaborate on your personal experience?

@Joebone I have not used bid/ask testing, yet. I don't have enough experience with Multicharts backtesting engine enough to say. However, on NT, the backtesting results were always accurate for the product I was trading except for being 1 tick optimistic. That's market orders, of course. So, that was pretty obvious and easy to account for when comparing the live and backesting results. MC probably is similar in that regards.

The most accurate fills engine I know is running Rithmic on paper/sim account. Taking the same trades in live and Rithmic, results were very similar. It was slightly optimistic in that I would get filled on a first touch more often on the simulator. Most of the systems I have built are looking for way more then a couple ticks and so just the standard backtesting results have been fairly informative as to performance.

Right, limit order exceeds will be pessimistic while fill on limit order will be optimistic. You can calculate your toxic fill ratio by counting the number of times you get filled without taking X tick heat before getting X tick profit. IB for futures might not be ideal. I had a bad experience with them, at least but if it works for you that's great. If you are trying to do very precise fills you might look at Rithmic supported broker.

If you are trying to trade futures spreads, you might look at trading the exchange supported spreads. I do not have any experience with spreads.

@Joebone I have not used bid/ask testing, yet. I don't have enough experience with Multicharts backtesting engine enough to say. However, on NT, the backtesting results were always accurate for the product I was trading except for being 1 tick optimistic. That's market orders, of course. So, that was pretty obvious and easy to account for when comparing the live and backesting results. MC probably is similar in that regards.

The most accurate fills engine I know is running Rithmic on paper/sim account. Taking the same trades in live and Rithmic, results were very similar. It was slightly optimistic in that I would get filled on a first touch more often on the simulator. Most of the systems I have built are looking for way more then a couple ticks and so just the standard backtesting results have been fairly informative as to performance.

Right, limit order exceeds will be pessimistic while fill on limit order will be optimistic. You can calculate your toxic fill ratio by counting the number of times you get filled without taking X tick heat before getting X tick profit. IB for futures might not be ideal. I had a bad experience with them, at least but if it works for you that's great. If you are trying to do very precise fills you might look at Rithmic supported broker.

If you are trying to trade futures spreads, you might look at trading the exchange supported spreads. I do not have any experience with spreads.

Thanks for this insight.

What about Data storage and cloud computing? would you mind sharing your best practices there? Why was IB not a suitable futures broker?

@Joebone

Well, this was many many years ago. I had traded with OEC at the time for over a year with no problems. The problem was MC and IB would get confused as to my position and I was scalping off the MC DOM. This meant my positions didn't match. I would try to flat and get reversed. Sometimes multiple times. The micros didn't exist then to test things. At that IB constantly updated their platform and so MC would need to be updated. I moved to the IB booktrader which seemed like it would be acceptable. Well, again the stops aren't real exchange stops unless you set a configuration. It was just problem after problem that I never experience with a dedicated FCM. Eventually I got it working "ok" but gave up on the MC. IB also uses snapshot data or did at the time while providers like Rithmic have full level 3 data or CQG. I imagine most pros are using CQG or Rithmic. Rithmic is expensive though and may not be worth it.

I can say that more recently I have used NT 8 connected to AMP and it was rock solid. I have grandfather license for it, as well. I am not executing through MC yet but hoping it will be solid like NT at least when connected to a broker like AMP. I moved my system development to MC because TS is really having problems now and it is easier to develop compared NT. IN NT, I was spending too much time trying to do trivial stuff like setup my windows.

As for cloud computing, the dominant leader is AWS. In terms of cost, I am not convinced it is worth it. You can get much more powerful PC at home and own it outright. You would be running likely a Windows box vs cloud services. You would need just to spec out appropriate EC2 instance. The benefit would be if you were planning to travel and wanted a box you can access remotely. If you wanted to use SQL server, you could run it as a local instance. You can schedule any backup configuration and it is not expensive. If you wanted to be able to move your configuration, you could run everything inside your EC2 in a virtual machine: see Hyper-V.

@Joebone

As for cloud computing, the dominant leader is AWS. In terms of cost, I am not convinced it is worth it. You can get much more powerful PC at home and own it outright. You would be running likely a Windows box vs cloud services. You would need just to spec out appropriate EC2 instance. The benefit would be if you were planning to travel and wanted a box you can access remotely. If you wanted to use SQL server, you could run it as a local instance. You can schedule any backup configuration and it is not expensive. If you wanted to be able to move your configuration, you could run everything inside your EC2 in a virtual machine: see Hyper-V.

Do you store all your data on a home PC? How do you deal with the potential of a power outage or internet loss?

@joebone

Yes, I monitor my systems in real-time. I have in the past used Hyper-V with different virtual machines for working on systems while running live. There are many cloud storage solutions Dropbox, Onedrive, Google drive, etc. for backups. As for power outage, if you have a laptop setup and at the ready then that can handle that. As for internet loss, if you have a cell-phone data then you could use it in an emergency but you need to test because it may not work as expected (didn't in my case-- internet worked but couldn't get TS to connect!!). If you are trading size or where stops are critical, I would just use OCO orders with exchange-held stops. Storage space is cheap these days, too-- if you want to backup locally.

As far as I can see, the main benefit to running in the cloud is if you aren't going to be home to monitor your systems then you can run an EC2 instance and have access when travelling. But if you are home most of the time, I don't see a huge advantage. In any regards, it should be easy to move my workload to the cloud if the need arises. If you are only away for short periods, you can monitor using remote access software like Logmein or Anydesk but I wouldn't trust it for more than a couple days away.

If you want to better manage multiple PC's at home, you could use J5 create USB cable and a monitor with 2 inputs or remote desktop. As well, there are monitors supporting multiple devices like the Dell 32 with built in KVM switch (don't have a need for it personally). I cannot see a need for more then 2 boxes at home and that's not required.

Regarding data, if you are referring to market data then I am not collecting data. I have used TS data or IQFEED in past. But, I have not collected my own data historically though I am looking into it for bid/ask testing.

If you really want to run in the cloud, I would suggest setup an appropriate EC2 instance and see how it works. The one thing with running multiple instances is to make sure you accidently disconnect the live data when working remotely-- this can be resolved with little thought like using CQG and Rithmic. I could see myself moving operations to an EC2 instance at some point in the future-- but for now I am okay running locally and moving shouldn't be that difficult.

Personally, I see the broker as the biggest risk. I have had a lot of trouble with TS on equities with delayed fills and so forth. I would imagine most systematic developers mitigate the risks you fear through their sizing and holding period or usage of OCO orders.

Thanks very much for those suggestions I will take them under consideration.

Regarding data, if you are referring to market data then I am not collecting data. I have used TS data or IQFEED in past. But, I have not collected my own data historically though I am looking into it for bid/ask testing.

What do you mean by this? Do you trade on Tick or Minute bars? where do you get your historical data? CQG?

@joebone I have used IQFEED in the past. I am using TS now. I found both adequate but TS data is free. I prefer to backtest on tick data but I use minute data for long historical backtesting results. I have only used CQG and Rithmic for real-time data.

I gather you are collecting the bid/ask and tick data in real-time for backtesting? Keep in mind you can run proxy-studies in the real-time-- assuming the real-time paper trading account is accurate. For example, let us imagine you have a strategy that runs on minute bars (5, 10, whatever) and you want to know how accurate your results are. Well, you can setup in the paper account a proxy strategy that buys and sells every 1 minute bar: so you get can get a lot of data fast. Obviously, conditions can change with volatility but even over a few days, you would have several thousand data points to inform your decision on what is realistic in terms of slippage.

That said, I have not found a way to backtest scalping systems that use the order book and order flow adequately which I do use in discretionary scalping.