Chart loading. Data backfilling.

Questions about MultiCharts and user contributed studies.
DRCM
Posts: 164
Joined: 17 Apr 2011
Location: England
Has thanked: 65 times
Been thanked: 8 times

Chart loading. Data backfilling.

Postby DRCM » 01 May 2011

Hello,

I created 6 workspaces with 9 charts in each (18 symbols, 15min.,1 hour and 1 day each symbol). Backfilled all data from IQFeed first time, it gives approx. 2 yr of forex and 4 yr of stocks data. Saved everything. If i want to open all my charts now it takes about half an hour to backfill again. Is it normal ?
Does MultiCharts backfill data from IQFeed or from local database when you use it after initial backfill? Thanks

User avatar
TJ
Posts: 6518
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Re: Chart loading. Data backfilling.

Postby TJ » 01 May 2011

DRCM wrote:Hello,

I created 6 workspaces with 9 charts in each (18 symbols, 15min.,1 hour and 1 day each symbol). Backfilled all data from IQFeed first time, it gives approx. 2 yr of forex and 4 yr of stocks data. Saved everything. If i want to open all my charts now it takes about half an hour to backfill again. Is it normal ?
Does MultiCharts backfill data from IQFeed or from local database when you use it after initial backfill? Thanks

one thing to check...

before you shut down the computer,
check your TaskManager
In the Processes tab
under the Description column,
make sure all the MultiCharts processes are exited.

If the user shuts down the computer before the processes are finished,
MultiCharts would not have a chance to save the data to the database.

Image

DRCM
Posts: 164
Joined: 17 Apr 2011
Location: England
Has thanked: 65 times
Been thanked: 8 times

Re: Chart loading. Data backfilling.

Postby DRCM » 01 May 2011

Hello,

Yes. When I shut MultiCharts down, no MC processes are running in TaskManager. But when I open MC it backfilling data half an hour again. I checked C:/ProgramData/TS Support/MultiCharts/Database and TSSTORAGE.GDB file shows 101,144 KB . So, it`s about 100MB of data stored in local database and i guess it should be something like that.
Does it mean that MC loading data from local database so slowly? :O

User avatar
TJ
Posts: 6518
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Re: Chart loading. Data backfilling.

Postby TJ » 01 May 2011

Which version of MultiCharts you are using?


I would go to online chat and let technical support check out your computer.
I remember someone asked similar question recently, but I cannot find the thread at the moment.

DRCM
Posts: 164
Joined: 17 Apr 2011
Location: England
Has thanked: 65 times
Been thanked: 8 times

Re: Chart loading. Data backfilling.

Postby DRCM » 01 May 2011

Hi,

I use MC 7 beta 3
Thanks, TJ

btw, I tried to open MC in offline mode and it takes exactly the same time as in online mode.

User avatar
Stan Bokov
Posts: 963
Joined: 18 Dec 2009
Has thanked: 367 times
Been thanked: 300 times

Re: Chart loading. Data backfilling.

Postby Stan Bokov » 02 May 2011

Yes, please come to Live Chat so we can investigate.

DRCM
Posts: 164
Joined: 17 Apr 2011
Location: England
Has thanked: 65 times
Been thanked: 8 times

Re: Chart loading. Data backfilling.

Postby DRCM » 03 May 2011

Hello,

MC support team analyzed all my workspaces and sent an email today saying,
" 9 charts on each workspace with 13 years of data on every chart = 300 000 fifteen minute bars. This is a lot of data. So 30 minutes to backfill all your charts from the database is a normal behavior."
Can someone tell me then, what point to backfill data from database if it takes same time to do it straight from IQFeed? Probably I don`t understand how this software works. sorry
I use AB as well and everytime I open software all old data already there ( 2.7 GB, 1min. database, about 200 symbols ), I don`t need to backfill anything except missing data for the last days. My current MultiCharts database only 100 MB and half an hour of wastage every time i close and open software. ?????????????????????
Thanks

User avatar
TJ
Posts: 6518
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Re: Chart loading. Data backfilling.

Postby TJ » 03 May 2011

DRCM wrote:Hello,

MC support team analyzed all my workspaces and sent an email today saying,
" 9 charts on each workspace with 13 years of data on every chart = 300 000 fifteen minute bars. This is a lot of data. So 30 minutes to backfill all your charts from the database is a normal behavior."
Can someone tell me then, what point to backfill data from database if it takes same time to do it straight from IQFeed? Probably I don`t understand how this software works. sorry
I use AB as well and everytime I open software all old data already there ( 2.7 GB, 1min. database, about 200 symbols ), I don`t need to backfill anything except missing data for the last days. My current MultiCharts database only 100 MB and half an hour of wastage every time i close and open software. ?????????????????????
Thanks


From my perch... I can see 2 different issues:

1. It is a lot of data. 9 charts of 18 symbols of 13 yrs of data is more than 300k bars.

2. Regardless of number of bars on the chart, did tech support verified that you are populating the chart from your computer's database, and not re-retrieving the data from a distance server again?


ps. if you already have the data you need, you canturn off the backfill.
File > Preferences.. > Data Server Mode
uncheck Download missing historical data

User avatar
TJ
Posts: 6518
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Re: Chart loading. Data backfilling.

Postby TJ » 03 May 2011

ps. you can open QuoteManager, check the event log to see if MultiCharts is waiting for data from the server.

DRCM
Posts: 164
Joined: 17 Apr 2011
Location: England
Has thanked: 65 times
Been thanked: 8 times

Re: Chart loading. Data backfilling.

Postby DRCM » 03 May 2011

Hi TJ,

I agree it`s a lot of data ( if 100 MB is a lot), but only for initial backfill from IQFeeD. With AB it takes same time to get it from IQFeed ( ~30mins ), but then when data is on your computer AB opens it instantly.You can close software and open again and again it`s on your charts. No any backfills from local database or etc. It can see data straightaway. How MC software can take such a long time(~ 30mins) to get those 100MB from local database? If honestly, I`didnt expect to find such kind of problem, first time i thought i`ll have to go through settings to configure something out, then started thinking that MC can`t find database and then backfill starts itself from IQFeeD, but seems it is "normal behavior".
However, I think MC is a good soft for auto, manual, DOM trading with many different brokers implemented and etc. but only if you work with couple of symbols not a big database or at least if you have plenty of time to wait untill your charts loads.
In my case I`ll have to use two platforms MC for realtime trading only and do all tech. analysis things in separate software. a bit disappointed
Thanks
p.s. TJ, I want to work online and get real time data , if i switch off backfill the latest data will be held.
p.s I just checked QuoteManager event logs and no backfill from IQFeeD, only says connected to IQ RT data. So it means MC backfilling data from locally.

User avatar
Stan Bokov
Posts: 963
Joined: 18 Dec 2009
Has thanked: 367 times
Been thanked: 300 times

Re: Chart loading. Data backfilling.

Postby Stan Bokov » 04 May 2011

Dear DRCM,

I can see how it can be frustrating from the point of view of a trader. Let me take a look at it, maybe something can be optimized in the way we pull up charts from the local database. I will let you know about my findings.

janus
Posts: 746
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 82 times

Re: Chart loading. Data backfilling.

Postby janus » 06 May 2011

I agree DRCM. It shouldn't be taking that long on subsequent reloads if you reload MC regularly. I can understand it if you don't restart MC for several days because on the next reload MC will load all the historical data from the historical feed. I think this is happens if a data feed provides historical minute data, then real-time minute data is not stored in the database. If you then exit and start MC again soon afterwards, it shouldn't take anywhere near the same amount of time since it should be reading all the new historical data that was stored in the local database. In other words, it should be behaving much like AB.

User avatar
TJ
Posts: 6518
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Re: Chart loading. Data backfilling.

Postby TJ » 06 May 2011

Please check your session time.

If you have set it to 24 hrs, and the symbol does not trade over night,
MultiCharts will still try to look (wait) for data that's not there.

User avatar
swz168
Posts: 120
Joined: 30 Sep 2010
Location: Germany
Has thanked: 47 times
Been thanked: 32 times

Re: Chart loading. Data backfilling.

Postby swz168 » 07 May 2011

TJ wrote:Please check your session time.

If you have set it to 24 hrs, and the symbol does not trade over night,
MultiCharts will still try to look (wait) for data that's not there.


I've always wondered why Multicharts took several seconds to display a chart. Now I know the reason. I've changed my exchange times and MC displays the chart with practically no delay.

Thanks TJ!

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 51 times

Re: Chart loading. Data backfilling.

Postby Tresor » 07 May 2011

Speaking of QM, I noticed, although not reported in PM, some inconsistencies, similar (but not the same) to issues raised by Janus:
https://www.multicharts.com/pm/viewissue ... _no=MC-185
https://www.multicharts.com/pm/viewissue ... _no=MC-191

Will do so when I find time.

I think that both (i) Quote Manager and (ii) the way it interacts with MultiCharts will need to undergo changes sooner or later.

janus
Posts: 746
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 82 times

Re: Chart loading. Data backfilling.

Postby janus » 07 May 2011

Tresor, yes the issues I raised in PM are related but not the same. One of the issues I raised was to inconsistency between what the editor in QM shows and what MC is displaying when watching a real-time chart. The problem I see is mostly not of MC's making but of IB's making. If the real-time and historical data were exactly the same (and they should be) then MC could save all the real-time data that's stored in the cache into the database upon exit, but it doesn't, perhaps for that reason. I would recommend that MC provide an option. The option would have a number of choices:

1. The real-time data is stored in the cache for performance reasons. When you exit MC, no data is stored in the database, and the data in the cache is cleared automatically. Currently MC does this for 1-minute data for IB. So, when the user starts up MC next time, it will backfill by requesting historical data, which is then stored in the database. Then it starts collecting and displaying real-time data using the cache again. This choice is similar to the current way MC operates, and perhaps can be the same. I just don't know enough of how the different types of data (ie, tick, 1-minute, daily) behave for historical versus real-time data collection from the various data providers to make a complete assessment as to what is the best approach. Perhaps the current way is OK.

2: As well as collecting real-time data, also collect historical-data using the historical data feed and store this with the real-time data in the cache. Alternatively, if it has minimal impact on the performance, which I think is the case, store the historical data immediately into the database rather than storing it temporarily in the cache. When you exit MC, only the historical data, if it's kept in the cache, is stored into the database and the real-time data is cleared. If it's stored directly into the database upon collection, then there's no delay on exit, which would be a significant improvement on exit speed. This way MC doesn't take as long to start again because it no longer is required to backfill with historical data that it has already collected and saved. It only needs to backfill up to the time it was exited last time. This can be a significant improvement in speed when users leave their MC continuously running for days and have lots of charts open. This option would not be used for data providers who only have one data feed, or in the case of IB got their act together and fixed their major problem as outlined elsewhere by others and myself.

3: Only ever collect historical data where it's available even during real-time mode. Not sure if this is efficient but it's certainly possible. On exit, MC stores this data from the cache into the database. In the case of IB, the data displayed will be the same at all times, unlike the situation now where a chart can look different after re-starting MC.

4: Only ever collect real-time data and store all such data in the cache into the database when exiting MC. When MC is re-started, it only backfills up to the time of the last exit time. This would be the most efficient and fasted method. The trouble with this though is the data in the database ends up being a mixture of real-time and historical data, and in the case of IB would be different. Not sure about other data providers. At least with this option the charts will look the same more often so back-testing results will change less often after re-starting MC. However, the results won't always be the same as those using choice #3, at least with IB.

I will consider placing a feature request in PM if I get enough interest here.

User avatar
TJ
Posts: 6518
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Re: Chart loading. Data backfilling.

Postby TJ » 07 May 2011

swz168 wrote:
TJ wrote:Please check your session time.

If you have set it to 24 hrs, and the symbol does not trade over night,
MultiCharts will still try to look (wait) for data that's not there.


I've always wondered why Multicharts took several seconds to display a chart. Now I know the reason. I've changed my exchange times and MC displays the chart with practically no delay.

Thanks TJ!


You are welcome !

Good trading.

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 485 times

Re: Chart loading. Data backfilling.

Postby Dave Masalov » 16 May 2011

Dear DRCM,

As Stan already wrote, we understand that current behavior can be frustrating from the point of view of a trader. At the moment MC sends data requests on per session basis and plots n-minute charts out of minute data every time. We will see what can be done to improve this behavior in the future.

As a workaround, please try to disable MultiCharts log files in the Windows registry: Start --> Run --> regedit:

HKEY_CURRENT_USER\Software\TS Support\BugSlayer
Trace\Enable="False"


Return to “MultiCharts”