Hello,
I created 6 workspaces with 9 charts in each (18 symbols, 15min.,1 hour and 1 day each symbol). Backfilled all data from IQFeed first time, it gives approx. 2 yr of forex and 4 yr of stocks data. Saved everything. If i want to open all my charts now it takes about half an hour to backfill again. Is it normal ?
Does MultiCharts backfill data from IQFeed or from local database when you use it after initial backfill? Thanks
Chart loading. Data backfilling.
- TJ
- Posts: 7743
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Chart loading. Data backfilling.
one thing to check...Hello,
I created 6 workspaces with 9 charts in each (18 symbols, 15min.,1 hour and 1 day each symbol). Backfilled all data from IQFeed first time, it gives approx. 2 yr of forex and 4 yr of stocks data. Saved everything. If i want to open all my charts now it takes about half an hour to backfill again. Is it normal ?
Does MultiCharts backfill data from IQFeed or from local database when you use it after initial backfill? Thanks
before you shut down the computer,
check your TaskManager
In the Processes tab
under the Description column,
make sure all the MultiCharts processes are exited.
If the user shuts down the computer before the processes are finished,
MultiCharts would not have a chance to save the data to the database.
Re: Chart loading. Data backfilling.
Hello,
Yes. When I shut MultiCharts down, no MC processes are running in TaskManager. But when I open MC it backfilling data half an hour again. I checked C:/ProgramData/TS Support/MultiCharts/Database and TSSTORAGE.GDB file shows 101,144 KB . So, it`s about 100MB of data stored in local database and i guess it should be something like that.
Does it mean that MC loading data from local database so slowly? :O
Yes. When I shut MultiCharts down, no MC processes are running in TaskManager. But when I open MC it backfilling data half an hour again. I checked C:/ProgramData/TS Support/MultiCharts/Database and TSSTORAGE.GDB file shows 101,144 KB . So, it`s about 100MB of data stored in local database and i guess it should be something like that.
Does it mean that MC loading data from local database so slowly? :O
- TJ
- Posts: 7743
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Chart loading. Data backfilling.
Which version of MultiCharts you are using?
I would go to online chat and let technical support check out your computer.
I remember someone asked similar question recently, but I cannot find the thread at the moment.
I would go to online chat and let technical support check out your computer.
I remember someone asked similar question recently, but I cannot find the thread at the moment.
- Stan Bokov
- Posts: 963
- Joined: 18 Dec 2009
- Has thanked: 367 times
- Been thanked: 302 times
Re: Chart loading. Data backfilling.
Hello,
MC support team analyzed all my workspaces and sent an email today saying,
" 9 charts on each workspace with 13 years of data on every chart = 300 000 fifteen minute bars. This is a lot of data. So 30 minutes to backfill all your charts from the database is a normal behavior."
Can someone tell me then, what point to backfill data from database if it takes same time to do it straight from IQFeed? Probably I don`t understand how this software works. sorry
I use AB as well and everytime I open software all old data already there ( 2.7 GB, 1min. database, about 200 symbols ), I don`t need to backfill anything except missing data for the last days. My current MultiCharts database only 100 MB and half an hour of wastage every time i close and open software. ?????????????????????
Thanks
MC support team analyzed all my workspaces and sent an email today saying,
" 9 charts on each workspace with 13 years of data on every chart = 300 000 fifteen minute bars. This is a lot of data. So 30 minutes to backfill all your charts from the database is a normal behavior."
Can someone tell me then, what point to backfill data from database if it takes same time to do it straight from IQFeed? Probably I don`t understand how this software works. sorry
I use AB as well and everytime I open software all old data already there ( 2.7 GB, 1min. database, about 200 symbols ), I don`t need to backfill anything except missing data for the last days. My current MultiCharts database only 100 MB and half an hour of wastage every time i close and open software. ?????????????????????
Thanks
- TJ
- Posts: 7743
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Chart loading. Data backfilling.
From my perch... I can see 2 different issues:Hello,
MC support team analyzed all my workspaces and sent an email today saying,
" 9 charts on each workspace with 13 years of data on every chart = 300 000 fifteen minute bars. This is a lot of data. So 30 minutes to backfill all your charts from the database is a normal behavior."
Can someone tell me then, what point to backfill data from database if it takes same time to do it straight from IQFeed? Probably I don`t understand how this software works. sorry
I use AB as well and everytime I open software all old data already there ( 2.7 GB, 1min. database, about 200 symbols ), I don`t need to backfill anything except missing data for the last days. My current MultiCharts database only 100 MB and half an hour of wastage every time i close and open software. ?????????????????????
Thanks
1. It is a lot of data. 9 charts of 18 symbols of 13 yrs of data is more than 300k bars.
2. Regardless of number of bars on the chart, did tech support verified that you are populating the chart from your computer's database, and not re-retrieving the data from a distance server again?
ps. if you already have the data you need, you canturn off the backfill.
File > Preferences.. > Data Server Mode
uncheck Download missing historical data
- TJ
- Posts: 7743
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Chart loading. Data backfilling.
ps. you can open QuoteManager, check the event log to see if MultiCharts is waiting for data from the server.
Re: Chart loading. Data backfilling.
Hi TJ,
I agree it`s a lot of data ( if 100 MB is a lot), but only for initial backfill from IQFeeD. With AB it takes same time to get it from IQFeed ( ~30mins ), but then when data is on your computer AB opens it instantly.You can close software and open again and again it`s on your charts. No any backfills from local database or etc. It can see data straightaway. How MC software can take such a long time(~ 30mins) to get those 100MB from local database? If honestly, I`didnt expect to find such kind of problem, first time i thought i`ll have to go through settings to configure something out, then started thinking that MC can`t find database and then backfill starts itself from IQFeeD, but seems it is "normal behavior".
However, I think MC is a good soft for auto, manual, DOM trading with many different brokers implemented and etc. but only if you work with couple of symbols not a big database or at least if you have plenty of time to wait untill your charts loads.
In my case I`ll have to use two platforms MC for realtime trading only and do all tech. analysis things in separate software. a bit disappointed
Thanks
p.s. TJ, I want to work online and get real time data , if i switch off backfill the latest data will be held.
p.s I just checked QuoteManager event logs and no backfill from IQFeeD, only says connected to IQ RT data. So it means MC backfilling data from locally.
I agree it`s a lot of data ( if 100 MB is a lot), but only for initial backfill from IQFeeD. With AB it takes same time to get it from IQFeed ( ~30mins ), but then when data is on your computer AB opens it instantly.You can close software and open again and again it`s on your charts. No any backfills from local database or etc. It can see data straightaway. How MC software can take such a long time(~ 30mins) to get those 100MB from local database? If honestly, I`didnt expect to find such kind of problem, first time i thought i`ll have to go through settings to configure something out, then started thinking that MC can`t find database and then backfill starts itself from IQFeeD, but seems it is "normal behavior".
However, I think MC is a good soft for auto, manual, DOM trading with many different brokers implemented and etc. but only if you work with couple of symbols not a big database or at least if you have plenty of time to wait untill your charts loads.
In my case I`ll have to use two platforms MC for realtime trading only and do all tech. analysis things in separate software. a bit disappointed
Thanks
p.s. TJ, I want to work online and get real time data , if i switch off backfill the latest data will be held.
p.s I just checked QuoteManager event logs and no backfill from IQFeeD, only says connected to IQ RT data. So it means MC backfilling data from locally.
- Stan Bokov
- Posts: 963
- Joined: 18 Dec 2009
- Has thanked: 367 times
- Been thanked: 302 times
Re: Chart loading. Data backfilling.
Dear DRCM,
I can see how it can be frustrating from the point of view of a trader. Let me take a look at it, maybe something can be optimized in the way we pull up charts from the local database. I will let you know about my findings.
I can see how it can be frustrating from the point of view of a trader. Let me take a look at it, maybe something can be optimized in the way we pull up charts from the local database. I will let you know about my findings.
Re: Chart loading. Data backfilling.
I agree DRCM. It shouldn't be taking that long on subsequent reloads if you reload MC regularly. I can understand it if you don't restart MC for several days because on the next reload MC will load all the historical data from the historical feed. I think this is happens if a data feed provides historical minute data, then real-time minute data is not stored in the database. If you then exit and start MC again soon afterwards, it shouldn't take anywhere near the same amount of time since it should be reading all the new historical data that was stored in the local database. In other words, it should be behaving much like AB.
- TJ
- Posts: 7743
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Chart loading. Data backfilling.
Please check your session time.
If you have set it to 24 hrs, and the symbol does not trade over night,
MultiCharts will still try to look (wait) for data that's not there.
If you have set it to 24 hrs, and the symbol does not trade over night,
MultiCharts will still try to look (wait) for data that's not there.
- swz168
- Posts: 120
- Joined: 30 Sep 2010
- Location: Germany
- Has thanked: 47 times
- Been thanked: 32 times
Re: Chart loading. Data backfilling.
I've always wondered why Multicharts took several seconds to display a chart. Now I know the reason. I've changed my exchange times and MC displays the chart with practically no delay.Please check your session time.
If you have set it to 24 hrs, and the symbol does not trade over night,
MultiCharts will still try to look (wait) for data that's not there.
Thanks TJ!
Re: Chart loading. Data backfilling.
Speaking of QM, I noticed, although not reported in PM, some inconsistencies, similar (but not the same) to issues raised by Janus:
https://www.multicharts.com/pm/viewissu ... _no=MC-185
https://www.multicharts.com/pm/viewissu ... _no=MC-191
Will do so when I find time.
I think that both (i) Quote Manager and (ii) the way it interacts with MultiCharts will need to undergo changes sooner or later.
https://www.multicharts.com/pm/viewissu ... _no=MC-185
https://www.multicharts.com/pm/viewissu ... _no=MC-191
Will do so when I find time.
I think that both (i) Quote Manager and (ii) the way it interacts with MultiCharts will need to undergo changes sooner or later.
Re: Chart loading. Data backfilling.
Tresor, yes the issues I raised in PM are related but not the same. One of the issues I raised was to inconsistency between what the editor in QM shows and what MC is displaying when watching a real-time chart. The problem I see is mostly not of MC's making but of IB's making. If the real-time and historical data were exactly the same (and they should be) then MC could save all the real-time data that's stored in the cache into the database upon exit, but it doesn't, perhaps for that reason. I would recommend that MC provide an option. The option would have a number of choices:
1. The real-time data is stored in the cache for performance reasons. When you exit MC, no data is stored in the database, and the data in the cache is cleared automatically. Currently MC does this for 1-minute data for IB. So, when the user starts up MC next time, it will backfill by requesting historical data, which is then stored in the database. Then it starts collecting and displaying real-time data using the cache again. This choice is similar to the current way MC operates, and perhaps can be the same. I just don't know enough of how the different types of data (ie, tick, 1-minute, daily) behave for historical versus real-time data collection from the various data providers to make a complete assessment as to what is the best approach. Perhaps the current way is OK.
2: As well as collecting real-time data, also collect historical-data using the historical data feed and store this with the real-time data in the cache. Alternatively, if it has minimal impact on the performance, which I think is the case, store the historical data immediately into the database rather than storing it temporarily in the cache. When you exit MC, only the historical data, if it's kept in the cache, is stored into the database and the real-time data is cleared. If it's stored directly into the database upon collection, then there's no delay on exit, which would be a significant improvement on exit speed. This way MC doesn't take as long to start again because it no longer is required to backfill with historical data that it has already collected and saved. It only needs to backfill up to the time it was exited last time. This can be a significant improvement in speed when users leave their MC continuously running for days and have lots of charts open. This option would not be used for data providers who only have one data feed, or in the case of IB got their act together and fixed their major problem as outlined elsewhere by others and myself.
3: Only ever collect historical data where it's available even during real-time mode. Not sure if this is efficient but it's certainly possible. On exit, MC stores this data from the cache into the database. In the case of IB, the data displayed will be the same at all times, unlike the situation now where a chart can look different after re-starting MC.
4: Only ever collect real-time data and store all such data in the cache into the database when exiting MC. When MC is re-started, it only backfills up to the time of the last exit time. This would be the most efficient and fasted method. The trouble with this though is the data in the database ends up being a mixture of real-time and historical data, and in the case of IB would be different. Not sure about other data providers. At least with this option the charts will look the same more often so back-testing results will change less often after re-starting MC. However, the results won't always be the same as those using choice #3, at least with IB.
I will consider placing a feature request in PM if I get enough interest here.
1. The real-time data is stored in the cache for performance reasons. When you exit MC, no data is stored in the database, and the data in the cache is cleared automatically. Currently MC does this for 1-minute data for IB. So, when the user starts up MC next time, it will backfill by requesting historical data, which is then stored in the database. Then it starts collecting and displaying real-time data using the cache again. This choice is similar to the current way MC operates, and perhaps can be the same. I just don't know enough of how the different types of data (ie, tick, 1-minute, daily) behave for historical versus real-time data collection from the various data providers to make a complete assessment as to what is the best approach. Perhaps the current way is OK.
2: As well as collecting real-time data, also collect historical-data using the historical data feed and store this with the real-time data in the cache. Alternatively, if it has minimal impact on the performance, which I think is the case, store the historical data immediately into the database rather than storing it temporarily in the cache. When you exit MC, only the historical data, if it's kept in the cache, is stored into the database and the real-time data is cleared. If it's stored directly into the database upon collection, then there's no delay on exit, which would be a significant improvement on exit speed. This way MC doesn't take as long to start again because it no longer is required to backfill with historical data that it has already collected and saved. It only needs to backfill up to the time it was exited last time. This can be a significant improvement in speed when users leave their MC continuously running for days and have lots of charts open. This option would not be used for data providers who only have one data feed, or in the case of IB got their act together and fixed their major problem as outlined elsewhere by others and myself.
3: Only ever collect historical data where it's available even during real-time mode. Not sure if this is efficient but it's certainly possible. On exit, MC stores this data from the cache into the database. In the case of IB, the data displayed will be the same at all times, unlike the situation now where a chart can look different after re-starting MC.
4: Only ever collect real-time data and store all such data in the cache into the database when exiting MC. When MC is re-started, it only backfills up to the time of the last exit time. This would be the most efficient and fasted method. The trouble with this though is the data in the database ends up being a mixture of real-time and historical data, and in the case of IB would be different. Not sure about other data providers. At least with this option the charts will look the same more often so back-testing results will change less often after re-starting MC. However, the results won't always be the same as those using choice #3, at least with IB.
I will consider placing a feature request in PM if I get enough interest here.
- TJ
- Posts: 7743
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Chart loading. Data backfilling.
You are welcome !I've always wondered why Multicharts took several seconds to display a chart. Now I know the reason. I've changed my exchange times and MC displays the chart with practically no delay.Please check your session time.
If you have set it to 24 hrs, and the symbol does not trade over night,
MultiCharts will still try to look (wait) for data that's not there.
Thanks TJ!
Good trading.
- Dave Masalov
- Posts: 1712
- Joined: 16 Apr 2010
- Has thanked: 51 times
- Been thanked: 489 times
Re: Chart loading. Data backfilling.
Dear DRCM,
As Stan already wrote, we understand that current behavior can be frustrating from the point of view of a trader. At the moment MC sends data requests on per session basis and plots n-minute charts out of minute data every time. We will see what can be done to improve this behavior in the future.
As a workaround, please try to disable MultiCharts log files in the Windows registry: Start --> Run --> regedit:
HKEY_CURRENT_USER\Software\TS Support\BugSlayer
Trace\Enable="False"
As Stan already wrote, we understand that current behavior can be frustrating from the point of view of a trader. At the moment MC sends data requests on per session basis and plots n-minute charts out of minute data every time. We will see what can be done to improve this behavior in the future.
As a workaround, please try to disable MultiCharts log files in the Windows registry: Start --> Run --> regedit:
HKEY_CURRENT_USER\Software\TS Support\BugSlayer
Trace\Enable="False"