DDE Collected data lost after program not responding error

Questions about MultiCharts and user contributed studies.
jl12
Posts: 197
Joined: 29 Jun 2007
Has thanked: 8 times
Been thanked: 8 times
Contact:

DDE Collected data lost after program not responding error

Postby jl12 » 12 Mar 2008

After copying and pasting a window MC locked up, all the dde data that had been collected during the day up to this time was lost.
This loss of data on following a problem has happened before.
Is data only saved to the database after exiting normally.

jl12
Posts: 197
Joined: 29 Jun 2007
Has thanked: 8 times
Been thanked: 8 times
Contact:

Postby jl12 » 14 Mar 2008

This problem has occure again all todays data collected via dde and quote manager has been lost

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 17 Mar 2008

If tsserver.exe has been closed using Task Manager the collected quotes will not be saved since we flush data on exit and the operation hasn't been completed properly. The flush process may take minutes and some customers don't wait and kill the program. So we are going to add flushing in progress dialog.
If you can reproduce freezing of MC let us know how.

jl12
Posts: 197
Joined: 29 Jun 2007
Has thanked: 8 times
Been thanked: 8 times
Contact:

Postby jl12 » 12 Apr 2008

It would be good if data could be saved at regular intervals

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 18 Apr 2008

Hi jl12,

The problem with saving data at regular intervals is that it's a waste of CPU. We have chosen to save one large chunk of data on exit.

However, saving data at regular intervals gives additional security and prevents data losses.

What we might do in future MC versions is create two options: save data at regular intervals or on exit.

T6
Posts: 6
Joined: 18 Apr 2008
Location: Sweden

Saving DDE

Postby T6 » 18 Apr 2008

It would be nice if we could save data from DDE to dbase during the day. Lost data destroy all our charts and indicator. So, i think it is more important to save them reg. during the day than saving CPU.
I talked to Sergee about it earlier. Complete the helpfile how to exit without losing data. And set up a message during save so we don´t shut down our computers and losing data.
It has happened to me many times before. So complete the helpfile, a message during save and if possible, save during the day.

Without complete data we don´t need any MC and we are blind. :)

jl12
Posts: 197
Joined: 29 Jun 2007
Has thanked: 8 times
Been thanked: 8 times
Contact:

Postby jl12 » 18 Apr 2008

What concerns me most is if my computer is frozen and i have to kill the pc there is no way to save the data.

dupl
Posts: 158
Joined: 04 Jul 2007

Re: Saving DDE/IB or eSignal EOD

Postby dupl » 18 Apr 2008

And set up a message during save so we don´t shut down our computers and losing data.

Yes! Excuse, this is not understandable, why you do not inform the users that killing tsServer.exe means, losing data!

Collecting data from IB or eSignal EOD also, is a lot of work an time! And users wonder, why they have to download it on next day partly a second time... :cry:

Minimum is a progressbar for saving the data, if exiting MC.

Thanks

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 23 Apr 2008

Hi,

We believe that informing the users that the data is going to be lost if a process is killed or if the program hasn't been terminated normally is reduntant because it should be self-understandable.

To prevent losses of large amounts of data, we will add the 'data flush' option.

dupl
Posts: 158
Joined: 04 Jul 2007

Postby dupl » 23 Apr 2008

We believe that informing the users that the data is going to be lost if a process is killed or if the program hasn't been terminated normally is reduntant because it should be self-understandable.


This is a fallacy! Because a normal user does not watch the taskmager and the particular processes...

If a normal customer exit a programm and the prgramm is closed, than he believes that, it is closed! And this happend to me in all my software programms over 10 years.

But MC is a exception from this rule. :idea: And that is why it is dangerous to you if you dont inform the customers about it...

Shuting down the computer is killing tsServer.exe, but the customer thinks it is closed with MC! :!:

T6
Posts: 6
Joined: 18 Apr 2008
Location: Sweden

Postby T6 » 24 Apr 2008

Hi,

We believe that informing the users that the data is going to be lost if a process is killed or if the program hasn't been terminated normally is reduntant because it should be self-understandable.

To prevent losses of large amounts of data, we will add the 'data flush' option.


If you look in the helpfile there is no info how to close down QM or in what order to do that to save the data.
I have lost data many times because of that. Even tried to get info from Sergee.
No one understod the problem.

What is "data flush"? Is it dumping data to dbase or what. I thougt you was flushing on the toilet. ;-)) That means skipping data to me!?

Why not a progressmeter?

Nick
Posts: 490
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Postby Nick » 28 Apr 2008

Flushing data should be an 'asynchronous' process. I guess MC has some sort of internal scheduling when it has nothing much to do it could flush the cache.

fs

Postby fs » 28 Apr 2008

We believe that informing the users that the data is going to be lost if a process is killed or if the program hasn't been terminated normally is reduntant because it should be self-understandable.


This is a fallacy! Because a normal user does not watch the taskmager and the particular processes...

If a normal customer exit a programm and the prgramm is closed, than he believes that, it is closed! And this happend to me in all my software programms over 10 years.

But MC is a exception from this rule. :idea: And that is why it is dangerous to you if you dont inform the customers about it...

Shuting down the computer is killing tsServer.exe, but the customer thinks it is closed with MC! :!:


When you kill a process Windows always give you a warning that killing a process can cause loss data.

Frankly if you are complaining about losing data because you a kill a process, which is clearly not the correct way of terminating a process and Windows is giving you a warning when you do, then you are just looking for something to complain about.

There are several database/storage related programs which flush data on exit asynchronously even after it appears the program exited. Unless you have looked at Task Manager over the last 10 years every single time you exit a program, you cannot say that all your programs worked like that over the last 10 years. If you did, you would have known that MC is not unique in this regard.

Do you also want car manufacturers to give you a warning that you might get hurt if you crash your car into a tree to stop instead of stopping gracefully?

I'd rather have MC use their resources to solve some other real bugs instead of spending resources on warning people not to kill processes.

- Fanus

dupl
Posts: 158
Joined: 04 Jul 2007

Postby dupl » 30 Apr 2008

When you kill a process Windows always give you a warning that killing a process can cause loss data.


I sad also:
Shuting down the computer is killing tsServer.exe, but the customer thinks it is closed with MC! :!:


Do you look in the Windows Taskmanager if you had shut down a Email programm etc., if all processes of this Email programm are down? :shock: And than you are shure that you can close your computer without losing emails? No you dont! You close simply your Email programm and than simply your computer thats it!
But you can't do this with MC, because tsServer.exe etc are still running and saving data (e.g. 5-15 minutes!) and the user dont know this. No normal user scrabbles the Windows Taskmanager after exiting a program... if he wants to close the computer!

This is what I mean, this is unthinkable.

processes of MC:
  1. tsServer.exe
  2. MultiCharts.exe
  3. QuoteManager.exe
  4. StudyServer.exe
  5. MCActiveX.exe
  6. PLEditor.exe
  7. TradingServer.exe

http://forum.tssupport.com/viewtopic.ph ... =exe#19013

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 07 May 2008

Hi guys,

As I mentioned before, in future versions of MC, a progress bar will be shown on exiting the program. It will tell you if MultiCharts is in the process of saving data and how much data is left to save. Also, we will add an option that will allow saving the received data at regular intervals while working with the program.

It will solve the issues discussed in this thread.

dupl
Posts: 158
Joined: 04 Jul 2007

Postby dupl » 09 Jun 2008

There ist a second problem!

if I shut down MC tsServer.exe will be NOT closed.
Only if I shut down also QM! :idea:

User avatar
TJ
Posts: 6585
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 971 times
Been thanked: 1907 times

Postby TJ » 09 Jun 2008

Hi guys,

As I mentioned before, in future versions of MC, a progress bar will be shown on exiting the program. It will tell you if MultiCharts is in the process of saving data and how much data is left to save. Also, we will add an option that will allow saving the received data at regular intervals while working with the program.

It will solve the issues discussed in this thread.


This is great. Thanks !

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 10 Jun 2008

There ist a second problem!

if I shut down MC tsServer.exe will be NOT closed.
Only if I shut down also QM! :idea:


This is normal behavior. And this is exactly what we've been talking about in this thread: if you shut down MultiCharts some of the processes might still be running - for example tsServer.exe flushing all the collected data into the database.

Regards.

dupl
Posts: 158
Joined: 04 Jul 2007

Postby dupl » 15 Jan 2009

Hi guys,

As I mentioned before, in future versions of MC, a progress bar will be shown on exiting the program. It will tell you if MultiCharts is in the process of saving data and how much data is left to save. Also, we will add an option that will allow saving the received data at regular intervals while working with the program.

It will solve the issues discussed in this thread.


We have still this problem.
My MC crashed today and now its loading all data again from IB. I have 13 workspaces open and not one chart is shown now!
Its really slow, because IB historical data server says very often "Historical data request pacing violation"!

Its really important to save the data and not to lose them all by crash or something else!

Please!

brodnicki steven
Posts: 407
Joined: 01 Jan 2008
Been thanked: 3 times

Postby brodnicki steven » 15 Jan 2009

Hi guys,

As I mentioned before, in future versions of MC, a progress bar will be shown on exiting the program. It will tell you if MultiCharts is in the process of saving data and how much data is left to save. Also, we will add an option that will allow saving the received data at regular intervals while working with the program.

It will solve the issues discussed in this thread.


**Marina- please make the interval selectable for data flushing in the future version.(every 5 minutes etc)

I noticed that it takes-3-4 minutes to save the data, before I shut down and that's on a pretty fast quad core machine.

** when will v5.3 be released ?

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 16 Jan 2009

Hu guys.

Dupl, what most probably happened was that you had MultiCharts running and collecting data for a long time. MultiCharts compresses and writes data into the database when you close the program. In your case, since the program did not terminate its work properly, the collected data could not be compressed and written into the database.

Steven, at some point we actually tried implementing the idea that you’re talking about, i.e. flushing data every 5 minutes. The problem with this solution is that compressing and writing data into the database is a very resource-intensive operation. Database need to be unpacked, the data is then compressed and written into the database. If this were to take place every 5 or every 10 minutes, you would see MultiCharts all of a sudden using a lot of CPU every 5 or 10 minutes. The program’s performance will be seriously compromised.

What we are planning to do instead, is save all the received data into temporary files (it is actually already done in the current MC versions). When MultiCharts is shut down, the data will be compressed and flushed into the database. If the program cannot be terminated properly, the data will still be there in the temporary folder and will simply be flushed into the database on next program’s start. This will ensure that the data is not only saved but saved correctly. In the past, some users, for example Bill Schamp and his students, had issues with their charts because fragments of data were corrupted when something went wrong with program’s termination and data being saved.

User avatar
TJ
Posts: 6585
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 971 times
Been thanked: 1907 times

Postby TJ » 16 Jan 2009

can you also make a manual save button?

I trade the ES as well as the HSI.

right now I have to exit and restart MC after the ES market, so that the data can be
saved. A manual button would save a lot of restarting.

dupl
Posts: 158
Joined: 04 Jul 2007

can you also make a manual save button?

Postby dupl » 22 Jan 2009

can you also make a manual save button?
I trade the ES as well as the HSI.
right now I have to exit and restart MC after the ES market, so that the data can be
saved. A manual button would save a lot of restarting.


Yes really! Saving data is 50% the work of MC in my case.

Today TWS crashed MC lost connection, didnt reconnect! I tried to open new chart and so on...
OK close MC and QM.
Start them again...
But it didnt reconnect to TWS
Close again MC and QM
Than I saw tow tsServer.exe in taskmanager!!! Ah, so I likked this second tsServer.exe by hand.
Start them again... MC and QM.
and load again my lost data! :cry:

can you also make a manual save button?

brodnicki steven
Posts: 407
Joined: 01 Jan 2008
Been thanked: 3 times

Postby brodnicki steven » 22 Jan 2009

My process every night when TWS shuts down, is to save workspaces and shut down MC. Then I close the datamanager and watch either the hard drive led or tsServer.exe in taskmanager, when usage goes to zero, it's safe to re-boot the PC. (usually takes3-5 minutes)
TS2ki would save RT data at very short intervals and there wasn't an issue with CPU usage, I don't know how they did it though but it was very efficient. I'd love to see MC use a similar method.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 26 Jan 2009

Hi steven,

From what we know, TS2000i didn't use to save data at regular intervals either. The data was saved in a demporary folder and then flushed into the database when the program was closed.

The most resource intensive part of saving data to a permanent database is unpacking the already existing data and then packing it all again after a new data segment has been added.

In MC Gold, the process of saving data will be both efficient and safe: if data cannot be properly stored when the program is closed, it will be saved when the program is started again.

T6
Posts: 6
Joined: 18 Apr 2008
Location: Sweden

Sawing DDE-data during the day

Postby T6 » 06 Feb 2009

Hi TSSUPPORT!

Still waiting for the function you promiised us. Long ago :( The first time i asked SERGEE promised that it should take about a mont. As you can se it is a bit longer than that.
I really need that function. My comp. is going down at least once a week. hen i am loosing data.

An other thing is when you send IMAGE via E-mail. The name is the same all the time. You can't even change is manually in the mail. Am i the only one who was seen that problem. All the time since i began witw MC. 2007.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 13 Feb 2009

Hi T6,

If you are talking about a new mechanism of saving data that would prevent losing quotes collected while working with MultiCharts in case of crashes, unexpected terminations etc, it will be available in the upcoming MC 5.0 beta 3. At the moment, there is a temporary cache file that saves all the received data. On closing MultiCharts, the permanent database is unpacked, the new data is placed in there and the database is compressed and packed again. In the upcoming MC 5.0 beta 3, if something goes wrong during this process and the data is not saved into the permanent database, it will not be lost. The quotes will still be there in the temporary cache file. These quotes will then be flushed into the permanent database on the next MC launch. With this new mechanism in place, you might only lose a few seconds worth of data.

As for sending images as an e-mail, you can open an attached image right from your email, save it with a new name and attach the renamed file to your e-mail message.

Regards.

brodnicki steven
Posts: 407
Joined: 01 Jan 2008
Been thanked: 3 times

Postby brodnicki steven » 13 Feb 2009

I'm not sure how TS2ki stored data but it was never a cpu issue. The Global server also had the option of doing an immediate "Nightly Maintenance" whenever you wanted to, I believe that stored any data still in memory, immediately.
It sounds like you have a good solution in the gold release.

T6
Posts: 6
Joined: 18 Apr 2008
Location: Sweden

Postby T6 » 16 Feb 2009

Hi!

Thank´s in advance. Just perfect. It will save me a lot of trouble in the future.

Regards

T6

dupl
Posts: 158
Joined: 04 Jul 2007

Postby dupl » 06 Mar 2009

Hi T6,

If you are talking about a new mechanism of saving data that would prevent losing quotes collected while working with MultiCharts in case of crashes, unexpected terminations etc, it will be available in the upcoming MC 5.0 beta 3. At the moment, there is a temporary cache file that saves all the received data. On closing MultiCharts, the permanent database is unpacked, the new data is placed in there and the database is compressed and packed again. In the upcoming MC 5.0 beta 3, if something goes wrong during this process and the data is not saved into the permanent database, it will not be lost. The quotes will still be there in the temporary cache file. These quotes will then be flushed into the permanent database on the next MC launch. With this new mechanism in place, you might only lose a few seconds worth of data.


Has somebody experienced the new mechanism of saving data?

T6
Posts: 6
Joined: 18 Apr 2008
Location: Sweden

Postby T6 » 06 Mar 2009

Hi dupl. Thank´s. I just downloaded the new beta. Is perhaps it is fixed in that version.

Thank´s for your information

Regards T6

dupl
Posts: 158
Joined: 04 Jul 2007

Postby dupl » 06 Mar 2009

Hi T6,
nice not to be alone! :D
How is the best way to test it?
Today e.g. I opened a lot of data, my MultiCharts.exe task had more than 1,2GB "memory usage history".
But if I closed a workspace than around 500MB less memory were used, but when opening again the workspace it loads a lot of data again from IB and I get all the time this pacing violation from IB in the event log.

Could it be a way to test the new mechanism of saving data to close a workspace an reopened it without closing MC?


Return to “MultiCharts”