Writing to a text file: Risk of congestion???  [SOLVED]

Questions about MultiCharts and user contributed studies.
arjfca
Posts: 1292
Joined: 23 Nov 2010
Has thanked: 725 times
Been thanked: 223 times

Writing to a text file: Risk of congestion???

Postby arjfca » 30 May 2014

Hello

Before starting to code a routine to write to a text file, I want to make sure that in real life it could be possible.

The goal is to add a function to an indicator that is use by all my charts that under a given condition will write info to a text file. Since I may have 50 to 70 charts open and running under same instrument with different time scale, many of them are subject to have to write to the same text file at the same time.

Is there a risk of congestion? I so, any suggestion on how to prevent it

A good weekend to all

Martin

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Writing to a text file: Risk of congestion???

Postby bowlesj3 » 30 May 2014

You could write it with the fileappend command as a CSV file (ending in a new line of course) to do a fast test. I suspect there would be no problem. I have written from two or three charts and had no problem that I am aware of. However it was not critical data. it was debugging data. The only problem if it is critical is there is no return code to see if the write failed. You could maintain a count in a GV and each write would grab the count, add to it, update the GV with the new value then do the write with the count on the file as one of the fields. You would edit this count from excel or something. At least you would know if it was correct. They may get out of sequence however but I think that would be very rare.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Writing to a text file: Risk of congestion???

Postby TJ » 30 May 2014

Hello
::
Is there a risk of congestion? I so, any suggestion on how to prevent it
::
Martin
It should be not be a problem, because today's computer uses multi-core CPU; housekeeping duties (eg writing to text file) are handled by one core, you still have the other cores to perform your calculations.

A SSD would definitely help.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Writing to a text file: Risk of congestion???

Postby TJ » 30 May 2014

ps. you can always write a test code, load it on 70 charts, then test run it on a papertrading account.

User avatar
Smoky
Posts: 507
Joined: 03 Dec 2010
Location: Thailand
Has thanked: 97 times
Been thanked: 115 times

Re: Writing to a text file: Risk of congestion???

Postby Smoky » 31 May 2014

No need to do the test I have done for you, 10 charts with 15 indicators Writing in the same CSV file tick by tick, you will be suprised at how fast you fill your SSD ;) and all the datas are clean (in my opinion this is one feature that works best)

arjfca
Posts: 1292
Joined: 23 Nov 2010
Has thanked: 725 times
Been thanked: 223 times

Re: Writing to a text file: Risk of congestion???

Postby arjfca » 31 May 2014

No need to do the test I have done for you, 10 charts with 15 indicators Writing in the same CSV file tick by tick, you will be suprised at how fast you fill your SSD ;) and all the datas are clean (in my opinion this is one feature that works best)
Thanks Smoky

The end goal is to have that file be written and read by both MC & Excel. MC to write and Excel Read and erase selected data in the file

OK for the SSD

Martin :)

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Writing to a text file: Risk of congestion???

Postby TJ » 31 May 2014

SSD has limited write/erase life,
you will destroy the disk in short order if you are to write and erase repeatedly.
You should use RAM disk if the data is not needed permanently.
You can always save permanent data to harddisk if desired.

User avatar
Smoky
Posts: 507
Joined: 03 Dec 2010
Location: Thailand
Has thanked: 97 times
Been thanked: 115 times

Re: Writing to a text file: Risk of congestion???  [SOLVED]

Postby Smoky » 31 May 2014


The end goal is to have that file be written and read by both MC & Excel. MC to write and Excel Read and erase selected data in the file

Martin :)
Sorry but MC write protect the CSV file, you can only read or make a copy to work,
but no writing/erasing with Excel ;)

when you open this CSV file (in real time MC is writing) open office ask you to make a copy or to open it in read only mode ;)

arjfca
Posts: 1292
Joined: 23 Nov 2010
Has thanked: 725 times
Been thanked: 223 times

Re: Writing to a text file: Risk of congestion???

Postby arjfca » 31 May 2014


The end goal is to have that file be written and read by both MC & Excel. MC to write and Excel Read and erase selected data in the file

Martin :)
Sorry but MC write protect the CSV file, you can only read or make a copy to work,
but no writing/erasing with Excel ;)

when you open this CSV file (in real time MC is writing) open office ask you to make a copy or to open it in read only mode ;)
OK, My plan was to use .txt file. I will postpone the project.

Martin

User avatar
JoshM
Posts: 2195
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1544 times
Been thanked: 1565 times
Contact:

Re: Writing to a text file: Risk of congestion???

Postby JoshM » 01 Jun 2014

SSD has limited write/erase life,
you will destroy the disk in short order if you are to write and erase repeatedly.
While true, this shouldn't have to be a real concern: with writing 10GB/day, a 80GB SSD has over 18 years till write exhaustion, and larger SSD drives take even longer (source).

User avatar
JoshM
Posts: 2195
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1544 times
Been thanked: 1565 times
Contact:

Re: Writing to a text file: Risk of congestion???

Postby JoshM » 03 Jun 2014

The goal is to add a function to an indicator that is use by all my charts that under a given condition will write info to a text file. Since I may have 50 to 70 charts open and running under same instrument with different time scale, many of them are subject to have to write to the same text file at the same time.
There is a risk of congestion because it seems that MultiCharts batches the write to file commands. Which is understandable because, as I understand it, `FileAppend` opens and closes the file every time. So the `FileAppend` from Symbol A will need to wait till Symbol B has completed its `FileAppend` process. If there are only a few symbols, this works good, but with plenty a "traffic jam" occurs.

At a certain point (on my pc with 8 charts and around 20 instances of the indicator below), the file is updated in (quick) steps but not fluently any more.

In some cases, there is also a line missing from the output, so writing to a file continuously does not seem very reliable, especially if you have a lot of charts (in my opinion).

Example indicator:

Code: Select all

{
Indicator: Example_WriteToFileAtTheSameTime

MultiCharts forum discussion topic: Writing to a text file: Risk of congestion?,
http://www.multicharts.com/discussion/viewtopic.php?f=1&t=46648

3-6-14

Note: Turn the 'Skip Identical Ticks' option off.
}

Variables:
fileName("C:\Temp\mcWriteTest.txt"),
stringToWrite("");

// No processing on non-realtime data
if (GetAppInfo(aiRealTimeCalc) <> 1) then
#return;

// Generate string
stringToWrite = Text(
Formattime("HH:mm:ss.", ComputerDateTime),
NumToStr(MilliSecondsFromDateTime(ComputerDateTime), 0),
";",
GetSymbolName,
";", NumToStr(Close, 5), ";",
NumToStr(Random(100), 0),
NewLine);

FileAppend(fileName, stringToWrite);
Generates an output like:

Code: Select all

18:08:13.699;EUR/USD;1.36220;3
18:08:13.711;EUR/USD;1.36220;61
18:08:13.711;EUR/USD;1.36220;88
18:08:13.726;EUR/USD;1.36220;26
18:08:13.738;EUR/USD;1.36220;76
18:08:14.487;EUR/USD;1.36221;39
18:08:14.488;EUR/USD;1.36221;57
18:08:14.503;EUR/USD;1.36221;2
18:08:14.519;EUR/USD;1.36221;2
18:08:14.534;EUR/USD;1.36221;23
18:08:14.548;EUR/USD;1.36221;84
18:08:15.234;USD/CAD;1.09140;64
18:08:15.248;USD/CAD;1.09140;45
18:08:16.731;AUD/CAD;1.01003;24
18:08:16.749;AUD/CAD;1.01003;52
18:08:17.481;GBP/USD;1.67504;74
18:08:17.496;GBP/USD;1.67504;96
18:08:18.885;EUR/USD;1.36221;85
18:08:18.885;EUR/USD;1.36221;3
18:08:18.900;EUR/USD;1.36221;46
18:08:18.900;EUR/USD;1.36221;10
18:08:18.915;EUR/USD;1.36221;71
18:08:18.929;EUR/USD;1.36221;71
18:08:19.244;USD/JPY;102.44600;54
18:08:19.264;USD/JPY;102.44600;90
18:08:19.388;USD/JPY;102.44600;78
18:08:19.401;USD/JPY;102.44600;97
18:08:19.994;EUR/USD;1.36221;86
18:08:19.994;EUR/USD;1.36221;5
18:08:20.17;EUR/USD;1.36221;94
18:08:20.32;EUR/USD;1.36221;9
18:08:20.45;EUR/USD;1.36221;90
18:08:20.58;EUR/USD;1.36221;93
18:08:20.289;GBP/CHF;1.50199;86
18:08:20.303;GBP/CHF;1.50199;60
18:08:20.541;USD/CAD;1.09140;60
18:08:20.553;USD/CAD;1.09140;70
18:08:20.695;GBP/USD;1.67504;47
18:08:20.709;GBP/USD;1.67504;36
18:08:20.724;GBP/USD;1.67504;11
18:08:20.742;AUD/NZD;1.09762;41
18:08:20.749;GBP/CHF;1.50203;72
18:08:20.756;GBP/CHF;1.50203;85
18:08:20.756;AUD/NZD;1.09762;30
18:08:20.788;EUR/USD;1.36218;40
18:08:20.790;EUR/USD;1.36218;34
18:08:20.792;AUD/CHF;0.82980;55
18:08:20.802;AUD/CHF;0.82980;1

18:08:20.803;EUR/USD;1.36218;49
18:08:20.805;GBP/USD;1.67504;90
18:08:20.815;EUR/USD;1.36217;61
18:08:20.815;GBP/USD;1.67504;82
18:08:20.816;EUR/USD;1.36217;74
18:08:20.828;GBP/USD;1.67504;45
18:08:20.828;EUR/USD;1.36218;77
18:08:20.840;EUR/USD;1.36217;14
18:08:20.851;GBP/CHF;1.50203;1
18:08:20.853;GBP/CHF;1.50203;13
18:08:20.853;EUR/USD;1.36218;70
18:08:20.867;EUR/USD;1.36217;53
(Btw, I did not run into an error message, even though some writes were done on the same millisecond time stamp).

But instead of just identifying an issue ;), I'd also like to offer three potential workarounds:

1) How many times per second (or per minute) would you like to update the file? If this is not continuous (say, every 5 seconds), perhaps you could create a global variable string that each indicator appends to. Then create one indicator that, once every five seconds, retrieves the GV string and write it to the file (and then reset it). But in that situation I'm note sure if there is absolutely no data loss.

2) Perhaps you could create a file for each symbol, and then read each file in a separate worksheet in Excel . Then you'll need one 'master worksheet' that combines them all.

3) You could batch the write to file. For example, don't write every tick update to the file immediately, but store all ticks updates in a string, and as soon as you have a certain amount of price updates, write this whole string in one go to the file. In that case, you bring down the amount of `FileAppend` open/close actions considerably. (This is the option I'd personally use).

But these three points are based on the assumption that you need to write to file a very often (i.e., continuously). If that is not the case, the odds are much smaller than in my example above that this happens at the same time.

So to summarise, if the `FileAppend` actions do not happen on the same millisecond (give or take 2-3 milliseconds), I would not be concerned about the risk of congestion.

The best test would be to use an indicator, like the one above, and open the file that is written to and see how fluently it updates. On my pc, time gaps of around 4-5 milliseconds were still being written fluently, but as soon as the write actions happened on the same millisecond, gaps started occurring and the file was updated in 'steps'.

(Edit: If the size of your string increase, so does the risk of congestion. When testing, write a string of the same length (or longer) to see how it would perform).

User avatar
Smoky
Posts: 507
Joined: 03 Dec 2010
Location: Thailand
Has thanked: 97 times
Been thanked: 115 times

Re: Writing to a text file: Risk of congestion???

Postby Smoky » 03 Jun 2014

As I tell you, if you use this syntax there are not congestion ...

Code: Select all

Vars: Enrg(""), Enrg2(""), Enrg3(""), FileLog("C:\CSV\LOG.csv");

Enrg = dat + ";" + hour +";" +order +";"+ numtostr(close,5) +";" ....+

print(file(Filelog),Enrg+Enrg2);
I think print command is buffered ;)


Return to “MultiCharts”