Difference between revisions of "Importing Data"

From MultiCharts
Jump to navigation Jump to search
Line 11: Line 11:
 
To import data:
 
To import data:
 
# Select a symbol (see [[Operating_Symbol_List#Selecting Symbol(s) in Symbol List|Selecting Symbol(s) in Symbol list]] section). If the selected symbol contains quotes with the same timestamp as in the file, they'll be overwritten without warning. Make sure the symbol's price scale matches the price scale of the input data file. The program always uses the symbol's price scale when importing.
 
# Select a symbol (see [[Operating_Symbol_List#Selecting Symbol(s) in Symbol List|Selecting Symbol(s) in Symbol list]] section). If the selected symbol contains quotes with the same timestamp as in the file, they'll be overwritten without warning. Make sure the symbol's price scale matches the price scale of the input data file. The program always uses the symbol's price scale when importing.
# Make sure the file you wish to import is a 1 Tick, Minutes or 1 Day. If a file contains bars, but has no static time increment, this is a Tick\Volume Bar file and must be processed using [[<span style="color: red;">'''LINK'''</span> /110_ASCII_Mapping.html name="";  |ASCII Mapping]].
+
# Make sure the file you wish to import is a 1 Tick, Minutes or 1 Day. If a file contains bars, but has no static time increment, this is a Tick\Volume Bar file and must be processed using [[Mapping_ASCII|ASCII Mapping]].
 
# On the '''File''' menu, click '''Import Data''', or click the '''Import Data''' http://www.multicharts.com/img/wiki/toolbar/QM/05_DataImp.png icon on the toolbar.
 
# On the '''File''' menu, click '''Import Data''', or click the '''Import Data''' http://www.multicharts.com/img/wiki/toolbar/QM/05_DataImp.png icon on the toolbar.
 
# Select '''ASCII''' or press '''Ctrl+I'''.
 
# Select '''ASCII''' or press '''Ctrl+I'''.

Revision as of 17:25, 23 February 2012

The program supports importing from most known text (ASCII) files. The principal requirement for the file is a clear structure, data integrity and absence of logical errors.

See [[LINK /107_Data_Import.html name="2"; |Invalid ASCII files]] section for information on the most common file faults.

The program is based on analyzing several hundreds of quote files, most in unique formats, which served as principles for file structure analysis. Despite successful automatic recognition of over 95% of files, there are still situations requiring the user to interfere, mostly because of improper file structure or if the program cannot make up a conclusion on its structure.

Note: For more information on known import limitations see [[LINK /107_Data_Import.html name="3"; |ASCII Import Limitations]] section. Be careful when importing files.


How to import ASCII data

To import data:

  1. Select a symbol (see Selecting Symbol(s) in Symbol list section). If the selected symbol contains quotes with the same timestamp as in the file, they'll be overwritten without warning. Make sure the symbol's price scale matches the price scale of the input data file. The program always uses the symbol's price scale when importing.
  2. Make sure the file you wish to import is a 1 Tick, Minutes or 1 Day. If a file contains bars, but has no static time increment, this is a Tick\Volume Bar file and must be processed using ASCII Mapping.
  3. On the File menu, click Import Data, or click the Import Data 05_DataImp.png icon on the toolbar.
  4. Select ASCII or press Ctrl+I.
  5. Indicate the Path. The path can be typed in, pasted from the clipboard or selected using the mouse. The selected path persists after converting the file so on next operation the previous path will be used by default. If it is necessary to move up a level, click the Up icon. To change the method of displaying files click the rightmost icon Views and select an option from the drop down menu.
  6. Select the desired file from the indicated folder.
  7. If the Contents of the selected file field the information is garbled, choose the proper encoding in the File origin.
  8. Make sure the Resolution is valid. If the value is not correct, the possible causes may be the reason:
    • The Date field is not defined or not properly defined
      To set date:
      1. Click the Date column heading.
      2. Select Date and the corresponding template.
      3. Click Custom if there is no such template.
      4. Create a template. If the number of symbols in the date changes, use an asterisk * to replace the symbol, e.g. 9/9/2004 changed to 10/9/2004. The number of symbols changes from 6 to 7.
      5. Apply the created template by selecting it from the list.
    • The Time filed is not defined or not properly defined
      To set time:
      1. Click the Time column heading.
      2. Select Time and the corresponding template.
      3. If the template does not exist, click Custom.
      4. Create a template. If the number of symbols in the date changes, use an asterisk * to replace the symbol, e.g. 8:30 changes to 10:10. The number of symbols changes from 4 to 5.
      5. To set a time template, e.g. 5:22 PM, the template must look like H:MM tt@PM, where tt means the postfix placeholder and after @ the PM value stands.
      6. Apply the created template by selecting it from the list.
    • Wrong detection of the starting line of quotes
      To set Start at Row You should indicate how many lines to skip.
      After those parameters have been properly set, the program analyses the file again and sets the proper Resolution.
  9. Select Field Ask, Bid or Trade to define what king of data the file contains.
  10. In the Time Zone please select the time zone of the quotes' origin.
  11. If the price or volume needs to be multiplied or divided, select a proper factor in Price Multiplier or Volume Multiplier.
  12. To select the time period for importing data set the Start Date/Start Date Time and End Date/End Date Time.

Invalid ASCII files

In most cases the invalid files are lacking the required fields, lacking data in some column, mixed up time and dates, irregular bar intervals. The program filters most faults and informs the user of them, but it is not possible in each and every case. Please see the analysis of typical invalid files below.

Invalid structure

Example 1. Two of the four necessary bar components are missing:
05-12-69  46.4  33.9
12-12-69  51.7  34.5
19-12-69  55.2  31

Example 2. Date is missing:
1.2061, 1.2065, 1.2061, 1.2065
1.2065, 1.2069, 1.2065, 1.2069
1.2069, 1.2070, 1.2066, 1.2066

Incomplete data

Example 1. File has a damaged string:
2/28/2002  10:45  31505  31530  31480  31480  125  86
2/28/2002  10:50  31480  31505  31480  31500  44    42
2/28/20

Example 2. File has variable number of columns:
СС12-D,10/01/1995,3.51,3.53,3.5,3.5,0,2643200,,,
СС12-D,11/01/1995,3.44,3.53,3.43,3.44,0,501600,0,,
СС12-D,12/01/1995,3.45,3.5,3.41,3.45,0,1545600,0.017544,,
СС12-D,13/01/1995,3.34,3.47,3.34,3.44,0,2353600,-0.038184,,
СС12-D 16/01/1995,3.39,3.41,3.34,3.35,0,1897600,-0.023325,,
СС12-D,17/01/1995,3.47,3.5,3.38,3.41,0,7425600,-0.011461,,
СС12-D,18/01/1995,3.45,3.5,3.41,3.5,0,766400,0.002903,-3.518377,

Logical errors

Example 1. The file has a 60 minute interval, but with 1759 instead of 1800:
<TICKER>,<PER>,<DTYYYYMMDD>,<TIME>,<OPEN>,<HIGH>,<LOW>,<CLOSE>
EURUSD,I,19980420,160000,1.1005,1.1010,1.1005,1.1010
EURUSD,I,19980420,170000,1.1003,1.1003,1.1003,1.1003
EURUSD,I,19980420,175900,1.1012,1.1012,1.1012,1.1012
EURUSD,I,19980420,200000,1.1012,1.1016,1.1009,1.1015
EURUSD,I,19980420,210000,1.1017,1.1020,1.1000,1.1006

Example 2. Time is not ordered chronologically:
GBPUSD,19890627,0,2200,15457,B,N,N
GBPUSD,19890627,0,2200,15462,A,N,N
GBPUSD,19890627,1,0100,15485,B,N,N
GBPUSD,19890627,1,0100,15493,A,N,N

ASCII Import Limitations

Important: Local data storage saves the following resolution types: 1 Tick, 1 Minute, 1 Day. All other resolutions are sampled from those.

Other data types may not be added to the database - use [[LINK /110_ASCII_Mapping.html name=""; |ASCII Mapping]] to process such files.