Difference between revisions of "Importing Data"

From MultiCharts
Jump to navigation Jump to search
(9 intermediate revisions by 5 users not shown)
Line 6: Line 6:
 
# Select the file location and click  
 
# Select the file location and click  
 
# Click '''Next'''
 
# Click '''Next'''
# Select select th einstruments to import  
+
# Select the instruments to import  
 
# Click '''Next'''
 
# Click '''Next'''
 
# Click '''Finish'''.
 
# Click '''Finish'''.
 
  
 
== Importing ASCII data ==  
 
== Importing ASCII data ==  
Line 22: Line 21:
  
 
<br>
 
<br>
 +
 +
== What values can be imported ==
 +
 +
For different data types different values can be imported. Some of them are obligatory and marked below as '''bold'''.
 +
* '''Tick data:'''
 +
'''Date''', '''Time''', '''Price''', '''Volume'''.
 +
 +
* '''Minute data:'''
 +
'''Date''', '''Time''', '''Open''', '''High''', '''Low''', '''Close''', '''Volume''', Up Volume, Down Volume, Unchanged Volume, Up Ticks, Down Ticks, Unchanged Ticks, Total Ticks.
 +
 +
* '''Daily data:'''
 +
'''Date''', '''Open''', '''High''', '''Low''', '''Close''', '''Volume''', Up Volume, Down Volume, Unchanged Volume, Up Ticks, Down Ticks, Unchanged Ticks, Total Ticks, Open Interest.
 +
 +
 
== How to import ASCII data ==  
 
== How to import ASCII data ==  
  
Line 29: Line 42:
 
# Select a symbol (see [[Operating_Symbol_List#Selecting Symbol(s) in Symbol List|Selecting Symbol(s) in Symbol list]] section). If the selected symbol contains quotes with the same timestamp as in the file, they'll be overwritten without warning. Make sure the symbol's price scale matches the price scale of the input data file. The program always uses the symbol's price scale when importing.
 
# Select a symbol (see [[Operating_Symbol_List#Selecting Symbol(s) in Symbol List|Selecting Symbol(s) in Symbol list]] section). If the selected symbol contains quotes with the same timestamp as in the file, they'll be overwritten without warning. Make sure the symbol's price scale matches the price scale of the input data file. The program always uses the symbol's price scale when importing.
 
# Make sure the file you wish to import is a 1 Tick, Minutes or 1 Day. If a file contains bars, but has no static time increment, this is a Tick\Volume Bar file and must be processed using [[Mapping_ASCII|ASCII Mapping]].
 
# Make sure the file you wish to import is a 1 Tick, Minutes or 1 Day. If a file contains bars, but has no static time increment, this is a Tick\Volume Bar file and must be processed using [[Mapping_ASCII|ASCII Mapping]].
# On the '''File''' menu, click '''Import Data''', or click the '''Import Data''' http://www.multicharts.com/img/wiki/toolbar/QM/05_DataImp.png icon on the toolbar.
+
# On the '''File''' menu, click '''Import Data''', or click the '''Import Data''' [[File:QM_Import.png]] icon on the toolbar.
 
# Select '''ASCII''' or press '''Ctrl+I'''.
 
# Select '''ASCII''' or press '''Ctrl+I'''.
 
# Indicate the '''Path'''. The path can be typed in, pasted from the clipboard or selected using the mouse. The selected path persists after converting the file so on next operation the previous path will be used by default. If it is necessary to move up a level, click the '''Up''' icon. To change the method of displaying files click the rightmost icon '''Views''' and select an option from the drop down menu.
 
# Indicate the '''Path'''. The path can be typed in, pasted from the clipboard or selected using the mouse. The selected path persists after converting the file so on next operation the previous path will be used by default. If it is necessary to move up a level, click the '''Up''' icon. To change the method of displaying files click the rightmost icon '''Views''' and select an option from the drop down menu.
# Select the desired file from the indicated folder.
+
# Select the desired file from the indicated folder. Only one file can be imported at a time.
 
# If the '''Contents of the selected file''' field the information is garbled, choose the proper encoding in the File origin.  
 
# If the '''Contents of the selected file''' field the information is garbled, choose the proper encoding in the File origin.  
 
# Make sure the '''Resolution''' is valid. If the value is not correct, the possible causes may be the reason:<br>
 
# Make sure the '''Resolution''' is valid. If the value is not correct, the possible causes may be the reason:<br>
Line 41: Line 54:
 
#*# Click '''Custom''' if there is no such template.
 
#*# Click '''Custom''' if there is no such template.
 
#*# Create a template. If the number of symbols in the date changes, use an asterisk * to replace the symbol, e.g. 9/9/2004 changed to 10/9/2004. The number of symbols changes from 6 to 7.
 
#*# Create a template. If the number of symbols in the date changes, use an asterisk * to replace the symbol, e.g. 9/9/2004 changed to 10/9/2004. The number of symbols changes from 6 to 7.
 +
#*#* Allowed symbols in template for Date: '''m''' - month, '''d''' - day, '''y''' - year;
 +
#*#* '''*''' sign means that length of chunk is undefined;<br><div style="background-color: #E5F6FF;">'''Example''': To convert date ''1 Feb 2002'' (2/1/2002) type ''*d mmm yyyy'' in the template field.</div>
 
#*# Apply the created template by selecting it from the list.  
 
#*# Apply the created template by selecting it from the list.  
 
#* The Time filed is not defined or not properly defined
 
#* The Time filed is not defined or not properly defined
Line 48: Line 63:
 
#*# If the template does not exist, click '''Custom'''.
 
#*# If the template does not exist, click '''Custom'''.
 
#*# Create a template. If the number of symbols in the date changes, use an asterisk * to replace the symbol, e.g. 8:30 changes to 10:10. The number of symbols changes from 4 to 5.
 
#*# Create a template. If the number of symbols in the date changes, use an asterisk * to replace the symbol, e.g. 8:30 changes to 10:10. The number of symbols changes from 4 to 5.
#*# To set a time template, e.g. 5:22 PM, the template must look like H:MM tt@PM, where tt means the postfix placeholder and after @ the PM value stands.  
+
#*#* Allowed symbols in template for Time: '''h''' - hour, '''m''' - minute, '''s''' - second,'''f''' - millisecond '''t''' - time modifier AM/PM.
 +
#*#* '''*''' sign means that length of chunk is undefined;
 +
#*#* '''@''' sign splits the whole template and time modifier PM lexem.<br><div style="background-color: #E5F6FF;">'''Example''': To convert time ''9:20 PM'' type ''*h:*m tt@pm'' in the template field.</div><div style="background-color: #E3FBE5;">'''Note''': Templates are stored in recognition engine's local settings. Time modifier can be in any language supported by current operating system.</div>
 
#*# Apply the created template by selecting it from the list.
 
#*# Apply the created template by selecting it from the list.
 
#* Wrong detection of the starting line of quotes
 
#* Wrong detection of the starting line of quotes
 
#*: To set '''Start at Row''' You should indicate how many lines to skip.<br><div style="background-color: #F8F6CD;">After those parameters have been properly set, the program analyses the file again and sets the proper Resolution.</div>
 
#*: To set '''Start at Row''' You should indicate how many lines to skip.<br><div style="background-color: #F8F6CD;">After those parameters have been properly set, the program analyses the file again and sets the proper Resolution.</div>
# Select '''Field Ask''', '''Bid''' or '''Trade''' to define what king of data the file contains.
+
# Select '''Field Ask''', '''Bid''' or '''Trade''' to define what kind of data the file contains. It is only possible to import only '''one''' quote field at a time.
 
# In the '''Time Zone''' please select the time zone of the quotes' origin.
 
# In the '''Time Zone''' please select the time zone of the quotes' origin.
 
# If the price or volume needs to be multiplied or divided, select a proper factor in '''Price Multiplier''' or '''Volume Multiplier'''.
 
# If the price or volume needs to be multiplied or divided, select a proper factor in '''Price Multiplier''' or '''Volume Multiplier'''.
Line 58: Line 75:
  
 
<br>
 
<br>
 +
 
== Invalid ASCII files ==
 
== Invalid ASCII files ==
 
In most cases the invalid files are lacking the required fields, lacking data in some column, mixed up time and dates, irregular bar intervals. The program filters most faults and informs the user of them, but it is not possible in each and every case. Please see the analysis of typical invalid files below.
 
In most cases the invalid files are lacking the required fields, lacking data in some column, mixed up time and dates, irregular bar intervals. The program filters most faults and informs the user of them, but it is not possible in each and every case. Please see the analysis of typical invalid files below.

Revision as of 15:03, 11 April 2019

Importing .qmd data (MultiCharts data format)

  1. Open QuoteManager window
  2. In QuoteManager main menu select File, then point to Import Data and click Import Instruments…
  3. Select the file location and click
  4. Click Next
  5. Select the instruments to import
  6. Click Next
  7. Click Finish.

Importing ASCII data

The program supports importing from most known text (ASCII) files. The principal requirement for the file is a clear structure, data integrity and absence of logical errors.

See Invalid ASCII files section for information on the most common file faults.

The program is based on analyzing several hundreds of quote files, most in unique formats, which served as principles for file structure analysis. Despite successful automatic recognition of over 95% of files, there are still situations requiring the user to interfere, mostly because of improper file structure or if the program cannot make up a conclusion on its structure.

Note: For more information on known import limitations see ASCII Import Limitations section. Be careful when importing files.


What values can be imported

For different data types different values can be imported. Some of them are obligatory and marked below as bold.

  • Tick data:

Date, Time, Price, Volume.

  • Minute data:

Date, Time, Open, High, Low, Close, Volume, Up Volume, Down Volume, Unchanged Volume, Up Ticks, Down Ticks, Unchanged Ticks, Total Ticks.

  • Daily data:

Date, Open, High, Low, Close, Volume, Up Volume, Down Volume, Unchanged Volume, Up Ticks, Down Ticks, Unchanged Ticks, Total Ticks, Open Interest.


How to import ASCII data

To import data:

  1. Open QuoteManager window
  2. Select a symbol (see Selecting Symbol(s) in Symbol list section). If the selected symbol contains quotes with the same timestamp as in the file, they'll be overwritten without warning. Make sure the symbol's price scale matches the price scale of the input data file. The program always uses the symbol's price scale when importing.
  3. Make sure the file you wish to import is a 1 Tick, Minutes or 1 Day. If a file contains bars, but has no static time increment, this is a Tick\Volume Bar file and must be processed using ASCII Mapping.
  4. On the File menu, click Import Data, or click the Import Data QM Import.png icon on the toolbar.
  5. Select ASCII or press Ctrl+I.
  6. Indicate the Path. The path can be typed in, pasted from the clipboard or selected using the mouse. The selected path persists after converting the file so on next operation the previous path will be used by default. If it is necessary to move up a level, click the Up icon. To change the method of displaying files click the rightmost icon Views and select an option from the drop down menu.
  7. Select the desired file from the indicated folder. Only one file can be imported at a time.
  8. If the Contents of the selected file field the information is garbled, choose the proper encoding in the File origin.
  9. Make sure the Resolution is valid. If the value is not correct, the possible causes may be the reason:
    • The Date field is not defined or not properly defined
      To set date:
      1. Click the Date column heading.
      2. Select Date and the corresponding template.
      3. Click Custom if there is no such template.
      4. Create a template. If the number of symbols in the date changes, use an asterisk * to replace the symbol, e.g. 9/9/2004 changed to 10/9/2004. The number of symbols changes from 6 to 7.
        • Allowed symbols in template for Date: m - month, d - day, y - year;
        • * sign means that length of chunk is undefined;
          Example: To convert date 1 Feb 2002 (2/1/2002) type *d mmm yyyy in the template field.
      5. Apply the created template by selecting it from the list.
    • The Time filed is not defined or not properly defined
      To set time:
      1. Click the Time column heading.
      2. Select Time and the corresponding template.
      3. If the template does not exist, click Custom.
      4. Create a template. If the number of symbols in the date changes, use an asterisk * to replace the symbol, e.g. 8:30 changes to 10:10. The number of symbols changes from 4 to 5.
        • Allowed symbols in template for Time: h - hour, m - minute, s - second,f - millisecond t - time modifier AM/PM.
        • * sign means that length of chunk is undefined;
        • @ sign splits the whole template and time modifier PM lexem.
          Example: To convert time 9:20 PM type *h:*m tt@pm in the template field.
          Note: Templates are stored in recognition engine's local settings. Time modifier can be in any language supported by current operating system.
      5. Apply the created template by selecting it from the list.
    • Wrong detection of the starting line of quotes
      To set Start at Row You should indicate how many lines to skip.
      After those parameters have been properly set, the program analyses the file again and sets the proper Resolution.
  10. Select Field Ask, Bid or Trade to define what kind of data the file contains. It is only possible to import only one quote field at a time.
  11. In the Time Zone please select the time zone of the quotes' origin.
  12. If the price or volume needs to be multiplied or divided, select a proper factor in Price Multiplier or Volume Multiplier.
  13. To select the time period for importing data set the Start Date/Start Date Time and End Date/End Date Time.


Invalid ASCII files

In most cases the invalid files are lacking the required fields, lacking data in some column, mixed up time and dates, irregular bar intervals. The program filters most faults and informs the user of them, but it is not possible in each and every case. Please see the analysis of typical invalid files below.


Invalid structure

Example 1. Two of the four necessary bar components are missing:
05-12-69  46.4  33.9
12-12-69  51.7  34.5
19-12-69  55.2  31

Example 2. Date is missing:
1.2061, 1.2065, 1.2061, 1.2065
1.2065, 1.2069, 1.2065, 1.2069
1.2069, 1.2070, 1.2066, 1.2066


Incomplete data

Example 1. File has a damaged string:
2/28/2002  10:45  31505  31530  31480  31480  125  86
2/28/2002  10:50  31480  31505  31480  31500  44    42
2/28/20

Example 2. File has variable number of columns:
СС12-D,10/01/1995,3.51,3.53,3.5,3.5,0,2643200,,,
СС12-D,11/01/1995,3.44,3.53,3.43,3.44,0,501600,0,,
СС12-D,12/01/1995,3.45,3.5,3.41,3.45,0,1545600,0.017544,,
СС12-D,13/01/1995,3.34,3.47,3.34,3.44,0,2353600,-0.038184,,
СС12-D 16/01/1995,3.39,3.41,3.34,3.35,0,1897600,-0.023325,,
СС12-D,17/01/1995,3.47,3.5,3.38,3.41,0,7425600,-0.011461,,
СС12-D,18/01/1995,3.45,3.5,3.41,3.5,0,766400,0.002903,-3.518377,


Logical errors

Example 1. The file has a 60 minute interval, but with 1759 instead of 1800:
<TICKER>,<PER>,<DTYYYYMMDD>,<TIME>,<OPEN>,<HIGH>,<LOW>,<CLOSE>
EURUSD,I,19980420,160000,1.1005,1.1010,1.1005,1.1010
EURUSD,I,19980420,170000,1.1003,1.1003,1.1003,1.1003
EURUSD,I,19980420,175900,1.1012,1.1012,1.1012,1.1012
EURUSD,I,19980420,200000,1.1012,1.1016,1.1009,1.1015
EURUSD,I,19980420,210000,1.1017,1.1020,1.1000,1.1006

Example 2. Time is not ordered chronologically:
GBPUSD,19890627,0,2200,15457,B,N,N
GBPUSD,19890627,0,2200,15462,A,N,N
GBPUSD,19890627,1,0100,15485,B,N,N
GBPUSD,19890627,1,0100,15493,A,N,N


ASCII Import Limitations

Important: Local data storage saves the following resolution types: 1 Tick, 1 Minute, 1 Day. All other resolutions are sampled from those.

Other data types may not be added to the database - use ASCII Mapping to process such files.