+1 888 340 6572
MultiCharts Project Management
previous_open_issue.png
Go to the previous open issue
previous_issue.png
Go to the previous issue (open or closed)
star_faded.png
Please log in to bookmark issues
feature_request_small.png
Open Feature request MC-2262

QM: Add tools for cleaning data

action_vote_minus_faded.png
1
Votes
action_vote_plus_faded.png
next_issue.png
Go to the next issue (open or closed)
next_open_issue.png
Go to the next open issue
Description

Since datafeeds are not perfect, one needs to clean erroneous data or know of /fill-in the gaps.
Right now, a trader has to change each and every bad tick/value manually.

Would be very helpful if MC could automate these tasks!

1) Have a tool that would scan the symbol's data in a database and report:
- gaps in data
- values >xpts away from the norm

2) add ability to change data in a database - either programmatically by the trader or automatically with several pre-set options for the most common data problems.

Comments (2)
#1
user-offline.png  Alex MultiCharts (Alex MultiCharts)
Jun 19, 2017 - 09:14
At the moment we consider that available tools should be enough to manage the received data: reloading functionality, Edit Data window functionality, data export, etc.

If you want us to improve currently available functionality, please let us know the following:

  1. How do we know that gaps in data are incorrect? As there are thousand of instrument that are no updating frequently and instruments with breaks within the session.
  2. Where do we get information about the "norm" you are referring to? We have dozens of data vendors with various provided data.
  3. How do you consider data management in MC should be made from your point of view? Please provide details.


#2
user-offline.png  Zheka (Zheka)
Jun 19, 2017 - 11:03

Alex,
current tools are all manual labor.

1) Gaps can be measured against an instrument's session template. Any serious user will take care to set them appropriately, incl. breaks within a session. Or against default Exchange settings (which MC updates more or less regularly).
(you might need to resurrect Sess1..and Sess2Endtime keywords).

2) 'Norm' is surrounding values for the bar (O-H-L-C) and the deviation can be left for the user to specify (like x% or y pts).
Same concept as with RT tick filtering, implemented by both by IB and several other software packages.

3) There can be buttons to :
- "unmark" "empty" periods in the database - so that MC tries to re-download them itself;
- apply filtering (as specified earlier by the user) to the data in the database - by substituting "bad" values
* "bad" open-> with previous bar close and "bad" close ->with next' bar's open
* "bad" high-low - > with the closest open or close +/- say 50% of the range (over last 30 bars);

Even if implementation is not perfect from the fist attempt, it will still be better than nothing and will improve after user feedback. I am also sure you are in a better position to advise how "best in class", professional-grade "data cleaning" looks like.

The key is to automate as much of it as it possible.

History
Issue basics
  • Type of issue
    Feature request
  • Category
    Not determined
  • Targeted for
    Not determined
  • Status
    Need Details
User pain
  • Type of bug
    Not triaged
  • Likelihood
    Not triaged
  • Effect
    Not triaged
Affected by this issue (0)
There are no items
People involved
  • Posted by
    user-offline.png  Zheka (Zheka)
  • Owned by
    Not owned by anyone
  • Assigned to
    Not assigned to anyone
  • Subscribers
    2 subscriber(s)
    Click here to show the list of subscribers
Times and dates
  • Posted at
  • Last updated
Issue details
  • Resolution
    Not determined
Attachments (0)
There is nothing attached to this issue
Commits (0)
There are no code checkins for this issue
Duplicate issues (0)
This issue does not have any duplicates