SEVERE BUG: Autotrading - Complete Breakdown

Questions about MultiCharts and user contributed studies.
TraderJ
Posts: 37
Joined: 14 Jul 2009
Location: Germany

SEVERE BUG: Autotrading - Complete Breakdown

Postby TraderJ » 14 Jul 2009

I am using Multicharts 5.0 Release 2369.

I am autotrading FOREX with IB (Interactive Brokers) on a 1-min time frame for about a week now.

The strategy uses LIMIT orders and allows for unlimited entries and exits per bar.
The limits are readjusted up to several times within a minute, but usually less.
When a limit is to be modified, MC cancels the current order and submits a new order with the new limit. (A modification would be better.)

Now I observed the following bug: MC wants to cancel the current order, but just at this very moment it gets executed (this is an educated guess). Now the following things happen:
1. MC pops tons of error messages on my screen. (no problem)
2. MC does not realize that it now has a position which it should be taking care of by attaching a STOP and a PROFIT TARGET to it. (very bad)
3. Instead, it keeps sending a new LIMIT order to IB. This order gets immediately cancelled because there is not enough margin to enter another position. MC finds this out and tries to solve the problem by resending the same order. (bad)

This problem seems to be rare as I already had dozens of successful trades.
It seems to be a timing problem (race condition).

I would be very glad to see this bug fixed.

brendanh
Posts: 154
Joined: 07 Apr 2007

Postby brendanh » 14 Jul 2009

I also autotrade forex via IB and have had overfills. I'm on v5.5 beta 2 and use a 5 minute time frame. I was short 50,000 on Monday morning, and my signal produced a reverse long trade. Instead of a 100,000 buy, Multicharts sent two orders for 150,000(!) to IB to put me long 250,000 instead of 50,000.

I have seen wrong reversal amounts in the past, so I don’t think this problem is exclusive to 5.5 beta 2.

janus
Posts: 747
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 83 times

Re: SEVERE BUG: Autotrading - Complete Breakdown

Postby janus » 14 Jul 2009

I'm not sure if it's related but I infrequently get a race condition message in the QuoteManager Event Log after I've done a fair amount of backfilling with MC (eg, 3 months of trade, bid and ask of more than a couple of contracts). MC from that point on fails to download and backfill anything. To reset it I exits MC, QM and tws and restart all three. Both issues must be examined and resolved in any case.

SUPER
Posts: 622
Joined: 03 Mar 2007
Has thanked: 99 times
Been thanked: 80 times

Postby SUPER » 15 Jul 2009

I also autotrade forex via IB and have had overfills. I'm on v5.5 beta 2 and use a 5 minute time frame. I was short 50,000 on Monday morning, and my signal produced a reverse long trade. Instead of a 100,000 buy, Multicharts sent two orders for 150,000(!) to IB to put me long 250,000 instead of 50,000.

I have seen wrong reversal amounts in the past, so I don’t think this problem is exclusive to 5.5 beta 2.


I have not experinced such situation on future contracts so far. I would really like to try test out the forex market and will appreciate if you can provide some basic information:

1)How often have you come accross such situations?
2)Are you using live or simulated account?
3)Are you using IOG system?
4)Your Automation setting?

Regards
Super

brendanh
Posts: 154
Joined: 07 Apr 2007

Postby brendanh » 15 Jul 2009

1)How often have you come accross such situations?
Perhaps 1-2 per week.
2)Are you using live or simulated account?
Live
3)Are you using IOG system?
No
4)Your Automation setting?
Async, assume same as on chart.

Also, occasionally I've enabled automation while in a position, and MC hasn't sent the stop & limit exits to IB, although I've not had this since I upgraded to beta 2 last week.

Super, can you return the favour and answer the above yourself? Do you have any automation problems? I trade futures via TS at the moment, but am looking to move everything to MC\IB.

SUPER
Posts: 622
Joined: 03 Mar 2007
Has thanked: 99 times
Been thanked: 80 times

Postby SUPER » 15 Jul 2009

repeat
Last edited by SUPER on 17 Jul 2009, edited 1 time in total.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 16 Jul 2009

Thank you for the information. We will analyze it and let you know within a few days.

SUPER
Posts: 622
Joined: 03 Mar 2007
Has thanked: 99 times
Been thanked: 80 times

Postby SUPER » 17 Jul 2009

1)How often have you come accross such situations?
Perhaps 1-2 per week.
2)Are you using live or simulated account?
Live
3)Are you using IOG system?
No
4)Your Automation setting?
Async, assume same as on chart.

Also, occasionally I've enabled automation while in a position, and MC hasn't sent the stop & limit exits to IB, although I've not had this since I upgraded to beta 2 last week.

Super, can you return the favour and answer the above yourself? Do you have any automation problems? I trade futures via TS at the moment, but am looking to move everything to MC\IB.


I use IOG in my systems and my exit orders are market orders. My entry orders are Stop / Limit and I use Async mode with partial fill ticked and convert unfilled strategy orders replacment set at "0" seconds ( unfortunate time out limitation with IOG mode at present). I never enable automaition with open position. I have not had any problems so far on live trading single contract.

geektrader
Posts: 100
Joined: 17 Jul 2009
Location: Germany

Postby geektrader » 17 Jul 2009

I am having this problem as well. Sometimes when TWS is disconnected from Market Data Server, TWS still tries to send an order and if it doesn´t get executed within a few seconds, it resends this order over and over. Then when TWS is back online, it executes ALL the orders TWS sent in the inactive period and hence position size can easily exceed 4 or 8 times of what you wanted and then autotrading runs completely out of sync.

I am also trading Forex with IB when this problem occurs.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Re: SEVERE BUG: Autotrading - Complete Breakdown

Postby Andrew Kirillov » 30 Jul 2009

I am using Multicharts 5.0 Release 2369.

I am autotrading FOREX with IB (Interactive Brokers) on a 1-min time frame for about a week now.

The strategy uses LIMIT orders and allows for unlimited entries and exits per bar.
The limits are readjusted up to several times within a minute, but usually less.
When a limit is to be modified, MC cancels the current order and submits a new order with the new limit. (A modification would be better.)

Now I observed the following bug: MC wants to cancel the current order, but just at this very moment it gets executed (this is an educated guess). Now the following things happen:
1. MC pops tons of error messages on my screen. (no problem)
2. MC does not realize that it now has a position which it should be taking care of by attaching a STOP and a PROFIT TARGET to it. (very bad)
3. Instead, it keeps sending a new LIMIT order to IB. This order gets immediately cancelled because there is not enough margin to enter another position. MC finds this out and tries to solve the problem by resending the same order. (bad)

This problem seems to be rare as I already had dozens of successful trades.
It seems to be a timing problem (race condition).

I would be very glad to see this bug fixed.




This is correct. MultiCharts attaches brackets to a position as soon as the complete order filled. It is important to mention that you can see such behavior in many platforms and NT6.5 and TS8.6 are among them. IB handles it properly and attachés the orders to partially filled position and modifies them as soon as new orders are filled.
We will improve this in the future.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 30 Jul 2009

I also autotrade forex via IB and have had overfills. I'm on v5.5 beta 2 and use a 5 minute time frame. I was short 50,000 on Monday morning, and my signal produced a reverse long trade. Instead of a 100,000 buy, Multicharts sent two orders for 150,000(!) to IB to put me long 250,000 instead of 50,000.

I have seen wrong reversal amounts in the past, so I don’t think this problem is exclusive to 5.5 beta 2.


We tried to reproduce it, but failed. We used different symbols for charting and execution.
If it is reproducible please send us your WSP, signals (if possible) and logs after the issue happens.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 30 Jul 2009

I am having this problem as well. Sometimes when TWS is disconnected from Market Data Server, TWS still tries to send an order and if it doesn´t get executed within a few seconds, it resends this order over and over. Then when TWS is back online, it executes ALL the orders TWS sent in the inactive period and hence position size can easily exceed 4 or 8 times of what you wanted and then autotrading runs completely out of sync.

I am also trading Forex with IB when this problem occurs.


Could you give us your WSP and signals?

geektrader
Posts: 100
Joined: 17 Jul 2009
Location: Germany

Postby geektrader » 30 Jul 2009

It can easily be reproduced this way for example: Turn on "Pre-Cautions" within TWS and also enabled those for the API. Then run a strategy and once it sends it first order, TWS will ask if you want to turn off "Pre-Cautions" for the API. Wait about 5 Minutes and then click "Yes" and see what happens now. There are about 50x times the same order and they all get executed. So this means while TWS was waiting for input, Multicharts didn´t get a fill confirmation and resent the order over and over. This is the same behavior as when TWS is "hanging" for a few minutes, all the orders then get executed several times, but with the "Pre-Cautions"-option it is reproduceable the easiest way.

brendanh
Posts: 154
Joined: 07 Apr 2007

Postby brendanh » 31 Jul 2009

I also autotrade forex via IB and have had overfills. I'm on v5.5 beta 2 and use a 5 minute time frame. I was short 50,000 on Monday morning, and my signal produced a reverse long trade. Instead of a 100,000 buy, Multicharts sent two orders for 150,000(!) to IB to put me long 250,000 instead of 50,000.

I have seen wrong reversal amounts in the past, so I don’t think this problem is exclusive to 5.5 beta 2.


We tried to reproduce it, but failed. We used different symbols for charting and execution.
If it is reproducible please send us your WSP, signals (if possible) and logs after the issue happens.


I use the same IB symbol for charting and trading. However, both the bid and ask price series, perhaps this is the difference with your test environment? I had another overfill today, this time just an exit, not a reverse. I was long 50,000, and MC had me exit 50,000 twice, the duplicate orders were within half a pip of each other. I had this problem when I first started autotrading using TS data. We traced the cause to TS data for the forex symbol having higher precision than IB. Switching to IB data solved.

I suspect my problem now is to do with using both bid & ask data. I started using both to solve chart\real world seperation caused by signals acting on bid series only, but IB filling at the ask, so there was a 2-3 pip area that would cause desyncronization:

http://forum.tssupport.com/viewtopic.php?t=6323

Looking forward to the tracking center as this will make troubleshooting autotrading problems much easier.

geektrader
Posts: 100
Joined: 17 Jul 2009
Location: Germany

Postby geektrader » 31 Jul 2009

I am also using Bid/Ask for the Auto-trading, if that might be the cause...

janus
Posts: 747
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 83 times

Postby janus » 31 Jul 2009

It can easily be reproduced this way for example: Turn on "Pre-Cautions" within TWS and also enabled those for the API. Then run a strategy and once it sends it first order, TWS will ask if you want to turn off "Pre-Cautions" for the API. Wait about 5 Minutes and then click "Yes" and see what happens now. There are about 50x times the same order and they all get executed. So this means while TWS was waiting for input, Multicharts didn´t get a fill confirmation and resent the order over and over. This is the same behavior as when TWS is "hanging" for a few minutes, all the orders then get executed several times, but with the "Pre-Cautions"-option it is reproduceable the easiest way.


Now that's a major issue, not necessarily a bug. I hope something is done about it soon because I often get at least one very short period every day where there's a connection loss. It may last only for a few seconds to a minute or so but if it happened at the same time MC submits an order then I may get the same problem. Perhaps it's an issue with IB not sending back an appropriate message to MC that the order was accepted by TWS but was not yet submitted to the exchange due to a momentary connection loss. There needs to be a more comprehensive handshaking procedure between TWS and MC. If this is not possible because IB can't provide such information back to MC then I'm afraid the only other way is to assume the order was sent to the exchange. Not sure if this is appropriate as it might lead to other issues. However, I rather MC did this than submit multiple orders. It all depends on what condition MC decides to re-send the orders. Is it because it didn't get any response back from TWS? If so, then is it the same condition if TWS was not running at all? If these two cases are distinguishable (which should be possible) then I'd rather MC not post multiple orders as long as it knows TWS is running even though it didn't respond back. Of course, the best way is for MC to know what TWS knows, namely when there's a connection loss or a wait for a response by the user to confirm the order through IB's window interface. Then MC can take the appropriate course of action.

SUPER
Posts: 622
Joined: 03 Mar 2007
Has thanked: 99 times
Been thanked: 80 times

Postby SUPER » 02 Aug 2009

janus,

Will use of function GetAppInfo(aiRealTimeCalc) in your code help to establish if there is any break in data feed from IB, just a wild thought.

Defination of function reads: will return a value of 1 if the calling application’s calculations are based on real-time data; otherwise, will return a value of 0

May be we need another function something along the lines: GetAppInfo(BrokerConnection) to return 1 if connected or 0 if disconnected.

Regards
Super

janus
Posts: 747
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 83 times

Postby janus » 02 Aug 2009

SUPER, not sure if the first idea would work. Your second idea could be useful. The study could first check if the connection is working and if it is then and only does the study execute the strategy orders. Will need to override these somehow when doing things like optimization (eg, MC simply assumes the connection is OK). Still, this is not 100% right. It's possible, no matter how unlikely that after receiving confirmation the connection is OK, it goes offline the instant the order is sent by the study. Then we have the same problem as before. The only sure way is to make the handshaking between IB and MC more intelligent as I described, if possible.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 03 Aug 2009

It can easily be reproduced this way for example: Turn on "Pre-Cautions" within TWS and also enabled those for the API. Then run a strategy and once it sends it first order, TWS will ask if you want to turn off "Pre-Cautions" for the API. Wait about 5 Minutes and then click "Yes" and see what happens now. There are about 50x times the same order and they all get executed. So this means while TWS was waiting for input, Multicharts didn´t get a fill confirmation and resent the order over and over. This is the same behavior as when TWS is "hanging" for a few minutes, all the orders then get executed several times, but with the "Pre-Cautions"-option it is reproduceable the easiest way.



We can’t reproduce the same issue on our side.

Environment:
- TWS Build 895.3
- MC 5.5 version
- wsp: AAT, 0 sec. replace to market for both; Channel Breakout LE + SE, Pyr - off, 10000 contracts; DataFeed - Interactive Brokers, 2 Symbols EUR.USD resolution 2 seconds, Bid & Ask based strategy calculating.
d
Case:
1) Delete TWS settings TWS (delete folder %ProgramFiles%/Jts/darykq for "edemo" login)
2) Run TWS, login.
3) Go to Main menu, Configure->API->All API Settings. Uncheck "Bypass Order Precautions for API Orders".

4) Go to main menu, Configure->Presets->Forex->EUR->Precautionary Settings. Ñòàâèì Size Limit 1000.
Click ÎÊ.
5) Run ÌÑ, open wsp
6) Enable AT
7) As soon as new bar arrives, new orders will be sent to TWS and we will see TWS message with a question about Order Precautions for API Orders.
Wait several dozens of seconds and click Yes.
8) TWS terminal will show lots of orders in ‘New’ state (they are not transmitted to exchange, but in TWS)+ a couple of price orders sent to the broker.
6) The market position is equal on MultiCharts(the chart market position) and TWS and there is no N times difference.
Your suggestions how to reproduce it are welcome.
Thank you

geektrader
Posts: 100
Joined: 17 Jul 2009
Location: Germany

Postby geektrader » 29 Oct 2009

Long time no reply to this thread, but I see the bug hasn´t been fixed. Today there was a typical situation which shows this bug again.

My automated strategy was running and submitted a "buy 80000" order at 11:15:28, then a Profit-Target (Sell 80000) order at 11:19.38. The Sell order didn´t get filled for about 10 seconds as I saw, then Multicharts just resubmitted the sell order again automatically, altough my system didn´t show any additional signals.

After a few more seconds, TWS filled both sell orders, pushing the position at the broker to Short 80000, altough it should be actually Flat. This is also what the Strategy in Multicharts said, that it is "Flat" now, altough the position at broker is now short 80000, through the re-submission of the same order twice.

That´s the same problem as discussed above in the thread again.

I hope this gets fixed somewhen soon as I can´t let my automated system run unattended because of this.

Just for info: I am trading EURUSD through IB via Multicharts 5.5 in Sync-Mode.

User avatar
TJ
Posts: 6518
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Postby TJ » 29 Oct 2009

this is serious problem.

you should document bugs like this, even though you might think this is the same bug, or that MC knows about it already....

I would record the following:

1. OS version
2. MC version
3. TWS version
4. TWS log
5. MC log
6. Timg and Sales
7. strategy settings, e.g. IOG, multiple orders, sync/async, etc.

.
Last edited by TJ on 01 Nov 2009, edited 1 time in total.

marekj
Posts: 51
Joined: 07 Aug 2009

Postby marekj » 30 Oct 2009

From my knowledge about current Mutlicharts bugs, situation described be geektrader CAN happen. Every time, when one of the open orders is modified (partially filled, filled, cancel, priced of the order changes) Multicharts will cancel all other orders (and resubmit them, what is completely wrong; imagine this situation if you trade manually from the dome!)

Now, add to it IB late trade reporting (usually delayed depends of instrument) and disaster is ready.

TWS log files can proof it easily. I try to push Multicharts team as hard as possible to correct this situation (no order should be cancel if other order is modified), but so far I cannot hear any commitment or time frame.

geektrader
Posts: 100
Joined: 17 Jul 2009
Location: Germany

Postby geektrader » 31 Oct 2009

That is to bad that they don´t take such a serious bug seriously. Gonna have to look for another trading-software if nothings changes soon...

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 02 Nov 2009

That is to bad that they don´t take such a serious bug seriously. Gonna have to look for another trading-software if nothings changes soon...


Geektrader,
The statement “That is to bad that they don´t take such a serious bug seriously.” looks at least strange considering the fact that on Mon Aug 03, 2009 we replied with the exact steps what we did to reproduce the problem and failed. You haven’t responded for several months after that you say that we don’t take it seriously!
We analyze and fix each and every issue we can reproduce. It looks you don’t want to cooperate or this issue is not important for you.
If you still want to get it fixed helps us and provides information what we did wrong.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 02 Nov 2009

From my knowledge about current Mutlicharts bugs, situation described be geektrader CAN happen. Every time, when one of the open orders is modified (partially filled, filled, cancel, priced of the order changes) Multicharts will cancel all other orders (and resubmit them, what is completely wrong; imagine this situation if you trade manually from the dome!)

Now, add to it IB late trade reporting (usually delayed depends of instrument) and disaster is ready.

TWS log files can proof it easily. I try to push Multicharts team as hard as possible to correct this situation (no order should be cancel if other order is modified), but so far I cannot hear any commitment or time frame.


Marekj,
The issue you are referring to is not related to this topic. What is the reason to put the comments to this thread? It is misleading other customers…
Your issue is not a bug. This is the way the software places orders. It could be critical in your case to have the orders modified, but it is not critical for hundreds of other strategies. This is why we have the feature in the to-do list.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 02 Nov 2009

That is to bad that they don´t take such a serious bug seriously. Gonna have to look for another trading-software if nothings changes soon...


Our assumption is that this error is rare since IB is quote robust, but sometimes it happens. It is possible it returns us that the order is canceled, but actually it is still pending. It is just a guess.
This is why we need either steps to reproduce (if it is an obvious error) or logs if the issue is rare.

TraderJ
Posts: 37
Joined: 14 Jul 2009
Location: Germany

A suggestion to fix the bug

Postby TraderJ » 06 Feb 2010

After quite some time I would like to report back with some new information. First of all, I would like to thank the MultiCharts team for creating such a powerful and easy to use software! MultiCharts is by far the most powerful backtesting platform I could find. Above all of its really strong features, its feature to use BID and ASK data for precise backtesting is simply a killer feature. No other platform offers that. Of course I still have a wishlist - one of my whishes is: Instead of the strange and limited Powerlanguage, I would like to use a modern high level language like C# (or Java) for clean state of the art programming. Code reuse is difficult with Powerlanguage. The best I found was to copy code which is not good.

Back to the topic - autotrading problems:
Because I needed a solution, I moved my autotrading from MultiCharts to a platform which supports low level programming. This platform is called OpenQuant (uses C#) and is ultimately powerful for automated trade execution, so I would love to see its features in MultiCharts. I found out in my early attempts that the new platform had similar problems as MultiCharts. I found out that these problems arose from the fact that IB tends to send events as a burst. This means an event such as "Order Filled" is often followed by a bunch of other events almost immediately after the first event. An example might be "Order Status Changed", then "Order Filled", then "Order Done" and so on. This behaviour of IB TWS can be dealt with easily by simply waiting 2 seconds after receiving an event before taking any actions. Within this delay, all related events have been fired by IB TWS. Now it is safe to look at the new situation (open position size, open orders and their sizes etc.) and to take new actions.

Another problem that I found is that IB does not report a position originating from a partial fill as a position. So you must look at the remaining size of all open orders to calculate the current position size.

After dealing with these two specialities of IB TWS, I found my autotrading with OpenQuant very stable. I am using scalping strategies which generate lots of trades in a single day.

I hope that I have helped you to understand the problems with autotrading on IB TWS so you can improve autotrading with MultiCharts.

Feel free to contact me if you have any questions. I would be glad to help.

janus
Posts: 747
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 83 times

Re: A suggestion to fix the bug

Postby janus » 06 Feb 2010

TJ you make many valid comments. I have mixed feelings with EL. It's good for basic stuff but when my code becomes really complex, it has many major drawbacks. That's one reason I now write most of my code externally in another language. I have used VC++ and VB.Net but now use FB as it's more than enough for my purposes as it has most of features of these other languages yet easier to use. I'm actually a FORTRAN guy from way back but FB is great for me. I don't have a requirement for complex numbers (which is what FORTRAN is useful for primarily). That's where some other packages like NT and AB are better for sophisticated programmers as they use C++ or C# like coding built-in. However, I think the vast majority of traders don't need such sophistication and is overkill, which is why EL is good too. So, I don't know what the final answer is. I doubt very much MC will ever dump EL as there are too many users already. I think the DLL approach I use is a good enough compromise. I now have windows gui add-ons to MC. For example, I can manually buy/sell/cover/block orders with point and click - something I think that MC said they will provide in the future. It allows me to override or supplement the auto-traded orders. So, once you start writing stuff in DDL you can do pretty much anything. I like to get into the habit of using EL just to plot signals and execute and manage orders. I can understand how there may be some issues but so far I can overcome them using global variables.

What I like to see is the ability to "kick start" a study from a DLL so that I can pass on a market order immediately without waiting for the next tick update.

As you and I indicated elsewhere the way MC manages orders is unusual, and to be honest unnecessarily complex. A simple buy/sell is sufficient without any restrictions on past history. I won't go into details as I've already explained it elsewhere. I just mention it here as it would be a major breakthrough if MC does make the order management process simpler and more aligned with how virtually all traders trade manually, and how most other systems do it. Then it would be a simpler take to do what NT does in terms of scaling in/out and different pyramiding strategies.


Return to “MultiCharts”