Symbol mapping between IQFeed and IB

Questions about MultiCharts and user contributed studies.
joe107c
Posts: 12
Joined: 17 Jul 2013
Has thanked: 2 times

Symbol mapping between IQFeed and IB

Postby joe107c » 03 Feb 2022

hi,
I use IQFeed as my data source and IB as my broker.
Below is my setting.
I turn on SA and my orders were rejected, and there are the errors from IB.


Is there something wrong with my settings?
Attachments
p5.png
(57.47 KiB) Not downloaded yet
p4.png
(60.4 KiB) Not downloaded yet
p3.png
(76.62 KiB) Not downloaded yet
p2.png
(33.08 KiB) Not downloaded yet
p1.png
(90.63 KiB) Not downloaded yet

joe107c
Posts: 12
Joined: 17 Jul 2013
Has thanked: 2 times

Re: Symbol mapping between IQFeed and IB

Postby joe107c » 03 Feb 2022

what does 'Expired cache for symbol TQQQ' means? and how to solve that problem?
Attachments
p6.png
(130.27 KiB) Not downloaded yet

User avatar
Kate MultiCharts
Posts: 575
Joined: 21 Oct 2020
Has thanked: 7 times
Been thanked: 144 times

Re: Symbol mapping between IQFeed and IB

Postby Kate MultiCharts » 03 Feb 2022

Hello joe107c,

Please try re-configuring the Symbol Mapping for the problematic symbols.
Find the detailed guide here.

If that doesn't help, please come to the live chat Monday-Friday from 5:30 AM till 1:00 PM ET so that we could connect to your PC remotely and check the issue.

PK1
Posts: 102
Joined: 12 Jun 2011
Has thanked: 42 times
Been thanked: 12 times

Re: Symbol mapping between IQFeed and IB

Postby PK1 » 18 Apr 2022

Hello,

I have the same issue with 3 of my custom futures, came out of nowhere. Since then at open of the workspace the notification shows up and on /ES even a few days data gap on the daily chart, which I never had and impossible to correct (I can only reload but MC is leaving out some days).

I've checked the link above but there was no solution how to deal with the problem "Expired cache for symbol ***. Remove and re-add the symbol mapping for this symbol".

What I did:

For /MCL I removed the symbol mapping, closed everything and re-added all future-contracts, including the expired ones. This re-adds the mapping. But the notification of the expired cache still appears, together with another red notification "No security definition has been found for the request". So as a result MC tells me to correct the mapping which is done by deleting it and MC was adding itself again (identical mapping) but still complains. So the problem is something else.

Would be great to know what the real cause for the expired-cache notification is.

AddOn:
It looks like the problem occurs with every custom contract I use for the futures and the problem is causing three such red notifications. Changing things in the Mappings-Dialogue triggers these notifications because MC is connecting IB and fails. One of the notification message shows invalid value in field #541 which might have to do with a wrong format of the expiration date of one futures contract. But the problem is with quite a few custom contracts, so it I can't to much here.

User avatar
Kate MultiCharts
Posts: 575
Joined: 21 Oct 2020
Has thanked: 7 times
Been thanked: 144 times

Re: Symbol mapping between IQFeed and IB

Postby Kate MultiCharts » 22 Apr 2022

Hello PK1,

Please try the following steps:
1. Close all the charts for the problematic symbols.
2. Re-add the symbols from the data source (both for your data feed and broker).
3. Go to File -> Broker Profiles -> Symbol Mapping -> IB -> Remove all the Symbol Mapping lines for problematic symbols.
4. Go to the main MultiCharts window -> Help -> Support -> Open Logs folder.
5. Close all MultiCharts components.
6. Delete DAT files starting with IB in the Logs folder (see step 4).
7. Reopen MultiCharts.
8. Configure Symbol Mapping for the required symbols.

PK1
Posts: 102
Joined: 12 Jun 2011
Has thanked: 42 times
Been thanked: 12 times

Re: Symbol mapping between IQFeed and IB

Postby PK1 » 31 May 2022

Hello PK1,

Please try the following steps:
1. Close all the charts for the problematic symbols.
2. Re-add the symbols from the data source (both for your data feed and broker).
3. Go to File -> Broker Profiles -> Symbol Mapping -> IB -> Remove all the Symbol Mapping lines for problematic symbols.
4. Go to the main MultiCharts window -> Help -> Support -> Open Logs folder.
5. Close all MultiCharts components.
6. Delete DAT files starting with IB in the Logs folder (see step 4).
7. Reopen MultiCharts.
8. Configure Symbol Mapping for the required symbols.
Hi Kate,
the hint with those dat-files was very good. I've tried that and MC is now also loading parts where the data had many gaps. But on top I had to reload the data too. But the actual problem seems to be fundamental when it comes to future-contracts overriding the contract-specifications from the same future 10 years ago -> ESH2 for 2012 vs ESH2 for 2022 as an example (same for the other contracts). Looking at the QM one can see that there's the old expiration von 2012, which is getting updated with re-adding the mapping. This also explains the many 541 messages I was mentioning from IB which MC gets when fetching data for ESH2 with the 10 year old expiration date.

So now the "contract-definition" for 2012 is lost, the 2012 data with its OHLC is still there. Overall I've saved ES data since around 2008/9 with MC and minute and "tick"-data is only available until 2018 more or less due to that missing handling of MC when it comes to overriding future-contracts. This is quite disappointing bc I'm doing much backtests with MC and R. So now I needed clean and to transfer all data to another database in order not to have more data loss.

So right now MC still tells me that my custom contracts for ES, NQ, CL and RTY have expired cache + a message that the individual contract (for example ESM2, where I re-added the mapping) has expired cache. All that after following your steps many times, so that's not solving the problem.

1. I wish I knew what else is necessary in order to get the contract-specifications in MC correct and getting rid of those red messages.
2. MC really needs a proper handling of those overriding contract future specifications. I regularly add all new futures, which has of course same notation like the 10 y old contract but other exp-date. Means, the correct handling of MC would be:
2.1. with adding a new future-contract which is already in the MC Database from 10y ago -> MC is UPDATING the contract-spec (especially the exp-date), hence afterwards MC won't receive those 541-error from TWS-API anymore.
2.2. Having a contract with data from now and 10ys ago MC should be able to handle that its actually not the same contract, rather a subcontract. I'm quite sure the ConId from IB is different here, so treating them as the same contract is wrong. A different ConId should always be treated differently even with same Symbol, that's how IB is doing it in its API. Otherwise fetching futures data via MC over many years has a deadly limit of consecutive years.

About 5ys ago MC-Support has changed some contract-symbols from 2008 and 2009 from ESx8 and ESx9 to ES08 and ES09 due to the same problem. This can't be a solution and in the end that data is lost for me too.

User avatar
Kate MultiCharts
Posts: 575
Joined: 21 Oct 2020
Has thanked: 7 times
Been thanked: 144 times

Re: Symbol mapping between IQFeed and IB

Postby Kate MultiCharts » 03 Jun 2022

Hi PK1,

MultiCharts uses ContID, but this field is modified by IB directly, and this is the very reason for the “Expired cache for symbol” message appearing during trading. To get rid of it, one needs to delete the DAT files and reconfigure symbol mapping as described in my previous post.

However, it seems like in your case there’s also an issue with Custom Futures plotting.

IB API supports futures contracts symbology with 1 year number (for example ESM2). That means that approximately 2 years after the futures contract’s expiration, it should be renamed (double-click the symbol -> change the Symbol Name on the Futures tab) or deleted (right-click -> Delete Symbol) in QuoteManager. For example, ESM8 should be renamed into ESM08 or ESM2008. It should be done manually. It is not safe for this process to be performed by MultiCharts automatically, and it’ll lead to workspace changes.

Manual renaming will allow plotting charts for the expired symbols until you delete them (or delete data for them). Contracts should be renamed when they stop appearing in the symbol list when adding symbols from data source (with the “Include Expired Contracts” box is enabled): QuoteManager -> Instrument -> Add Symbol -> From Data Source.

Here’s an example. ESM8 expired 06-2018. For some time when data for this symbol is requested, “No security definition…” messages will be returned by IB. But approximately in 2024 a futures contract with the exact same name will appear in the IB API symbol list, but it’ll be a 2028 contract now. That’s when the issue you reported might be reproduced.

So, you’ll need to leave as they are those symbols that IB provides data for and that are available in the IB symbol list. The old ones should be renamed so that there are 2 (preferably 4) numbers to mark the year. If you trade on Custom Futures, you’ll also need to re-configure symbol mapping after each rollover.


Return to “MultiCharts”