Link multichart signal to python classifier  [SOLVED]

Questions about MultiCharts and user contributed studies.
mickatrade
Posts: 114
Joined: 12 Jul 2015
Has thanked: 20 times
Been thanked: 16 times

Link multichart signal to python classifier

Postby mickatrade » 24 Aug 2019

Hello,

I develop strategies which are traded with portfolio trader (no intrabar order generation)
I would like to link my algos to a python machine learning classifier.
The process would be this one.

1/ Entry conditions are met
2/ The strategy send an array of values to the python program
3/ the python program convert the array to an image
4/ the image is evaluated by the model classifier
5/ the classifier return to the strategy the tag (good or bad) of the image
6/ based on the tag value, the entry is allowed or not.

The hard part for me is to make communication between multicharts and python. i really dont know how to handle it.
Hope you can help.
Thx


MJ

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Link multichart signal to python classifier

Postby hughesfleming » 26 Aug 2019

Hello MJ,

You can try sending your array to sharedvar server. You can then read your array into python calling the dll using python ctypes. I have done this and it works. It is probably faster to write the result to a text file and read it . This way you can also back test. You can call your code at end of each bar using Lua.

Good luck with your project.

Alex

mickatrade
Posts: 114
Joined: 12 Jul 2015
Has thanked: 20 times
Been thanked: 16 times

Re: Link multichart signal to python classifier

Postby mickatrade » 26 Aug 2019

Hi,
Thank you for your reply.
I didn't know sharedvar, i'll give it a try.
so far instead of sending data from MC to python, i'm writing a csv file.
What i dont how to do is to trigger the python script to open it and send an answer (writing a new csv file).
An other concern is , with all the process outside MC ( file opening, model query , file writing), is there a way to make MC script to wait until it receive the answer to open or not the trade.

Regards,
MJ

User avatar
Smoky
Posts: 507
Joined: 03 Dec 2010
Location: Thailand
Has thanked: 97 times
Been thanked: 115 times

Re: Link multichart signal to python classifier

Postby Smoky » 26 Aug 2019

https://fx1.net/sharedvar.php

there is some post here about this wonderful tool

mickatrade
Posts: 114
Joined: 12 Jul 2015
Has thanked: 20 times
Been thanked: 16 times

Re: Link multichart signal to python classifier

Postby mickatrade » 27 Aug 2019

Hello MJ,

You can try sending your array to sharedvar server. You can then read your array into python calling the dll using python ctypes. I have done this and it works. It is probably faster to write the result to a text file and read it . This way you can also back test. You can call your code at end of each bar using Lua.

Good luck with your project.

Alex
Hi Alex,

i'm performing test of sharedvar.
I can send MC data to the sharedvar, but for some reason, i can read them with python . it seems the handle value send by MC is incorect.
so i cant read svgetBool, svgetDouble, svgetInt, svgetString.
regards,
MJ
Attachments
sharedvar.png
(202.69 KiB) Downloaded 432 times

User avatar
Smoky
Posts: 507
Joined: 03 Dec 2010
Location: Thailand
Has thanked: 97 times
Been thanked: 115 times

Re: Link multichart signal to python classifier

Postby Smoky » 27 Aug 2019

Can't read your MC code post it with tag.

but , first you can controle your writing with sharevar server ...

mickatrade
Posts: 114
Joined: 12 Jul 2015
Has thanked: 20 times
Been thanked: 16 times

Re: Link multichart signal to python classifier

Postby mickatrade » 27 Aug 2019

Hi smoky,

You can find the code below.
I can see the values on the sharedvar monitor application, but i can't get all them when i query the server from python.
I cant understand why, but it seems handle int value returned is incorrect when used in the python.

Code: Select all

external: "SharedVar-a64.dll", int, "svInit" ,string;
external: "SharedVar-a64.dll", bool, "svServerRunning";
external: "SharedVar-a64.dll", bool, "svSetString" ,int,string,string;
external: "SharedVar-a64.dll", string, "svGetString",int,string;
external: "SharedVar-a64.dll", string, "svVersion";
external: "SharedVar-a64.dll", bool, "svSetValue",int,string,double;
external: "SharedVar-a64.dll", int, "svRealmCount";

var : myserv(false),intrabarpersist initval(-1),strbool(False),varstr(""),db(False),intrealm(0);

once begin
cleardebug;
initval = svInit("MC-" +getsymbolname);
end;


myserv = svServerRunning();
strbool = svSetString(initval,"Close","Test sharedvar");
db = svSetValue(initval,"MCChart handle value",initval);
intrealm = svRealmCount();

print(initval);
print("Handle value for " + getsymbolname + " : " + NumToStr(initval,0));

plot1(initval,"graph");

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Link multichart signal to python classifier

Postby hughesfleming » 27 Aug 2019

Hi Mj,

I will try and post a simple example later today. It has been a while since I looked at this. You are probably running into an issue with C types and Python types.

regards,

Alex

mickatrade
Posts: 114
Joined: 12 Jul 2015
Has thanked: 20 times
Been thanked: 16 times

Re: Link multichart signal to python classifier

Postby mickatrade » 31 Aug 2019

Hi Mj,

I will try and post a simple example later today. It has been a while since I looked at this. You are probably running into an issue with C types and Python types.

regards,

Alex
Hi,
I 've 2 issues with SharedVar app.

MC side (sharedvar-a64.dll):
I ve converted all dll function to powerlanguage function. the only one that wont work as expected is svGetString.
Inside a study i use the svSetString function, i can see the value on sharedvar app, i immediately (next code line) call svGetString for the same variable and get weird characters.

Code: Select all

//********print**********
Get string from variable named Close :Ðp).
Get string from variable named Un nom :Ðp).
Python side (jupyter notebook sharedvar-w64.dll)
While mc study is running, i run my notebook which create a Realm name : "Python reader".
When i loop through Realms value, i get MC realm and Python realm names =>Ok
But when want to see the variable inside each realm, i'm only get python realm variable names. => Not Ok
python notebook.png
(13.89 KiB) Downloaded 356 times
I'm stuck .
Hope you can help.

MJ

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Link multichart signal to python classifier  [SOLVED]

Postby hughesfleming » 31 Aug 2019

Hi MJ,

Hopefully this will help. I am running this Python code inside of Rapidminer as I prefer to use that for many machine learning tasks especially in the early phase of testing ideas. This requires that I use dataframes but the code should be straight forward.

In this example, I am exporting six data points from Multicharts and generating an ID and datetime stamp.

Code: Select all

//Generate Realms
OutputDateString = FormatDate(OutputDateFormat, ELDateToDateTime(Date));
bartime = FormatTime("HH:mm:ss", DateTime );



once begin



BarID = svinit("ID_"+NumtoStr(barinterval,0));
VOpenLink = svinit("VOpen_"+NumtoStr(barinterval,0));
VHighLink = svInit("VHigh_"+NumtoStr(barinterval,0));
VLowLink = svInit("VLow_"+NumtoStr(barinterval,0));
VCloseLink = svInit("VClose_"+NumtoStr(barinterval,0));
SSHighLink = svInit("VSSHigh_"+NumtoStr(barinterval,0));
SSLowLink = svInit("VSSLow_"+NumtoStr(barinterval,0));


svRemoveAll(BarID);
svRemoveAll(Closelink);
svRemoveAll(VOpenLink);
svRemoveAll(VHighLink);
svRemoveAll(VLowLink);
svRemoveAll(VCloseLink);
svRemoveALl(SSHighLink);
svRemoveAll(SSLowLink);



end;







if barstatus(0) = 2 then
begin
svSetValue(BarID,OutputDateString+" "+bartime,barnumber);
svSetValue(VOpenLink,OutputDateString+" "+bartime,VOpen);
svSetValue(VHighLink,OutputDateString+" "+bartime,VHigh);
svSetValue(VLowLink,OutputDateString+" "+bartime,VLow);
svSetValue(VCloseLink,OutputDateString+" "+bartime,VCLose);
svSetValue(SSHighLink,OutputDateString+" "+bartime,SSHigh);
svSetValue(SSLowLink,OutputDateString+" "+bartime,SSLow);
end;
On the Python side, the code looks like this....

Code: Select all

import pandas as pd
from ctypes import *

def rm_main():
lib = cdll.LoadLibrary("SharedVar-w64.dll")
linkID =(lib.svInit("ID_65"))
link1 =(lib.svInit("VOpen_65"))
link2 =(lib.svInit("VHigh_65"))
link3 =(lib.svInit("VLow_65"))
link4 =(lib.svInit("VClose_65"))
link5 =(lib.svInit("VSSHigh_65"))
link6 =(lib.svInit("VSSLow_65"))


lib.svGetDouble.restype = c_double
lib.svVarName.restype = c_wchar_p
MaxBars = lib.svVarCount(linkID)

data=[]

for x in range(1,MaxBars):
BarDateTime = lib.svVarName(linkID, x)
varID = lib.svGetInt(linkID,BarDateTime)
varValue1 = lib.svGetDouble(link1,BarDateTime)
varValue2 = lib.svGetDouble(link2,BarDateTime)
varValue3 = lib.svGetDouble(link3,BarDateTime)
varValue4 = lib.svGetDouble(link4,BarDateTime)
varValue5 = lib.svGetDouble(link5,BarDateTime)
varValue6 = lib.svGetDouble(link6,BarDateTime)



data.append((varID,BarDateTime,varValue1,varValue2,varValue3,varValue4,varValue5,varValue6))

cols = ['ID','DateTime','VOpen','VHigh','VLow','VClose','SSHigh','SSLow']
df = pd.DataFrame(data, columns=cols)
df = df.sort_values(['ID'], ascending=True)
return(df)
The only tricky part is this part which is dealing with the differences between python types and c types. This probably where you are having your problem.

Code: Select all

lib.svGetDouble.restype = c_double
lib.svVarName.restype = c_wchar_p
The end result is a nicely formatted table ready for machine learning.
DataTable.JPG
(125.85 KiB) Downloaded 343 times
I hope this helps.

Regards,

Alex

mickatrade
Posts: 114
Joined: 12 Jul 2015
Has thanked: 20 times
Been thanked: 16 times

Re: Link multichart signal to python classifier

Postby mickatrade » 31 Aug 2019

Hi MJ,

Hopefully this will help. I am running this Python code inside of Rapidminer as I prefer to use that for many machine learning tasks especially in the early phase of testing ideas. This requires that I use dataframes but the code should be straight forward.

In this example, I am exporting six data points from Multicharts and generating an ID and datetime stamp.

Code: Select all

//Generate Realms
OutputDateString = FormatDate(OutputDateFormat, ELDateToDateTime(Date));
bartime = FormatTime("HH:mm:ss", DateTime );




once begin



BarID = svinit("ID_"+NumtoStr(barinterval,0));
VOpenLink = svinit("VOpen_"+NumtoStr(barinterval,0));
VHighLink = svInit("VHigh_"+NumtoStr(barinterval,0));
VLowLink = svInit("VLow_"+NumtoStr(barinterval,0));
VCloseLink = svInit("VClose_"+NumtoStr(barinterval,0));
SSHighLink = svInit("VSSHigh_"+NumtoStr(barinterval,0));
SSLowLink = svInit("VSSLow_"+NumtoStr(barinterval,0));


svRemoveAll(BarID);
svRemoveAll(Closelink);
svRemoveAll(VOpenLink);
svRemoveAll(VHighLink);
svRemoveAll(VLowLink);
svRemoveAll(VCloseLink);
svRemoveALl(SSHighLink);
svRemoveAll(SSLowLink);



end;







if barstatus(0) = 2 then
begin
svSetValue(BarID,OutputDateString+" "+bartime,barnumber);
svSetValue(VOpenLink,OutputDateString+" "+bartime,VOpen);
svSetValue(VHighLink,OutputDateString+" "+bartime,VHigh);
svSetValue(VLowLink,OutputDateString+" "+bartime,VLow);
svSetValue(VCloseLink,OutputDateString+" "+bartime,VCLose);
svSetValue(SSHighLink,OutputDateString+" "+bartime,SSHigh);
svSetValue(SSLowLink,OutputDateString+" "+bartime,SSLow);
end;
On the Python side, the code looks like this....

Code: Select all

import pandas as pd
from ctypes import *

def rm_main():
lib = cdll.LoadLibrary("SharedVar-w64.dll")
linkID =(lib.svInit("ID_65"))
link1 =(lib.svInit("VOpen_65"))
link2 =(lib.svInit("VHigh_65"))
link3 =(lib.svInit("VLow_65"))
link4 =(lib.svInit("VClose_65"))
link5 =(lib.svInit("VSSHigh_65"))
link6 =(lib.svInit("VSSLow_65"))


lib.svGetDouble.restype = c_double
lib.svVarName.restype = c_wchar_p
MaxBars = lib.svVarCount(linkID)

data=[]

for x in range(1,MaxBars):
BarDateTime = lib.svVarName(linkID, x)
varID = lib.svGetInt(linkID,BarDateTime)
varValue1 = lib.svGetDouble(link1,BarDateTime)
varValue2 = lib.svGetDouble(link2,BarDateTime)
varValue3 = lib.svGetDouble(link3,BarDateTime)
varValue4 = lib.svGetDouble(link4,BarDateTime)
varValue5 = lib.svGetDouble(link5,BarDateTime)
varValue6 = lib.svGetDouble(link6,BarDateTime)



data.append((varID,BarDateTime,varValue1,varValue2,varValue3,varValue4,varValue5,varValue6))

cols = ['ID','DateTime','VOpen','VHigh','VLow','VClose','SSHigh','SSLow']
df = pd.DataFrame(data, columns=cols)
df = df.sort_values(['ID'], ascending=True)
return(df)
The only tricky part is this part which is dealing with the differences between python types and c types. This probably where you are having your problem.

Code: Select all

lib.svGetDouble.restype = c_double
lib.svVarName.restype = c_wchar_p
The end result is a nicely formatted table ready for machine learning.

DataTable.JPG

I hope this helps.

Regards,

Alex
Hi Alex,

Thanks for your reply.
With your code, i understand what i wasn't doing right.
I didn't know i had to init the realm on both MC and Python with the same string parameter. Now it works like a charm.
Thank you again for your great help.
Now i can connect MC to python using SharedVar or Lua dll and it's great.
BTW, contacted fx1 about the dvGetString function for the Sharedvar-a64.dll et they confirmed the issue.

Regards,

MJ

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Link multichart signal to python classifier

Postby hughesfleming » 01 Sep 2019

Great MJ, I am glad that you got it working. Enjoy the rest of your weekend. Give Rapidminer a try. It is especially useful for validating your time series models. Once you are done with your testing, you can then call your python code directly. It is very easy use and is a huge time saver.

Regards,

Alex

mickatrade
Posts: 114
Joined: 12 Jul 2015
Has thanked: 20 times
Been thanked: 16 times

Re: Link multichart signal to python classifier

Postby mickatrade » 01 Sep 2019

Hi,
i'll give Rapidminer a try, this tool looks great.
Regards,
MJ

wilkinsw
Posts: 662
Joined: 21 Apr 2013
Has thanked: 154 times
Been thanked: 104 times

Re: Link multichart signal to python classifier

Postby wilkinsw » 18 Feb 2020

Very interesting thread.

Are there any free alternatives to rapidminer people are using?

hughesfleming
Posts: 275
Joined: 22 Apr 2014
Has thanked: 70 times
Been thanked: 72 times

Re: Link multichart signal to python classifier

Postby hughesfleming » 18 Feb 2020

Hi @wilkinsw,

After 30 days, the trial turns into a community edition. A limit of 10,000 rows and single threaded. It is still extremely capable. You can also run R and Python within your process, deal with databases etc. It is an amazing tool. Single threaded is not the limitation that you might think it is. Many machine learning operators like neural nets lack repeatability when multithread. That is not what you want when back testing so you would have to switch it off anyway.

regards,

Alex

User avatar
Smoky
Posts: 507
Joined: 03 Dec 2010
Location: Thailand
Has thanked: 97 times
Been thanked: 115 times

Re: Link multichart signal to python classifier

Postby Smoky » 12 Oct 2022

your python code doesnt work on my side, python version ?


Return to “MultiCharts”