Anyone doing preprocessing?
Hello,
I have some calculations that happen on every bar regardless of Inputs for my Signal. Other than exporting data in an indicator then importing it as a instrument. Is there any way to preprocess data to improve optimization speed?
Pre-Processing
-
TJ
- Posts: 7742
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2222 times
Re: Pre-Processing
Yes, put it in a variable. Put it in a function.
ps. If you need coding help, you need to post your codes.
ps. If you need coding help, you need to post your codes.
Re: Pre-Processing
Hey TJ, Thanks as always.Yes, put it in a variable. Put it in a function.
ps. If you need coding help, you need to post your codes.
Lets say for instance I am calculating an array of moving averages. No matter how I optimize my signal I will still calculate the next bar of the entire array. For this example lets assume the MA array is already in a function. and I am filling my Array by passing it into the function.
Code: Select all
Array:
Averages_Array[100](0);
Vars:
Price_value(0);
Price_value = C;
Average_Array_Function(Price_value, Averages_Array)
Inputs:
Average_1(0),
Average_2(0),
Average_3(0),
Average_4(0);
Vars:
Condition_1(0),
Condition_2(0);
Condition_1 = Averages_Array[Average_1] > Averages_Array[Average_2];
Condition_2 = Averages_Array[Average_3] > Averages_Array[Average_4];
If condition_1 then buy next bar at open;
If condition_2 then sell next bar at open;
Do you have a suggestion of how to preprocess this Moving average array to speed up optimization?
-
TJ
- Posts: 7742
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2222 times
Re: Pre-Processing
Buy a faster CPU ?Do you have a suggestion of how to preprocess this Moving average array to speed up optimization?
Re: Pre-Processing
So apart from exporting the MAs then importing them as instruments there isnt a way I could pre process something that will be the exact same calculation on every version of the optimization?Buy a faster CPU ?Do you have a suggestion of how to preprocess this Moving average array to speed up optimization?
-
ABC
- Posts: 718
- Joined: 16 Dec 2006
- Location: www.abctradinggroup.com
- Has thanked: 125 times
- Been thanked: 408 times
- Contact:
Re: Pre-Processing
joebone,
one idea could be to compute everything once and store the results in Maps/Lists using EasyLanguage Collections. Then you can access these values during your optimization. If this really improves your performance will depend on the amount of values you need to access each bar, so I would try to minimize this as much as possible.
Additionally, you will likely have to perform these calculations and storing before your actual optimization and since the optimization will likely be spread across different CPU cores, sharing the Maps/Lists might not work and you would have to read the values in from a text file at the beginning of every optimization iteration.
Regards,
ABC
one idea could be to compute everything once and store the results in Maps/Lists using EasyLanguage Collections. Then you can access these values during your optimization. If this really improves your performance will depend on the amount of values you need to access each bar, so I would try to minimize this as much as possible.
Additionally, you will likely have to perform these calculations and storing before your actual optimization and since the optimization will likely be spread across different CPU cores, sharing the Maps/Lists might not work and you would have to read the values in from a text file at the beginning of every optimization iteration.
Regards,
ABC
-
ABC
- Posts: 718
- Joined: 16 Dec 2006
- Location: www.abctradinggroup.com
- Has thanked: 125 times
- Been thanked: 408 times
- Contact:
Re: Pre-Processing
joebone,
a more practical question would be why you compute an array with 100 averages when each strategy only uses four? Computing just the four averages you need per strategy might already show a decent speed improvement (I realize that you might only post a generic example here, so this approach might not be beneficial for your actual use case).
Regards,
ABC
a more practical question would be why you compute an array with 100 averages when each strategy only uses four? Computing just the four averages you need per strategy might already show a decent speed improvement (I realize that you might only post a generic example here, so this approach might not be beneficial for your actual use case).
Regards,
ABC
Re: Pre-Processing
Yes, this was just an example. The thought came to me while I was reading about "Pre-cooking" data for image processing. I was just seeing if anyone was doing something like this here.joebone,
a more practical question would be why you compute an array with 100 averages when each strategy only uses four? Computing just the four averages you need per strategy might already show a decent speed improvement (I realize that you might only post a generic example here, so this approach might not be beneficial for your actual use case).
Regards,
ABC
Another example could be a bunch of complex convolutions or a cross correlation matrix. Anything really that you would have no problem calculating on live bars trading but when doing an optimization they become cumbersome because you recalculate them every time you run an optimization and the calculation may be exactly the same each time.
You point remains valid that I should focus on reducing unnecessary calculations though.
-
ABC
- Posts: 718
- Joined: 16 Dec 2006
- Location: www.abctradinggroup.com
- Has thanked: 125 times
- Been thanked: 408 times
- Contact:
Re: Pre-Processing
joebone,
a good approach might be to test this and measure the performance difference. When things become complex even reading in text files with the results can give you a speed advantage compared to performing the calculations every time.
Regards,
ABC
a good approach might be to test this and measure the performance difference. When things become complex even reading in text files with the results can give you a speed advantage compared to performing the calculations every time.
Regards,
ABC
Re: Pre-Processing
joebone,
a good approach might be to test this and measure the performance difference. When things become complex even reading in text files with the results can give you a speed advantage compared to performing the calculations every time.
Regards,
ABC
Thanks for the help on thinking about this one ABC
