The Evaluation Shell (Zorro S)

The strategy evaluation shell is an addendum to the Zorro project. Its purpose is determining the ideal parameters and functions for a given trading strategy, generating a portfolio of optimal asset, algo, and timeframe combinations, and predicting its live perfomance under different market situations. The Zorro evaluation shell can make the best of a strategy, especially of 'indicator soup' strategies that combine many different indicators and analysis functions.

A robust trading strategy has to meet several criteria:

• It must exploit a real and significant market inefficiency. Random-walk markets cannot be algo traded.
• It must work in all market situations. A trend follower must survive a mean reverting regime.
• It must work under many different optimization settings and parameter ranges.
• It must be unaffected by random events and price fluctuations.

The shell evaluates parameter combinations by these criteria. The robustness under different market situations is determined through the R2 coefficient, the parameter range robustness with a WFO profile (aka Cluster Analysis), the price fluctuation robustness with oversampling. A Montecarlo analysis finds out whether the strategy is based on a real market inefficiency. More about WFO profiles and Montecarlo analysis can be found on financial-hacker.com/why-90-of-backtests-fail, more about oversampling on financial-hacker.com/better-tests-with-oversampling.

Zorro already offers functions for all these tests, but they require a large part of code in the strategy, more than for the algorithm itself. The evaluation shell skips the coding part. It can be simpy attached to a strategy script. It makes all strategy variables accessible in a panel, adds optimization and money management as well as support for multiple assets and algos, runs automated analysis processes, and builds the optimal portfolio of strategy variants. It compiles unmodified under lite-C and under C++, and supports .c as well as .cpp scripts.

With almost 1500 lines, the shell is probably the largest Zorro script so far, and goes far beyond other strategy evaluation software on the market. It comes with source code, so it can be easily modified and adapted to special needs. It is restricted to personal use; any commercial use or redistribution, also partially, requires explicit permission by oP group Germany. Since it creates a panel and a menu, it needs a Zorro S license to run. Theoretically you could remove the panel and menu functions and use the shell with the free Zorro version. This is allowed by the license, but would require clumsy workarounds, like calling functions by script and manually editing CSV files.

Overview

Developing a successful strategy is a many-step process, described in the Black Book and briefly in an article series on Financial Hacker. The evaluation shell cannot replace research and model selection. But it takes over when a first, raw version of the strategy is ready. At that stage you're usually experimenting with different functions for market detection and generating trading signals. It is difficult to find out which indicator or filter works best, since they are usually interdependent. Market detector A may work best with asset B and lowpass filter C on time frame D, but this may be the other way around with asset E. It is very time consuming to try out all combinations.

The evaluation shell solves that task with a semi-automated process.

Creeate  Jobs

Generate  Summary

Cluster  Analysis

Correlation  Analysis

Montecarlo  Analysis

Final  Portfolio

The first step is generating sets of parameter settings, named jobs. Any job is a variant of the strategy that you want to test and possibly include in the final portfolio. You can have parameters that select betwen different market detection algorithms, and others that select between different lowpass filters. The parameters are edited in the variables panel, then saved with a mouse click as a job. Instead of entering values in the panel, you can also edit the jobs directly with a spreadsheet program or a text editor.

The next step is an automated process that runs through all jobs, trains and tests any of them with different asset, algo, and time frame combinations, and stores their results in a summary. The summary is a CSV list with the main performance parameters of all jobs. It is sorted by performance: The best performing jobs are at the top. So you can see at a glance which parameter combinations work with which assets and time frames, and which are not worth to examine further. You can repeat this step with different global settings, such as bar period or optimization method, and generate multiple summaries in this way. 

The next step in the process is cluster analysis. Every job in a selected summary is optimized multiple times with different WFO settings. These settings are taken from - you guessed it - a separate CSV file that may contain a regular WFO matrix, a list of irregular cycles/datasplit combinations, or both. For reducing the process time, only profitable jobs with rising equity curves - determined by a nonzero R2 coefficent - get a cluster analysis. You can also further exclude jobs by removing or outcommenting them in the summary.

After this process, you likely ended up with a couple survivors in the top of the summary. The surviving jobs have all a positive return, a steady rising equity curve, shallow drawdowns, and robust parameter ranges since they passed the cluster analysis. But not all of them are suited for the final portfolio. The purpose of a strategy portfolio is diversifiction, but this would not work with a set of variants that are all tightly correlated and have their drawdowns all at the same time. You want a balanced portfolio with uncorrelated algorithms. Automatically reducing the portfolio to the combinations with the smallest correlation is planned for a future shell version, but at the moment it involves a manual selection. Check the equity curves and keep only the best of all variants with similar curves or the same assets, algos, and time frames. They can now be combined with a mouse click to the final balanced portfolio.

But you're not finished yet. Any selection process generates selection bias. Your perfect portfolio will likely produce a great backtest, but will it perform equally well in live trading? To find out, run a Montecarlo analysis. This is the most important test of all, since it can determine whether your strategy exploits a real market inefficiency. If the Montecarlo analysis fails with the final portfolio, it will likely also fail with any other parameter combination, so you need to run it only at the end. If your system passes Montecarlo with a p-value below 5%, you can be relatively confident that the system will return good and steady profit in live trading. Otherwise, back to the drawing board.
 

The following sections cover the various operating modes of the shell. For a quick example of an evaluation process, scroll down to the tutorial.

Step 1: Attaching the shell

You can attach the shell to an existing script in several ways. For the full process it must take control over all variables and replace its asset/algo loops But for adding cluster analysis, Montecarlo analysis, oversampling, various walk-forward optimization methods, and different training objectives, you only need to insert two lines. Include eval.h at the begin of the script:

#include "eval.h"

... and eval.c at the end:

#include "eval.c"

You can find a script example with attached shell in Workshop6a. When you start it, the [Result] button will change to [Start], a new menu will be available under [Action], and a panel with many parameter settings will pop up:

The panel and menu are the reason why a Zorro S license is required, since panel functions are not supported by the free version. The variables in the panel are mostly self explaining:

The settings have only effect when your script does not set those parameters itself. If it does, the script will override all settings in the panel, except for the _WFO_Cycles and _WFO_OOS values. They override the script, since they are used for the cluster analysis.

The Min and Max columns display the valid ranges of the variables, Mode lets you determine whether the variable is global, i.e. used for all jobs, or specific to a particular job. More about jobs below. Edited values in the panel will be remembered at the next start. Changes are indicated with a white background. Exceeding the valid range is allowed, but indicated with a red background.
 

Action menu

After starting the script in [Train] mode, a click on [Action] opens a dropdown menu. It contains only a few items at first:

When nothing is selected, clicking on [Start] will just run a test or training with the current variable settings in the panel (if not overridden by the script). After a successful backtest, the performance report will pop up with the price curve, the trades, and the equity curve.

When you select Cluster Analysis or Montecarlo analysis, a click on [Start] will run the whole process with the current settings. Dependent on the selected template, Cluster analysis will generate either a WFO profile or a heatmap like this:

The x axis is the number of WFO cycles, the y axis is the OOS period in percent. The numbers in the fields are the resulting strategy performance, selected by _Criteria. You want as much red in the heatmap as possible. The above example, with the Pessimistic Return Ratio (PRR) as metric, has only 13 red fields out of 25. This job would not pass the CA analysis.

Montecarlo Analysis is normally only applied to the final strategy with all assets and algorithms. It generates a histogram from results with randomized price curves (red) and the original price curve (black). This one below indicates that you better don't trade that system live, since its great walk-forward performance is not to trust:

Another one from a stragegy that generated less performance in the walk-forward analysis, but can be traded since it exploits a real market inefficiency:

Train mode will always use walk forward optimization, even when none is defined in the script. Test mode will only run a backtest; so the strategy should have been trained before. After changing any variable, train again. If a variable range was exceeded or historical data was missing for the selected backtest period, you'll get an error message. Training errors will store the error message in Log\Errors.txt and abort.
 

Step 2: Creating jobs

The next step of evalution is letting the shell control your script variables. This allows you to experiment with many different parameter settings and optimization ranges, and testing them all in an automated process. Any such variant is a 'job'. As a side benefit, putting script variables in a panel also allows you to observe their bar-to-bar behavior with the Visual Debugger.

Instead of eval.h, now include evars.h at the begin of the script:

#include "evars.h"

Below the #include statement, put all relevant script variables that you want to observe, optimize, or experiment with in a list. Add comments and put an END_OF_VARS statement at the end of the list. Example (from Workshop6b):

var CNTR_BPPeriod; //= 30, 20..40, 5; Bandpass period in bars
var CNTR_BPWidth; //= 0.5, 0.2..1; Bandpass width
var CNTR_FisherPeriod; //=500, 300..700; Fisher transform period
var CNTR_Threshold; //= 1,0.5..1.5,0.1; Bandpass threshold
var CNTR_Fisher; //= 500,300..500; Time frame of Fisher transform
var CNTR_TrailDist; //=4, 0..10; Distance factor for trailing
var TRND_Mode; //= 0, 0..1; 0 - Lowpass, 1 - Butterworth filter
var TRND_Detect; //= 0, 0..1; 0 - MMI, 1 - Hurst
var TRND_LPPeriod; //= 300,100..700,50; Lowpass/Butterworth period in bars
var TRND_MMIPeriod; //= 300,100..500; MMI/Hurst Filter time period
var StopDist; //= 10,5..30,5; Stop distance in ATR units
var ATRPeriod; //=100, 30..150; ATR period for stop/trail distances
END_OF_VARS

This encapsulates the variables in a C struct named V. Thus, in the script the variables are accessed with a preceding V., for instance V.ATRPeriod. They all must be of type var. It you need an integer variable, as in a switch/case statement, use an (int) typecast. Replace all your optimize calls with _optimize and pass the variable as sole parameter. Example:

Stop = _optimize(V.StopDist) * ATR(V.ATRPeriod);

The startup panel will now look like this (Workshop6b):

The added script variables appear at the end with a blue background. For illustrating the process, we have added two new variables to workshop 6 for selecting between two market regime detectors (Hurst and MMI) and two filters (Lowpass and Butterworth). The variable names and the comments matter. Observe the following rules:

• If the variable belongs exclusively to a certain algo, let its name begin with the algo name (f.i. TRND_Mode). Otherwise it is assumed that the variable is common for all algos.
• The default value is given with '= value' in the comment. If omitted, the variable is set to 0.
• If the variable is to be limited by a range, enter it with 'min..max' following a comma after the default value.
• If the variable is to be optimized, enter the step width following a comma after the range. If the step width is 0, the variable is not optimized, even if it appears in an _optimize call in the script. In this way, different jobs can optimize different variables.
• After the default value or range, enter a semicolon ';' and then a short description of the variable.

The Action menu got some new items:

In the Mode colum the grey system variables can be switched with a mouse click between two modes. In the global setting they affect all jobs, in the job setting they are job specific and loaded from a job file. Normally you'll use the global setting for them. The blueish strategy variables are always job specific.

Experiment with different variable settings and save them to the Jobs folder (or any other folder). Load all of them with Load Multiple Jobs. Four example jobs from workshop 6 are included, but you normally have a lot more jobs for thorougly testing a strategy, When you click [Start] after loading one or multiple jobs, they are all trained. At the end of the process, a bell will ring, and the Summary will open in the editor. It is a list of all trained jobs with their performance parameters. Any job that you train adds to the summary. If a summary entry of that job already exists, it is updated with the new results. The jobs with the best performances are at the top of the summary. The metrics are explained below.

If you activated Cluster Analysis after loading jobs from the summary, WFO profile or matrices are generated for any profitable job, and the percentage of positive training runs with different WFO/OOS settings is stored in the summary under CA%. If an otherwise profitable job got a bad cluster analysis result, like less than 60%, remove it from the summary.

The summary

Every job and every asset, algo, and timeframe variant of a job is listed as a line in the summary. The top jobs are at the begin. The summary is stored in CSV format, so it can be further evaluated in a spreadsheet program. The fields:

Step 3: Creating multiple variants

For finding out which asset, algo, and time frame combinations work best with which job, you can test it with any combination. Since this interferes with your asset/algo loops (if any) in the script, it has to be modified further:

First, put all assets, algos, and timeframes you want to test in #define _ASSETS, _ALGOS, _TIMEFRAMES statements:

 #define _ASSETS "EUR/USD","USD/JPY"
#define _ALGOS "TRND","CNTR"
#define _TIMEFRAMES 60,240  // in minutes

The syntax is the same as in a loop statement, so you can also use the Assets keyword to test all assets from the current asset list. Next, replace your main asset and algo loops with a single assetLoop() call. Assume your run function previously looked like this:

function rum()
{
  BarPeriod = 60; 
  LookBack = 2000; 

  while(asset(loop("EUR/USD","USD/JPY")))
  while(algo(loop("TRND","CNTR")))
  {
    if(strstr(Algo,"TRND")) {
      TimeFrame = 1;
      tradeTrend();
    } else if(strstr(Algo,"CNTR")) {
      TimeFrame = 4;
      tradeCounterTrend();
    }
  }
}

It now becomes this (see workshop6c.c):

 #define _ASSETS "EUR/USD","USD/JPY"
#define _ALGOS "TRND","CNTR"
#define _TIMEFRAMES 60,240  // in minutes

function run()
{
  BarPeriod = 60; 
  LookBack = 2000; 

  assetLoop();

  if(strstr(Algo,"TRND")) 
    tradeTrend();
  else if(strstr(Algo,"CNTR")) 
    tradeCounterTrend();
}

If your script has no asset/algo loops, replace your first asset() call with assetLoop(). Mind the Algo comparison with strstr. Using strcmp or '=' for algo selection would not do anymore, since algo names will now get a number appended. If you start the system now, it assigns different colors to the variables since it recognizes from their name to which algo they belong.

The Action menu is now complete:

Results and reports

The test results for any job are automatically stored in subfolders of the job folder. In this way any job leaves a 'papertrail', consisting of text, CSV, image, or html files from its training runs. All reports can be opened with a plain text editor such as the included Notepad++. The .csv files can also be opened with Excel, and the charts with an image viewer like the included ZView.

Aside from the summary and the CA or MA charts, the following reports are produced when jobs are processed (for details see exported files):

*.txt - the last performance report.
*history.csv - the performance history of the job. Any test or training run adds a line to the history.
*.png - the chart with the equity curve, trades, and indicators.
*.htm - a page with parameter charts that visually display the effect of parameters on the performance.
*.log - the event log of the backtest.
*train.log - the training log with the objective results for all tested parameter combinations.
*pnl.csv - the equity curve of the backtest, for further evaluation.
*trd.csv - the trade list of the backtest, for further evaluation.
*par.csv - the results of genetic or brute force training runs, for further evaluation.
Log\Errors.txt - the last errors encountered (if any) while training or testing.
 

Very concise tutorial

• First, the summary of job performances. Select the Workshop6c script and click [Train].
• From the [Action] scrollbox, select Load Multiple Jobs.
• 4 example jobs beginning with "W6..." are included in the Job folder. Double click one of them.
• Click [Start]. All jobs with their asset, algo, timeframe variants are now trained.
• At the end of the process, the performance summary will pop up in the editor.
   
• Next, cluster analysis. Click [Train] and select Load Jobs from Summary.
• Double click on Job\Workshop6c_Summary.csv that was generated in the previous step.
• From the [Action] scrollbox, select Run Cluster Analysis.
• Now you have the choice between several predefined clusters. Select WFO_Cluster3x3.
• After a lengthy process, the summary appears with the CA% column of the best jobs populated,
   
• Next, portfolio creation. Click [Train] and select Create Algos from Summary.
• Double click on Job\Workshop6c_Summary.csv. You'll see a list of algo variants from the top of the summary.
• Click [Start] to train. The perfomance report will pop up at the end.
   
• Finally, the Montecarlo Reality Check. Click [Train]. The algos are automatically loaded,
• From the [Action] scrollbox, select Run MonteCarlo Analysis.
• At the end of the process you'll get the p-value and a histogram of the real vs. reandom results.

Narurally, training hundreds of jobs and variants will take its time. Get a cup of coffee inbetween.
 

Remarks

• Compare algo names not with '=', but with strstr(), as in the above example. First, it is needed because the algos generated by the shell get a number appended. Second, it is needed when you want to run your strategy with VC++. VC++ will give no error message when comparing strings with '=', but will just skip the clause.
• Make sure that Lookback is sufficient for running all loaded jobs. Some jobs need up to 800 bars for the lookback period. When extending or reducing the backtest years, adapt WFO_Cycles accordingly.
• Training can be greatly sped up by activating multiple threads. If a thread encounters an error, the optimization will stop. The error message can be found at the end of the file Log\Errors.txt.
• Training results are only kept for last training run or the final portfolio. When you switch jobs or change parameters, always train again.
• Do not open results in Excel while the shell is running. Excel will block any access to its open files.

See also:

► latest version online