Price History (and how to get it)

Historical price data is used in [Test] and [Train] mode, and optionally also in [Trade] mode for pre-loading the LookBack period. The needed data period can be determined with StartDate and EndDate. If historical prices from the desired period are missing, they can be downloaded with the assetHistory function or the Download script (see below). All downloaded prices are usually aligned to UTC time. The broker's time zone does not matter. Therefore it's normally no problem to use price data from different sources for the simulation and for trading.

Zorro stores historical price data in files in the History folder, together with optional  additional data such as spread and volume ("Level-1 data"), expiration dates, option chains, or order book content ("Level-2 data"). Each file name is composed from the asset name (with special characters such as '/' removed), an optional year number, and the extension. The usual formats are ".t6" for candles with spread and volume data, ".t1" for plain ask/bid quotes (see also M1 vs. T1), and ".t8" for options and futures. For instance, "EURUSD_2010.t6" contains prices and tick volumes of the EUR/USD asset in 2010. It is also possible to import arbitrary user-specific formats in a dataset and retrieve historical data from it. The .t6 price data files included with Zorro are minute based, but any other time base can also be used as long as it's equal or less than the bar period used in the script.

Price data in different formats can not be mixed. You can use .t1, .t6, and/or .t8 price data in the backtest of a strategy, but not change the format between years, for instance .t6 from 2016 and .t1 from 2015. But it is no problem to load option prices from a .t8 file and the underlying prices, with a different time base, from a .t6 or .t1 file.

A .t1 or .t6 price data file is a simple list of T1 or T6 structs in descending order (newest price first).

typedef struct T1
DATE time; // timestamp of the tick in UTC, OLE date/time format
float fVal; // price data, positive for ask and negative for bid
} T1; typedef struct T6 { DATE time; // timestamp of the end of the tick in UTC, OLE date/time format float fHigh,fLow; float fOpen,fClose; float fVal,fVol; // additional data, like ask-bid spread, volume etc. } T6;

Historical data of options and futures are stored in a .t8 file that includes option specific parameters such as type, strike, and expiration date. It is a list of CONTRACT structs in descending order:

typedef struct CONTRACT
  DATE  time;   // time in UTC, OLE date/time format
  float fAsk,fBid; // premium without multiplier
  float fVal,fVol;  // additional data, like open interest, volume etc.
  float fUnl;   // underlying price
  float fStrike; // strike price
  long  Expiry; // YYYYMMDD format

For downloading and reading data files with a resolution of less than one minute (such as .t1 files), Zorro S is required. For converting price data from an external source to the Zorro format, separate it into years, convert it to a list of T6 or T1 structs, and store it in a binary file with the name described above. The structs are stored in reverse order, i.e. the most recent tick comes first. The OLE DATE format is a double float value, counting days since midnight 30 December 1899; hours, minutes, and seconds are represented as fractions of days. The timestamps are supposed to be the UTC time at the end of the tick when the Close price is sampled. If you have price data based on local time, convert it to UTC (you can find some example code here). The exception is daily (D1) price history, where the timestamps are normally the local time at the begin (00:00) of the trading day. This is the format in which daily price history is usually stored. If the price data contains bid prices, you can convert them to ask prices by adding the spread. Using ask or bid price history has however normally no noticeable effect on backtest results.

The fOpen, fClose, fHigh and fLow data streams can be separately accessed in the script. The fVal and fVol data can be used to store and access additional data, such as spread, quote frequency, trade volume, or rollover. Data in .t1 format can be used for additional information, such as external indicators or statistics such as the VIX (in that case set Detrend at 16 for disabling the automatic fixing of price data). All data streams are automatically synchronized to the streams of the first asset call in the script; so it does not matter if some data is only available in daily or weekly 'ticks'. The asset function can be used to select between different data files, while the price functions select between the four data streams inside the file.

Historical data in arbitrary CSV formats can be converted to .t1, .t6, or .t8 with the dataParse function. Exporting or converting historical data to and from other formats is also easy due to the simple format - in the Strategy folder you can find several small scripts for this purpose. Download.c (see below) downloads historical data in .t6 or .t1 format from Yahoo, IB, FXCM, or Oanda. CSVtoHistory converts data from arbitrary .csv formats, f.i. from HistData, Quandl, or Yahoo, to the Zorro .t6 format. CSVExport.c exports the complete price history of the selected asset to a .csv spreadsheet, and CSVFromHistory converts a particular .t6 file to a .csv spreadsheet. Example code for automatically downloading .csv price data from the Internet or from a provider API can be found on the http page. A script for generating .t8 historical data for options from the yield rate, dividends, and underlying price history can be found here. Example scripts for converting history files from complex .csv or .nxc formats to the .t1 format can be found here.

The Download script

Download.c is a small script for downloading price history from the Internet or from a broker or signal service. It offers also a convenient way for retrieving broker-specific asset data, such as lot size, margin, and rollover, directly from the broker API. The script opens a control panel with buttons (blue) and entry fields (yellow):

Asset List

Download or update historical price data for the current year

Adding a new asset to the Asset scrollbox

Downloading historical M1 price data for a new asset

Downloading historical tick-based price data

Downloading historical daily data from Yahoo™

Simulating a different broker account Updating the recent price history of all assets

See also:

Bars, file, Dataset, asset parameters, assetHistory

► latest version online