Price History (and how to get it)

Historical price data is used in [Test] and [Train] mode, and optionally also in [Trade] mode for pre-loading the LookBack period. The needed data period can be determined with StartDate and EndDate. If historical prices from the desired period are missing, they can be downloaded with the assetHistory function or the Download script (see below). All downloaded prices are usually aligned to UTC time. The broker's time zone does not matter. Therefore it's normally no problem to use price data from different sources for the simulation and for trading.

Zorro stores historical price data in files in the History folder, together with optional  additional data such as spread and volume ("Level-1 data"), expiration dates, option chains, or order book content ("Level-2 data"). Each file name is composed from the asset name (with special characters such as '/' removed), an optional year number, and the extension. The usual formats are ".t6" for candles with spread and volume data, ".t1" for plain ask/bid quotes (see also M1 vs. T1), and ".t8" for options and futures. For instance, "EURUSD_2010.t6" contains prices and tick volumes of the EUR/USD asset in 2010. It is also possible to import arbitrary user-specific formats in a dataset and retrieve historical data from it. The .t6 price data files included with Zorro are minute based, but any other time base can also be used as long as it's equal or less than the bar period used in the script.

Price data in different formats can not be mixed. You can use .t1, .t6, and/or .t8 price data in the backtest of a strategy, but not change the format between years, for instance .t6 from 2016 and .t1 from 2015. But you can load option prices from a .t8 file and the underlying prices, with a different time base, from a .t6 or .t1 file.

A .t1 or .t6 price data file is a simple list of T1 or T6 structs in descending order (newest price first).

typedef struct T1
{
DATE time; // timestamp of the tick in UTC, OLE date/time format
float fVal; // price data, positive for ask and negative for bid
} T1; typedef struct T6 { DATE time; // timestamp of the end of the tick in UTC, OLE date/time format float fHigh,fLow; float fOpen,fClose; float fVal,fVol; // additional data, like ask-bid spread, volume etc. } T6;

Both .t1 and .t6 data can contain the current ask-bid spread, which is returned by the marketVal function. For backtesting with variable spread, set Spread = marketVal() in the script.

 Historical data of options and futures are stored in a .t8 file that includes option specific parameters such as type, strike, and expiration date. It is a list of CONTRACT structs in descending order:

typedef struct CONTRACT
{
  DATE  time;   // time in UTC, OLE date/time format
  float fAsk,fBid; // premium without multiplier
  float fVal,fVol;  // additional data, like open interest, volume etc.
  float fUnl;   // underlying price
  float fStrike; // strike price
  long  Expiry; // YYYYMMDD format
  long  Type;   // PUT, CALL, FUTURE, EUROPEAN, BINARY
} CONTRACT;

Example code for converting historical options data from .csv to .t8 format can be found under contract.

Historical order book data is stored in a .t2 file that contains the ask/bid quotes and the quoted volume:

typedef struct T2
{
  DATE time;  // timestamp in UTC, OLE date/time format
  float fVal; // price quote, positive for ask and negative for bid
  float fVol; // volume / size
} T2; 

For downloading and reading data files with a resolution of less than one minute (such as .t1 files), Zorro S is required. For converting price data from an external source to the Zorro format, separate it into years, convert it to a list of T6 or T1 structs, and store it in a binary file with the format and name described above. The structs are stored in reverse order, i.e. the most recent tick comes first. The OLE DATE format is a double float value, counting days since midnight 30 December 1899; hours, minutes, and seconds are represented as fractions of days. The timestamps are supposed to be the UTC time at the end of the tick, i.e. the time of the Close price. If you have price data based on local time, convert it to UTC (you can find some example code here). The exception is daily (D1) price history, where the timestamps are the local date with no time offset (otherwise a bar ending at 00:00 would have the date of the next day). If the original price data contains bid prices, you could convert them to ask prices by adding the spread; if the price data contains time stamps from the tick start, you could shift them to the tick end by adding the bar period when converting the file. However, both is normally not very relevant for backtest results.  

The fOpen, fClose, fHigh and fLow data streams can be separately accessed in the script. The fVal and fVol data can be used to store and access additional data, such as spread, quote frequency, trade volume, or rollover. Data in .t1 format can be used for additional information, such as external indicators or statistics such as the VIX (in that case set Detrend at 16 for disabling the automatic fixing of price data). All data streams are automatically synchronized to the streams of the first asset call in the script; so it does not matter if some data is only available in daily or weekly 'ticks'. The asset function can be used to select between different data files, while the price functions select between the four data streams inside the file.

Historical data in arbitrary CSV formats can be converted to .t1, .t6, or .t8 with the dataParse function. Exporting or converting historical data to and from other formats is also easy due to the simple format - in the Strategy folder you can find several small scripts for this purpose. Download.c (see below) downloads historical data in .t6 or .t1 format from Yahoo, IB, FXCM, or Oanda. CSVtoHistory converts data from arbitrary .csv formats, f.i. from HistData, Quandl, or Yahoo, to the Zorro .t6 format. CSVExport.c exports the complete price history of the selected asset to a .csv spreadsheet, and CSVFromHistory converts a .t6 file to a .csv spreadsheet. Example code for automatically downloading .csv price data from the Internet or from a provider API can be found on the http page. A script for generating .t8 historical data for options from the yield rate, dividends, and underlying price history can be found here. Example scripts for converting history files from complex .csv or .nxc formats to the .t1 format can be found here.

For looking into .t8, .t6, .t2, or .t1 files and displaying their content in a panel, use the History script.

The Download script

History files for all example strategies are available on the Zorro Download page. For more price histories you can use Download.c, a small script for downloading prices and asset parameters from the Internet or from a broker or signal service. You can download either the history of a single asset, or of all assets of a given asset list. The script opens a control panel with toggle buttons (blue) and entry fields (yellow):

History  
AAPL Asset or asset list to download
2016-2017 Time period to download
M1 Data resolution (M1, Tick, EOD)
Ask/Bid Data type (Ask/Bid, Trades, Unadjusted, Assetlist)
Broker Data source (Broker, Stooq, AlphaVantage)
Download! Click here to start the download.

Asset or asset list

Enter here the symbol of the asset in the form needed by the broker or price source, f.i. "AAPL.US" for downloading AAPL data from Stooq. If the name contains the word "Assets" or ends with ".csv", price data for all assets from the given asset list is downloaded.

Data resolution

Data type

Data source

Not all combinations make sense. For instance, you can normally not download M1 data or asset parameters from an Internet data source. Check with your broker which data types are available. If a price history file of the same asset already exists, the new prices are appended at the end; if the file was already updated on the same or previous day, no data is downloaded.

See also:

Bars, file, Dataset, asset parameters, assetHistory

► latest version online