Foundations of System Design

Foundations of System Design

The best system provides instant gratification and constant satisfaction.

Introduction

This chapter examines many key system design issues. Now that you understand some basic principles of system design, you can consider more complex issues. And as you understand these issues, you can design more powerful systems.

We will begin by asking the question: Do markets trend? The answer to the next big question, whether you should trade with the trend or against the trend, is that you should trade with the trend. This chapter presents some test results to support this answer. You can then ask whether you should or should not optimize your trading system. We explore here how well you can predict future performance based on optimization of historical data.

The chapter begins the discussion on risk control issues by addressing whether the initial stop is a problem or a solution and discussing the different types of risk you may face in your trading. You should consider these issues early in your design process. We then look at the different types of data you can use for your testing and what difference, if any,

40 Foundations of System Design

they make. Finally, the chapter explains what is found as well as what is lacking in the system performance summary.

At the end of this chapter, you will be able to:

1. Explain how you can diagnose trending markets.

2. Know whether to use a trend-following or countertrend strat-

^

3. Explain the benefits and pitfalls of optimization.

4. Understand the type of risks you may encounter.

5. Know how to select data for tests.

6. Effectively use the performance summary of system testing results.

7. Understand and explore what is not covered in the performance summary.

8. Explain why system design has its limits.

Diagnosing Market Trends

You can design a profitable trading strategy if you can correctly and consistently diagnose whether a market is trending. In simple terms, the market exists in two states: trending and ranging. A market is trending if it moves steadily in one direction. If the market is going back and forth within a relatively narrow price range, then it is ranging.

Longer-term strategies are likely to succeed in trending markets, and shorter-term strategies in ranging markets. As always, the market may not make a crisp transition from trending to ranging and back again. Sometimes the market begins to range only to break out into a trend, or vice versa.

There are many different ways to determine if a market is trending. Clearly, you must make a number of trade-offs, and these trade-offs largely define your answer. For example, one well-known measure is the average directional index (ADX) developed by Welles Wilder Jr. (see bibliography for references). This is usually a built-in function in most technical analysis software programs. The ADX describes double-smoothed, absolute market momentum. A rising ADX line usually indicates trend. You have to choose the number of days to calculate the ADX; the sensitivity of the indicator decreases as the time increases. A

Diagnosing Market Trends 41

value of 14 days is common, although 18 days works well. You must also define two reference levels to screen out false signals. An ADX value of 20 is useful as a reference levelthat is to say a market is not trending unless the rising 18-day ADX is above 20. A second useful barrier level is 40, which says that when the ADX rises above 40 and then turns down, a consolidation is likely. You will find that in particularly strong trends, the 'hook' from above 40 often signals just a brief consolidation phase. The trend then has a strong second 'leg' toward higher highs or lower lows.

Sometimes you will find that the ADX will rise above 20 in markets that are in a broad trading range. Another quirk is that the ADX can head lower even though prices march steadily and smoothly in either direction. In short, this is not a perfect indicator. The main difficulty with the ADX is that it has two levels of smoothing, which produces disconcerting lags between price movement and indicator response. Chapter 5 shows that the absolute level of the ADX indicator is not as useful for system design as is its trend.

An indicator that is more directly based on market momentum, and that responds more predictably than the ADX, is the range action verification index (RAVI). This strategy, which focuses on identifying ranging markets, is different from the ADX, which looks at how much of today's price action is beyond yesterday's price bar.

To define RAVI, we begin by selecting the 13-week simple moving average, since it represents a quarter of a year. Because we want to use daily data, we convert the 13-week SMA into the equivalent 65-day SMA of the close. This is the long moving average. The short moving average is chosen as only 10 percent of the long moving average, which is 6.5 days, or, rounding up, 7 days. Thus, we use 7-day and 65-day simple moving averages. This choice of lengths is purely arbitrary. Next, the RAVI is defined as the absolute value of the percentage difference between the 7-day SMA (7-SMA) and the 65-day SMA (65-SMA):

RAVI = Absolute value (100 x (7-SMA-65-SMA)/65-SMA)

An arbitrary reference level of 3 percent means a market is ranging if the RAVI is less than 3 percent, and trending strongly if the RAVI is greater than 3 percent. In some markets, such as Eurodollars, this is too high a hurdle. Hence, you may want to experiment with a smaller level, such as 1 percent, or use a relative measure, such as a 65-day SMA of the

I RAVI. You can also require that the RAVI be above 3 percent and rising

I for there to be a strong trend.

I

Foundations of System Design

Note the following design features of the RAVI: (1) There is only one level of smoothing. (2) The 7-day moving average is relatively sensitive, so that the lags between price action and indicator action should be small. (3) Markets can still move more quickly than the RAVI indicates. You can verify this by looking at the currency markets. (4) Markets in a slowly drifting, choppy trend will pin the RAVI below 3 percent, indicating ranging action.

Figure 3.1 compares the 18-day ADX (bottom graph) to the RAVI (middle graph) with a horizontal line at the 3 percent RAVI level. There is a general similarity between the two indicators, with the RAVI responding more quickly than the ADX because it has only one level of smoothing versus two levels for the ADX. A double-smoothed RAVI indicator created by smoothing the RAVI with a 14-day SMA is very similar to the 18-day ADX, as shown in Figure 3.2. Thus the ADX closely describes double-smoothed momentum and can lag price movements.

We now compare the ADX and RAVI and use them both to measure how often trends occur. In this example, we use continuous contracts from January 1, 1989, through June 30, 1995, a rising 18-day ADX above 20, and a rising RAVI greater than 3 percent. The ADX and RAVI are considered to be rising if today's value is greater than the value 10 days ago. These choices of length and reference levels are arbitrary.

Figure 3.1 Comparison between the ADX (bottom) and RAVI (middle) to measure ranging behavior.

Diagnosing Market Trends 43

in Jul Aug Sep Oct Nov Dec

Figure 3.2 A double-smoothed RAVI (solid line) compared to the 18-day ADX (dotted line) shows that the two indicators are very similar.

The calculations shown in Table 3.1 suggest that markets seem to show some form oftrendiness about 20 to 40 percent of the time. Some markets, such as the 10-year T-note, have not shown very strong trends as measured by the RAVI. However, this may just be due to using a 3 percent barrier with the RAVI to measure trend strength. The 'soft' markets, such as coffee and sugar, show the highest tendency to trend. Other fundamentals-driven markets, such as cotton, copper, and crude oil, also show a tendency to have strong trends, with a RAVI rating above 35 percent. The more mature markets, such as S&P-500 and U.S. bond markets, show fewer strong trends than the softs. RAVI calculations correctly tagged the prolonged sideways ranging action in gold with a low rating of 15.8.

A separate calculation showed that the average length of these trending intervals was about 15 to 18 days in most markets, with values ranging from as low as 1 to more than 30. Thus, the trending phase of these markets was long enough to allow profitable trading. These calculations show that markets have provided sufficient opportunities for trend-following systems in the 'trendless nineties.'

In summary, you can use momentum-based indicators to measure ranging or trending action. The calculations show that markets have trends lasting 15 to 18 days on average. Hence, trend-following strate-

44 Foundations of System Design

Table 3.1 Proportion of market days showing definite trend, using ADX and RAVI

ADX Rising, RAVI Rising, Market (1/1/89-6/30/95) ADX>20 RAVI > 3

gies are worth considering for system design. The next section examines whether you should use trend-following strategies over the long run.

To Follow the Trend or Not?

If you are not a large hedger or an institutional trader, you can follow either of two basic strategies when you design a trading system. You can be a trend follower, or you can take antitrend positions. If you are a trend follower, you will typically take intermediate-term positions. In contrast, with a countertrend strategy, you take shorter-term positions that anticipate trends. This section explores both strategies and shows that a trend-following approach is more likely to be profitable over the long run than an antitrend approach.

Table 3.2 shows test results for a stochastic-oscillator-based antitrend trading system provided with System Writer Plus software from Omega Research. The stochastic oscillator is a range-location oscillator that shows where today's close is within its trading range over the last x days. If the close is near the top of the range, then oscillator values are greater than 80. The next move in prices will probably be toward the lower end of the range. Similarly, if the close is near the lower

To Follow the Trend or Not? 45 Table 3.2 Stochastic-oscillator antitrend trading system results

Paper Number Percent- Largest Biggest Consecu- Intraday

Profits of age of Winner Loser tive Drawdown

($) Trades Winners ($) ($) Losers ($)

end of the range, then oscillator values are below 20. We assume that the next move will take prices toward the top of the range. The 'range' between the .r-day high and low changes continuously. Hence, this oscillator cannot predict the amplitude of the next move.

The system tested uses a 10-day period to calculate the so-called fast-K and fast-D moving averages. When the fast-K is above the fast-D line, the system buys on the open and vice versa. The System Writer Plus software guide gives the exact method for the calculations.

This example uses continuous contracts for seven unrelated markets, allows $100 for slippage and commissions, and uses a $1,500 initial money management stop. The test period was from May 26, 1989, through June 30, 1995. This simple system was a net loser over these markets. It also had substantial drawdowns, largely due to the many successive losing trades. Note the large number of trades and the relatively low proportion of winners.

The main implication of these calculations is that although markets may trend for short periods only, the profits during trending periods can far exceed the profits during trading ranges. The reason for this is that the amplitude of price moves during trends is many times the amplitude during trading ranges.

This example assumes that you pay the 'discounted' trading commissions offered on the street. If your trading commissions are very low or negligible, then the antitrend strategy, with its high trading frequency, takes on a different dimension.

46 Foundations of System Design

Table 3.3 Impact of trading costs on profitability of antitrend trading strategies (dollars)

Table 3.3 compares paper profits with and without slippage and commissions (S&C). The difference in profitability is striking. The stochastic oscillator system performance improved significantly with low commissions. This result indicates that an antitrend strategy would not be attractive if you had to pay high commissions.

There are a number of 'antitrend' strategies. Table 3.4 presents another set of calculations using a different trading strategy to illustrate this point. The moving average crossover (MAXO) system is the simplest trend-following strategy, but it can also be used as an antitrend strategy. For example, if the shorter moving average crosses over the longer moving average, you can go short in an antitrend strategy. Of course, this 'upside' crossover would be a signal to buy long in a trend-following strategy.

Table 3.4 Comparison of trading systems using 5-day and 20-day simple MAXO tests, 5/89-6/95 (dollars)

To Follow the Trend or Not? 47

Here we have arbitrarily picked 5-day and 20-day moving averages as examples of short- to intermediate-term averages. The test period was from May 26, 1989, through June 30, 1995, with $100 for slippage and commissions and a $1,500 initial stop. The antitrend strategy was a net loser on average, with significant potential for intraday drawdowns. The trend-following strategy cut the average loss by 79 percent and drawdown is lower by 39 percenta better situation on both counts.

Table 3.5 presents another combination: the moving average antitrend and trend-following strategies with 7-day and 50-day simple moving averages. This combination is good for no-nonsense trend following. The assumptions are the same as before: $100 for slippage and commissions and a $1,500 initial stop with the calculations performed from May 26, 1989, through June 30, 1995.

Under antitrend trading, the 7/50-day SMA combination was also a net loser. On the other hand, it was a net winner with trend following, with profitability across all seven markets. The trend-following strategy had approximately one-fifth the drawdowns of the antitrend approach. Thus, the trend-following approach was the better choice on both counts.

These calculations show that a trend-following strategy is probably the better choice for the average position trader. However, the antitrend strategy may be attractive if you have low commission costs and little slippage.

The example tests in this chapter used arbitrary combinations of moving averages. However, you can test your system over historical data

Table 3.5 Comparison of performance for 7-day and 50-day simple MAXO tests, 5/89-6/95 (dollars)

48 Foundations of System Design

to find other combinations with better performance. Optimization is the process of finding the 'best' performing variable set on historical data. The next section examines whether optimization is a good design strategy.

To Optimize or Not to Optimize?

If you have a computer, you can easily set up a search to find the 'optimum' values for a system over historical data. The results can be truly astonishing. Imagine your profits if you could only have known ahead of time what the most profitable parameter combination was going to be. Therein lies the rub. The unfortunate fact is that parameters that work best on past data rarely provide similar performance in the future.

The term 'optimization' is used rather loosely here to include all the activities affecting selection of parameter values in a trading system. We have already seen the difficulties of curve-fitting a model. You can also consider lower levels of optimization, in which you test variables over a broad range of values and markets, and try to select the one you like 'best.' But the real issue is not whether a particular set is the best. It is whether you believe sufficiently in the system to trade it without deviations. The primary benefit of optimization may be that you improve your comfort level with a particular system.

The problem with system optimization is that past price patterns do not repeat exactly in the future. The same is true of intermarket relationships. Although broad relationships follow from historical data, there can be differences in the time-lags between events and the relative magnitudes of the effects.

You must also resolve other conflicts. For example, you must choose the period you will use to optimize your trading system values. As you will quickly discover, the values you choose depend on the length of the test period. You must also determine how often you will reop-timize your system in the future. You must then prescribe the time for which the optimized values are valid.

For example, you may decide to use 3 years of data to optimize the values and recalculate them after 3 months. Thus, one solution may be to reoptimize after 3 months on the latest 3 years of data available. This is equivalent to retraining your favorite neural net. If you do reoptimize, you must determine how to treat trades that may be open from the previous period or values of the trading system.

To Optimize or Not to Optimize? 49

You must also decide if you want to use the same values of your system parameters on all markets. If not, you will have to optimize the system on each market separately. In that case, you must keep up a program of reoptimization and recalibration for each of your systems over every market that you trade. Is all this effort worth the trouble? The results of deterministic testing do not support any attempts at finding the 'best' or optimized variables.

Consider the following test using actual deutsche mark futures contracts. The rollover dates are the twenty-first day of the month before expiration. For simplicity, we will trade just one contract, allowing $100 for slippage and commissions, with a $1,500 initial money management stop. We will use a variation of the moving average crossover system, trading not the crossover, but a 5-day breakout in prices after the crossover. Thus, if the shorter moving average was above the longer moving average, then a 5-day breakout above the highs would trigger a long entry. Also included is a simple exit condition, ending the trade on the close of the twentieth day in the trade. One attractive feature of this arbitrary system is that the lengths of the short and long moving average can be optimized.

The calculations are simplified by fixing the length of the short average to a 3-day simple moving average of the close. The length of the longer simple moving average varies from 20 to 50 days, with an increment of 5 days. The test period was from November 14, 1983, through November 21, 1989. The performance of the various models was observed 3, 6, 9, and 12 months into the future. As Tables 3.6 and 3.7 show, there is no predicting how the model will do over a future period. The relative rankings change from period to period without any pattern or consistency.

Table 3.6 Data showing that past performance does not predict future performance

20 31,238 -2,200 -1,538 1,863 650 25 28,275 -2,475 -3,112 -488 -2,300 30 24,175 338 -300 2,325 2,113 35 18,088 338 63 2,175 1,963 40 15,475 338 -525 2,625 4,000 45 7,950 338 -4,363 2,038 3,600 50 7,013 338 -4,363 -1,800 -238

50 Foundations of System Design

Table 3.7 Data showing that relative rankings from the past do not predict future relative ranks

We next test the hypothesis that if the optimization period were closer to the actual trading period, the predictions would be more reliable. However, as Tables 3.8 and 3.9 show, there is again no way to predict what the model will do in the succeeding periods. This should be expected because there is no cause-and-effect relationship between our optimized model and market forces. Since we are merely fitting a model to past data, we are not capturing all the fundamental and psychological forces driving the market. Our poor ability to predict the future based only on past price data is not surprising.

Let us carry our argument one step forward. Because we do not capture any cause-and-effect relationships, optimization on one market should have little or no benefit for trading other markets. Indeed, as Table 3.10 shows, optimizing a system on one market (here the deutsche mark) does little to improve performance in other markets.

Table 3.8 Data showing that bringing the optimization period closer to the trading period (11 /88-11 /89) does not predict future performance

20 3,525 -1,625 -1,000 2,650 2,438 25 5,225 -1,900 -2,575 400 -^13 30 4,250 5,338 4,713 7,688 8,475 35 ' 513 5,338 4,713 7,213 8,000 40 63 5,338 4,437 6,213 8,813 45 -2,800 5,338 3,138 4,913 7,638 50 -1,525 5,338 913 2,688 5,413

To Optimize or Not to Optimize? 51

Table 3.9 Data showing that relative rankings over recent past (11 /88-11 /89) do not predict future relative ranks

Any optimization exercise has many potential benefits. The first benefit is recognition of the type of market conditions under which the trading system is unprofitable. For any rules that you can construct, you can find market action that produces losses. This happens because the market triggers the signal, and then does just the opposite instead of following through.

The second benefit is verification of the general ideas underlying the model. For example, you can check to see if the model is profitable in trending markets or trendless markets. You have designed the rules to be profitable under certain market assumptions. The optimization exercise allows you to verify if your broad assumptions are correct.

A third benefit is understanding the effect of initial money management stops. You can quantify what level of initial stop allows you to

Table 3.10 Data showing that optimization over one market does not predict performance in other markets

52 Foundations of System Design

capture the majority of potential profits. For example, if your stop is too wide, your losing trades will be relatively large. On the other hand, if your stop is too close to the starting position, you will be stopped out frequently. Your loss per trade will be small. However, the higher frequency of losing trades means your total drawdown could exceed a larger initial stop.

The biggest benefit of optimization is reinforcing your beliefs about a particular trading system. Ultimately, it is more important for you to implement the trading system exactly as planned. Hence, any testing you do that allows you to understand system performance and become more comfortable with its profit and loss characteristics will help you to execute it with greater confidence in actual trading.

The main point of this section is that you cannot assume your system is going to be as profitable in the future as it has been in the past. This raises the issue of how you control your risks to cope with uncertain future performance. The next section presents risk-control ideas.

Initial Stop: Solution or Problem?

Many traders have raised stop placement to an art form because it is not clear if the initial stop is a solution or a problem. The answer depends on your experiences. Often, the stop acts as a magnet for prices. It seems the market hits the stop, only to reverse and resume the previous trend. Thus, initial stops can easily test your patience. Even so, initial stops should be an essential part of managing trading risk. This section discusses some general issues related to selecting an initial stop. Detailed examples appear in the following chapters.

If you use an initial stop at all, use stops that follow money-management rules but are derived from system design and market volatility. A good idea is to use a 2 percent of equity initial stop, and then use maximum adverse excursion (MAE), a distribution of the worst loss in winning trades, to select the dollar value of the stop for a particular system. Relate the MAE to some measure of market volatility before calculating the number of contracts. Thus, the initial stop meets three criteria:

money management, MAE, and volatility.

Another issue involves whether you should place your stop loss order with your broker. Many traders will have a well-defined exit price, but will not place an order in the market. They like to monitor the market in real time, and will place the exit order themselves if needed. This is termed the 'discretionary initial stop.' If you have good discipline and

Initial Stop: Solution or Problem? 53

judgment, the discretionary initial stop could work well for you. However, if you cannot monitor the market continuously, it may be prudent to enter the exit order with your broker.

What values of the initial stop should you use during system testing? That depends on the type of data you have and the nature of the system design. The issue is whether to use a tight stop or a loose stop. A tight stop may have a dollar value less than $500 per contract. A loose stop could be as high as $5,000.

Let us assume you have only daily data. In this case, it is difficult to test a tight stop accurately because the exact track of prices during the day is unknown. Suppose you are trading the bond market, and the typical daily range is $1000. Now, say you want to test a $100 stop with daily data. Most system-testing software will stop you out on the day of entry because it does not know the exact track of prices. Of course, if you have intraday data, then you can more accurately test a $100 stop. Thus, if your stop is very tight, you need intraday data for accurate tests.

There are two broad types of systems, those that are self-correcting and those that are not self-correcting. Self-correcting systems have rules for long and short entries. Such systems will eventually generate a long signal for short trades and vice versa. Because these systems are self-correcting, the reverse signal will limit losses, even without an initial stop. Of course, the losses will depend on market volatility, and easily could be as large as -$10,000 per contract.

Systems that are not self-correcting include those that trade the long side or the short side only. Thus, you could get a false short signal and remain short through a long up trend. The losses in these systems can be unlimited, and hence must be protected by an initial stop. A onesided system with an exit strategy can become self-correcting. The exit strategy will limit losses in a one-sided system by closing out the trade at some preselected point. For example, a self-correcting, longside-only system has an exit stop at the most recent 14-day low.

You can get a better feel for the efficiency of entry rules if you test a self-correcting system without initial stops. However, if the system is not self-correcting, then you must test it with an initial stop. There is still the issue of how wide the stop should be. Relatively wide stops, defined as three times the 10-day average of the daily range, are a good choice. In this way the stop has a smaller influence on results than do the entry rules. If you like tight stops, then use intraday data, or use an amount larger than the recent daily trading range.

Your data set will strongly influence the results of your initial stop selection. If your data set has many trading range markets, then a tight stop will produce whipsaw losses. Even though each loss may be small,

Foundations of System Design

the sum of a series of losses can be large. A loose stop will prevent whip-saw losses in a trading range. If the market is trending, then the value of the initial stop is not critical. Thus, a trending market will rescue a system with tight stops, and you can get some astonishing results.

Relatively loose stops, between $1,500 and $5,000, work well. If the stops are relatively 'loose' then there is little difference between nearby values. Conversely, if the stop is 'tight,' then small changes in the stop can produce big swings in equity. Hence, the system tests in this book use daily data and stops ranging from $1,000 to $5,000.

Often, the point of discussion in this book does not depend on the amount of the stop. Sometimes the loose stop is a necessary design feature. In such cases the reason for choosing the wider stop is stated. Ultimately, if you do not like my stop, you can retest the system to suit your preferences.

Some actual calculations will clarify this discussion. Here we use the standard 20-day channel breakout on the close (CHBOC) trading system. This system buys on the close if today's close is higher than the highest high of the last 20 days. The short sale condition is symmetrical.

20-day CHBOC with varying initial stop

Initial stop ($)

Figure 3.3 Profit increases steadily and then levels off as the initial stop increases.

Initial Stop: Solution or Problem? 55

The system sells short on the close if today's close is lower than the lowest low of the past 20 days. We will test this system on the coffee market, which has seen much volatility as well as strong trends. We will vary the initial stop from $0 to $8,000 in $500 increments and allow $100 for slippage and commissions.

Consider for a moment what the $0 initial stop means. The system goes long or short on the close. Thus, the trade will remain open only if prices continue to move strongly beyond today's close. This is the toughest stop you can impose because the only trades that survive are the ones that are profitable immediately.

Observe that profits increase steadily as we loosen the initial stop (see Figure 3.3). There was a surprising profit of $158,103 with a $0 initial stop on just 20 (of 434) trades. This confirms a common piece of market wisdom that the best trades are profitable immediately. It also confirms that only 5 percent or so of the trades are the 'big ones.' So you should work hard not to miss them.

Figure 3.4 shows that a tight stop can produce a drawdown greater than using no stop at all. More and more trades recover their losses and

Changes In MIDD for 20-day CHBOC on Coffee

Initial stop ($)

Figure 3.4 As we loosen the initial stop, MIDD first increases and then stops declining.

56 Foundations of System Design

close at a profit as the stop widens. Eventually the stops are so large that they have little effect, and so MIDD stabilizes.

The initial stop cuts off fewer trades as we loosen it (see Figure 3.5), and hence the total number of trades produced by CHBOC decreases. Once the stop is 'too loose' (more than $3,000 or so), it has little effect, and the number of trades stops declining.

Only 5 percent of the trades are profitable with a $0 stop. The percentage of winners increases quickly as we loosen the initial stop until the stop has little effect (see Figure 3.6). As we loosen the stop, more of the winning trades can survive the vagaries of market action.

As you may expect, the worst losing trade increases as we loosen the stop (see Figure 3.7, page 58). This occurs because the worst case with a $0 stop reflects slippage due to a weak opening. However, as we loosen the stop, the losing trade from a false signal can survive longer.

The highest average 10-day trading range in the coffee market over the last 20 years was approximately $5,025. The average value was $1,015 and the standard deviation was $641. The cumulative distribu-

Number of trades for 20-day CHBOC on Coffee

Initial stop ($)

Figure 3.5 The number of trades drops and levels off as we loosen the initial stop.

Initial Stop: Solution or Problem? 57

Changes In percent profitable trades, 20-day CHBOC on Coffee

Figure 3.6 The proportion of profitable trades increases and levels off as we loosen the initial stop.

tion (Figure 3.8, page 59) shows that a stop of $3,000 exceeds 98.3 percent of all the 10-day average trading range values seen in coffee over the last 20 years. Hence, $3,000 should be a loose stop. Figures 3.3 through 3.7 show that the changes in performance begin to level off beyond $3,000. Thus, you can view stops greater than $3,000 as 'very loose' stops. A $500 stop that covers less than 20 percent of all observed values of the 10-day average daily range qualifies as a 'tight' stop.

You can now use the cumulative frequency distribution to select a stop based on market volatility. An arbitrary stop may be too tight or too loose. This analysis assumes that you use the same dollar stop on every trade. If you vary the initial stop on every trade then this analysis will be of little use to you. We already know that stops are hit more frequently during trading range markets. Hence, you could use some measure of trendiness to vary your initial stop.

Many traders feel an aversion to taking a big loss, even though they have no problem taking many small ones. The maximum drawdown usually decreases as the stop increases (see Figure 3.4). Thus, you should

58 Foundations of System Design

Variation In biggest losing trade: 20-day CHBOC on Coffee

CM CM CO CO 'tf T 1^ 10 <D tO f*- h-.

-1000

-2000

-7000

-8000

-9000

Initial stop ($)

Figure 3.7 The worst losing trade increases as we loosen the stop.

try to take the long-term view when you set your stops. If you use a constant stop based on system design, then use loose stops. If you set the stop differently for each trade, then you have probably mastered the fine art of placing stops.

The risk of being stopped out is highest near trade inception, as shown by the calculations in Table 3.11, page 60. This table shows the effect on the length of the average losing trade of using no stop, a $1,500 stop, and a variable stop. A simple 20-day CHBOC model, with no exits other than an initial money management stop, is used, allowing $100 for slippage and commissions. The tests were over a 6-year period commencing May 26, 1989, using continuous contracts.

The data in Table 3.11 show that inserting an initial money management stop of $1,500 reduced the length of the average losing trade by approximately 40 percent to 17 days from 28 days. These calculations confirm that the risk of being stopped out is highest near trade inception. The average winning trade was typically 2 to 3 times longer than the average losing trade.

Initial Stop: Solution or Problem? 59

Cumulative frequency distribution average 10-day daily range in coffee

250 750 1250 1750 2250 2750 3250 3750 4250 4750 5250 Range ($)

Figure 3.8 The cumulative frequency distribution of the 10-day average daily range shows that an initial stop of $3,000 or more covers 98.3 percent of all trading ranges.

If you look more closely at Table 3.11, you will see that for some markets, such as gold, sugar, and soybeans, the length of the average losing trade did not decrease much even after adding a stop. This means that the volatility in these markets is not as large as, say, the currency or bond market. An approximate initial stop that will produce an average losing trade length of 10 or 11 days is also shown in Table 3.11. The S&P-500 index futures contract and coffee were the two most volatile markets, followed by cotton, Swiss franc, and the U.S. bond markets. Conversely, gold, sugar, and crude oil were relatively less volatile. Hence, you may find it useful to consider overall market volatility when placing your initial stop.

In summary, you can get a better feel for system performance if you use loose stops with a self-correcting system. If a stop is 'tight,' then a small change in the stop can affect long-term performance. If a stop is 'very loose,' then changing the stop will have little effect. As you loosen your initial stop, the profits increase and then change more slowly. This

60 Foundations of System Design

Table 3.11 Effect of initial money management stop on length of average losing trade

means that once you pass some volatility threshold, increasing the initial stop adds little value.

Another reason to use loose stops is that you cannot properly test stops that are smaller than the daily price range. Ideally, you should base your initial stop on money management guidelines, the maximum adverse excursion of the system, and on market volatility. There are many ways to select an initial stop; once you pick a method, you should use it consistently.

Does Your Design Control Risks?

As you design your trading system, remind yourself that one of your key goals is to control the downside risk. You will quickly discover that risk is a many-splendored thing. This section briefly discusses some of the areas of risk you may wish to consider as you take a portfolio-level look at your trading system.

Does Your Design Control Risks? 61

A trailing stop is a popular method to control portfolio volatility and protect profits. A trailing stop is simply a stop order that is placed some fixed distance away from the highest profit point in the trade. When the market reverses, or when market volatility increases, this stop will be touched off and will protect your profits. If you are using long-term systems that are slow to react to trend changes, then such a stop may smooth out your equity curve.

An important type of risk arises from correlation among markets. You know that correlated markets move roughly together. A good example is the currency markets such as the Swiss franc and deutsche mark (see Figure 2.10). These markets tend to experience broadly similar moves versus the U.S. dollar. As we saw in chapter 2, trading correlated markets in the same portfolio is equivalent to trading multiple contracts in a single market. This may increase your risk to market events such as unexpected and unexplained news events.

There is an execution risk to your portfolio due to market liquidity or lack of it. For example, lightly traded markets can produce significant slippage. You. experience slippage getting in and out, reducing profits, and increasing losses. In these markets your paper testing may not adequately account for slippage and commissions, thereby overestimating potential profitability.

Liquidity can be a particular problem near major holidays, such as Christmas and New Year's Day. During these thin market periods, it is common to see large one-day moves (see Figure 3.9) that can scramble the best-laid risk-control plans. These moves do not change the underlying trend, but can be difficult to model when you test your trading system.

Global trading produces a new set of risks to your portfolio. If news events occur when the U.S. markets are closed, then large price moves could occur in foreign markets. This is particularly true for currencies such as the Swiss franc, Japanese yen, or deutsche mark; energy markets such as crude oil; and metals markets such as gold and silver. Often, an emotional reaction in foreign markets will produce a large opening gap stopping you out at extraordinary slippage. You may find that your profits are lower than anticipated due to these large opening gaps. Then to make matters worse, the markets may stage a recovery to close well inside your stop loss point. Thus, round-the-clock trading adds new risks to your portfolio.

The DM contract in 1995 showed some large gaps during a volatile period (see Figure 3.10, page 63). Large overnight moves in foreign currency markets produced these large gaps, which are difficult to simulate correctly in historical testing. The first encircled gap was for $2,112.50,

62 Foundations of System Design

Dec

Figure 3.9 Increased volatility in the British pound market caused by thin markets at year end 1994 is clearly visible.

a big move against you if you were short. The island reversal in the middle ellipse in late March 1995 also left huge gaps, about $1,300 per contract. The gap circled in May was about $1,500. Here, your signal a day off on either side would show significantly different results.

The large intraday ranges in this contract also increases the difficulty with entering a market on the close. For example, you may lose a big move if you had the right signal on the right day, but entered the market on the close rather than on a stop. Say you had a sell order at 71.80 stop close only. ^bur fill would have been after a slippage of $2,400, quite unacceptable to most traders (see Figure 3.11).

The type of data you use often poses hidden risk. Consider a situation in which you are using weekly data to develop your trading system. Let us suppose you generate a signal at the Friday close, and purchase with a delay on Tuesday open. Since daily opening gaps are missing from weekly data, you can easily underestimate the slippage from actual trading. Another potential problem area is using systems that generate signals this week and ask you to trade next week. You could have a large move this week, and have missed a big portion of the profits by the time you enter the trade next week.

Your system could also experience a time-based risk. For example, the best moves seem to occur when the market moves rapidly immedi-

Does Your Design Control Risks? 63

Figure 3.10 The large gaps are due to big moves overnight in overseas markets. Note the large close-to-open gaps in the bars highlighted by the ellipse in the middle of the chart.

16

Figure 3.11 A stop close only sell order would have a $2,400 slippage due to market volatility.

64 Foundations of System Design

ately after a signal. Suppose the market consolidates immediately after giving you a new breakout signal. The risk of being stopped out is significantly higher in a sideways trend. Hence, you may want a filter that will exit within 5 days of entry if the trade shows a loss.

Another quirky situation arises when you get a new signal very close to a rollover date. It is possible to generate an entry signal on the contract about to expire, but not on the next active futures contract. In this case you must decide whether to take the signal as is, and then rollover immediately or in a few days, or just to wait until the next active contract generates its signal.

In testing with continuous contracts, you could easily underestimate the effects of rollovers on trading costs and profitability. You must also resolve the issue of where to place your initial stop on the new rollover position. Your real position may hit the stop, while your continuous contract merrily rolls along with its position intact.

This discussion does not include all types of risk, but highlights why you should consider risk control early in your trading system design process.

Data! Handle with Care!

You have many choices when you select data for your system testing. You should therefore exercise great care in choosing your test data because they have a big influence on test results.

Choose your data vendor with care, since data vendors differ in accuracy, depth of coverage, and reporting conventions. For example, there can be differences in the opening price of a contract between data vendors. Another policy difference is how errors are detected and corrected. You want a vendor with 'clean' data all around.

If you are testing futures contracts, use a continuous contract or the actual contracts with rollovers. Unlike a cash market index, futures data are not continuous: contracts are dormant, become active, and expire. Hence, to produce a continuous, albeit 'synthetic,' data stream, different contracts must be combined in some consistent manner. You have two choices when you combine futures contracts. You can preserve the prices or price levels, at the cost of correctly preserving the amplitude of price movements, or, alternately, you can preserve the amplitude of price movements at the cost of adjusting prices. Continuous contracts that preserve amplitudes are preferred for testing. Each type of continuous contract will give slightly different numerical results.

Data! Handle with Care! 65

A good choice is using actual data with rollovers if the software allows it. Results from continuous contracts come closest to data with rollovers, but they tend to underestimate the number of trades and hence the losses due to slippage and commissions. They also underestimate the difficulty with placing stops on rollover trades. For example, after rollovers, you can get stopped out on a real trade, while the system trading the continuous contract continues to hold its position.

Other than type of data, you must also choose the amount of data. In general, the more data you can use, the better, because you can then test a model over a wide variety of market conditions. There is also a statistical requirement, usually quoted as sufficient data to produce 30 or more trades over the test period. The idea here is simply that the more trades you have in a given sample, the better your estimate of the average properties for the entire universe. Hence, you can use the average properties (such as average trade) to estimate how the system will do in the future.

Although 30 trades may not seem like a large number, you may have difficulty obtaining 30 trades with weekly data using certain trend-following models. Thirty trades is probably too small a number to fully eliminate any effects of the data used for the tests. Hence, more than 100 tests is preferable, if possible, and you can obtain sufficient numbers by combining tests over multiple markets and multiple time frames. Later in the book you will learn how to generate synthetic data to generate a large sample of trades.

The point of this discussion is that different input data will give you different results. Besides, you cannot be certain that your trading will ever achieve those results because you do not actually trade the synthetic data. We now look at the results of testing a dual-moving average crossover system using 5-day and 50-day simple moving averages, an initial money management stop of $1,500, and $100 for slippage and commissions.

Table 3.12 was created by Continuous Contractor software from Tech Tools, Inc. to show different types of continuous contracts for the Swiss franc. Over the same period, model performance was calculated using actual contracts with automatic rollover on the twenty-first day of the month prior to expiration. The cautionary tale here is that test results vary widely with the type of data used.

Focus on the number of trades for the same entry and exit rules. There are 111 trades with the actual data, almost 37 percent more than the average number of trades created using continuous contracts. The extra trades result primarily from rolling over existing positions into the next active contract.

66 Foundations of System Design

Table 3.12 Comparison of test results with different types of continuous contracts

Profit MIDD of Wins Win/Loss Data Type ($) ($) Trades (%) Ratio

Actual with rollovers 17,963 -21,663 111 40 1.80

Continuous type 38/13 18,450 -24,813 79 31 2.74

Continuous type 49/25 20,413 -22,137 77 31 2.89

Continuous type 55/25 20,350 -21,115 86 34 2.42

Continuous type 56/25 10,625 -27,800 91 31 2.43

Continuous type 60/25 39,862 -18,363 70 35 3.12

Notice also that the ratio of average winning to losing trades is the smallest (1.80) with actual data. This occurs because rolling over positions decreases the amount of profit in winning trades. During a long trend the continuous contract goes on without any rollovers. Hence, the duration of the trade is longer and therefore its profit figure is also greater with continuous contracts. Thus, continuous contracts may underestimate the number of trades and overestimate the win/loss ratio and the duration of trades.

Choosing Orders for Entries and Exits

You have three basic choices for orders that you use to initiate or exit your trades: market, stop, or limit orders. There are three philosophies at work here. One says to get your price, implying you should use limit orders or stop limit orders to get into the trade. Another philosophy says to guarantee entry into trade, implying you should use market orders or stop orders. The third philosophy insists that you should exit positions with market orders, but can enter on a stop or a limit.

Timed-market orders on the open or the close are a good way to both exit and enter positions. Many traders recommend entering on the open, and avoiding the rush of orders at the close. Using stop orders can cause you extra slippage if the market opens beyond the stop price.

Remember that there can be divergences between what the testing software assumes and what actually happens in practice. You may get a fill from the software, but not from the floor due to the peculiarities of the market. For example, the software testing daily data has no idea if there were fast market conditions on a particular day. Hence, you may or

Understanding Summary of Test Results 67

may not get a fill when the software says you did. It is important to understand how the software fills a particular order.

One other important feature is the difference between the signal day (or date) and execution day (or date). This means you can get your signal and open the trade the same day. For example, if you are trading a simple moving-average crossover system, you can calculate that a close beyond a particular point will give you a crossover today. Hence, your order may say buy (or sell) at x, stop close only. Alternatively, you can generate your signals after the close, and enter them the next day. The latter approach is preferable only because it is simpler, and when coupled with an order to enter on the open, it is a reliable way to measure system performance.

Entering tomorrow on a stop above the high or low of today is an effective filter when compared to buying tomorrow on the open. In effect, it filters out some whipsaw trades. Generating orders today and entering on the next day on a stop, on the open or on the close, is a consistent and realistic way to assess model performance.

Understanding Summary of Test Results

This discussion of the detailed summary of test results found in technical analysis programs uses in part the report from Omega Research's TradeStation software. The purpose of the summary is to show how a particular trading system would have done on historical data.

The summary shown in Table 3.13 is for the British pound continuous contract for the 65-day simple moving average, three consecutive closes (65sma-3cc) trend-following system. The 65sma-3cc trading system is discussed in detail in the next chapter. The summary here is for all trades, long and short. The software shows the same information for long trades only and for all short trades.

The summary is broken down into five blocks. The top-most block describes profitability. The second block gives physical trade count. The third block presents average trade data. The fourth block shows trade duration or length data. Finally, the fifth block gives important information on drawdowns, profit factor, and returns.

The total net profit is the difference between the gross profit and gross loss. The gross profit is the sum of the profits on all profitable trades. Similarly, the gross loss is the sum of losses on all losing trades. The open trade profit or loss is the value on a trade still open at the end of the test period. The net profit is an important figure that influences

68 Foundations of System Design

Table 3.13 Typical performance summary for 65sma-3cc system

Total net profit ($) 155,675.00

Gross profit ($) 266,918.75

Total number of trades 71

Number of winning trades 32

Largest winning trade ($) 40,768.75

Average winning trade ($) 8,341.21

Average win/average loss 2.92

<

Maximum consecutive 6

winners

Average number of bars in 123

winners

Maximum intraday -27,881.25

drawdown ($)

Profit factor 2.40

Account size required ($) 30,881.25

Open trade profit or loss ($) -1,212.50

Gross loss ($) -111,243.75

Percentage profitable 45

Number of losing trades 39

Largest losing trade ($) -7,993.75

Average losing trade ($) -2,852.40

Average trade (win and loss) 2,192.61 ($)

Maximum consecutive losers 7

Average number of bars in 29 losers

Maximum number of 1

contracts held

Return on account (%) 504

other calculations below. Note that the profit factor shown in block five is simply the absolute value of the ratio of gross profit divided by gross loss. In trading system design, a profit factor of more than 1 is highly desirable, since it says that gross profits exceeded gross losses over the test period.

The trade count block shows the total number of trades, and the breakdown into number of winning and losing trades. The percentage of winning trades is a function of both the trading system rules and the test data, and helps influence the risk of ruin. Naturally, the larger this number, the better. It is common to have trend-following systems report in with a winning percentage of 30 to 50 percent. A number above 60 percent is difficult to find, and anything over 70 percent is remarkable.

The average trade performance block merely combines data from the two blocks above to report average numbers. The largest winning trade and largest losing trade are new numbers in this block. They are usually functions of the test data, trading system rules, and risk control specifications. If you do not use stops and the markets are volatile, there will be a large losing trade. Exceptional trends can give you a large winning trade. Beware if the largest winning trade is more than 50 percent

Understanding Summary of Test Results 69

of your net profits. It probably means you should deduct this amount from net profits to evaluate true system potential.

The average winning trade is simply the ratio of gross profit divided by number of winning trades. The ratio of the average winning to average losing trade is useful for calculating risk of ruin. This is called the payoff ratio, and is a function of the test data, trading system rules, and the length of trades. The typical trend-following systems will return values greater than 2.

The average trade reported in the third block is one of the most important numbers in the summary. It is simply the ratio of net profit divided by the total number of trades. This number depends on the test data and trading system rules. This number would ideally be as large as possible. If this number is negative or less than $200, avoid trading this system unless you test it on other markets and other time frames. This number is the statistical edge for this system.

The trade duration block gives the length of the average winning and losing trades (average number of bars in winners equals length of average winning trade). This ratio should be greater than 1, and it could be greater than 5 for trend-following systems. Ask yourself if you would be comfortable holding a trade for the number of days shown in the length of average winning trade. Do you have the discipline to stay with a trade that lasted twice as long as the average winning trade? If you are not patient, this may be a difficult task, and you might miss out on a mega-trade.

Alternately, ask if the length of an average winning trade coincides with your trading horizon. If the length of the average trade it is too long or too short, test the system first over more data and then over other markets. If you are still not comfortable with this number, you should consider changing your trading system.

The maximum consecutive winners and losers data will vary with the test period. Maximum consecutive losers have a great influence on your drawdowns. You should carefully examine the period when the consecutive losers occur to understand under what conditions your trading system will produce large losses.

As a rough rule of thumb, ask yourself if you could tolerate twice the number of consecutive losers as the number reported for maximum consecutive losers. This will tell you how to set your money management guidelines to avoid serious drawdowns. Ask yourself also if you would hold a losing trade as long as the average losing trade number suggests.

The last block shows the maximum intraday drawdown. Ask yourself if you could tolerate a number twice as large. The account size and

70 Foundations of System Design

return on margin numbers are not very useful. The profit factor, as discussed above, should be greater than 1.

What the Performance Summary Does Not Show

The test summary leaves out some important information, highlighted below. You may wish to examine these factors in greater detail.

One simple ratio is the recovery factor (RF). RF is absolute value of the ratio of net profit to maximum intraday drawdown, and it measures how far you recovered from the depth of the drawdown. In Table 3.13 the recovery factor is approximately equal to 5.6 (155,675/27,881). This number should be greater than 2, and the higher the better. It tells whether the potential benefits over the long haul are worth the aggravations caused by the drawdowns.

Another useful value is the adjusted gross profit, in which the largest winning trade is deducted from the gross profit. To penalize the system, do not correspondingly deduct the largest losing trade. The rationale here is not to expect to get the periods with large profits, but that a period of losses comparable to the worst losses in the test period is likely. The profit factor is then recalculated to see if it is still greater than 1. For the data in Table 3.13, the adjusted gross profit is $155,675 - 40,769 = $114,906. The adjusted profit factor is then 114,906/111,244 = 1.03. This is a sharp reduction from the reported profit factor of 2.40. Thus, a more realistic assessment of this system is that it will produce a small net profit over time.

The summary also does not give a histogram of your trades. You may wish to export your data to a spreadsheet to look for the maximum favorable excursion and maximum adverse excursion. These quantities will be explained in chapter 4 with the 65sma-3cc system.

The summary does not give you any feel for the variation in test results. It does not give a standard deviation of trade profits and losses for all trades. The variability is another important item you should calculate, using a spreadsheet if needed. The variation tells you what you can expect for volatility of returns.

You cannot get an idea of how a typical trade evolves in time from the test summary. For example, it does not tell you the average profit or maximum profit or loss on a day-in-trade basis. It does not show what happened on day 1 in the trade, or day 10 in the trade. A typical trade template, by Chande and Kroll, as discussed in The New Technical Trader

A Reality Check 71

(see bibliography), would help you understand the time-price evolution of a typical trade.

In addition, the test summary does not give a realistic impact of slippage. The software provides fills in a manner that may not be representative of fills in the real world. It is safer to assume that you will experience greater slippage than the model. In some instances, the software will give you a fill that you could not have obtained in practice. If this happened to be a big winner, you may overestimate trade profitability. Hence, you are better off using the average trade numbers to assess system performance, since they have averaged out the effects over many trades.

The performance summary also does not give any idea of how many successive .v-month periods would have been profitable. For example, it is useful to know how many successive 6-month periods have been profitable over a 5-year period. You could use any time interval you like. This breakdown tells you how quickly you can expect to get out of drawdowns, and is a vital piece of information for your mental approach to trading the system.

The most important factor to recognize is that the test summary does not tell you how the system will perform in the future. Your test results are hostage to your data. You should look below the surface of the results to get a better understanding of your system tests.

Ideally, you should examine the results on a trade-by-trade basis on the charts to understand how your system rules worked. This will reinforce your trading beliefs, and give you a good feel for when the system does or does not work. A study of unprofitable trades often reveals flaws in your logic. Convince yourself that you want to follow this system because its rules make money under market conditions that are likely to repeat in the future. A trade-by-trade review may also strengthen your ability to use discretion in trade entries or exits.

A Reality Check

This section sounds a note of warning before you proceed: Test results are not what they seem. You should recognize that trading systems are designed with the benefit of hindsight. This is true because you know, a priori, what the market has done in the past. Any trading system you design or optimize reflects your view of past market action. You may state your understanding in a generalized way that avoids the dangers of

72 Foundations of System Design

curve-fitting. However, it is worth recognizing that the influence of hindsight is difficult to eliminate.

It is also important to recognize that past price patterns may not repeat in precisely the same way. Hence, because the exact future sequence of trades is unpredictable, your system may not achieve profits or losses similar to the hypothetical system. It should be easy to conclude that past results are not indicative of future results because neither market action nor trader reaction is predictable.

There is another key problem area with simulated trades. Hypothetical trades from a trading system design exercise have not been entered in the markets and do not represent actual trading. They do not accurately reflect the effects of market liquidity, slippage, bad fills, overnight trading, or fast markets. They also do not reflect a trader's psychology accurately since each and every signal is assumed to be executed with identical simplifying assumptions.

You, the trader, are perhaps the most capricious variable in the trading system. Because system testing is performed in an emotional vacuum, there is no assurance that you will execute all signals from a trading system without deviation. Thus, the biggest slippage could occur not in the markets, but at the source if you fail to enter orders as required.

As you will see in chapter 8 on data scrambling, it is possible to encounter market conditions that generate a long string of losing trades or one huge loss. Just because the probability that an event occurs is very small, this does not mean that it will not occur. The usual distribution of trades from a typical trading system has 'fat' tails. This simply means that the probability that unusual market conditions will occur is much greater than you might expect from a normal distribution. Hence, system testing results will often underestimate market risks.

Thus, when you design trading systems, be aware that your hypothetical results do not accurately predict system performance in the future. In general, you should view any trading system results with all due caution.