Dam risk and the owner’s dilemma15 September 2000
Recent literature on risk analysis for dams suggests that the historic controversies concerning practical quantification of risk have yet to be resolved. Des Hartford takes a look back at why the presently unacceptable situation has arisen, and charts the future path necessary to establish sound risk analysis which can be used confidently in dam safety practice
Dam risk and its management has always been inherent to good dam safety practice, albeit implicitly. The idea of explicitly considering dam risk and making safety assessments on the basis of risk, although first proposed by Casagrande over thirty five years ago1, gained prominence in the 1990s; as did the controversies surrounding the practice of risk assessment. That quantification of risk might be difficult was recognised at that time with one experienced observer noting that what Casagrande was proposing was beyond the capability of most in the profession.2 The 1970s saw attempts to apply economic risk analysis concepts to decision-making concerning spillway adequacy. They were restricted to cases where there was no threat to life, the idea of assigning a monetary value to life in economic risk analysis having been dismissed in 1973.
There was little in the way of progress in the quantification of risk posed by dams during the 1970s and by 1980 one view held that it might be logically impossible to analytically derive failure probabilities for dams.3 It was proposed that the base rate of dam failures of 10-4 as derived from the historic record should be used as a default value in benefit-cost analyses of dams. Clearly this was better than the value of zero used up to then and there were certain qualifiers concerning the distribution of costs and benefits.
This proposal was not left unchallenged4 and well founded objections to the idea of using the historic base rate failure frequency, as a basis for estimating the probability of failure of an individual dam, were raised. Dr Rasmussen, of nuclear industry fame, is even reputed to have remarked that not enough is known about dams to perform a risk analysis.5 Around the same time (1983), and in response to the difficulties of analytically determining dam failure probabilities, a so-called judgmental technique emerged. In terms of the judgmental approach ‘the investigator attempts to quantify his judgment based on all available information. The judgmental statement may be made directly in terms of annual probability of failure of the dam due to a particular condition (eg probability of failure due to internal erosion = 1x10-3 annually), in terms of the chance of failure over a specified remaining operational life of the dam, or as a fraction of the probability associated with other modes (eg about twice the risk attributable to flooding and overtopping.’6 Two distinctly different philosophies concerning how to estimate probabilities of failure of individual dams emerged in the early 1980s — the analytical and the judgmental — with apparently irreconcilable differences between the two philosophies. The analytical view held that it was not feasible, using the techniques of the day, to analytically derive failure probabilities and that in some cases it might be logically impossible. Whereas the judgmental view held that it was always possible to know all failure modes and mechanisms and judge the associated probabilities. To further complicate matters, there were statistics of dam failures with the result that risk-based dam safety concepts involved three types of probability: analytical, empirical and judgmental with a hybrid empirical/ judgmental approach being deemed the most practical6 at that time.
The literature account of the judgmental approach6 provides no guidance as to the process for arriving at the judgement of failure probability. Further, there was no indication as to the process whereby the empirical record could be adjusted to estimate failure probabilities for individual cases. In the 1980s the ‘empirical/judgmental’ approach found limited use in risk-based decision analysis at the US Bureau of Reclamation, and it served a useful purpose by providing a basis for comparing decision options. The approach did not find widespread application by regulated dam owners, in part because of the reservations of dam safety regulators.7 Furthermore, any need for analytical rigour or mathematical correctness of the interpretation and use of probabilities did not necessarily pose insurmountable problems if mathematical ‘problems’ were consistent between options because the results were not used as an absolute measure of risk and if mathematical ‘problems’ were consistent between options. Risk-based approaches to decision-making are distinctly different to what is now regarded as a risk assessment. Unfortunately, the term risk assessment was used at that time to describe what was in fact a risk estimate.6 Like the controversy concerning the two philosophies of probability, controversy caused by confusion in the use of risk terms in dam safety originated in the early 1980s.
In 1984, building on the work of Casagrande and recognising the problem of adequacy of experience to estimate risks, Professor R V Whitman provided, for discussion purposes, the first indication as to how risks posed by dams might be estimated8 in practice. Importantly, the proposed procedure was de-compositional with event trees being used to illustrate the failure mechanism. In addition to providing a basis for determining failure probabilities through analytical means, Whitman also provided the framework for what would now be termed a quantitative risk assessment. Although Whitman’s suggestions provided a bridge between the analytical and empirical/judgmental philosophies, the empirical/judgmental approach was the only technique adopted in the applications of risk-based concepts that prevailed in the 1980s.6,7,9 At the end of the 1980s there were clear, and to a degree irreconcilable, differences in professional opinion concerning how best to manage the safety of dams — traditional (deterministic standards) or risk (probabilistic). There were also important differences of opinion between proponents of the risk assessment approach.
These differences of opinion were divisive and a source of great controversy. They also created serious difficulties for owners of dams regardless of the size of the dams or their financial capacity to control the risks inherent to the structures. In the absence of regulatory acceptance of risk-based approaches, there was no incentive to do anything more than apply the risk-based decision approach within the context of accepted deterministic practice.
The owner’s dilemma was becoming increasingly serious. They were trapped in a vicious circle of ever-increasing dam safety standards, spiralling costs of incremental safety improvements, and market driven demands for improved economic efficiency. The divisions within the profession further exacerbated the problem, as those working in the field of dam safety had thus far been unable to provide a satisfactory solution that would address owners’ often conflicting economic and legal concerns and which would be acceptable to the courts, prevailing regulatory instruments and the public. This said, it would be most inappropriate to infer that the dam safety profession is at fault. The issues are complex and the resolution of the owner’s dilemma goes beyond engineering.
The owner’s dilemma
The owner’s dilemma is not a new phenomenon, and has been around in various forms since dams were first constructed; manifesting itself very clearly every time a dam failed in the past. The failure of the Dale Dyke dam and subsequent events in England in the 1860s10 provide a valuable illustration of one aspect of the owner’s dilemma.
It is possible that current trends in the corporatisation and/or privatisation of dam ownership might not fully appreciate the nature of the owner’s dilemma and the lessons learned from failures such as that of the Dale Dyke dam failure.
In the early 1990s the Australian Committee on Large Dams (ANCOLD) and Canada’s BC Hydro independently began to address the owner’s dilemma. According to ANCOLD: ‘the immediate objective of many of those advocating risk assessment in current practice is to provide defensible design solutions as economic optima that are likely to be of lower cost than those that result from a traditional engineering standards approach to design.’11 The judgmental approach to risk analysis that emerged in the 1980s and was practised by some owners in the 1990s.
The question — does the judgmental approach resolve the engineering component of the owner’s dilemma? — naturally arose. Unfortunately, the modern literature on risk analysis for dams is decidedly unhelpful in this regard as two opposite views, together with all possible views in between these two extremes, are frequently presented. At one extreme there is the view that methodologies for estimating the chance of dam failure are poorly developed and, at the present time, do not provide a defensible basis for the conclusive sign off on the safety status of a dam.12 At the other extreme, there is the view that methods are available for estimating the probability of failure of dams for use in quantitative risk assessment for all failure modes.13 The former view dominates the literature and there is a significant body of supporting empirical evidence. In the light of such diversity of opinion, owners can not rely on emerging approaches to risk assessment to resolve the owner’s dilemma. In fact, the modern literature on quantitative risk assessment for dams is increasingly suspect as there is clear evidence that it is being contaminated by recycled ideas, many of which have been previously debunked and others that can be classed as recycled and wrong. This adds a new dimension to the dilemma, one which owners need to be aware of if they rely on the literature and/or those who write on the subject.
Clearly, the owner’s dilemma must be addressed and engineering is central to its resolution. Risk analysis is the engineering component of risk assessment and the question as to whether or not risk can be estimated in a reliable way must be answered. Engineers can begin by answering the following question: is a judgmental estimate of risk derived using the expert opinion method equivalent to the estimate of risk derived through analytical procedures? A detailed example of the former, which can be completed in a matter of two or three weeks.14 A condensed example of the latter, which took teams of subject matter experts three years to complete, is summarised in figures 1a-g.15 Certainly, there is no equivalence in cost and level of effort. However, in searching for an answer to the question of equivalence of estimates of risk, experience in applying both methods is obviously beneficial.
Unfortunately, this simplified and apparently practical approach is not as straightforward or practical as it appears, if one abides by the rules of the exercise of judgement in engineering and the principles of estimating probabilities. One has to look no further than the Challenger Space Shuttle accident to question the validity of engineer’s judgements of probability and see how tragic the consequences should the advice lead to a bad decision. Judgmental probabilities are more correctly termed subjective probabilities, where the theory of subjective probability forms an entire branch of the mathematics of probability.
Objective (empirical frequency) probabilities and subjective probabilities, are not the same types of probability. The simplest way of obtaining objective probabilities is to perform repeated trials, whereas the simplest method of obtaining a subjective probability is to simply ask the person what their degree of belief is. After all a subjective probability is just someone’s opinion.16 However, asking someone what their degree of belief is, is equally surely the worst way of obtaining subjective probabilities.16 The challenge was and still is to ensure that the probabilistic statement of opinion is well founded.
The apparently practical empirical/ judgmental approach to probability estimation was clearly some kind of hybrid of two different concepts. Fundamentally, quantitative risk assessment of the form that was proposed in the 1990s11,17 requires an objective statement of the probability of failure of individual dams for comparison with objective criteria. The empirical record, which characterises the properties of the population of dams, can never be indicative of the probability of failure of individual dams (statistics do not apply to individuals). Also, judgmental probabilities are subjective — they have no objective meaning and only exist in the mind of the person making the estimate. The hybrid as applied to individual dams can not have an objective meaning as it is made up of two parts, neither of which exists in the real world.
For subjective probabilities to be well founded, it is important that those making the estimates have substantive (subject matter) expertise and normative expertise, ie be well calibrated.18 They should have basic training in probabilistic reasoning (a notoriously difficult discipline16) and be familiar with the mathematics of probability. They should also know how to transform evidence and experience into coherent statements of probability. Finally, because of the extent to which judgement pervades dam engineering, they should be people of judgement.
If anything, given the extensive role of judgement in engineering practice, the judgmental quality should have been the most readily available. Di Biagio and Høeg19 provide a very eloquent account of where judgement in engineering comes from. Together with the experience in estimating subjective probabilities outlined above, they provide an insight into the make-up of project teams charged with assigning judgmental probabilities of dam responses. Therefore, the team should be made up of: •People of judgement: who have theoretical knowledge, experimental evidence, empirical experience and the ability to integrate all elements of the exercise in a logical and transparent manner.
•People with substantive expertise: who are subject matter experts in each of the physical processes involved in the failure mechanism.
•People who also have normative expertise: who are demonstrably well calibrated (have a good track record in guessing right), trained in probabilistic reasoning; and have detailed knowledge of the dam under investigation, its properties, its design and performance and its vulnerabilities.
Clearly, there is an enormous difference between asking someone to quantify their ‘opinion’ about the outcome of an event concerning a dam, and asking properly qualified people of judgement (in the accepted engineering sense) to transparently transform their knowledge, experience and evidence into their subjective probability of the event. Obviously, this is not routine engineering and certainly not commodity engineering. The expert opinion approach is fraught with difficulties — risk analysis experts outside the dams business do not admit opinion. According to Kaplan20: ‘we should never ask an expert for his opinion. What we want from an expert is his experience, his information, his evidence.’
Judgement of probability versus opinion
The difficulties of making judgements of probability (as opposed to asking someone’s opinion) are manifestly obvious. Such judgements of probability are very uncertain because the data are scarce and difficult to relate to the case at hand (all dams are unique). Moreover, it is debatable whether it is really judgement that is being exercised, because those who make these estimates usually lack rational mechanical models on which to base their beliefs. The dam safety analyst must estimate (ie assign) probabilities subjectively in part, often describing them as engineering judgements of probabilities. Such guesses are used in lieu of probabilistic estimates of parameters and states made by means of observations and rational physical models. These guesses contaminate the risk analysis, degrading quality. In the absence of realistic theories and mathematical structure, anyone can say anything, and in accordance with some interpretations of the theory of subjective probability they would all be right.
The practicality of making judgements of probability concerning the failure of dams clearly poses an enormous challenge. Setting aside any complexities associated with the mathematics, as they are numerous but essentially tractable if appropriate mathematical expertise is brought to bear on the problem, the most significant practical difficulties are found to be with the exercise of judgement concerning the behaviour of dams at or near failure.
The exercise of judgement in engineering practice requires an adequate theoretical basis, validated by experiments and observations. The role of theory in practical engineering is well established and its importance has long been recognised in dam engineering. Professors W J M Rankine and K Terzaghi (practical engineers of undoubted excellence) were acutely aware of the need for adequate theories and their role in practice.21 The advice provided by Terzaghi concerning the role of theory in geotechnical engineering practice and the role of rigorous and simplified solutions is also applicable to the exercise of judgement in estimating failure probabilities of dams.
According to Terzaghi: ‘The ability to obtain rigorous solutions is not a prerequisite for successful work in the field of soil mechanics. For both the research man and the practising engineer it is sufficient to know the general procedure by means of which the rigorous solutions are obtained. The rigorous solution of the problems should be left to professional mathematicians.’ However, Terzaghi made it perfectly clear that practice is based on theory and that adequate theories are a necessary part of practical engineering and the exercise of judgement in engineering practice.
Immediate observations include: •Estimating risks posed by dams requires adequate analytical theories to provide a basis for developing practical approaches to analysing risks.
•The empirical record of dam failures and incidents can not in itself provide an adequate basis for inferences concerning failure risks in individual cases.
•If judgmental probabilities were to be genuinely characterised as judgements, in the established engineering sense as opposed to quantified statements of opinion, they would be based on analytically derived predictions. In other words, the apparent differences between the analytical probability philosophy and the judgmental probability philosophy cease to exist if proponents of the judgmental probability philosophy adopt the established approach to exercising judgement in engineering practice.
In the absence of adequate theories of dam failure mechanisms on which to base judgements of probability, and given the philosophical problems associated with mixing objective and subjective probabilities, the empirical/judgmental approach to estimating probabilities of dam failure becomes an intellectual and philosophical failure. The subjectively estimated probability of failure of a dam is no more a property of the dam than the betting odds on a horse are a property of the horse.
Traditional dam safety practice relies heavily on the judgement of knowledgeable engineers, whose practice is firmly grounded in sound, empirically validated and professionally accepted theories. It might be argued that engineering practice that is devoid of adequate theories and effective means of demonstrating that solutions are viable is arguably not engineering, and judgement that is exercised in the absence of evidence, experience and a line of logical reasoning that links the data with the expectation is arguably not judgement.
Opinions of experts should be replaced by expert’s experience, information and evidence. Empirical data and experience alone are inadequate, as every situation encountered in practice is unique and not fully represented by the historic data. No amount of Monte Carlo simulation, especially if applied to empirical statistical regression equations, can make up for inadequacies in theories, experimental data, empirical evidence and sound judgement. Ensuring the exercise of judgement complies with defined procedures to ensure robustness and tractability is imperative as ‘it is no longer acceptable to present decisions as matters of judgement without further explanation, still less to cloud them in the pretence of being matters of fact.’22 The resolution of the engineering component of the owner’s dilemma resides in the use of well tried and tested approaches to engineering analysis, and the adoption of established scientific procedures to develop new theories of the performance of dams at and near failure.
The owner’s dilemma will not solve itself and owners should not expect others to resolve it for them. As with the benefits of dam ownership, the owners own the risk that was created by the dam and they have a responsibility to manage it. However, this can not be done in isolation of government, as decisions concerning the tolerability of risks posed by dams are a classic function of government.
Good risk management requires good risk information — information that is generated through risk analysis. Risk analysis goes beyond risk estimation as it is always possible to make an estimate of anything. Owners have a responsibility to ensure that they have adequate analytical techniques available to them to effectively manage, characterise and understand their risks. Many of the risk analysis challenges are too large and complex for one organisation to resolve. Although pressures of competition often inhibit co-operation between owners, a case can be made that co-operation in the area of risk analysis for dam safety is of benefit to everyone in this era of competition. The Canadian Electricity Association’s Dam Safety Interest Group’s research project, A Guide to Dam Risk Management,23 is an example of a co-operative international initiative sponsored by owners from different countries with diverse interests working towards a common goal — a framework for resolving the risk analysis component of the owner’s dilemma.
Finally, there is a clear need to improve the quality of the literature. The level of debate concerning risk analysis of dams needs to be raised and professional practice continually improved to ensure that Professor De Mello’s concern about pseudo-professional analysing of risks24 (Professor De Mello is not alone in holding this view) is consigned to the history books.
If these vitally important issues are not addressed scepticism and lack of confidence in risk analysis in dam safety practice will become even more entrenched, and the dam safety risk component of the owner’s dilemma will remain unresolved.