FY 2003 Mainstem/Systemwide proposal 200303200

Section 1. Administrative

Project location

Reasonable and Prudent Alternatives (RPAs)

Section 2. Past accomplishments

Section 3. Relationships to other projects

Section 4. Budget for Planning and Design phase

Task-based budget

Outyear objectives-based budget

Outyear budgets for Planning and Design phase

Section 5. Budget for Construction and Implementation phase

Task-based budget

Outyear objectives-based budget

Outyear budgets for Construction and Implementation phase

Section 6. Budget for Operations and Maintenance phase

Task-based budget

Outyear objectives-based budget

Outyear budgets for Operations and Maintenance phase

Section 7. Budget for Monitoring and Evaluation phase

Task-based budget

Outyear objectives-based budget

Outyear budgets for Monitoring and Evaluation phase

Section 8. Estimated budget summary

Itemized budget

Total estimated budget

Cost sharing

Reviews and recommendations

This information was not provided on the original proposals, but was generated during the review process.

Comment:

If funded, Proposal #34008 in the Innovative Solicitation "Use a Multi-Watershed Approach to Increase the Rate of Learning from Columbia Basin Watershed Restoration Projects" would seem to overlap the objectives of this proposal. This is an awkward situation because funding decisions on proposals submitted under the Innovative Solicitation may not be complete.

This proposal may be premature and appears to duplicate some efforts in ongoing projects in other provinces. The proposal should more clearly explain its relationship to the ongoing projects and the overall RME planning activities in proposal #35033 of which the PI's agency is a cooperating member. This project proposal should also linked to others being submitted: e.g., 35016 (A Pilot Study to Test Links Between Land Use / Land Cover Tier 1 Monitoring Data and Tier 2 and 3 Monitoring, Feist); 35019 (Develop and Implement a Pilot Status and Trend Monitoring Program for Salmonids and their Habitat in the Wenatchee and Grande Ronde River Basins); 35048 (NWFWC Salmon Data Management, Analysis and Access for Research, Monitoring and Evaluation Programs). The relationship to these proposals should be more clearly specified, e.g., are any of these projects necessary for the success of this proposal? A primary contribution of this proposal would be to implement 2-3 pilot studies of effectiveness monitoring projects. This seems to overlap the objectives of proposal 35019 from the same agency. The proposals should be coordinated to avoid duplication of effort.

Action Agency/NMFS RME Group Comments:

HABITAT ACTION EFFECTIVENESS RESEARCH SUBGROUP -- Does the proposal satisfy the objectives of RPA 183?

The proposals intent to provide a mechanism to coordinate and prioritize implementation of projects, provide design guidelines for monitoring, and implement several pilot projects does not fully satisfy RPA requirements.

Elements that are Lacking.

Much of the work proposed here is already underway within the Action Agencies RME framework. What this proposal offers that the AER team is not currently doing is the implementation of several pilot projects. These pilot projects can be used to test the methods and guidelines established by the AER team. In addition, the pilot studies can test cause-effect linkages between management actions and the proposed indicators. Reviewers believe this is an important component of AER.

Means and Opportunities to Strengthen Proposal.

The study proposes to develop pilot projects aimed at grazing control, barrier removal, and installation of irrigation diversion screens. Providing information on how these studies will be developed or the methods that will be used would clearly strengthen the proposal. It is not clear if the pilot studies intend to test the protocols (and selected indicators) developed by the AER team, or if the pilot studies will "intensively" investigate the web of mechanistic relationships in the stream ecosystems (the latter is referred to as "intensive effectiveness research" by the AER team).

Feasibility of Proposed Work.

More information on the development of the pilot studies is needed to ascertain the feasibility of the proposed work. It is not clear how the studies will be developed, nor is it clear if the sponsor intends to implement and test the protocols developed by the AER team. Reviewers sense that the sponsor intends to develop their own monitoring criteria and guidelines.

OCEAN AND ESTUARY SUBGROUP -- Action item addressed - 183. Pilot projects have already been chosen that do not include the estuary. Unless that focus is going to be expanded, this proposal does not address the estuary.

ISRP Remarks on RME Group Comments: The ISRP agrees with the RME group assessment that "More information on the development of the pilot studies is needed to ascertain the feasibility of the proposed work" and it is unclear if the proposal would meet the needs of RPA 183.

Comment:

Comment:

Comment:

We note the overlap of the overall objectives of this proposal with those submitted by ESSA in their "innovative project proposal." The approach proposed by ESSA is that of an observational study leading to comparison of watersheds or stream reaches by standard statistical methods, e.g., regression modeling techniques and other empirical methods. The ESSA approach leads to standard, acceptable scientific inferences, however we grant that the approach does not lead to the same level of "cause and effect" relationships as true treatment-control experiments.

Even though it is important to include controls and create a true experimental design, we caution the sponsor on the difficulty of designing adequate paired or BACI (treatment-control) experiments on the scale indicated. It is difficult to find and maintain the number of replicates of treatment and control stream reaches necessary to carry out the design and analysis. Also, the presence of confounding factors introduces extreme variation in measured variables (e.g., in evaluation of the effects of additional large wood in streams, some of the streams may have livestock fenced out and others may not). The ISRP's experience with projects like the Idaho Supplementation Study (ISS) lead us to believe that the fundamental elements of the design of the proposed pilot projects may not be feasible. In the ISS, study sponsors had extreme difficulty obtaining and maintaining treatment and control streams that can be captured in a rigorous experimental design. Similar logistical problems in the proposed pilot projects will likely limit conclusions that can be drawn.

The sponsors propose to work with other ongoing project managers to implement their pilot monitoring projects. This will require extensive regional buy-in and therefore to be successful, this project should be coordinated with other monitoring projects to ensure compatibility of objectives, common methods and protocols. This coordination could be accomplished under the CBFWA proposal #35033.

There are severe deficiencies with the present proposal to develop a proposal. However, the ISRP grants that if the objectives could be met, such a project would provide critical monitoring of habitat recovery efforts in the Columbia Basin. If the Council agrees to extend the period for consideration of funding of this and perhaps other monitoring proposals (e.g., new data to be captured by StreamNet and the NMFS Proposal #35048, Tier I monitoring proposed by #35016, and Tier III monitoring proposed by #35020) then the ISRP could review the set at a later time.

Comment:

Comments
NMFS proposal. Inappropriate to comment.

Already ESA Required?
No

Biop?
yes

Comment:

It is unclear that implementation of this proposal can do much more than 35019 in cooperation with the ESSA project (34008, as originally proposed). This proposal, 35020, claims to address the questions about whether classes of projects are effective, and whether a single project is effective, and differentiates these questions from the examination of effectiveness of “spatial classes” of projects conducted under 35019. However, monitoring conducted under 35019 may help address the single project and classes of projects questions within its spatial scope. The ISRP does not see the value added of this project over the ESSA project in cooperation with the status and trend monitoring in 35019. The proposal does not make the distinction clear, other than to purport that it will meet requirements of RPA 183 and to indicate that it will coordinate with the ESSA project.

The Council should carefully evaluate whether the ESSA project, 34008, as originally proposed has been compromised (or improved) during the Bonneville contracting process. Indeed, the author of this proposal, 35020, stated that he worked with Bonneville to draft the statement of work for project 34008. If the project 34008 was modified by the author of 35020, then the author may have a strong conflict of interest in the present proposal, 35020. Also, it may no longer be appropriate for the ISRP to recommend the ESSA project if the statement of work does not closely match the original objectives, tasks, and methods reviewed by the ISRP.

The BiOp RPAs relevant to this proposal put the Action Agencies in a no-win position with more-or-less an impossible task, namely to prove cause and effect relationships in an eight-year (now five-year) timeframe. Reaching this RPA goal is made even more difficult by the approach proposed here; i.e. the use of observational studies to prove cause and effect. A study of the type proposed in 35020 was instigated to prove that the oil spilled by the Exxon Valdez in Prince William Sound, 1989, caused certain injuries to the local floral and fauna. Fourteen years later, scientists are still arguing about the interpretation of results from those studies. It is now a widely accepted conclusion that “smoking causes cancer,” but one only has to look at the long history of arguments to see the difficulty of establishing such relationships based on observational studies. The proponents of 35020 and the authors of the BiOp have depended too heavily on inappropriate use of statistical tests of null hypotheses.

The ISRP recommends that they either: (1) rely on estimation of effects and build models for predicting effects based on measures of habitat improvement actions and other predictor variables, or (2) design a program similar to the Intensive Watershed Monitoring program developed in the State of Washington (described in the soon to be might be better served by reviewing the RPAs with the objective of devising more workable interpretations.

Formal tests of null hypotheses are not appropriate for the stated objectives of the proposal and some of the tasks listed in the RPAs of the BiOp, because the test results depend not only on the magnitude of effects, but also on the sample sizes. In the present setting, tests of null hypotheses can result in a false confidence in the results, i.e., results may be statistically significant, but of no biological importance. Alternatively, effects may be important biologically, but results are not statistically significant because of high variances or small sample sizes. Instead, analytical methods should include estimation of effects with measures of precision of the estimates, perhaps expressed as confidence intervals.

The proponents propose to match treatment sites with control sites on a large set of covariates and remove hidden bias quoting Rosenbaum (1995). By definition, it is not a straightforward process to establish that comparisons are free from hidden bias because Rosenbaum (p. Vii) defines hidden bias as due to differences between the treated and control groups on variables that are not recorded. That is what makes it “hidden.” The adjustments that can be made are from “non-hidden” biases. Quoting from Rosenbaum (p. 136) “Still, all observational studies are sensitive to sufficiently large biases, and large biases have occurred on occasion … A sensitivity analysis shows how biases of various magnitudes might alter conclusions, but it does not indicate whether biases are present or what magnitudes are plausible.”

The proposal indicates that a critical determinant of success in effectiveness monitoring is the rapid and free exchange of data. Given that the lack of data sharing among the entities listed is a serious problem, what mechanisms are in place to ensure that this takes place? In earlier comments we recommended coordination through #35033, but this CBFWA proposal is not explicitly mentioned in 35020.

Comment:

Comment:

Comments: