2010 EM Validation Task Force
The initial meeting of the Electron Microscopy Validation Task Force (EM VTF) was held on September 28-29, 2010. Twenty highly regarded EM scientists from around the world assembled at Rutgers University in New Brunswick NJ to consider possible validation standards for 3D maps and models produced by electron microscopy reconstruction methods. The 2-day workshop was co-chaired by Richard Henderson (MRC, Cambridge) for map validation and Andrej Sali (UCSF) for model validation. Initial recommendations are now published. The full citation is:
Richard Henderson, Andrej Sali, Matthew L. Baker, Bridget Carragher, Batsal Devkota, Kenneth H. Downing, Edward H. Egelman, Zukang Feng, Joachim Frank, Nikolaus Grigorieff, Wen Jiang, Steven J. Ludtke, Ohad Medalia, Pawel A. Penczek, Peter B. Rosenthal, Michael G. Rossmann, Michael F. Schmid, Gunnar F. Schröder, Alasdair C. Steven, David L. Stokes, John D. Westbrook, Willy Wriggers, Huanwang Yang, Jasmine Young, Helen M. Berman, Wah Chiu, Gerard J. Kleywegt, Catherine L. Lawson, Outcome of the First Electron Microscopy Validation Task Force Meeting, Structure, Volume 20, Issue 2, 8 February 2012, Pages 205-214, ISSN 0969-2126, 10.1016/j.str.2011.12.014.
Agenda
|
Tuesday, September 28, 2010 CABM, Rutgers University (Room 010) Morning and Afternoon Sessions are open to the Rutgers/UMDNJ Community (flyer) |
|||
| 9:00am | Welcome Wah Chiu Introduction Helen Berman Validation of X-ray structures overview Gerard Kleywegt |
||
| 10:00am | Break | ||
| 10:15am |
Overview of Charge/Questions to Committees Cathy Lawson | ||
| 10:30am | Map Validation Session Richard Henderson, Chair | ||
| 10:30 Richard Henderson 10:45 Niko Grigorieff 11:00 Pawel Penczek | |||
| 11:15 Steve Ludtke 11:30 Wen Jiang 11:45 Ed Egelman | |||
| 12:00 Ohad Medalia 12:15 Alasdair Steven | |||
| 12:30pm | Lunch | ||
| 1:40pm | Model Validation Session Andrej Sali, Chair | ||
| 1:45 Andrej Sali 2:00 Michael Rossmann 2:15 Matt Baker | |||
| 2:30 Gunnar Schroeder 2:45 Willy Wriggers | |||
| 3:00pm | Coffee Break | ||
| 3:15pm | Additional Presentations, Discussion | ||
| 5:00pm | Meeting Adjourns - Transportation back to the Heldrich | ||
| 7:00pm | Dinner at Soho on George | ||
|
Wednesday, September 29, 2010 The Heldrich Hotel (Rooms Neilson and Dickson) |
|||
| 09:00am | Presentations and discussions to prepare report for full group | ||
| Maps: Room Neilson Richard Henderson, Chair | |||
| Models: Room Dickson Andrej Sali, Chair | |||
| 11:45am | Lunch | ||
| 01:00pm | Full group discussions of Maps and Models in Room Neilson | ||
| 03:30pm | Full group presentations in Room Neilson | ||
| 05:00pm | Meeting adjourns | ||
| 07:00pm | Dinner at Christopher's (Heldrich Hotel) | ||
| revised 27 September 2010 |
|||
Attendees
| EMDataBank Organizing Committee | ||
|
Dr. Helen M. Berman Dr. Wah Chiu |
Dr. Gerard Kleywegt Dr. Cathy Lawson |
|
| EM Models Discussion Group | EM Map Discussion Group | |
|
Dr. Andrej Sali (Chair) Dr. Matthew Baker Dr. Batsal Devkota Dr. Peter Rosenthal Dr. Michael Rossmann Dr. Gunnar Schroeder Dr. Willy Wriggers |
__ |
Dr. Richard Henderson (Chair) Dr. Bridget Carragher Dr. Kenneth Downing Dr. Edward H. Egelman Dr. Joachim Frank Dr. Nikolaus Grigorieff Dr. Wen Jiang Dr. Steven J. Ludtke Dr. Ohad Medalia Dr. Pawel Penczek Dr. Michael Schmid Dr. Alasdair C. Steven |
| Other Participants |
||
|
Dr. Margaret Gabanyi Dr. Zukang Feng |
Dr. John Westbrook Dr. Huanwang Yang Dr. Jasmine Young |
|
EM VALIDATION TASK FORCE CHARGE AND QUESTIONS
Charge to the Committee
The EM Validation Task Force (EM-VTF) is charged with advising members of EMDataBank.org on approaches to validate maps deposited to EMDB and models deposited to PDB that are obtained from cryoEM data.
Specifically, the EM-VTF is to provide recommendations for EM structure validation criteria and tools. Such validation tools can be based either on existing or proposed software; these tools would become freely available for use by individual laboratories. Validation tools will not be used by EMDB or PDB as a basis to “reject” maps or models, but rather to flag potential problems for a depositor or user to be aware of.
The assignment for the 1st meeting is to discuss assigned questions (see below), decide on priorities and develop plans for creating the recommendations. Recommendations will be assembled into "white papers" that can be used to seek support for development and implementation of the validation tools, with aim explicitly to publish a combined "EM Validation" White Paper in a journal.
Questions
The questions below were drafted by the EM-VTF organizing committee and discussion group leaders for consideration by the map and model discussion groups, and are intended to stimulate discussion. All EM-VTF members are welcome to address any of the questions.
Maps
(1) How can map accuracy (both noise level and overall “correctness”) be assessed? What statistical values provide an indication of the quality of a reconstruction? What tests can be recommended that can be used to validate image processing parameters? In modelling in general, there are 3 quality criteria of a “model”, such as a density map or an explicit structural model. There is in general an ensemble of maps (or structural models) that are consistent with raw data, thus there is the precision of the ensemble, the accuracy of a particular map (or model), and the degree of the coverage of the measured system by the map (or a model). Is there any utility in this perspective for describing the quality of a map, not only model(s) based on the map?
(2) How should map resolution be reported? Currently it is self-reported by depositors as a single value with text description, e.g., “determined by FSC at 0.5 cutoff”. Should EMDB require the complete Fourier Shell Correlation information from depositors? Are there other useful indicators/tools?
(3) What density manipulation/filtering procedures should be specified for the deposited map densities? Should any procedures be disallowed? Examples: negative density truncation, profile fitting, low or high pass filtering, B-factor sharpening, cropping, masking, FOM weighting.
(4) Would it be desirable to have a tool to validate map point group/helical symmetry and define orientation and position with respect to a standard frame?
(5) What parameters should be used to indicate reconstruction quality, noise level and signal-to-noise ratio from 3D tomogram and sub-tomogram averages?
Models
(1) What kind of structural models (eg, atomic, coarse-grained, single models, ensembles of models) do we expect based on EM maps?
(2) What general criteria in principle describe the quality of a “model”, including an explicit macromolecular structural model based on an EM map (eg, precision, accuracy, coverage)?
(3) What can we learn from other assessment efforts (eg, X-ray, NMR, homology and ab initio protein structure modelling, SAXS, integrated modelling, EM map) and should we strive for a common language and framework across these fields, allowing us to understand the commonalities and differences, thus benefit from concepts and tools developed by a much larger community?
(4) How should the fit of an atomic model into an EM map be evaluated? Are there suitable local (e.g., per residue, per secondary-structure element, or per domain) measures? Are there suitable global fit measures? (How) would assessment vary with map resolution, or with fitting protocol (e.g., rigid-body vs. annealing)?
(5) How can we evaluate (esp. for tomograms) that a fitted model is the correct one (it could conceivably be the wrong molecule!) or that the solution is unique/optimal? Is there any measure of quality-of-fit or signal-to-noise for (a) selecting a model to fit to a feature and (b) the selected orientation/position?
(6) We know how to validate the stereochemistry/geometry of atomic models, but how should we "value" the results when applied to EM models? How should we handle cases were rigid-body placement/refinement of a high-resolution X-ray structure has been carried out, the validation statistics will say nothing about the quality of the EM model (except for clashes)? And how should we handle cases where refinement/model-building has been done (there may be thousands of outliers even in a good model just because the resolution is so low)? And (how) should we use clashes?
(7) The precise taxonomy/protein sequence of an EM sample (a) may not be known or (b) the only high-resolution structures available for fitting may be from a species homolog. How should PDB handle the following situations: model deposited with wrong sequence(s), model deposited based on homology-modelling?
(8) How are errors in EM maps and errors in models coupled? This interdependence is possibly going to be increasingly explicitly mutual (eg, via iterative map / model calculations). How to take into account the quality of maps when estimating the quality of models based on these maps?
posted 8/24/10 CL
Materials for Review
Related Workshops
- September 2004 CryoEM Structure Deposition Workshop
- November 2005 Outcome of a Workshop on Structural Biological Models
- July 2008 Protein Models Workshop
- January 2010 Modelling of CryoEM Map Workshop
- Upcoming CryoEM Modelling Challenge and January 2011 PSB Workshop
Other Validation Task Forces
Example Validation Reports (X-ray)
- PDBe Example Report (pdf file)
- RCSB PDB Example Report (pdf file)
- PDBj Example Report (pdf file)
EM Metadata Dictionaries
Source: vtf.emdataresource.org archived content. Outdated links have been removed, April 2022