Kaylin R. Clements Emily J. Wilkins 20250522 spreadsheet https://doi.org/10.5066/P13TZJ7B Kaylin R. Clements James English Emily J. Wilkins Megan A. Moore Rudy Schuster 2025 publication Engagement of end-users and fellow experts is fundamental to the creation of useful scientific tools. We conducted a survey-based research project to assist scientists responsible for creating scientific tools by evaluating their engagement of end-users and experts in the development of these tools. Our study includes a suite of six United States Geological Survey (USGS) projects that designed and implemented end-user and expert engagement to produce scientific tools for natural resource managers. USGS teams engaged end-users and experts (hereafter “participants”) across the United States from federal, state and local governments; universities; Tribes; Territories; and non-governmental organizations in the design and development of scientific tools intended to support end-users in their work. An online questionnaire sent to 113 participants (98 responded) measured satisfaction across several indicators, including preference for the frequency of engagement activities, whether there were sufficient opportunities to provide feedback, trust that feedback was considered, sufficient engagement of the necessary perspectives, and the level of functionality the tool would have for end-users. The data were collected as a part of a larger study that investigated lessons learned about end-user and expert engagement in the development of scientific tools. Results were shared with the scientists leading each of the projects and we gathered their lessons learned based on these results and their own experiences to create a guide of end-user and expert engagement. The data should be used to explore effective strategies for implementing effective engagement. 20240430 20240619 Online survey Complete None planned United States -179.6484 -50.2734 72.2891 12.8975 ISO 19115 Topic Category biota None Online survey Evaluation End-user Engagement Coproduction USGS Thesaurus social sciences information sciences customer support and user feedback USGS Metadata Identifier USGS:67d0bb59d34ecfe34cc87ce1 None. Please see 'Distribution Info' for details. None. Users are advised to read the dataset's metadata thoroughly to understand appropriate use and data limitations. Emily Wilkins U.S. Geological Survey, Fort Collins Science Center Supervisory Research Social Scientist mailing and physical

2150 Centre Avenue Bldg C

Fort Collins Colorado 80526 United States 9702269338 ewilkins@usgs.gov Thank you to the Biological Threats and Invasive Species Research Program for proposing and investing in this project and for supporting engagement and coproduction of science tools. Windows 11 Enterprise, Qualtrics survey software, Excel, SurveyResponses_PartnerEngagementEvaluation.csv, File size: 16 KB Qualtrics 2020 application/service Copyright © 2020 Version: April 2024 https://www.qualtrics.com The survey was pretested using Qualtrics survey software with eight individuals knowledgeable about the projects or with expertise in stakeholder engagement or coproduction research. Feedback from this pre-test group, including suggestions and clarifications about wording in the questions, answer choices, and introduction of the survey, were incorporated. This was followed by a review of the questions by nine project leads on the projects whose participants were part of the survey’s sample population as well as a USGS employee who oversees the projects. This final review resulted in two minor edits to wording to improve clarity, but otherwise the survey was approved by all project leads. We checked the responses in comparison to possible answer choices and data collected falls within the expected range for the variables being measured. These data are a result of an online survey with a response rate of 63.48% and a completion rate of 80.53% for those who responded (n = 113). The initial sample population included 178 individuals representing federal, state, Tribal, Territorial, private, non-governmental, and academic organizations. This data was collected as part of a study of six projects led by USGS scientists to produce scientific tools. Over the course of one or more years, each project facilitated engagement activities that primarily included potential end-users of the intended scientific tool or subject matter experts in a topical area related to the tool (hereafter, “participants”). Project leads across all six projects typically used the term “partner” in reference to these participants. Thus, the term "partner" is used in the survey questions as well. As part of these engagement efforts, each project team systematically recruited and facilitated engagement activities with participants. This survey was distributed to these participants to evaluate the process of engagement. The survey included 17 questions (and followed the requirements for the Department of Interior (DOI) Programmatic Clearance for Customer Satisfaction Surveys) and was approved through the requirements of the Office of Management and Budget (OMB Control # 1040-0001.). We developed the survey using existing research on coproduction and stakeholder engagement to produce actionable science in the contexts of natural resource management and medical research (Bamzai-Dodson et al., 2021; Bamzai-Dodson and McPherson, 2022; Meadow and Owen, 2021; Wall et al., 2017). Each question measured a specific indicator identified in the research on coproduction of actionable science (Lavallee et al., 2012; Ray and Miller, 2017; Wall et al., 2017). We prioritized indicators based on how actionable the feedback about that indicator would be for project leads. For example, learning whether participants would like to have less frequent or more frequent virtual group meetings could easily be interpreted and acted upon. In addition, project teams were consulted on what types of questions would provide most useful information to them. This resulted, for example, in the inclusion of a question asking participants how functional they believed the projects’ deliverables would be for their work as well as what level of engagement they perceived they were involved in. Furthermore, the survey served as a formative evaluation as most projects were ongoing and had not released final products. Answer options typically followed a Likert-type scale, such as a 7-point range from Strongly disagree to Strongly agree. The survey was pretested using Qualtrics survey software with eight individuals knowledgeable about the projects or with expertise in stakeholder engagement or coproduction research. Feedback from this pre-test group, including suggestions and clarifications about wording in the questions, answer choices, and introduction of the survey, were incorporated. This was followed by a review of the questions by nine project leads on the projects whose participants were part of the survey’s sample population as well as a USGS employee who oversees the projects. This final review resulted in two minor edits to wording to improve clarity, but otherwise the survey was approved by all project leads. The sample population included 178 individuals representing federal, state, Tribal, Territorial, private, non-governmental, and academic organizations who were invested in the final outcomes of the projects, excluding USGS staff. Following best practices for web-based survey research (Dillman et al., 2014), after initial distribution of the survey on April 30, 2024 using Qualtrics survey software, two reminder emails were sent to the distribution list. The survey closed June 19th, 2024. One hundred and thirteen people accessed the introduction page and advanced at least to the first question of the survey. Fifteen of these respondents answered too few questions to be included in analysis. Another seven did not finish the survey completely but provided responses to several questions that were included in some of the analysis. Ninety-one respondents finished the survey completely. This resulted in a response rate of 63.48% and a completion rate of 80.53% for those who responded. Bamzai-Dodson, A., Cravens, A. E., Wade, A. A., & McPherson, R. A. (2021). Engaging with stakeholders to produce actionable science: a framework and guidance. Weather, Climate, and Society, 13(4), 1027-1041. https://doi.org/10.1175/WCAS-D-21-0046.1 Bamzai-Dodson, A., & McPherson, R. A. (2022). When Do Climate Services Achieve Societal Impact? Evaluations of Actionable Climate Adaptation Science. Sustainability, 14(21), 14026. https://doi.org/10.3390/su142114026 Lavallee, D. C., Williams, C. J., Tambor, E. S., & Deverka, P. A. (2012). Stakeholder engagement in comparative effectiveness research: how will we measure success?. Journal of Comparative Effectiveness Research, 1(5), 397-407. https://doi.org/10.2217/cer.12.44 Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method. John Wiley & Sons. Meadow, A. M., & Owen, G. (2021). Planning and Evaluating the Societal Impacts of Climate Change Research Projects: A guidebook for natural and physical scientists looking to make a difference. Ray, K. N., & Miller, E. (2017). Strengthening stakeholder-engaged research and research on stakeholder engagement. Journal of comparative effectiveness research, 6(4), 375-389. https://doi.org/10.2217/cer-2016-0096 Wall, T. U., Meadow, A. M., & Horganic, A. (2017). Developing evaluation indicators to improve the process of coproducing usable climate science. Weather, Climate, and Society, 9(1), 95-107. https://doi.org/10.1175/WCAS-D-16-0008.1 20240619 SurveyResponses_PartnerEngagementEvaluation.csv Comma Separated Value (CSV) file containing data. Producer Defined Q1Project Response to the following question: "For the purpose of this survey, we are asking that you respond based on your experience with just one project, even if you have been involved in multiple. Please select the project with which you have been the most involved. Use this project as the basis for your answers for the rest of the questions in this survey." Multiple choice options included the six individual projects evaluated by survey respondents. This question was required. Respondents' subsequent answers are attributed to the project identified in this question. Three different types of tools were being created by the projects: web tools, scientific protocols, and information products. Producer Defined 99 Respondent did not answer the question Producer defined Information Product 1 A project creating the first of two information products included in the survey. End-users and experts participating in this project's engagement activities responded to the survey. Producer defined Information Product 2 A project creating the second of two information products included in the survey. End-users and experts participating in this project's engagement activities responded to the survey. Producer defined Scientific Protocol 1 A project creating the first of two scientific protocols included in the survey. End-users and experts participating in this project's engagement activities responded to the survey. Producer defined Scientific Protocol 2 A project creating the second of two scientific protocols included in the survey. End-users and experts participating in this project's engagement activities responded to the survey. Producer defined Web Tool 1 A project creating the first of two web tools included in the survey. End-users and experts participating in this project's engagement activities responded to the survey. Producer defined Web Tool 2 A project creating the second of two web tools included in the survey. End-users and experts participating in this project's engagement activities responded to the survey. Producer defined Q2OpportunitiesFeedback Response to the following question: "Please indicate your level of agreement with the following statement: The project team has provided me with sufficient opportunities to provide feedback about the project." Answer options included: Strongly disagree, Disagree, Somewhat disagree, Neutral, Somewhat agree, Agree, and Strongly agree. Producer Defined 99 Respondent did not answer the question Producer defined Somewhat agree The respondent somewhat agrees with the statement "The project team has provided me with sufficient opportunities to provide feedback about the project." Producer defined Strongly agree The respondent strongly agrees with the statement "The project team has provided me with sufficient opportunities to provide feedback about the project." Producer defined Agree The respondent strongly agrees with the statement "The project team has provided me with sufficient opportunities to provide feedback about the project." Producer defined Neutral The respondent does not disagree nor agree with the statement "The project team has provided me with sufficient opportunities to provide feedback about the project." Producer defined Strongly disagree The respondent strongly disagrees with the statement "The project team has provided me with sufficient opportunities to provide feedback about the project." Producer defined Somewhat disagree The respondent somewhat disagrees with the statement "The project team has provided me with sufficient opportunities to provide feedback about the project." Producer defined Disagree The respondent disagrees with the statement "The project team has provided me with sufficient opportunities to provide feedback about the project." Producer defined Q3TrustFeedback Response to the following question: "Please indicate your level of agreement with the following statements: I trust that the project team has considered the feedback I have given." Answer options included: Strongly disagree, Disagree, Somewhat disagree, Neutral, Somewhat agree, Agree, and Strongly agree. Producer Defined 99 Respondent did not answer the question Producer defined Somewhat disagree The respondent somewhat disagrees with the statement "I trust that the project team has considered the feedback I have given." Producer defined Agree The respondent agrees with the statement "I trust that the project team has considered the feedback I have given." Producer defined Strongly agree The respondent strongly agrees with the statement "I trust that the project team has considered the feedback I have given." Producer defined Somewhat agree The respondent somewhat agrees with the statement "I trust that the project team has considered the feedback I have given." Producer defined Neutral The respondent does not disagree or agree with the statement "I trust that the project team has considered the feedback I have given." Producer defined Strongly disagree The respondent somewhat disagrees with the statement "I trust that the project team has considered the feedback I have given." Producer defined Disagree The respondent disagrees with the statement "I trust that the project team has considered the feedback I have given." Producer defined Q4-9FrequencyEmails Response to the following question: "For each of the following types of interaction that you have participated in, what is your perception of the frequency of this type of interaction? For types of interaction that you have not participated in, please choose "N/A": Emails Answer options included: Far too little, Slightly too little, The right amount, Slightly too much, Far too much, N/A Producer Defined N/A "Not Applicable": The respondent did not interact with the project team in the form of emails. Producer defined 99 Respondent did not answer the question Producer defined Slightly too little The respondent perceives there is slightly too little interaction with the project team in the form of emails. Producer defined The right amount The respondent perceives there is the right amount of interaction with the project team in the form of emails. Producer defined Far too little The respondent perceives there is far too little interaction with the project team in the form of emails. Producer defined Slightly too much The respondent perceives there is slightly too much interaction with the project team in the form of emails. Producer defined Far too much The respondent perceives there is far too much interaction with the project team in the form of emails. Producer defined Q4-9FrequencyOne-on-one Response to the following question: "For each of the following types of interaction that you have participated in, what is your perception of the frequency of this type of interaction? For types of interaction that you have not participated in, please choose "N/A": One-on-one calls or meetings Answer options included: Far too little, Slightly too little, The right amount, Slightly too much, Far too much, N/A Producer Defined N/A "Not Applicable": The respondent did not interact with the project team in the form of one-on-one calls or meetings. Producer defined 99 Respondent did not answer the question Producer defined Slightly too little The respondent perceives there is slightly too little interaction with the project team in the form of one-on-one calls or meetings. Producer defined The right amount The respondent perceives there is the right amount of interaction with the project team in the form of one-on-one calls or meetings. Producer defined Far too little The respondent perceives there is far too little interaction with the project team in the form of one-on-one calls or meetings. Producer defined Slightly too much The respondent perceives there is slightly too much interaction with the project team in the form of one-on-one calls or meetings. Producer defined Far too much The respondent perceives there is far too much interaction with the project team in the form of one-on-one calls or meetings. Producer defined Q4-9FrequencyPresentations Response to the following question: "For each of the following types of interaction that you have participated in, what is your perception of the frequency of this type of interaction? For types of interaction that you have not participated in, please choose "N/A": Presentations (example: updates or webinars about the project) Answer options included: Far too little, Slightly too little, The right amount, Slightly too much, Far too much, N/A Producer Defined N/A "Not Applicable": The respondent did not interact with the project team in the form of presentations. Producer defined 99 The respondent did not answer the question Producer defined Slightly too little The respondent perceives there is slightly too little interaction with the project team in the form of presentations. Producer defined The right amount The respondent perceives there is the right amount of interaction with the project team in the form of presentations. Producer defined Far too little The respondent perceives there is far too little interaction with the project team in the form of presentations. Producer defined Slightly too much The respondent perceives there is slightly too much interaction with the project team in the form of presentations. Producer defined Far too much The respondent perceives there is far too much interaction with the project team in the form of presentations. Producer defined Q4-9FrequencyVirtualGroupMeetings Response to the following question: "For each of the following types of interaction that you have participated in, what is your perception of the frequency of this type of interaction? For types of interaction that you have not participated in, please choose "N/A": Virtual group meetings Answer options included: Far too little, Slightly too little, The right amount, Slightly too much, Far too much, N/A Producer Defined N/A "Not Applicable": The respondent did not interact with the project team in the form of virtual group meetings. Producer defined 99 Respondent did not answer the question Producer defined Far too little The respondent perceives there is far too little interaction with the project team in the form of virtual group meetings. Producer defined Slightly too little The respondent perceives there is slightly too little interaction with the project team in the form of virtual group meetings. Producer defined The right amount The respondent perceives there is the right amount of interaction with the project team in the form of virtual group meetings. Producer defined Slightly too much The respondent perceives there is slightly too much interaction with the project team in the form of virtual group meetings. Producer defined Far too much The respondent perceives there is far too much interaction with the project team in the form of virtual group meetings. Producer defined Q4-9FrequencyIn-PersonInteractions Response to the following question: "For each of the following types of interaction that you have participated in, what is your perception of the frequency of this type of interaction? For types of interaction that you have not participated in, please choose "N/A": In-person interactions Answer options included: Far too little, Slightly too little, The right amount, Slightly too much, Far too much, N/A Producer Defined N/A "Not Applicable": The respondent did not interact with the project team in the form of in-person interactions. Producer defined 99 Respondent did not answer the question Producer defined Slightly too little The respondent perceives there is slightly too little interaction with the project team in the form of in-person interactions. Producer defined The right amount The respondent perceives there is the right amount of interaction with the project team in the form of in-person interactions. Producer defined Far too little The respondent perceives there is far too little interaction with the project team in the form of in-person interactions. Producer defined Slightly too much The respondent perceives there is slightly too much interaction with the project team in the form of in-person interactions. Producer defined Far too much The respondent perceives there is far too much interaction with the project team in the form of in-person interactions. Producer defined Q4-9FrequencyOtherTEXT Answer written in by respondent beside the option "Other" to the following question: "For each of the following types of interaction that you have participated in, what is your perception of the frequency of this type of interaction? For types of interaction that you have not participated in, please choose "N/A"." Producer Defined 99 Respondent did not answer the question Producer defined 88 Respondent did not have another type of interaction to add as an "Other" option. Producer defined I'd love to get some sort of periodic update, e.g., quarterly, in the form of a simple e-mail...progress, products, plans, etc... Respondent added "periodic updates", which was not an option in the list of types of interaction they could choose from. Producer defined Trainings Respondent added "trainings", which was not an option in the list of types of interaction they could choose from. Producer defined Chats Respondent added "chats", which was not an option in the list of types of interaction they could choose from. Producer defined Other ways to provide input Respondent added "other ways to provide input", which was not an option in the list of types of interaction they could choose from. Producer defined Introductory webinars Respondent added "introductory webinars", which was not an option in the list of types of interaction they could choose from. Producer defined Q4-9FrequencyOther Respondents' perception of the frequency of the interaction they described in Q4-9FrequencyOtherTEXT Answer options included: Far too little, Slightly too little, The right amount, Slightly too much, Far too much, N/A Producer Defined 99 Respondent did not answer the question Producer defined 88 This question did not apply to the respondent. Producer defined Slightly too little The respondent perceives there is slightly too little interaction with the project team in the form of interaction they indicated in Q4-9FrequencyOtherTEXT. Producer defined Far too little The respondent perceives there is far too little interaction with the project team in the form of interaction they indicated in Q4-9FrequencyOtherTEXT. Producer defined The right amount The respondent perceives there is the right amount of interaction with the project team in the form of interaction they indicated in Q4-9FrequencyOtherTEXT. Producer defined Slightly too much The respondent perceives there is slightly too much interaction with the project team in the form of interaction they indicated in Q4-9FrequencyOtherTEXT. Producer defined Far too much The respondent perceives there is far too much interaction with the project team in the form of interaction they indicated in Q4-9FrequencyOtherTEXT. Producer defined Q10NecessaryPerspectives Response to the following question: "Please indicate your level of agreement with the following statement: "The project team is engaging partners with the necessary subject matter expertise and management perspectives to inform the project." Answer options included: Strongly disagree, Disagree, Somewhat disagree, Neutral, Somewhat agree, Agree, and Strongly agree. Producer Defined 99 Respondent did not answer the question Producer defined Strongly disagree The respondent strongly disagrees with the statement "The project team is engaging partners with the necessary subject matter expertise and management perspectives to inform the project." Producer defined Agree The respondent agrees with the statement "The project team is engaging partners with the necessary subject matter expertise and management perspectives to inform the project." Producer defined Strongly agree Respondent strongly agrees with the statement "The project team is engaging partners with the necessary subject matter expertise and management perspectives to inform the project." Producer defined Somewhat agree The respondent somewhat agrees with the statement "The project team is engaging partners with the necessary subject matter expertise and management perspectives to inform the project." Producer defined Neutral The respondent does not disagree or agree with the statement "The project team is engaging partners with the necessary subject matter expertise and management perspectives to inform the project." Producer defined Somewhat disagree The respondent somewhat disagrees with the statement "The project team is engaging partners with the necessary subject matter expertise and management perspectives to inform the project." Producer defined Disagree The respondent disagrees with the statement "The project team is engaging partners with the necessary subject matter expertise and management perspectives to inform the project." Producer defined I don't know The responded does not know whether the project team is engaging partners with the necessary subject matter expertise and management perspectives to inform the project. Producer defined Q11Involvement Response to the following question: "Which of the following best describes your involvement in this project?" Answer options included: I am informed about the project’s progress, final products and outputs. I am consulted for feedback on certain aspects of the project, such as analyses, design, implementation, products and outputs. I collaborate with the project team to formulate solutions and design final products and outputs. I am a coequal to rest of the project team; I provide foundational input and recommendations to project development. Producer Defined 99 Respondent did not answer the question Producer defined I am informed about the project’s progress, final products and outputs. The respondent was kept informed about the project’s progress, final products and outputs. Producer defined I collaborate with the project team to formulate solutions and design final products and outputs. The respondent collaborated with the project team to formulate solutions and design final products and outputs. Producer defined I am consulted for feedback on certain aspects of the project, such as analyses, design, implementation, products and outputs. The respondent was consulted for feedback on certain aspects of the project, such as analyses, design, implementation, products and outputs. Producer defined I am a coequal to rest of the project team; I provide foundational input and recommendations to project development. The respondent was a coequal to the rest of the project team; they provided foundational input and recommendations to project development. Producer defined Q12Functionality Response to the following question: "How functional do you believe the scientific tool(s) or product(s) this project is producing will be to support your invasive species work?" Answer options included: Not at all (Not well-developed at all; not functional) Minimal (Very limited in scope, scale, or function) Moderate (Generally functional with notable insufficiencies or limitations) Good (Gaps may exist for minor elements) Robust (Well-developed and highly functional) Producer Defined 99 Respondent did not answer the question Producer defined Moderate (Generally functional with notable insufficiencies or limitations) The respondent believed the functionality of the scientific tool(s) or product(s) the project was producing would be moderate (generally functional with notable insufficiencies or limitations). Producer defined Good (Gaps may exist for minor elements) The respondent believed the functionality of the scientific tool(s) or product(s) the project was producing would be good (gaps may exist for minor elements). Producer defined Robust (Well-developed and highly functional) The respondent believed the functionality of the scientific tool(s) or product(s) the project was producing would be robust (well-developed and highly functional). Producer defined Minimal (Very limited in scope, scale, or function) The respondent believed the functionality of the scientific tool(s) or product(s) the project was producing would be minimal (very limited in scope, scale, or function). Producer defined Not at all (Not well-developed at all; not functional) The respondent believed the scientific tool(s) or product(s) the project was producing would not be functional at all (not well-developed at all; not functional). Producer defined Q13Experience Response to the following question: "How do you feel about your experience with this project team?" Answer options included: Extremely dissatisfied Dissatisfied Somewhat dissatisfied Neither satisfied nor dissatisfied Somewhat satisfied Satisfied Extremely satisfied Producer Defined 99 Respondent did not answer the question Producer defined Somewhat dissatisfied The respondent felt somewhat dissatisfied with their experience with the project team. Producer defined Somewhat satisfied The respondent felt somewhat satisfied with their experience with the project team. Producer defined Satisfied The respondent felt satisfied with their experience with the project team. Producer defined Extremely satisfied The respondent felt extremely satisfied with their experience with the project team. Producer defined Neither satisfied nor dissatisfied The respondent felt neither satisfied nor dissatisfied with their experience with the project team. Producer defined Dissatisfied The respondent felt dissatisfied with their experience with the project team. Producer defined Extremely dissatisfied The respondent felt extremely dissatisfied with their experience with the project team. Producer defined GS ScienceBase U.S. Geological Survey mailing address

Denver Federal Center, Building 810, Mail Stop 302

Denver CO 80225 United States 1-888-275-8747 sciencebase@usgs.gov Unless otherwise stated, all data, metadata and related materials are considered to satisfy the quality standards relative to the purpose for which the data were collected. Although these data and associated metadata have been reviewed for accuracy and completeness and approved for release by the U.S. Geological Survey (USGS), no warranty expressed or implied is made regarding the display or utility of the data for other purposes, nor on all computer systems, nor shall the act of distribution constitute any such warranty. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government. Digital Data https://doi.org/10.5066/P13TZJ7B None 20250715 FORT Data Management U.S. Geological Survey, Fort Collins Science Center FORT Data Management mailing

2150 Centre Avenue Bldg C

Fort Collins Colorado 80526 United States 9702269100 fortdatamanagement@usgs.gov FGDC Biological Data Profile of the Content Standard for Digital Geospatial Metadata FGDC-STD-001.1-1999