Other News

Request for Proposals: Evaluation of J. Christopher Stevens Virtual Exchange Initiative Programs

Issue Date: December 16, 2022

Proposal Deadline: February 9, 2023

Contact: Henry Shepherd, stevensinitiative@aspeninstitute.org

Introduction

The Aspen Institute requests applications for the independent evaluation of the performance and effect of the J. Christopher Stevens Virtual Exchange Initiative (JCSVEI). The application should take the form of a monitoring and evaluation plan covering evaluation and analysis of the approximately 16 grantees that will conduct virtual exchange programs for approximately 22,000 young people in the United States and across the Middle East and North Africa (MENA) during the period from April 2023 through June 2025.

The J. Christopher Stevens Virtual Exchange Initiative was launched in 2015 to continue the legacy of U.S. Ambassador to Libya J. Christopher Stevens, who devoted his life to building bridges through open and respectful dialogue and person-to-person diplomacy. He served the majority of his Foreign Service career in the Middle East and North Africa, a region he grew to love, and eventually served as U.S. Ambassador to Libya. The Initiative, conceived and developed in close partnership with Ambassador Stevens’ family, seeks to give a generation of young people the kind of meaningful international experience that shaped Ambassador Stevens as a young man. The Initiative is a U.S. Department of State Bureau of Educational and Cultural Affairs program administered by the Aspen Institute.

The Initiative builds on the pioneering work of the virtual exchange community, which uses technology to connect young people across continents and cultures. Virtual exchange provides opportunities to connect young people from diverse places using everyday technology for collaborative learning and interaction through sustained and facilitated engagement. While virtual exchange can cover any topic or subject matter and can vary substantially in the length of programming, technology or platform(s) used, learning outcomes and activity types, all virtual exchange includes a core component of cross-cultural connection and collaboration. Through virtual exchange, youth have access to a substantive international exchange experience by collaborating and learning with their peers abroad without having to leave their communities.

Recognizing the impact that virtual exchange promises, and capitalizing on advances in technology, the aim is to make life-changing, cross-cultural experiences available to young people, and for the experience to be mutually beneficial to participants on all sides of the exchange. Learn more about virtual exchange here.

By summer 2023, prior to the start of programs to be evaluated through this contract, the Initiative will expand its reach to nearly 75,000 young people in 17 MENA countries and the Palestinian Territories, and in 48 U.S. states, Puerto Rico, one tribal community, the U.S. Virgin Islands, and Washington, D.C. Learn more about the Initiative, its activities, its funders, and the programs it has supported here.

Scope and Basis for This Evaluation

The evaluation sought by this request for proposals is intended to establish the plan for – and carry out the work of – collecting, analyzing, and reporting data required by the JCSVEI’s Performance Monitoring Plan (PMP), which is a component of the cooperative agreement between the Department of State and the Aspen Institute. The PMP is based on the MODE framework developed and maintained by the Bureau of Educational and Cultural Affairs. Learn more about the MODE framework here. The items that the Aspen Institute seeks to have covered by this evaluation are included in the Abridged JCSVEI Performance Monitoring Plan that is an annex to this request for proposals. That document includes a final section containing custom items that the Aspen Institute seeks the evaluator to include in their plans for designing and carrying out data collection, analysis, and reporting. Some items from the PMP agreed between the Department of State and the Aspen Institute are not included in the abridged PMP and are not expected to be collected, analyzed, or reported on by the evaluator because the Aspen Institute is able to collect and report them itself.

Proposals should be structured to correspond to the following sections, which explain the competencies sought in the evaluator as well as the areas of work and the anticipated deliverables of the project. The applicant should be clear how they would approach working at this scale and with this range of partners serving young people in disparate contexts.

Section 1: Experience and Capacity in Relevant Contexts (25%)

Experience and Capacity on Projects of this Scale and Breadth (5%): This evaluation will cover virtual exchange programs conducted by approximately 16 grantees during the period from April 2023 through June 2025. These programs are anticipated to serve approximately 22,000 young people. Some of the programs are conducted as part of primary school, secondary school, or higher education for-credit courses and others are conducted as extracurricular activities. Most programs include a combination of synchronous and asynchronous communication.

Experience and Capacity at the Primary/Secondary and Higher Education Levels (10%): Six of the programs serve young people at the primary or secondary school level and 10 of the programs serve young people at the higher education or young professional level.

Experience and Capacity in the U.S. and MENA Region (10%): The programs will be conducted across 18 countries in the Middle East and North Africa and the Palestinian Territories and in 42 U.S. States and the District of Columbia.

The evaluator should describe their experience working on projects of this scale, working in the primary/secondary and higher education contexts, and working in the United States and in the Middle East and North Africa. The applicant should make clear whether they would be hiring or contracting any additional individuals or entities to provide expertise in any of these areas.

Section 2: Data Collection (20%)

Participant Survey Preparation (10%): The evaluator will maintain an administrative account on an online survey management system and will create sub-accounts for the Aspen Institute and for each grantee organization. The evaluator will create a survey template including the items listed in the abridged PMP that are labeled as being collected via a post-program survey. The evaluator will implement survey logic or take other steps to ensure that respondents see the appropriate survey items and instructions. The evaluator should maintain the survey template in English as well as Arabic, retaining an Arabic language expert for this purpose if needed. In rare cases, grantees may request to administer surveys on separate platforms, in which case the evaluator will give that grantee a template document and the grantee will submit a draft survey for approval to ensure it complies with Aspen Institute and Department of State requirements.

The Aspen Institute may ask the evaluator to add items to the survey template during the period of the contract, either as a result of the item being introduced by the Department of State or as an additional item that is unique to the JCSVEI. Grantees may also request support from the evaluator to add items to their survey that are unique to their program. The evaluator will add these items to the survey template or to the survey for the specific grantee as needed.

Grantee-Reported Administrative Data (5%): Some program information – identified as having the Data Source of “administrative records/data” in the abridge PMP annex – is compiled and shared by grantees rather than collected via surveys. The Aspen Institute currently collects (and anticipates continuing to collect) this data from grantees via an evaluation reporting template that must be submitted after grantees complete a cohort of their virtual exchange program. For example, a grantee that conducts a fall program that concludes in November is required to submit data disaggregated by participating institution, identifying which U.S. institutions are community colleges or Title I schools and which MENA institutions do not use English as the primary language of instruction. The evaluator is expected to analyze and include this data in its semi-annual reports.

Site Visits (5%): The evaluator will plan and conduct site visits of six grantee programs, a representative mix of programs (age level, location, grant type, etc.), the identity and timing of which will be determined later. For the purpose of planning and budgeting, there should be one in-person site visit planned in the United States and two in-person site visits planned as part of one trip in one country to be determined later in the MENA region. (Due to the number of grantees, the contractor will not be expected to conduct site visits of all grantee programs. The Aspen Institute will conduct visits or other oversight of the other programs.) Site visits should include observing virtual exchange activities while they are being conducted, focus groups with youth participants, and interviews with facilitators, program staff, and other relevant stakeholders such as institutional leaders to collect qualitative data from virtual exchange programs. The evaluator should account for the steps that may be required to get permission for conducting site visits, particularly when they involve educational institutions, minors, or other settings where advance permission may be required. The proposal may include a mix of virtual and in-person visits. The travel and logistics for in-person visits, whether in the United States, the Middle East, or North Africa, should be clearly described and justified. Site visits in the MENA region or virtually with participants in that region may be best conducted in Arabic or another language other than English; the evaluator should plan to retain a translator or data collection team member who speaks the appropriate language for a portion of the site visits if necessary. Within 10 business days following each visit, the evaluator will submit a short report of no more than two or three pages to the Institute (and shared with the grantee whose program was visited), describing what occurred during the visit, any notable data that was collected, and recommendations for improving the program. If the evaluator encounters or observes any urgent or concerning issues during the site visit, the evaluator must notify the Institute immediately.

Section 3: Capacity building (20%)

Grantee Training (10%): The evaluator will conduct an onboarding meeting with new grantees during Spring 2023. The evaluator will conduct three instances of the meeting for subsets of the grantees. Throughout the period of the contract, the evaluator will provide technical assistance to grantees via one-on-one and group communication by email and meetings, with an emphasis on addressing common and persistent challenges in data collection and reporting methods, with particular emphasis on helping grantees attain high survey response rates. This support and capacity building is intended to help grantees finalize their surveys, understand and implement the ECA MODE framework survey questions, prepare their facilitators/partners to circulate the surveys, get reliable responses and high survey response rates, organize and host site visits, and refine their programs based on the lessons learned over the course of the grant.

Survey Technical Assistance (5%): The evaluator will work proactively with grantees to ensure grantees’ survey drafts are complete, up to date, consistent with the ECA MODE framework and approved template (or any additions, subtractions, or changes are approved by the Aspen Institute), and functioning properly well in advance of the survey implementation date, which should be during the final week or session of a virtual exchange program. The evaluator will communicate with grantees prior to and during the early stages of each academic term to collect information about when grantees’ programs will begin and each; the evaluator will maintain and update as needed a database with this information and share access to the database with the Aspen Institute. The evaluator will monitor survey responses and follow up with grantees to ensure survey administration is occurring appropriately, notifying the Aspen Institute of any significant issues or shortfalls in response rates. The evaluator will analyze survey data, checking and cleaning it consistent with standards in the field, and include it in the reports described below.

Grantee Community of Practice (5%): The evaluator will plan, convene, and facilitate semi- annual videoconference meetings for all grantees to gather as a community of practice to share reflections on their experience conducting virtual exchange programs and evaluating their programs.

Section 4: Data Analysis, Reporting, and Dissemination (20%)

Data Analysis (5%): The proposal should address how the evaluator would meet the data analysis expectations laid out in the ECA MODE framework and attached PMP as well as any additional calculations and analyses the evaluator proposes to effectively and appropriately provide insights into how programs are being conducted and what effect they are having on participants.

Evaluation Reports (10%): The plan should include, for each of the two years of programming covered, a brief evaluation memo following the Fall academic term (and include Summer programming data) and a longer report on grantee evaluation late each summer following the Spring academic term. The memo following the Fall academic term should include participant demographic and survey data and does not need to include prose beyond a short introduction and explanations. The report following the Spring academic term should be approximately 8 to 12 pages and should include demographic, survey, and site visit data from the whole academic year (the just-concluded Spring term as well as the preceding Fall and Summer terms). The data should be presented disaggregated by the summer, fall, and winter/spring terms, as well as being presented aggregated for the whole academic year; for example, the first report would show data separately for the summer 2023, fall 2023, and winter/spring 2024 terms, as well as aggregated from all of those terms. The report should also contain a brief analysis of trends and findings emerging from the data, and a synthesis of common challenges and promising practices that emerged from the preceding year of grantee virtual exchange programs. This report may have a short appendix attached. Reports should be written in a way that can be easily excerpted or adapted for a general audience that is aware of virtual exchange but not expert on evaluation; they should be clear, concise, and readable and do not need to be produced with publication- ready layout or graphic design.

Dissemination (5%): The plan should include the evaluator’s participation in one online or in- person presentation or outreach or training event with or on behalf of the Institute over the course of the approximately two years covered by this agreement.

Section 5: Risk and Safety (5%)

Risk and Safety (5%): The evaluator should demonstrate familiarity with the challenges specific to the contexts where this project will be carried out. This includes laws and norms surrounding data collection and storage (particularly demographic or personally identifiable information); laws and norms about adult/evaluator interaction, especially with minors; sensitivities about certain topics both in the United States and in the MENA region, especially at the primary/secondary education level; and other topics. Issues in these areas can pose risks to the health and safety of youth participants, facilitators, and others associated with the programs and they can harm institutional partnerships, grantee and other organizations, and the JCSVEI as a whole. The evaluator will be expected to work proactively with the Aspen Institute and grantees to solicit and address any feedback or concerns about survey questions, other data collection methods, or any other aspects of the evaluation.

Section 6: Cost-effectiveness (10%)

Cost-effectiveness (10%): The proposal should present a clear plan for carrying out the work and should correspond to an attached budget that includes all expenditures and an attached budget narrative that describes all expenditures in detail. This section of the proposal should include an overview of the staffing plan, with a description of the role each staff member would have in carrying out the project. All costs must be consistent with any relevant federal guidelines. The assessment of this criterion will consider not only whether the total requested budget and the items within it are competitive, but also whether they are realistically sufficient to successfully carry out the work.

Proposal Parameters and Guidance

Timeline: The proposal should have a period of performance from April 1, 2023, through September 30, 2025.

  • April 2023: Inception and contractor onboarding; the evaluator collaborates with the Institute and Department of State to develop a draft evaluation framework
  • May 2023: The evaluator begins engagement with grantees, refines the evaluation framework, and develops a survey template with grantee and ECA input
  • June 1, 2023, through June 30, 2025: Conduct the evaluation of grantee virtual exchange
  • March 15, 2024: First evaluator memo is due (reporting on Summer 2023 and Fall 2023 survey data).
  • September 15, 2024: First evaluator report is due (reporting on the whole preceding academic year including Summer 2023, Fall 2023, and Spring 2024).
  • March 15, 2025: Second evaluator memo is due (reporting on Summer 2024 and Fall 2024 survey data).
  • September 15, 2025: Second evaluator report is due (reporting on the whole preceding academic year including Summer 2024, Fall 2024, and Spring 2025).

Proposal Length and Format: The proposal should be no longer than 20 pages, not counting attachments. It should be structured to correspond to the numbered sections above, which explain the competencies sought in the evaluator as well as the areas of work and the anticipated deliverables of the project.

Attachments: In addition to the proposal described above, please provide these attachments:

  1. A timeline, including major areas of activity and all milestones and deliverables;
  2. Budget, including international and domestic in-person site visits;
  3. Budget narrative, providing details and justification for all items in the budget;
  4. CVs for evaluator staff members, including at least one member with expertise in the Middle East and North Africa.

Submission Deadline and Method: Proposals are due no later than 5:00 p.m. U.S. Eastern Time on Thursday, February 9, 2023. Proposals must be submitted by email to stevensinitiative@aspeninstitute.org.

Selection Criteria:

    1. Experience and Capacity in Relevant Contexts (25%)
      • Experience and Capacity on Projects of this Scale and Breadth (5%)
      • Experience and Capacity at the Primary/Secondary and Higher Education Levels (10%)
      • Experience and Capacity in the S. and MENA Region (10%)
    2. Data Collection (20%)
      • Participant Survey Preparation (10%)
      • Grantee-Reported Administrative Data (5%)
      • Site Visits (5%)
    3. Capacity Building (20%)
      • Grantee Training (10%)
      • Survey Technical Assistance (5%)
      • Grantee Community of Practice (5%)
    4. Data Analysis, Reporting, and Dissemination (20%)
      • Data Analysis (5%)
      • Evaluation Reports (10%)
      • Dissemination (5%)
    5. Risk and Safety (5%)
    6. Cost-effectiveness (10%)

Questions about this request for proposals should be directed to Henry Shepherd at stevensinitiative@aspeninstitute.org.

More News

See All News