it’s a common scenario: a principal, excited to begin personalized learning in his/her building, hears requests from teachers for a certain k-5 adaptive digital program for math and uses his/her discretionary budget to purchase a school-wide license. meanwhile, at the district office, the curriculum director for math has just approved a new textbook-based curriculum which comes with an online version of the text. in the it department, fears of student privacy breaches bring about a limit on which programs are permitted on the network. and in the educational technology department, if the district has had the foresight and ability to prioritize having such a position, there is someone trying to juggle it all.
all of the parties in this scenario are well-intentioned and want what’s best for students and teachers. they have prioritized personalized learning and know that integrated digital content is one of the core four ways to do it. but they lack:
1) a vision for how digital content and tools will support personalized learning
2) an alignment between their offline and online content and the role of each tool
3) an inclusive process to review, demo, and select digital content and tools
this white paper focuses on the third component: the need for a formalized, repeatable, and
the importance of developing a process to review, demo, and select digital tools
considering the scenario discussed above, why and how would a process for reviewing, demoing, and selecting digital content and tools be helpful?
formalization: everyone involved knows what the process is and follows a standard procedure that leads to predictable results.
repeatability: lightning doesn’t have to strike twice. if a digital tool is needed for another content area or grade level, the process is documented and able to be repeated.
scalability: if personalized learning is rolling out in waves of schools, it is important to have a process that works just as well for one or two schools as 100.
budget considerations: process that everyone knows and follows will lead to a reduction in duplicate spending and potential discounts for larger implementations (i.e. individual versus school versus districtwide license discounts).
stakeholder engagement: from teachers to parents and students, all stakeholders will be more involved in the process and will understand why certain purchasing decisions are made.
accountability: a standard process can lead to greater accountability across multiple roles: teachers and schools are held accountable for usage, district leaders are held accountable for why certain decisions are made, and providers are held accountable for the service and product they deliver.
many districts and schools use an organic approach to digital content procurement, which can work in the short-term but can lead to redundancies, sticky questions about the budget, and a tendency towards “shiny object syndrome.” as ryan russell, assistant superintendent for innovation and improvement at the metropolitan school district of warren township (in) says, “previously, the selection of digital tools was ‘person-centric’ meaning that one individual identified a tool and was in control of the budget. what we ended up purchasing varied depending on the person with the power at that time. we could get distracted by the ‘sparkle and glitter’ of the tools and we lumped them all into one category: digital tools.”
this is not to say that the process should remain rigid -- the creators of the process should keep a design-thinking mindset to constantly reflect and iterate on the needs of the users and the intended outcomes. as you will see below, we outline a recommended process broken into three parts: part a: review, part b: demo, and part c: select. the process that you create may include some or all of these elements based on the unique needs of your district.
before beginning the steps below, ensure that you have:
district case study: marion central school district (ny)
when superintendent don bavis took the reins of marion central school district (mcsd), a small rural district about 30 minutes outside of rochester, ny, he knew that personalized learning would be a key focus area. he communicated to key stakeholders about “the one initiative initiative” through an op-ed in a local newspaper and then combined his vision for personalized learning with his team’s vision. the district’s vision became: “we personalize learning today to empower every black knight to thrive tomorrow“ (the black knight being the district mascot). his team then kept this vision in mind as they went through the curriculum mix framework with 瑞士vs喀麦隆亚盘赔率 . they determined that while the elementary and middle school had appropriate adaptive ela and math programs that met the district’s vision, the high school lacked foundational content across all content areas that supported teachers in their efforts to personalize. high school teachers were crafting many of their lessons independently and could benefit from a provider that could provide both instruction and practice for students with minimal teacher lift.
in the first part of the process, you will use your identified area of need to determine the ideal process and timeline for selection; to cast a wide net of providers and tools that could potentially address your need; and to narrow to a list of providers to invite forward to demo (part b).
the person who leads the digital content selection process may vary widely from district to district depending on size and staffing, but the selection of one lead is essential. some districts may want to create a small team to support the lead in his or her work, but one and only one person should be ultimately accountable for moving the work forward.
this person might be:
if possible, this person should have the ability to ensure that decisions are not made in silos and to interact with stakeholders from the superintendent to principals, teachers, and students.
once a lead has been established, the lead will need to take a look at the remaining steps of the process and determine which to include (depending on the needs of the district and the focus area selected).
once the process has been solidified, the lead will establish a timeline. establishing a timeline is essential as districts often discover hidden obstacles at the end of the process with procurement, account setup, and training which can throw a wrench into their carefully laid plans. it may be easiest to use the principle of backwards design when planning your timeline. get feedback from school leaders as to when they would like to begin using this tool and backwards map from there, factoring in time for the selection process, procurement, account setup, and training.
it may be helpful to backwards plan and establish the following dates:
target review date: the date by when you will have made a first pass at reviewing the market to determine which providers might address your area of need and have narrowed them down to a demo list of 3-5 providers.
target demo completion date: the date by when you will have demoed your short list of providers, received feedback from key stakeholders, and narrowed down to a final list of 2-3 providers.
target selection date: the date by when you will have made your final selection decision, ensuring there is enough time for procurement, account setup, and training prior to the target use date.
target use date: the date when school leaders would like to begin using this tool.
one district that has given a good deal of thought to determining a lead, process, and timeline for selection is freehold township elementary school district (nj). pamela nathan, assistant superintendent for curriculum and instruction, led the selection of digital content and tools for her district last spring with the support of her superintendent ross kasun. one of the early decisions nathan and kasun made was to form a “digital vetting committee” to assist with the selection process. educators from across the district filled out an application for a position on the committee which, for teachers, provided an additional stipend because of an expectation that they would spend approximately an additional ten hours on committee work outside of the school day.
nathan stressed that the process their team established was “consultative but not consensus-building.” even though the digital vetting committee would be able to give its feedback, the ultimate timeline, process, and final decision would rest with the digital content lead.
questions to consider
district case study: syracuse city school district (ny)
one of the key decision-makers for digital content and tools for syracuse city school district (scsd) is manami tezuka, director of library media services. tezuka works closely with nate franz, the assistant superintendent of teaching and learning, to craft a portfolio of digital content and tools that provides teachers in personalized learning classrooms with the opportunity to choose among options. teachers in scsd know to call tezuka if they have a question about anything related to digital content, including feedback about a vendor’s product, training, or support.
this step can be the most challenging for many district leaders. an internet search for “best ela digital content” or “most popular learning management system” will only get you so far.
thankfully, if you’ve been using the 瑞士vs喀麦隆亚盘赔率 phase one and phase two white papers you will have a very specific area of need (for example: we need an adaptive ela program for grades k-2 that also provides the ability to assign lessons) that can help you to focus.
how do you identify a “long-list” of programs out there that might meet your need?
as you conduct your initial research, you’ll want to begin compiling your “long-list.” collect the name of the provider, the subject area(s) and/or grade level(s) served, links to the provider’s website/any reviews and notes on your initial impression. this initial research does not yet need to be in-depth, as that step will come later.
your “long-list” should consist of 5-7 providers that upon initial research appear to meet the needs you have identified. in the case of some very specific needs, you may only find that 3-5 providers exist to meet them, and that is ok. start your "long-list" with as many viable options as you can find, but do not exceed 5-7.
questions to consider
step 3: narrow providers to a demo list and develop a demo rubric
once you have a “long-list” of 5-7 providers, you will want to narrow it to 3-5 contenders to move along to the demo part of the process. how do you make the decision of which 3-5 providers to demo? this may vary widely depending on your district’s needs and timeframe. for instance, you may have a long window to demo and want to increase knowledge on as many providers as possible. or you may have a shorter window and already have an idea of a few providers that have caught your eye.
one easy way to “make a first cut” is to list out the “must-haves” versus “nice-to-haves” for your area of need.
example - focus area of need: adaptive ela program for k-2
must-haves
nice-to-haves
when deciding on your “must-haves” versus your “nice-to-haves,” jaraun dennis, chief technology officer at uinta school district one (wy), recommends keeping one question at the forefront of your mind: “what are you hiring digital content to do?” he says, “so many times district staff purchase a digital tool asking it to do x, but it really does y. you’ve got to make sure you know what you want to hire your digital content to do and make sure that it actually does that thing.” russell of warren agrees: “the most helpful part of the work we’ve done has been the concept of categorizing tools to identify specific purposes -- digital or not. before we lumped all online tools together when really they can do very different things.”
what you’re hiring your digital content provider to do (your “must-haves” and “nice-to-haves”) will vary depending on your district’s needs. for some providers, it may be easy to determine if they meet your “must-haves” through a quick web search of the product’s website or the edsurge edtech index. for others, you might not be sure yet if they do or do not meet your qualifications. that is ok -- you will learn more as you begin to talk to teachers who have used the products or receive demos on these products.
at this point, you may want to develop a more in-depth rubric to score providers before receiving demonstrations for them. this is an important step because it will ensure that you’re not swayed by a new “flashy” aspect of a provider if it does not address your specific area(s) of need. this will also help you know which questions to ask providers during their demonstration as well as help providers know which aspects of their products to present on, rather than giving you an “everything but the kitchen sink” presentation.
heather van looy, program specialist for instructional technology for fulton county schools (ga), has thought a great deal about what makes the right fit for digital content or tools. in her previous role as an instructional coach at river trail middle school, she was the go-to person in her school for teachers who had questions about particular providers. in her current role, she supports leaders across the district as they’re making digital content decisions. she believes that the first question you need to ask yourself is: are you reviewing a digital content provider or a digital tool? by this, she means is this product going to be used primarily for instruction (i.e. digital content) or creation (i.e. digital tool)? based on that, she thinks about different considerations. on the following pages you’ll find a list of considerations compiled with input from van looy and other district leaders.
consideration for all digital content and tools
use:purpose:
specific considerations for digital tools
purpose:
use:
with these considerations in mind, you’re now ready to develop a demo rubric for your district.
here are a few example demo rubrics from our partner districts as well as our own:
• fulton county schools
• uinta school district one
• dc public schools
• 瑞士vs喀麦隆亚盘赔率
questions to consider
district case study: fulton county schools (ga)
“it’s like the wild wild west out there!” we heard this from district administrators at fulton county schools (fcs), referring to the plethora of digital content and tools being used across the district’s almost 100 schools. fcs wanted to find a way to vet what they called the “grey market” of tools being purchased and used by schools and teachers. the grey market was impeding the district’s effort to coalesce around a single provider for ela and math. in conjunction with 瑞士vs喀麦隆亚盘赔率 , fcs created a digital content evaluation tool that vetted the products in the grey market based on key qualifications outlined by the district. products must achieve at least a certain score on this tool in order to be approved for use in the district.
"so many times district staff purchase a digital tool asking it to do x, but it really does y. you've got to make sure you know what you want to hire your digital content to do and make sure it actually does that thing." —jaraun dennis, chief technology officer at uinta school district one (wy)
step 4: initiate contact with providers and conduct initial demo
congratulations! you are armed with both a clear focus and measures of success for meeting your area of need. you have made it to the step of contacting the providers. again, we recommend that you not begin to contact providers before your initial research and determination of your demo rubric categories. as van looy of fulton says, “don’t contact providers unless you’re really serious because you might not be able to get rid of them!” you want to make sure for your sake and the sake of a provider that you don’t waste each other’s time.
how can you initiate contact with providers? if you happen to know the name of your regional representative for a product, that can be the easiest and most direct way. don’t know who that is? try leveraging your network to see if anyone knows who the best person to speak to is. as dennis of uinta puts it, “reaching out to other districts in your state can be very helpful. they’ve likely been through this process and can give you tips and tricks on working with providers.”
if you’re not able to get a direct line to the provider, one easy fall-back is to fill out a request for information on the product’s website. you will want to describe in the comment or notes section of your request that you make purchasing decisions for your district so that it will set you apart from one-off teacher or principal requests.
if you don’t get a response in a timely manner through a phone call or email, that is a “bad sign” says russell of warren. “how they treat the district through the demo process is how they will treat you once you’ve purchased the product. you may even want to give the account managers deadlines -- it will show if they prioritize your district.” what does the “initial demo” process look like? we recommend:
1. phone call with regional representative - you may first want to start with a quick 15-30 minute phone call with the regional representative to get a few of your burning questions answered. prioritize asking about your “must-have” qualifications so that you can easily know if it is worth investing more time and energy in this provider. we recommend that only the project lead attends this call.
2. webinar demo of the product - request a 30-45 minute demonstration of the product. this will be a chance to see the product in action. you may want to include a few other key team members in this webinar to get other opinions. if possible, see if you can get demo accounts of the product from the provider before the demo so that you can ask targeted questions.
3. in-person demo - if the provider is meeting your key requirements after the webinar, you may want to request an in-person demonstration for more stakeholders to see and interact with the product.
questions to consider
step 5: host in-person demo or digital content fair
easily the high point of any digital content selection process is the in-person demo or digital content fair. a well-run digital content fair serves many purposes: 1) you can learn a great deal about the prospective providers; 2) you can involve stakeholders at multiple levels; 3) you can get feedback from a variety of people at once.
as with all parts of this process, your ability to host either an intimate in-person demo with just your core team versus a full-day digital content fair with hundreds of participants will depend on your district’s needs, culture, and timeline. you might even need to skip the fair altogether -- “we tried to hold a digital content fair but our district is 90 miles from an airport!” says dennis of uinta. in that case, a webinar or virtual demo may be more feasible.
if you’re opting to host an in-person digital content fair, these can range from half to full-day events, with attendees generally being a representative sample of the school leaders and teachers who will be using the product. bavis of marion says, “we were very purposeful in how we selected teachers to participate in our digital content fair. we were sure to include some teachers who we thought might be ‘nay-sayers.’ before our fair, i’d say 80% of our teachers did not truly understand what digital content was -- they were accustomed only to its use in credit recovery. the fair was a great way to help push their understanding as to how digital content could be helpful for more than that.”
russell of warren agrees. “based on the culture of your district, some teachers may feel like curricular decisions don’t impact them,” he said. “we wanted to make an intentional shift to include them.”
here are some of the keys to a good digital content fair:
make sure the right providers are invited
for each focus area you’re targeting, you will want to invite 3-4 providers. if you’re targeting more than one focus area, this could mean that you have 10+ providers attending. you may want to stay organized with a tracker like this one from charleston county school district.
set expectations with the providers
ensure providers understand who will be attending the fair -- is it mainly teachers, school leaders, or district staff? ensure they also understand your district’s context, your musthaves and nice-to-haves, and anything else you’re looking for. as russell of warren says, “be transparent with vendors -- if you can get teachers on board, we will make it work,” meaning that if the provider impresses the most important customers (i.e. the teachers or students), then the district team is sold. here are example letters to providers (fulton county schools, charleston county school district example 1, example 2).
create a detailed schedule
ensure you have the time and space for all providers. consider how long each session with a provider will be; if you’re able to let providers present more than once; if you’ll have a individual rooms for providers to demo; or if you’ll have a big open fair where participants will be able to flow through freely. here are example schedules (marion central school district, fulton county schools, charleston county school district)
invite key stakeholders
consider if you’ll invite teachers, school leaders, or district staff. consider your focus area and who would be best to provide feedback on that focus area. here are two examples from fulton county schools that show communication with participants: example 1 and example 2.
collect feedback from stakeholders on providers
your goal should be to collect the necessary information from the providers in order to fill out your demo rubric created in the previous step. feedback from school leaders, teachers, and other district office staff during a digital content fair can be extremely useful to completing that rubric. here are examples of surveys or demo guides used during or directly after digital content fairs. (fulton county schools, charleston county school district, freehold township elementary school district, loudoun county public schools)
stay organized
hosting a digital content fair involves many people and moving pieces. consider starting a project plan like this one from charleston county school district.
questions to consider
district case study: charleston county school district (sc)
charleston county school district’s goal is to ensure that schools across their district have equitable access to digital content and tools, while still providing autonomy to personalize learning with the program that works best for their students. they found that they had a wide variety of tools being used across the district but needed some commonality to measure the correlation between digital content usage and nwea scores. in order to meet those goals, they decided to hold a digital content fair for over 200 participants from 87 schools, with 12 vendors attending. every principal and one to two other representatives from each school were invited to attend sessions and provide feedback.
step 6: get feedback from key stakeholder groups
coming out of your in-person demos, you should have a variety of feedback from stakeholders on their initial impressions of digital content providers. if possible, during your in-person demo request a variety of demo accounts to use over the course of weeks or months. consider requesting:
the variety of accounts will allow you to experience the product beyond just the student level and will expose the type of data and reporting you will receive at the various levels (teacher versus school leader versus district).
encourage teachers to go into the demo student account to see if it may meet your identified need. sometimes spending an hour in a student account of a potential provider is an eyeopening experience.
if possible, consider inviting students to help serve as “beta testers” for the products as well. as dennis of uinta says, “if students aren’t engaged with a program, you’re going to waste your money. earning credentials, creating an avatar, customization -- these are all ways that students enjoy engaging in programs.” he says that you should consider asking the students these questions:
at this point, some districts may also choose to start a small pilot of one or two providers to see which one will best meet their needs before expanding on a larger scale.
you can view another pilot proposal form from freehold township elementary school district here.
questions to consider
step 7: narrow to final shortlist and ask for pricing information
based on the feedback you received from stakeholders at the digital content fair and afterward through the “beta test,” you should have your eye on the final 1-3 providers that you’d like to move to the final selection phase of the process.
before moving forward with your finalists, you may want to reach back out to the providers who provided a demo but were not ultimately selected. inform them of your decision to move forward with other providers, give them the reasons for your decision (if you wish), and share with them a timeline of when you may be reevaluating your digital content so that they can reach back out to you at that point.
when you’re ready to move forward, contact your finalists for additional pricing information or a pricing quote. some providers price by the student, while others will price by the teacher or building. for most providers, there are discounts available if you purchase a site-wide or district-wide license. you may find it helpful to request information on:
you will also want to ask if these prices include any necessary professional development or training or if that is purchased separately and inquire about any extra fees such as an account setup fee. sometimes pricing for digital content can be as confusing as a cell phone bill with hidden fees!
van looy of fulton says, “it may be more beneficial to buy a site license even if you only need a few licenses because of the price breaks. you have to do the math.
bavis of marion encourages leveraging buying organization membership. in his region of the wayne-finger lakes, there is a buying organization (boces) relationship that ensures the small districts in the region are able to take advantage of larger-scale pricing discounts. “encourage providers to work with your boces if you’re able,” he says.
questions to consider
step 8: complete selection rubric, select final provider, and negotiate
if you are using a selection rubric, now is the time to fill it out for the final 1-3 providers. this is a step that many districts skip but leads to sticky questions later about why a final decision was made.
as you’re looking over the rubric scores and thinking about making your final selection, here are a few negotiation tips:
questions to consider
step 9: share, train, implement and reflect
congratulations! you have purchased a new digital content program or tool. your next steps will be to:
share the final decision with stakeholders
nathan and kasun of freehold shared the final decisions with their “digital vetting committee” then with the rest of the staff. they also gave a presentation to their board on the digital content providers that were chosen.
begin setting up accounts with the provider
this may take some time depending on the provider’s ability to integrate with your district’s student information system (sis).
schedule any trainings needed
most digital content providers have required or recommended professional development before starting use. many districts, including freehold township and charleston county, will designate one person per building to be the point person, expert, or turn-key trainer. you’ll want to consider what structures you will put in place in the long-term to be able to support the professional development of your teachers beyond the provider’s initial training.
set expectations for use
wesley of orange shared that after the initial pilot was completed and a provider was selected, the district wrote a usage plan that specified:
usage
implement
as implementation begins, consider ways you can encourage your staff to utilize the product to its full amount. you could hold a “launch day” celebration or a usage competition like “march math madness.” many providers offer printable certificates or wall trackers to track progress that excite students and teachers alike. you can see other ways for students and teachers to track usage in our white paper on the core four of personalized learning.
reflect
reflection should occur on both a short- and long-term basis. frequently consider: are we providing adequate training and support to our staff? are we using the program with fidelity? what student data is this program providing and how can we target our instruction more effectively based on this data?
on a longer-term basis, your over-arching question should be: is this tool helping us achieve our district’s vision for personalized learning? consider creating set intervals at which you will get feedback on the provider throughout the year or will reflect more formally on its usage and effectiveness at meeting the vision and focus area of need. some districts will start this process at six
here are few other thought processes that district leaders undergo when considering whether or not to renew a program:
questions to consider
there is no one right way to select digital content, but there are a lot of things to do to get it right. this white paper has provided an extensive list of steps to take and questions to consider as you begin the process. even if your district has already started the
about the