Windows Application Testing: Are You Piloting More Than 15%?

All Posts

Windows Application Testing: Are You Piloting More Than 15%?

Annual Windows updates are here to stay. As Microsoft introduces Windows 11 this year it’s moving to annual feature updates during the second half of the year. If you are on Windows 10, don’t panic, Microsoft has said it will support at least one semi-annual channel until October 2025. It’s going to provide support for enterprise Windows 11 updates for 36 months which may give you a little more time, but for your company to take advantage of the latest features, you’re probably going to stick with yearly rollouts so it’s time to streamline your testing process.

If you’ve recently moved to Windows 10, you’ll be used to the luxury of having months to pre-test applications to ensure they are compatible with the release. Sadly, that luxury is no more. Even if you’ve been managing Windows 10 updates for a while and thinking of moving to Windows 11, you could be feeling overwhelmed with this constant annual update cycle.  

You may have already worked out a way to reduce the amount of testing you do – after all there is no getting away from testing completely. But if you are pre-testing or piloting more than 15% of your estate it’s going to be a challenge to complete your updates within that annual timeframe. So, if you want to find out a way to reduce the time you spend on testing and piloting, read on. 

Choose your categories 

First you need to categorize your applications – and if you want to know more about that – read our blog ‘How to Categorize Applications Before Rolling Out Updates’. The categories that you place your applications in will define how you treat them during the update: 

  • Tier 1: Mission Critical Apps: These are the applications that support essential business services or critical users. There is no getting away from lab testing and certifying that these will work with the update ahead of rollout. Once you identify a list of your critical apps, you’ll likely discover that these are at most 10% of your estate – and usually less. 

  • Tier 2: Important Apps: These are the applications that are being used widely across the organization, so if there’s an issue when the update is rolled out, it could affect a lot of users, or business operations – but not in a critical way. We recommend testing these by rolling out software pilots, testing with small groups of users that you can work with to get feedback on before rolling out more widely. 

  • Tier 3: Low-risk apps: These are usually commercial off the shelf (COTS) or home-grown apps that are non-essential, used for day-to-day operations and likely leveraged by small groups of users based on department, geography or another factor. You can start rolling out updates to users, adding more users to deployment rings or waves as you go and only investigate if you hear of any issues. 


Use your tools, systems, and databases to find out how you should categorize your enterprise applications. The majority will fall into the low-risk category, which should already reduce the amount of time you spend on testing. But how should you go about managing your pilot testing for the Tier 2- important applications? 


Defining your pilots 

Once you have identified the applications that should be Tier 2, you then need to define a process for pilot testing. Do this by: 

  • Assigning a risk score to applications – and define your level of risk – that could be based on the number of people that rely on that application (refer to your source data to find this information). 
  • Define the number of users or pilot test rings you want to assign to each level of risk. 
  • Roll out pilot waves accordingly by first picking typical or friendly users – the ones that have helped you in the past with feedback. Just make sure that they have okayed the update before you roll out further. 

Once you’ve rolled out ‘X’ number of pilots (however many you have defined) to satisfy your requirements based on the risk score you have assigned to your Tier 2 applications, and users have reported that everything is ok, you can begin rolling out to more and more users in the organization. 

By categorizing your applications this way, you can reduce pre-testing and pilots to around 15% of your estate. Sound good? Even better, that’s going to reduce the time you spend on testing to around a few weeks, or at most a couple of months – which is much more manageable when tackling annual Windows servicing programs. 

If you want to simplify things further, you could consider adopting automation for more than 50% of the manual, repetitive tasks that you will perform when inventorying, categorizing and pilot testing your applications. By doing this you can cut the time and effort as well as the errors created by manual input – making the process much smoother. 

Schedule a demo with ReadyWorks to understand how applying intelligent automation can help cut the time and effort of testing ahead of your enterprise Windows update. 

Related Posts

The Cost of Dark Data

The amount of data we are producing is rising at a dramatic rate. Statista predicts that b...

The Cybersecurity Risks caused by Human Error and How to Avoid Them

Through its research on the Psychology of Human Error, Stamford University released a repo...

How to Stay Off the Cyber Security Data Breach Naughty List

As the number of cyberattacks continues to rise across the globe, many are warning that th...