Monday, February 17, 2020

Running NServiceBus on Azure WebJobs

On this post we will address a common and cost-saving approach: how to deploy NServiceBus on Azure WebJobs.
Photo by True Agency on Unsplash

Most of us isn't there yet with microservices. But that doesn't mean that we shouldn't upgrade our infrastructure to reduce costs, or think proactively about performance, security, deployments, and scaling using simpler/newer technologies.

In this article we will discuss how to deploy NServiceBus endpoints on Azure WebJobs, including:
  • Why use WebJobs;
  • How to upgrade and NServiceBus endpoints as Azure WebJobs;
  • The necessary refactorings;
  • How to deploy our WebJobs;
  • Building, debugging, testing and production considerations;

Introduction

If you're running NServiceBus you probably use NServiceBus.Host. On Azure, the traditional way to run your backend using NServiceBus is by either deploying them to Cloud Services or Windows Services by utilizing NServiceBus.Host. The major problems with either of them is that, by being Platform on a Service (PAAS) you will have to patch, secure, and manage each of them separately. Plus, these technologies cannot be scaled out easily and usually cost way more than Azure App Services.

A good alternative to upgrading these services without significant changes while benefiting from a more modern, scalable and maintanable technology would be migrating these services to Azure WebJobs. Let's see how.

Azure WebJobs

Microsoft describes WebJobs as:
a feature of Azure App Service that enables you to run a program or script in the same context as a web app, API app, or mobile app. There is no additional cost to use WebJobs.
In other words, a WebJob is nothing more than a script or an application run by Azure. Currently supported formats are:
  • .cmd, .bat, .exe (using Windows cmd)
  • .ps1 (using PowerShell)
  • .sh (using Bash)
  • .php (using PHP)
  • .py (using Python)
  • .js (using Node.js)
  • .jar (using Java)
Note: as of the creation of this post, Azure still didnt' support WebJobs on App Service on Linux.

Types of WebJobs

WebJobs can be triggered and continuous. The differences are:
  • Continuous WebJobs: start immediately, can be run in parallel or be restricted to a single instance.
  • Triggered WebJobs: can be triggered manually or on a schedule and run on a single instance selected by Azure.
As backend services like NServiceBus and MassTransit traditionally run continuously on the background, this post will focus on continuous WebJobs.

Benefits of running NServiceBus on WebJobs

So what are the benefits of transitioning our NServiceBus hosts to WebJobs? In summary, NServiceBus running on WebJobs:
  • help reducing costs as no VMs or Cloud Services are required;
  • can easily scale up and out with your Azure App Service instance;
  • releases you from maintaining/upgrading/patching and securing your cloud resources;
  • is way simpler to deploy;
  • differently than NServiceBus Host, are not being deprecated. 
So let's review how it works.

Migrating NServiceBus backends to WebJobs

Migrating NServiceBus backend to WebJobs couldn't be simpler. However, NServiceBus's official documentation does not clearly describes a migration process so let's address the details here. Essentially we will have to:
  • Transform out host project in a console application;
  • Add a startup class refactoring the endpoint initialization;
  • Reference Microsoft.Azure.WebJobs so we can use the WebJob Api (optional);

Transforming our Host in a Console Application

The first part of our exercise requires converting our NServiceBus endpoint to a WebJob. Since WebJobs are essentially executable files we can start by simply transforming our endpoint project from a Class Library to a Console Aplication:

Referencing the Microsoft.Azure.WebJobs NuGet package

As recommended, to leverage the Azure Api we'll have to add the Microsoft.Azure.WebJobs NuGet package to our solution. After that package is added, we'll also have to refactor our  Main  method to correctly initialize and shutdown NServiceBus on Azure.

Adding a Statup class

Next, we have to add a startup class to our project. Essentially the compiler just needs a static void Main() method inside our solution so the project can be initialized. This is a simple example:
From here, not much will change. In summary, you will have to:
  • Remove the NServiceBus.Host pakage from the solution
  • Remove IWantToRunWhenEndpointStartsAndStops as no longer necessary
  • Refactor some of your settings because your deployment will likely change.
  • Optionally, add some sort of centralized logging like ApplicationInsights telemetry. Outside of the scope of this post but, you have to assume that now your backend will run in multiple instances and having a consolidade logging infrastructure wouldn't hurt.

Potential Problems

If you updated to the 3.x series of the Microsoft.Azure.WebJobs NuGet package, you probably realized that Microsoft aligned .NET Core and the .NET Framework on this release, probably already preparing for .NET 5. While that's excellent news, I you may also have conflicting dependencies and build errors as the one listed below.
error CS0012: The type 'Object' is defined in an assembly that is not referenced. You must add a reference to assembly 'netstandard, Version=2.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
That error can be fixed by:
  1. Referencing the NETStandard.Libray NuGet package on your WebJob;
  2. Adding <Reference Include="netstandard" /> just below the <ItemGroup> section on your WebJob's csproj file.

Building, testing and debugging the solution

After upgrading packages, refactoring code and fixing dependencies issues, we'll now have to fix potential build errors, and assert that the unit-tests are passing for the solution solution. Since I don't expect any major issues here, guess we can move ahead and review necessary changes for debugging.

Debugging

Because we transformed our NServiceBus endpoint in a console app, we remain able to start it on debug as previously. However there are some important details ahead. Make sure your WebJob project is set to start when debugging. To do so, we have to configure our solution to start multiple projects at the same time by right-clicking your solution, clicking Startup scripts, selecting Multiple startup projects and setting the Action column to Start for the projects you want to start.
Now set your Azure Storage connection string and the your project should start when debugging your solution integrated with Azure.

Running Locally

But if you want to run the WebJob using the development connection string ("UseDevelopmentStorage=true"), you will realize that the initialization fails. WebJobs can't run with the Azure Storage Emulator:
System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException:
   Failed to validate Microsoft Azure WebJobs SDK Storage account.
   The Microsoft Azure Storage Emulator is not supported, please use a Microsoft Azure Storage account 
hosted in Microsoft Azure.
   at Microsoft.Azure.WebJobs.Host.Executors.StorageAccountParser.ParseAccount(String connectionString,
String connectionStringName, IServiceProvider services)
   at Microsoft.Azure.WebJobs.Host.Executors.DefaultStorageAccountProvider.set_StorageConnectionString(String value)
   at Microsoft.Azure.WebJobs.JobHostConfiguration.set_StorageConnectionString(String value)
   at ATIS.Services.Program.d__1.MoveNext() in C:\src\ATIS\ATIS.Services\Program.cs:line 49
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
So if we have a console application ready to run, why do even have to start the Jobhost at all?

That's a common scenario in which we want to run a different logic on debug and release modes, I usually resort to preprocessor directives where I have different implementations for debug and release. The next snippet shows it on line 15:

Going Async

To finish, let's make code asynchronous. The two previous snippets could be refactored into  something like:

Deployment Considerations

Now let's discuss deployment. Three important things to note:
  1. Variable collisions - you will probably have to change or rename variables since some of them may overlap;
  2. Changes in the deployment process - to be run as a WebJob, your backend will have to be deployed with your web app;
  3. Transformations - will probably have to change some transformations so they're also available for the backend.

Deploying our WebJob

A continuous webjob should be deployed with your Azure App Service on App_data/jobs/continuous. Triggered jobs should go into the App_data/jobs/triggered folder. The screenshot below shows both types running under my App Service:
Another way to confirm that is by using the Azure Serial Console and cding into that folder:

    Changing the Deployment Process

    So how do we get our WebJobs on App_data/jobs/continuous ? Well, that will obviously depend on how you're deploying your services. The most common deployment strategies are:
    1. ClickOnce from Visual Studio
    2. Custom PowerShell scripts
    3. Using an automated deployment tool (ex. Azure DevOps, CircleCI, AppVeyor, Octopus Deploy, etc)
    4. By hand ๐Ÿ˜ข 
    Let's discuss the two most common ways: NuGet packages and PowerShell Scripts.

    NuGet Packaging

    A common way to package code is building NuGet Packages. I won't extend much into that as it's outside of the scope of this post but I want to highlight that getting our project within our NuGet package is very simple. If you're already building NuGet packages, by simply add a reference to your project on the <files> section, we're telling msbuild to package our project with our web application:

    PowerShell

    If your CI/CD supports PowerShell, we could add the below snippet in a step just before the release:
    # PowerShell Copy-Item ..\MyApp.Backend\bin\release App_Data\jobs\continuous\MyApp.Backend -force -recurse

    Post-build event

    Another alternative would be running a specific command you your post-build event. Just keep in mind that this would also slow your local builds unless if add some contitional around it:
    # xcopy xcopy /Q /Y /E /I ..\MyApp.Backend\bin\release App_Data\jobs\continuous\MyApp.Backend

    Testing Considerations

    With the deployment out of the way, let's what should be considered when testings:
    1. Performance - I didn't see any degradation performance changes but that could not be your case. Test and compare the performance of this implementation.
    2. Failures - A crashing WebJob won't crash your App Service but, have you tested edge cases?
    3. Scale - the number of instances can be different from your current setup. Can you guarantee that no racing conditions exist? 
    4. Logging - Do you need to change how your application logs its data? Are the logs centralized and easily accessible?
    5. Remoting - Because you departed Windows VMs and Cloud Services doesn't mean that you can't access the instance remotely. The Azure Serial Console is an excellent tool to manage and inspect some aspects of your job.

      Production Considerations

      Still there? So let's finish this post with some considerations about running NServiceBus on WebJobs in production. I expect you tested your application against the items highlighted on the previous section.

      I'd recommend that before going to production your team,
      • build some metrics - around the performance before deploying so you know what to expect;
      • use Azure Deployment Slots - to validate production before setting it live;
      • doubletriple-checking your configurations - because it's a new deployment to a new environment and some configurations were changed, weren't they?
      • keep an eye on the logs - as we always do, right? ๐Ÿ˜Š 
      • do a post-portem after the deployment - so your team reflects on the pros/cons of this transtion.

      Final Thoughts

      Migrating NServiceBus from VMs to WebJobs was a refreshing and cost-saving experience. Over time, we felt the heavy burden of managing VMs (security patches, firewalling, extra configuration, redundancies, backups, storage, vNets, etc) not to mention how difficult it is to scale them out. Because webjobs scale out with the app service and add virtually no extra cost to it, we definitely gained in a lot of fronts.

      Some of the positive impacts I saw:
      • quicker deployments
      • easier to scale out
      • cheaper to run
      • more secure
      • reduced zero ops
      • simpler deployments
      • decent performance
      I will keep monitoring the service and update this post with findings in the future. Keep tuned!

      More about NServiceBus?

      Want to read other posts about NServiceBus, please also consider:

      References

      See Also

      For more posts about NServiceBus, please click here.

      Share

      Any comment about this page? Please reach out on Twitter