Running NServiceBus on Azure WebJobs

On this post we will learn how to build and deploy NServiceBus on Azure WebJobs.

Photo by True Agency on Unsplash

Most of us aren't there yet with microservices. But that doesn't mean we shouldn't upgrade our infrastructure to reduce costs, increase performance, enhance security, simplify deployment, and scaling using simpler/newer technologies. On this article let's review how to deploy NServiceBus on Azure WebJobs and discuss:

• why use WebJob
• how to upgrade your NSB endpoints as Azure WebJob;
• the refactorings you'll need
• how to build, debug, test and deploy our WebJob;
• production considerations

If you're starting a new project or have a light backend, I'd recommend you to consider going serverless on Azure Functions or AWS Lambda with SQS.

Introduction

On Azure, NServiceBus is usually deployed on Cloud Services or Windows Services. The major problem with both is that, by being Platform on a Service (PAAS) they are difficult to update, secure, scale out and have to be managed separately.

Depending on your usage, migrating your NSB backend to Azure WebJobs could be a good alternative to reduce costs, maintenance and increase security. Plus, since webjobs are scaled out automatically with your Azure App Services, you would literally get autoscaling on your NSB backend for free!

Azure WebJobs

So let's start by quikcly recapping what are webjobs. According to Microsoft, webjobs are

a feature of Azure App Service that enables you to run a program or script in the same context as a web app, API app, or mobile app. There is no additional cost to use WebJobs.

In other words, a WebJob is nothing more than a script or an application run by Azure. Currently supported formats are:

• .cmd, .bat, .exe (using Windows cmd)
• .ps1 (using PowerShell)
• .sh (using Bash)
• .php (using PHP)
• .py (using Python)
• .js (using Node.js)
• .jar (using Java)

As of the creation of this post, Azure still didnt' support WebJobs on App Service on Linux.

Types of WebJobs

WebJobs can be triggered and continuous. The differences are:

• Continuous WebJobs: start immediately, can be run in parallel or be restricted to a single instance.
• Triggered WebJobs: can be triggered manually or on a schedule and run on a single instance selected by Azure.

Since backend services like NServiceBus and MassTransit traditionally run continuously on the background, this post will focus on continuous WebJobs.

Benefits of running NServiceBus on WebJobs

So what are the benefits of transitioning our NServiceBus hosts to WebJobs? In summary, this appoach will:

• reduce your costs as no VMs or Cloud Services are required
• allow your backend to scale up automatically with your Azure App Service
• eliminates your concerns about maintenance/upgrade/patch and security
• is way simpler to deploy
• differently than NServiceBus Host, is not being deprecated

So let's review how it works.

Migrating NServiceBus backends to WebJobs

Migrating NServiceBus backend to WebJobs couldn't be simpler. Since, NSB's official documentation does not clearly describes a migration process, let's address it here. Essentially you'll have to:

• transform your host project in a console application
• add a startup class and refactor the endpoint initialization
• add a reference Microsoft.Azure.WebJobs so you can use the WebJob Api (optional)

Transforming our Host in a Console Application

The first part of our exercise requires converting our NServiceBus endpoint to a WebJob. Since WebJobs are essentially executable files we can start by simply transforming our endpoint project from a Class Library to a Console Aplication:

Referencing the WebJob package

As recommended, to leverage the Azure Api we'll have to add the Microsoft.Azure.WebJobs NuGet package to our solution. After that package is added, we'll also have to refactor our Main method to correctly initialize and shutdown our bus.

Adding a Startup class

Next, we have to add a startup class to our project. Essentially the compiler just needs a static void Main() method inside our solution so the project can be initialized. This is a simple example:
From here, not much will change. In summary, you will have to:

• Remove the NServiceBus.Host pakage from the solution
• Remove IWantToRunWhenEndpointStartsAndStops as it's no longer necessary
• Refactor some of your settings because your deployment will likely change.
• Optionally, add some sort of centralized logging like Application Insights since your backend will run in multiple instances. And having a consolidated logging infrastructure wouldn't hurt.

Potential Problems

If you updated to the 3.x series of the Microsoft.Azure.WebJobs NuGet package, you probably realized that Microsoft aligned .NET Core and the .NET Framework on this release, already preparing for .NET 5. While that's excellent news, I you may also have conflicting dependencies and build errors as the one listed below.

error CS0012: The type 'Object' is defined in an assembly that is not referenced. You must add a reference to assembly 'netstandard, Version=2.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.

That error can be fixed by:

1. Referencing the NETStandard.Libray NuGet package on your WebJob;
2. Adding <Reference Include="netstandard" /> just below the <ItemGroup> section on your WebJob's csproj file.

Building, testing and debugging the solution

After upgrading packages, refactoring code and fixing dependencies issues, we'll now have to fix potential build errors, and assert that the unit-tests are passing for the solution solution. Since I don't expect any major issues here, guess we can move ahead and review necessary changes for debugging.

Debugging

Because we transformed our NServiceBus endpoint in a console app, we remain able to start it on debug as previously. However there are some important details ahead. Make sure you set your WebJob project to start when debugging. To do so, we have to configure our solution to start multiple projects at the same time by right-clicking your solution, clicking Startup scripts, selecting Multiple startup projects and setting the Action column to Start for the projects you want to start.

Now set your Azure Storage connection string so you can debug your project with Azure.

Running Locally

But if you want to run the WebJob using the development connection string ("UseDevelopmentStorage=true"), you will realize that the initialization fails. WebJobs can't run with the Azure Storage Emulator:

System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException:
   Failed to validate Microsoft Azure WebJobs SDK Storage account.
   The Microsoft Azure Storage Emulator is not supported, please use a Microsoft Azure Storage account 
hosted in Microsoft Azure.
   at Microsoft.Azure.WebJobs.Host.Executors.StorageAccountParser.ParseAccount(String connectionString,
String connectionStringName, IServiceProvider services)
   at Microsoft.Azure.WebJobs.Host.Executors.DefaultStorageAccountProvider.set_StorageConnectionString(String value)
   at Microsoft.Azure.WebJobs.JobHostConfiguration.set_StorageConnectionString(String value)
   at ATIS.Services.Program.d__1.MoveNext() in C:\src\ATIS\ATIS.Services\Program.cs:line 49
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()

So if we have a console application ready to run, why do even have to start the Jobhost at all?

That's a common scenario in which we want to run a different logic on debug and release modes, I usually resort to preprocessor directives where I have different implementations for debug and release. The next snippet shows it on line 15:

Going Async

To finish, let's make code asynchronous. The two previous snippets could be refactored into  something like:

Deployment

Now let's discuss deployment. Three important things to note:

1. Variable collisions - you will probably have to change or rename variables since some of them may overlap;
2. Changes in the deployment process - to be run as a WebJob, your backend will have to be deployed with your web app;
3. Transformations - will probably have to change some transformations so they're also available for the backend.

Deploying your WebJob

A continuous webjob should be deployed with your Azure App Service on App_data/jobs/continuous. Triggered jobs should go into the App_data/jobs/triggered folder. The screenshot below shows them running in my AppService:

Another way to confirm that is by using the Azure Serial Console and cding into that folder:

Changing the Deployment Process

So how do we get our WebJobs on App_data/jobs/continuous? Well, that will obviously depend on how you're deploying your services. The most common deployment strategies are:

1. ClickOnce from Visual Studio
2. Custom PowerShell scripts
3. Using an automated deployment tool (ex. Azure DevOps, CircleCI, AppVeyor, Octopus Deploy, etc)
4. By hand 😢 

Let's discuss the two most common ways: NuGet packages and PowerShell Scripts.

NuGet Packaging

A common way to package code is building NuGet Packages. I won't extend much into that as it's outside of the scope of this post but I want to highlight that getting our project within our NuGet package is very simple. If you're already building NuGet packages, by simply add a reference to your project on the <files> section, we're telling msbuild to package our project with our web application:

PowerShell

If your CI/CD supports PowerShell, we could add the below snippet in a step just before the release:

# PowerShell Copy-Item ..\MyApp.Backend\bin\release App_Data\jobs\continuous\MyApp.Backend -force -recurse

Post-build event

Another alternative would be running a specific command you your post-build event. Just keep in mind that this would also slow your local builds unless if add some contitional around it:

# xcopy xcopy /Q /Y /E /I ..\MyApp.Backend\bin\release App_Data\jobs\continuous\MyApp.Backend

Testing Considerations

With the deployment out of the way, let's what should be considered when testings:

1. Performance - I didn't see any degradation performance changes but that could not be your case. Test and compare the performance of this implementation.
2. Failures - A crashing WebJob won't crash your App Service but, have you tested edge cases?
3. Scale - the number of instances can be different from your current setup. Can you guarantee that no racing conditions exist? 
4. Logging - Do you need to change how your application logs its data? Are the logs centralized and easily accessible?
5. Remoting - Because you departed Windows VMs and Cloud Services doesn't mean that you can't access the instance remotely. The Azure Serial Console is an excellent tool to manage and inspect some aspects of your job.

Production Considerations

Still there? So let's finish this post with some considerations about running NServiceBus on WebJobs in production. I expect you tested your application against the items highlighted on the previous section.

I'd recommend that before going to production that you:

• build some metrics - around the performance before deploying so you know what to expect;
• use Azure Deployment Slots - to validate production before setting it live;
• doubletriple-check your configuration - because it's a new deployment to a new environment and some configurations were changed, weren't they?
• keep an eye on the logs - as we always do, right? 😊 
• do a post-mortem after the deployment - so your team reflects on the pros/cons of this transition.

Final Thoughts

Migrating NServiceBus from VMs to WebJobs was a refreshing and cost-saving experience. Over time, we felt the heavy burden of managing VMs (security patches, firewalling, extra configuration, redundancies, backups, storage, vNets, etc) not to mention how difficult it is to scale them out. Because WebJobs scale out automatically with the App Service at virtually no extra cost, we definitely gained a lot with this change. Some of the positive impacts I saw were:

• quicker deployments
• easier to scale out
• cheaper to run
• more secure
• reduced zero ops
• simpler deployments
• decent performance

If you're starting a new project or have a light backend, I'd recommend you to consider going serverless on Azure Functions or AWS Lambda with SQS.

More about NServiceBus?

Want to read other posts about NServiceBus, please also consider:

• Custom Security on NServiceBus Endpoints
• MassTransit, a real alternative to NServiceBus?
• Async Request/Response with MassTransit, RabbitMQ, Docker and .NET core

References

• NServiceBus Host
• Azure App Services
• Self Hosting in Azure WebJobs
• Azure Storage Emulator
• What is PaaS? Platform as a Service
• What is Azure Application Insights?
• MSBuild - Visual Studio
• Developer Command Prompt for Visual Studio
• Create a NuGet package using MSBuild

See Also

• Microservices in ASP.NET
• My journey to 1 million articles read
• Adding Application Insights telemetry to your ASP.NET Core website
• Async Request/Response with MassTransit, RabbitMQ, Docker and .NET core
• Creating ASP.NET Core websites with Docker
• Deploying Docker images to Azure App Services
• Send emails from ASP.NET Core websites using SendGrid and Azure
• Hosting NuGet packages on GitHub
• Configuration in .NET Core console applications
• How to build and run ASP.NET Core apps on Linux 

How to enable ASP.NET error pages using Azure Serial Console

It's possible to enable ASP.NET error pages on Azure by using the new Azure Serial Console. Let's see how.

By default, ASP.NET web applications running on a remote server set the customErrors property to "RemoteOnly". That means that, unless you're running on the local server, you won't be able to view the original error and the stack trace related it. And that's a good thing! A lot of successful hacks derive from understanding the exception messages and working around them.

But what if you're testing a new server, a new deployment process or just released a new feature and need to enable the error pages very quickly? Well, if you're using Azure, you can use Azure Serial Console to do the job. No SSHing, no RDPing or uploading of configurations to the remote environment. Let's see how.

Azure Serial Console

Today we will use Azure Serial Console. According to Microsoft:

The Serial Console in the Azure portal provides access to a text-based console for virtual machines (VMs) and virtual machine scale set instances running either Linux or Windows. This serial connection connects to the ttyS0 or COM1 serial port of the VM or virtual machine scale set instance, providing access independent of the network or operating system state. The serial console can only be accessed by using the Azure portal and is allowed only for those users who have an access role of Contributor or higher to the VM or virtual machine scale set.

In other words, Azure Serial Console is a nice, simple and accessible tool that can be run from the Azure portal allowing us to interact with our cloud resources including our Azure App Services.

Accessing the console

To access the console for your web application, first we find our Azure App Service in the Portal by clicking on App Services:

Selecting the web site we want to open:

And click on Console on the Development Tools section. You should then see a shell similar to:

Using the Console

Now the fun part. We are ready to interact with our App Service directly from that shell. For starters, let's get some help:

The above screenshot shows some of the administrative commands available on the system. Most of them are standard DOS command prompt utilities that you probably used on your Windows box but never cared to learn. So what can we do?

Linux Tools on Azure Serial Console

Turns out that Redmond is bending to the accessibility, ubiquity and to the power of POSIX / open source tools used and loved by system administrators such as ls, diff, cat, ps, more, less, echo, grep, sed and others. So before jumping to the solution, let's review what we can do with some of these tools.

Example 1: a better dir with ls

Example 2: Creatting and appending content to files using echo, pipes and cat

Example 3: getting disk information with df

Example 4: viewing mounted partitions with mount

Example 5: Displaying differences between files using diff

Example 6: Getting kernel information using uname

Example 7: Even curl and scp is available!

Disabling Custom Errors

Okay, back to our problem. If you know some ASP.NET, you know that the trick is to modify the customErrors Element (ASP.NET Settings Schema) and set the property to   Off   . So let's see how we can change that configuration using a command line tool.

Backing up

Obviously we want to backup our web.config. I hope that's obvious with:

cp web.config web.config.orig

Using sed to replace configuration

Now, we will use sed (a tool available on the GNU operating system that Linux hackers can't live without) to change the setting directly from the console. I'm a sed geek and use it extensively in a Hugo project I've been working on (thousands of markdown files). Together with Go, the i3 window manager, Vim, ranger and grep, my Fedora workstation becomes an ideal development environment. Now, back to .NET...

Testing the Patch

We can safely test if our changes will work by typing:

sed 's/RemoteOnly/Off' web.config

Applying the Patch

Let's jump right to how to replace our customErrors element from   RemoteOnly   to   Off   ? The solution is this simple one-liner script:

sed -i 's/RemoteOnly/Off/' web.config

Switching Back

Now, obviously we may want to switch back. That's why it was important to backup your web.config before. We can switch back by replacing the changed web.config with the original:

rm web.config
mv web.config.orig web.config

Or by running sed again, this time with the parameters inverted:

sed -i 's/Off/RemoteOnly/' web.config

Security Considerations

I hope I don't need to repeat that it's unsafe to leave error pages off on your cloud services. Even if they are simply a playground, there are risks of malicious users pivoting to different services (like your database) and accessing confidential data. Please disable them as soon as possible.

What about Kudu?

Yes, Azure Kudu allows editing files on a remote Azure App Service by using a WISIWYG editor. However, we can't count on that always, everywhere. Remember, with the transition to a microservice-based architecture, more and more our apps will run on serverless and containerized environments meaning tools like that wouldn't be available. So the tip presented on this post will definitely stand the test of time! 😉

Final Thoughts

Wow, that seems a long post for such a small hack but I felt the need to stress certain things here:

1. Developers shouldn't be afraid to use the terminal - I see this pattern especially with Microsoft developers assuming that there should always be a button to do something. The more you use the terminal, the more confident you'll be with the tools you're using regardless of where you are. 
2. Microsoft is moving towards Linux and you should too - The GNU tools prove an unimaginable asset to know. Once you know how to use them better, you'll realize that your toolset grows and you get more creative getting things faster. Plus, the ability to pipe output between them yields unlimited possibilities. Don't know where to start? WSL is the best way to learn the Linux on Windows 10.
3. Be creative, use the best tool for the job - choose wise the tool you use. Very frequently the command line is the fastest (and quickest) way to accomplish most of the automatic workflow. And it can be automated!

Conclusion

The Azure Serial Console can be a powerful tool to help you manage, inspect, debug and run quick commands against your Azure App Service and your Virtual Machines. And combined with the Linux tools it becomes even more powerful!

And you, what's your favorite hack?

References

• Azure Serial Console
• Microsoft Docs | customErrors Element (ASP.NET Settings Schema) 
• Serial Console for Linux VMs
• Serial Console for Windows VMs
• The Basics of Using the Sed Stream Editor to Manipulate Text in Linux

See Also

• Microservices in ASP.NET
• My journey to 1 million articles read
• Adding Application Insights telemetry to your ASP.NET Core website
• Creating ASP.NET Core websites with Docker
• Send emails from ASP.NET Core websites using SendGrid and Azure
• Hosting NuGet packages on GitHub
• Configuration in .NET Core console applications
• How to create a Ubuntu Desktop on Azure
• Building and Running ASP.NET Core apps on Linux
• Windows Subsystem for Linux, the best way to learn Linux on Windows
• How I fell in love with i3
• Why I use Fedora Linux