Showing posts with label MassTransit. Show all posts
Showing posts with label MassTransit. Show all posts

Tuesday, June 1, 2021

Microservices in ASP.NET

Microservices is the last significant change in modern development. Let's learn some tools and related design patterns by building a simplified e-commerce website using modern tools and techniques such as ASP.NET Core and Docker.
Photo by Adi Goldstein on Unsplash

For some time we've been discussing tools and technologies adjacent to microservices on this blog. Not randomly though. Most of these posts derived from my open-source project aspnet-microservices, a simple (yet complicated 😉) distributed application built primarily with .NET Core and Docker. While still work in progress, the project demoes important concepts in distributed architectures.

What's included in the project

This project uses popular tools such as:
On the administrative side, the project also includes:

Disclaimer

When you create a sample microservice-based application, you need to deal with complexity and make tough choices. For the aspnet-microservices application, I deliberately chose to balance complexity and architecture by reducing the emphasis on design patterns focusing on the development of the services themselves. The project was built to serve as an introduction and a start-point for those looking forward to working of Docker, Compose and microservices.

This project is not production-ready! Check Areas for Improvement for more information.

Microservices included in this project

So far, the project consists of the following services:

  • Web: the frontend for our e-commerce application;
  • Catalog: provides catalog information for the web store;
  • Newsletter: accepts user emails and stores them in the newsletter database for future use;
  • Order: provides order features for the web store;
  • Account: provides account services (login, account creation, etc) for the web store;
  • Recommendation: provides simple recommendations based on previous purchases;
  • Notification: sends email notifications upon certain events in the system;
  • Payment: simulates a fake payment store;
  • Shipping: simulates a fake shipping store;

Technologies Used

The technologies used were cherry-picked from the most commonly used by the community. I chose to favour open-source alternatives over proprietary (or commercially-oriented) ones. You'll find in this bundle:
  • ASP.NET Core: as the base of our microservices;
  • Docker and Docker Compose: to build and run containers;
  • MySQL: serving as a relational database for some microservices;
  • MongoDB: serving as the catalog database for the Catalog microservice;
  • Redis: serving as distributed caching store for the Web microservice;
  • RabbitMQ: serving as the queue/communication layer over which our services will communicate;
  • MassTransit: the interface between our apps and RabbitMQ supporting asynchronous communications between them;
  • Dapper: lightweight ORM used to simplify interaction with the MySQL database;
  • SendGrid: used to send emails from our Notification service as described on a previous post;
  • Vue.js and Axios.Js to abstract the frontend of the Web microservice on a simple and powerful  JavaScript framework.

Conventions and Design Considerations

Among others, you'll find in this project that:
  • The Web microservice serves as the frontend for our e-commerce application and implements the API Gateway / BFF design patterns routing the requests from the user to other services on an internal Docker network;
  • Web caches catalog data a Redis data store; Feel free to use Redis Commander to delete cached entries if you wish or need to.
  • Each microservice has its own database isolating its state from external services. MongoDB and MySQL were chosen as the main databases due to their popularity.
  • All services were implemented as ASP.NET Core webapps exposing the endpoints /help and /ping so they can be inspected from and observed automatically the the running engine.
  • No special logging infrastructure was added. Logs can be easily accessed via docker logs or indexed by a different application if you so desire.
  • Microservices communicate between themselves via Pub/Sub and asynchronous request/response using MassTransit and RabbitMQ.
  • The Notification microservice will eventually send emails. This project was tested with SendGrid but other SMTP servers should work from within/without the containers.
  • Monitoring is experimental and includes Grafana sourcing its data from a Prometheus backend.

Technical Requirements

To run this project on your machine, please make sure you have installed:

If you want to develop/extend/modify it, then I'd suggest you to also have:

Running the microservices

So let's get quickly learn how to load and build our own microservices.

Initializing the project

Get your copy by cloning the project:
git clone https://github.com/hd9/aspnet-microservices

Next open the solution src/AspNetContainers.sln with Visual Studio 2019. Since code is always the best documentation, the easiest way to understand the containers and their configurations is by reading the src/docker-compose.yml file.

Debugging with Visual Studio

Building and debugging with Visual Studio 2019 is straightforward. Simply open the AspNetMicroservices.sln solution from the src folder, build and run the project as debug (F5). Next, run the dependencies (Redis, MongoDB, RabbitMQ and MySQL) by issuing the below command from the src folder:

docker-compose -f docker-compose.debug.yml up

Running the services with Docker Compose

In order to run the services you'll need Docker and Docker Compose installed on your machine. Type the command below from the src folder on a terminal to start all services:
docker-compose up
Then to stop them:
docker-compose down
To remove everything, run:
docker-compose down -v
To run a specific service, do:
docker-compose up <service-name>
As soon as you run your services, Compose should start emitting on the console logs for each service:
The output of our docker-compose command

You can also query individual logs for services as usual with docker logs <svc-name>. For example:

~> docker logs src_catalog_1
info: CatalogSvc.Startup[0]
      DB Settings: ConnStr: mongodb://catalog-db:27017, Db: catalog, Collection: products
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://[::]:80
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app

Database Initialization

Database initialization is automatically handled by Compose. Check the docker-compose.yml file to understand how that happens. You'll find examples on how to initialize both MySQL and MongoDB.

Dockerfiles

Each microservice contains a Dockerfile in their respective roots and understanding them should be straightforward. If you never wrote a Dockerfile before, consider reading the official documentation.

Docker Compose

There are two docker-compose files in the solution. Their use is described below:
  • docker-compose.yml: this is the main Compose file. Running this file means you won't be able to access some of the services as they'll not be exposed.
  • docker-compose.debug.yml: this is the file you should run if you want to debug the microservices from Visual Studio. This file only contains the dependencies (Redis, MySQL, RabbitMQ, Mongo + admin interfaces) you'll need to use when debugging.

Accessing our App

If the application booted up correctly, go to http://localhost:8000 to access it. You should see a simple catalog and some other widgets. Go ahead and try to create an account. Just make sure that you have the settings correctly configured on your docker-compose.yml file:
Our simple e-commerce website. As most things, its beauty is in the details 😊.

    Admin Interfaces

    You'll still have available admin interfaces for our services on:
    I won't go over the details about each of these apps. Feel free to explore on your own.

    Monitoring

    Experimental monitoring is available with Grafana, Prometheus and cadvisor. Open Grafana at http://localhost:3000/ and login with admin | admin, select the Docker dashboard and you should see metrics for the services similar to:

    Grafana capturing and emitting telemetry about our microservices.

    Quick Reference

    As a summary, the microservices are configured to run at:

    The management tools are available on:

    And you can access the databases at:
    • MySql databases: use Adminer at: http://localhost:8010/, enter the server name (ex. order-db for the order microservice) and use root | todo as username/password.
    • MongoDB: use MongoExpress at: http://localhost:8011/. No username/password is required.

    Final Thoughts

    On this post I introduce to you my open-source project aspnet-microservices. This application was built as a way to present the foundations of Docker, Compose and microservices for the whole .NET community and hopefully serves as an intuitive guide for those starting in this area.

    Microservices is the last significant change in modern development and requires learning lots (really, lots!) of new technologies and new design patterns. This project is by far complete and should not be used in production as it lacks basic cross-cutting concerns any production-ready project would need. I deliberately omitted them for simplicity else I could simply point you to this project. For more information, check the project's README on GitHub.

    Feel free to play with it and above all, learn and have fun!

    Source Code

    As always, the source code is available on GitHub at: github.com/hd9/spnet-microservices.
    Posted by
    Labels: , , , , , , , , , , , , , , , ,

    Tuesday, December 1, 2020

    Distributed caching in ASP.NET Core using Redis, MongoDB and Docker

    Redis is the world's most popular caching database. Let's review how to implement distributed caching in ASP.NET Core using Redis, MongoDB and Docker Compose.
    Photo by Christian Nielsen on Unsplash

    One of the things that every modern website needs is caching. After all, we don't want to be alerted at 2AM being informed that our services are down because we had a spike in usage which our databases couldn't handle.

    One common solution to reducing the stress in our applications is placing a fast caching service between our website and our database. Modern caching implementations include requirements around decreasing response time, distributed caching (sharing the same cache between multiple web instances) and cost reduction. Most implementations today use Redis (a super-fast an in-memory key–value database) as a cache service sitting in front of a database of choice.

    On this post we will implement a fictional ASP.NET Core e-commerce website using MongoDB as database and Redis as a cache service, both running on Docker with Docker Compose so that we can understand how it all works together.

    On this post we will:
    • Scaffold an ASP.NET Core website
    • Implement a catalog service using MongoDB
    • Implement distributed caching using Redis
    • Run our dependencies using Docker Compose
    • Setup Redis Commander and Mongo Express to view/manage our services

    Setting up an ASP.NET Core website

    Let's quickly scaffold an ASP.NET Core website using the command line with:
    dotnet new mvc -n AspNetDistributedCaching
    Then, add the below configuration to your appsettings.json file:
      "Mongo": {
        "ConnectionString": "mongodb://localhost:27017",
        "Db": "catalog",
        "Collection": "products"
      },
      "Redis": {
        "Configuration": "localhost",
        "InstanceName": "web"
      }
    Next, add the config classes and bind these configs. In case you missed, feel free to review how configurations work in ASP.NET Core 3.1 projects on a previous article.

    Setting up dependencies

    Let's now setup our dependencies: Redis, MongoDB and the management interfaces  Redis Commander and Mongo Express. Despite sounding complicated, it's actually very simple if we use the right tools: Docker and Docker Compose.

    Docker Compose 101

    Without much extension, let me briefly re-introduce Docker Compose:
    Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. See the list of features.
    Setting Compose in your project is actually very simple. For starters, paste this on a docker-compose.yml file on the root of the project. We'll add the services and their respective configurations next:
    version: '3.7'
    services:
      # we'll add our services on the next steps

    Configuring our MongoDB instance

    Next, let's configure MongoDB. Paste the snippet below at the bottom of your docker-compose.yml file. It will instruct Compose to init a Mongo instance called catalog-db and initialize the catalog database from the ./db.js file:
      catalog-db:
        image: mongo
        environment:
          # MONGO_INITDB_ROOT_USERNAME: root
          # MONGO_INITDB_ROOT_PASSWORD: todo
          MONGO_INITDB_DATABASE: catalog
        volumes:
        - .db.js:/docker-entrypoint-initdb.d/db.js:ro
        expose:
          - "27017"     ports:
          - "3301:27017"

    Configuring our Redis instance

    As with MongoDB, let's now setup our Redis cache. Paste this at the bottom of your docker-compose.yml file:
      redis:
        image: redis:6-alpine
        expose:
          - "6379"
        ports:
          - "6379:6379"

    Configuring  the Management interfaces

    Let's now setup management interfaces for Redis - Redis Commander and Mongo - Mongo Express to access our resources (I'll show later how do you use them). Again, paste the below on your docker-compose.yml file:
      # Mongo Express: tool to manage our Mongo database
      mongo-express:
        image: mongo-express
        restart: always
        ports:
          - "8011:8081"
        environment:
          - ME_CONFIG_MONGODB_SERVER=catalog-db
          # MONGO_INITDB_ROOT_USERNAME: root
          # MONGO_INITDB_ROOT_PASSWORD: todo
        depends_on:
          - catalog-db

      # Redis Commander: tool to manage our Redis container from localhost
      redis-commander:
        image: rediscommander/redis-commander:latest
        environment:
          - REDIS_HOSTS=redis
        ports:
          - "8013:8081"
        depends_on:
          - redis

    Querying Catalog Data

    Obviously in order to cache the data, we should have it first. So let's implement a simple MongoDB wrapper using the Repository Pattern which I'll call CatalogRepository. Its interface looks like:
    public interface ICatalogRepository
    {
        Task<IList<Category>> GetCategories();
        Task<Category> GetCategory(string slug);
        Task<Product> GetProduct(string slug);
        Task<IList<Product>> GetProductsByCategory(string slug);
    }
    For the concrete implemtation, don't forget to add the MongoDB.Driver NuGet Package. I show a simple query below. To view it in full, check this demo's repo:
    public async Task<IList<Category>> GetCategories()
    {
        var c = _db.GetCollection<Category>("categories");
        return (await c.FindAsync(new BsonDocument())).ToList();
    }
    Next, let's set up DI for this guy using ASP.NET's DI framework:
    services.AddTransient<ICatalogRepository, CatalogRepository>();
    To view the code in full, check it on this demo's repo at github.com/hd9/aspnet-distributed-caching.

    Caching Catalog Data

    With the repository working, let's implement the caching. I'll divide this task in:
    1. setting up Redis with distributed caching
    2. implementing a Service class and
    3. adding the caching logic to the service class.

    Setting up the Redis Initialization

    First, add NuGet references to Microsoft.Extensions.DependencyInjection. and Microsoft.Extensions.Caching.StackExchangeRedis. Then add the add a call to services.AddStackExchangeRedisCache in ConfigureServices:
    services.AddStackExchangeRedisCache(o =>
    {
        o.Configuration = cfg.Redis.Configuration;
        o.InstanceName = cfg.Redis.InstanceName;
    });

    Implementing a Service Class

    The next part consists of implementing the caching logic. Such logic will live in CatalogService which implements the service design pattern and abstracts from the Controller either the repository and the caching implementations. Its first important part is the constructor which looks like this:
    public CatalogSvc(
        ICatalogRepository repo,
        IDistributedCache cache)
    {
        _repo = repo;
        _cache = cache;
    }

    Adding the caching logic

    Then, every method of the above interface are similarly implemented. For example, the GetCategories method that feeds the landing page looks like this:
    public async Task<IList<Category>> GetCategories()
    {
        return await GetFromCache<IList<Category>>(
            "categories",
            "*",
            async () => await _repo.GetCategories());
    }
    And our GetFromCache private method that generalizes caching logic for our catalog service looks like:
    private async Task<TResult> GetFromCache<TResult>(
        string key,
        string val,
        Func<Task<object>> func)
    {
        var cacheKey = string.Format(_keyFmt, key, val);
        var data = await _cache.GetStringAsync(cacheKey);

        if (string.IsNullOrEmpty(data))
        {
            data = JsonConvert.SerializeObject(await func());

            await _cache.SetStringAsync(
                cacheKey,
                data);
        }

        return JsonConvert.
            DeserializeObject<TResult>(data);
    }
    The interesting bits of the above piece of code is that before searching the database (which's abstracted by a Func<> parameter), it searches the cache with GetStringAsync. If it finds, it returns the cached data cast as the provided type (TResult) by deserializing it from its string value. In case the cache key is not present, it will invoke the target function and cache its result as a Json string in the Redis cache.

    To use it, first we wire it up to the DI container:
    services.AddTransient<ICatalogSvc, CatalogSvc>();
    So we can properly inject it and use it in our controllers:
    public HomeController(ICatalogSvc svc)
    {
        _svc = svc;
    }

    Management Interfaces

    To finish up, let's quickly review how to access the management interfaces for Redis and Mongo Express.

    Accessing Mongo Express

    To view/modify your catalog, you can access Mongo Express at: http://localhost:8011/. I changed from the original port 8081 to 8011 since many services run on that port and if it was your case, Compose would fail. But feel free to change that configuration on your docker-compose.yml file. As previously mentioned, this database is auto-initialized from the db.js file by Docker Compose. Here's a quick glance of Mongo Express displaying our catalog data:

    Accessing Redis Commander

    Redis Commander is a frontend for vieweing and managing Redis. On this demo I run it on http://localhost:8011/. As previously, feel free to change the port on your docker-compose.yml file. Here's Redis Commander showing our cached data:

    Running the Services

    The last part is to describe how to run the services. As a .NET developer, you're already used to debug and run your solutions with Visual Studio - the same applies here. The only thing that remains is how to run the dependencies? As mentioned, it's as simple as running the below command from the project's root with Docker Compose:
    docker-compose up
    You should see the services starting in the backend similar to this:
    To shutdown, run:
    docker-compose down
    Finally, to remove everything, run:
    docker-compose down -v

    Final Thoughts

    On this article we reviewed how to use Redis, a super-fast key-value document database in front of MongoDB database serving as a distributed service. On this example we still leveraged Docker and Docker Compose to simplify the setup and the initialization of our project so we could get our application running and test it as quickly as possible.

    Redis is one of the world's most used and loved databases and a very common option for caching. I hope you also realized how Docker and Docker Compose help developers by simplifying rebuilding complex environments like this.

    Source Code

    As always, the source code for this article is available on my GitHub.

    References

    Posted by
    Labels: , , , , , , , , , , , , , ,

    Monday, November 2, 2020

    Async Request/Response with MassTransit, RabbitMQ, Docker and .NET core

    Let's review how to implement an async resquest/response exchange between two ASP.NET Core websites via RabbitMQ queues using MassTransit
    Photo by Pavan Trikutam on Unsplash

    Undoubtedly the most popular design pattern when writing distributed application is Pub/Sub. Turns out that there's another important design pattern used in distributed applications not as frequently mentioned, that can also be implemented with queues: async requests/responses. Async requests/responses are very useful and widely used to exchange data between microservices in non-blocking calls, allowing the requested service to throttle incoming requests via a queue preventing its own exhaustion.

    On this tutorial, we'll implement an async request/response exchange between two ASP.NET Core websites via RabbitMQ queues using MassTransit. We'll also wire everything up using Docker and Docker Compose.

    On this post we will:
    • Scaffold two ASP.NET Core websites
    • Configure each website to use MassTransit to communicate via a local RabbitMQ queue
    • Explain how to write the async request/response logic
    • Run a RabbitMQ container using Docker
    • Test and validate the results

    Understanding MassTransit Async Requests

    If you understand how to wire everything up, setting up async request/response with MassTransit is actually very simple. So before getting our hands into the code, let's review the terminology you'll need to know:
    • Consumer: a class in your service that'll respond for requests (over a queue on this case);
    • IRequestClient<T>: the interface we'll have to implement to implement the client and invoke async requests via the queue;
    • ReceiveEndpoint: a configuration that we'll have to setup to enable our Consumer to listen and respond to requests;
    • AddRequestClient: a configuration that we'll have to setup to allow our own async request implementation;
    Keep that info in mind as we'll use them in the following sections.

    Creating our Project

    Let's quickly scaffold two ASP.NET Core projects by using the dotnet CLI with:
    dotnet new mvc -o RequestSvc
    dotnet new mvc -o ResponseSvc

    Adding the Dependencies

    The dependencies we'll need today are:

    Adding Configuration

    The configuration we'll need  is also straightforward. Paste this in your RequestSvc/appsettings.json:
    "MassTransit": {
        "Host": "rabbitmq://localhost",
        "Queue": "requestsvc"
    }
    And this in your ResponseSvc/appsettings.json:
    "MassTransit": {
        "Host": "rabbitmq://localhost",
        "Queue": "responsesvc"
    }
    Next, bind the config classes to those settings. Since I covered in detail how configurations work in ASP.NET Core 3.1 projects on a previous article I'll skip that to keep this post short. But if you need, feel free to take a break and understand that part first before you proceed.

    Adding Startup Code

    Wiring up MassTransit in ASP.NET DI framework is also well documented. For our solution it would look like this for the RequestSvc project:
    services.AddMassTransit(x =>
    {
        x.AddBus(context => Bus.Factory.CreateUsingRabbitMq(c =>
        {
            c.Host(cfg.MassTransit.Host);
            c.ConfigureEndpoints(context);
        }));
       
        x.AddRequestClient<ProductInfoRequest>();
    });

    services.AddMassTransitHostedService();
    And like this for the  ResponseSvc project:
    services.AddMassTransit(x =>
    {
        x.AddConsumer<ProductInfoRequestConsumer>();

        x.AddBus(context => Bus.Factory.CreateUsingRabbitMq(c =>
        {
            c.Host(cfg.MassTransit.Host);
            c.ReceiveEndpoint(cfg.MassTransit.Queue, e =>
            {
                e.PrefetchCount = 16;
                e.UseMessageRetry(r => r.Interval(2, 3000));
                e.ConfigureConsumer<ProductInfoRequestConsumer>(context);
            });
        }));
    });

    services.AddMassTransitHostedService();
    Stop for a second and compare the differences between both initializations. Spot the differences?

    Building our Consumer

    Before we can issue our requests, we have to build a consumer to handle these messages. In MassTransit's world, this is the same consumer you'd build for your regular pub/sub. For this demo, our ProductInfoRequestConsumer looks like this:
    public async Task Consume(ConsumeContext<ProductInfoRequest> context)
    {
        var msg = context.Message;
        var slug = msg.Slug;

        // a fake delay
        var delay = 1000 * (msg.Delay > 0 ? msg.Delay : 1);
        await Task.Delay(delay);

        // get the product from ProductService
        var p = _svc.GetProductBySlug(slug);

        // this responds via the queue to our client
        await context.RespondAsync(new ProductInfoResponse
        {
            Product = p
        });
    }

    Async requests

    With consumer, configuration and the startup logic in place, it's time to write the request code. In essence, this is the piece of code that will mediate the async communication between the caller and the responder using a queue (abstracted obviously by MassTransit). A simple async request to a remote service using a backend queue looks like:
    using (var request = _client.Create(new ProductInfoRequest { Slug = slug, Delay = timeout }))
    {
        var response = await request.GetResponse<ProductInfoResponse>();
        p = response.Message.Product;
    }

    Running the dependencies

    To run RabbitMQ, we'll use Docker Compose. Running RabbitMQ with Compose is as simple as running the below command from the src folder:
    docker-compose up
    If everything correctly initialized, you should expect to see RabbitMQ's logs emitted by Docker Compose on the terminal:
    To shutdown Compose and RabbitMQ, either click Ctrl-C or run:
    docker-compose down
    Finally, to remove everything, run:
    docker-compose down -v

    Testing the Application

    Open the project from Visual Studio 2019, and run it as debug (F5) and VS will open 2 windows - one for RequestSvc and another for ResponseSvc. RequestSvc looks like this:

    Go ahead and run some queries. If you got your debugger running, it will stop in both services allowing you to validate the exchange between them. To reduce Razor boilerplate the project uses VueJS and AxiosJs so we get responses in the UI without unnecessary roundtrips.

    RabbitMQ's Management Interface

    The last thing worth mentioning is how to get to RabbitMQ's management interface. This project also allows you to play with RabbitMQ at http://localhost:8012. By logging in with guest | guest and clicking on the Queues tab you should see something similar to:
    RabbitMQ is a powerful message-broker service. However, if you're running your applications on the cloud, I'd suggest using a fully-managed service such as Azure Service Bus since it increases the resilience of your services.

    Final Thoughts

    On this article we reviewed how to implement an asynchronous request/response using queues. Async resquests/responses are very useful and widely used to exchange data between microservices in non-blocking calls, allowing the resqueted service to throttle incoming requests via a queue preventing its own exhaustion. On this example we still leveraged Docker and Docker Compose to simplify the setup and the initialization of our backend services.

    I hope you liked the demo and will consider using this pattern in your applications.

    Source Code

    As always, the source code for this article is available on my GitHub.

    References

    See Also

    Posted by
    Labels: , , , , , , , , , , , , , , , , , , ,

    Wednesday, July 15, 2020

    Hosting NuGet packages on GitHub

    On this post let's review how to build, host and consume our own NuGet packages using GitHub Packages
    Photo by Leone Venter on Unsplash

    Long gone are the days we had to pay to host our NuGet packages. Today, things have changed. We have many options to host our own NuGet packages for free (including privately if we wish) including in our own GitHub repositories. On this tutorial let's review how to build our own packages using .NET Core's CLI, push them to GitHub and finally, how to consume from our own projects.

    About NuGet

    NuGet is a free and open-source package manager designed by Microsoft and used extensively in the .NET /.NET Core ecosystem. NuGet is the name of the tool and of the package itself. The most common repository for NuGet packages is NuGet.org hosting more than 200k packages! But we can host our own packages on different repos (including private ones) such as GitHub Packages. NuGet is bundled with Visual Studio and with the .NET Core SDK so you probably have it already available on your machine.

    About GitHub Packages

    GitHub Packages is GitHub's free offering for those wanting to host their own packages. GitHub Packages allows hosting public and private packages. The benefits of using GitHub Packages is that it's free, you can share your packages privately or with the rest of the world, integrate with GitHub APIs, GitHub Actions, webhooks and even create complex end-to-end DevOps workflows. For more information about GitHub Packages, click here.

    Why build our own packages

    But why build our own packages? Mainly because packages simplify using and distributing self-contained and reusable software (tools, libraries, etc) in a clean and organized way of doing so. Beyond that, other common reasons are:
    1. sharing packages with someone else (and possibly the world)
    2. sharing that package privately with your coworkers so they can be used in different projects.
    3. packaging software so it can be installed or deployed elsewhere.

    Building NuGet Packages

    So let's get started and build our first NuGet package. The project we'll build is a simple library consisting of POCOs I frequently use as standard onfiguration bindings when developing microservices: Smtp, Redis, RabbitMQ, MassTransit and MongoDB. I chose this example because this is the type of code we frequently duplicate, so why not isolate them in a shareable package and keep our codebase DRY?

    Creating our project

    To quickly create my project let's use the .NET Core CLI (feel free to use Visual Studio if you will):
    dotnet new classlib -o HildenCo.Core
    Then I'll add those config classes. For example the SmtpOptions looks like:
    public class SmtpOptions
    {
        public string Host { get; set; }
        public int  Port { get; set; }
        public string Username { get; set; }
        public string Password { get; set; }
        public string FromName { get; set; }
        public string FromEmail { get; set; }
    }

    Creating our first NuGet package

    Let's then create our first package. The simplest way to do so is by configuring it via Visual Studio. For that, select the Project and Alt-Enter it (or right-click it with the mouse) to view Project Properties and check Generate NuGet package on build on the Package tab:
    Don't forget to add relevant information about your package such as Id, Name, Version, Authors, Description, Copyright, License and RepositoryUrl. All that information is required by GitHub:
    If you prefer, you can edit the above metadata directly in the csproj file.
    Now, build again to confirm our package was built by inspecting the Build Output in VS (Ctrl-W, O):
    1>------ Build started: Project: HildenCo.Core, Configuration: Debug Any CPU ------
    1>HildenCo.Core -> C:\src\nuget-pkg-demo\src\HildenCo.Core\bin\Debug\netstandard2.0\HildenCo.Core.dll
    1>Successfully created package 'C:\src\nuget-pkg-demo\src\HildenCo.Core\bin\Debug\HildenCo.Core.0.0.1.nupkg'.
    ========== Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========
    Congrats! You now have built your first package!
    Don't forget to add RepositoryUrl with your correct username/repo name. We'll need it to push to GitHub later.

    Creating our package using the CLI

    As always, the CLI may be a better alternative. Why? In summary because it allows automating package creation on continuous integration, integrating with APIs, webhooks and even creating end-to-end DevOps workflows. So, go ahead and uncheck that box and build it again with:
    dotnet pack --configuration Release
    This time, we should see this as output:
    Microsoft (R) Build Engine version 16.6.0+5ff7b0c9e for .NET Core
    Copyright (C) Microsoft Corporation. All rights reserved.

      Determining projects to restore...
      All projects are up-to-date for restore.
      HildenCo.Core -> C:\src\nuget-pkg-demo\src\HildenCo.Core\bin\Release\netstandard2.0\HildenCo.Core.dll
      Successfully created package 'C:\src\nuget-pkg-demo\src\HildenCo.Core\bin\Release\HildenCo.Core.0.0.1.nupkg'.
    TIP: You may have realized that we now built our package as release. This is another immediate benefit from decoupling our builds from VS. On rare occasions should we push packages built as Debug.

    Pushing packages to GitHub

    With the basics behind, let's review how to push your own packages to GitHub.

    Generating an API Key

    In order to authenticate to GitHub Packages the first thing we'll need is an access token. Open your GitHub account, go to Settings -> Developer Settings -> Personal access tokens, click Generate new Token, give it a name, select write:packages and save:

    Creating a nuget.config file

    With the API key created, let's create our nuget.config file. This file should contain the authentication for the package to be pushed to the remote repo. A base config is listed below with the fields to be replaced in bold:
    <?xml version="1.0" encoding="utf-8"?>
    <configuration>
        <packageSources>
            <clear />
            <add key="github" value="https://nuget.pkg.github.com/<your-github-username>/index.json" />
        </packageSources>
        <packageSourceCredentials>
            <github>
                <add key="Username" value="<your-github-username>" />
                <add key="ClearTextPassword" value="<your-api-key>" />
            </github>
        </packageSourceCredentials>
    </configuration>

    Pushing a package to GitHub

    With the correct configuration in place, we can push our package to GitHub with:
    dotnet nuget push ./bin/Release/HildenCo.Core.0.0.1.nupkg --source "github"
    This is what happened when I pushed mine:
    dotnet nuget push ./bin/Release/HildenCo.Core.0.0.1.nupkg --source "github"
    Pushing HildenCo.Core.0.0.1.nupkg to 'https://nuget.pkg.github.com/hd9'...
      PUT https://nuget.pkg.github.com/hd9/
      OK https://nuget.pkg.github.com/hd9/ 1927ms
    Your package was pushed.
    Didn't work? Check if you added RepositoryUrl to your project's metadata as nuget uses it  need it to push to GitHub.

    Reviewing our Package on GitHub

    If you managed to push your first package (yay!), go ahead and review it in GitHub on the Package tab of your repository. For example, mine's available at: github.com/hd9/nuget-pkg-demo/packages and looks like this:

    Using our Package

    To complete the demo let's create an ASP.NET project to use our own package:
    dotnet new mvc -o TestNugetPkg
    To add a reference to your package, we'll use our own nuget.config since it contains pointers to our own repo. If your project has a solution, copy the nuget.config to the solution folder. Else, leave it in the project's folder. Open your project with Visual Studio and open the Manage NuGet Packages. You should see your newly created package there:
    Select it and install:
    Review the logs to make sure no errors happened:
    Restoring packages for C:\src\TestNugetPkg\TestNugetPkg.csproj...
      GET https://nuget.pkg.github.com/hd9/download/hildenco.core/index.json
      OK https://nuget.pkg.github.com/hd9/download/hildenco.core/index.json 864ms
      GET https://nuget.pkg.github.com/hd9/download/hildenco.core/0.0.1/hildenco.core.0.0.1.nupkg
      OK https://nuget.pkg.github.com/hd9/download/hildenco.core/0.0.1/hildenco.core.0.0.1.nupkg 517ms
    Installing HildenCo.Core 0.0.1.
    Installing NuGet package HildenCo.Core 0.0.1.
    Committing restore...
    Writing assets file to disk. Path: C:\src\TestNugetPkg\obj\project.assets.json
    Successfully installed 'HildenCo.Core 0.0.1' to TestNugetPkg
    Executing nuget actions took 960 ms
    Time Elapsed: 00:00:02.6332352
    ========== Finished ==========

    Time Elapsed: 00:00:00.0141177
    ========== Finished ==========
    And finally we can use it from our second project and harvest the benefits of clean code and code reuse:

    Final Thoughts

    On this post we reviewed how to build our own NuGet packages using .NET Core's CLI, pushed them to GitHub and finally described how to consume them from our own .NET projects. Creating and hosting our own NuGet packages is important for multiple reasons including sharing code between projects and creating deployable artifacts.

    Source Code

    As always, the source code for this post is available on GitHub.

    See Also

    Posted by
    Labels: , , , , , , , , , , , , , , , , , ,

    Monday, February 17, 2020

    Running NServiceBus on Azure WebJobs

    On this post we will learn how to build and deploy NServiceBus on Azure WebJobs.
    Photo by True Agency on Unsplash

    Most of us aren't there yet with microservices. But that doesn't mean we shouldn't upgrade our infrastructure to reduce costs, increase performance, enhance security, simplify deployment, and scaling using simpler/newer technologies. On this article let's review how to deploy NServiceBus on Azure WebJobs and discuss:

    • why use WebJob
    • how to upgrade your NSB endpoints as Azure WebJob;
    • the refactorings you'll need
    • how to build, debug, test and deploy our WebJob;
    • production considerations
    If you're starting a new project or have a light backend, I'd recommend you to consider going serverless on Azure Functions or AWS Lambda with SQS.

    Introduction

    On Azure, NServiceBus is usually deployed on Cloud Services or Windows Services. The major problem with both is that, by being Platform on a Service (PAAS) they are difficult to update, secure, scale out and have to be managed separately.

    Depending on your usage, migrating your NSB backend to Azure WebJobs could be a good alternative to reduce costs, maintenance and increase security. Plus, since webjobs are scaled out automatically with your Azure App Services, you would literally get autoscaling on your NSB backend for free!

    Azure WebJobs

    So let's start by quikcly recapping what are webjobs. According to Microsoft, webjobs are
    a feature of Azure App Service that enables you to run a program or script in the same context as a web app, API app, or mobile app. There is no additional cost to use WebJobs.
    In other words, a WebJob is nothing more than a script or an application run by Azure. Currently supported formats are:
    • .cmd, .bat, .exe (using Windows cmd)
    • .ps1 (using PowerShell)
    • .sh (using Bash)
    • .php (using PHP)
    • .py (using Python)
    • .js (using Node.js)
    • .jar (using Java)
    As of the creation of this post, Azure still didnt' support WebJobs on App Service on Linux.

    Types of WebJobs

    WebJobs can be triggered and continuous. The differences are:
    • Continuous WebJobs: start immediately, can be run in parallel or be restricted to a single instance.
    • Triggered WebJobs: can be triggered manually or on a schedule and run on a single instance selected by Azure.
    Since backend services like NServiceBus and MassTransit traditionally run continuously on the background, this post will focus on continuous WebJobs.

    Benefits of running NServiceBus on WebJobs

    So what are the benefits of transitioning our NServiceBus hosts to WebJobs? In summary, this appoach will:
    • reduce your costs as no VMs or Cloud Services are required
    • allow your backend to scale up automatically with your Azure App Service
    • eliminates your concerns about maintenance/upgrade/patch and security
    • is way simpler to deploy
    • differently than NServiceBus Host, is not being deprecated
    So let's review how it works.

    Migrating NServiceBus backends to WebJobs

    Migrating NServiceBus backend to WebJobs couldn't be simpler. Since, NSB's official documentation does not clearly describes a migration process, let's address it here. Essentially you'll have to:
    • transform your host project in a console application
    • add a startup class and refactor the endpoint initialization
    • add a reference Microsoft.Azure.WebJobs so you can use the WebJob Api (optional)

    Transforming our Host in a Console Application

    The first part of our exercise requires converting our NServiceBus endpoint to a WebJob. Since WebJobs are essentially executable files we can start by simply transforming our endpoint project from a Class Library to a Console Aplication:

    Referencing the WebJob package

    As recommended, to leverage the Azure Api we'll have to add the Microsoft.Azure.WebJobs NuGet package to our solution. After that package is added, we'll also have to refactor our Main method to correctly initialize and shutdown our bus.

    Adding a Startup class

    Next, we have to add a startup class to our project. Essentially the compiler just needs a static void Main() method inside our solution so the project can be initialized. This is a simple example:
    From here, not much will change. In summary, you will have to:
    • Remove the NServiceBus.Host pakage from the solution
    • Remove IWantToRunWhenEndpointStartsAndStops as it's no longer necessary
    • Refactor some of your settings because your deployment will likely change.
    • Optionally, add some sort of centralized logging like Application Insights since your backend will run in multiple instances. And having a consolidated logging infrastructure wouldn't hurt.

    Potential Problems

    If you updated to the 3.x series of the Microsoft.Azure.WebJobs NuGet package, you probably realized that Microsoft aligned .NET Core and the .NET Framework on this release, already preparing for .NET 5. While that's excellent news, I you may also have conflicting dependencies and build errors as the one listed below.
    error CS0012: The type 'Object' is defined in an assembly that is not referenced. You must add a reference to assembly 'netstandard, Version=2.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.
    That error can be fixed by:
    1. Referencing the NETStandard.Libray NuGet package on your WebJob;
    2. Adding <Reference Include="netstandard" /> just below the <ItemGroup> section on your WebJob's csproj file.

      Building, testing and debugging the solution

      After upgrading packages, refactoring code and fixing dependencies issues, we'll now have to fix potential build errors, and assert that the unit-tests are passing for the solution solution. Since I don't expect any major issues here, guess we can move ahead and review necessary changes for debugging.

      Debugging

      Because we transformed our NServiceBus endpoint in a console app, we remain able to start it on debug as previously. However there are some important details ahead. Make sure you set your WebJob project to start when debugging. To do so, we have to configure our solution to start multiple projects at the same time by right-clicking your solution, clicking Startup scripts, selecting Multiple startup projects and setting the Action column to Start for the projects you want to start.
        Now set your Azure Storage connection string so you can debug your project with Azure.

        Running Locally

        But if you want to run the WebJob using the development connection string ("UseDevelopmentStorage=true"), you will realize that the initialization fails. WebJobs can't run with the Azure Storage Emulator: 
        System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException:
   Failed to validate Microsoft Azure WebJobs SDK Storage account.
   The Microsoft Azure Storage Emulator is not supported, please use a Microsoft Azure Storage account 
hosted in Microsoft Azure.
   at Microsoft.Azure.WebJobs.Host.Executors.StorageAccountParser.ParseAccount(String connectionString,
String connectionStringName, IServiceProvider services)
   at Microsoft.Azure.WebJobs.Host.Executors.DefaultStorageAccountProvider.set_StorageConnectionString(String value)
   at Microsoft.Azure.WebJobs.JobHostConfiguration.set_StorageConnectionString(String value)
   at ATIS.Services.Program.d__1.MoveNext() in C:\src\ATIS\ATIS.Services\Program.cs:line 49
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
        So if we have a console application ready to run, why do even have to start the Jobhost at all?

        That's a common scenario in which we want to run a different logic on debug and release modes, I usually resort to preprocessor directives where I have different implementations for debug and release. The next snippet shows it on line 15:

        Going Async

        To finish, let's make code asynchronous. The two previous snippets could be refactored into  something like:

        Deployment

        Now let's discuss deployment. Three important things to note:
        1. Variable collisions - you will probably have to change or rename variables since some of them may overlap;
        2. Changes in the deployment process - to be run as a WebJob, your backend will have to be deployed with your web app;
        3. Transformations - will probably have to change some transformations so they're also available for the backend.

        Deploying your WebJob

        A continuous webjob should be deployed with your Azure App Service on App_data/jobs/continuous. Triggered jobs should go into the App_data/jobs/triggered folder. The screenshot below shows them running in my AppService:
        Another way to confirm that is by using the Azure Serial Console and cding into that folder:

          Changing the Deployment Process

          So how do we get our WebJobs on App_data/jobs/continuous? Well, that will obviously depend on how you're deploying your services. The most common deployment strategies are:
          1. ClickOnce from Visual Studio
          2. Custom PowerShell scripts
          3. Using an automated deployment tool (ex. Azure DevOps, CircleCI, AppVeyor, Octopus Deploy, etc)
          4. By hand 😢 
          Let's discuss the two most common ways: NuGet packages and PowerShell Scripts.

          NuGet Packaging

          A common way to package code is building NuGet Packages. I won't extend much into that as it's outside of the scope of this post but I want to highlight that getting our project within our NuGet package is very simple. If you're already building NuGet packages, by simply add a reference to your project on the <files> section, we're telling msbuild to package our project with our web application:

          PowerShell

          If your CI/CD supports PowerShell, we could add the below snippet in a step just before the release:
          # PowerShell Copy-Item ..\MyApp.Backend\bin\release App_Data\jobs\continuous\MyApp.Backend -force -recurse

          Post-build event

          Another alternative would be running a specific command you your post-build event. Just keep in mind that this would also slow your local builds unless if add some contitional around it:
          # xcopy xcopy /Q /Y /E /I ..\MyApp.Backend\bin\release App_Data\jobs\continuous\MyApp.Backend

          Testing Considerations

          With the deployment out of the way, let's what should be considered when testings:
          1. Performance - I didn't see any degradation performance changes but that could not be your case. Test and compare the performance of this implementation.
          2. Failures - A crashing WebJob won't crash your App Service but, have you tested edge cases?
          3. Scale - the number of instances can be different from your current setup. Can you guarantee that no racing conditions exist? 
          4. Logging - Do you need to change how your application logs its data? Are the logs centralized and easily accessible?
          5. Remoting - Because you departed Windows VMs and Cloud Services doesn't mean that you can't access the instance remotely. The Azure Serial Console is an excellent tool to manage and inspect some aspects of your job.

            Production Considerations

            Still there? So let's finish this post with some considerations about running NServiceBus on WebJobs in production. I expect you tested your application against the items highlighted on the previous section.

            I'd recommend that before going to production that you:
            • build some metrics - around the performance before deploying so you know what to expect;
            • use Azure Deployment Slots - to validate production before setting it live;
            • doubletriple-check your configuration - because it's a new deployment to a new environment and some configurations were changed, weren't they?
            • keep an eye on the logs - as we always do, right? 😊 
            • do a post-mortem after the deployment - so your team reflects on the pros/cons of this transition.

            Final Thoughts

            Migrating NServiceBus from VMs to WebJobs was a refreshing and cost-saving experience. Over time, we felt the heavy burden of managing VMs (security patches, firewalling, extra configuration, redundancies, backups, storage, vNets, etc) not to mention how difficult it is to scale them out. Because WebJobs scale out automatically with the App Service at virtually no extra cost, we definitely gained a lot with this change. Some of the positive impacts I saw were:
            • quicker deployments
            • easier to scale out
            • cheaper to run
            • more secure
            • reduced zero ops
            • simpler deployments
            • decent performance
            If you're starting a new project or have a light backend, I'd recommend you to consider going serverless on Azure Functions or AWS Lambda with SQS.

            More about NServiceBus?

            Want to read other posts about NServiceBus, please also consider:

            References

            See Also

              Posted by
              Labels: , , , , , , , , , , , , , , , ,

              About the Author

              Bruno Hildenbrand      
              Principal Architect, HildenCo Solutions.