Showing posts with label NServiceBus. Show all posts
Showing posts with label NServiceBus. Show all posts

Monday, November 2, 2020

Async Request/Response with MassTransit, RabbitMQ, Docker and .NET core

Let's review how to implement an async resquest/response exchange between two ASP.NET Core websites via RabbitMQ queues using MassTransit
Photo by Pavan Trikutam on Unsplash

Undoubtedly the most popular design pattern when writing distributed application is Pub/Sub. Turns out that there's another important design pattern used in distributed applications not as frequently mentioned, that can also be implemented with queues: async requests/responses. Async requests/responses are very useful and widely used to exchange data between microservices in non-blocking calls, allowing the requested service to throttle incoming requests via a queue preventing its own exhaustion.

On this tutorial, we'll implement an async request/response exchange between two ASP.NET Core websites via RabbitMQ queues using MassTransit. We'll also wire everything up using Docker and Docker Compose.

On this post we will:
  • Scaffold two ASP.NET Core websites
  • Configure each website to use MassTransit to communicate via a local RabbitMQ queue
  • Explain how to write the async request/response logic
  • Run a RabbitMQ container using Docker
  • Test and validate the results

Understanding MassTransit Async Requests

If you understand how to wire everything up, setting up async request/response with MassTransit is actually very simple. So before getting our hands into the code, let's review the terminology you'll need to know:
  • Consumer: a class in your service that'll respond for requests (over a queue on this case);
  • IRequestClient<T>: the interface we'll have to implement to implement the client and invoke async requests via the queue;
  • ReceiveEndpoint: a configuration that we'll have to setup to enable our Consumer to listen and respond to requests;
  • AddRequestClient: a configuration that we'll have to setup to allow our own async request implementation;
Keep that info in mind as we'll use them in the following sections.

Creating our Project

Let's quickly scaffold two ASP.NET Core projects by using the dotnet CLI with:
dotnet new mvc -o RequestSvc
dotnet new mvc -o ResponseSvc

Adding the Dependencies

The dependencies we'll need today are:

Adding Configuration

The configuration we'll need  is also straightforward. Paste this in your RequestSvc/appsettings.json:
"MassTransit": {
    "Host": "rabbitmq://localhost",
    "Queue": "requestsvc"
}
And this in your ResponseSvc/appsettings.json:
"MassTransit": {
    "Host": "rabbitmq://localhost",
    "Queue": "responsesvc"
}
Next, bind the config classes to those settings. Since I covered in detail how configurations work in ASP.NET Core 3.1 projects on a previous article I'll skip that to keep this post short. But if you need, feel free to take a break and understand that part first before you proceed.

Adding Startup Code

Wiring up MassTransit in ASP.NET DI framework is also well documented. For our solution it would look like this for the RequestSvc project:
services.AddMassTransit(x =>
{
    x.AddBus(context => Bus.Factory.CreateUsingRabbitMq(c =>
    {
        c.Host(cfg.MassTransit.Host);
        c.ConfigureEndpoints(context);
    }));
   
    x.AddRequestClient<ProductInfoRequest>();
});

services.AddMassTransitHostedService();
And like this for the  ResponseSvc project:
services.AddMassTransit(x =>
{
    x.AddConsumer<ProductInfoRequestConsumer>();

    x.AddBus(context => Bus.Factory.CreateUsingRabbitMq(c =>
    {
        c.Host(cfg.MassTransit.Host);
        c.ReceiveEndpoint(cfg.MassTransit.Queue, e =>
        {
            e.PrefetchCount = 16;
            e.UseMessageRetry(r => r.Interval(2, 3000));
            e.ConfigureConsumer<ProductInfoRequestConsumer>(context);
        });
    }));
});

services.AddMassTransitHostedService();
Stop for a second and compare the differences between both initializations. Spot the differences?

Building our Consumer

Before we can issue our requests, we have to build a consumer to handle these messages. In MassTransit's world, this is the same consumer you'd build for your regular pub/sub. For this demo, our ProductInfoRequestConsumer looks like this:
public async Task Consume(ConsumeContext<ProductInfoRequest> context)
{
    var msg = context.Message;
    var slug = msg.Slug;

    // a fake delay
    var delay = 1000 * (msg.Delay > 0 ? msg.Delay : 1);
    await Task.Delay(delay);

    // get the product from ProductService
    var p = _svc.GetProductBySlug(slug);

    // this responds via the queue to our client
    await context.RespondAsync(new ProductInfoResponse
    {
        Product = p
    });
}

Async requests

With consumer, configuration and the startup logic in place, it's time to write the request code. In essence, this is the piece of code that will mediate the async communication between the caller and the responder using a queue (abstracted obviously by MassTransit). A simple async request to a remote service using a backend queue looks like:
using (var request = _client.Create(new ProductInfoRequest { Slug = slug, Delay = timeout }))
{
    var response = await request.GetResponse<ProductInfoResponse>();
    p = response.Message.Product;
}

Running the dependencies

To run RabbitMQ, we'll use Docker Compose. Running RabbitMQ with Compose is as simple as running the below command from the src folder:
docker-compose up
If everything correctly initialized, you should expect to see RabbitMQ's logs emitted by Docker Compose on the terminal:
To shutdown Compose and RabbitMQ, either click Ctrl-C or run:
docker-compose down
Finally, to remove everything, run:
docker-compose down -v

Testing the Application

Open the project from Visual Studio 2019, and run it as debug (F5) and VS will open 2 windows - one for RequestSvc and another for ResponseSvc. RequestSvc looks like this:

Go ahead and run some queries. If you got your debugger running, it will stop in both services allowing you to validate the exchange between them. To reduce Razor boilerplate the project uses VueJS and AxiosJs so we get responses in the UI without unnecessary roundtrips.

RabbitMQ's Management Interface

The last thing worth mentioning is how to get to RabbitMQ's management interface. This project also allows you to play with RabbitMQ at http://localhost:8012. By logging in with guest | guest and clicking on the Queues tab you should see something similar to:
RabbitMQ is a powerful message-broker service. However, if you're running your applications on the cloud, I'd suggest using a fully-managed service such as Azure Service Bus since it increases the resilience of your services.

Final Thoughts

On this article we reviewed how to implement an asynchronous request/response using queues. Async resquests/responses are very useful and widely used to exchange data between microservices in non-blocking calls, allowing the resqueted service to throttle incoming requests via a queue preventing its own exhaustion. On this example we still leveraged Docker and Docker Compose to simplify the setup and the initialization of our backend services.

I hope you liked the demo and will consider using this pattern in your applications.

Source Code

As always, the source code for this article is available on my GitHub.

References

See Also

Monday, August 20, 2018

Exploring MassTransit InMemory Scheduled Messaging using RabbitMQ and .NET Core

On this post, let's explore MassTransit's scheduler messaging system using RabbitMQ

On a previous post, I demoed how to create a MassTransit client/server application using RabbitMQ, .NET Core and Linux. Today we will explore another very interesting functionality: the ability to schedule messages to send them in the future. My experiences with MassTransit so far have been fantastic, however, there are a few use cases that I still would like to test. On this post, we cover the scheduled message use case, testing the in-memory solution.

Persistence Requirements

In order to keep our data persisted for the scheduler to use we'll need to configure MassTransit's storage with one of the following services:
  • Quartz.Net in a hosted server
  • Azure Service Bus
  • RabbitMQ by installing a plugin
  • a test-driven In-Memory implementation
On this post, we'll spike out the in-memory solution due to its simpler requirements but the behaviour should be equivalent for different transports.

Referencing Packages

MassTransit's Scheduling Api utilizes Quartz.Net. So, for it to work, we will need the to add a reference to the MassTransit.Quartz package to your project with:
$ dotnet add reference <project-name> MassTransit.Quartz --version 5.1.3
Once the reference is added, run dotnet restore to load the necessary extension methods to do the initialization.

Initialization

The initialization code for the in-memory implementation as simple as adding a call to UseInMemoryScheduler() on your bus configuration.
Using the in-memory scheduler uses non-durable storage. If the process terminates, any scheduled messages will be lost, immediately, never to be found again. For any production system, using a standalone service is recommended with persistent storage.

Sample Code

The code below shows a simple implementation of MassTransit and its scheduling system:

Running the Demo app

So, I run my app and my code schedules a message to be sent 3 seconds after sent by the user. This is my output:

Conclusion

Hope this serves as an introduction to the scheduling feature within MassTransit. I've been using MassTransit for a couple of years now and definitely would recommend it as a framework for your distributed systems. Want to learn more about MassTransit? Please consider reading the following posts:

    Source Code

    The source for this post is located on my GitHub page.
    In case you're interested, I recently pushed a more updated MassTransit/Docker/.NET Core 3.1 implementation to GitHub here: https://github.com/hd9/masstransit-rabbitmq

    See Also

    Monday, August 13, 2018

    Creating a MassTransit client/server application using RabbitMQ, .NET Core and Linux

    Let's test the versatile MassTransit framework using RabbitMQ, .NET Core and Linux and see if it can serve as a reliable messaging system.
    On a previous post we introduced MassTransit on this blog and presented and some reasons why it may be a real nice alternative to your system. Today we will review a simple use case: I will simulate a client/server architecture by building two console applications using MassTransit as the service bus, RabbitMQ as the transport running on Docker, .NET Core.

    Sounds complicated? Let's take a look.

    Installing RabbitMQ

    The quickest way to install RabbitMQ locally is by using RabbitMQ's official Docker image. Assuming you have docker installed on your machine, pulling and running it is as simple as:
    $ docker run --hostname rmqhost --name rmqcontainer -p 15672:15672 -p 5672:5672 rabbitmq:3.7.5-management
    Before running that command, let's examine what each part means:
    • --hostname rmqhost : sets the host name
    • --name rmqcontainer : sets the name of the container
    • -p 15672:15672 : maps container port 15672 to host port 15672 so you can access on your localhost
    • -p 5672:5672 : maps container port 5672 to host port 5672 so you can access on your localhost
    • rabbitmq:3.7.5-management : the name of the image to donwload and run. I chose that one because it has a nice UI to manage RabbitMQ in case you want to play with it.
    If you don't have that image yet, Docker will pull it for you and initialize a container based on the parameters above.

    Once the download is complete, Docker will init RabbitMQ for us. On my Fedora box, I get:

    With your image loaded and you can then access the Management UI on http://localhost:15672. Login with guest | guest:
    Cool. Now that we have RabbitMQ running, let's take a look at MassTransit.

    Building a shared .NET POCO Contract

    Since we're building a client and a server, we also need to build a shared contract they can access. Because the client and the server may (and should) run on different servers, the common solution is to build a class library and reference it on both projects.

    In .NET Core we simply type in the terminal:
    dotnet new console -n MassTransit.RabbitMQ.Contracts
    Then open that project and add this simple message:
    That's it. Build that project to validate everything is ok and let's move to the client.

    Building a MassTransit RabbitMQ Client

    As previously said, the client will need to do two important things:
    • connect to the bus so it can start sending messages
    • send messages that the server recognizes. That's why we created the contract on the step above.
    You can initialize your client with a command similar to the one executed above. Once you have your project created, the important bits are: adding a reference to the contract project previously created, initializing the bus and publishing a message to RabbitMQ.

    Initializing the bus

    To initialize the bus, write a code similar to:

    Publishing a message

    Now, we should be able to publish a message with a code like: Build this project with $ dotnet build to make sure everything is ok but don't run it yet. We still need to build the server.

    Building a MassTransit RabbitMQ Server

    The simple server is pretty similar to the client with the difference that:
    • the server does not publish messages
    • the server contains handlers (consumers) to messages published
    Build a new console app and reference the contract project created above. The gist below shows how to init a service that will run on a console terminal and handle the CreateAccount message:

    Testing

    Now, let's test both the client and the server to see if they're working ok. Don't forget that now you'll need your RabbitMQ instance running. I suggest running the apps side by side and let's see how that goes:

    My Client App:
     My Server App:
    And my RabbitMQ console:
    So, from the above, you see that the messages sent from the client are reaching the server. Remember, those applications do not know about each other and the only coupling between them is the standalone contract project that both references.

    Conclusion

    With all that we reach the end of this post. I hope this demo showed how simple it is to create a MassTransit client/server application using RabbitMQ, .NET Core. I've been using MassTransit for a couple of years now and definitely would recommend it as a base framework for your distributed systems. Want to learn more about MassTransit? Please consider reading the following posts:

    Source Code

    The source code used on this post is available on GitHub.

    In case you're interested, I recently pushed a more updated MassTransit/Docker/.NET Core 3.1 implementation to GitHub at: github.com/hd9/masstransit-rabbitmq

    See Also

    Monday, August 6, 2018

    MassTransit, a real alternative to NServiceBus?

    Understand how MassTransit could be a real alternative when building distributed systems on the .NET Platform.
    Photo by Markus Spiske on Unsplash

    Looking for an free and open-source alternative to NServiceBus? Maybe MassTransit could be what you are looking for. Let's understand the platform and how it could be used on your next project.

    What is MassTransit?

    MassTransit  is a lightweight service bus for building distributed .NET applications. The main goal is to provide a consistent, .NET friendly abstraction over the message transport (RabbitMQ, Azure Service Bus, etc.). MassTransit is not a new project. They've been around since 2007 and were created as an alternative to NServiceBus. In early 2014, the framework was rewritten to support asynchronous programming as well as leveraged the power of messaging platform. The code was also rewritten, resulting in an entirely new, completely asynchronous, and highly optimized framework for message processing.

    Why MassTransit

    Like NServiceBus, MassTransit helps decoupling your backend from your frontend (and in general, decoupling services), leveraging enterprise design patterns like CQRS and Eventual Consistency. Some of the features you will find in MassTransit are:
    • providing support for messages, sagas
    • supporting different transports
    • allows automated or custom retries on failures
    • asynchronous requests/responses
    • poison message handling
    • exception management
    • custom serialization
    • message correlation
    • routing
    • scheduling
    • support for modern technologies like Azure Service BusApache KafkaAzure Event Hub and Amazon SQS

    Customizations

    MassTransit is also extremely customizable and as mentioned previously, can run on different transports (RabbitMQ, Azure Service Bus, etc) providing enormous benefits as both are strong and stable platforms with different characteristics. It also supports the new .NET Standard on .NET Core and runs on multi-platforms.

    Cloud Support

    todo

    Transports

    MassTransit includes full support for several transports, most of which are traditional message brokers. RabbitMQ, ActiveMQ, Azure Service Bus, Apache Kafka, Azure Event Hub and Amazon SQS are supported.

    Sample Code

    With that introduction, let's review some code. If worked with NSB before, you're probably looking for three things:
    1. How to initialize the bus
    2. How to initialize the host (your backend, where your handlers run)
    3. How to initialize a client (where you send messages from).
    Please take a look at examples below to understand how it all works.

    Initializing the Bus

    The below code shows how to initialize a simple MassTransit Bus:

    Initializing a Server (Host)

    The below code shows how to initialize a simple MassTransit server:

    Initializing a Client

    The below code shows how to initialize a simple MassTransit client that talks to the above service:

    Other aspects to consider

    Before we end, I'd like to bring to your attention other things you should evaluate before committing on such an important framework to your organization. I'd recommend that you research about:
    • Support: NServiceBus, being backed by a commercial organization deserves a point on this item. Having a commercial organization backing the product may potentially be a more compelling argument to your employer. NSB offers multiple support options including consulting and on-site training.
    • Online Documentation: both frameworks have good documentation but NSB is definitely ahead on this criteria. You will find way more documentation about NSB.
    • Community: both MT and NSB have decent online communities, with the latter being bigger and more active.
    • Access to current technologies: MT and NSB will definitely provide you access to modern technologies like cloud and serverless but NSB seems to be ahead on that regard.
    • Active development: both NSB and MT have very active developments.
    • Open-Source: I preferably like to have access to the source code of the product I'm using, in case there are issues
    • Ecosystem: Particular, NSB's parent company, offers lots of other products that are worth considering. These products integrate well with NServiceBus and depending on your use of the platform may 

    Final Thoughts

    This ends our quick introduction about MassTransit. I've been using NServiceBus for quite some time and while I recognize its value, I wanted to explore viable (and cheaper) alternatives for my open-source projects. I've been using MT for a couple of years I can confirm that it's a solid alternative to NServiceBus. MassTransit today is also a bigger and more solid project than it was 5 years ago and definitely deserves your attention.

    However, choosing a critical framework like this is not only about code. I recommend you to also evaluate other criteria such as support, documentation, training and ecosystem. On pretty much every criteria (except for price) NServiceBus is ahead. And maybe those things will count for your organization.

    More about MassTransit

    Want to learn more about MassTransit? Please consider reading these articles:

    Source Code

    As always, the source code is available on my GitHub.

    References

    See Also

      Monday, September 25, 2017

      Custom Security on NServiceBus Endpoints

      See how to enhance the security on NServiceBus endpoints by using generic and elegant code.


      On a previous post, we discussed why as developers, it's important to consider security. Then we discussed how to secure our Asp.Net MVC application using a mix of custom role/claims providers. Today well's see how we can protect our NServiceBus endpoints.

      NServiceBus

      NServiceBus (NSB) is a messaging and workflow for .NET and .NET Core that,
      Supports a variety of messaging patterns and workflows on multiple transports like MSMQ, RabbitMQ, Azure, and Amazon SQS. Developers focus on core logic, fully abstracted from the underlying infrastructure. Runs on .NET or .NET Core on Windows, Linux, or in Docker containers.
      In other words, It helps us separating the concerns in our application, create simpler services using distributed messaging, simplify our workflow, easy to test and helping us implement interesting design patterns like CQRS.

      Securing NserviceBus Endpoints

      But how could we integrate a robust, generic security authorization in a backend?

      On previous post we discussed how to implement SecurityController, a db-independent security mechanism to our Asp.Net website. In order to use it in our business layer, we need to inject its dependencies (roles and groups) via its Init factory method described below:
      public class SecurityController
      {
          public static void Init(List<Role> roles, List<Group> groups)
          {
              allRoles = roles ?? new List<Role>();
              allGroups = groups ?? new List<Group>();
              allPerms = allRoles.SelectMany(r => r.Permissions).Distinct().ToList();
          }
      }
      How would we integrate our custom groups and roles in the NSB backend?

      NServiceBus.INeedInitialization

      Within NSB, we can use INeedInitialization interface to hook our code in the NSB initialization pipeline. By using INeedInitialization you are supposed to implement this inerface:
      public void Customize(BusConfiguration busConfiguration)

      Our custom SecurityRegistry

      When we initialize our NSB endpoint, we know that NSB will invoke classes that implement that interface, including our own SecurityRegistry:
       public class SecurityRegistry : INeedInitialization
      {
          public void Customize(BusConfiguration busConfiguration)
          {
              // todo :: load roles, groups from your repository...

              // init the security controller...
              SecurityController.Init(roles, groups);
          }
      }
      Now, let's review how do we plug our validator in the NSB pipeline so it gets executed whenever a new message is executed in our service layer.

      NServiceBus.IMutateIncomingMessages

      The elegant way of doing that using NSB is trough message mutators. Yes, a fancy name for a plugin, filter or whatever else you want to name it. Message mutators allow we injecting our own business logic before the messages reach your NSB Message handler. With mutators, you can mutate incoming and outcoming messages. Since we're talking about security here, don't think you wanna mutate outcoming messages, right

      Creating a MessageMutator

      So, let's go back to the code: we know that we need to create a class that would implement IMutateIncomingMessages where we want to validate commands submitted to our service layer. Please, just note that, in order for this to automatically work, we will need:
      • a base generic Command where we can now beforehand who submitted what, when;
      • a common user;
      • the permission associated to a particular message;
      • access to our SecurityController that will validate if that user has access to submit that command or not.
      The code below shows a sample validator implementing IMutateIncomingMessages:
      public class PermissionValidator : IMutateIncomingMessages
      {
          private CommandBase cmd;
          private User user;

          public object MutateIncoming(object message)
          {
              ValitatePermission(message);
              return message;
          }

          private void ValitatePermission(object message)
          {
              cmd = message as CommandBase;
              if (cmd == null)
                  return;

              // tries to load custom security permissions from command
              var pa = cmd.GetType().GetCustomAttribute(typeof (RequirePermissionAttribute), true) as RequirePermissionAttribute;
             
              // if class not decorated with RequirePermission, nothing to validate
              if (pa == null)
                  return;

              var user = LoadUser();

              if (user is null)
                  throw new SecurityException("Is the User null?");

              ValidatePermission(pa.Permission);           
          }

          private User LoadUser()
          {
              if (cmd.SubmittedBy == null)
                  return null;

              // todo :: load our user from repo...
              return user;
          }      

          private void ValidatePermission(string permission)
          {
              // todo :: add your custom permission validation...
          }
      }

      The RequirePermissionAttribute

      The last part in the puzzle is how to automatically map permissions to commands. This can be easily done by creating a custom attribute like:
      public class RequirePermissionAttribute : Attribute
      {
          public string Permission { get; set; }

          /// <summary>
          /// Validates only if the SubmittedBy is an existing User
          /// </summary>
          public RequirePermissionAttribute()
          {        
          }
         
          /// <summary>
          /// Validates if the SubmittedBy is an existing User AND if that user has specified permission
          /// </summary>
          /// <param name="p"></param>
          public RequirePermissionAttribute(string p)
          {
              Permission = p;
          }
      }
      That attribute could be now used to decorate commands present in our service layer like:
      [RequirePermissionAttribute(Permissions.UpdateUser)]
      public class UpdateUser : ContentCommandBase
      {
          [Required]
          [StringLength(100)]
          public string Name { get; set; }
          // etc...
      }

      Associating permissions and Commands

      The last step is to build the association between permissions and commands. The line below does exactly that. The code binds permission to its associated commands:
      // load custom security permissions from command
      var pa = cmd.GetType().GetCustomAttribute(typeof (RequirePermissionAttribute), true) as RequirePermissionAttribute;

      Conclusion

      On this post we reviewed how we can inject our custom security logic into a messaging framework like NServiceBus. Since most of these frameworks have extension points in their pipelines, it shouldn't be complicated to do a similar approach with MassTransit, for example. If want to read other posts about NServiceBus, please also consider:

      References

        See Also

          About the Author

          Bruno Hildenbrand