Microservices in ASP.NET

Microservices is the last significant change in modern development. Let's learn some tools and related design patterns by building a simplified e-commerce website using modern tools and techniques such as ASP.NET Core and Docker.

Photo by Adi Goldstein on Unsplash

For some time we've been discussing tools and technologies adjacent to microservices on this blog. Not randomly though. Most of these posts derived from my open-source project aspnet-microservices, a simple (yet complicated 😉) distributed application built primarily with .NET Core and Docker. While still work in progress, the project demoes important concepts in distributed architectures.

What's included in the project

This project uses popular tools such as:

• ASP.NET Core
• Docker, Docker Compose
• MongoDB, MySQL, Redis and Dapper
• RabbitMQ, MassTransit
  
• Vue.js and Axios.Js

On the administrative side, the project also includes:

• Db management: Redis Commander, Adminer and Mongo Express
• Administrative interfaces: to manage use services such as RabbitMQ Management Console.
• Monitoring: with Grafana, Prometheus and cadvisor.
• Log indexing (experimental): with the ELK stack.
  

Disclaimer

When you create a sample microservice-based application, you need to deal with complexity and make tough choices. For the aspnet-microservices application, I deliberately chose to balance complexity and architecture by reducing the emphasis on design patterns focusing on the development of the services themselves. The project was built to serve as an introduction and a start-point for those looking forward to working of Docker, Compose and microservices.

This project is not production-ready! Check Areas for Improvement for more information.

Microservices included in this project

So far, the project consists of the following services:

• Web: the frontend for our e-commerce application;
• Catalog: provides catalog information for the web store;
• Newsletter: accepts user emails and stores them in the newsletter database for future use;
• Order: provides order features for the web store;
• Account: provides account services (login, account creation, etc) for the web store;
• Recommendation: provides simple recommendations based on previous purchases;
• Notification: sends email notifications upon certain events in the system;
• Payment: simulates a fake payment store;
• Shipping: simulates a fake shipping store;

Technologies Used

The technologies used were cherry-picked from the most commonly used by the community. I chose to favour open-source alternatives over proprietary (or commercially-oriented) ones. You'll find in this bundle:

• ASP.NET Core: as the base of our microservices;
• Docker and Docker Compose: to build and run containers;
  
• MySQL: serving as a relational database for some microservices;
• MongoDB: serving as the catalog database for the Catalog microservice;
• Redis: serving as distributed caching store for the Web microservice;
  
• RabbitMQ: serving as the queue/communication layer over which our services will communicate;
• MassTransit: the interface between our apps and RabbitMQ supporting asynchronous communications between them;
• Dapper: lightweight ORM used to simplify interaction with the MySQL database;
• SendGrid: used to send emails from our Notification service as described on a previous post;
  
• Vue.js and Axios.Js to abstract the frontend of the Web microservice on a simple and powerful  JavaScript framework.
  

Conventions and Design Considerations

Among others, you'll find in this project that:

• The Web microservice serves as the frontend for our e-commerce application and implements the API Gateway / BFF design patterns routing the requests from the user to other services on an internal Docker network;
• Web caches catalog data a Redis data store; Feel free to use Redis Commander to delete cached entries if you wish or need to.
  
• Each microservice has its own database isolating its state from external services. MongoDB and MySQL were chosen as the main databases due to their popularity.
  
• All services were implemented as ASP.NET Core webapps exposing the endpoints /help and /ping so they can be inspected from and observed automatically the the running engine.
• No special logging infrastructure was added. Logs can be easily accessed via docker logs or indexed by a different application if you so desire.
  
• Microservices communicate between themselves via Pub/Sub and asynchronous request/response using MassTransit and RabbitMQ.
• The Notification microservice will eventually send emails. This project was tested with SendGrid but other SMTP servers should work from within/without the containers.
• Monitoring is experimental and includes Grafana sourcing its data from a Prometheus backend.
  

Technical Requirements

To run this project on your machine, please make sure you have installed:

• Docker Desktop (Mac, Windows) or Docker Engine (Linux)
• Docker Compose
• .NET SDK 3.1 or greater
• A git client
• A Linux shell or WSL

If you want to develop/extend/modify it, then I'd suggest you to also have:

• a valid GitHub account
• Visual Studio 2019
• (or) Visual Studio Code

Running the microservices

So let's get quickly learn how to load and build our own microservices.

Initializing the project

Get your copy by cloning the project:

git clone https://github.com/hd9/aspnet-microservices

Next open the solution src/AspNetContainers.sln with Visual Studio 2019. Since code is always the best documentation, the easiest way to understand the containers and their configurations is by reading the src/docker-compose.yml file.

Debugging with Visual Studio

Building and debugging with Visual Studio 2019 is straightforward. Simply open the AspNetMicroservices.sln solution from the src folder, build and run the project as debug (F5). Next, run the dependencies (Redis, MongoDB, RabbitMQ and MySQL) by issuing the below command from the src folder:

docker-compose -f docker-compose.debug.yml up

Running the services with Docker Compose

In order to run the services you'll need Docker and Docker Compose installed on your machine. Type the command below from the src folder on a terminal to start all services:

docker-compose up

Then to stop them:

docker-compose down

To remove everything, run:

docker-compose down -v

To run a specific service, do:

docker-compose up <service-name>

As soon as you run your services, Compose should start emitting on the console logs for each service:

The output of our docker-compose command

You can also query individual logs for services as usual with docker logs <svc-name>. For example:

~> docker logs src_catalog_1
info: CatalogSvc.Startup[0]
      DB Settings: ConnStr: mongodb://catalog-db:27017, Db: catalog, Collection: products
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://[::]:80
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app

Database Initialization

Database initialization is automatically handled by Compose. Check the docker-compose.yml file to understand how that happens. You'll find examples on how to initialize both MySQL and MongoDB.

Dockerfiles

Each microservice contains a Dockerfile in their respective roots and understanding them should be straightforward. If you never wrote a Dockerfile before, consider reading the official documentation.

Docker Compose

There are two docker-compose files in the solution. Their use is described below:

• docker-compose.yml: this is the main Compose file. Running this file means you won't be able to access some of the services as they'll not be exposed.
  
• docker-compose.debug.yml: this is the file you should run if you want to debug the microservices from Visual Studio. This file only contains the dependencies (Redis, MySQL, RabbitMQ, Mongo + admin interfaces) you'll need to use when debugging.

Accessing our App

If the application booted up correctly, go to http://localhost:8000 to access it. You should see a simple catalog and some other widgets. Go ahead and try to create an account. Just make sure that you have the settings correctly configured on your docker-compose.yml file:

Our simple e-commerce website. As most things, its beauty is in the details 😊.

Admin Interfaces

You'll still have available admin interfaces for our services on:

• Grafana: http://localhost:3000/
• MySQL Admin: http://localhost:8010/
• Mongo Express: http://localhost:8011/
• RabbitMQ dashboard: http://localhost:8012/
• Redis Commander: http://localhost:8013/
• The ELK Stack (Experimental): http://localhost:5601/app/kibana#/home

I won't go over the details about each of these apps. Feel free to explore on your own.

Monitoring

Experimental monitoring is available with Grafana, Prometheus and cadvisor. Open Grafana at http://localhost:3000/ and login with admin | admin, select the Docker dashboard and you should see metrics for the services similar to:

Grafana capturing and emitting telemetry about our microservices.

Quick Reference

As a summary, the microservices are configured to run at:

• Web: http://localhost:8000
• Catalog: http://localhost:8001
• Newsletter: http://localhost:8002
• Order: http://localhost:8003
• Account: http://localhost:8004
• Recommendation: http://localhost:8005
• Notification: http://localhost:8006
• Payment: http://localhost:8007
• Shipping: http://localhost:8008

The management tools are available on:

• Grafana: http://localhost:3000/
• MySQL Admin: http://localhost:8010/
• Mongo Express: http://localhost:8011/
• RabbitMQ dashboard: http://localhost:8012/
• Redis Commander: http://localhost:8013/
• The ELK Stack (Experimental): http://localhost:5601/app/kibana#/home.

And you can access the databases at:

• MySql databases: use Adminer at: http://localhost:8010/, enter the server name (ex. order-db for the order microservice) and use root | todo as username/password.
• MongoDB: use MongoExpress at: http://localhost:8011/. No username/password is required.
  

Final Thoughts

On this post I introduce to you my open-source project aspnet-microservices. This application was built as a way to present the foundations of Docker, Compose and microservices for the whole .NET community and hopefully serves as an intuitive guide for those starting in this area.

Microservices is the last significant change in modern development and requires learning lots (really, lots!) of new technologies and new design patterns. This project is by far complete and should not be used in production as it lacks basic cross-cutting concerns any production-ready project would need. I deliberately omitted them for simplicity else I could simply point you to this project. For more information, check the project's README on GitHub.

Feel free to play with it and above all, learn and have fun!

Source Code

As always, the source code is available on GitHub at: github.com/hd9/spnet-microservices.

References

• Microservices - Design Patterns
  
• Microservices architecture | Microsoft Docs
• API Gateway
• Dockerfile
• Docker Networking
• Docker logs
• Docker Compose
• Create a web API with ASP.NET Core and MongoDB
• Docker images for ASP.NET Core
• Development workflow for Docker apps
• Dockerize an ASP.NET Core application

See Also

• My journey to 1 million articles read
• Deploying Docker images to Azure App Services
• Pushing Docker images to ACR - Azure Container Registry
• Distributed caching in ASP.NET Core using Redis, MongoDB and Docker
• Async Request/Response with MassTransit, RabbitMQ, Docker and .NET core
• Configuration in .NET Core console applications
• Adding Application Insights telemetry to your ASP.NET Core website
• Building and Hosting Docker images on GitHub with GitHub Actions
• Deploying Docker images to Azure App Services
• Hosting Docker images on GitHub
• How to push Docker images to ACR - Azure Container Registry
• Send emails from ASP.NET Core websites using SendGrid and Azure
• Sam Newman: Monolith to Microservices
• github.com/dotnet-architecture/eShopOnContainers

Patching RavenDB Metadata

On this post, let's understand how simple it is to patch the metadata of your documents using RavenDB Studio

On a previous post we talked about how to query metadata using RavenDB. Now, let's take a look at some ways to patch our metadata in RavenDB.

Patching Records with RavenDB

Patching records is RavenDB can be easy or less easy =). The easy way is doing it from RavenDB Studio. There you can do it by collection or index and write some JavaScript to it.

Patching using RavenDB Studio

Here's a simple example on how I can patch the Name and IsActivated fields in my Users collection:

Patching metadata using RavenDB Studio

But, you may be required to patch the metadata, how to do it? The answer is: Using the @metadata dictionary. Let's take a look.

Final Thoughts

In the example above, we're updating the document metadata property "Raven-Clr-Type" with "MyApp.Core.User, MyApp.Core":

this["@metadata"]["Raven-Clr-Type"] = "MyApp.Core.User, MyApp.Core";

You can obviously patch whatever you want in your metadata. In the previous example, I was patching the namespace of an entity that was refactored in my application.

Hope it helps!

See Also

• My journey to 1 million articles read
• A simple introduction to RavenDB
• An in-depth review of the RavenDB Cloud
• Installing and Running RavenDB on Windows and Linux
• Running RavenDB on Docker
• Querying RavenDB Metadata
• Creating ASP.NET Core websites with Docker
• Send emails from ASP.NET Core websites using SendGrid and Azure

Querying RavenDB Metadata

Need to query your RavenDB metadata? Read to understand.

We've been discussing RavenDB on this blog. On this post, I want to address a feature that's commonly asked: how to query metadata in RavenDB.
There are two ways to do this:

• using an pre-defined index: adding metadata to an index and querying it;
• using a dynamic: using dynamic indexes in the studio.

Querying Metadata in RavenDB using an Index

This should be your standard approach, basically resumed in:

1. Access the document's metadata from your index;
2. Query the indexes, using the metadata property as if it were a normal property.

 Here's what you should do to access a document's metadata from your index:

Once that set, here's how you would query it:

Simple enough, isn't it? This is the best option as your data will be always searched using the index causing minimum stress to your database.

Querying Metadata in RavenDB dynamically using the Studio

But sometimes is unavoidable: users will come to you and ask you to run some queries. What if you don't have that property indexed? You have to options:

1. add that property to an index in rebuild the index - not recommended on production;
2. run a query against the dynamic index in RavenDB. The dynamic index allows you to write your queries against the collection and RavenDB will, behind the scenes, create the index for you. Note that we shouldn't be doing this frequently because it adds some stress to the server as it will build an index to process that query but

In order to query for the metadata in the studio, here's how you should do it:

So, using the below Lucene query allows you to query a document's metadata using RavenDb. Just remember to select the dynamic index. In this case, I'm using the dynamic/Users index.

@metadata.Raven-Clr-Type:"HildenCo.User, HildenCo.Domain"

Querying metadata using Lucene in C#

If I wanted, we can also use the client api to query RavenDB Lucene from my C# application. The following example shows us how to do it:

Conclusion

On this post I tried to show a few ways to query metadata. Hope it helps.

See Also

• My journey to 1 million articles read
• A simple introduction to RavenDB
• Installing and Running RavenDB on Windows and Linux
• An in-depth review of the RavenDB Cloud
• Patching RavenDB Metadata
• Creating ASP.NET Core websites with Docker
• Send emails from ASP.NET Core websites using SendGrid and Azure

Installing and Running RavenDB on Windows and Linux

Let's install and run RavenDB on Windows and Linux and learn how it works.

On a previous post  I introduced RavenDB on this blog. On this one let's review how to install or run a standalone RavenDB instance on our machine.

On this post we will cover:

1. Installing and running on Windows;
2. Installing and running on Linux;
3. Using the RavenDB console tool;
4. Creating a new database;

Downloading RavenDB

First off, navigate to the RavenDB downloads page and download the server version for the environment you're working on. The currently supported platforms are: Windows, Linux, OSX, Raspberry PI and Docker.

To download your image, select Server, Stable and the appropriate version for your environment. Aaccept the terms, click on the .ZIP Package download button to download the image to your disk.

Running standalone RavenDB on Windows

On Windows, once the download is completed, extract all those files in a folder and you'll see two PowerShell files: run.ps1 and setup-as-service.ps1.

Open the Powershell terminal, cd into the folder you extract your files and run .\run.ps1 You'll then see some outputs the RavenDB service is emitting for us when running as a standalone instance:

A new window will open for you where you'll need to configure a cluster and/or security. For now, let's skip the cluster configuration and go with the Unsecure option. 

This configuration is enough for this demo and simple development efforts. Clicking on it, RavenDB Studio will open on the default Url: http://127.0.0.1:8080/studio/index.html.

That's it! The standalone instance is running and you can start testing RavenDB on your Windows box.

Installing on RavenDB Windows

To install you your machine,  open the PowerShell terminal as an administrator and run the setup-as-service.ps1 script.

If all goes well, you'll  have to accept the user license agreement and proceed with the instance configuration.

Note that:

• the installation requires administrator privileges
• will use port 8080 for "unsecure" installs or 443 for secure (options selected during the installation)

Configuring the new Instance

After installed, you'll have to configure your instance as shows the image below. For a development setup, you should be good with the Unsecure option.

Clicking on it, will prompt you for Http/Tcp ports and IP address. Leaving empty will use the defaults. Click "Restart Server" and RavenDB should be installed.

The RavenDB Service

Once installed, on Windows, we can view the service status using the Get-Service Powershell cmdlet:
Get-Service -Name RavenDB

For more information, please visit: https://ravendb.net/docs/article-page/4.0/csharp/start/installation/setup-wizard

Running standalone RavenDB on Linux

The Linux installation is similar: download your Linux image from the RavenDB downloads page, unzip it and run the script. Let's see how it works.

Download the image by selecting Linux x64 from the downloads page and download it using Firefox:

Once downloaded, extract the bz2 pkg on a local folder:

Cd into that folder and run "run.sh". I should then see:

Installing RavenDB on Linux

Installing RavenDB on Linux is very similar to Windows. You run the run.sh shell script and select the installation option on the command line.

The RavenDB Console

After installed, basic manipulation of the server can be done either by using the Raven UI (Raven Studio) or by using the console previously opened.

For example, when I type help on my shell in Fedora, I get:

From the console, You can do things like restarting/shutting down the server, like exporting/importing data, reading logs and viewing server stats. Just type the commands shown. For example, to shut the instance down, I should type: shutdown.

Creating a Database

The final needed step before touching code is to create a database. For that do: Databases -> New Database:

Enter a DB name (for example, "Blog") and click "Create"):

Clicking on that Database creates our database and takes us to the its page. From there we basically can view our documents (records), create, query, patch, view logs, stats, etc:

Next Steps

Now that RavenDB is running and the database is created, the next step is to start interacting with it. You can use either RavenDB Studio or the client Api (with C#, Java, Python, etc). For more details, check my simple introduction to RavenDB.

Don't forget that RavenDB is also available on the cloud. Check the article An in depth review of the RavenDB Cloud for more information.

Conclusion

Hope this post shows how is a simple introduction on how to install RavenDB on Windows and Linux boxes. For more information, check the official documentation.

References

• RavenDB - Gettting Started
• RavenDB Studio
• RavenDB - Client Api

See Also

• My journey to 1 million articles read
• Creating ASP.NET Core websites with Docker
• Send emails from ASP.NET Core websites using SendGrid and Azure
• A simple introduction to RavenDB
• An in depth review of the RavenDB Cloud
• Why I use Fedora Linux
• Running RavenDB on Docker
• Querying RavenDB Metadata
• Patching RavenDB Metadata

A simple introduction to RavenDB

RavenDB may be an excellent alternative for developers looking for a reliable, robust and friendly NoSql database. Read to understand.

In case you never heard of it, RavenDB is a fast, reliable and extensible NoSql database. Being a NoSql database, it has benefits and drawbacks over Sql databases. On this post, let's take a look at how RavenDB behaves with Linux, containers and .NET Core.

If you're interested in more information on why to use a NoSql database, please check this article.

Why RavenDB?

With so many big names in the NoSql market, why even consider RavenDB?

Easy to use

Raven is extremely easy and simple to use, either by developers and non-technical people. In a couple of minutes, non-technical users were able already to interact with the database with no major hiccups.

Multiplatform

RavenDB is multiplatform and can be installed on Linux, Windows, Mac, Docker and even on a Raspberry Pi. Installation is also straight-forward. On Windows and Linux, testing is as simple as downloading the executable, extracting and running the executable. On Docker, a simple docker pull solves.

Raven Studio

Raven Studio is the front-end to interact with RavenDB. It's the app responsible for making the user experience enjoyable for developers and non-developers alike. It’s included with any license, including the free community version.

Multiple Clients

RavenDB client can be accessed using the major programming languages in the market, including C#, Java, Node, Python, Ruby and Go!

Transactional NoSQL

RavenDB is the first non-relational database to achieve ACID across the entire database. Maintain the best of SQL while boosting your capacity to the next level.

Extensive Management Features

You can set up a distributed data cluster in minutes. Replicate your database in real time so you can be everywhere at once, and always available to your users.

Simple API with an embedded ORM

This is especially important to developers. It means easily creating/updating/deleting records, quicker to test and to release, no migration scripts, simple and powerful api. For example, this is how I insert a record in the database:

Example: a very simple insert operation in Raven.

Simple NoSQL Querying

RavenDB has powerful ways to do query including geospacial, faceted, full-text and map-reduce operations. But since most of the time, what you want is very easily available to you via the api and indexes.

Example: a very simple "select * from Category" query in RavenDB.

Extensive Querying Types

Apart from the simplicity shown above, RavenDB brings awesome querying support such as Full text search, Spatial, Facets and a built-in graph api.

Index Support

Indexes in RavenDB are one of the strongest points. Being a NoSql database, it's an intelligent solution to avoiding multiple requests to the database by merging multiple tables. We will also revise this in future posts.

In-Memory Database

RavenDB has an in-memory database that you can use to persist data from your application (and replace SqlLite for example) or, better yet, use in your tests so you have real db operations (in-memory) without the hassle of creating mocks and simulating your tests.

Extensions / Bundles

You can extend the database with bundles and extensions. Some come already built in for us, others can be created through what they call Bundles. It's more of a technical solution but very interesting feature to have available. For example, these are the out of the box bundles that come with a standard RavenDB installation:

Powerful and Extensive API

Apart from (most of the time) simple operations, we have lots of advanced features in the client api to manage everything from tables (collections), records, bulk operations to database indexes, etc.

RavenDB Cloud

Yes, RavenDB is also available as a SAAS on its RavenDB Cloud. Ideally, it serves really well to small teams so that you get a hosted your databases without any need to manage VMs on your favorite cloud provider.

I tested and liked it! Please check this post for more information about the RavenDB Cloud.

Final Thoughts

In almost 20 years working with software development, I know how difficult it is to manage and evolve databases during the project life cycle. Being agile today requires releasing as fast as possible and complex operations like data migration can have a big impact on your domain. My experience with RavenDB, is that development, patches, migrations and even clustering are greatly simplified.

And because it makes sense to do polyglot persistence, why not use RavenDB, beside your existing Sql instance of a or even side by side with your MongoDB, CouchDB, Redis and Cassandra? So go ahead, install it on your machine (or test its Cloud) and give it a try!

See Also

• My journey to 1 million articles read
• An in depth review of the RavenDB Cloud
• Installing and Running RavenDB on Windows and Linux
• Creating ASP.NET Core websites with Docker
• Send emails from ASP.NET Core websites using SendGrid and Azure
• Running RavenDB on Docker
• Querying RavenDB Metadata
• Patching RavenDB Metadata

Seven Databases in Seven Weeks, 2nd Edition

The book is an interesting heads-up on databases being used on different fields throughout the world.

You may or may not have heard about polyglot persistence. The fact is that more and more, software projects are making use of different technologies. And when it comes to the database world, the number of options is immense: it can be relational, document or columnar databases, key-value stores, graph databases, not to mention other cloud infrastructure options like service buses, blobs, storage accounts and queues.

What to use? Where? And how does that affects developers?

Choosing a database is perhaps one of the most important architectural decisions a developer can make. In that regard, I'd like to recommend a very interesting book that addresses some of that discussion: Seven Databases in Seven Weeks, 2nd Edition.

Why you should read this book

Because the book:

• provides practical and conceptual introductions to Redis, Neo4J, CouchDB, MongoDB, HBase, PostgreSQL, and DynamoDB
• introduces you to different technologies encouraging you to run your own experiences
• revises important topics like the CAP Theorem
• will give you an overview of what’s out there so you can choose the best tool for the job
• explore some cutting-edge databases available - from a traditional relational database to newer NoSQL approaches
• and make informed decisions about challenging data storage problems
• tackle a real-world problem that highlights the concepts and features that make it shine

Conclusion

Whether you're a programmer building the next big thing, a data scientist seeking solutions to thorny problems, or a technology enthusiast venturing into new territory, you will find something to inspire you in this book.

Reference

• Seven Databases in Seven Weeks, 2nd Edition

See Also

• My journey to 1 million articles read
• Skill Up: A Software Developer's Guide to Life and Career
• Hello, Startup - A book for developers building their startups
• Non-technical skills for software developers
• Interview questions for QA analysts