Showing posts with label RavenDB. Show all posts
Showing posts with label RavenDB. Show all posts

Thursday, December 12, 2019

An in depth review of the RavenDB Cloud

The RavenDB Cloud may be an excellent choice for developers looking for a SAAS database on the cloud.

The RavenDB team recently launched the RavenDB Cloud. Since I have been working with  RavenDB for some time, I thought it would be interesting to investigate the Raven Cloud more in-depth.

In this article we will learn:

  1. About RavenDB;
  2. About cloud integrations (AWS, Azure, Google Cloud);
  3. How to create an account;
  4. How to deploy you free AWS instance;
  5. How to manage your server;
  6. How to interact with it through code;
  7. Potential risks;
  8. What's next?

About RavenDB

I’m a big fan of RavenDB. I've been using it on production for the last 5 years and the database has been fast, reliable, secure and given that it provides a robust and friendly C# Api and lots of interesting features, it was definitely a good choice for our product.

If you want to know more about my RavenDB experience, please click here.

RavenDB 4 introduces many welcome enhancements. Below I highlight my favourites:
  • Management Studio – much faster, customizeable and intuitive.
  • Speed – Raven 4 is faster than 3.5. All Raven 4 tested features (Db imports, queries, patches and custom implementation) showed significant improvements against the previous version.
  • Security – the authentication now happens via certificates instead of usernames/passwords and the databse offers encrypted storage and backups out of the box.
  • New Server Dashboard – the new dashboard offers a holistic overview of the cluster, nodes and databases deployed on a cloud account.
  • Clustering – Raven 4 works on a cluster fashion instead of an instance fashion. This brings better performance, stability and consistency for applications.
  • Auto-Backups: you can setup the database to run periodic backups (incremental or not). This feature is also present on the RavenDB helping to reduce the reliance on personalized jobs and/or scripts. Backups can be encrypted and uploaded to different servers including blob storages on AWS and Azure.
  • Ongoing Tasks – the new Manage Ongoing Tasks interface simplifies the configuration and deployment of important services such as ETL, Sql Replication and Backups.
  • RQL – RQL is the new Raven Query Language - a mix of LINQ and JavaScript. Much clearer and intuitive than Lucene. See the Queries section for more information.
With all that information now let’s review some of the most interesting aspects of the RavenDB Cloud.

Licensing

Currently, three types of licenses are available: Free, Developer and Production. The main differences are:
  • Only one free version per cloud account
  • The free version runs only on AWS US East 1
  • Some features (such as SQL replication) are only available on the Production version
If you would like to test RavenDB, remember that you can also have local database.
Other aspects of the the free version are (as of Dec, 2019):

Services

Which services are available on the free version? In summary, a free RavenDB Cloud license allows to:
  • Create a new RavenDB instance
  • Manage the RavenDB server and databases
  • Configure some aspects of the database (other were not available with the free version)
  • Import an  existing database
  • Query data using a simple console tool
  • Test partially the SQL replication feature
  • Test partially the backup feature

AWS, Azure and Google Cloud Integration

The RavenDB Cloud can aslo be deployed on AWS and Azure’s most popular regions and they're also working on the Google Cloud. That means the database could sit on the same datacenter as your application reducing the latency between your services.

Pricing

The price varies by tier and time utilization. This is the estimated pricing from the Development Tier (Dec 09, 2019):

Creating an Account

The process to create an account is simple. Just go to cloud.raven.net, click Get started for free:
Enter your email and proceed with the setup. You should get an access token to access your created account.

Creating your Instance

To create your database, log in to your account utilizing the email token and click Add Product:
Specify the required information:
Aand review your request to proceed with the deployment:

Deployment

With all the above information submitted, the deployment process starts. It took me around 2 minutes to have the new cluster provisioned. Once deployed, you should see it on the Raven Cloud portal:

Accessing the Instance

In order to access that cluster, clicking on the Manage button takes us to the new Management Studio also available on RavenDB 4. New users will be required to install a new certificate. Once logged in, here’s what the new RavenDB Studio looks like:
Some of the interesting features of the RavenDB Studio are:
  • a nice overview of our databases on the right. From there you can quickly view failed indexes, alerts and errors.
  • database telemetry including cpu, memory, storage and indexing
  • database management
  • a global overview of the database cluster 
  • real-time cluster monitoring
  • customizeable

Creating Databases

Let's now create and import databases. The process to create a new database is simple. Click Databases -> New Database:

Enter the database name and some other properties and the database is quickly created. The deployment of a new database takes less than 5 seconds. An empty database utilizes aproximately 60 Mb on disk.

Costs

Unfortunately I can't provide an estimate on costs but I'd like to recall that you should consider also consider costs for:
  • networking: networking costs will vary and likely increase your costs.
  • storage: backup costs will also vary and will probably increase your costs.

Replication

Replication is done directly through Raven Studio either on creation or on settings. The database administrator sets how many nodes of the cluster he’d like to use and chooses between dynamic/manual replication and Raven handles all the rest. This is an important feature as it allows your database to be up in case one node within your cluster fails.

Importing Databases

I also tested the database important and more importantly, if it would be easy to migrate data between Raven 3.5 and Raven 4. Luckily the process of importing databases didn’t change much and a new RavenDB 4 accepts most of the imported data successfully. This is how the import process looks like:
For the record, my cloud instance imported of 1.5 million records in just over 2:40 minutes:

Managing your Server

From the Manage your Server section you'll have access to tools such as cluster, client configuration, logs, certificates, backups, traffic, storage, queries and more. You can see what's available below to manage your cluster:

Database Tools

Under Manage Ongoing Tasks you will also find interesting database-specific resources:

Backups

Backups are handled directly from Management Portal -> Database -> Settings -> Manage Ongoing Tasks tool:

I was pleasantly surprised that they offer backups to AmazonS3 buckets and Azure blob storage by default:
Once setup, you'll see that the automatic backup runs periodically:

Backup Considerations

Other imporant considerations regarding backups are:
  • The Free and Production tiers are regularly and automatically backed up.
  • You can define your own custom backup tasks, as you would with an on-premises RavenDB server.
  • A mandatory-backup task that stores a full backup every 24 hours 
  • An incremental backup happens every 30 minutes
  • Backups created by the mandatory backup routine are stored in a RavenDB Cloud
  • You will have no direct access to backups
  • You can view and restore them using your portal's Backups tab and the management Studio.
  • Mandatory-backup files are kept in RavenDB's own cloud.
  • RavenDB offers 1 GB per product per month for free
  • The backup storage usage is measured once a day, and you'll be charged each month based on your average daily usage.

External Replication

We can also configure external replication by Selecting a Database / Settings / Manage Ongoing Tasks / Add Task / External Replication. The screenshot below shows the options:

SQL Replication

The Raven Cloud also supports SQL replication. Unfortunately this feature was not available on the free version. From what I could test, the feature didn’t change much from Raven 3.5 and runs well.
The next step is to write a transformation script. For example:
All this done, the Raven will automatically replicate its records to the remote SQL database. This is excellent for reporting and, in case you need, querying your NoSql data from a traditional SQL database.

Scheduled Backups

Scheduled backups can also be customized and are available at Database / Setup / Manage Ongoing Tasks / Add Task:

Scaling

Being clustered by default, RavenDB 4 can be easily scaled via the Portal. The documentation describes in detail how it can be configured. Below, a screenshot provided by RavenDB on how it should work (the feature isn't available on the free tier):

Security

The RavenDB Cloud offer strong security features including:
  • Authentication: RavenDB uses X.509 certificate-based authentication. All access happens via certificates, all instances are encrypted using HTTPS / TLS 1.2 / X.509 certificates.
  • IP restriction: you can choose which IP addresses your server can be contacted by.
  • Database Encryption: implemented at the storage level, with XChaCha20-Poly1305 authenticated encryption using 256 bit keys.
  • Encryption at Rest: the raw data is encrypted and unreadable without possession of the secret key.
  • Encrypted Backups: your mandatory backup routines produce encrypted backup files. 

For more information, please read RavenDB on the Cloud: Security.

Cluster Api

There's also a Cluster API to managing the cluster. It allows your devops team to script database operations including:

For example, we can dynamically add a node to a cluster by running the following PowerShell script:
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$clientCert = Get-PfxCertificate -FilePath <path-to-pfx-cert>
Invoke-WebRequest -Method Put -URI "http:///admin/cluster/node?url=&tag=&watcher=&assignedCores= -Certificate $cert"

Or with the equivalent in cURL:
curl -X PUT http:///admin/cluster/node?url=&tag=&watcher=&assignedCores= --cert

Database Api

Since we’re talking devops, it’s important to note that the client api only manages nodes on the cluster. By itself, that feature still isn’t sufficient to start a new working environment since databases, indexes and data would be required.

In case you're interested, I created a RavenDb.Cloud spike tool to review database operations.

Querying the Cloud Database

A major change happened on Raven 4: RQL replaces Lucene as the default language for queries and patches. If you don’t run RavenDB yet, you shouldn’t be alarmed. But for folks migrating from RavenDB 3.5, that potentially will be a significant impact requiring bigchanges on the code.

The good news is that RQL comes with important changes and multiple improvements on the new Management Studio. The UI is now friendlier, faster and simpler to use, query and export data.

RQL’s the syntax is a mix of .NET’s LINQ and JavaScript. Overall, it’s elegant, clean and simple to use. It also makes querying and patching the Raven database simpler. However, for users currently relying on Lucene, this may represent a risk as those queries will have to be migrated (and subsequently, extensively tested) to RQL.

Further Reference:
Breaking the language barrier
Querying: RQL - Raven Query Language

Running queries on the Portal

Querying using RQL from is straightforward. .NET developers should recognize the language since it's very similar to LINQ:

Customizating Queries

The Studio allows us to choose which columns should be returned by the results table:

Patching Data

The patch tool accepts RQL. For example, this is how we apply a simple patch using the new syntax:

Managing the database using the C# api

When we consider a database on the cloud we should ask how is the support for automation. RavenDB provides a powerful C# api that I show below some of the operations (source here):

Risks

With every new technology, there are risks. It’s important to understand that even if RavenDB is a mature technology (and their developers are bright!), there are risks that should be considered with this and any new platform. I highlight:
  • Costs: I wasn't able to determine the overall cost mainly because all values provided by RavenDB are estimates. If the costs are an important requirement for the migration, a more in-depth evaluation should be performed.
  • Performance: I didn't invest much time testing the performance of the Raven Cloud. The good new is that Raven 4 is much faster than 3.5 and is potentially offered on the same Cloud provider / region as your application.
  • Hidden Costs: as previously said, all prices listed are estimates. It’s probable that other costs will be added to your bill at the end of the month.
  • RQL – RQL is the new way to run queries against the Raven 4 database. However, due to the amount and complexity of some of our queries relying on Lucene (the old advanced way of querying the Raven database), migrating all the complex queries to RQL will be a challenge in terms of time and testing efforts necessary. 
  • Major changes on the API: ignore this if you're new to RavenDB. But, if you were using RavenDB 3.5 there aresignificant changes on the RavenDB 4 Api. Broken dependencies will potentially be: business logic (if implicitly coupled to RavenDb 3.5), indexes, tests and tools. Also, Lucene queries, Map-Reduce indexes, patches and logic that contains bulk-insert operations will likely have to be upgraded.
  • NServiceBus: the Raven 4 api requires conflicting libraries with NServiceBus. So it may be possible that a RavenDB upgrade will also require you a NServiceBus upgrade.

For Further Investigation

My short experience with the RavenDB Cloud was solid. However I would like to highlight other topics that could potentially be researched in the future:
  • Full Cost Estimate – all the costs on this post are estimates are subject to variation. Most of these estimates were provided by RavenDB on the Raven Cloud website. It’s highly probable that on a real production environment, costs will be bigger. But, for what the Raven Cloud provide, I still find their prices very attractive.
  • Performance Benchmarks – I personally didn’t do any performance benchmark when testing the Raven Cloud. Based on this exercise, I did realize that both the local and the cloud versions or RavenDB 4 showed a good increase in the overall performance.
  • Security – No security tests were performed as outside of the scope of the spike. My understanding is that security is way beefier on Raven 4. But how secure is it?
  • SQL Integration – The free version doesn’t support sql replication. It’s a very important feature for those that need some sort of reporting. Probably a good reason to go to the dev/prod subscriptions.
  • Backup/Restore – The backup/restore feature wasn’t tested because the only available option for the AWS free version was on S3 storage. Worth investigating if considering using the Raven Cloud on production. My experience with a local install of Raven 4 is that it’s reliable and super fast!
  • Smuggler – The smuggler tool is available on the Raven 4 Api. I built a simple console tool to manage databases and import/export data. The source code is available here.
  • Cluster Api – since the free version does not include clustering, I couldn’t test the Api. However, since the Raven Apis are extensive and well document, I don’t expect any problems with that.

Conclusion

This ends the Raven Cloud evaluation, thanks for reading! Hopefully this quick look on these features helped clarigying what RavenDB offers. For me, RavenDB is a very strong alternative on the NoSql market and its cloud brings significant benefits for teams looking forward to reducing costs.

I dare to say that, for all it provides, RavenDB Cloud is a strong contender against MongoDB Atlas, Elastic Search and Azure CosmosDB.

References

See Also

Posted by
Labels: , , , , , , , , , ,

Monday, July 30, 2018

Patching a property from a json string in RavenDB

On this post, let's understand how to patch data using a json string in RavenDB

Because RavenDB is a NoSql database and because it stores its data internally as Json strings, sometimes when patching, we can simplify things immensely by using common JavaScript resources.

Up to Raven 3.5 the patch window accepted JavaScript (including running fancy lodash commands). So, being JavaScript, it's also valid to declare our own functions and/or use standard JavaScript functions including JSON.parse().

The Problem

My patch required to update one property (ContentName) from another (OriginalContent). Turns out that OriginalContent is a string that may or may not contain Json data and I need to access the property Name of that - if valid - Json object.

Sounds complicated? How do we do that?

The Solution

Take a look at the patch below to understand what I mean:


In this case my OriginalContent property stores the generic Json string that I pulled from an integration endpoint. That property is a string and can be anything: null, empty, a valid or invalid Json string, a simple string or a complex json data structure.

And what did I do on that patch?

I created a function to simply parse that property from a string to a Json object. If parsed, my code then tries to access its Name property. Then I set my ContentName property to, if available, the parsed Name from the Json string.

See Also

Posted by
Labels: ,

Monday, July 23, 2018

Patching RavenDB Metadata

On this post, let's understand how simple it is to patch the metadata of your documents using RavenDB Studio
On a previous post we talked about how to query metadata using RavenDB. Now, let's take a look at some ways to patch our metadata in RavenDB.

Patching Records with RavenDB

Patching records is RavenDB can be easy or less easy =). The easy way is doing it from RavenDB Studio. There you can do it by collection or index and write some JavaScript to it.

Patching using RavenDB Studio

Here's a simple example on how I can patch the Name and IsActivated fields in my Users collection:

Patching metadata using RavenDB Studio

But, you may be required to patch the metadata, how to do it? The answer is: Using the @metadata dictionary. Let's take a look.

Final Thoughts

In the example above, we're updating the document metadata property "Raven-Clr-Type" with "MyApp.Core.User, MyApp.Core":

this["@metadata"]["Raven-Clr-Type"] = "MyApp.Core.User, MyApp.Core";

You can obviously patch whatever you want in your metadata. In the previous example, I was patching the namespace of an entity that was refactored in my application.

Hope it helps!

See Also

Posted by
Labels: , , ,

Monday, July 16, 2018

Querying RavenDB Metadata

Need to query your RavenDB metadata? Read to understand.
We've been discussing RavenDB on this blog. On this post, I want to address a feature that's commonly asked: how to query metadata in RavenDB.
There are two ways to do this:
  • using an pre-defined index: adding metadata to an index and querying it;
  • using a dynamic: using dynamic indexes in the studio.

Querying Metadata in RavenDB using an Index

This should be your standard approach, basically resumed in:
  1. Access the document's metadata from your index;
  2. Query the indexes, using the metadata property as if it were a normal property.
 Here's what you should do to access a document's metadata from your index:

Once that set, here's how you would query it:

Simple enough, isn't it? This is the best option as your data will be always searched using the index causing minimum stress to your database.

Querying Metadata in RavenDB dynamically using the Studio

But sometimes is unavoidable: users will come to you and ask you to run some queries. What if you don't have that property indexed? You have to options:
  1. add that property to an index in rebuild the index - not recommended on production;
  2. run a query against the dynamic index in RavenDB. The dynamic index allows you to write your queries against the collection and RavenDB will, behind the scenes, create the index for you. Note that we shouldn't be doing this frequently because it adds some stress to the server as it will build an index to process that query but
In order to query for the metadata in the studio, here's how you should do it:

So, using the below Lucene query allows you to query a document's metadata using RavenDb. Just remember to select the dynamic index. In this case, I'm using the dynamic/Users index.

@metadata.Raven-Clr-Type:"HildenCo.User, HildenCo.Domain"

Querying metadata using Lucene in C#

If I wanted, we can also use the client api to query RavenDB Lucene from my C# application. The following example shows us how to do it:

Conclusion

On this post I tried to show a few ways to query metadata. Hope it helps.

See Also

Posted by
Labels: , , , , , , ,

Monday, June 11, 2018

Running RavenDB on Docker

Developers are now able to run RavenDB on Docker. On this post, let's review how that works.

RavenDB 4.0 brought us ability to be run Docker containers. On this post, let's review in detail how that works and test it on a Linux box. But before, let's jump directly to a quick review on how to get it installed on our Linux boxes.

Installing Docker

Installing Docker on Ubuntu/Fedora workstations shouldn't be more complicated than running these 2 lines below on the terminal. Windows users please check this page out.

Installing on Ubuntu

# In Ubuntu and Debian-based distros
sudo apt-get install docker

Installing on RH-based distros (Fedora, CentOS, etc)

# In RH based distros
sudo dnf install docker

Running a Docker RavenDB Instance

Now the fun part! Running RavenDB on Docker is super simple. The RavenDB team on Docker Hub already provided a Docker image that can be downloaded directly from Docker Hub. Assuming that your machine has internet connection, we just need to run the command below on a terminal:
sudo docker run -p 8080:8080 ravendb/ravendb
For example, when I ran that command on my Fedora machine, I get:

Reviewing the Previous Command

Let's review what happened on the above screenshot:
  • Docker detected that I didn't have the image available locally so it pulls the binary directly from Docker Hub
  • After completed the download, the RavenDB initialization starts
  • We then see RavenDB initialization messages such as Tcp listening on 172.17.0.2:38888. This is Raven's telling us its default TCP port. Note that we didn't map that port out of the container, so it will not be available
  • Once the "Running non-interactive" message is outputted, the instance is ready to be tested.

Accessing the Instance

With the RavenDB docker instance running, let's see how to access it. Since we mapped the internal and external container ports to 8080, I can access my instance by navigating to https://localhost:8080/ The "Welcome message" shows up as for a regular installation. It's because that docker instance is treated as a brand new installation.

Configuring RavenDb's Docker Instance

Now, let's proceed with the configuration of our RavenDB instance. I have outlined everything here in case you want to review it. TLDR, click on "Unsecure" to start initialize a development version:

Then Raven Studio is open for us. This is what we need to start interacting with the database:

Post-Installation

That's it! You now have a simple RavenDB Docker node running on your machine. Please note that In Docker, we can also use port mapping with the -p flag, To do so, we'll need to check the box "Customize external IP and Ports" and supply the external IP address as well as the exposed ports. So I then can create container using:
$ sudo docker run -t -p 38889:38888 -p 443:8080 ravendb/ravendb
I recommend exploring other parameters when running the RavenDB docker container. More information is available on this page.

Conclusion

In this post I demoed how easy it is to have RavenDB running on Docker on your machine. Ideal solution for those who just a simple instance to start playing with. Also notice how simple it was to run RavenDB on my Fedora Linux workstation.

Try it out. This is a fast, simple and quick way to test RavenDB on your machine.

References

See Also

Posted by
Labels: , , , , ,

Monday, June 4, 2018

Installing and Running RavenDB on Windows and Linux

Let's install and run RavenDB on Windows and Linux and learn how it works.

On a previous post  I introduced RavenDB on this blog. On this one let's review how to install or run a standalone RavenDB instance on our machine.

On this post we will cover:
  1. Installing and running on Windows;
  2. Installing and running on Linux;
  3. Using the RavenDB console tool;
  4. Creating a new database;

Downloading RavenDB

First off, navigate to the RavenDB downloads page and download the server version for the environment you're working on. The currently supported platforms are: Windows, Linux, OSX, Raspberry PI and Docker.

To download your image, select Server, Stable and the appropriate version for your environment. Aaccept the terms, click on the .ZIP Package download button to download the image to your disk.

Running standalone RavenDB on Windows

On Windows, once the download is completed, extract all those files in a folder and you'll see two PowerShell files: run.ps1 and setup-as-service.ps1.

Open the Powershell terminal, cd into the folder you extract your files and run .\run.ps1 You'll then see some outputs the RavenDB service is emitting for us when running as a standalone instance:
A new window will open for you where you'll need to configure a cluster and/or security. For now, let's skip the cluster configuration and go with the Unsecure option. 

This configuration is enough for this demo and simple development efforts. Clicking on it, RavenDB Studio will open on the default Url: http://127.0.0.1:8080/studio/index.html.
That's it! The standalone instance is running and you can start testing RavenDB on your Windows box.

Installing on RavenDB Windows

To install you your machine,  open the PowerShell terminal as an administrator and run the setup-as-service.ps1 script.

If all goes well, you'll  have to accept the user license agreement and proceed with the instance configuration.

Note that:
  • the installation requires administrator privileges
  • will use port 8080 for "unsecure" installs or 443 for secure (options selected during the installation)

Configuring the new Instance

After installed, you'll have to configure your instance as shows the image below. For a development setup, you should be good with the Unsecure option.
Clicking on it, will prompt you for Http/Tcp ports and IP address. Leaving empty will use the defaults. Click "Restart Server" and RavenDB should be installed.

The RavenDB Service

Once installed, on Windows, we can view the service status using the Get-Service Powershell cmdlet:
Get-Service -Name RavenDB

For more information, please visit: https://ravendb.net/docs/article-page/4.0/csharp/start/installation/setup-wizard

Running standalone RavenDB on Linux

The Linux installation is similar: download your Linux image from the RavenDB downloads page, unzip it and run the script. Let's see how it works.

Download the image by selecting Linux x64 from the downloads page and download it using Firefox:
Once downloaded, extract the bz2 pkg on a local folder:

Cd into that folder and run "run.sh". I should then see:

Installing RavenDB on Linux

Installing RavenDB on Linux is very similar to Windows. You run the run.sh shell script and select the installation option on the command line.

The RavenDB Console

After installed, basic manipulation of the server can be done either by using the Raven UI (Raven Studio) or by using the console previously opened.

For example, when I type help on my shell in Fedora, I get:
From the console, You can do things like restarting/shutting down the server, like exporting/importing data, reading logs and viewing server stats. Just type the commands shown. For example, to shut the instance down, I should type: shutdown.

Creating a Database

The final needed step before touching code is to create a database. For that do: Databases -> New Database:
Enter a DB name (for example, "Blog") and click "Create"):
Clicking on that Database creates our database and takes us to the its page. From there we basically can view our documents (records), create, query, patch, view logs, stats, etc:

Next Steps

Now that RavenDB is running and the database is created, the next step is to start interacting with it. You can use either RavenDB Studio or the client Api (with C#, Java, Python, etc). For more details, check my simple introduction to RavenDB.

Don't forget that RavenDB is also available on the cloud. Check the article An in depth review of the RavenDB Cloud for more information.

Conclusion

Hope this post shows how is a simple introduction on how to install RavenDB on Windows and Linux boxes. For more information, check the official documentation.
Posted by
Labels: , , , , , , , , , ,

About the Author

Bruno Hildenbrand