Intro

In the last post HowTo: Web Application with Spring-Boot, Kotlin and Mongodb, a short tutorial was provided to create a simple web application. We have written a small Spring-Boot application utilizing Spring-Data-MongoDB to read and write documents in MongoDB. In this post we will extend the simple example from the previous post by following:

  1. We will enrich the current document, which currently consists only of id and name, with further fields, to a more complex document
  2. We will provide a Frontend to present our documents
  3. We will make our application reactive

The following excursion, embedded or separate documents, provides a brief explanation of the advantages and disadvantages of embedded and separate documents.

Following this, we will update the documents and the corresponding repositories. In addition, we are going to create a lookup operation  using Custom-Repository.

The section Frontend briefly describes how we can use the Angular-CLI to develop an Angular 4 app for our user interface.

Finally, the last section, Becoming Reactive, briefly describes the advantages of reactive applications and which steps have to be taken to update our example application to a reactive application.

Excursion: embedded or separate documents

In document-oriented databases there are basically two approaches to model documents consisting of more complex structures:

  1. the documents will be stored as separate documents or
  2. all the information is embedded in a main document.

Embedded documents

The advantage of document-oriented databases is that you can forego many join operations. This is achieved by embedding additional documents in the main document. Another important advantage for embedded documents are atomic operations. This ensures, without additional application-side checks, that an operation is either successful as a whole or not.

The example below shows an example of an embedded document

In this document, the user John Doe has multiple addresses that are embedded in the main document. If another address is added, it will be appended to the address array.

This makes it possible to query the entire document using a simple query.

In addition, we can search for documents on basis of specific attributes, whereby we get here all the documents that match the attributes

The simpler queries, however, have the disadvantage that they lose their flexibility. So it is not possible to query only specific addresses and to receive a corresponding document. Instead, the main document is always selected and the irrelevant information has to be suppressed:

The size of the document also does matter. The maximum size of a MongoDB document can be 16 MB. When queried for documents, they are loaded as a whole into the Memory or transferred to the client. Depending on what information is stored (images, HTML, text), many data can be transferred or processed which are not relevant. For example, our fictitious user John Doe could be a successful real estate broker who has several hundred addresses that consist of both successfully passed and open offers. By now, it is not possible to select only a subset, for example all open offers.

Separate documents

As an alternative to embedded documents, it is also possible to separate the documents in different collections. This has the advantage that each document exists independently of another document and we can access it accordingly. This simplifies the queries against individual documents but has the disadvantage that one must link the documents to query the entire data set. The example from Listing 1 with separate documents:

In order to link the documents you can make a join using the $lookup function since MongoDB version 3.2:

The content of the fields (user_id) are checked for identity, and an array (addresses) with all found documents from the collection address is attached to the document with the id john-doe from the collection user. The resulting JSON is identical to the JSON in Listing 1.

Since the documents are saved separately, the developer is responsible to set the documents in relation. In order to avoid ambiguities and thus produce false-positive results, the value should be unique across the collection. The “email” field can be used. To define a field at MongoDB as unique, an index must be created

For the unique index you must consider the following:

  • If documents already exist in the collection with the corresponding field and duplicates, they must be resolved before a unique index can be created
  • If documents already exist in the collection but without the field on which the unique index is to be created, the field must be created for each document
  • The field is unique across the entire collection, not just in one document.

Another drawback of separate documents is that operations across multiple documents are not atomic. The developer must ensure that a consistent state is restored when an error or problem occurs.

Mashup

Of course, it is also possible to use a mixed form. We could create a document in which the user has a maximum number of embedded addresses and all other addresses are stored and referenced in a separate collection and a separate document. This would have the advantage that, on the one hand, the complete document with the relevant addresses can be queried and the additional addresses can only be reloaded as required. This reduces the number of irrelevant data which is loaded but not used.

Best Practice?

When modeling documents and structures, you should consider how the application will access the data and what data have to be displayed. While for relational data the model should be normalized (usually up to the third normal form) in order to avoid redundancies and duplicates, the document in a document-oriented database should be modeled as to how the data is accessed. The goal is to keep join operations low and to obtain all relevant data with one single query.

Let us stay with the example of the real estate agent John Doe: our main requirement for the application could be that we want to select houses that are offered for sale and which broker offers which houses.

As already described, we could store all the data in one document. We create the main document ‘user‘ and embed all related addresses. Now we can query for users and get a complete listing of all addresses and houses. So far so good. But we also get houses that have already been sold. If we want to sort the addresses or suppress all the addresses whose houses have been sold or do not meet our expectations (distance, price, other), we have to solve this via application-side logic (this is probably also possible via a map-reduce function or via an aggregate function, but we wanted to reduce the complexity with MongoDB and not to increase it by complex queries).

Even with these three requirements, searching, sorting and filtering, you can see that for this use case, it is not sufficient to save everything in one document. Instead, it would be more effective to separate the documents and load only the documents that match the query. In this case, the question arises under which criteria the documents are to be separated. We could group all the addresses of sold and unsold houses in separate documents. However, we need additional logic to update the documents if a house is sold or once a house is available again for sale. The presumably best solution would be to treat each address as a stand-alone document and to refer to it via some key fields.

We already see that we are moving towards a relational model, and a document-based database may not be the right solution for these requirements.

So rather stay relational?

Let us now take a look at our configurations (see Post 1 and Post 2). Our configuration consists of a name, a description, an overview image and the configuration as an XML file. In addition, each configuration can be rated and commented on.

The main task of our application is to provide configuration and maybe, view comments. Creating or updating existing configurations is rather the exception. Since the comments are output in the order in which they were created, both sorting and filtering are superfluous. Also the deleting or updating of existing comments will be an exception.

As a schema it therefore makes sense to create a document for each configuration, in which all comments are stored. The advantage of this is that comments are only valid for one configuration and are deleted when the configuration will be deleted. Because of the rare write operations and simple queries, the separation of documents would not bring any advantage.

Conclusion

Unfortunately, there is no general recommendation for creating a reasonable schema. When modeling, the developers should be aware of the applications requirements and create their own documents accordingly. For an application that requires frequent write operations and whose data is going to be frequently updated, it may be more sensible to separate the documents and make them retrievable individually. Even if the data must be aggregated differently and no parent document can be identified, it is possible and may make more sense to separate documents. However, if you have dependent data, such as blog page with ten to twenty blog posts, where each page can be represented as a separate document, it may make sense to summarize this data to embedded documents.

Extend Documents

In the first step, we look at the configurations requirements and then decide how to model them most adeptly. Since this is merely an example, we limit ourselves to the following three requirements:

  1. Our example application is intended to provide an overview of all the configurations already created.
  2. On the details page, the user should see the detail information of the configuration, such as, description, creator’s name, number of ratings, download the file, and display the reviews including comment and author.
  3. The user has the possibility to create his own configuration via a simple input form.
    Based on this, there are two possibilities, as our configuration documents can look. We could embed all comments into the configuration document. Any newly added comment would be written to position 0 (first digit in an array).

Picture 1 – Insert new document in embedded Rating Array

This would have the disadvantage that the entire document including the comments will be already loaded on the overview page. For documents with a small number of comments, this would not be a problem because the size of the document will be comparatively small. However, since the comments can grow as desired, we could get performance problems in the future. herefore, we want to separate the comments and the configurations. We store a document for each configuration and a document for all comments to this configuration. The documents to be saved are as follows:

Picture 2 – Insert document in embedded Rating array and reference parent document with User document

Once we have decided how to store the data, we need to make sure which attributes we want to store.

The following properties are defined for our configuration:

  • Each configuration should have a unique name
  • Each configuration should have an overview image
  • Each configuration should have a description
  • Each configuration contains an XML file
  • Each configuration is created by a user

And for a comment, the following properties apply:

  • Each comment belongs to a configuration
  • Each comment is created by a user
  • Each comment consists of a rating and a description

Next, let’s add a little structure to our example. In the last tutorial we wrote everything in the main class. As our application is growing, we should separate the classes, to provide a clear overview within the project. In the future, the individual classes can be found in the following packages

Picture 3 – Package overview

 

  • mvc.rest.controller contains all controller
  • data.document contains all documents-classes
  • data.repository contains all repositiory

 

After the packages are created and the classes have been moved accordingly, we take care of the ConfigurationDocument.

We change the name of the document from “test” to “configuration” and expand the document by the appropriate fields. We have some fields that we do not want to change (Kotlin val is an equivalent to Java’s final), including the Id, the name and the author. For the files we use a string instead of a byte array, since we encode the values as base64-string.

Next, we will look at the comment document:

Since we have a comment document for each configuration document, we enter the individual ratings into the main-rating document. For the reference, we use the configurationId field where the Id of the configuration document to which the comments are to be assigned is stored.

Repository

Since we now have an additional document which we want to access, we must create a corresponding repository interface. This is done in the same way as the ConfigurationDocumentRepository (https://blog.novatec-gmbh.de/how-to-web-application-with-spring-boot-kotlin-and-mongodb/#spring-data-rest). In addition, we create a method to search for documents belonging to a specific configuration.

Unlike the previous tutorial, we do not rely on our own methods to work. Instead, we are writing unit tests that we can use to ensure this. The easiest way is to create a separate database for the test cases, where we can then usually create the same documents as on the real database. MongoDB creates a database as soon as we access a Database, which does not exist. Also MongoDB will create a collection, if it tries to access a document in a collection, which might not exist. With Spring-Boot and Spring-Data we can rely on Spring-Date to create a MongoDB connection and Database specified in the application.properties.

Also we can create an application.properties file in the folder test/resources, which gives us access to the same properties as in a normal application.properties, but limited to the “test” environment. You can then set the database using the key spring.data.mongodb.database

Picture 4 – Package overview

We then create our first unit test, which checks whether the findAllByName method returns the expected result.

A brief explanation of the code:

  • Line 8-15: In order for the tests to be reproducible and not corrupted by existing or incorrect data, we delete all documents in the collection and then create a document before each test.
  • Line 18-21: This is our default document that is created.
  • Line 23-32: This is the actual test. As a passing parameter, our method expects a name, as well as a Pageable object, to control how many documents we receive. Then we search for all documents called shazaam and get the first element from the result set. We then compare the values from the expected and the actual objects.

In the same way, we also create a test-case for RatingDocumentRepository.

Aggregation function (blocking only)

In our example application, we separated the comments and the configurations. This is because we will display many configurations on the overview page and do not want to load unnecessary data. However, we want to display the comments on the details page of a configuration. We now have the option to first load the configuration using the Id and then use the same configurationId to load the comments. Alternatively, we can write an aggregation function that loads the configuration with embedded comments. With Spring Data, it is possible to write your own functions, or overwrite existing repository functions.

Similar to Spring-Data’s own query functions, it is also important for extensions that the naming conventions expected by Spring-Data are respected. This means that the interface has to be named identically to the repository name and have to contain the suffix ‘Custom’. The class name must also be identical to the name of the repository and the name have to contain the post-fix ‘Impl’. This makes it possible for Spring-Data to recognize the class automatically. In order to use the function you have to name the functions like find…By, get…By, read…By, query…By or count…By.

Our aggregate function is going to combine two documents. To do this, we use the aggregation function ‘lookup‘ similar to our example from Separate documents. The rating comments will then be embedded in the configuration document. We therefore extend the ConfigurationDocumentRepository and create the Interface ConfigurationDocumentRepositoryCustom. We then implement the interface in the class ConfigurationDocumentRepositoryImpl.

Then we can provide our own aggregate function like this:

Now that we get a document with more embedded documents as a result, we need a class that represents this result. To do this, we create a helper class named ConfigurationWithRatings, whose only task is to represent the aggregated document.

Finally, we are updating our test-case by a short unit test to ensure that our function also delivers the expected result.

Frontend

First of all: to make this tutorial not too big, I will only describe the most important functions. The entire source code of the Frontend can be found in the repository at https://github.com/nkolytschew/blog_post_examples/tree/master/frontend. The source code of the Frontend is for demonstration purposes and  shows how a simple Angular-Project can be created quickly and with little effort. Please keep in mind, that many good practices and style guides are being violated.

We take an Angular for this example because we can develop a Frontend with little effort. To create an app, we use Angular-CLI. In order to use Angular-CLI, we must first install the current version of NodeJS and NPM (https://nodejs.org/en/download/current/). Then, you can install Angular-CLI using the command line (https://github.com/angular/angular-cli#installation).

With Angular-CLI specific commands, we

Picture 5 – Angular App – Project Structure

can now conveniently create commands, services, classes, or the entire project using the console/command promt.It usually does not matter where the Angular project is created. However, I recommend that you create a folder (frontend, for example) in our example main-project, in which the Angular app will be initialized.

Using the command

we will create a minimal application called simple-app in the directory where we are currently located.
After the application is created, we start it with the command

Then we can access the app via the URL http://localhost:4200. If everything was successful you get an overview page with the Angular icon and the message “Welcome to app!!”

Our sample application will have

  • an overview showing the configurations,
  • a form in which new configurations are created,
  • a form in which the existing configurations can be edited and
  • a detailed view in which the ratings and comments are displayed in addition to the configuration details.

To keep the Frontend as simple as possible, we are going to create all functions in one single page.

Angular treats a component as a page with a directive, which contains the source code an the page logic, an HTML template and Styling-Rules. To create your first page, use the Angular-CLI command

In the src/app directory this command will create a new directory with the name of the component. In addition this command also creates

  • an HTML file for the template,
  • a CSS file for the styling rules,
  • a spec.ts file for unit test and
  • a TypeScript file for application logic

Finally the component will be added to the NgModule class (more information here: https://angular.io/guide/bootstrapping).

Configuration Overview

For the overview page, we can use and overwrite the existing Template within app.component.html and application logic within the app.component.ts files. In the first step, we only want to display the configurations that we get from our Spring Data REST repository as JSON. To do this, we need to execute a GET request to our ConfigurationDocumentRepository (URL for this example: http://localhost:8080/configurations).

Note: to make sure the requests to the repository will be successful, we must add an @CrossOrigin (value = “http://localhost:4200”) annotation. Otherwise, all requests from the Frontend to the Backend will be blocked.

As shown in Listing 3, we use the HTTP module provided by Angular to submit request against our REST-Repository. In the getConfigList() method, we issue a GET request and receive an observable (http://reactivex.io/documentation/observable.html).

Observables are used to observe something (for example, functions, objects, methods) and to react to changes. Similar to Java Stream, nothing happens as long as there is no terminal operation. Thus it is possible for us to execute many different queries “parallel“. For the Observable to perform the operations, we must subscribe to the Observer instance using the subscribe() function. This will inform “us” about changes. With the subscribe() function, three functions can be specified as parameters

  1. onNext – treats the success and contains the data
  2. onError – handles the error and contains the Error object
  3. onComplete – treats completion after next or error has been completed

The return value of all HTTP functions is an observable, which allows the requests to start independently from each other and to process their tasks. In addition, Observables offer the possibility to cancel their request. This is particularly important in asynchronous environments where the result of a function or object is known at a later time. In a synchronous environment, the application would wait until a function fulfills its task, and then continues processing.
If we have successfully subscribed to an observable, we can process the response in the onNext() function. In our case, we get a JSON, which we can assign directly to a variable and then process it in the HTML template.

The JSON we get from Spring-Data-REST looks as follows:

To access the individual fields of the JSON, we must of course know what they are called. In our case this is easy because it is our object and we know how the JSON should look like. Of course, there is also a generic or programmatic way, where the unknown JSON can be traversed and you can display the values and assign them to your own fields. Typically you iterate over each key-field of the JSON and print or assign the key and value to your variables:

Add configuration

Picture 6 – Overview and Add Configuration Page – Screenshot

For the next step, we define a form with to can create new configuration. To insert this configuration into the Database using Spring Data REST, we must perform a POST-Request with a JSON, that has the essential configuration attributes. In our example, we want to upload the name, a description, an author, a thumbnail, and then the actual configuration XML-file and save it in our database

Extract of the HTML template

In the form tag, we define the name of our form (configForm) and which function has to be called at submit (saveConfig). Then we assign the values of the input fields to the corresponding attributes of the form (#configName=”ngModel”).

In the saveConfig() function, a JSON will be generated from the form and a POST is sent to our Backend application.
The POST function looks as follows:

and the corresponding JSON body from the POST:

Becoming Reactive

Now that we have a web application that allows us to view, create, and delete configurations and comments, we will try to make them reactive in the last step.

Currently, we have a small application whose Frontend works asynchronously and is able to respond to events, but our Backend is still synchronized/blocking and requires the processing of the requests one after the other. A reactive application, on the other hand, is capable of processing multiple requests at the same time. In simple terms reactive means that you can deal with asynchronous data streams.In our small example application this does not matter, but it can lead to bottlenecks in more complex enterprise applications, since threads and resources can be blocked while waiting for the result of another function.

How can an application become reactive?

A typical imperative application, defines a sequence of statements, and the order in which these statements are executed. The result is then waited for.

Figure 6 shows how the typical workflow of an imperative code would look.

Figure 7 – Simple blocking workflow

In the main function, three sub-functions are started, each of which calculates or reads something from a database. Each subsequent function waits for a result from the previous function to be available.

How would this look in the reactive model? Instead of a sequence of functions, we have streams. The result of a transformation (map, scan, filter, flatMap, etc.) is another stream, etc. A stream represents a sequence of events.

Figure 8 – Reactive Streams workflow

Each event can contain either a value, an error or a “terminated” signal. To get to the content of an event, we write functions that are executed

  • as soon as a value is available,
  • if an error occurs
  • or as soon as we receive the “terminated” signal

Then we need an observer to subscribe to the stream and trigger the actual processing.
The reactive programming model is particularly suitable for web applications, which enables many user interactions.

Updating our Web application

Having briefly looked at the theory, we want to make our application reactive. Since Spring supports the reactive programming model only from version 5.0, we must replace the existing dependencies with reactive dependencies. This includes, for example, Spring-Data-MongoDB for the asynchronous data access and Spring-WebFlux for the reactive HTTP and WebSocket client.

In our example, we will modify the build.gradle file and increase the version of SpringBoot and change the dependencies.

After we have updated the build file, we have to adjust the repositories. Instead of MongoRepository, we now inherit from ReactiveMongoRepository. This changes the return types of List, Collection, or Entity-Type to Flux, or Mono. Flux and Mono are reactive data types that can contain a sequence of 0..n, or 0..1 element. In general, they offer similar functions as the Stream API to process, transform and merge these sequences with other sequences.

Unit tests

After we have modified the respositories and inherited from the ReactiveMongoRepository class, the data from the database will be already returned as an asynchronous stream. As a result we have to adapt our test cases because otherwise we get “Unresolved reference …” errors. This is because Flux and Mono, as a distributor, only describe the asynchronous process without actually processing the data. We have to subscribe to the distributor, so that it can start the processing and at some point we get the data or an error. But since we are in a test case, we do not want to wait until the data is available, instead we can block the processing until the process is finished, so we can get the data directly and complete the test case. For the blocking we are going to use the following functions

  • block () – for mono; Blocks until a value is available
  • blockFirst () – at Flux; Until we get the first item
  • blockLast () – at Flux; Until we get the last element
  • toIterable () – at Flux; Changes to a synchronous list
  • toStream () – at Flux; Changes to a synchronous Java stream

Controller

After updating the repositories and the test cases, we are going to create controller to pass the data to the front end. Unlike the rest repository, we are now personally responsible to provide the functions we need. For the ConfigurationController, this means that we need the following functions:

  • Read all configurations
  • Read a configuration using an id
  • Save a configuration
  • Update a configuration
  • Delete a configuration using an id

The controller is a simple RestController, which transfers the database response as JSON to the Frontend.

Similar to the rest repository, we add an @CrossOrigin annotation so that the requests from the Frontend to the Backend are not blocked, and an @RequestMapping annotation so that the request URL remains the same and we do not have to adjust the URL in the Frontend.

Creating the RatingCommentController we proceed similarly: we create the controller with the methods we need and add the @CrossOrigin and @RequestMapping annotations.

Finally, we need to change the configurationListJson object in the Frontend HTML template. While we were embedded the result data in the fields “embedded” and “configurationDocuments” from the JSON result of Spring-Data-REST, we now can directly access the data in the JSON result of the RestController

Difference between a Spring Data REST JSON response and RestController JSON response

This means that we no longer have to navigate through “._embedded.configurationDocuments” in the template in order to get the content and therefore we can remove it:

Summary

What have we done?

First we expanded our small application and extended our simple documents to more complex documents. Subsequently, we have provided an user interface for the user instead of printing simple JSON. In the UI, we have already made sure that the synchronous requests to the server are processed asynchronously. Afterwards, we also designed the Backend asynchronously, more specifically reactive.

And why did we do that?

Admittedly, in our small application, we have hardly gained any advantages at first sight. We even had a little extra effort because we had to write our own controllers. However, we are now able to cancel requests that take too “long“. In addition, we are also able to execute different requests in parallel or send our request to another service if the first does not work or take to long, without blocking the further processing.

The (functional) reactive programming gives us many possibilities to react independently to events rather than to process the order of functions as before and to wait until the previous function is completed.

Sources

GitHub Repository

  1. Simple Angular Frontend
  2. Reactive MongoDB, Spring-WebFlux with Kotlin
  3. Reactive MongoDB, Spring-WebFlux with Java 8

Leave a Comment

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close