Showing posts with label QA. Show all posts
Showing posts with label QA. Show all posts

Thursday, April 1, 2021

Building and testing Node.js apps with Azure DevOps Pipelines

Learn how to built, test and run code coverage analysis on a Node.js app using Azure DevOps Pipelines
Photo by Chirstopher Gower on Unsplash

Setting up an Azure DevOps pipeline is not too complicated. On this tutorial, let's learn how to setup a simple pipeline to a basic Node.js app so we can build, test and capture test results and report code coverage analysis, exposing them to everyone in the team.

On this tutorial we will:

  • Create a new DevOps Pipeline
  • Build our project
  • Configure DevOps to read the outputs of our tests and test coverage

Sample Project

This tutorial can be executed using this repository. 

Setting up a Connection

To get started, we have to grant Azure DevOps access to your external repo, GitHub on this case. To configure a new connection, click on Project Settings on the bottom left in your navigation bar:

Then click on Boards -> GitHub connections:

Now, click on New Connection:

And click Connect your GitHub account:

Finish the integration by entering your username/password and authorizing access to your GitHub account and repositories.

Creating a new DevOps Pipeline

Now that your Azure DevOps has access to your GitHub repos, let's build our first pipeline. Click on Pipelines -> New Pipeline on top right to create a new one:

Choose a source repository

Next, select where your project is located. Since we're using GitHub for this demo, click on it:

And choose, one of your existing repositories:

Configuring the Pipeline

Next, DevOps will ask you to specify your project type. Since we're building a Node.js project, choose it from that list:

Click Next and you should see a default build yaml for your new project that should look more a less like this:

  vmImage: 'ubuntu-latest'

- task: NodeTool@0
    versionSpec: '10.x'
  displayName: 'Install Node.js'
- script: |
    npm install
    npm run build
  displayName: 'npm install and build'

Running our first build

If all your dependencies can be satisfied by npm, most likely your project can be built so click Save and run so that DevOps builds your project for the first time:

By clicking on the columns on the left, we can even inspect the result of the build. For example, here's the output of npm install and build:

Coming back to out pipeline page, we get our first successful build:

Step 4 - Setup Unit Test reporting

But as good as the initial template is, it does not auto-enable unit-testing and code coverage reporting, so let's set it up. To change the pipeline in the future, click on the Dots menu -> Edit Pipeline:

You'll be redirected to the editor. On the Tasks menu on the right, type test results and click on Publish Test Results to add it as a new task in our azure-pipelines.yml file:

Next screen, select JUnit, follow the configuration below and click Add:

Step 5 - Setup Code Coverage reporting

Similarly, type code coverage on the Tasks menu on the right, to add the code coverage task:

And set:

  • Coverage tool: Cobertura
  • Summary file: coverage/cobertura-coverage.xml

Review and Save

The above steps should have added something similar to the below on your azure-pipelines.yml:

- task: PublishTestResults@2
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/TEST-*.xml'
- task: PublishCodeCoverageResults@1
    codeCoverageTool: 'Cobertura'
    summaryFileLocation: 'coverage/cobertura-coverage.xml'
The menus and buttons we clicked are nothing more than helpers to add the above code. Feel free to explore the editor and properties for each field since it contains intellisense.

Click Save and Run again to produce a new build. Once your build is finished, it should show up in the pipeline's main screen:

Reviewing Test Results and Test Coverage

To wrap it up, let's review our test results/coverage reports. Clicking on the second build should take you to the build page:

Highlighted in the above image, both are links that take us to the tests page so we can review test results:

Cool, isn't it? Clicking on the cover tests link above, we're taken to the Code Coverage page where we're able to see the code coverage in our project:


On this article we reviewed how to setup a simple Azure DevOps pipeline to build, test and report code coverage for a basic NodeJS app exposing them in the build. Azure DevOps is a pretty powerful CI/CD system but unfortunately, often ignored by the community. Feel free to explore it!

Source Code

As always, the source code for this post is available on my GitHub.


See Also

Monday, October 2, 2017

Interview questions for QA analysts

Hiring a QA engineer? Here are some ideas.

We were asked to provide feedback on what I would like to see in a potential QA analyst. Apart from the traditional and essential HR screening (which they know way better than us) this post is about what I would like to see in a QA engineer working in my team.

Conceptual / Basic Questions

Before everything, we're interested in knowing if the person has solid understanding on why that work is being done, why it's important and how to get good results out of it. So, we would like to know, for example:
  • In your understanding, what are the benefits of QA?
  • In your opinion, what is the role of a QA analyst in a project development?
  • What is severity and priority of a bug? Give examples;
  • Define bug triage;
  • Expect some sort of calculation using severity, number of tests, dependent tasks, etc;
  • How to Estimate Testing effort;

Define why do we test software?

Expect the answer to include thinks like:
  • the process of assuring that the product being developed is meeting all requirements.
  • the reason to perform testing is to find bugs and make sure that they get fixed.

Why do we do QA?

Expect the answer to include thinks like:
  • to find the bugs before the product is released to customer.
  • to improve the quality of the product
  • to evaluate that the product is according to requirements

When is it the best moment to start QA in a project?

Expect the answer to include thinks like:
  • A good time to start the QA is from the beginning of the project startup.
  • This will lead to plan the process which will make sure that product coming out meets the customer quality expectation.
  • QA also plays a major role in the communication between teams. It gives time to step up the testing environment.
  • The testing phase starts after the test plans are written, reviewed and approved.

What are the key challenges of software testing?

Expect the answer to include thinks like:
  • Application should be stable enough to be tested.
  • Testing always under time constraint
  • Understanding requirements, Domain knowledge and business user perspective understanding
  • Which tests to execute first?
  • Testing the Complete Application
  • Regression testing
  • Lack of skilled testers.
  • Changing requirements
  • Lack of resources, tools and training

Define a testing lifecycle?

There is no standard testing life cycle, but I like to have following phases and would expect some (if not all) of them being mentioned:
  • Test Planning (Test Strategy, Test Plan, Test Bed Creation)
  • Test Development (Test Procedures, Test Scenarios, Test Cases)
  • Test Execution
  • Result Analysis (compare Expected to Actual results)
  • Defect Tracking
  • Reporting

List different types of tests

I expect at least 5 of the following:
  • Manual testing
  • Smoke testing
  • Regression testing
  • Automated testing
  • Stress Test
  • Load Test
  • Performance Test
  • Exploratory Testing

Programming Skills

For me, developers (as sysadmins/devops engineers), would do their job way more efficiently if they knew how to programm. If she knows how to program, then I would ask:
  • to write a simple function that would, for example, write a string backwards;
  • if the person knows html, css? Exercise that a little;
  • does the person know how to hit a restful api to use browserstack, for example;
  • does the person know javascript;
  • does the person know some other scriptting language so he/she would be able to write some automation;

SQL / NoSQL Questions

I would also very interested in knowing if the person knows SQL. Not necessarily super well but well enough to deal with basic sql instructions and be independent. Also, in case you use NoSQl, ability to query your db in your query language like for example, MongoDb's query syntax or Lucene.

Software Automation Questions

In order to maximize the return of a QA in the team, test automation is a must. That said, the person needs to have enough programming skills write reproducible tests using selenium for example. Personally, I would like to know if the person: 
  • thinks automation is important?
  • knows when to automate?
  • knows what are the benefits of automating?
  • knows what are the downsides of automating?
  • have written automated tests?
  • have used selenium?
Having already working selenium is a big advantage for me.

Infrastructure Questions

Expect the answer to include thinks like:
  • What's the difference between a build and a release?
  • Yes, I have used Azure or AWS before;
  • Yes, I know how to use ftp;
  • Yes, I know how to you check if a server is up;
  • have previously used git, github?
  • have previously you/written a wiki?
  • have previously used Firebug / Chrome Dev tools 
  • is able to inspect, communicate js/html issues;

Board/Offline Exercise

While I don't think we should always use the board, it would be interesting to use the board to check how the person would improvise on two scenarios:

Scenario 1 - Write a test document based on a real use case

In this scenario, I would present a simple use case such as how to create an account on my system (please, use a simple enough for your own system):
  1. Navigate to the landing page of my site;
  2. Click on the Register button;
  3. Enter essential information to create account;
Then, ask user to:
  • List what he/she would have validated?
  • What boundary tests could be written?
  • Ask user to write test matrix;

Scenario 2 - Write a bug report

Still on the create account use case, let's assume that the system should email the new user a confirmation token but it didn't. I then would be interested in:
  • How would the person write a report in case the email didn't reach your inbox? 
  • If the person would document simple stuff as:
    • Date of the test, url, browser version, etc;
    • Steps:
      • clicked on button X;
      • Entered the following info...;
      • Clicked submit;
    • Expected result;
    • Actual result;
That is what I, as a developer, would like to see in a basic bug reported: enough information to reproduce and fix the test. Less than that may be insufficient to reproduce it meaning the probably   the candidate is not a good fit.


I consider the following, differential to have:
  • Selenium/Automation experience;
  • Cloud experience;
  • Being able to discuss technical details with the developer
  • Linux/MacOS experience;


So, this is all that came to my mind. There's probably more to it but I think that the list above covers a good, very strong candidate. Note that this is by no means a checklist but how I describe a very strong QA candidate and I would be very happy to have someone with the above skills in my team.

About the Author

Bruno Hildenbrand      
Principal Architect, HildenCo Solutions.