Aura – continuous-integration

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Basic Jenkins configuration

Discover the basic Jenkins configuration

Introduction

Aura uses Jenkins as continuous integration (CI) tool. In particular, the in-house installation in DCIP.

Webhook	URL	Content type
Pull Request	https://ci.tid.es/pro-dcip-aura-01/ghprbhook/	`application/x-www-from-urlencoded`
Push	https://ci.tid.es/pro-dcip-aura-01/github-webhook	`application/x-www-from-urlencoded`

Configuration

Configure a webhook in a github repository

Prerequirement: administrative access to Github repository and to dcip are must.

In the Github web of your repository, go to Settings and to Webhooks, as can be seen in the following picture:
Click on Add webhook, the following window will be shown:
- Fill the URL of the webhook and its content type format
- Choose Let me select indivial events and select only Pull requests. Remember to uncheck Pushes that’s checkec by default.
- Set the webhook as Active and click on the Add webhook button to save the configuration.

Jenkins jobs

ℹ️ Prerequirement: administrative access to dcip is a must.

First thing is to create a new set of Jenkins jobs for your brand new respository, so from the main aura Jenkins page, you should run _component-setup job.
Choose Build with parameters and fill the data of your repository in this case aura-bot-platform. Remember to use ssh as repository URL.

This job will generate a new set of jobs specific for the current repository and with its very same name.

Generate a dedicated slave to run these jobs

Add a Dockerfile to the root of your repository (follow the one in aura-bot-platform). In this case, a new slave definition is needed to run the jobs with a node10 docker container.

To build it, just build the job _docker-slave-setup with parameters:

This will generate automatically the job _docker-slave-builder that will process, configure and publish a docker image based on the provided Dockerfile. Slaves using this image will be available under the label repository-name, in this case: aura-bot-platform.

Build PR job

This is one of the automatically jobs generated by the _component-setup job. It will run whenever a Pull Request is requested in the repository, if it counts on the corresponding webhook.

The basic configuration provided automatically is OK, but in Aura we should apply a couple of changes to handle it properly:

Check Restrict where this project can be run and write the just created aura-bot-platform labeled slaves.
To access github, select the already available credentials (usually contint (contint user for GITHub/PDIHub)).
A couple of environment variables should be configured:
- NPM_TOKEN: it should be added in “Build Environment” section, binding a secret text, with the specific credential.
- DOCKER_HOST: it should be added in “Build Environment” section, binding a secret text, with the specific credential.
The next step is to include execution of the existing delivery/pipelines/pr.sh script during the building phase of the job.
- In Build section, just add an Execute shell step with the following content, depending on the project, further variables should be needed:
```
#!/bin/bash
set -e
export DOCKER_HOST=${DOCKER_HOST_VM}
git remote set-url origin git@github.com:Telefonica/aura-bot-platform.git
delivery/pipelines/pr.sh
```

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Github as orchestrator

How to use a Github repository to orchestrate the Release Train Manager processes

Release Train Manager Repository

To organize the work, we use a Github repository. This repository contains classic branches for the RTM development cycle itself and a set of branches reserved for RTM orchestration. The default base branch of the orchestrator is master.

The classic branches contain the prefixes:

feat/description
fix/description
doc/description
etc.

The specific branches for the orchestrator contain a descriptive name and the JIRA identifier:

main/name_reference#id: To orchestrate new features.
release/name_reference#id: To orchestrate releases.
hotfix/name_reference#id: To orchestrate hotfix on a release.
fasttrack/name_reference#id: To orchestrate features on a release.

To manage the RTM processes, we use Pull Requests (PR) in this repository. Each of these PRs has a base branch and a target branch, depending on the operation to be performed:

- main       -> Base: master. Target: main/name_reference#id
- release    -> Base: master. Target: release/name_reference#id
- hotfix     -> Base: a release branch. Target: hotfix/name_reference#id
- fasttrack  -> Base: a release branch. Target: fasttrack/name_reference#id

This PR is generated automatically when a commit is made for the first time in one of these special branches.

Pull Request Body

In the body of the generated PR, all the configuration information for the branch is displayed and the Pull Request will contain all the history of processes executed on it in comment mode.

An example of PR body of a main branch

The information for the PR body is obtained from a configuration file and is updated as the different phases of the orchestrator are executed.

The Github Repository Structure

/
├─ .github/
│  ├─ PULL_REQUEST_TEMPLATE/
│  ├─ actions/
│  ├─ workflows/
├─ docs/
├─ resources/
│  ├─ templates/
├─ src/
│  ├─ tools/
│  │  ├─ build-docker/
│  │  ├─ build-actions/
│  ├─ toolkit/
│  │  ├─ nodejs/
│  │  │  ├─ package-manager/
│  │  │  │  ├─ operations/
├─ branch-config.yml

branch-config.yml

This file contains all the configuration associated with a special branch. In this file, the involved equipment, phases, steps, etc. are configured.

All the information is included in Branch Configuration section.

.github/actions

This folder contains Github Actions necessary to perform all operations on repositories, data persistence and communications with Jenkins, among others.

resource/templates

This folder contains templates for the branch-config.hml file for each branch type:

_main_      -> `main-branch-config.yml`
_hotfix_    -> `hotfix-branch-config.yml`
_release_   -> `release-branch-config.yml`
_fasttrack_ -> `fasttrack-branch-config.yml`

src/tools

It contains the tools needed to build the docker actions and images.

src/toolkit

It contains the code SDKs as a complement to the Github actions. These utilities will eventually become actions someday.

src/toolkit/nodejs

It contains the NodeJS toolkit. More information in [NodeJS Toolkit].
⚠️ NodeJS toolkit section WIP

.github/workflows

We define stage as a set of steps to be performed in a coordinated manner by each team. Until all the teams involved in the phase have completed all the steps of a phase, it will not be possible to move on to the next phase.

For example, for example, we can define the INIT phase in which the teams have to create the specific branches in their repositories and activate an environment where they can deploy. We would have that each team would have an INIT phase in its configuration, with a step to generate the branch or branches in its repositories, and the Devops team would also have another set of steps to prepare the environment.

All these configurations are explained in the Branch Configuration section.

In order to synchronize the work between the teams and perform these operations we will use Github Workflows. These workflows are explained in the [Workflows] section.
⚠️ Workflows section WIP

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Jenkins pipelines definition

Understand how the Continuous Integration system works in Aura system in detail

Introduction

During the refactor phase, the way to declare the configuration of the different repositories has been redefined by trying to simplify the delivery/pipelines/Jenkinsfile file statement.

The main objective of the refactor is “to simplify”. This is attempted to achieve as follows:

All repositories must execute the same stages (predefined).
Repositories with a specific language should execute the same tasks in stages.
Scripts to execute tasks should be common and defined in jenkins-libraries-aura repository to be reused.

General structure project

The files associated with the continuous integration system are found in the delivery folder, which has the following structure:

delivery/
├── docker                          # Folder to Dockerfile
├── pipelines/Jenkinsfile.aura      # Jenkinsfile
└── scripts                         # Utility script for pipelines

Scripts added to the delivery/scripts folder should be specific tasks for this repository. Otherwise, the scripts should be in the resources/org/aura/script folder and that can be reused.

Add script to delivery/scripts only if it is completely necessary.

Jenkinsfile example

@Library(['aura']) _

auraPipeline {
    language = 'node'
}

The previous definition is sufficient for a nodeJS type repository to execute the default continuous integration tasks defined for an Aura nodejs project.

Definition of pipelines

Common libraries

Although it is possible to directly define the tasks to be executed with groovy here, there are a number of predefined scripts to run common tasks that can be used by including the aura and devops libraries:

@Library(['aura','devops']) _

Pipeline configuration

auraPipeline {
    slave,          // Slave where pipeline will be executed. Default: 'aura-bot-platform-14'
    nature,         // Pipeline nature: Default: 'standard' (the refactor nature)
    language,       // Language used by the pipeline
    artifacts,      // List of artifacts that need to be generated.
    stages,         // List of stages. Default: default stages
    modifiedStages, // List of modified stages.
    options,        // Pipeline options.
    postJobs        // Post execution jobs.
}

slave property

Slave property indicates the slave where Jenkins stages will be executed.

If a slave is not specified explicitly, aura-bot-platform-14 will be finally used.

nature property

The nature property indicates the definition of pipeline that will be executed on the list defined in ‘jenkins-libraries-aura’ project.

The simplification in the definition of pipelines has reduced the list to a single generic pipeline defined as standard.

It is still possible to execute existing pipelines, previous to refactor. Simply, the value of nature should indicate the pipeline to be executed.

This is deprecated in favor of the new standard pipeline.

language property

The language property helps the standard pipeline to execute the appropriate tasks depending on the language of the repository itself.

The system is prepared to handle repositories whose source code is written in a single main language.

artifacts property

List of artifacts that will be generated.

stages property

Although the standard pipeline defines a series of default stages depending on the type of ‘push’ that is done on the repository, this field allows you to define the stages manually.

modifiedStages property

In some cases, it is necessary to modify the configuration of some default stage in standard pipeline, without defining all the stages again (as we would do on the stages property).

For example, we can modify the test stage for the ‘pr’ type:

modifiedStages = [
    'pr': [
        new Stage(name: 'test', description: 'Test', options: [ skipLint: true] )
    ]
]

or modify the test stages for all types (pr, release, etc):

modifiedStages = [
    '*': [
        new Stage(name: 'test', description: 'Test', options: [ skipLint: true] )
    ]
]

options property

The following options are available:

Option	Description	type	Default
deleteBuildOnSkip	Delete build on skip	boolean	true
uploadGeneratedArtifacts	Upload generated artifacts from ‘artifacts’ directory	boolean	false

postJobs property

Jenkins jobs that will be executed once the current job ends.

Defining artifacts

The artifacts are components that we can generate during the execution of the pipeline. All artifacts must extend from the Artifact class and depending on its type will generate one component or another.

Currently, there are only two types of components (although it can grow in the future): AuraComponent and AuraLibrary.

AuraComponent

A type component AuraComponent is used to generate a Docker image with the following configuration options:

Option	Description	type	Default
name	Component name	String
version	Version for the docker image	String
envVersionVar	Environment version variable (as `AURA_BOT_VERSION`)	String
scripts	List of names of configuration scripts	List
initScripts	List of names of initialization scripts	List
buildArguments	Docker build arguments	List
dockerImageName	Docker image name	String
dockerFile	Docker file name	String	‘Dockerfile’
dockerPromotionTag	Docker promotion tag	String	’latest’

Example:

    artifacts = [
        new AuraComponent(
            name: "aura-bot", 
            version: "\$BOT_UC_VERSION", 
            dockerImageName: "aura/aura-bot-uc", 
            dockerFile: "Dockerfile", 
            scripts: ["librariesListFile.sh", "cfBotPlatformVersion.sh"], 
            buildArguments: ["BOT_BASE_VERSION=\$BOT_BASE_VERSION", "UC_VERSION=\$UC_VERSION", "UC_REVISION=\$PRODUCT_REVISION"]),
    ]

AuraLibrary

A type component AuraLibrary is used in repositories containing several libraries (multi-repository). In this way, it is possible to execute the pipeline for each of the libraries defined in the list of components.

If a multi-repository follows the structure of adding each library within the “packages” directory, the stages in standard pipeline are able to automatically get the library list, so in most cases it is not necessary to define configuration by hand.

In case you need to define the library list, the following options are available:

Option	Description	type	Default
name	Component name	String
path	Component path (Ex. packages/my-library)	String
versionStrategy	Version strategy: distag, old	String	distag

Example:

    artifacts = [
        new AuraLibrary( name: "aura-json-schema-generator", path: "packages/aura-json-schema-generator")
    ]

How to define my own stages?

The standard pipeline defines the following stages by default:

// Default stages
'default': [
    'pr'     : [
        new Stage(name: 'decrypt', description: 'Decrypt'),
        new Stage(name: 'initialization', description: 'Initialization'),
        new Stage(name: 'build', description: 'Build'),
        new Stage(name: 'test', description: 'Test')
    ],
    'release': [
        new Stage(name: 'checkSkip', description: 'Checkout skip'),
        new Stage(name: 'decrypt', description: 'Decrypt'),
        new Stage(name: 'initialization', description: 'Initialization'),
        new Stage(name: 'build', description: 'Build'),
        new Stage(name: 'test', description: 'Test'),
        new Stage(name: 'versioning', description: 'Versioning'),
        new Stage(name: 'publish', description: 'Publish'),
        new Stage(name: 'promote', description: 'Promote'),
        new Stage(name: 'deploy', description: 'Deploy')
    ]
]

Although the standard pipeline supplies the previous list of stages by default, it is possible to define stages manually, indicating in the repository that stages will be executed for each type.

There are two different ways to do this:

Modify or add stages on the standard pipeline. For this you must use the modifiedStages property. If a stage is defined with the name of an existing one, it is becoming overwritten. In case the added stage has a name different from the existing default stage, the new stage will be added in the execution.
Refine completely the list of stages to be executed. As indicated above in the options, the stages property allows you to define the list of stages that will be executed.

Example to execute only the test stage in the ‘pr’ type:

stages = [
    'pr': [
        new Stage(name: 'test', description: 'Test')
    ]
]

What tasks are executed for each stages?

Each Stage executes a series of task based on the value of the language property.

Node

Stage	Description
checkSkip*	Perform a check to know if it can do skip of the build.
decrypt*	Decrypt files that need it.
initialization	Gets versions for AuraComponent type artifacts.
build	Run `node/build.sh` script for all `Library` type components (the scripts can get libraries automatically).
test	Run `node/test.sh` script for all `Library` type components (the scripts can get libraries automatically). After execution, reports are generated for: Test, Coverage and LiNT.
versioning	Run `node/versioning.sh` script for all `Library` type components (the scripts can get libraries automatically).
publish	If the `onlyPack` option is added, the `npmPack.sh` script will be executed. Otherwise, the `publish.sh` script will be executed.
promote*	For each `AuraComponent` type artifact performs a docker `tag` and `push` command
deploy*	For each `AuraComponent` type artifact performs a deployment for the component

The value * at the end of stage’s name indicates that it is a default stage and it has not been modified for the node language.

Scripts used in the execution process for the node language:

node
├── build.sh        # Execute "npm install".
├── npmPack.sh      # Execute "npm install", "npm pack" and move the generated packages to the _artifacts_ folder.
├── publish.sh      # Check if it is not published and runs "npm install" and "npm publish".
├── test.sh         # Execute "npm run coverage-jenkins" and "npm run lint-jenkins".
└── versioning.sh   # Detects changes in the library to increase version (only if publishConfig exists in package.json)

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Main branches stages

Description of the different stages for main branches in the Release Train Manager

The following diagram shows the Release Train Manager stages for main branches:

All phases of the main branches are managed by the workflow: .github/workflows/push-to-main.yml.

name: Push To Main
on:
  push:
    branches:
      - "main/**"
      - "hotfix/**"

jobs:
  load-configuration:
  ########
  # INIT #
  ########
  prepare-init-process:
  protect_branch_main:
  init-bot-module:
  init-devops-module:
  init-legacy-module:
  init-models-module:
  init-nlp-module:
  init-training-nlp-module:
  init-qa-module:
  #########
  # START #
  #########
  start-bot-module:
  start-qa-module:
  ####################
  # PREPARE TO MERGE #
  ####################
  block-close:
  prepare-to-merge-bot-module:
  prepare-to-merge-training-nlp-module:
  prepare-to-merge-qa-module:
  prepare-to-merge-legacy-module:
  prepare-to-merge-devops-module:
  #########
  # CLOSE #
  #########
  close-bot-module:
  close-devops-module:
  close-qa-module:
  close-legacy-module:
  close-nlp-module:
  close-training-nlp-module:
  ############
  # FINALIZE #
  ############
  finalize-bot-module:

  process-result:

Main Init stage

In this phase, the branches are generated in the repositories and, if there is a configured environment, it will be deployed. As in every stage, each team defines its own steps, but some of them are shared and executed by any of the teams.

Common steps

Create branch

This step gets the list of repositories of the team (from branch-config.yml file, teams.*.repositories) which step is being executed and creates the corresponding main branch in all of them. If the branch already exists, it just goes on.

Protect Branch

Once all the initial changes are applied in each package of each main branch, they are protected to force the creation of a Pull Request to merge changes on them. It is executed after every stage of the RTM, because it needs the branch to be unprotected during its execution, to make changes automatically.

Unprotect Branch

RTM stages need to remove repositories’ branches protection to apply changes automatically in the code. It is executed at the beginning of almost any stage of the RTM.

Check conflicts

This step is executed during prepare-to-master stage to check if there is any conflict between master and the main branch in any configured repository. Any conflict will be reflected in the RTM as a message in the corresponding PR.

Init Bot Steps

Bot: Install dependencies

In this step, the dependencies defined in teams.bot.dependencies are installed. The tgz of the NPM module is obtained and installed in the Github repository. This is necessary because the main branch works in isolation and uses Github as the main repository.

Bot: Store Locales

In this step, all the locales files of every package in every repository are stored in DB. They are stored in the collection called branch-locales.

Bot: Prepare Repository

In this step, all the dependencies of every package in every repository are updated. Internal node packages (those belonging to @telefonica NPM organization) in master branch are read from npm registry and a x.y.z version is used. But each main branch uses for the internal dependencies a private GitHub npm registry and the dependencies are tagged with the decodedName of the branch: version: 7.5.0-main-my-feature.0.

Packages are read in order, so first those packages without internal dependencies are processed. They are published in GitHub registry and those packages depending on them are then processed, going on recursively.

Bot: Create Changelog

This step is responsible for creating the changelog scaffolding for the current version.

Bot: Install Tools

A new project is created in POEditor to include the language resources to be contained in this main branch. The project is created by auradev@tid.es user and its name will be the branch decoded name: main/feature-desc#id -> main-feature-desc-id. This project is intended to contain all the changes (creations or updates) of locales for the issues being resolved in this main branch.

Main Start stage

In this phase, all the components configured in branch-config.yml and that were prepared during the init stage will be deployed if an environment has been configured for the current branch. Afterwards, the sanity test plan is launched just to validate that the environment is properly settled. If no environment is configured (for instance, if a branch is just to review documentation or to prepare something specifically for a given OB, such as language trainings) tests are not executed.

Start Bot Steps

Bot: Trigger Deploy Components

A push is launched in the repositories of the components to force them to be deployed from CI job running in Jenkins. After each component is deployed, Jenkins sends a notification to GitHub to inform of the status, that is updated in the RTM API database.

Bot: Validate Deploy Components

This step is in charge of validating that all the components that are configured for the current branch have been deployed. It reads the information from the RTM API database that is updated with every notification from Jenkins, as explained in the previous step.

Main Prepare-To-Merge stage

This phase aims to merge master changes in the current main branch, deploy the updated version in the branch in configured environment, if exists, and launch the complete functional test plan to validate the full code base before merging to master.

Prepare-To-Merge Bot Steps

Bot: Update Locales

This step is in charge of updating the locales files in all repositories doing a merge into the main poEditor project and Aura-Bot PoEditor project.

Bot: Merge and Update Versions

Once the eventual conflicts have been resolved in all repositories configured in the branch, the content of master and main branches are merged together. In the happening of any change, tagged versions are automatically updated to provoke a new deployment of the components.

Bot: Publish Modules

If new versions have been generated in any of the bot npm libraries, a new version of them is published and updated in all the components.

Main Close stage

In this phase, all the components will be deployed in master and the libraries published in NPM. The versions of the modules will be updated if the repositories have been changed along the development in the main branch.

Close Bot Steps

Bot: Prepare versions to master

As already mentioned, bot versions in master are not tagged but in x.y.z format. This step calculates the next version of each library and component automatically depending on the existing version in master, the original semantic version and the number of changes in its tagged version in the main branch. If a library or a component has been updated in a main branch, no matter the number of commits needed for the change, its semantic version in master will be increased by 1 in the minor counter. For instance, if the version in master for library my-library is 6.2.0, and my-library has some changes in the current branch, its version in master after merging the main branch, will be 6.3.0.

Bot: merge to master

If new versions have been generated in the libraries, changes are merged and the libraries are published in @telefonica private npm registry.

Finalize step

In this phase, all the branches created, packages installed in Github and the POEditor project are deleted.

Finalize Bot Steps

Bot: Remove Main Branches

The main branch created in the repositories is removed.

Bot: Uninstall Tools

This step removes the POEditor project created for the language resources of this main branch.

Bot: Remove Packages

This step removes the packages created in the branch for each library.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Branch configuration

Description of the process for the configuration of branches in the Release Train Manager

Branch configuration file

The configuration of branches is defined in the branch-config.yml file.

Each stage defined in the schema is fully described in the Stages section.

Properties

Property	type	description	default	mandatory
name	string	Name of the branch. The format is: `[main\|hotfix\|fasttrack\|release]/description#jira-id` Example: `name: main/doc-review#14820`	none	true
release_branch	string	Name of the base branch from which the specific branch is generated.	master	In hotfix and fasttrack branches
release_version	string	Version of the base branch from which the branch will be generated.	none	In release branches
environment	string	English letter name to indicate the environment number to be generated to deploy the components used in the special branch.	none	If an environment is needed: from `one` to `ten`
mode	string	It specifies whether the branch that has been generated is to make a “real” change to the repositories to perform a test.	production	Use `developer` for test purposes
version	string	Version of configuration file.	1.0.0	Not used yet
description	string	Brief description of the changes for which the branch has been generated.	Fake text	true
jira	Jira	Jira references and tags referring to the task that the branch solves.	Fake data	true
teams	Map<string,Team>	List of teams involved in the resolution of the task solved by this branch.	none	At least one
main_protection	Object	Protection that will be applied to main branch when created in the RTM repository. This configuration can be partial and will be merged with the default configuration. All the documentation concerning the protection of a branch can be found at Github	Defined by Template	true
protection	Object	Default configuration property, this configuration will be merged between custom protection configurations of teams or main branches. All the documentation concerning the protection of a branch can be found at Github	Defined by Template	true

Jira Property

Property	type	description	default	mandatory
tags	string[]	String words used in Jira to categorize the feature or bug.	Fake data	true
references	string[]	Jira URLs where the tasks that the branch solves once the life cycle is completed are	Fake data	true

Team Object

It contains a set of common properties and others customized by each team. In the following table, the properties that are shared by all teams are described.

Property	type	description	default	mandatory
repositories	string[]	List of repository names involved in the changes to be made to the branch.	Defined by Template	true
protection	Object	Security configuration of the branches to be generated in the repositories. This configuration can be partial and will be merged with the default configuration. Example: protection: restrictions: users: [] teams: - aura-ci - aura-bot-global-dev - aura-bot-ext-devs apps: [] All the documentation concerning the protection of a branch can be found at Github	Defined by Template	true
stages	Map<string,string[]>	Map of stages in which the team has a series of steps to perform. All steps that must be performed to complete the phase.	Defined by Template	true
tasks	string[]	List of custom tasks to be completed before closing the PR for the branch. These tasks will be converted into a checkbox component in the body of the Pull Request	Fake data	false

Team Bot Custom Properties

Description of the bot’s custom properties.

Property	type	description	default	mandatory
update_dependencies	Object[] of {name: string, version: string}	List of objects with name and version. This property is used to update dependencies on the modules in the repositories defined by aura-bot. All the modules that use one of the libraries in the list, will be updated to the version that has been specified.	[]	false
dependencies	Object[] of {name: string, version: string}	List of objects with name and version. This property contains those @telefonica libraries that are needed in the Github library repository to isolate dependencies. At the INIT stage, the system will pick up that library from NPM and publish it to Github.	[]	false
node_version	number	Version of node to use in the bot’s Workflows.	14	true
tools	Object	It currently contains only one property, POEditor, which specifies the creation of a separated project to manage the localization strings.	Defined by Template	false

Branch config File

name: master
environment: none
mode: production
version: 1.0.0
description: Text to describe the release changes
jira:
  tags:
    - MyTag
  references:
    - "https://jira.tid.es/browse/AURA-12941"
    - "https://jira.tid.es/browse/AURA-13210"
teams:
  bot:
    update_dependencies: []
    repositories:
      - aura-aog-bridge
      - aura-services-api
      - aura-bot-common
      - aura-clients
      - aura-bot-libraries
      - aura-bot-libraries-test
      - aura-bot-platform
      - aura-bridge
      - aura-code-generator
      - aura-locale-importer
      - aura-movistar-libraries-utilities
      - aura-terms-and-conditions-api
      - aura-utilities
      - aura-kpis-uploader
      - aura-docs
    node_version: 14
    dependencies:
      - name: baikal-sdk
        version: 0.10.1
      - name: baikal-sdk
    tools:
      po_editor:
        project_base: Aura-Bot
    protection:
      restrictions:
        users: []
        teams:
          - aura-ci
          - aura-bot-global-dev
          - aura-bot-ext-devs
        apps: []
    stages:
      init:
        - install-dependencies
        - create-branch
        - store-locales
        - prepare-repository
        - create-changelog
        - protect-branch
        - install-tools
      start:
        - unprotect-branch
        - trigger-deploy-components
        - protect-branch
        - validate-deploy-components
      prepare-to-merge:
        - update-locales
        - check-conflicts
        - merge-and-update-versions
        - publish-modules
        - trigger-deploy-components
        - validate-deploy-components
      close:
        - unprotect_branch
        - prepare-versions-to-master
        - merge-to-master
        - validate-deploy-components
      finalize:
        - remove-main-branches
        - uninstall-tools
        - remove-packages
    tasks:
      - Review documentation before closing
      - Review PoEditor content before closing
      - Review performance before closing
  devops:
    repositories:
      - aurak8s
    stages:
      init:
        - create-branch
        - initial-commit
        - check-environment
        - create-environment
        - deploy-environment
        - unprotect-tests-branch
        - upload-environment-files
        - protect-tests-branch
        - protect-branch
      prepare-to-merge:
        - check-conflicts
        - merge
      close:
        - check-environment
        - close-environment
        - delete-environment
        - merge-to-master
    protection:
      restrictions:
        users: []
        teams:
          - aura-ci
          - aura-devops-admin
          - aura-global-cto
        apps: []
  qa:
    repositories:
      - aura-tests
    protection:
      required_status_checks:
        strict: false
        contexts: ['continuous-integration/jenkins/pr-head']
      enforce_admins: true
    tests:
      smoke_tests_enabled: false
      wa_mobile_tests_enabled: false
    stages:
      init:
        - create-branch
        - prepare-repository
        - protect-branch
      start:
        - sanity-tests
      prepare-to-merge:
        - unprotect-branch
        - check-conflicts
        - merge
        - protect-branch
        - smoke-tests
      close:
        - unprotect-branch
        - restore-repository
        - unprotect-master
        - merge-to-master
        - protect-master
        - smoke-tests

  nlp:
    repositories:
      - aura-cognitive-container
    repository_components: complex_logic,context-4p-synchronizer,context_provisioning,context,nlp-provisioning,nlp
    stages:
      init:
        - create-branch
      prepare-to-merge:
        - check-status-container
      close:
        - merge-to-master
    tasks: []

  training_nlp:
    repositories:
      - aura-nlpdata-es
    release_branch: release/7.0
    stages:
      init:
        - create-branch
      prepare-to-merge:
        - create-all-main-prs
        - check-status-repositories
      close:
        - merge-to-master
    tasks: [ ]

  legacy:
    repositories:
      - aura-cognitive-containers
    protection:
      restrictions:
        users: []
        teams:
          - aura-ci
          - aura-platform-tech-leads
          - aura-global-cto
        apps: []
    stages:
      init:
        - create-branch
        - prepare-repository
        - protect-branch
      prepare-to-merge:
        - check-conflicts
        - unprotect-branch
        - merge
        - protect-branch
      close:
        - unprotect-branch
        - update-version
        - unprotect-master
        - merge-to-master
        - protect-master

main_protection:
  restrictions:
    users: []
    teams:
      - aura-ci
      - aura-global-cto
    apps: []

protection:
  required_status_checks:
    strict: false
    contexts: []
  enforce_admins: false
  required_pull_request_reviews:
    dismissal_restrictions:
      users: []
      teams:
        - aura-ci
    dismiss_stale_reviews: true
    require_code_owner_reviews: true
    required_approving_review_count: 2
    bypass_pull_request_allowances:
      users: []
      teams:
        - aura-ci
  restrictions:
    users: []
    teams: []
    apps: []
  required_linear_history: false
  allow_force_pushes: false
  allow_deletions: false
  block_creations: false
  required_conversation_resolution: false

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Managing locales in RTM

How locales files and POEditor project are managed in RTM.

How to manage locales

To avoid updating the locales with the base project in the main branches in the RTM INIT process, there is a job store-locales, that is responsible for storing the locales of each component/library in the branch-locales collection with the following structure:

{
   "branch": "main/fake",
   "authentication-api": {
        "de-de": "{\n \"authentication:authenticator-error.description\": [\n \"Bei der Authentifizierung ist ein Fehler aufgetreten. Bitte versuche es später noch einmal\"\n ],\n ....",
        "es-cr": "{\n \"authentication:A1004.authenticator-error.description\": [\n \"Parece que has olvidado tu usuario o contraseña de acceso a Mi Movistar.\"\n ] ....}"
   },
    "aura-bot-platform": {
        "es-cr": "{\n \"context-filter:multimsisdn-users-not-allowed.onboarding\": [\n \"Pulsa el micrófono y di lo que quieras. Consulta la ayuda para saber qué cosas puedes preguntarme.\"\n ],\n \"context-filter:user-info-not-accessible\": [\n \"Sorry\"\n ],\n \"core:bypass.close.words\": [\n \"cancelar\",\n \"cerrar\",\n \"salir\",\n \"desconectar\"\n ],\n \"core:empty.response\": [\n \"Ok, estarei aqui sempre que você precisar.\"\n ] }\n",
        "de-de": "{\n \"context-filter:multimsisdn-users-intent-not-allowed.text\": [\n \"Entschuldigung, auf diese Daten kann ich aufgrund deines Vertrages nicht zugreifen. Zukünftig kann ich das sicherlich, aber bis dahin kann ich dir mit Fragen rund um o2 Services weiterhelfen. Wenn du mehr Informationen über deinen Vertrag erhalten möchtest, gehe einfach in deinen [Mein o2 Bereich](www.o2online.de/meino2).\"\n ],\n \"context-filter:user-info-not-accessible\": [\n \"Sorry\"\n ],\n \"core:bypass.close.words\": [\n \"cancelar\",\n \"cerrar\",\n \"salir\",\n \"desconectar\"\n ],\n \"core:empty.response\": [\n \"Ok, estarei aqui sempre que você precisar.\"\n ] }\n"
     }
}

During the development of the main branch, if it is necessary to update the locales of any component, it will be done using the locale-update script, which will have been updated by the following locale-importer command in the creation phase of the branches of each component/library:

aura-locale-importer -u -s Db -pkg <package_name> -m library -b core -d ./locale -f -mb <main-branch-name>
The environment variables SECRET_TOKEN and BRANCH_CONFIG_APIKEY are required for its execution.

This command will do a merge between the local files stored in DB for that component/library and the resources created/updated in the POEditor project main-branch-name generating the new locales files.

In the execution of the prepare-to-merge of the bot, an update-locales step has been added, which updates the locales of each component/library by merging the content of the base project in POEditor (Aura-Bot) with the POEditor project of the main branch.

Endpoints locales

For the management of the locales in BD, the following endpoints have been created in aura-release-train-branches: - /branches/{branch}/locales - /branches/{branch}/locales/{package}

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Release Train Manager

Release Train Manager (RTM) implementation and architecture

Introduction to RTM development methodology

The “release train” software development methodology is a way of planning the delivery of software according to a predetermined and regular schedule, as if it were a “train timetable”. This schedule is public for all the teams that must contribute to the delivery and represents a commitment on their part, as the schedule must be adhered to.

If we use the metaphor: “the train will take the passengers who are ready to travel”, i.e., only the software that is ready to be integrated will be integrated. For this it is necessary to work on the different functionalities in parallel and in isolation without affecting the rest.

We start from a prioritized backlog. The tasks will enter the implementation phase in order of priority.

Github is used for the organization of the process. See here the details of Github as orchestrator.
The file and properties for the configuration of branches is included in Branch configuration.
Learn how locales files and the POEditor project are managed in RTM: Managing locales.

RTM overview

The important thing when we are working on a new feature is to be able to completely isolate the development, environment and testing. To do this, environments will be generated per feature and git branches will be used for the different repositories involved.

RTM Overview

INIT: initial processing of a feature (select repos involved, create branches on those repos and create environment).
TF (Tests Feature): daily smoke tests on feat/new-feature branches.
TFPR (Tests Feature PR): full regression tests on the master merge and the feat/new-feature branch.
TM (Tests Master): complete daily/weekly regression tests on the master branches.
PR Fx (PR Feature): PRs on the feat/new-feature branch.
PR Mx (PR Master): PRs on the master branch.

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Aura Continuous Integration

Find in this section key information related to Aura’s Continuous Integration system

Introduction

Aura uses Jenkins as CI orchestrator, mainly by the use of Jenkinsfiles with some internally developed libraries.

Our Jenkins runs in DCIP instance.

Access to these documents to find key information regarding:

Docs:

Mon, 01 Jan 0001 00:00:00 +0000

Repositories configuration file: aura-config.json

The purpose of the aura-config.json file is to provide a description of the repository to make it easier for releases and CI flows to manage each repository

Description

The aura-config.json configuration file can be created at the root of each Aura project to specify configuration that can be used to automate tasks.

Structure

Sections

{
    "release": {
        // Release information
    },
    "docs": {
        // Information associated with documentation (for example aura-docs)
    }
}

Fields

Field	Type	Description
name	string	Library or package name
description	string	Library or package description
multiRepository	boolean	In deployment tasks, it is used to know if the project contains other projects (packages)
release.name	string	Current release name . It can be used by projects like aura-release-manager to increase versions automatically
release.version	string	Current release version. It follows the https://semver standard and it can be used by aura-release-manager and jenkins to increase versions automatically

Fields in bold are required

Visual example

{
    "name": "aura-bot-libraries",
    "description": "aura-bot-libraries contains all global libraries required in aura-bot.",
    "multiRepository": true,
    "release": {
        "name": "tasos",
        "version": "4.0.0"
    }
}

Interfaces

AuraConfig

/**
 * @interface AuraConfig
 * @description Interface to aura-config.json file
 */
export interface AuraConfig {
    /**
     * Library or package name
     */
    name: string;
    /**
     * Library or package description
     */
    description: string;
    /**
     * In deployment tasks, it is used to know if the project contains other projects (packages)
     */
    multiRepository?: boolean;
    /**
     * Release information
     */
    release?: AuraConfigRelease;
}

AuraConfigRelease

/**
 * @interface AuraConfigRelease
 */
export interface AuraConfigRelease {
    /**
     * Contains the current release name . It can be used by projects like aura-release-manager
     * to increase versions automatically
     */
    name: string;
    /**
     * Contains the current release version. It follows the https://semver standard and it can be used by
     * aura-release-manager and jenkins to increase versions automatically
     */
    version: string;
}

Project using aura-config.json

Project name	Used configuration	Used in
aura-bot-libraries	release	jenkins

JSON Schema

aura-config.schema.json