Orchestrator

sfpowerscripts:orchestrator is a group of commands (or topic in the CLI parlance) that enables one to work with multiple packages in a mono-repo through the development lifecycle or stages. The current version of the orchestrator features these commands

A typical pipeline

To understand the orchestrator, let's take a look at a typical CI/CD pipeline for a package based development in a program that has multiple environments. For brevity, prepare and validate states are not discussed.

Let's dive into the pipeline depicted above, there are two basic pipelines in play

• CI Pipeline: A pipeline that gets triggered on every merge to the trunk. During this process, the following stages happen in sequence.

  • ​quickbuild a set of changed packages (packages without validating for dependency or code coverage)

  • ​deploy to a Development Sandbox. This process ensures the upgrade process of a package is accurate and you could also do a quick round of validation of your packages coming in from a scratch org.

  • Once deploy is successful, the pipeline proceed to build the set of changed packages (but this time with dependency validation and code coverage check)

  • The pipeline could then publish these validated packages to an artifact repository for deployment into higher environments for further testing.

  Each of this stage could have a pre-approval step modelled like the example shown below

• CD Pipeline: A Continuous Delivery Pipeline that gets triggered manually or automatically (every day on a scheduled time interval) deploying a set of the latest validated packages to a series of environment. The sequence of stages include

  • Fetch the Artifacts from the artifact repository using the provided release definition

  • ​Deploy the set of packages say to System Testing environment

  • Upon successful testing, the same set of packages progress to the System Integration Test environment and so forth

  • If the packages are successful in all of the testing, the packages are marked for promotion

  • The promoted packages are then deployed to production.

Take a note of each stage in the pipeline above and the key functionality required, such as build, deploy, fetch etc, this is typically done by inserting the equivalent sfdx commands into your CI/CD pipeline definition. As your number of packages grow, it not only is hard to maintain but is error prone. This is where sfpowerscripts orchestrator simplifies the pipeline to a one time setup. All the stages are driven by sfdx-project.json, which ensures zero maintenance to the pipelines. Each stage of the above pipeline could be modelled by using equivalent sfpowerscripts orchestrator commands

Orchestrator commands

1. ​prepare (sfdx sfpowerscripts:orchestrator:prepare): Prepare command helps you to build a pool of prebuilt scratch orgs which include managed packages as well as packages in your repository. This process allows you to considerably cut down time in re-creating a scratch org during a pull request validation process when a scratch org is used as Just-in-time CI environment. In simple terms, it reduces time taken in building a scratch org to validate your changes in an incoming pull request. This command also have an option to pull artifacts from your artifact repository, so that say you can prebuild your validation orgs, say from validated set of packages.

2. ​validate (sfdx sfpowerscripts:orchestrator:validate): This command goes in pair with the prepare command. It fetches a scratch org from the pool already pre prepared (by the prepare command) and deploys/unit tests the changed packages.

3. ​build (sfdx sfpowerscripts:orchestrator:build/quickbuild) : This command builds all the packages in parallel wherever possible by understanding your manifest and dependency tree. Goodbye to the sequential builds, where you fail in the n-1th package and have to wait for the next hour. This command brings massive savings to your build (package creation) time. Also use the quickbuild variant, which builds unlocked package without dependency check in intermittent stages for faster feedback.

4. ​deploy (sfdx sfpowerscripts:orchestrator:deploy): So you have built all the packages, now this command takes care of deploying it, by reading the order of installation as you have specified in your sfdx-project.json. Installs it one by one, deciding to trigger tests etc. and provide you with the logs if anything fails

5. ​promote (sfdx sfpowerscripts:orchestrator:promote) : Promote enables the packages to be deployable to production. This explicit stage prevents incorrectly tested packages to reach production

6. ​publish (sfdx sfpowerscripts:orchestrator:publish) : Publish lets you publish the built artifacts into an artifact registry during publish stages of your pipeline.

Controlling Aspects of the Orchestrator

Orchestrator utilizes additional properties mentioned along with each package in your sfdx-project.json which can be used to control what the orchestrator should work with each package.

Handling multiple ignore files

Sometimes, due to certain platform errors, some metadata components need to be ignored during quickbuild or validate or any other stages. sfpowerscripts offer you an easy mechanism, which allows to switch .forceignore files depending on the stage.

Add this entry to your sfdx-project.json and as in the example below, mention the path to different files that need to be used for different stages

 "plugins": {
        "sfpowerscripts": {
            "ignoreFiles": {
                "prepare": ".forceignore",
                "validate": ".forceignore",
                "quickbuild": "forceignores/.buildignore",
                "build": "forceignores/.validationignore"
            }
        }

Ignoring a package on any particular stage from being being processed by the orchestrator

Utilize the ignoreOnStage descriptor to mark which packages should be skipped by the lifecycle commands.