Baseline patching

IMPORTANT

Currently the dry run setting has no effect if baseline is enabled, hence it's improssible to create dry-run plans inside a pipeline with baseline feature enabled.

General information

Baseline patching allows to install exactly the same vendor update patches on many environments. The baseline is automatically built based on the patches that were successfully installed on the first environment in the pipeline. All installed patches from the first step are divided into groups based on OS type and version that is retrieved from SSM metadata PlatformName and PlatformVersion fields. For example Windows Server 2012 is treated as a separate baseline source than Windows Server 2016. Patches are retrieved from the corresponding group created in the first step. In later steps the decision on what patches to install on a particular machine will be made based on that machine’s OS type and version. Please note that we provide best effort solution and if machines states differ between steps, patching can result in missing packages, which will be reported.

Operating systems

Supported

• Windows Server 2008 - 2019, including R2 versions
• Amazon Linux 2012.03 - 2018.03
• Amazon Linux 2
• CentOS 6.x - 8.x
• Red Hat Enterprise Linux (RHEL)
• SUSE Linux
• SUSE Linux Enterprise Server (SLES)
• Ubuntu Server 14.04 LTS and later

Setting up

Creating

Baseline patching configuration is available on a pipeline level. Please note that for baseline to work properly, machines with each patched OS type and version must be included in the first step (i.e. if the second step contains an Ubuntu 14.04 machine the first step must also contain at least one Ubuntu 14.04 machine).

To create new baseline pipeline in the User interface go from left navigation bar to Plan pipelines > New pipeline. To change existing pipeline into baseline go to Plan pipelines, search for plan and click Edit. Then, in both cases, switch the checkbox Enable baseline on.

System requirements

Each machine must have SSM agent installed and be registered to AutoPatcher. This is the same process as in standard patching, which is described in Setting up SSM agent and in Adding a machine to AutoPatcher. Please note that existing machines may require policy update to allow access to SSM baseline documents (the arn:aws:ssm:*:*:document/AWS-RunPatchBaseline and arn:aws:ssm:*:${AutoPatcherAccountID}:document/AP-RunPatchBaseline resources should be allowed for ssm:SendCommand action, the details are here).

All Linux machines must have Python 2.7 or higher installed.

CentOS machines must have the yum-utils package installed and GPG key imported. In CentOS 7 that can be done by following command:

sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

More details about system requirements for the AWS-RunPatchBaseline SSM document (which is used for installing the packages in a baseline patching) are presented on the Official AWS SSM documentation page.

Patching

Windows

Baseline for Windows leverages the default patch baseline provided by AWS. There, patches are installed when they have classification set to CriticalUpdates or SecurityUpdates and they have severity set to Critical or Important. After the first step, baseline builder collects all the installed packages on machines from SSM. Exact statuses are: Installed, InstalledOther and InstalledPendingReboot. InstalledOther indicates that patch was installed by OS provider. InstalledPendingReboot indicates that installed patch requires reboot. After reboot, status is changed to Installed. When build process is completed, packages are passed as a InstallOverrideList argument for the next steps, so that following machines can only install updates from provided packages.

Linux

In Linux, AutoPatcher uses modified SSM baseline document, that allows to use custom repository during update. This ensures that none of the installed packages will be missing, when the next steps are executed. Custom repository is built after the first step and contains all of the installed packages for particular OS type and version, retrieved directly from corresponding package manager. Nonetheless custom baseline document still leverages on SSM baseline mechanism, so each distribution uses its own, default baseline. This means that patches are filtered as following:

Error handling and manual approval

The step approval mechanism for second step in a pipeline with the baseline option enabled is different than that without that option (third and later steps behave the same as in usual pipelines). The following subsections describe the flow depending on the patching policies the second step can have.

Success or approval (default)

If the first step fails, the baseline building process does not start automatically. It starts only after the user manually approves the second step. While baseline is building, the events from the second step are in the APPROVED state and are changed to NEW automatically when the baseline building process finishes successfully. If it fails (which in general is not expected to happen) the events from the first step are marked as DONE_ERROR_BASELINE and the second step is changed back to NOT_APPROVED. In this situation the user can make a decision to continue patching anyway (i.e. the baseline building failure was a false-positive or in other way acceptable to proceed) by approving the second step again (so called double approval). Then the events from the first step are changed back to their original states and the second step is marked as NEW.

Patch anyway

If the first step fails, the baseline building process is started anyway (similar to success or approval, the second step events are temporarily marked as APPROVED for that time). However, if the building fails, the first step marked id as DONE_ERROR_BASELINE and the second as NOT_APPROVED thus requiring manual approval from the user (similar to the double approve mechanism described in success or approval section).

Approval required

In this case the baseline building starts only after the user manually approves the second step, regardless of the first step status. The baseline building process failure is handled in the same manner as for other policies.