AEM PreUpgrade Tasks and Activites

Pre-Upgrade Maintenance Tasks

Follow below maintenance tasks to ensure that the system is ready & it can be rolled back if any issues occur:

v  Ensure Sufficient Disk Space

v  Fully Back Up AEM

v  Back Up Changes to /etc

v  Generate The quickstart.properties File

v  Configure Workflow and Audit Log Purging

v  Install, Configure, and Run The Pre-Upgrade Tasks

v  Disable Custom Login Modules

v  Remove Updates From The /install Directory

v  Stop Any Cold Standby Instances

v  Disable Custom Scheduled Jobs

v  Execute Offline Revision Cleanup

v  Execute Datastore Garbage Collection

v  Upgrade the Database Schema If Needed

v  Delete Users that Might Hinder the Upgrade

v  Rotate Log Files

Ensure Sufficient Disk Space

                     When executing the upgrade, in addition to the content and code upgrade activities, a repository migration will need to be performed. Migration will create a copy of the repository in the new Segment Tar format. As a result, we need enough disk space to retain a second, potentially larger, version of your repository.

Fully Back Up AEM

                    AEM should be fully backed up before beginning the upgrade. Make sure to back up your repository, application installation, data store, and Mongo instances if any applicable.

Back Up Changes to /etc

                  Upgrade process does maintaining and merging existing content and configurations from under the /apps and /libs paths in the repository.  For changes made to the /etc path, including Context Hub configurations, it is necessary to re-apply these changes after the upgrade. While upgrade will make a backup copy of any changes that it cannot merge under /var, its recommend to back up these changes manually before beginning the upgrade.

Generate the quickstart.properties File

                   When starting AEM from the jar file, a quickstart.properties file will be generated under crx-quickstart/conf. If AEM has been started with the start script in the past, this file will not be present and the upgrade will fail. Make sure to check for the existence of this file and restart AEM from jar file if it is not present.

Configure Workflow and Audit Log Purging

                     The WorkflowPurgeTask and com.day.cq.audit.impl.AuditLogMaintenanceTask tasks require separate OSGi configurations and will not work without them. If pre-upgrade task execution fail, missing configurations is likely the reason. So make sure to add OSGi configurations for these tasks.

Install, Configure, and Run The Pre-Upgrade Tasks

            Pre-upgrade maintenance tasks that before had to be performed manually are being optimized and automated. Pre-upgrade maintenance optimization added to AEM 6.3 enables a unified way to trigger these tasks and be able to inspect their result on demand.

All tasks included in the pre-upgrade optimization step are compatible with all versions from AEM 6.0 onwards.

For upgrading from AEM 6.2

  Download and install the package pre-upgrade-tasks-content-cq62-1.2.4 on the instance to be upgrade. Default Configuration of the Pre-Upgrade Health Checks – Ran runAllPerUpgradeTasks() method

How to Use It

          The PreUpgradeTasksMBean OSGI component comes preconfigured with a list of pre-upgrade maintenance tasks that can be run all at once. You can configure the tasks by following the below procedure:

Go to the Web Console http://serveraddress:serverport/system/console/configMgr

Search for "preupgradetasks", then click on the first matching component, name is com.adobe.aem.upgrade.prechecks.mbean.impl.PreUpgradeTasksMBeanImpl.

Modify the list of maintenance tasks that need to run as shown below:

Default Configuration of the Pre-Upgrade Health Checks

              Access the MBeans by going to the JMX Console http://serveraddress:serverport/system/console/jmx

         Search for PreUpgradeTasks and click it. Select any method from the Operations section and select Invoke like in below window.

runAllPreUpgradeTasks()

ACTION

Runs all the pre-upgrade maintenance tasks in the list.

Remove Updates From The /install Directory

              Remove any service packs, feature packs or hotfixes that have been deployed through the crx-quickstart/install directory on the local file system. This prevent the inadvertent installation of old hotfixes and service packs on top of the new AEM version after the update has completed.

Stop Any Cold Standby Instances

              Using TarMK cold standby, stop any cold standby instances. These guarantee an efficient way to come back online in case of issues in the upgrade.

After the upgrade has completed successfully, the cold standby instances will need to be rebuilt from the upgraded primary instances.

Disable Custom Scheduled Jobs

               Disable any OSGi scheduled jobs that are included in your application code.

Created Purge configs for Workflow and Audit Logs

https://blogs.perficientdigital.com/2017/09/14/aem-6-3-when-runallpreup-gradetask-fails/

Run tar offline compaction process

Run below compaction commands as part of pre-upgrade maintenance tasks to compact the repository.

Check for Check points that needs to be deleted in later command.

        java -jar oak-run.jar checkpoints installPath /crx-quickstart/repository/segmentstore

Next, delete the unreferenced checkpoints

        java -jar oak-run.jar checkpoints  installPath/crx-quickstart/repository/segmentstore rm-all

Final step is to run the compaction and wait for it to complete (tail the logs!)

       java -jar oak-run.jar compact installPath /crx-quickstart/repository/segmentstore

Download respective oak-run jar version, based on version of AEM being used.

Unpack AEM 6.5 jar.

              After successfully completion of pre-upgrade tasks run below command by placing AEM 6.5 jar under 6.2 folder & unpack 6.5 version by using below command.

java -Xmx4096m -jar AEM_6.5_Quickstart.jar –unpack

Now need to run below command to change repository to segment tar

java -Xmx4096m -XX:MaxPermSize=2048m -jar AEM_6.5_Quickstart.jar -v -x crx2oak -xargs -- --load-profile segment-no-ds

Use name of crx2oak jar and place it under ‘crx-quickstart\opt\extensions’ - crx2oak-1.10.0-all-in-one, use name of crx2oak in the command.

Note: For file system data store we need update the node profile, refer link for more info.

https://helpx.adobe.com/experience-manager/6-5/sites/deploying/using/in-place-upgrade.html

After completion of above command below log is thrown in command prompt.

To complete migration you need to manually move the following files in the following order:

                                                Source |  Operation |                                        Destination

    ...192_HEAP\crx-quickstart\repository\segmentstore | -- MOVE -> | ...\crx-quickstart\repository-crx2oak-backup-20...

    ...\crx-quickstart\repository-segment-tar-20191... | -- MOVE -> | ...192_HEAP\crx-quickstart\repository\segmentstore

Under UNIX you can try with the following commands:

    mv crx-quickstart\repository\segmentstore crx-quickstart\repository-crx2oak-backup-20191115-130058\segmentstore

    mv crx-quickstart\repository-segment-tar-20191115-130044\segmentstore crx-quickstart\repository\segmentstore

  Under Windows you can try with the following commands:

    move crx-quickstart\repository\segmentstore crx-quickstart\repository-crx2oak-backup-20191115-130058\segmentstore

    move crx-quickstart\repository-segment-tar-20191115-130044\segmentstore crx-quickstart\repository\segmentstore

Now Start AEM 6.5.

It starts with new segment tar repository & also it performs re-indexing. Wait to see below message in log.

        “org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing completed”

Navigate to product information console to see the product version

Referrences: https://helpx.adobe.com/experience-manager/6-3/sites/deploying/using/pre-upgrade-maintenance-tasks.html

https://helpx.adobe.com/experience-manager/6-5/sites/deploying/using/in-place-upgrade.html

AEM Schedulers

          Apache Sling Scheduler enables you to easily schedule jobs within your application. Jobs can be executed at a specific time, regularly at a given period or at the time given by a cron expression by leveraging the Sling scheduler service.

Scheduler in AEM
For Creating a scheduler in AEM, follow below steps -
- Create an OSGi configuration to read the scheduler specific values from the user i.e. cron expression, the name of the scheduler, custom parameter etc.
- Create a sling scheduler which displays the custom parameter at an interval specified by the cron expression.

import org.apache.sling.commons.scheduler.ScheduleOptions;
import org.apache.sling.commons.scheduler.Scheduler;
import org.osgi.service.component.annotations.Activate;
import org.osgi.service.component.annotations.Component;
import org.osgi.service.component.annotations.Deactivate;
import org.osgi.service.component.annotations.Modified;
import org.osgi.service.component.annotations.Reference;
import org.osgi.service.metatype.annotations.Designate;
import org.demo.core.services.CustomSchedulerConfig;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
 * @author Sailaxman
 *
 * A Sling Scheduler demo using OSGi R6 annotations
 *
 */
@Component(immediate = true, service = CustomScheduler.class)
@Designate(ocd = CustomSchedulerConfig.class)
public class CustomScheduler implements Runnable {

private static final Logger log = LoggerFactory.getLogger(CustomScheduler.class);
private String customParam;

/**
 * Id of the scheduler based on its name
 */
private int schedulerId;

/**
 * Scheduler instance injected
 */
@Reference
private Scheduler scheduler;

/**
 * Activate Method to initialize stuff
 */
@Activate
protected void activate(CustomSchedulerConfig config) {
 // Getting the scheduler id
schedulerId = config.schdulerName().hashCode();

 // Getting the custom parameter
customParam = config.customParam();
}

/**
 * Method Modifies the scheduler id on modification
 */
@Modified
protected void modified(CustomSchedulerConfig config) {
//Removing the scheduler
removeScheduler();

// Updating scheduler id
schedulerId = config.schdulerName().hashCode();

//Again adding the scheduler
addScheduler(config);
}

/**
 * Method deactivates the scheduler and removes it
 */
@Deactivate
protected void deactivate(CustomSchedulerConfig config) {
//Removing the scheduler
removeScheduler();
}
/**
 * This method removes the scheduler
 */
private void removeScheduler() {
log.info("Removing scheduler: {}", schedulerId);
//Unscheduling/removing the scheduler
scheduler.unschedule(String.valueOf(schedulerId));
}
/**
 * Method adds the scheduler
 */
private void addScheduler(CustomSchedulerConfig config) {
// Check if scheduler is enabled or not
if(config.enabled()) {
// Scheduler option takes the cron expression as a parameter and run accordingly
ScheduleOptions scheduleOptions = scheduler.EXPR(config.cronExpression());
// Adding params
scheduleOptions.name(config.schdulerName());
scheduleOptions.canRunConcurrently(false);

// Scheduling job
scheduler.schedule(this, scheduleOptions);
log.info("Scheduler added");
} else {
log.info("Scheduler is Disabled");
}
}
/**
 * Overridden run method to execute
 */
@Override
public void run() {
log.info("Custom Scheduler is now running using the passed custom paratmeter, customParam {}", customParam);
}
}

• First, we are registering the class as a service and implementing the Runnable interface. At the same time, using @Desginate annotation, we are linking the OSGi configuration created in the previous section with this class.
• Now, we are injecting the org.apache.sling.commons.scheduler.Scheduler dependency.
• In the activate() method, we are reading the required values. Then we are getting the schedulerId from the scheduler name.
• The modified() method recalculates the schedulerId in case the OSGi configuration is modified.
• In the addScheduler() method, we are registering the scheduler using the Scheduler API.
• The run() method will be defining our task. Here we are just printing the customParameter in the logs.

AEM Asset Off Loading

Setting up Asset Offloading

Steps to be involved:-
     1. Set topology for the Master and Worker instances. Add Worker in Master & Master in Worker topology configurations.
     2. Master is configured with DAM Update Asset Offloading Workflow to run instead of DAM Update Asset.
     3. When assets are uploaded on to Master, an offloading job sends asset info to Worker i.e. from Master -> Worker.
     4. Worker executes actual DAM Update Asset Workflow on each & every asset.
     5. With Reverse replication, the package with processed assets & its renditions are replicated back to Master from Worker i.e. Master <- Worker.

Set Topology For Instances:

• Login into OSGi on the Master instance http:// <host>:<port>/system/console/topology
• Configure 'sling discovery service' with topology connector url also note Sling ID and here we need to configure Configure Discovery.Oak Service at

                 http:// <host>:<port>/system/console/configMgr/org.apache.sling.discovery.oak.Config

• In Master topology update hostname and IP address of Worker topology

                              

• Now Login into OSGi console of Worker i.e. http:// <host>:<port>/system/console/topology
• Note instance Sling ID and here needs to configure Configure Discovery.Oak Service at

                     http:// <host>:<port>/system/console/configMgr/org.apache.sling.discovery.oak.Config

• Configure sling discovery service with topology connector url ( Topology connector url of Master http://master:port/libs/sling/topology/connector)
• Verify that Worker Outgoing topology connectors is pointed to the Master AEM

After topology configuration is done it looks like below with instances sling id.

                   

   Configure Topic Consumption

• On Master access: http://<hostname>:<port>/libs/granite/offloading/content/view.html to configure topic.
• Search for ‘Topic: com/adobe/granite/workflow/offloading’
• Disable Topic for the Master instance:

                        

   Offloading processing of DAM Assets

1. On the Master instance go to Workflow console and access launchers tab: http://<hostname>:<port>/libs/cq/workflow/content/console.html
2. For DAM Update Asset workflow launcher update Workflow drop down value from “DAM Update Asset” to “DAM Update Asset Offloading”.

                                             

       3. Open Workflow launcher console on the Worker, open DAM Update Assets workflow and uncheck Transient Workflow, save changes.

                            

   Use Forward replication

1. On Worker instance, open configuration of OffloadingDefaultTransporter (http://<hostname>:<port>/system/console/configMgr/com.adobe.granite.offloading.impl.transporter.OffloadingDefaultTransporter).
2. Change value of the property default.transport.agent-to-master.prefix from offloading_outbox to offloading.

                               

   Turning off transport packages

1. On Master, open configuration of OffloadingDefaultTransporter at http://<hostname>:<port>/system/console/configMgr/com.adobe.granite.offloading.impl.transporter.OffloadingDefaultTransporter
2. Disable the property Replication Package (default.transport.contentpackage).
3. Repeat same step on Worker:

                       

   Disabling the transport of workflow model

1. Open Master access Workflow console http://<hostname>:<port>/libs/cq/workflow/content/console.html.
2. Open the Models tab.
3. Open the DAM Update Asset Offloading workflow model.
4. Open step properties & Arguments tab, deselect Add Model to Input & Add Model to Output both options like below.

                    

         Save changes done to the model.

   Optimizing the polling interval

           Workflow offloading is implemented using an external workflow on the Master, that polls for the completion of the offloaded workflow on the Worker. The default polling interval for the external workflow processes is five       seconds. Adobe recommends to increase polling interval of the Assets offloading step to at least 15 seconds to reduce the offloading overhead on the master.

1. Open Workflow console on Master http://<hostname>:<port>/libs/cq/workflow/content/console.html.
2. Open the Models tab and open DAM Update Asset Offloading workflow model.
3. Open the step properties for the DAM Workflow Offloading step and open Commons tab, and adjust the value of the Period property.
4. Save changes to model.

   Turning off automatic agent management:

       Adobe recommends to turn off automatic agent management because it does not support binary-less replication and can cause confusion when setting up a new offloading topology. Moreover, it does not automatically support the forward replication flow required by binary-less replication.

1. Open Configuration Manager from the URL http://<hostname>:<port>/system/console/configMgr.
2. Open the configuration for OffloadingAgentManager (http://<hostname>:<port>/system/console/configMgr/com.adobe.granite.offloading.impl.transporter.OffloadingAgentManager).
3. Disable automatic agent management.

                  

        Repeat above step same on the Worker instance.

AEM Workflow Purging

Workflow Purging:

                  Situations like content syncing & migration, development issues, bulk execution or some other unpredictable situations can lead to huge number of workflow instances to run concurrently on AEM environment.

                 Concurrent running workflow instances could lead to performance issues on the server. Completed workflow instances should be purged as nodes in the repository gets accumulated and it would be beneficial to clean them to manage disk space and performance. It`s essential to purge instances to manage size of JCR repository. So, workflow purging is one of the maintenance activity that must be carried out.

Workflow Purge Scheduler

             Workflow Purge Scheduler is categorized as ‘Sling Service Factory’ so we can create an instance configuration we can configure it on Felix Console and manage the workflow purge scheduling as per our requirements.

•     Every time when we launch workflow a new workflow instance gets created (like any of the activities i.e. asset uploading, publishing etc.).

•    Once workflow completes with the stages mentioned (successful/aborted/terminated), it’s archived and never gets deleted.

•    Workflow purging needs to be done to clean up archived workflow instances.

•    Workflow purging can be done based on 3 categories for the workflows

• Based on Workflow model

• Based on Workflow Completion status

• On Age of the workflow instance

Enabling Workflow Purge Scheduler

It’s not a pre-configured feature in AEM. We need to enable AEM to automatically purge workflows.

·         Log into AEM osgi configuration console, host:port/system/console/configMgr to configure automatic workflow purging

·         Search for a configuration ‘Adobe Granite Workflow Purge Configuration’ which will be managing the purge schedule of the workflow instances. As it is a Service Factory we can create multiple instances of this config for different criteria’s and run parallel.

·         For adding another instance configuration, use ‘+’ sign provided on configuration.

The factory PID of configuration is ‘com.adobe.granite.workflow.purge.Scheduler’.

PID of instance will be set when the configuration is saved.

A unique ID is appended to the Factory PID of the configuration like below com.adobe.granite.workflow.purge.Scheduler-myIdentifier where myidentifier is the unique id mentioned above.

·         Configure below to set up scheduler to purge the workflows. It has following fields:

             Job Name: Name of particular purge instance.

            Workflow Status: Options like ‘Completed’ and ‘Running’ we can choose which status of workflows to purge.

Models to purge: It`s a multi field to provide different models to purge. Configure ids of the models that are required for purge.

            Workflow Age: Enter number of days after which the workflows have to be purged.

Provide values in inputs as required.

           Workflow instances are stored as nodes on AEM. Long running instances where website users or automated jobs can run workflows, this adds large amounts of content to repository leading to slowness of AEM server and it disk space grows rapidly.

We can also use JMX console to purge repository of completed workflows at below urls.

·         To manually purge workflow, execute the operation purgeCompleted on the console with domain com.adobe.granite.workflow.

Click on maintenance for the domain mentioned above.

Based on the type of scenarios required for workflows we can execute different operation provided in the console.

     

     Currently on an average took few seconds or less than a minute while purging active/completed workflows on localhost. Average time for completion of process would differ based on content use of workflows, workflow instances it got created & currently running/completed workflows.

      Adobe in default recommends Weekly Maintenance Window for Workflow Purge task. Adobe default suggested timing for the daily maintenance window is 2 to 5 AM and to run weekly maintenance window will execute b/w 1 and 2 AM on Saturdays.

§  Workflow Purge Scheduler URL: https://host:portNo/system/console/jmx/com.adobe.granite.workflow%3Atype%3DMaintenance

§  Statistics of purged workflows:

       https://host:portNo /system/console/jmx/com.adobe.granite.workflow%3Atype%3DStatistics