Distributed Agents

The Distributed Agent Framework has been a key component of Extended ECM for several versions. This blog post hopes to help Administrators understand how the Distributed Agent Framework operates and give them a deeper look at the inner workings.

As Extended ECM continues to leverage the Distributed Agent Framework more in the future an understanding of it becomes more vital for Administrators.

Distributed Agent Framework Basics

The Distributed Agent Framework is a scalable solution, allowing for tasks to be executed across the cluster with redundancy. This framework is leveraged by many components of Extended ECM such as Facets, Notification Center, Records Management, Recycle Bin, Workflow Item Handler steps, and much more.

The Distributed Agent Framework is made up of three key components. Tasks, which portions of work to be completed. Workers which are the threads that execute the tasks. Then the Distributed Agent Agent (DAAgent) which manages the assignment of tasks to the workers.

Tasks

Tasks are small portions of work to be completed. They are submitted to the Distributed Agent Framework by a component in Extended ECM. There are two main types of tasks Simple Tasks, and Job Chains.

Distributed Agent Agent

The Distributed Agent Agent is responsible for looking at the inbound tasks that have entered the Distributed Agent Framework. Removing any duplication and assigning those tasks out to workers factoring in load balancing, with the priority of the task. The Distributed Agent Agent also handles errors from the Workers, and quarantines problematic tasks.

Workers

The Workers are Extended ECM threads which are dedicated to processing Tasks. They are assigned work by the Distributed Agent Agent. They can be optionally configured to only run specific tasks or allow all tasks to be executed by that worker.

Tasks

As tasks are created, they are placed into the WorkerQueue table. The tasks are created with several pieces of information that help the Distributed Agent Framework manage them. Tasks are assigned a priority by the Extended ECM component that created them, and a fingerprint of the task. This fingerprint allows the Distributed Agent Framework to identify tasks that are identical in nature, and deduplicate them. Additionally, the task can be created with an Activation Date set in the future. The task will not be executed until the Activation Date has been passed.

Simple Tasks

Simple Tasks which are self-contained, they may initiate another task but are not dependent on any other tasks. They also must complete in a single database transaction.

Job Chains

Job Chains which have one parent task, and many child tasks. Once all the child tasks are complete, the parent task will be run.

Distributed Agent Agent

The Distributed Agent Agent (DAAgent) can be viewed as the ‘Manger’ of tasks that have been submitted to the Distributed Agent Framework. Multiple DAAgents can be installed in a Extended ECM deployment, however only one DAAgent can be ‘active’ at any given time. Any other installed DAAgent will be in a suspended state and may be enabled in the event of an outage of the primary DAAgent.

The Distributed Agent Agent (DAAgent) operates in a cycle performing several different activities for the Distributed Agent Framework. Each process in this cycle is done sequentially.

Processing new inbound tasks

When tasks are newly created, they are inserted into a table in the database called WorkerQueue. The DAAgent will look at this table in blocks of 250 tasks in a First-In-First-Out method.

Within the block that it has taken from WorkerQueue the DAAgent will compare the fingerprints of the tasks and remove any duplicates. The deduplicated block of tasks will then be inserted into the WorkerQueuePending table.

The DAAgent will continuing taking new blocks of tasks from WorkerQueue until it has exhausted all the tasks in the table, or it has been looping over tasks for 30 seconds.

Distributed Agent Tasks that have a future “ActivationDate” will not be picked up by the DAAgent, they will remain in the WorkerQueue table until the “ActivationDate” has passed. This allows for solutions leveraging the Distribute Agent Framework to queue up specific tasks to be executed in the future.

Assign tasks to Workers

The DAAgent will evaluate if there are any idle workers, or workers which can accept more tasks into their queue. It does this by evaluating how many tasks are assigned to each worker in the WorkerQueueCurrent table.

If there are workers available, the DAAgent will then pull tasks from the WorkerQueuePending table and assign them to workers by moving those tasks to the WorkerQueueCurrent table.

Additionally at this phase the DAAgent will check the status of the Workers by pulling data from the WorkerStatus table. If a worker has gone offline, that workers assigned tasks will be reassigned.

Clean up WorkerQueueCurrent

Each task in WorkerQueueCurrent will be updated by the Worker when it has completed the task, or if there was an error. The DAAgent will scan this table and remove completed work to the WorkerQueueComplete table or in the event of an error to the WorkerQueueError table.

Evaluate Job Chains

Some tasks are created Job Chains. The DAAgent will check to see if all child tasks have been completed. Once all the child tasks have been completed it will mark the parent task to be enabled and executed by a worker.

Workers

The Distributed Agent Workers iterate through the tasks assigned to them in a First-In-First-Out method. Each Distributed Agent Worker can have up to 4,500 tasks assigned to it. As they execute the tasks that are assigned to them, they validate that there is not another task running with the same fingerprint in the cluster.

If another worker is executing a task with the same fingerprint, the 2nd worker will delay execution of that task until the previous task has been completed.

As tasks are executed the Worker will switch into the context of the user that submitted the task, executing the task with their permissions set.

Once a worker has completed a task, it will update the WorkerStatus table with its current state. It will also update the WorkerStatus table each time it queries for new work in the event it is idle.

Performance Considerations

Each Extended ECM deployment will be unique and may require an Administrator to familiarize themselves which tasks are most often created on their system and adjust workers accordingly.

Each Distributed Agent Worker can be configured to execute specific tasks ahead of others or ignore specific tasks. Administrators may want to prioritize some tasks such as Notification Center messages, or Records Management activities over tasks which are less important to their end users.

Back to post listing