Conductor Simple Queue System (SQS) Support

Amazon's SQS is a queue where one subscriber for a queue gets notified when a new message is published.

SNS is a topic where each subscriber on a queue gets notified with a copy of the message.

Business case

Short term, we need SQS in conductor to send/receive messages with MDM.
We'll need SQS when integrating with the MyPath/cccApply team for student application/profile data.
Long term, SQS is an attractive alternative to REST calls and an option we'll want to build upon. So, we want something robust and extensible.

Overview

The "out of box" solution for Amazon SQS in Conductor is deemed insufficient for our needs. The reasons are:

It uses deprecated AWS API calls
There are no hooks for AWS configuration or Guice binding for AWS security credentials. You must modify the plugin source code to let it pick up AWS security credentials.
There seems to be no configuration for the AWS queue name. Documentation and examples have this "hard coded" in mock tests in java - not in conductor workflows or tasks or property files. It seems we would need to modify their source to configure the queue names or any custom queue properties we may want.

So, the included plugin solution seems more like a starting point or an example to build from, not a drop-in working solution that's ready to use.

Design

The below diagram shows an example usage of SQS to integrate with CCC Apply and MDM. This is an example, full details and design of this are contained elsewhere.

We'll create a new Guice module that implements com.google.inject.Module or extends com.google.inject.AbstractModule

We'll add this class to the CSV of classes implementing a Guice Module:

    conductor.additional.modules=org.ccctech.apigateway.conductor.SpringBootstrap,org.ccctech.apigateway.conductor.modules.sqs.SQSModule

In the service-conductor repo, the property file is config-extended.properties

Plugin

The plugin examples from Conductor live in their contribs package

The ContribsModule.java (for SQS) and NatsModule.java (for NATS) give different implementations for queues and we borrow heavily from both.

The plugin class should only need to override the configure method and provide Guice bindings - analgous to Spring injection with @Resource

EventQueueProvider

We'll need a class that implements EventQueueProvider. EventQueueProvider is a core conductor interface that exposes queues and allows them be hooked into queues used in Conductor workflows:

public interface EventQueueProvider { 
    ObservableQueue getQueue(String queueURI);
}

Implementation of the getQueues(..) method will return a class implementing ObservableQueue. The SQS implementation of ObservableQueue will need to use an instance of AmazonSQS class to interact with the AWS SQS for things like receiving, acknowledging, deleting and publishing messages.

Deployment

We'll want each instance of conductor to subscribe to SQS and help distribute the load. So, nothing special with regards to one instance versus another.

With this implementation will come the need for AWS credentials to be provided at the container level. Examples of this that worked elsewhere included the s3-sync module.

Limitations

This design is only concerned about SQS queues - not topics. Many times an SQS queue will be a subscriber to an SNS topic, then there's guaranteed delivery to (one) SQS subscriber. and any subscribers
Bootstrap code exits that adds Spring dependency injection as a way to make that code more similar to the rest of our code. However, Conductor uses Google Guice and it's likely we should port that code to a Guice only DI framework and remove the additional Spring code - as Conductor won't use it anyway.

CCCTC Docs