Copyright (c) 2019 California Community Colleges Technology Center
Licensed under the MIT license.
A copy of this license may be found at https://opensource.org/licenses/mit-license.php

Service Conductor

This module is part of the API Gateway.

The service-conductor module extends the Netflix Conductor OSS package. Netflix Conductor is the workflow engine that provides queues, scheduling, load balancing, monitoring and exception handling for calls to microservices.

Prerequisites

Instructions and pre-requisites for running these services together can be found in the overall API Gateway documentation here: development-environment.

Local Development Setup

Background: Conductor workflow and task definitions are defined in various scripts in init-scripts. Workflow configuration is pulled from the service-conductor-config repository.

  • The ssh private key is set locally when running build.sh in Step 3 during "Building and Running service-conductor Locally" or with the below command.
  • Additional configuration for our custom integrations are outlined in Conductor-Customization.
  • Refer to the Netflix docs for general configuration of Conductor.

Follow the steps below to get service-conductor working locally.

Step 1. Set your AWS credentials into the local environment using the following:

export AWS_ACCESS_KEY_ID="...your aws key..."
export AWS_SECRET_ACCESS_KEY="...your aws secret..."

  • You must provide credentials to access the configured SQS Queues running on AWS. See the AWS Security Credentials page for more information on getting your keys.

  • If you need a CCCTC AWS account set up because you don't have one, submit a request to Infiniti at: https://infiniticg.zendesk.com/).

Step 2. If you need to incorporate scheduling support:

Use the CCCTC Crontab service. Conductor has no built in scheduling system.

Step 3. Set Slack notifications:

To incorporate the notification workflow that runs periodically to find FAILED workflows use one of the two following options for local development:

Option 1:

  • Set up a Slack webhook to send to your private channel in Slack (so that only you can see your annoying tests).

  • Follow directions here: https://api.slack.com/incoming-webhooks

    Once you have your URL, add it to your environment:

      export SLACK_NOTIFICATION_API=(your url)

Option 2:

  • As an alternative to the above, contact Ben Stout to get access to the shared devops Slack webhook and add it to your bash profile.

Step 4. To work with SQS queues and event messages:

  • To maintain message order (FIFO), we should use a FIFO SQS queue, not the "standard" queue.

  • For testing an SQS queue, Conductor will subscribe to a queue linked to the movies test service. The queue is configured in Docker with the property.

      CCCTC_MOVIES_INBOUND=gateway_movies-list_dev.fifo

    • To subscribe to events, a Conductor event handler must be defined for the queue as seen in 7-events.sh.

    • The queue name must end in '.fifo' and refer to a FIFO queue

    • If the queue doesn't exist on AWS, we'll create the queue with default config

Building and Running service-conductor

Typically you'll run all three components of the API-Gateway stack locally: service-router, service-conductor and service-workers.

You can run service-conductor locally using one of the two following methods:

  • Docker and the CCCTC Private Registry, or

  • Build and run the Docker images locally

Pulling and Running service-conductor Locally Using the CCCTC Private Registry

Use the instructions below to run and build service-conductor using the CCCTC private registry.

Prerequisite Follow the instructions on adding your user / pass to the registry here to set your credentials for pulling from the CCCTC private registry.

Step 1. To allow multiple services to communicate with each other, they must all join the same Docker network. Ensure the 'ccctc' network is created locally by running the below:

docker network create ccctc

Step 2. Use the following command if you have NOT built the images locally but want to pull already-built images:

export DOCKER_REGISTRY=registry.ccctechcenter.org/

Note: You can then switch back to local images with:

unset DOCKER_REGISTRY

Step 3. Login to the CCCTC Docker registry:

docker login registry.ccctechcenter.org:5000

Step 4. Use the following command in the root directory to run Conductor locally using the CCCTC base image:

./build.sh local restart

Step 5. After running the above docker-compose command you can CTRL-C to stop the service-conductor processes. To ensure proper cleanup, follow that with:

docker-compose down

Building and Running service-conductor Locally

See below on how to build and run images locally.

Step 1. To allow multiple services to communicate with each other the services must all join the same Docker network. Ensure the 'ccctc' network is created locally by running the below:

docker network create ccctc

Step 2. Build the conductor-server image (with your changes) by executing the following build script from the module root directory:

./build.sh local

This packages all the workflows, configs and scripts into an image you can run locally. Normally, we run Conductor as a Docker image only.

Note: You may also want to build and restart just conductor-server, which can be done with this command:

./build.sh local restart

Running Conductor locally

After building the images locally, use the following command in the root directory to run Conductor locally:

docker-compose up

After running the above docker-compose command, you can CTRL-C to stop the service-conductor processes. To ensure proper cleanup, follow that with:

docker-compose down

Netflix OSS Base Image
Netflix forked codehttps://github.com/ccctechcenter/conductor
Docker hub imageshttps://hub.docker.com/repository/docker/ccctechcenter/conductor

Docker images are auto built when a tag matching the pattern *.RELEASE is pushed to above fork.

This repo uses the ccctechcenter/conductor:server-[VERSION] and ccctechcenter/conductor:ui-[VERSION] images.

The steps below outline how to use a new Docker Hub image.

  1. For this service-conductor repo, create a branch as usual, i.e., feature/CONDUCTOR-283_upgrade-conductor

  2. Modify the docker-compose.yml, compose-base.yml and Dockerfile files with the new version.

  3. After validating in CI, create a pull request and merge into develop as normal.

  4. Finally, modify the service-router and service-worker POM files to use the new conductor library version on maven central.

See Conductor-Customization for adding new features and extending the base image.

Resources

Sentry IO

Sentry IO is error tracking software that has been integrated in the project. The integration relies on the existence of an environment variable SENTRY_DSN (the data source name). The data source name is a URL that provides endpoint information to the sentry IO subsystem. When the environment variable is not present, the Sentry IO subsystem is completely disabled.