Enabling Services Documentation

Enabling Services Troubleshooting Documentation

Release and Deploy Process

The standard deploy process can be seen here.

The College Adaptor deploy process needs to be updated, but for context you can refer to the Confluence document here.

SuperGlue Data Integrations Framework Overview

The SuperGlue Integrations Framework facilitates data exchange and workflows across multiple CCCTC and external systems. Below is a graphic depicting the processing and delivery of multiple measures student placements to colleges via SuperGlue:

  1. Student applies to college via CCCApply and presses Submit

  2. Math and English placement recommendations are calculated using any/all available transcript data, in alignment with AB-705 statewide rulesets. See the Transcript data sources below)

    Transcript data sources (in priority order):

    • California College Guidance Initiative (CCGI) - CaliforniaColleges.edu
    • Cal-PASS Plus
    • California Department of Education (CDE) record (pending as MOU w/CCCCO was just signed)
    • Self-Reported via CCCApply opt-in screens

  3. Placement recommendations are delivered to the college via SuperGlue Placement data delivery to the colleges will be via one of the two methods below at each college’s request:

    • A downloadable .csv file via Placement Adapteor (AWS S3) (fall 2018)
    • Writing to a staging table in the college’s SIS via College Adaptor (spring 2019)

When Delivery of SuperGlue Transactions to a College Fails

Delivery of SuperGlue Transactions to a college can fail for a variety of reasons:

  • The college’s systems are down for a few hours of routine maintenance, or because an emergency shuts them down for several days or weeks. Wildfires, earthquakes and other natural disasters common to California can cause a systems shut down of many weeks in duration. When this happens, the colleges want to know that when they return to normal operation the student application data, placement data and any other transactions can be recovered/re-sent without permanent loss of this crucial data.

  • External-partner components may have an unexpected failure for any number of reasons, and the CCCTC team must have a process to follow to return the key component back to a functional status.

SuperGlue Transactions Failure Process

When a college’s systems are down and SuperGlue is unable to write transactions to staging tables or folder locations within the college’s network, the following processes are initiated:

  1. When delivering to a college’s SIS fails, the College Adaptor reports a downed state for a specific MIS code to New Relic. The Infiniti DevOps team receives the New Relic notifications. Sometimes these are expected (Need the ability for the adaptor to pickup pre-pending tasks in the queue for a long time due to adaptor down) blackouts due to routine backups or other routine processes at the colleges. The ability of the adaptor to process a pending workflow task is not hindered, but the attempt to deliver to the unresponsive SIS endpoint would generate an SQL delivery error and fail the workflow transaction.
  2. When the college adaptor sends “status down” in response to a conductor workflow, this indicates that the conductor workflow is working, but something may be happening at the district. At this point the CCCTC Support team will need to reach out via the district’s/college’s escalation contact email group. These email contacts are getting organized by Enabling Services CRMs. The district/college will have an escalation email for the Technology Center as well so that they can initiate communication. online.
  3. Conductor reports failed transaction errors in the slack channels listed below, a separate channel per environment. Failed transactions are stored in Conductor for two weeks.
    • #conductor-err-CI
    • #conductor-err-QA
    • #conductor-err-PILOT
    • #conductor-err-PROD
  4. When the college’s systems are back online after a long down cycle and SuperGlue transactions can be delivered again, there are different processes depending on the type of transaction.
    1. For CCCApply student applications, these transactions can be re-sent by a CCCTC staff with access to the CCCApply spam filter or by the replay of a failed Conductor college-adaptorWF workflow specific for the /apply URI.
    2. Should delivery of multiple measures student placements to the CSV file located within the College’s environment be stopped, the placement data will persist in the CCCTC’s AWS S3 folders until such time as the placement adapter is brought back up and requests them. The placement data is also preserved in S3 in an /archive folder after it's been downloaded to the college, in the event it's been lost by the college, or an audit needs to be performed.
    3. Should delivery of the placement data to AWS be interrupted, the placement data transactions would need to be re-send manually by a CCCTC staff member with access to Conductor. See Re-sending Failed Transactions From Conductor.

Re-Sending Failed Transactions From Conductor

Conductor workflows can be filtered in order to isolate transactions which failed to complete successfully. These transactions will contain a status of FAILED.

CCCApply applications which are expected to be delivered to an SIS staging table will be presented to the Adaptor using a college-adaptorWF workflow transaction. To isolate applications targeted to a specific college, the following free filter can be submitted with the full string option enabled. uri=apply, httpMethod=POST, parameterMap={mis=nnnnnn represents the MIS code of the college that the application is targeted for.

The resulting set of filtered transactions may contain COMPLETED as well as FAILED status. Further filtering can be accomplished by setting the Status filter to FAILED in order to focus on these specific transactions. Clicking on the UUID reference for a transaction will open a more detailed presentation. Within this new presentation, the ability to replay the transaction becomes possible by clicking on the button labeled RETRY.

A similar process can be utilized for MMPS delivery failures utilizing an alternate filter.

uri=student-placements, httpMethod=POST, parameterMap={mis=nnn

MMPS-Specific Integration Failure Process

If the Multiple Measures Placement Service (MMPS) process stops or fails for any reason in PILOT, the Enabling Services team should monitor it and follow the process below.

  1. Monitor the #conductor-err-pilot Slack channel for every single error. (Until a problem is fixed, we’ll keep getting errors and that makes the channel very noisy.)
  2. For every error, resolve it using the following steps:

    1. Determine the potential root cause based on the Slack channel conductor-err-pilot error message, Conductor, and the items below: Note: Gurpreet confirmed that she believe both Martin and Ramya on the ES team have both Conductor and Postman access.

      Issue: College Adaptor, i.e. if the college-adaptor-wf is failing

      Action:

      • Contact the college with the college adaptor that has the issue to ensure it isn’t down (using the email list you have for college staff)
      • Check: Verify the issue by running GET /terms via Postman; if you get a 500 response, then you know that college’s college adaptor is definitely down
      • Can’t resolve? Go to step ii below to create a JIRA and assign to a developer.

      Issue: CCGI, i.e. a 500 error is occuring for a CCGI workflow

      Action:

      • Check the Conductor workflow to confirm it is a CCGI issue, if possible.
      • Create a JIRA following step 2b, below. Copy Ben Stout, James Whetstone, and Ben Baird on the JIRA bug

      Issue: ERP issue, i.e. a 500 error is occuring for an ERP workflow

      Action:

      • Check the Conductor workflow to confirm it is an ERP issue, if possible.
      • Create a JIRA following step 2b, below. Copy Ben Stout, James Whetstone, and Dan Lamoree on the JIRA bug

    2. Create a JIRA for the development team in the Conductor: Technology Platform BRB Backlog sprint. (i.e. CCCTC JIRA Dashboard -> Conductor: Technology Platform -> Backlog -> scroll down to BRB -> Create Issue link below the BRB board -> ensure the bug icon is selected -> start typing the bug/issue.

    3. Enter details of the bug or issue--i.e. The steps to reproduce, any logs, Postman errors, or workflow identifiers, any potential third-party causes, etc.

    4. Assign the JIRA to the appropriate developer (Ben Stout, James Whetstone, and/or any third-party application contact (Ben Baird for CCGI and Dan Lamoree for ERP).

    5. Click the Start watching this issue link for the JIRA so you can see email updates as the issue is worked/resolved.

  3. Once the error is resolved, follow up with the college, if College Adaptor related, or with QA (Gurpreet) to confirm resolution is in place.
Super Glue →