Copyright (c) 2019. California Community Colleges Technology Center
Licensed under the MIT license.
A copy of this license may be found at https://opensource.org/licenses/mit-license.php

Canvas Data Warehouse (aka Canvas Data)

This microservice provides a connector to the Canvas Data Warehouse (by Instructure).

Part of this connector is to manage all tke access keys and authentication requirements when connecting to Canvas Data.

The endpoints allow downloading Canvas data for local access and, e.g., importing into another data warehouse.

Configuration

Standard Spring profile property files are used.

Running canvas-datawarehouse

Instructions below explain how to build and run this module in various ways.

IntelliJ

The IntelliJ Run Configuration can be configured as:

Main class: org.ccctech.canvas.datawarehouse.CanvasDWMicroservice
VM options: -Dspring.profiles.active=dev,local
Working Directory: $MODULE_DIR$ 
Environment Variables: canvasDW_misKeys_001.secret=[secret];canvasDW_misKeys_001.key=[key]

NOTE: The authenticationProperties key and secret above are expired and you'll have to create one associated to your account in the Canvas Data Portal

Docker

Configuration

Private registry credentials

By using the remote registry, you can run the 'latest' docker image and not need to build it locally.

To pull images from the remote private registry, follow the instructions on adding your user / pass to the registry are available here: https://cccnext.jira.com/wiki/spaces/DEVOPS/pages/129302935/Steps+to+deploy+Private+Docker+Registry+Authenticating+new+users

Docker network

To allow multiple services to communicate with each other, they must all join the same docker network. Ensure the 'ccctc' network is created locally by running the below:

docker network create ccctc
Docker Environment variables

The docker-compose.yml file uses the following environment variable:

VariableDescription
DOCKER_REGISTRYURL to remote private registry or "" for local.
CANVAS_DATAWAREHOUSE_001_APIKEYCanvas Data API KEY (from college or canvas test instance
CANVAS_DATAWAREHOUSE_001_APISECRETCanvas Data API SECRET (from college or canvas test instance

Set the environment variables from a terminal as in:

NOTE: The API_SECRET_001 and API_KEY_001 below are expired and you'll have to create one associated to your account in the Canvas Data Portal

export DOCKER_REGISTRY=registry.ccctechcenter.org:5000/
export CANVAS_DATAWAREHOUSE_001_APIKEY=9fcd0bb8caccf9086566e9586900118950c09a9c
export CANVAS_DATAWAREHOUSE_001_APISECRET=08170d40b5bb9694c3eaa5eb1eced2fef93d7012

and switch from remote to local registry by unsetting the DOCKER_REGISTRY with:

unset DOCKER_REGISTRY

Running

From the module root directory, start the container with:

docker-compose up

Stopping canvas-datawarehouse

With the above docker-compose command, you can CTRL-C to stop the processes. To ensure proper cleanup, follow that with:

docker-compose down

Testing canvas-datawarehouse Service

Postman

Import tests

Note: If you've modified the tests or environment in postman you should export them and check them in. (see below)

  1. Clone this git repo so you have a local copy.
  2. Open the Postman desktop app (Note: Postman app for Chrome will soon be removed, so upgrade to desktop app)
  3. Remove existing collections and environments for this repo to ensure you get a clean copy below.
  4. In the upper left, click the Import button
  5. The popup dialog, will show a "Drop files here" area. The easiest thing to do is open an explorer/file-browser and browse to your local copy of this git repo and into the src/test/postman folder
  6. Drag the collection and any environment files you want into the above popup.
  7. You'll get notifications at the top of the postman app that the environment files were imported (named with a "Copy" if the environment already existed). If the collection already exists, postman will prompt to overwrite your existing collection or import the new one as a copy.

Viewing API Documentation

With the latest latest tests imported (see Import tests)

  1. In the postman desktop application, select the collection

  2. Select the right arrow ('>') on the right side of collection to bring up actions for the collection as seen below:

    postman_collection_actions

  3. Click the View in web button to bring up the API documentation in a browser.

  4. In the browser window, in the upper right you can view details of the endpoints on the left pane. If you select an environment in the upper right of the page, and language in the right pane, the tests in the right will update with the syntax you can use for the various languages (cURL, jQuery, Ruby, Python Requests, Node, PHP, Go).

Modifying tests

If you modify and want to share your test, follow the below steps.

  1. In the postman desktop application, select the collection
  2. Select the ellipses ('...') on the right side of collection to bring up commands for the collection
  3. Choose the Export command
  4. In the next popup, choose the Collection v2.1 (recommended) option and click the Export button
  5. Browse to your local copy of this git repo and into the src/test/postman folder
  6. You can now commit and push the change to git as appropriate.

Automated Tests

During the build process, Jenkins utilizes the above to test the build and deployment in various environments. The build tests use a specific collection (defined in pom.xml) following the pattern {repo_name}_build-tests.postman_collection]. Any other collections checked in and shared in this folder are ignored by Jenkins.

You can run the build tests locally from the commandline with:

mvn process-test-resources -Ppostman,dev

Swagger

http://localhost/swagger-ui.html

Data Sync file list

When running in IDE

curl -s -H "Content-Type: application/json" \
    -X GET "http://localhost:10101/dataSync/fileList?mis=001" | python -m json.tool

or in docker

curl -s -H "Content-Type: application/json" \
    -X GET "http://localhost/dataSync/fileList?mis=001" | python -m json.tool

Limitations