Copyright (c) 2019. California Community Colleges Technology Center
Licensed under the MIT license.
A copy of this license may be found at https://opensource.org/licenses/mit-license.php
Canvas Data Warehouse (aka Canvas Data)
This microservice provides a connector to the Canvas Data Warehouse (by Instructure).
Part of this connector is to manage all tke access keys and authentication requirements when connecting to Canvas Data.
The endpoints allow downloading Canvas data for local access and, e.g., importing into another data warehouse.
Configuration
Standard Spring profile property files are used.
Running canvas-datawarehouse
Instructions below explain how to build and run this module in various ways.
IntelliJ
The IntelliJ Run Configuration can be configured as:
Main class: org.ccctech.canvas.datawarehouse.CanvasDWMicroservice
VM options: -Dspring.profiles.active=dev,local
Working Directory: $MODULE_DIR$
Environment Variables: canvasDW_misKeys_001.secret=[secret];canvasDW_misKeys_001.key=[key]
NOTE: The authenticationProperties key and secret above are expired and you'll have to create one associated to your account in the Canvas Data Portal
Docker
Configuration
Private registry credentials
By using the remote registry, you can run the 'latest' docker image and not need to build it locally.
To pull images from the remote private registry, follow the instructions on adding your user / pass to the registry are available here: https://cccnext.jira.com/wiki/spaces/DEVOPS/pages/129302935/Steps+to+deploy+Private+Docker+Registry+Authenticating+new+users
Docker network
To allow multiple services to communicate with each other, they must all join the same docker network. Ensure the 'ccctc' network is created locally by running the below:
docker network create ccctc
Docker Environment variables
The docker-compose.yml file uses the following environment variable:
Variable | Description |
---|---|
DOCKER_REGISTRY | URL to remote private registry or "" for local. |
CANVAS_DATAWAREHOUSE_001_APIKEY | Canvas Data API KEY (from college or canvas test instance |
CANVAS_DATAWAREHOUSE_001_APISECRET | Canvas Data API SECRET (from college or canvas test instance |
Set the environment variables from a terminal as in:
NOTE: The API_SECRET_001 and API_KEY_001 below are expired and you'll have to create one associated to your account in the Canvas Data Portal
export DOCKER_REGISTRY=registry.ccctechcenter.org:5000/
export CANVAS_DATAWAREHOUSE_001_APIKEY=9fcd0bb8caccf9086566e9586900118950c09a9c
export CANVAS_DATAWAREHOUSE_001_APISECRET=08170d40b5bb9694c3eaa5eb1eced2fef93d7012
and switch from remote to local registry by unsetting the DOCKER_REGISTRY with:
unset DOCKER_REGISTRY
Running
From the module root directory, start the container with:
docker-compose up
Stopping canvas-datawarehouse
With the above docker-compose command, you can CTRL-C to stop the processes. To ensure proper cleanup, follow that with:
docker-compose down
Testing canvas-datawarehouse Service
Postman
Import tests
Note: If you've modified the tests or environment in postman you should export them and check them in. (see below)
- Clone this git repo so you have a local copy.
- Open the Postman desktop app (Note: Postman app for Chrome will soon be removed, so upgrade to desktop app)
- Remove existing collections and environments for this repo to ensure you get a clean copy below.
- In the upper left, click the Import button
- The popup dialog, will show a "Drop files here" area. The easiest thing to do is open an explorer/file-browser and browse to your local copy of this git repo and into the src/test/postman folder
- Drag the collection and any environment files you want into the above popup.
- You'll get notifications at the top of the postman app that the environment files were imported (named with a "Copy" if the environment already existed). If the collection already exists, postman will prompt to overwrite your existing collection or import the new one as a copy.
Viewing API Documentation
With the latest latest tests imported (see Import tests)
In the postman desktop application, select the collection
Select the right arrow ('>') on the right side of collection to bring up actions for the collection as seen below:
Click the View in web button to bring up the API documentation in a browser.
In the browser window, in the upper right you can view details of the endpoints on the left pane. If you select an environment in the upper right of the page, and language in the right pane, the tests in the right will update with the syntax you can use for the various languages (cURL, jQuery, Ruby, Python Requests, Node, PHP, Go).
Modifying tests
If you modify and want to share your test, follow the below steps.
- In the postman desktop application, select the collection
- Select the ellipses ('...') on the right side of collection to bring up commands for the collection
- Choose the Export command
- In the next popup, choose the Collection v2.1 (recommended) option and click the Export button
- Browse to your local copy of this git repo and into the src/test/postman folder
- You can now commit and push the change to git as appropriate.
Automated Tests
During the build process, Jenkins utilizes the above to test the build and deployment in various environments. The build tests use a specific collection (defined in pom.xml) following the pattern {repo_name}_build-tests.postman_collection]. Any other collections checked in and shared in this folder are ignored by Jenkins.
You can run the build tests locally from the commandline with:
mvn process-test-resources -Ppostman,dev
Swagger
http://localhost/swagger-ui.html
Data Sync file list
When running in IDE
curl -s -H "Content-Type: application/json" \
-X GET "http://localhost:10101/dataSync/fileList?mis=001" | python -m json.tool
or in docker
curl -s -H "Content-Type: application/json" \
-X GET "http://localhost/dataSync/fileList?mis=001" | python -m json.tool