Copyright (c) 2019 California Community Colleges Technology Center
Licensed under the MIT license.
A copy of this license may be found at https://opensource.org/licenses/mit-license.php
S3 Sync
This module connects to and synchronizes a provided list of files with an AWS S3 bucket (and folder).
Configuration
Standard Spring profile property files are used.
AWS Credentials
You must provide credentials to access the configured S3 bucket where this module will access.
See the AWS Security Credentials page for more information on getting your keys.
There are a few options to set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your environment. Below for IntelliJ and Docker are examples.
S3 bucket name mappings
The AWS S3 bucket to use is passed as a 'locationKey' in the URL to the microservice. An example is:
{{S3Sync_URL}}/sync/folder/001?locationKey=develop
Here, the locationKey is 'develop'. This value is mapped under the property 's3.bucketKeys' defined in application or the ENV specific files application-dev, application-ci, application-qa.
Folders in bucket
The URL into the micro also specifies a folder that sits relative to the root of the bucket. So the above URL would be in the mapped 'develop' bucket in a subfolder named '001'.
Running S3-sync
Instructions below explain how to build and run this module in various ways.
IntelliJ
The IntelliJ Run Configuration can be configured as:
Main class: org.ccctech.S3sync.S3SyncApplication
VM options: -Dspring.profiles.active=dev,local
Working Directory: $MODULE_DIR$
Environment Variables: AWS_ACCESS_KEY_ID=[YOUR_AWS_KEY]; AWS_SECRET_ACCESS_KEY=[YOUR_AWS_SECRET]; AWS_REGION=us-west-2
Note: see Working with AWS Credentials for the options and search order the AWS SDK uses to locate your credentials. The above IDE example specifies the keys as environment variables, but they can also be configured in the ~/.aws directory.
Docker
Docker-Configuration
Private registry credentials
By using the remote registry, you can run the 'latest' docker image and not need to build it locally.
To pull images from the remote private registry, follow the instructions on adding your user / pass to the registry are available here: https://cccnext.jira.com/wiki/spaces/DEVOPS/pages/129302935/Steps+to+deploy+Private+Docker+Registry+Authenticating+new+users
Docker network
To allow multiple services to communicate with each other, they must all join the same docker network. Ensure the 'ccctc' network is created locally by running the below:
docker network create ccctc
Docker Environment variables
The docker-compose.yml file uses the following environment variable:
Variable | Description |
---|---|
DOCKER_REGISTRY | URL to remote private registry or "" for local. |
AWS_ACCESS_KEY_ID | see AWS Credentials |
AWS_SECRET_ACCESS_KEY | see AWS Credentials |
AWS_REGION | us-west-2 |
Set the environment variables from a terminal as in:
export DOCKER_REGISTRY=registry.ccctechcenter.org:5000/
export AWS_ACCESS_KEY_ID=[YOUR_KEY]
export AWS_SECRET_ACCESS_KEY=[YOUR_SECRET]
and switch from remote to local registry by unsetting the DOCKER_REGISTRY with:
unset DOCKER_REGISTRY
Running
From the module root directory, start the container with:
docker-compose up
Stopping S3-sync
With the above docker-compose command, you can CTRL-C to stop the processes. To ensure proper cleanup, follow that with:
docker-compose down
Testing S3-sync Service
Postman
Import tests
Note: If you've modified the tests or environment in postman you should export them and check them in. (see below)
- Clone this git repo so you have a local copy.
- Open the Postman desktop app (Note: Postman app for Chrome will soon be removed, so upgrade to desktop app)
- Remove existing collections and environments for this repo to ensure you get a clean copy below.
- In the upper left, click the Import button
- The popup dialog, will show a "Drop files here" area. The easiest thing to do is open an explorer/file-browser and browse to your local copy of this git repo and into the src/test/postman folder
- Drag the collection and any environment files you want into the above popup.
- You'll get notifications at the top of the postman app that the environment files were imported (named with a "Copy" if the environment already existed). If the collection already exists, postman will prompt to overwrite your existing collection or import the new one as a copy.
Viewing API Documentation
With the latest latest tests imported (see Import tests)
In the postman desktop application, select the collection
Select the right arrow ('>') on the right side of collection to bring up actions for the collection as seen below:
Click the View in web button to bring up the API documentation in a browser.
In the browser window, in the upper right you can view details of the endpoints on the left pane. If you select an environment in the upper right of the page, and language in the right pane, the tests in the right will update with the syntax you can use for the various languages (cURL, jQuery, Ruby, Python Requests, Node, PHP, Go).
Modifying tests
If you modify and want to share your test, follow the below steps.
- In the postman desktop application, select the collection
- Select the ellipses ('...') on the right side of collection to bring up commands for the collection
- Choose the Export command
- In the next popup, choose the Collection v2.1 (recommended) option and click the Export button
- Browse to your local copy of this git repo and into the src/test/postman folder
- You can now commit and push the change to git as appropriate.
Automated Tests
During the build process, Jenkins utilizes the above to test the build and deployment in various environments. The build tests use a specific collection (defined in pom.xml) following the pattern {repo_name}_build-tests.postman_collection]. Any other collections checked in and shared in this folder are ignored by Jenkins.
You can run the build tests locally from the commandline with:
mvn process-test-resources -Ppostman,dev
Swagger
When running micro locally: http://localhost:10102/swagger-ui.html
Limitations
the /saveFile endpoint takes the entire file contents and doesn't use a muli-part form or other options - it's intended for relatively small files.