in

S3 Reverse Proxy with GET, PUT and DELETE methods and authentication (OpenID Connect and Basic Auth)


Go Doc CircleCI Go Report Card

Coverage Status Docker Pulls GitHub license GitHub release (latest by date)


Why ?

First of all, yes, this is another S3 proxy written in Golang.

I’ve created this project because I couldn’t find any other that allow to proxy multiple S3 buckets or to have custom templates with OpenID Connect authentication and also to get, upload and delete files.

Features

  • Multi S3 bucket proxy
  • Index document (display index document instead of listing when found)
  • Custom templates
  • AWS S3 Login from files or environment variables
  • Custom S3 endpoints supported
  • Basic Authentication support
  • Multiple Basic Authentication support
  • OpenID Connect Authentication support
  • Multiple OpenID Connect Provider support
  • Redirect to original host and path with OpenID Connect authentication
  • Bucket mount point configuration with hostname and multiple path support
  • Authentication by path and http method on each bucket
  • Prometheus metrics
  • Allow to publish files on S3 bucket
  • Allow to delete files on S3 bucket
  • Open Policy Agent integration for authorizations
  • Configuration hot reload
  • CORS support

Configuration

See here: Configuration

Templates

See here: Templates

Open Policy Agent (OPA)

See here: OPA and in the configuration here: OPA Configuration

API

GET

This kind of requests will allow to get files or directory listing.

If path ends with a slash, the backend will consider this as a directory and will perform a directory listing or will display index document. Example: GET /dir1/

If path doesn’t end with a slash, the backend will consider this as a file request. Example: GET /file.pdf

PUT

This kind of requests will allow to send file in directory.

The PUT request path must be a directory and must be a multipart form with a key named file with a file inside. Example: PUT --form file:@file.pdf /dir1/

DELETE

This kind of requests will allow to delete files (only).

The DELETE request path must contain the file name. Example: DELETE /dir1/dir2/file.pdf.

AWS IAM Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        // Needed for GET API/Action
        "s3:ListBucket",
        "s3:GetObject",
        // Needed for PUT API/Action
        "s3:PutObject",
        // Needed for DELETE API/Action
        "s3:DeleteObject"
      ],
      "Resource": ["arn:aws:s3:::<bucket-name>", "arn:aws:s3:::<bucket-name>/*"]
    }
  ]
}

Grafana Dashboard

This project exports Prometheus metrics. Here is an example of Prometheus dashboard that you can import as JSON file: dashboard.

This dashboard has been done and tested on Grafana 7.0.

Prometheus metrics

See here: Prometheus metrics

Deployment

Kubernetes – Helm

A helm chart have been created to deploy this in a Kubernetes cluster.

You can find it here: https://github.com/oxyno-zeta/helm-charts/tree/master/stable/s3-proxy

Docker

First, write the configuration file in a config folder. That one will be mounted.

Run this command:

docker run -d --name s3-proxy -p 8080:8080 -p 9090:9090 -v $PWD/conf:/proxy/conf oxynozeta/s3-proxy

TODO

  • Support more authentication and authorization systems
  • JSON response
  • Add tests

Want to contribute ?

Inspired by

Thanks

  • My wife BH to support me doing this

Author

  • Oxyno-zeta (Havrileck Alexandre)

License

Apache 2.0 (See in LICENSE)

GitHub

https://github.com/oxyno-zeta/s3-proxy




Samsung Galaxy M33 5G Might Feature a Massive 6000mAh Battery

Pop!_OS 21.10 Introduces a New Application Library, GNOME 40, and a Refresh Install Option