Clojars Worklog - 2020

This worklog covers work performed on Clojars under a limited contract from Clojurists Together and the Software Freedom Conservancy. The bulk of this work is moving the Clojars infrastructure from Rackspace to AWS, but also will include security improvements and general maintenance.

Changelogs: clojars-web, clojars-server-config

This month I worked on moving Clojars from an embedded SQLite database to PostgreSQL. This was the first step in allowing the Clojars server to be ephemeral, since we can't keep the db on disk to satisfy that. This included setting up PostgreSQL on the Rackspace server and setting up hourly/daily backups to Rackspace Cloudfiles along with adjustments to queries within the app that relied on SQLite quirks.

Changelogs: clojars-web, clojars-server-config

The work in January focused on moving the collected usage statistics storage from the server to Amazon S3. This required changing all code that generated and used the stats to use S3 instead. This also required modifying the Fastly configuration to serve the stats from S3.

I also changed the server Nginx config to redirect all requests made to the on-server repository to the CDN-backed repository to prepare for removing the repository from the server.

Changelogs: clojars-web, clojars-server-config

This month I moved the repository from Rackspace Cloudfiles to Amazon S3, and adjusted the Fastly CDN to serve artifacts from S3. This also involved modifying the feed generation logic to upload to the S3 repository bucket instead of to Cloudfiles.

I also set up PostgreSQL on Amazon RDS and modified the Rackspace server to use that database instead of the local one.

Changelogs: clojars-web, clojars-server-config

This month I moved the server from Rackspace to AWS EC2. This involved:

• Updating the app to send email via AWS SES
• Moving all email receiving (for users and aliases) from the server to Amazon Workmail
• Modifying the Ansible configuration to set up a new server on EC2
• Running a beta of the new server setup
• Switching all traffic to the new server
• Removal of all Cloudfiles dependencies from the codebase

Changelogs: clojars-web, clojars-server-config

This month included work finishing up the migration to AWS, including:

• Setting up a way to build new AMIs for the server
• Modifying the deploy process to allow new instances to deploy the current release
• Moving the server to an auto-scaling group to make it more robust and allow zero-downtime new AMI releasing
• Documenting the new server architecture
• Upgrading from Java 8 to Java 11

I then started on improving security. The work here was focused on adding a deploy token feature. This was released in early May, but 90% of the work was completed in April.

I also fixed an issue where the cookies weren't being set as secure.

Changelogs: clojars-web, clojars-server-config

I continued working on deploy tokens, adding:

• An endpoint that GitHub will use to report when a deploy token is found in source code that will notify the user and disable the token
• The option to scope a deploy token to a group or artifact

I also implemented the bulk of two-factor authentication. This wasn't released in May, but will be released by mid-June.

Part of the two-factor implementation is an internal eventing system that will make it easier to add additional email notifications in the future.

Changelogs: clojars-web, clojars-server-config

This month was focused on security and monitoring improvements. I:

• finished and released two-factor authentication
• implemented logging to provide more insight into the behavior of the application
• added rate-limiting to the password reset endpoint
• removed the ability to deploy artifacts using a password; a deploy token is now required