Clojars Worklog

Table of Contents


This worklog covers work performed on Clojars under a limited contract from Clojurists Together and the Software Freedom Conservancy. The bulk of this work is moving the Clojars infrastructure from Rackspace to AWS, but also will include security improvements and general maintenance.

December 2019

Changelogs: clojars-web, clojars-server-config

This month I worked on moving Clojars from an embedded SQLite database to PostgreSQL. This was the first step in allowing the Clojars server to be ephemeral, since we can't keep the db on disk to satisfy that. This included setting up PostgreSQL on the Rackspace server and setting up hourly/daily backups to Rackspace Cloudfiles along with adjustments to queries within the app that relied on SQLite quirks.

January 2020

Changelogs: clojars-web, clojars-server-config

The work in January focused on moving the collected usage statistics storage from the server to Amazon S3. This required changing all code that generated and used the stats to use S3 instead. This also required modifying the Fastly configuration to serve the stats from S3.

I also changed the server Nginx config to redirect all requests made to the on-server repository to the CDN-backed repository to prepare for removing the repository from the server.

February 2020

Changelogs: clojars-web, clojars-server-config

This month I moved the repository from Rackspace Cloudfiles to Amazon S3, and adjusted the Fastly CDN to serve artifacts from S3. This also involved modifying the feed generation logic to upload to the S3 repository bucket instead of to Cloudfiles.

I also set up PostgreSQL on Amazon RDS and modified the Rackspace server to use that database instead of the local one.

March 2020

Changelogs: clojars-web, clojars-server-config

This month I moved the server from Rackspace to AWS EC2. This involved:

  • Updating the app to send email via AWS SES
  • Moving all email receiving (for users and aliases) from the server to Amazon Workmail
  • Modifying the Ansible configuration to set up a new server on EC2
  • Running a beta of the new server setup
  • Switching all traffic to the new server
  • Removal of all Cloudfiles dependencies from the codebase

April 2020

Changelogs: clojars-web, clojars-server-config

This month included work finishing up the migration to AWS, including:

  • Setting up a way to build new AMIs for the server
  • Modifying the deploy process to allow new instances to deploy the current release
  • Moving the server to an auto-scaling group to make it more robust and allow zero-downtime new AMI releasing
  • Documenting the new server architecture
  • Upgrading from Java 8 to Java 11

I then started on improving security. The work here was focused on adding a deploy token feature. This was released in early May, but 90% of the work was completed in April.

I also fixed an issue where the cookies weren't being set as secure.

May 2020

Changelogs: clojars-web, clojars-server-config

I continued working on deploy tokens, adding:

I also implemented the bulk of two-factor authentication. This wasn't released in May, but will be released by mid-June.

Part of the two-factor implementation is an internal eventing system that will make it easier to add additional email notifications in the future.

June 2020

Changelogs: clojars-web, clojars-server-config

This month was focused on security and monitoring improvements. I:

Author: Toby Crawley

Created: 2020-07-04 Sat 09:19