Bootstrapping Trust on Kubernetes

Posted on Wed 15 November 2023 in Infrastructure • Tagged with engineering, aws, kubernetes

We use Kubernetes at $work, and since I am in charge of platform, Kubernetes is my problem. Here's an interesting problem when trying to secure your Kubernetes workload.

Our pods need to talk to each other over the network. Early on, we decided that each pod would receive a unique …


Continue reading

AWS Support Pricing

Posted on Sat 11 March 2023 in Infrastructure • Tagged with aws

Wherein I praise AWS support, complain about Amazon's stupid support billing model, tell you how to get out of paying support costs, and then hope that my loophole will poke AWS to provider cheaper organization-wide support plans.


Continue reading

Serializing CircleCI

Posted on Sun 17 April 2022 in Infrastructure • Tagged with ci, tooling

We extensively use CircleCI as our CI provider. We run our entire test suite on CircleCI machines for all new commits in all PRs across our microservice repositories.

image of circleci logo

Like most other CI providers, they integrate with Github's webhooks so that CircleCI gets pinged to start running CI tests whenever new …


Continue reading

SQS Performance (II)

Posted on Mon 22 November 2021 in Infrastructure • Tagged with aws, performance, sqs

I continue hunting my SQS performance white whale and learn that SQS hangs up every 80 messages.


Continue reading

SQS's Slow Tail Latency

Posted on Wed 11 August 2021 in Infrastructure • Tagged with aws, performance, sqs

We use AWS Simple Queue Service a lot. But man, the tail latency really sucks.


Continue reading

MySQL Proxies

Posted on Thu 06 August 2020 in Infrastructure • Tagged with aws, performance

Our Django application connects to MySQL to store all of its data. This is a pretty typical setup. However, you may not know that establishing connections to MySQL is rather expensive.

MySQL Connections

I have written previously about controlling API response times -- we take it very seriously! One thing that …


Continue reading

Rate Limiting

Posted on Tue 19 February 2019 in Infrastructure • Tagged with redis, abuse

Being DOSd by a CI error led us to build rate limiting into our product.


Continue reading

AWS Cognito Limitations

Posted on Wed 27 June 2018 in Infrastructure • Tagged with aws

AWS Cognito is nice to get up and running quickly with. But after extensive usage, I have a few problems with it.


Continue reading