Bassist · Improviser · Computer Wrangler

Your New Engineer is Working Too Many Hours

By S. Tilmon Edwards on 2015-07-31

Here’s a scenario: You’re an engineering manager, and you’ve just hired a new junior developer or three, fresh out of college. They aren’t producing much yet, but you know that’s ok. They’re right out of college, after all. It takes some time for new graduates to be productive the way you expect other engineers to be productive.

They don’t know that yet.

Consider the possibility that they aren’t working late because they’re ruthless and dedicated workers who only need exposure to your wisdom and experience to become a smashing success. They’re working late because they’re worried that they’re not meeting expectations. They know what everyone around them is producing, and they know they aren’t keeping up with that pace. They’re trying to compensate with time.

You could let them do that until they burn out. But you could also tell them it’s almost 6:00 and they should go home.

Why am I using swap space when I have plenty of RAM?

And what the heck is 'swappiness' anyway?
By S. Tilmon Edwards on 2015-07-29

Most savvy computer users are at least vaguely familiar with the concept of swap space, or the Windows page file. Here’s some disorganized, high-level information on Linux swap space.

The kernel can send memory to swap even when it doesn’t have to. The knob for controlling how likely it is to do this is called swappiness and it ranges from 0 to 100. On many systems, the default is set to 60. If set to 0, the kernel will only swap out pages to avoid actually running out of physical memory. If you set swappiness to 0, you should probably have lots of ram and a pretty good reason. Having RAM headroom is usually a good thing. Nobody wants to block their memory allocations on disk writes.

Having memory pages in swap isn’t necessarily bad. Sure, they’ll take longer to access if they’re needed, but computers are complex, and at any given time there is plenty of data in memory that just isn’t needed in a normal running system. Kernel code for drivers that aren’t needed by any of the hardware attached to that system probably don’t need to be taking up comparatively valuable RAM space.

What you might want to look out for, however, is swap thrashing. That happens when, for one reason or another, the kernel keeps sending memory to swap to and from swap. I would expect this to happen most often when the system doesn’t have quite enough physical RAM for the workload, but I’m sure there are plenty of other ways to induce that behavior.

You can use vmstat to monitor swap activity, among other useful things. As always, consult your local man pages for detailed usage. My favorite incantation is:

vmstat \ # Probably already installed on your system
  -w \   # Make the columns wider, to accomidate lots of digits of RAM.
  1      # And poll every second

Pay partiuclar attention to the si (swap in) and so (swap out) columns. Those columns indicate the number of bytes that were swapped during that polling interval.

The performance impact of swapping is dependent on a lot of factors, so without controlling for any of those, the only definitive guideline for whether swapping is hurting performance is to say that if si and so are flat zero, swapping is probably not the problem.

Otherwise, if it’s not zero, you’re swapping, and if it’s extremely not zero, you’re swapping even more than that. SSD-backed swap space will make swapping faster. Magnetic-backed network swap space will probably make swapping very painful. Keep your disk close, and your swap closer.

If I’m wrong about any of this stuff, please leave me a constructive comment so I can fix the problem.

Cfer Part 1: Creating a VPC with CloudFormation

Practical infrastructure, easy mode
By S. Tilmon Edwards on 2015-07-28

When I first encountered AWS CloudFormation, I was appalled by the format for about ten minutes. This is the criticism of it that I’ve heard most often: the unwieldy syntax makes it unapproachable. Writing it by hand is worse than lunch meetings. I don’t disagree.

However, I think CloudFormation makes a pretty decent intermediate language. Users of Elasic Beanstalk might notice a bunch of CloudFormation stacks in their account, all containing some kind of Amazon-produced EB magic. This is where CloudFormation excels: Machine-generated infrastructure.

To that end, I took a couple weekends and wrote Cfer, a DSL for generating CloudFormation templates in Ruby. I’m not the only one doing this. But I’ll run through an example template, which will help you build a basic VPC in AWS, and at the same time, address some of the features of Cfer that might make CloudFormation a little more appealing to you.

If you find the format of this post difficult to follow, you can see this same example as a fully functional template in examples/vpc.rb

This template creates the following resources for a basic beginning AWS VPC setup:

  1. A VPC
  2. A route table to control network routing
  3. An Internet gateway to route traffic to the public internet
  4. 3 subnets, one in each of the account’s first 3 availability zones
  5. A default network route to the IGW
  6. Associated plumbing resources to link it all together

Template Parameters

Template parameters allow you to use the same template to build multiple similar instances of parts of your infrastructure. Parameters may be defined using the parameter function:

parameter :VpcName, default: 'Example VPC'

Resources are created using the resource function, accepting the following arguments:

  1. The resource name (string or symbol)
  2. The resource type. See the AWS CloudFormation docs for the available resource types.

The VPC Resource

The VPC is the foundation of your private network in AWS.

resource :vpc, 'AWS::EC2::VPC' do

Each line within the resource block sets a single property. These properties are simply camelized using the ActiveSupport gem’s camelize function. This means that the cidr_block function will set the CidrBlock property.

cidr_block ''

Following this pattern, enable_dns_support sets the EnableDnsSupport property.

enable_dns_support true
enable_dns_hostnames true
instance_tenancy 'default'

The tag function is available on all resources, and adds keys to the resource’s Tags property. It accepts the following arguments:

  1. Tag name (symbol or string)
  2. Tag value
tag :DefaultVpc, true

Parameters are required at template generation time, and therefore may be referenced using the parameters hash anywhere in a template. This will render the parameter value as a string constant in the CloudFormation JSON output.

tag :Name, parameters[:VpcName]

Finally, we can finish this resource by closing the block that we started when we called the resource function.


The Internet Gateway

Instances in your VPC will need to be able to access the internet somehow, and an internet gateway is the mechanism for making this happen. Let’s create one.

If there are no properties to set on a resource, the do..end block may be omitted entirely

resource :defaultigw, 'AWS::EC2::InternetGateway'

Attaching the Gateway

For a gateway to be routable, it needs to be attached to a specific VPC using a “VPC Gateway Attachment” resource.

Fn::ref serves the same purpose as CloudFormation’s {"Ref": ""} intrinsic function.

resource :vpcigw, 'AWS::EC2::VPCGatewayAttachment' do
  vpc_id Fn::ref(:vpc)
  internet_gateway_id Fn::ref(:defaultigw)

The Route Table

A VPC also needs a route table. Every VPC comes with a default route table, but I like to create my own resources so that they’re all expressed in the template.

resource :routetable, 'AWS::EC2::RouteTable' do
  vpc_id Fn::ref(:vpc)

The Default Route

We also have to set up a default route, so that any traffic that the VPC doesn’t recognize gets routed off to the internet gateway.

The resource function accepts one additional parameter that was not addressed above: the options hash. Additional options passed here will be placed inside the resource, but outside the Properties block. In this case, we’ve specified that the default route explicitly depends on the VPC Internet Gateway.

As of this writing, this is actually a required workaround for this template. The gateway must be attached to the VPC before a route can be created to it, but since the gateway attachment isn’t actually referenced anywhere in this resource, we need to explicitly declare that dependency.

resource :defaultroute, 'AWS::EC2::Route', DependsOn: [:vpcigw] do
  route_table_id Fn::ref(:routetable)
  gateway_id Fn::ref(:defaultigw)
  destination_cidr_block ''

The Subnets

Naturally, you’ll also need networks. Like the route table, I like to create my own so that I have control of their configuration inside the template.

Notice Fn::select, Fn::get_azs and AWS::region in this snippet. These all map to the CloudFormation functions and variables of the same name.

We’ll use Ruby to create three identical subnets:

(1..3).each do |i|

The subnets themselves will be in the first three availability zones of the account. A more sophisticated template might want to handle this differently.

resource "subnet#{i}", 'AWS::EC2::Subnet' do
  availability_zone Fn::select(i, Fn::get_azs(AWS::region))
  cidr_block "172.42.#{i}.0/24"
  vpc_id Fn::ref(:vpc)

Now the subnet needs to be associated with the route table, so that hosts in the subnet are able to access the rest of the network and the internet.

resource "srta#{i}".to_sym, 'AWS::EC2::SubnetRouteTableAssociation' do
  subnet_id Fn::ref("subnet#{i}")
  route_table_id Fn::ref(:routetable)

We can use the output function to output the subnet IDs we’ve just created. We’ll go over how Cfer makes this useful in part 2 of this post.

output "subnetid#{i}", Fn::ref("subnet#{i}")

And of course, end the iteration block.


Finally, let’s output the VPC ID too, since we’ll probably need that in other templates.

output :vpcid, Fn::ref(:vpc)

Converging the Stack

Now that you have your template ready, you’ll be able to use Cfer to create or update a CloudFormation stack:

cfer converge vpc -t examples/vpc.rb --profile ${AWS_PROFILE} --region ${AWS_REGION}

Which should produce something like this.

Cfer Demo

Use cfer help to get more usage information, or check and Rakefile in the source repository to see how to embed Cfer into your own projects.

In Part 2…

In part 2, we’ll go over some additional features of Cfer. If you want a preview, you can check out the instance.rb example, which covers how you can use Cfer to create security groups, instances with automated provisioning, and how to automatically look up outputs from other stacks.

Cfer can be found on GitHub and RubyGems and is MIT licensed. Pull requests are welcome.

The Ides Language: Lambas and Coercion Rules

By S. Tilmon Edwards on 2014-01-16

The Ides programming language, for the uninitiated, is my own attempt at designing a modern, multiparidigm systems programming language. It is heavily inspired by Scala and Rust, and is built on LLVM. This post is a brief overview of some aspects of the language’s coercion rules as they apply to lambda expressions.

These ideas are works in progress, and feedback is welcome.

Inference and Coercion

Type inference and type coercion can be thought of as opposite operations. With type inference, types implicitly float upward through the expression tree:

val x = 5;

Conversely, with type coercion, types are implicitly forced downward through the expression tree:

val x : float32 = 5;

Both of these are used extensively in idiomatic Ides code.

Coercion to Functions

Ides functions implement the trait Function[Ret: Any, Args: Any...]1. Functions are first-class values that can be accepted as arguments to other functions. For example, suppose we have a function that allows you to set the random number generator used by some system:

def setRNG(rng: Function[int32]);

Ides allows any expression of type T to be coerced to a Function[R] (a function which accepts zero arguments and returns a value of type R) if T (the type of the expression) can be coerced to R (the return type).

This means that, if we determine that a single fair dice roll is a sufficient quality of randomness, we could define the random number generator as follows:


The expression will be coerced to an expression of type Function[int32], and will be evaluated when the resulting function is called.

Lambdas and Placeholders

A placeholder is an expression with no type, but which may be coerced to any type. Placeholders are only valid in an expression tree that the compiler will coerce to a function:

// BAD: f can't be assigned to a value of no type.
val f = :1;
// OK: :1 is coerced to the type of the first argument (int32),
//     and then coerced to the return type (float64).
val f = (:1 as Function[float64, int32]);

// OK: x = 5.0
val x = f(5);

It would be cumbersome if this was the only way to define lambdas. Ides also provides the -> operator, which allows for a more verbose, but more flexible way of creating lambda expressions:

// OK: Argument is named and typed explicitly.
val f = (x : int32) -> x;

// OK: x = 5
val x = f(5);

The special placeholder :0 always corresponds to the immediately enclosing function. :0 may be used in any function context, not including constructors.

Defining Valid Coercions

Ides provides two mechanisms for declaring new coercions: implicit auxilary constructors, and the CoerceTo[T] trait.

Implicit Auxilary Constructors

Ides borrows the concept of primary and auxilary constructors from Scala2. In addition to standard auxilary constructors, a programmer may define any number of implicit auxilary constructors:

class A(s: String) => {
    implicit def this(s: String) => A(s);

This defines the process for coercing a value of type String to a value of type A. Semantically, this is a coercion from a String.


A class may also implement the trait CoerceTo[T] by providing a single method, to() which returns a value of type T3:

class B(s: String) : CoerceTo[A] => {
    def to() : A => A(s);

This defines the process for coercing a value of type B to a value of type A. Semantically, this is a coercion to type A.

If multiple valid coercions exist, CoerceTo[T] will always take precedence over implicit constructors.

Some Fun Examples

Recursive calculation of the nth fibonacci number:

val fib = (n : int32) -> if (n <= 1) 1 else :0(n - 1) + :0(n - 2)

Calculating n factorial:

val factorial = (n : int32) -> (1 to n).fold(1, :1 * :2)
// Note: The second argument to fold() is actually a second lambda expression.
// :1 and :2 in this context do not refer to arguments to factorial() itself.

Lazy values, implemented without the need for explicit language support:

class Lazy[T : Any](func : Function[T]) : CoerceTo[T] => {
    // Coercions can be chained.
    // An expression of type T is implicitly coerced to a lambda
    // before being passed here:
    implicit def this(f: Function[T]) => this(f);

    // The Lazy[T] can be used as though it was a value of type T.
    // When coerced to a T, the actual value is retrieved once 
    // from the function and cached for further coercions.
    def to() : T => value match {
        case Some(v) => v
        case None => { value = Some(func()); value.get(); }

    private var value : Option[T];
  1. The type argument Ret, which is bound to subtypes of Any, is always present and represents the function’s return type. The type argument Args may be zero or more types, each a subtype of Any, representing the types of the arguments to the function.

  2. The Scala Language Specification § 5.3.1

  3. Unlike many other statically typed languages, Ides allows functions to be overloaded by return type, and uses coercion rules to disambiguate between otherwise identical methods. This allows CoerceTo[T] to be implemented for multiple types of T.

More C++11: Context Stacks

By S. Tilmon Edwards on 2014-01-06

In today’s post, more evil with C++ templates.

In the course of development of Ides I’ve repeatedly found that, in the highly recursive nature of code generation, I need a reliable way to keep track of context. For example, code generation within a function often needs to reference the actual function being generated, even though the compiler is deep inside an expression tree. There are several instances of this pattern appearing, and so I’ve made an attempt to generalize it into a variadic template class I’ve called MultiContext.

In a nutshell, MultiContext tracks multiple stacks of different heights and types alongside the call stack. If an item is placed into the MultiContext at some level N in the call stack, it will be popped when N returns or throws.

// The main case. T = the context type.
    template<typename... Args>
    class MultiContext : std::stack<Args>... {
        // MultiContextItem handles all stack operations for us, to ensure the stack stays sane.
        template<typename CT, typename CS>
        friend class MultiContextItem;
        template<typename A>
        A& Top() const { return ((std::stack<A>*)this)->top(); }
        template<typename A> void Pop() { ((std::stack<A>*)this)->pop(); }
        template<typename A> void Push(A& t) { ((std::stack<A>*)this)->push(t); }

Instances of a second class are created on the stack to ensure that items are always popped off the stack at the end of the proper scope:

template<typename T, typename S>
    class MultiContextItem {
        MultiContextItem(T& val, S& stack)
            : stack(stack), val(val)
            stack.template Push<T>(this->val);

        ~MultiContextItem() {
            // Sanity check.
            // We should be the only one with write access to the stack.
            // But evil exists.
            assert(stack.template Top<T>() == val);

            stack.template Pop<T>();

        MultiContextItem( const MultiContextItem<T, S>& );
        const MultiContextItem<T, S>& operator=( const MultiContextItem<T, S>& );
        S& stack;
        T& val;

For convenience, we define a macro that creates a MultiContextItem on the stack, assigned to a (hopefully) unique variable name, ensuring that items will be popped off the stack whether we return or throw an exception.

#define SETCTX_OTHER(arg, self) MultiContextItem<decltype(arg), decltype(self)> __stack_ctx##arg(arg, self);
#define SETCTX(arg) SETCTX_OTHER(arg, *this)

The second macro references this under the assumption that MultiContext will be inherited by a visitor class doing the recursive processing.

Now we’re able to support create multiple parallel stacks, bound to lexical scope.


int main() {
    int i = 5;
    int i2 = 10;
    std::string s = "test";
    MultiContext<int, double, std::string> ctx;
        SETCTX_OTHER(i, ctx);
        SETCTX_OTHER(s, ctx);
            SETCTX_OTHER(i2, ctx);
        assert(ctx.Top<int>() == 5);
        assert(ctx.Top<std::string>() == "test");