Mark Hudnall's website

How many servers do we need?

Fri, 09 Jun 2023 00:00:00 GMT

How much concurrency do we need to handle a given throughput and response time?

This is a fundamental question when scaling software systems. Since software systems are queues, let us try to answer it with queuing theory.

An initial guess using Little's law

Little's law states that the average number of jobs in a system ( $L$ ) is equal to the arrival rate of jobs ( $λ$ ) multiplied by the average time a job spends in the system ( $W$ ). In other words, $L = λ W$ .

Little's law can provide a simple estimate of the required concurrency — we can set concurrency to the average number of jobs in the system, given some known arrival rate (throughput) and service time (response time).

Consider a web server where average response time is 100ms. Suppose we'd like to handle a throughput of 100 requests per second. Using Little's law, we would conclude that, on average, 10 requests are "in progress" from clients' perspectives (either queued or actively being serviced). We might then think to set concurrency to 10 in order to effectively handle this load.

However, because Little's law works with averages, it doesn't account for variability in the requests' arrival and service times. In practice, requests are not uniformly distributed over time, and some requests may take longer than others. There is a non-zero probability that an incoming request might find all servers busy and need to wait.

So Little's law on its own is not enough. We need to consider a more sophisticated model.

Erlang's C formula

We would like to find some level of concurrency that constrains the probability of waiting to some target. To do this, we turn to Erlang's C formula, which provides us with the probability that an incoming job must wait because all servers are busy¹. Erlang's C formula is defined as follows:

C (L, S) = \frac{\frac{L^{S}}{S!} \times \frac{S}{S - L}}{\sum_{i = 0}^{S - 1} \frac{L^{i}}{i!} + \frac{L^{S}}{S!} \times \frac{S}{S - L}}

Here, $L$ is the average number of jobs in the system, or $λ W$ as above with Little's law. $S$ is the number of servers. $L$ is also referred to as "offered traffic" — Erlang's C formula was developed in the context of efficient telephone line usage so "offered traffic" thus refers to the amount of traffic that could be carried given a certain volume of calls and unlimited lines. For our purposes, it is average load, or the average number of requests expected to be in-progress from clients' perspectives, given a certain throughput and response time.

To find the ideal level of concurrency, we can start with some number of servers (the initial guess using Little's law) and then increment it until $C (L, S)$ is below some desired threshold (say 5% probability of waiting). We will write a script to do this.

We're faced with a challenge here, though. Due to the factorials, computing Erlang's C formula will underflow. To work around this, we will translate Erlang's C formula into log-probability space where underflows and overflows are less likely.

Deriving the logarithmic Erlang C formula

We start with the original Erlang's C formula:

C (L, S) = \frac{\frac{L^{S}}{S!} \times \frac{S}{S - L}}{\sum_{i = 0}^{S - 1} \frac{L^{i}}{i!} + \frac{L^{S}}{S!} \times \frac{S}{S - L}}

We take the natural log of each term in the formula:

\log (C (L, S)) = \log (\frac{L^{S}}{S!}) + \log (\frac{S}{S - L}) - \log (\sum_{i = 0}^{S - 1} \frac{L^{i}}{i!} + \frac{L^{S}}{S!} \times \frac{S}{S - L})

Here, we've used the property of logarithms $\log (a / b) = \log (a) - \log (b)$ .

We further simplify the above expression:

\begin{aligned} \log (C (L, S)) = S \log (L) & - \log Γ (S + 1) \\ + \log (S) - \log (S - L) \\ - \log (\sum_{i = 0}^{S - 1} e^{i \log (L) - \log Γ (i + 1)} \\ + e^{S \log (L) - \log Γ (S + 1) + \log (S) - \log (S - L)}) \end{aligned}

Here, we've used the property of logarithms, $\log (a^{b}) = b \cdot \log (a)$ , and replaced factorials with the Gamma function ( $n! = Γ (n + 1)$ for a non-negative integer $n$ ), as $\log (n!)$ can be computed as $\log Γ (n + 1)$ .

Note that the original denominator is now the log of a sum of exponentials. We can thus use the log-sum-exp trick to calculate these terms without underflow.

Finally, we exponentiate the log-probability to get back to the original probability space:

C (L, S) = \exp (\log (C (L, S)))

Python code

Let's look at how we'll calculate the minimum concurrency needed to meet a certain throughput, response time, and probability of queuing.

First, we'll choose an initial guess for the required number of servers using Little's law and a target throughput and response time. Then, we will update our initial guess using an iterative process. We will increment the number of servers until the probability of queuing, as given by our logarithmic Erlang's C formula, is below our desired max.

import math

def log_sum_exp(numbers):
    max_num = max(numbers)
    return max_num + math.log(sum(math.exp(x - max_num) for x in numbers))

def erlang_c(L, S):
    log_numerator = S * math.log(L) - math.lgamma(S + 1) + math.log(S) - math.log(S - L)
    denominator_terms = [i * math.log(L) - math.lgamma(i + 1) for i in range(S)] + [S * math.log(L) - math.lgamma(S + 1) + math.log(S) - math.log(S-L)]
    log_denominator = log_sum_exp(denominator_terms)
    erlang_c = math.exp(log_numerator - log_denominator)
    return erlang_c

def calculate_servers(lambda_, W, max_prob):
    lambda_ = float(lambda_)
    W = float(W)
    L = lambda_ * W
    S = math.floor(L + 1)
    print(f'Starting with {S} servers')
    prob = erlang_c(L, S)
    while prob > max_prob:
        print(f'With {S} servers, the queuing probability is {prob}. Trying {S + 1} servers.')
        S += 1
        prob = erlang_c(L, S)
    return S, prob

requests_per_second = 1000.0
response_time_s = 0.1
max_prob = 0.05

num_servers, prob = calculate_servers(requests_per_second, response_time_s, max_prob)
print(f'The minimum number of servers required is: {num_servers} (queuing probability {prob})')

When we run this script, we see the following output:

Starting with 101 servers
With 101 servers, the queuing probability is 0.8833145020395913. Trying 102 servers.
With 102 servers, the queuing probability is 0.7770966583320832. Trying 103 servers.
With 103 servers, the queuing probability is 0.6807968118039859. Trying 104 servers.
With 104 servers, the queuing probability is 0.593855705420894. Trying 105 servers.
With 105 servers, the queuing probability is 0.5157074268123086. Trying 106 servers.
With 106 servers, the queuing probability is 0.44578262621325715. Trying 107 servers.
With 107 servers, the queuing probability is 0.3835119575327338. Trying 108 servers.
With 108 servers, the queuing probability is 0.32832968002713697. Trying 109 servers.
With 109 servers, the queuing probability is 0.27967734713081166. Trying 110 servers.
With 110 servers, the queuing probability is 0.23700750028506098. Trying 111 servers.
With 111 servers, the queuing probability is 0.1997872798880628. Trying 112 servers.
With 112 servers, the queuing probability is 0.16750186339584125. Trying 113 servers.
With 113 servers, the queuing probability is 0.13965764259076413. Trying 114 servers.
With 114 servers, the queuing probability is 0.11578505827853611. Trying 115 servers.
With 115 servers, the queuing probability is 0.09544102105759475. Trying 116 servers.
With 116 servers, the queuing probability is 0.0782108608850327. Trying 117 servers.
With 117 servers, the queuing probability is 0.06370976519997625. Trying 118 servers.
With 118 servers, the queuing probability is 0.05158368436937104. Trying 119 servers.
The minimum number of servers required is: 119 (queuing probability 0.04150970303605052)

It's interesting to note that our initial guess has a very high probability of queuing. If we graph our findings, we see that Erlang's C formula is decreasing and convex in $S$ :

Beyond the Erlang model

Our model here gives the impression that throughput scales linearly with concurrency. In practice, it does not.

Amdahl's law and the Universal Scalability Law remind us that as we increase concurrency, total speedup is limited by sequential computation, as well as contention and coherence issues.

Amdahl's law states that any speedup using multiple processors is limited by the time needed for the sequential fraction of the program. Similarly, for USL, our system may depend on a shared resource like a database, or require coordination to keep data consistent across multiple servers. Ultimately, the throughput we get from increasing concurrency diminishes with scale.

Perhaps we could measure the contention and coherence factors empirically as we scale, in pursuit of a better model.

M/M/c queues

The analysis also relies on the M/M/c queue model, characterized by three properties:

Markovian (or Poisson) arrival process: Jobs arrive independently at a constant average rate. In simpler terms, the timing of the next job arrival does not depend on when the previous job arrived — this is what we mean by "Markovian". The arrivals could be at any time, and they are spaced apart randomly but with a constant average rate.
Markovian service times: Similar to arrivals, the time it takes to service the next job does not depend on how long it took to service the previous job, and there is a constant average rate.
c servers: There are c identical servers each operating at the same average service rate.

Many real-world systems don't adhere to these assumptions. For example, if clients retry requests or make sequential requests during page load, then arrivals are no longer memoryless — there are dependencies between request arrival times. Similarly, if servers contend for a common database under load, service times may no longer be independent. And if an application has peak times, then request arrival times are unlikely to be Markovian, since the arrival process depends on the time of day.

The model is a simplification, and we must always keep that in mind. It can still useful, if we are judicious in how we apply it. For example, perhaps it's better to use throughput and response time at peak times, rather than their averages, even if it means over-provisioning most of the time. We must recognize that there might not be a definitive answer, but instead, an ongoing exploration that balances the tension of theory, observation, and good old tinkering.

For an M/M/c queue. See the last section for caveats. ↩

Three things about Common Lisp

Sat, 27 Feb 2021 00:00:00 GMT

I continue to have my mind blown by Common Lisp. Here are three things I've been thinking about.

1. Error handling

Common Lisp made me realize I was living with a fixed mental model of error handling. Swimming in water, so to speak. To me, error handling meant try/catch or returning errors directly from functions a la golang or promises. In either case, control is relinquished at the error-site and bubbled up the stack to an error handler. The flow of control continues with the error handler.

The Common Lisp condition system, on the other hand, separates flow of control from the error handler. A low-level piece of code can define multiple ways to handle an error that it might throw. A higher-level function can then decide which path to actually take when the error is thrown. Control flow depends on that decision: it is possible to just skip the error and continue on with the low-level code, if that is one of the error handling options that the low-level code provides.

This is simply not possible in the try/catch paradigm without directly handling the error at the error-site, which means we lose generality: we are now stuck with a specific error handling implementation without the ability to customize.

The chapter, Beyong Exception Handling: Conditions and Restarts from Practical Common Lisp explains the condition system very well.

2. Dynamic variables

Dynamic variables are like global variables with a twist. We are used to lexical scoping, where a variable is defined in some scope and can be accessed by any inner scope that is literally nested within the outer one. Dynamic variables, on the other hand, are "dynamically scoped". When a dynamic variable is bound to a value, this binding lives on for the duration of that scope, surviving function calls. The binding is pushed onto a stack, so when the scope ends, the binding is popped and the variable returns to its former value. This differs from a global variable, where "re-binding" the variable merely sets it to a different value.

I've seen dynamic variables before in elisp and clojure. I've almost never used them in clojure, and I've used them sparingly in emacs when it was the only possible way to customize some value for the duration of a block. I equated dynamic variables with global variables, and I've seen the messiness of relying on global variables.

In Common Lisp... they just make sense for some reason. Maybe it's the fact that they're used so much more in Common Lisp code I've been working with, but I find it incredibly useful to have a value that is "globally" accessible (but only for a certain, well-defined lifetime) and to be able to customize that variable non-locally.

I see the pitfalls too. Since any function inside a scope can set a dynamic variable, it makes data flow unclear. Very easy to forget that the variable can be set non-locally and write code that assumes the value I set 3 lines above in a let binding is the current one.

There's a sort of similar pitfall with the condition system: my code cannot necessarily assume that throwing an error means control will end at that location.

3. Interactivity

Common Lisp is intensely interactive. It is possible (and seemingly common), for example, to embed a Swank server in a production deployment. This allows you to connect to the running production system in a REPL. You can change live data, debug problems, hotswap code, etc. You might have also heard the story of lisp at the JPL, where they debugged a problem over a REPL through space.

Interactivity is at the core of the language. Installing dependencies happens at the REPL inside the running lisp process, rather than through an out-of-process package manager. Unhandled errors automatically invoke the interactive debugger rather than panicking. Standalone executables, which are quite large, are actually lisp "images": dumps of all data from a running lisp process (running processes can be dumped and restored at any time).

This interactivity changes the programming experience dramatically. I frequently find myself writing programs from the "bottom up", starting with an inner piece that I incrementally evaluate in the REPL and build on. Moreover, the feedback loop is tight; writing lisp is fun.

There are drawbacks here too. The problem with "images" is that the textual code and the image can diverge. The symptoms are starting up a fresh REPL only to be greeted by "The variable FOO is unbound". The root cause is usually having set some variable during a REPL session to a value that is defined further down in the code. A minor headache, as long as it doesn't get deployed to production.

First impressions of Nix

Wed, 27 Jan 2021 00:00:00 GMT

Nix is package management from the future. But like other transmissions from our future selves, it's tough to understand what Nix really is, why it matters, and what you can do with it.

I started using Nix and I'm not looking back. Let me tell you how I think about Nix and why it's worth learning.

Caveat lector: Nix is difficult to get started with on macOS. For more thoughts on this and other pain points, see Challenges. I encourage perseverance (and perhaps a stiff drink) as you explore this brain-changing technology.

What is Nix?

Nix is a package manager like Homebrew or pacman. We can install ripgrep with Nix to get a feel for it:

$ nix-env --install --attr nixpkgs.ripgrep
installing 'ripgrep-12.1.1'
building '/nix/store/0l147binjrsqsq9bnh1zn2xnfnmxv2lf-user-environment.drv'...
created 708 symlinks in user environment

ripgrep is now installed:

$ rg --version
ripgrep 12.1.1
-SIMD -AVX (compiled)
+SIMD +AVX (runtime)

Where did Nix install it?

$ which rg
/Users/mark/.nix-profile/bin/rg

$ readlink $(which rg)
/nix/store/43m8p9f7zbjnf959rhx5mzls5c28y2gk-ripgrep-12.1.1/bin/rg

With Nix, package names (43m8p9f7zbjnf959rhx5mzls5c28y2gk-ripgrep-12.1.1) are constructed by hashing a package's source files with the names of all its dependencies. A human-readable name is then appended to this hash. This means that if the source of a package or any of its dependencies change, the name of the package changes too.

Packages reside in the Nix store, which is just the directory /nix/store. Nix requires that all package dependencies also be in the Nix store. In this way, the Nix store forms a graph where nodes are packages and edges are dependency relationships.¹

Nix also has an expression language, which is used to define packages. With Nix, these package definitions are called derivations. The syntax is somewhat alienating for newcomers, so let's look at an example to familiarize ourselves. Here is the derivation for ripgrep, which is available on nixpkgs, Nix's primary package repository:

{ stdenv
, fetchFromGitHub
, rustPlatform
, asciidoctor
, installShellFiles
, Security
, withPCRE2 ? true
, pcre2 ? null
}:

rustPlatform.buildRustPackage rec {
  pname = "ripgrep";
  version = "12.1.1";

  src = fetchFromGitHub {
    owner = "BurntSushi";
    repo = pname;
    rev = version;
    sha256 = "1hqps7l5qrjh9f914r5i6kmcz6f1yb951nv4lby0cjnp5l253kps";
  };

  cargoSha256 = "03wf9r2csi6jpa7v5sw5lpxkrk4wfzwmzx7k3991q3bdjzcwnnwp";
  cargoBuildFlags = stdenv.lib.optional withPCRE2 "--features pcre2";

  nativeBuildInputs = [ asciidoctor installShellFiles ];
  buildInputs = (stdenv.lib.optional withPCRE2 pcre2)
  ++ (stdenv.lib.optional stdenv.isDarwin Security);
  // ...
  meta = with stdenv.lib; {
    description = "A utility that combines the usability of The Silver Searcher with the raw speed of grep";
    homepage = "https://github.com/BurntSushi/ripgrep";
    license = with licenses; [ unlicense /* or */ mit ];
    maintainers = with maintainers; [ tailhook globin ma27 zowoq ];
  };
}

We are looking at an anonymous function that takes a single argument, an attribute set. An attribute set is like a hashmap where keys are strings. The attribute set used as input argument has been destructured as part of the function definition and contains the keys stdenv, fetchFromGitHub, rustPlatform, asciidoctor, etc. The function calls rustPlatform.buildRustPackage to define the package, passing in another attribute set that specifies the configuration necessary to build ripgrep. In particular, it specifies that the ripgrep source (the src key) should be fetched from GitHub. Many of the details are hidden inside the function buildRustPackage, which ultimately specifies that src should be built using cargo build.

Note that the above function only has access to its input arguments. These input arguments are either utilities functions and namespaces (like stdenv, fetchFromGitHub, and rustPlatform) or other packages (like asciidoctor and Security²), both defined using the Nix expression language.

If your package relies on other packages as build or runtime dependencies, you'll declare these dependencies using the Nix expression language, as ripgrep does above with nativeBuildInputs and buildInputs. Nix computes the full path to those dependencies (something like /nix/store/<a-hash>-your-dependency) and makes them available through $PATH or other environment variables when it actually builds your package. Nix makes sure those dependencies are built before it builds your package.

So when ripgrep is built by Nix, it is actually built using Nix-packaged rustc and cargo (/nix/store/189i8cpfdr758nplhviw59qcp539q4l2-rustc-1.49.0/bin/rustc and /nix/store/d6bynqj12ak4dqqgk1wvw3pwrxym9xw3-cargo-1.49.0/bin/cargo respectively at the time of writing).³

The big idea

To recap: Nix names packages by hashing their source files along with the names of all their dependencies. Thus, updating a package's source or any of its dependencies changes this hash, resulting in a new package with a new name. Nix puts packages and their dependencies in the Nix store, so it has a global view of all packages. Since changes to packages always result in a new name, the Nix store is append-only.

The implications of this are staggering:

Multiple versions of the same package can peacefully coexist on the same system, since they are guaranteed to have different names.
Once a package is built (for a given platform), it has a globally unique name and can thus be globally cached. In fact, Nix does this automatically, so most packages are actually just downloaded from cache.nixos.org rather than built from source on your computer.
Rollbacks to previous versions are trivial: just change symlinks back to what they were before. Nix tools like nix-env --rollback make this easy to do.

A motivating example: Homebrew and openssl

To better understand this paradigm shift, consider this issue regarding Homebrew, openssl, and rbenv or this similar one for pyenv. Or consider the dreaded dyld: Library not loaded issues when upgrading macOS. I have encountered these issues myself, resulting in hours of frustration. Multiply this by the # of engineers in your org and you have real productivity loss, especially if your org keeps dependencies up-to-date or mandates regular OS updates (which it should).

Now these problems can be solved by hand. In the first case, openssl v1.0.x was end-of-lifed, so Homebrew dropped the formula that lets you build it. Unfortunately, old ruby versions rely on this old version of openssl, so it must be manually installed and the ruby build retried with --with-openssl-dir=/path/to/openssl-v1.0.x⁴. In the second case, openssl is dynamically linked at runtime, but its dylib has disappeared during an OS upgrade. The path to the nonexistent dylib needs to be changed with install_name_tool on the binary that uses it.

But here be dragons. The highly upvoted and accepted answer for the dylib issue has you re-link the old, end-of-lifed version of openssl. Another answer has you install openssl using some random formula. These are bad ideas for something like openssl. Yet, I have done these things myself because, like most developers, I just want to get on with the task at hand.

So what's the underlying problem?

Rich Hickey sums it up well in his excellent talk, Spec-ulation. The situation is something like this:

Software has dependencies
Dependencies are referenced by name
Names are resolved by the "environment"
The environment changes out from under you, breaking name resolution

Unsurprisingly, this pattern — let's call it "naming stuff" — pervades software design:

Functions call other functions. Import statements set up the environment where function names can be resolved.
Import statements reference other namespaces. Package declarations (package.json, etc.) set up the environment where namespaces can be resolved.
Package declarations reference other packages by artifact name or URL. Package registries or DNS set up the environment where artifacts can be resolved.
Shells reference commands by name. $PATH sets up the environment where commands can be resolved to paths in the filesystem.
Filesystem paths are names too. inodes set up the environment where paths can be resolved to block locations on disk.

The problem is that associations between names and their referents are mutable. Names, while convenient for human beings, hide changes in the environment. The nature of the relationship between names and their referents is a longstanding philosophical question. Applied to programming, we might ask:

If I make a breaking change to a function, is it still the same function?

If I have openssl v1.0.2 and openssl v1.1.1, which one is "openssl"?

SemVer is supposed to resolve this dilemma by separating breaking and non-breaking changes into major and minor/patch versions. But since there is no formal relationship between artifact names and namespaces, SemVer doesn't really fix the underlying issue. This becomes clear in "transitive dependency hell", when Package A depends on B and C, where B depends on D v1.0 and C depends on D v2.0. What are we importing when we write import D?

Nix solves this problem for packages by making every name correspond one-to-one with a referent⁵. If a package's source or dependencies change, it is a different thing with a different name. There can be no question about which version the name "openssl" refers to, because Nix makes the name "openssl" impossible (or at least, makes it have only a single referent in the context of a given package that depends on it).

In essence, Nix makes the package environment append-only. New names can be added, but existing associations between names and their referents are immutable.

So to answer the age-old philosophical question, "ship of Theseus" is a symlink to /nix/store/j2pic1a5ijcpbs494vdhdbafn7y50ppr-ship-of-theseus that is updated to /nix/store/rmqbimj5mqnshfr3rgffk8h0flx7k911-ship-of-theseus when the first plank of wood is replaced.

Challenges

I ran into both practical and conceptual difficulties with Nix. Ultimately, my opinion is that Nix is hard to install and use for newcomers but that its problems can be overcome through education and automation.

Here is a non-exhaustive list of head-scratchers I've encountered in my Nix journey to date.

Installing on macOS Catalina and later

Nix is difficult to install on macOS. The primary issue is Catalina's root filesystem is read-only, so /nix cannot be created like a regular directory. Nix assumes that it has write access to /nix and derivations use absolute paths when referring to other objects in the Nix store. The workaround, which has been automated by the Nix installer, is to create a separate volume for the Nix store and mount it at /nix.

But the installer did not work for me; the Nix installer creates a volume that relies on the T2 chip's encryption at rest rather than FileVault, and my laptop does not have a T2 chip. This issue is documented here. As suggested by the docs, the fix is to manually create a Filevault-encrypted volume. The installer does not do this automatically because of race conditions between services that use Nix on startup and mounting + decrypting the volume, which causes some applications to fail on startup.

Thankfully, an open PR is set to fix this race condition and automate creating the Filevault-encrypted volume during installation. I used the preview installer in the open PR's description to get up and running.

nix-darwin

The next challenge was understanding and installing nix-darwin. nix-darwin's README describes it as:

Nix modules for darwin, /etc/nixos/configuration.nix for macOS.

Huh? Basically, nix-darwin gives you a way codify configuration for a macOS computer using the Nix expression language. Configuration ranges from networking and users to installed applications and CLI tools. This is the same role that NixOS plays for Linux, but less tightly integrated in nix-darwin due to macOS's more restrictive platform. With NixOS, I would configure my system by editing a file located at /etc/nixos/configuration.nix. With nix-darwin, the analogous file defaults to ~/.nixpkgs/darwin-configuration.nix.

For example, to define a user, I can write:

users.users.mark = {
  name = "mark";
  home = "/Users/mark";
  shell = pkgs.zsh;
};

And I can install system-wide packages using:

environment.systemPackages = [ pkgs.ripgrep ];

When I run darwin-rebuild switch, nix-darwin updates my system to reflect any changes to the config by building new packags and updating symlinks on my $PATH. If the update fails for any reason, I can roll back to the previous version instantaneously with darwin-rebuild --rollback.

nix-darwin is structured internally using NixOS modules, a convention where Nix files define available configuration options for some part of the system and then implement the configuration logic. The documentation helps with understanding the available configuration options.

I ran into some issues installing nix-darwin that I can't remember now, but they were solved using workaround in various GitHub issues.

Applications and spotlight

A big issue I ran into was making Nix-installed macOS applications like Firefox work well with Alfred. nix-darwin symlinks macOS applications built with Nix into ~/Applications/Nix Apps. This works fine if you use the dock to launch applications, but Spotlight does not index symlinks and so it is not possible to open these applications using Spotlight or Alfred.

At first, I tried to get Alfred to index my Nix store, but couldn't get this to work. A workaround suggested in this GitHub issue is to use AppleScript to create macOS aliases, which, unlike symlinks, are indexed by Spotlight. Using AppleScript seems like a hack. Another user suggests a similar workaround using a Swift script to create these aliases.

I ended up combining these workarounds into a custom module. You can see the Nix module in my config here. The module is enabled with the following line in my nix-darwin configuration:

services.link-apps = {
  enable = true;
  userName = config.users.users.mark.name;
  userHome = config.users.users.mark.home;
};

Applications that are installed with Nix are now aliased into ~/Applications/Nix, which can be indexed by Alfred. I also needed to configure Alfred to index aliases by going to Alfred Preferences -> Features -> Default Results -> Extras -> Advanced... and dragging in one of the aliases in $HOME/Applications/Nix.

It still feels like an ugly workaround, but it has worked well for my purposes ever since.

Paper-cuts

Beyond the installation issues and conceptual hurdles (Nix's novel paradigm of content-addressing packages), Nix gave me some paper-cuts. These tidbits of accidental complexity raise the barrier to entry but aren't inherent to Nix's new paradigm.

Nix expression language

Nix requires learning the Nix expression language, a purpose-built language that isn't re-usable in other contexts. I quickly needed to understand and wield the language when I ventured beyond just installing packages with nix-env into creating reproducible environment with nix-darwin + home-manager and, for projects, shell.nix files.

Now that I'm more familiar with it, I actually like it a lot. But it created a higher upfront cost to learning Nix. I could just as easily see Nix working with YAML, widely used in automation tooling, but I suppose that DSLs need to be learned irrespective of the host language, and Nix's expression language gives it the power of modularity. So maybe I'm just being grumpy.

Everything is an attribute set

Nix attribute sets are nice data structures. nixpkgs is basically a big, nested attribute set, hence installing a package with e.g. nix-env -iA nixpkgs.graphviz involves naming a derivation nested inside the top-level nixpkgs attribute.

nixpkg utility functions such as mkOption or buildRustPackage are also defined using attribute sets.

The facts that everything is an attribute set, the Nix expression language is dynamically typed, and documentation is sparse make discoverability a challenge. I often find myself poking around the nixpkgs source, hunting down function definition or derivations so I can see their inputs and understand what they do. As before, this steepens the learning curve.

CLI tools

Nix has many top-level CLI tools. There is nix, nix-env, nix-shell, nix-store, nix-build, nix-channel, darwin-rebuild (or nixos-rebuild), and others, for example. The self-titled tool, nix, is used to set global configuration, while the others are used for Nix's various subsystems.

These CLI tools reflect Nix's breadth, but I wish there was just one top-level nix command and everything else was a subcommand. nix does have subcommands, but there are sometimes slight differences in functionality. For example, there's nix develop but also nix-shell. They have differences, but which one should I be using?⁶

"Now I have two build tools"

I've been using Nix for projects as well. I think of Nix as a superset of language-specific package managers like ruby's bundler and version managers like rbenv. In addition to managing packages and language versions, I can use Nix to install CLI tools that aren't language specific. Combined with lorri and direnv, all the tools I need when working on a given project are available when I cd into its repo.

But I find that using Nix for package management in projects is awkward. It makes sense: Nix makes builds reproducible by creating a content-hash for each dependency, but we have to bring builds into Nix-land to do this. So there are additional tools that convert dependency declarations like a Gemfile or go.mod into a bunch of Nix derivations in a .nix file.⁷ I could see these getting out of date, especially when I'm new to Nix and my head is more in the language-specific tools (this could be solved with linting).

Moreover, there are subtle differences between the language tools and Nix. For example, both ruby and python allow packaging pre-compiled, platform-specific binaries rather than having users build from source. I don't know enough about what Nix does here under the hood, but it chooses to build from source and has failed for me in cases where the language specific tool doesn't. Theoretically, Nix builds packages in isolation, so perhaps the snags come from the impure parts of Nix that bridge to a specific platform. Or perhaps some packages have build instructions hard-coded for platforms that bypass Nix's attempts to sandbox builds. Either way, it's frustrating enough that, for the most part, I use Nix to bring in language-agnostic tooling but continue to use language-specific package managers.

Docker sort of solves this with multi-stage builds. But Docker is more about creating reproducible run environments (where the run environment can also be used to help build another container) rather than reproducible builds. It doesn't prevent you from specifying "latest" for your dependencies, whereas Nix always requires a hash, effectively pinning versions even if one is not explicitly specified. On the other hand, for Nix to create reproducible builds, the sources must be available somewhere.

Whatever, I can just pin versions using my language-specific package manager and get on fine with Docker. But I much prefer developing on the host if I can. Docker has real costs: the images can be huge and Docker is layered with complexity. Docker on macOS is resource hungry, slow, and probably always will be because it is virtualized. For deployment, I love Docker. Unfortunately, I haven't used NixOS or nixops enough to compare my experiences there.

Resources

Despite my gripes, I still think Nix is solarpunk. It is a category-defining tool, and I hope its ideas become the backbone of software development. It has an elegance, and I appreciate that a globally shareable cache that benefits all developers springs forth naturally from this.

A few resources have been invaluable on my Nix journey. Perhaps they will be helpful to you too.

Burke Libbey's "Nixology" videos. These are excellent, practical videos on using Nix on macOS. Just enough detail about what's going on under the hood to pique my interest. Start here.
Christine Dodrill's "How I Start: Nix". This was a huge help for understanding Nix in projects. Christine's other posts on Nix and NixOS are fantastic too.
Nix pills. The official guide. Excellent for understanding what's going on underneath but maybe not the best starting point for practical use.
Lawrence Dunn's "How Nix Derivation Instantiation Works": A great deep drive into how Nix derivations are instantiated, that is, how we go from Nix expressions to hashes and build instructions.
nix.dev. Useful supplemental docs and style guide.

Good luck.

Nix provides tools for inspecting the relationships between packages. For example, I can see all the dependencies of ripgrep by running nix-store --query --requisites /nix/store/43m8p9f7zbjnf959rhx5mzls5c28y2gk-ripgrep-12.1.1. ↩
The macOS Security framework packaged for Nix. ↩
You can verify this by running nix-shell '<nixpkgs>' -A ripgrep and then which cargo. nix-shell puts you into the environment that Nix uses to build a given package. ↩
Actually, ruby-build was reworked to stop looking for Homebrew's version of openssl so this shouldn't be necessary anymore. Instead, it will download the version of openssl that it needs if it doesn't already exist. pyenv still looks for Homebrew's openssl and so still suffers from this problem. ↩
For solving the same problem in code, I'm really excited about Unison. In particular, check out the documentation on Refactoring and modifying code in Unison. It's a trip. Unfortunately, it's hard to imagine Unison being used widely because it is both an entirely new paradigm and programming language. Of course, Nix is both of these things as well... ↩
FWIW, I use nix-shell. ↩
For Gemfile, it's bundix and for go.mod, it's vgo2nix. ↩

Widening circles

Mon, 02 Sep 2019 00:00:00 GMT

I live my life in widening circles
that reach out across the world.
I may not complete this last one
but I give myself to it.

I circle around God, around the primordial tower.
I’ve been circling for thousands of years
and I still don’t know: am I a falcon,
a storm, or a great song?

– Rainer Maria Rilke (translated by Joanna Macy and Anita Barrows)

The following generative art piece is made by repeatedly drawing circles with smaller and smaller diameters. Perlin noise is applied to both the colors and positions to produce some variation.

You can click the piece (or refresh the page) to see a new rendition of it.

The piece was inspired by the first 2 chapters of Generative Art by Matt Pearson, which has been a joy. Some selected quotes from its prose:

Without giving it much thought, we have turned into augmented beings existing in a world that is simultaneously real and virtual.

Computing is what a stream does as it finds its way downhill toward the ocean. It’s what the planets do as they move in their orbits. It’s what our bodies do as they maintain the balance needed to keep us upright. It’s what our DNA does as it unravels. Computing is what I’m doing now as I process these ideas and output them as text—and what your brain is doing as you read the words and form your own ideas as a result.

This is why I can say, without contradiction, that while I still find computers boring, I think computing is cool. The only place computers really come into it is in attempting to simulate these computations or creating new ones to rival those of the natural world.

With design, the intention is to produce a visual that produces the same response in everyone who sees it—the intention (such as “street,” “retro,” or “subtle”) should be mostly unambiguous. With art, you’re still aiming to produce a response, but if that response is different in different people that doesn’t matter. It’s fine for one person to like a piece while another sneers. Even better, if one viewer loves the work, we would hope another might hate it. If we can foster such an extremity of reaction, it’d be a measure of success. Perhaps the only cardinal sin of art is to be boring.

Like the landscape gardener, the lot of the generative artist is to take naturally evolving phenomena and to fashion them into something aesthetically pleasing. It’s finding that point of balance between the beautiful unruliness of the natural world and the desired order of our ape brains. A garden that is unkempt and overgrown is unpleasing to us because it’s too far into the realm of the chaotic, whereas concreting the area instead is the tidiest, most ordered of solutions, which also removes all beauty.

The sweet spot is between the two, where the grass is neat and evenly cut but still no two blades are alike or move in perfect synchronicity—where the colors of the flowers are evenly balanced, but not in a way that is exact and precise. The sweet spot is where the “art” lives.

With generative art, the autonomous system does all the heavy lifting; the artist only provides the instructions to the system and the initial conditions.

The artist’s role in the production process may be closer to that of a curator than a creator. You create a system, model it, nurture it, and refine it, but ultimately your ownership of the work produced may be no more than a parent’s pride in the work of their offspring.

The purpose of a tool is not only to extend our capabilities; it should also enhance the flow of our creativity. The newest software gives us the power to fashion the world in ways that would have been unimaginable only 50 years ago, but there is still a long way to go in terms of a natural artistry to their use.

Starting Figwheel in Emacs

Mon, 25 Apr 2016 00:00:00 GMT

When I use Emacs to write ClojureScript, I often jack in to CIDER just to start Figwheel and get to the ClojureScript REPL. After running M-x cider-jack-in, I would type in the REPL:

user> (use 'figwheel-sidecar.repl-api)
user> (start-figwheel!)
user> (cljs-repl)

Turns out, there's a faster way to do this using the cider-cljs-lein-repl variable. Add the following to your Emacs config:¹

(setq cider-cljs-lein-repl
      "(do (require 'figwheel-sidecar.repl-api)
           (figwheel-sidecar.repl-api/start-figwheel!)
           (figwheel-sidecar.repl-api/cljs-repl))")

Now, when I'm in a ClojureScript project, I run M-x cider-jack-in-clojurescript instead of M-x cider-jack-in. The value of cider-cljs-lein-repl is used to start the ClojureScript REPL, so figwheel starts automatically and puts me right into the cljs REPL. Magic!

I put this in the :config block of a use-package declaration for CIDER, if you're into that sort of thing. ↩

Depth-first and breadth-first thinkers

Mon, 21 Dec 2015 00:00:00 GMT

This piece is about working on teams.

Consider this problem: you're walking along a trail in the dense forest and you need to find your way out. In front of you, the trail forks over and over again as far as you can see. The exit might be miles away, or it might be a 5-minute walk down one of the forks – you're so turned around that you have no idea.

If you were a poet, you might take the road less traveled. In computing, however, this is a very clear tree traversal problem, and there are two methodical ways to go about it.

Depth-first search

With depth-first search, you go as far as you can down one path before you backtrack.

For instance, you keep walking down the rightmost path, ignoring all the forks, until you reach a dead-end. Then, you backtrack to the last fork with a road you hadn't taken and try that one, following the rightmost path like you did before. Reached a dead-end? You backtrack and repeat the process.

Breadth-first search

With breadth-first search, you explore all the paths closest to you before moving deeper.

For example, you walk down the path on the right until you reach a fork. If you don't see a way out, you backtrack and walk down the path on the left. Still nothing? You backtrack again, back to path on the right to explore the two paths that lead from there. Then you repeat with the path on the left.

You can imagine the trade-offs of both:

With depth-first search, you go really far down one path really quickly. That works out 👍for you if the exit happens to be far away, but if the exit is down the other path and really close, you won't discover it for a while.
With breadth-first search, it takes a long time to reach an exit that's far away, but you'll never miss an exit on a path that's really close by.

Tree traversal algorithms like DFS and BFS are used in many computing disciplines. For example, you can represent a chess game as a tree, where each "fork in the road" represents a different board configuration and each road is a possible move by one of the players. If you could explore this tree quickly, you could look ahead in the game and always make a move that gives you a leg up. Armed with this tree, a computer becomes a worthy chess opponent.

Thinking as a tree

When you're starting a project or exploring a new idea, your thinking is like a tree. Each problem you need to solve relates to the last. Each idea branches into many others, until finally, the full scope of your project or idea stands in front of you.¹

You'd obviously like to enumerate this scope. This begs a question you should ask yourself: How do I naturally walk this tree? Am I a depth-first or breadth-first thinker?

Working on a team

This process of discovery, like finding the exit in a winding forest, takes some time. And frequently, you're doing it with a slew of other folks in a room somewhere, possibly with a whiteboard.

When I'm working collaboratively like this, a couple things tend to happen:

Someone says something that I feel is irrelevant
Someone moves too quickly in one direction and I can't follow what they're saying

These encounters can be frustrating, but here's the thing: everyone in the room is traversing the tree. We're all just in different places along the BFT—DFT spectrum.

So the second question to ask yourself is: Which of my teammates are depth-first thinkers, and which are breadth-first?

When you pose this question, you begin to understand why a detail that feels irrelevant to you matters to someone else (what path are they on compared to my own?). You also come to know why someone's thinking can move so quickly in one direction (how deep down a path are they compared to me?)

Let's go back to the forest for a second.

Imagine now that, instead of being lost by yourself, all of your teammates are with you. By now, you've figured out whether you're a depth-first thinker or a breadth-first thinker. You'll be able to find an exit more quickly by nature of the fact that you have more people to help you.

But if your teammates all think like you, you'll go down the same path together and you won't be any better off than when you were alone.

People who think differently than you see things that you can't.

If you're working on a team, first recognize what kind of thinker you are. Then, recognize what kinds of thinkers your teammates are. You'll be more patient and create better ideas.

If you're building a team, seek out different kinds of thinkers. They'll give you vision that's impossible without them.

Otherwise, you'll miss the tree for the forest.

Thinking as a tree is a simplification. Your thinking is probably more like a graph since ideas between branches are also interrelated. The concept still holds, since one traverses a graph in the same manner. ↩

Try

Tue, 19 May 2015 00:00:00 GMT

I tie my self-worth to what I do and whether I succeed. I call this the “Do or do not, there is no try” philosophy.¹

But I can spend all day doing without really trying. When it comes to my self-worth, the more important philosophy is simpler:

Try.

When I follow the “Do or do not” philosophy, I pin my self-worth to whether I did something or didn’t, however improbable the odds of success. I end up judging myself by things that are out of my control. When the odds aren’t in my favor, I become afraid to even begin.

When I try, I’m pushed to my limit, which isn’t always far enough to succeed.

In other words, I fail.

At so many things.

When I try and fail, I learn. When I learn, I get better.

Doing has never been the destination, but rather, the false means I’ve come up with to achieve a truer goal: to live with myself and love myself.

Trying never purports to be a destination, but it leads me… somewhere. It brings into focus my edges, the uncomfortable demarcations of territory where beyond is what I can be, but am not yet.

As my edges become clear, I can begin to truly know them: winding coastlines formed of insecurity, doubt. I can poke and prod them and find their soft spots. I can accept them and love them as parts of me, as integral as my skills or confidence or doings.

And I can keep pushing.

Only a Sith deals in absolutes. ↩

Friends

Thu, 10 Apr 2014 00:00:00 GMT

The other night, I met an Israeli couple who recently moved from Tel Aviv to the Bay Area. They'd lived in Israel their whole lives and it was their first time living abroad in the US. She is getting her master's degree, and he is working in Silicon Valley.

During our conversation, I asked what the hardest part of moving to California was.

"It took some time to adapt to American culture. Being in the Bay Area surrounded by tech helps, because we can still speak that common language."

"What is the difference between US and Israeli culture?" I asked.

"How often do you see your parents?"

I told them my parents live in the South Bay, not too far away, but I still see them less than I should, once or twice a month.

"Back in Israel, I saw my parents almost every day. The friends I grew up with, I saw almost every day. In the US, you make friends, great friends, in college. You become close, but then your friends might move across the country for work or you might move away for work, and you make new friends depending on where you are and who you're with.

"Part of that is because Israel is so much smaller. It's easier to go out for a drink with your friends every day. And that made it a lot harder to learn how to do this."

He gestured out to the crowd of 15 or so people talking and having dinner in the small house. I was invited over by a friend I met the week before and I knew only 3 of the 15 people there.

"But walking into this house by myself to talk to people I've never met is terrifying for me too," I said.

"You have no idea. For you, this is normal. It might feel uncomfortable, but it happens all the time, and you adapt."

"Back home, relationships are ..." he paused, searching for the right word, "deeper. There, you learn how to connect deeply with people, the same people. Here, you learn how to interact with new people."

I thought about my roommates from college, who are now spread all over the world. We half-joke over Skype about how we're going to move into a 5-story house someday with our families and live together again. The reality is that they're far away, and who knows what forces will bring us back together as our lives take shape.

"Which kind do you think is better?" I asked, "The depth or the breadth?"

They both shook their heads without hesitation.

"The question isn't 'Which is better?' as much as 'How do I survive?'"

Kindness is a Skill

Fri, 27 Dec 2013 00:00:00 GMT

Imagine that I call you up one day (we're good friends, on first name basis for years and everything), and I ask you to build me a house. Of course, you're very willing to build me a house (because we're such good friends), but you don't know the first thing about it. But I implore you.

You are distraught because you don't know where to begin. It's not like you were born knowing how to build a house — that would be very improbable. It would be like if you were born knowing how to read or write a novel or public-speak or play the cello professionally. At any rate, you'd really, really like to build this house.

So you start from the beginning. Maybe you nail together two pieces of wood. But you've never wielded a hammer, so it's hard the first time. You practice nailing two pieces of wood together until it becomes easy, and then you build larger structures using more than two pieces of wood. After you've mastered that, maybe you try building a whole wall.

You keep building and practicing and eventually you build a house. And after you've built a couple houses, you become really good at it, and you build me a beautiful house and we become even better friends because you often come over and we drink tea and talk about books or just sit around.

But learning how to build houses was never easy. Remember when you thought you couldn't do it because you had no idea how to begin? When I asked you to build me a house, you could have told me, "I can't build you a house because I don't have the mind for it. I'm just not able to. It's not in my nature." A lot of people seem to take this attitude toward math and programming, that they'll never be able to figure it out, even though, like building a house, these things are iterative processes of learning that take time and effort. It's probably true that it's naturally easier for some people to build houses. But nobody was born knowing how to build a house. They had to figure it out, and figuring things out is hard. You can seek more skilled carpenters' and architects' advice, but at the end of the day, you still need to expend effort to develop your own house-building skill.

Now imagine that you and I are not friends and that we don't even know each other. Imagine instead that you believe you are selfish and kind of an asshole, but you wish to be kind. Sometimes you say terrible things and your behavior often benefits yourself at the expense of others.

But you feel guilty about the way you are.

You want to be kind.

But you aren't.

You look at your friends who are generous and who consistently stick their necks out for you in situations you'd probably just avoid, and you think to yourself, "Dammit, they are so kind."

"And I am not."

You feel even more guilty because you see other people being kind, and you're unsure why you aren't able to act that way. Is there something wrong with you deep down? Maybe you start to believe that it's not in your nature. And you feel even worse.

But maybe it occurs to you that feeling guilty about not being kind is like feeling guilty for not knowing how to build a house when your carpenter and architecture friends do it with such ease.

Maybe it occurs to you that nobody was born knowing how to be kind. (I mean, do you remember being a kid? They aren't always the nicest bunch.)

Maybe it occurs to you that learning anything when you know nothing is hard, but the more you build and practice, the better you become. The more you learn about what it means to be kind. The more you learn about what it means to empathize. The more you see the house instead of the two pieces of wood nailed together.

Maybe you begin to understand how being kind of an asshole is also a skill, and that being kind of an asshole more frequently makes it easier to be kind of an asshole.

Maybe you realize that kindness must be practiced, that kindness becomes easier, but greater kindness is always challenging.

Not because you're fighting against your nature, but because you're learning.

Handling GCM messages in the foreground and background

Wed, 13 Nov 2013 00:00:00 GMT

When the Clef Android app gets notifications, we want it to behave differently when it's open and in the foreground than when it's in the background.

This is a surprisingly difficult thing to do, since it's actually quite difficult to determine whether an app is in the background. You can use a static boolean or keep track of the shown activity (maybe in a custom Application class or another singleton), but that isn't always reliable. You might also have other requirements—what if you want to differentiate behavior based on which activity is in the foreground?

One solution is sendOrderedBroadcast, but I prefer something with a little less boilerplate.

All aboard the event bus

Instead of using BroadcastReceivers, we can use an event bus like Otto or EventBus to notify activities about an incoming GCM message. If no activities are registered on the bus, we can correctly place a notification in the notification drawer.

If you aren't familiar with event buses, the idea is simple. Any object (usually an Activity) can register with (or as I like to say, ride) the event bus:

bus.register(this);

The object then provides subscriber methods to run when the bus emits an event:

public void onEvent(NotificationEvent event) {
    Toast.makeText(this, 
        event.payload.getString("message"), 
        LENGTH_SHORT)
    .show();
}

When an object should stop riding the event bus, you can unregister it:

bus.unregister(this);

To actually emit events:

NotificationEvent event = new NotificationEvent(bundle);
bus.post(event);

This triggers a call to the onEvent(NotificationEvent event) method in every object registered on the bus. You can add different subscriber methods (such as onEvent(LocationEvent event)) and these method will only be run when an event of the same type is emitted (bus.post(new LocationEvent())) There is no limit to how many objects can register with the bus.

If there are no objects registered, a second event is automatically posted. With Otto, this event is called DeadEvent. With EventBus, it's called NoSubscriberEvent. This lets you separately handle cases where there are no subscribers (you might see where this is going).

You can also post any object. I like to create classes that logically separate events and suffix them with Event. For instance, the NotificationEvent class looks like this:

public class NotificationEvent {
    public Bundle payload;

    public NotificationEvent(Bundle extras) {
        this.payload = extras;
    }
}

Otto and EventBus

These libraries are very similar. Otto uses annotations (@Subscribe) to determine which subscriber methods to run, while EventBus uses a consistent method name (onEvent).

Unfortunately, both libraries have drawbacks. Otto doesn't let you register with the bus in a superclass and receive events in a subclass. This is annoying if you'd like to register in a BaseActivity that all of your activities extend from.

EventBus has a bug when trying to subscribe to the NoSubscriberEvent. If you register and unregister an object, the event never fires. Bummer. There's a pull request that adds a one line fix, but the maintainers haven't touched it yet.

I ended up choosing EventBus because I really wanted to be able to register with the bus in a base class¹, but again, they are basically the same for most purposes. I did have to add the patch in this pull request to make it work.

Changing app behavior for GCM messages

Here's how we use an event bus to do different things when the app is in the foreground or background.

We use a singleton event bus. The EventBus library provides one for you with EventBus.getInstance(). It's also easy to make a singleton bus with Otto—you can even use Dagger to inject the global bus where needed (to truly Square-ify your code).

Every activity registers with the bus when it is shown and unregisters when it is hidden. This ensures that only one activity receives events from the bus at a time. We also add a subscriber method to handle the notification. We accomplish this with a base class:

public abstract class BaseActivity extends Activity {
    @Override
    public void onResume() {
        super.onResume();
        EventBus.getInstance().register(this);
    }
    
    @Override
    public void onPause() {
        super.onPause();
        EventBus.getInstance().unregister(this);
    }

    public void onEvent(NotificationEvent notification) {
        startActivity(new Intent(BaseActivity.this, NotificationActivity.class));
        overridePendingTransition(R.anim.fade_in_animation, R.anim.fade_out_animation);
    }

}

Then, in our GCM Receiver, we post to the event bus:

public class GcmReceiver extends BroadcastReceiver {
    @Override
    public void onReceive(Context context, Intent intent) {
        GoogleCloudMessaging gcm = GoogleCloudMessaging.getInstance(context);
        String gcmMessageType = gcm.getMessageType(intent);
        if (GoogleCloudMessaging.MESSAGE_TYPE_MESSAGE.equals(gcmMessageType)) {
            Bundle payload = intent.getExtras();
            NotificationEvent notification = new NotificationEvent(payload);

            // Register ourselves so we can receive the NoSubscriberEvent notification if no
            // activities are registered
            EventBus.getDefault().register(this, NoSubscriberEvent.class);

            // Post the notification for an activity
            EventBus.getDefault().post(notification);
        }
        setResultCode(Activity.RESULT_OK);
    }

The real magic happens in this line:

    // Register ourselves so we can receive the NoSubscriberEvent notification if no
    // activities are registered
    EventBus.getDefault().register(this, NoSubscriberEvent.class);

We register the GcmReceiver with the event bus to receive only the NoSubscriberEvent. If no activity is being displayed, then nothing is registered for NotificationEvent. EventBus will then post the NoSubscriberEvent, which we can handle in GcmReceiver like so:

/*
 * Called when there are no subscribers to the NotificationEvent.
 *
 * This occurs when the app is in the background and no activities are riding the bus.
 */
public void onEvent(NoSubscriberEvent deadEvent) {
    // We're all done here
    EventBus.getDefault().unregister(this);

    NotificationEvent notification = (NotificationEvent) deadEvent.originalEvent;

    // Save the wee little notification for later consumption by an activity
    Mailbox.getInstance().put(notification);

    // Display the notification in the notification drawer
    buildNotification(notification);
}

buildNotification puts the notification in the notification drawer with a PendingIntent. We also save the notification in a global queue, so that when application is opened back up not from the notification drawer, we can process it as normal. There's probably a better solution than this, since it's possible for the Mailbox singleton to be deallocated when the app is in the background.

Both EventBus and Otto also support "sticky" events. Once you post an event with postSticky, EventBus caches it, so you can get the last event back with bus.getStickyEvent(TheEventClass.class). Otto allows you to use a default value by defining a method with a @Provides annotation. In this example, sticky events could replace storing the notification away in Mailbox.

Differentiating notifications

Your app probably sends more than one kind of GCM message. If that's the case, you can subclass NotificationEvent to handle different message types. When you register an activity with the event bus, you can write your subscriber functions so they only respond to certain kinds of notifications. This is a nice way to use the built-in type system, instead of doing something like payload.getString("type").equals(aMessageType) everywhere.

Conclusion

Event buses are awesome. There are many more use cases, such as communicating between fragments in a sane way (without using interfaces) or making callbacks from AsyncTasks (although, I prefer replacing AsyncTask with RxJava altogether²).

Though there is a workaround for Otto. ↩
Event buses could actually replace RxJava for a lot of simpler use cases. Otto and EventBus both provide some control over threading so you can do things in the background, and it might make for less complex looking code. ↩

Happiness and Fulfillment

Tue, 05 Nov 2013 00:00:00 GMT

Two patients lie in their death beds in a hospital. Maybe they're both surrounded by their family and friends. Maybe they're in the Northwest as the weather gets colder and they both realize independently, at slightly different times, that they'll never see the sun again. Maybe they each have regrets.

One thing I can say for certain, as the author of this little story, is that one patient's life was spent in the pursuit of happiness, while the other's was spent in the pursuit of fulfillment. Whether they each found it is unbeknownst to me. I also can only speculate what they thought as they quietly slipped away, as the steady beep of the EKG disappeared into a deafening solid note, which the nurses quickly cut out partly because we got the message and partly because the sound itself is so obnoxious.

In that instant, I suppose it doesn't matter whether the one found happiness or the other found fulfillment, or even which they chose to pursue—whether they spent their lives frolicking in the grass or anti-frolicking in a metaphorical windowless office. I suppose it doesn't matter to the two patients, at that very moment and every one thereafter, whether or not they figured it all out.

But for the rest of us, it does.

Every day, I wake up and struggle with whether I want to be myself and happy or whether I want to be better and fulfilled.

Happiness

I've spent most of my life in the pursuit of happiness¹, but the times I've actually felt happy are when I stopped trying. Happiness is an ephemeral moment when I'm not self-conscious, when I'm not worried about what I'm doing with my body or what I look like, when I'm completely present. It's taken me a long time to understand what that feels like and I think it's why people like dancing, and why I like music festivals².

When I'm present, I forget the past, I forget the future, I forget my aspirations. I forget who I am in my mind's eye, because I'm focused on being who I am³.

I get lost in the present. In a kind of twisted way, that presence also implies an absence. I'm here but I'm also not really there, because there's no context for me being there.

In that blissful moment⁴, however, I don't notice an absence. I simply accept the beauty that everything is the way it is.

It isn't enough

And then, something creeps into my psyche. I start to feel complacent. Stagnated.

How do I reconcile this complacency with my desire to be better, to do better, to not simply accept my experience and my fate as it is?

Fulfillment

Some would say you can't. When you feel that complacency, it's because life is about fulfillment, not happiness, and you aren't fulfilling yourself. And that's true—I feel guilty⁵ for not trying to better myself, because I know I can be more. I know I can be part of something greater. When I strive to better myself, when I build something, when I'm uncomfortable, I'm satisfied. I'm contributing. I've felt this way a lot while working on Clef.

It isn't enough either

You can say that you're happy when you're bettering yourself, but that's an issue of semantics, and I'd call that fulfillment.

When I strive to better myself, it's out of discontent with who I am now. That's pretty opposed to my happiness. Sometimes that discontent is totally concrete and I know exactly what I need to do, but most of the time, it's an itch more than anything else. A reminder in the back of my head that what I'm doing isn't enough.

I don't have a satisfying answer

If you're looking for someone to tell you that happiness isn't important, it isn't me. But I cannot tell you that fulfillment is the answer either, sorry.

The pursuit of fulfillment is torture. Perhaps if I were driven by some purely-good, higher purpose, it wouldn't be. But I'm driven by a feeling of incompleteness and my insecurities.

Trust yourself

You can't have only happiness or only fulfillment. You need to have a balance.

"Balance" is a pretty wholly disappointing answer, but I'm not surprised it isn't some grand revelation (seeing how people have been thinking about this sort of thing for a very long time).

It raises the next question: How do I know what the right balance is?

That isn't something I can answer for you. Hell, it's something I can barely answer for myself. But I try to approach it from a place of self-love, where I can begin to accept who I am, even when I'm not happy with who I am.

When I feel right with the world, that is how I am. When I feel discontent and I strive to better myself, that is also how I am. I can't fight against the way I feel in a given moment, because in that moment, I'm being me.

So when I want to float in the waves, I revel in that fact. And when I'm no longer content just floating, when I want to swim my ass off and ride one into the shore, I revel in that fact too.

Trusting myself to do both is hard. I worry that when I'm floating or when I'm swimming, I'll never have the desire for the other again. It isn't that I want to be one way or another—it's the self-doubt in either state that drives me crazy. So I have to trust that what I'm doing at any given moment is right for me, and listen to myself when it isn't.

When I am one of those patients, wondering whether I'll ever see the sun again, I probably still won't know whether I've "achieved" fulfillment or happiness. And it probably won't matter⁶.

What will be true is when I wanted to be better, I tried, and when I wanted to be me, I was.

And hopefully, the nurse will turn that obnoxious sound off sooner rather than later.

Well, I spent high school in the wholehearted pursuit of being cool, which I guess I did because I thought it would make me happy. ↩
The first music festival I ever attended was Sasquatch 2013. I'll have to write another post about how to approach festivals, but in a nutshell: festivals can be an incredibly safe space for exploring presence and being yourself. Some call it letting loose. ↩
Maybe a better way to say this is that there is no I to be focused on anything. ↩
That "moment" could last weeks or months. ↩
I wonder how much of that is a result of my own drive and how much is a result of constantly comparing myself to others. ↩
Or maybe it will! The living will never know. ↩

RxJava and Android: Just What the Doctor Ordered

Tue, 15 Oct 2013 00:00:00 GMT

I work on Android at Clef. Most of the time, though, it feels like I'm working against Android. The Activity and Service model is elegantly designed for interoperability, but when it comes down to implementation, it's often painful to work with.¹

But a bigger problem is handling user interaction. There are two very common patterns that I have trouble with:

Doing something in the background and updating the UI.
Responding to user interaction and updating a different part of the UI.

RxJava

In both cases, RxJava helps me keep my sanity. RxJava is a Java implementation of Functional Reactive Programming. If you aren't familiar with FRP, it's not as complicated as its name makes it sound. FRP is all about streams of data. A stream produces data at different points in time. An observer is notified whenever data in that stream and does something with it. It's a little different than the Observer pattern, since there's a focus on composition. You can map, filter, and reduce streams—that's where the "functional" part comes in.

Doing things in the background

The old way: AsyncTask

It's an inconvenient necessity that we have to worry about blocking the UI thread. Android provides AsyncTask to help ease this pain, letting you do work on the background with doInBackground and then delivering on the UI thread with onPostExecute. For the most part, the AsyncTasks I create are requests to Clef's API server that return JSON. But there are a few pitfalls with AsyncTask:

Error handling: as mttkay points out, there's no easy way to handle error messages in the UI for errors that happen in the background. I usually return null instead of the actual JSONObject and then later handle the case where the JSONObject is null. Ugh.
Code reuse/readability: Since API requests are all similar, I use a general APIRequestTask that I give a path and some parameters. I also pass in the caller which implements a APIRequestHandler interface, so the caller can handle the results.

In the activity/fragment:

APIRequestTask request = new APIRequestTask("/user/register", this);
request.putParameter("email", "test@example.com");
request.execute();

In APIRequestTask.java:

public interface NetworkRequestHandler {
    public void onRequestCompleted(JSONObject response, String url);
}

public NetworkRequestTask(String url, NetworkRequestHandler handler) {
    this.url = url;
    this.mHandler = handler;
    this.requestParameters = new ArrayList<NameValuePair>();
}

public void putParameter(String key, String value) {
    if (key != null && value != null) {
        requestParameters.add(new BasicNameValuePair(key, value));
    }
}

@Override
protected JSONObject doInBackground(Void... voids) {
    // Do the request
}

@Override
protected void onPostExecute(JSONObject response) {
    if (mHandler != null) {
        mHandler.onRequestCompleted(response, this.url);
    }
}

Though the generality is nice, with this pattern, if a calling Activity/Fragment makes multiple requests, it needs to handle responses in a single function by switching on the URL. This makes for code that isn't very readable, especially when you have to jump from where the request is executed to where the response is handled.

Once more, with feeling

Let's try that again with RxJava:

Observable.create(new Observable.OnSubscribeFunc<JSONObject>() {
    @Override
    public Subscription onSubscribe(Observer<? super JSONObject> observer) {
        try {
            // Do the request
            // ...
            observer.onNext(jsonResponse);
            observer.onCompleted();
        }
        catch (Exception e) {
            observer.onError(e);
        }

        return Subscriptions.empty()
    }
}).subscribeOn(Schedulers.newThread()).observeOn(AndroidSchedulers.mainThread())
  .subscribe(new Action1<JSONObject>() {
    @Override
    public void call(JSONObject response) {
        if (response.getBoolean("success")) {
            RegistrationActivity.this.messageView.setText("You're all set.");
        }
    }
}, new Action1<Throwable>() {
    @Override
    public void call(Throwable error) {
        Toast.makeText(RegistrationActivity.this,
            error.getMessage(),
            Toast.LENGTH_SHORT)
        .show();
    }
});

Observable.create creates a stream that makes a network request. When the request completes, the response is delivered from the stream to an observer. In this case, the observer is the first Action1 in the subscribe method. If there are any errors, they are delivered to the second Action1 passed into the subscribe method.

subscribeOn(Schedulers.newThread()) runs the OnSubscribeFunc on a thread, while observeOn(AndroidSchedulers.mainThread()) delivers the results to the main thread.

It's still a little verbose, but it's more similar to writing synchronous code. I use the response in code that directly follows the request. I also get error handling on the UI thread for free.

Composition

RxJava also lets you compose logically separated tasks. mttkay gives this example:

Observable<String> filePathObservable = downloadFileObservable().map(new Func1<File, String>() {
    @Override
    public String call(File file) {
        return file.getAbsolutePath();
    }
});

// now emits file paths, not `File`s
subscription = filePathObservable.subscribe(/* Observer<String> */);

Unfortunately, if return file.getAbsolutePath(); throws an error, we run into the same problem that we had with AsyncTasks: how do we handle the error when we don't have a reference to an observer? Instead of map, we must use flatMap. Here's an example where we tokenize a credit card into a Balanced uri and then use the uri in a request to our server:

final Card card = new Card(cardNumber, expiryMonth, expiryYear, cvc);

Observable.create(new Observable.OnSubscribeFunc<String>() {
    @Override
    public Subscription onSubscribe(Observer<? super String> observer) {
        try {
            Balanced balanced = new Balanced(Constants.BALANCED_MARKETPLACE_URI, NewCreditCardActivity.this);
            String cardURI = balanced.tokenizeCard(card);
            observer.onNext(cardURI);
            observer.onCompleted();
        } catch (Exception e) {
            observer.onError(e);
        }

        return Subscriptions.empty();
    }
})
.flatMap(new Func1<String, Observable<JSONObject>>() {
    @Override
    public Observable<JSONObject> call(String cardURI) {
        return Observable.create(new Observable.OnSubscribeFunc<JSONObject>() {
            @Override
            public Subscription onSubscribe(Observer<? super JSONObject> observer) {
                try {
                    JSONObject response = doTheRequest(cardURI); 
                    observer.onNext(response);
                    observer.onCompleted();
                }
                catch (Exception e) {
                    observer.onError(e);
                }
                return Subscriptions.empty();
            }
        });
    }
})
.subscribe(/* ... */);

Unfortunately, maintaining the error handling abilities makes the code much more verbose. However, it also makes it a lot easier to display errors to the UI and chain separate, dependent asynchronous operations in a readable way.

Edit: lungos in the comments pointed out that the using flatMap may be overkill. Any exceptions in the flatMap's OnSubscribeFunc will be caught by the try/catch block in the original Observable's OnSubscribeFunc. In fact, flatMap is not necessary here at all, because map in RxJava will wrap return values in an Observable automatically, making map and flatMap essentially the same. They aren't actually the same, but the nuance requires an explanation of monads and other theory that I don't want to get into.

Responding to user interaction

RxJava isn't just for background operations. While RxJava isn't yet as fully featured as its Objective-C counterpart, ReactiveCocoa, it's still really useful for making a responsive UI. It also enforces a separation of view and model logic.

For instance, say I'm displaying information about a pending payment. Instead of storing payment information as member variables of an activity, I store them in a separate Payment model object (which is good practice anyway). A user can select a shipping option that increases a total amount of the payment. I could create an onItemClickListener which updates the amount TextView, but this entangles the view logic with the underlying model logic.

With RxJava, I can use a Subject to automatically update the UI. Subjects are observables that you can both subscribe to and trigger updates on. With Subjects, you don't need a reference to an observer—you just emit the data on the subject itself.

public class Payment {
    private Integer totalAmount; 
    private ReplaySubject<Integer> totalAmountSubject = new ReplaySubject.create();

    /* ... */

    public void setTotalAmount(Integer newAmount) {
        this.totalAmount = newAmount; 
        this.totalAmountSubject.onNext(this.totalAmount);
    }

    public Observable<Integer> totalAmountObservable() {
        return this.totalAmountSubject.observeOn(AndroidSchedulers.mainThread());
    }

    /* ... */
}

Now, I can subscribe to totalAmountObservable in the Activity like so:

this.totalAmountSubscription = this.payment.totalAmountInCentsObservable().subscribe(new Action1<Integer>() {
    @Override
    public void call(Integer newAmount) {
        Log.d(TAG, "Total amount in cents has been updated!");
        TextView amountView = (TextView) findViewById(R.id.price);
        amountView.setText(Format.formatCents(newAmount));
    }
});

In the onItemClickListener (and anywhere else), I can now just use setTotalAmount, and the view is automatically updated to reflect the new amount.

If you've used Backbone.js or another JavaScript framework, this one-way binding might not be new to you, but it's nice to have it available on Android. There's also an open issue on GitHub for adding more UI hooks and make it even more usable.

Edit: @bryanstern pointed out that you can use a BehaviorSubject instead of a ReplaySubject to the same effect. The difference between them is that BehaviorSubject emits only the last item when it is subscribed to, while ReplaySubject emits all previous items. That means if you do subject.onNext(foo1); subject.onNext(foo2); you'll get foo1 and then foo2 with a ReplaySubject, but just foo2 with a BehaviorSubject. Check out the different kinds of subjects in the docs.

Final thoughts

RxJava has been a pleasure to use so far. There are a few snags, like remembering to include subscribeOn(Schedulers.newThread()) and observeOn(AndroidSchedulers.mainThread()), but being able to properly handle errors and write more readable code is essential.

What are your experiences using RxJava with Android? Am I doing anything egregiously wrong? Let me know in the comments.

You should follow me on Twitter.

For instance, detecting when an app is being backgrounded is simple on iOS because a delegate function is provided for you. On Android, however, we are forced to manually keep track of whether activities are open (or other similar hackery). Again, the Activity model is beautiful, but sometimes, we need to keep global state, dammit. ↩

Docking at Port

Tue, 24 Sep 2013 00:00:03 GMT

This post is part of a larger media experiment describing why, and how, Clef moved to Oakland. If you want to read more, visit the timeline at oakland.is/home or tell us your thoughts with #oaklandis on Twitter.

We woke up the next morning sore and tired from sleeping on the floor. We drove in to our new office and didn't know what to expect. What we found was a burgeoning community that welcomed us openly.

The Port at 101 Broadway sits on the edge of Jack London Square. A 30-second walk takes you to the water, where small boats wait at the docks and a hulking freighter unloads container after container with a tall crane off in the distance.

While the breeze out there is refreshing, all the good stuff is happening within the red brick walls of the Port.¹ We've been here three weeks, and it felt like home two weeks ago.

The first (real) day we arrived, we were exhausted from the move. Aaron and Ryan from ShopPad immediately invited us to play ping pong. Though we didn't actually play (we had to run into a meeting), I was caught off guard in the best way. At our old office, we were friendly with the other companies nearby, but never really interacted with them socially.²

And Michael, Zoey and Stephen helped with everything else.

Michael is one of the founders of the Port and an architect who helped design Pier 39. When we jokingly asked him for a bike room, he told us he was already making one.³

Stephen runs IT and general operations. When our router stopped working, Stephen got mobile hotspots for the whole co-working space.

And Zoey does community management. She designed a drip coffee bar that cuts down on plastic waste for everyone to use. We even had the crew over at Mr. Espresso give us a lesson on how to make perfect drip coffee.⁴

The staff (Michael, Zoey, Stephen and many others) and the co-workers at Port are passionate about what they do. And what they do isn’t the same buzzword-heavy software play that overcrowds co-working spaces across the Bay. They’re in music, law, art, home decor, cannabis, architecture, video production and even Antarctica.⁵ There is software too, of course, but it’s more focused. When we get together for weekly dinners or for a game of pool, we don’t just rehash the tech news of the day. We talk about a co-worker’s homemade wine, the best places to catch the sun setting over the Bay, experiences at Burning Man or what’s showing at next month’s Art Murmur.

Port isn’t more cultured or anything like that–it’s just more diverse. It isn’t contaminated by technical elitism or overt, cold competition. It’s filled with interested, disparate people who are making something real.

And they’re friendly as hell, too.

The last piece in this series is Oakland.

Well, that’s kind of an exaggeration. There are some great restaurants and events happening in Jack London. Check out Jack's Night Market for an awesome mix of food and art. ↩
Of course, that's our fault too. It's easy to get into the rhythm of just working at work and neglecting the other aspects of the people around you – and that goes for a company of any size. The difference at Port was that people made it easy to engage them. ↩
And it got built in the next 2 days. ↩
A drip coffee setup seems pretentious, but it’s a beautiful alternative to the water cooler. We’ve had drip coffee for about a week, and the number of interactions I’ve had with fellow co-workers has risen dramatically. It’s also about 10 bucks to make. ↩
One startup at Port is building a way for researchers in Antarctica to share their experiences with each other and the outside world. They previously worked at Palmer Station. ↩

The 10-Hour Move

Tue, 24 Sep 2013 00:00:02 GMT

Brennen promised it would be like ripping off a bandaid. In the end, he was right. But at the time, it felt like ripping off the bandaid, putting it back on, and then ripping it off a few more times for good measure.

We moved to Oakland in less than 10 hours.

That morning, we were working at our windowless, basement office in San Francisco. By the evening, we were sleeping on the floor of our new home in Oakland, ready to wake up the next day and drive to our new office at Jack London Square.

3pm

The Bay Bridge closes tomorrow. We have to drive over the bridge to meet the owner of a house anyway, so Brennen emails Port Workspaces and asks if we can sign a year-long lease–and move in all our stuff–today. They respond almost immediately and give us the go ahead.

We start disassembling the office.

5pm

Traffic is heavy on the bridge. We finally arrive at the house, just northwest of Lake Merritt. As we walked in, we spot the owner and the realtor in the backyard, watering some plants and laughing. We'd actually scheduled another house to visit at 5:30pm, but in 20 minutes of meeting the owner and seeing the place again, we are talking about signing the lease. It's the place we want to call home.

5:30pm

Since it's such short notice, Port isn't quite ready with keys for us. But we don't mind. We'd just driven south along Lake Merritt and realized we would be biking that route every day to work. Though the necklace of lights isn't yet lit, our faces are glowing.

6-7pm

One thing we notice immediately about Port is the sheer amount of glass (we actually have windows!). We get the idea to make our own whiteboards with panes of glass, hung on the wall with industrial velcro.¹

We take a trip to Home Depot to get panes of glass and some other materials, including velcro and a drill. Glass is much, much heavier and sharper than I would have expected, but I've also never handled glass before. As we drive away from Home Depot, I keep peering into the backseat, wondering whether the glass will shatter. When we get back to Port, we assemble the whiteboards and leave them on the floor to let the velcro adhesive dry.²

8:35pm (Back to San Francisco)

We still need to get the rest of our stuff from the old office, so we head back over the bridge. This is the part that feels like slapping back on the bandaid. When we come back to our old office to get the tables, it no longer feels ours. The shadows cast from street lights up above warp the walls. The office is done with us. We shove all the little items into boxes and load the car with tables.

9:30pm

It is time for a celebratory meal at Chipotle.³ As we eat our last burritos as founders in San Francisco, we overhear a group of guys at another table talking about a senior executive at a large company. "I heard she slept with X (another executive) when she was working there." There's that bandaid again.

11:30pm

We head back over the bridge, all very happy that we won't have to make another trip tonight. As we shuttle between the car and our office on the third floor, we hear the last train roll by and the bustle of people returning home from a late dinner.

Our old office is empty and a memory to us. It's weird to leave a place completely, to be in a routine in one moment and then gone the next, without the natural distancing of sleep.

But here we are, nearing midnight, standing in our new office with our whiteboards drying and our standing desks ready.

We still didn't have mattresses for our home (or even a fridge for that matter). But moving was like ripping off a bandaid, and underneath, I found the wound was fully healed. We had arrived.

The next piece in this series is One weird trick to furnish your new home for free. (Don't worry, it isn't spam.)

We later found out that Port would have painted our walls with dry-erase paint if we had asked. ↩
We'll cover how we did this in another post. ↩
It wasn't all that celebratory. I was really hungry and really, really irritable at the time. ↩

Moving Alone

Tue, 24 Sep 2013 00:00:01 GMT

I think box manufacturers purposefully make boxes just a little too awkwardly shaped and large. So when you try to move them by yourself, it’s nearly impossible, but when you have another person, it’s way too easy.

We'd missed it.

The lease was up and we hadn't found another place, so we had to move into my parents’ house in Los Altos.

Actually, I had to move into my parent’s house in Los Altos. Jesse was backpacking for the weekend with his father and brother in the Sierras, something he’d planned months ago, and Brennen was speaking at WordCamp in Providence. I wasn’t angry they were gone and I didn’t feel like they deserted me. I just had to move everything by myself, and that sucks. A certain surreal feeling arises when profound emotion is mixed with administrative tasks. I’ve felt it when family dies and everyone is crying and sad, but the mortician still needs papers signed for the cremation and arrangements need to be made for the funeral, and we have to decide what type of flowers to have and what box to pick to hold the ashes.¹ I’m still not sure whether the mundane tasks help ease emotional pain or make it worse.

I can’t in good faith compare moving to a family death–that would be ridiculous. But I was still terrified. I had no sense of permanence. How long would we be staying with my dad? Two weeks? A month? A year? How long would it be until the commute made me lose it?²

Working at a startup means experiencing uncertainty every day. That takes a toll, but we bear it willingly because we love what we’re doing. But I don’t want everything in my life to be like that. I want to come back to a home, not just a place to sleep. I need a little certainty too. I don’t want to think about where I’ll be living in two weeks when I’m trying to help build a company that will last years.

I was terrified, and I was alone. And there the boxes and suitcases were by the door, mocking me. So I did what people do when they’re scared and unsure about the monolithic future. I started the administrative tasks. I sat for a moment, grabbed two smaller bags and began loading up the car.

The next piece in this series is Wandering Eyes.

It’s weird because you’re not sure whether the mortician is trying to upsell you. I wonder if the mortician feels guilty about how easy that is. ↩
Turns out, it was about two weeks. See the 10 hour move. ↩

Slim Pickings in San Francisco

Tue, 24 Sep 2013 00:00:00 GMT

It gets foggy in Excelsior. Not as much as the Sunset, but the "wall of fog" up there kind of leaks down and casts a gray in the mornings and evenings.

On July 27th, though, the sun was rearing its head, and we were feeling good.

We were about to visit an apartment only a few blocks away with high hopes. Rent was only $3000 a month, the bedrooms looked sunny and big in the Craigslist post, and it was almost next door. It wouldn't be the palace we'd found for the summer, but it was close enough (and cheap enough) that we'd be happy.

I never considered $3000 a month cheap until I started looking for apartments in San Francisco. Every day we looked, we lowered our expectations a little more. It's amazing what relative pricing does to the brain - we started out with an absolute maximum of $3300, and by the end of the week, we were all looking at the $4000 apartments and wondering, "Why couldn't we?"

That day I was bluntly educated by the city of San Francisco about what $3000 a month gets you. The bedrooms were tiny and narrow, like slightly widened hallways and nothing like the pictures. The bathroom (only one) was very small, and even though I'm short,¹ I could barely fit my head underneath the shower head. But these were just gripes, things we could live with and get accustomed to over time.

We continued doing our walkthrough-I like to circle around a few times pretending like I know all the things a responsible renter is supposed to look for-until we realized: there's no kitchen. The three of us were piled into the bathroom, quietly wondering aloud where the kitchen was, when we heard another prospective tenant ask the owner the same question.

"Sorry, there's no kitchen. But you can put a hotplate right here."²

Brennen turned to me, flattened his mouth and said, "Nope," and we booked it out of there.

The place was not physically unlivable.³ It had bedrooms, it had bathrooms, it had a place where food could be cooked. But it was $3000 a month. I didn't want to live there for a year and pay that price. What pained me is that most people couldn't pay that price. I've read the articles about tech people invading the city, paying 10 bucks for a slice of bread and a cup of coffee. And there I was, a tech person who comes from privilege in San Francisco looking for an apartment, increasing the cost of living for everyone. I wasn't an artist or an SF local or a student. I was the reason why, for $3000 a month, you got an apartment with a little counter where you could put a hotplate.

I didn't want to be a part of that problem.

I didn't want to make people pay exorbitantly to just get by.

But dammit, I needed a place to live.

The next piece in this series is Moving Alone.

Hey, still above average. ↩
Something I (a naive college graduate) learned from apartment hunting: people lie on Craigslist. Well, they don't lie outright, they just don't tell you the whole truth until you actually get over there. Apartment hunting, like most human interaction, unfortunately, is all about due diligence. ↩
Though legally, it wasn't an in-law apartment as advertised, since an in-law apartment must have a kitchen. ↩

Mark Hudnall's website

How many servers do we need?

An initial guess using Little's law

Erlang's C formula

Deriving the logarithmic Erlang C formula

Python code

Beyond the Erlang model

M/M/c queues

Footnotes

Three things about Common Lisp

1. Error handling

2. Dynamic variables

3. Interactivity

First impressions of Nix

What is Nix?

The big idea

A motivating example: Homebrew and openssl

So what's the underlying problem?

Challenges

Installing on macOS Catalina and later

nix-darwin

Applications and spotlight

Paper-cuts

Nix expression language

Everything is an attribute set

CLI tools

"Now I have two build tools"

Resources

Footnotes

Widening circles

Starting Figwheel in Emacs

Footnotes

Depth-first and breadth-first thinkers

Depth-first search

Breadth-first search

Thinking as a tree

Working on a team

Footnotes

Try

Footnotes

Friends

Kindness is a Skill

Handling GCM messages in the foreground and background

All aboard the event bus

Otto and EventBus

Changing app behavior for GCM messages

Differentiating notifications

Conclusion

Footnotes

Happiness and Fulfillment

Happiness

It isn't enough

Fulfillment

It isn't enough either

I don't have a satisfying answer

Trust yourself

Footnotes

RxJava and Android: Just What the Doctor Ordered

RxJava

Doing things in the background

The old way: AsyncTask

Once more, with feeling

Composition

Responding to user interaction

Final thoughts

Footnotes

Docking at Port

Footnotes

The 10-Hour Move

3pm

5pm

5:30pm

6-7pm

8:35pm (Back to San Francisco)

9:30pm

11:30pm

Footnotes

Moving Alone

Footnotes

Slim Pickings in San Francisco