# rss_item

guid: http://tom.preston-werner.com/2008/10/18/how-i-turned-down-300k
title: How I Turned Down $300,000 from Microsoft to go Full-Time on GitHub
url: http://tom.preston-werner.com/2008/10/18/how-i-turned-down-300k.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="how-i-turned-down-300000-from-microsoft-to-go-full-time-on-github">How I Turned Down $300,000 from Microsoft to go Full-Time on GitHub</h1>

<p class="meta">18 Oct 2008 - San Francisco</p>

<p>2008 is a leap year. That means that three hundred and sixty six days ago,
almost to the minute, I was sitting alone in a booth at Zeke’s Sports Bar and
Grill on 3rd Street in San Francisco. I wouldn’t normally hang out at a sports
bar, let alone a sports bar in SOMA, but back then Thursday was “I Can Has Ruby”
night. I guess back then “I can has <strong>__</strong>_” was also a reasonable moniker to
attach to pretty much anything. ICHR was a semi-private meeting of like minded
Ruby Hackers that generally and willingly devolved into late night drinking
sessions. Normally these nights would fade away like my hangover the next
morning, but this night was different. This was the night that
<a href="http://github.com/">GitHub</a> was born.</p>

<p>I think I was sitting at the booth alone because I’d just ordered a fresh Fat
Tire and needed a short break from the socializing that was happening over at
the long tables in the dimly lit aft portion of the bar. On the fifth or sixth
sip, Chris Wanstrath walked in. I have trouble remembering now if I’d even
classify Chris and I as “friends” at the time. We knew each other through Ruby
meetups and conferences, but only casually. Like a mutual “hey, I think your
code is awesome” kind of thing. I’m not sure what made me do it, but I gestured
him over to the booth and said “dude, check this out.” About a week earlier I’d
started work on a project called <a href="http://github.com/mojombo/grit">Grit</a> that
allowed me to access Git repositories in an object oriented manner via Ruby
code. Chris was one of only a handful of Rubyists at the time that was starting
to become serious about Git. He sat down and I started showing him what I had.
It wasn’t much, but it was enough to see that it had sparked something in Chris.
Sensing this, I launched into my half-baked idea for some sort of website that
acted as hub for coders to share their Git repositories. I even had a name:
GitHub. I may be paraphrasing, but his response was along the lines of a very
emphatic “I’m in. Let’s do it!”</p>

<p>The next night, Friday, October 19, 2007 at 10:24pm Chris made the first commit
to the GitHub repository and sealed in digital stone the beginning of our joint
venture. There were, so far, no agreements of any kind regarding how things
would proceed. Just two guys that decided to hack together on something that
sounded cool.</p>

<p>Remember those amazing few minutes in Karate Kid where Daniel is training to
become a martial arts expert? Remember the music? Well, you should probably go
buy and listen to <a href="http://phobos.apple.com/WebObjects/MZStore.woa/wa/viewAlbum?i=260417864&amp;id=260417040&amp;s=143441">You’re The
Best</a>
by Joe Esposito in iTunes because I’m about to hit you with a montage.</p>

<p>For the next three months Chris and I spent ridiculous hours planning and coding
GitHub. I kept going with Grit and designed the UI. Chris built out the Rails
app. We met in person every Saturday to make design decisions and try to figure
out what the hell our pricing plan would look like. I remember one very rainy
day we talked for a good two hours about various pricing strategies over some of
the best Vietnamese egg rolls in the city. All of this we did while holding
other engagements. I, for one, was employed full time at Powerset as a tools
developer for the Ranking and Relevance team.</p>

<p>In mid January, after three months of nights and weekends, we launched into
private beta mode, sending invites to our friends. In mid February PJ Hyett
joined in and made us three-strong. We publicly launched the site on April 10th.
TechCrunch was not invited. At this point it was still just three 20-somethings
without a single penny of outside investment.</p>

<p>I was still working full time at Powerset on July 1, 2008 when we learned that
Powerset had just been acquired by Microsoft for around $100 million. This was
interesting timing. With the acquisition, I was going to be faced with a choice
sooner than I had anticipated. I could either sign on as a Microsoft employee or
quit and go GitHub full time. At 29 years old, I was the oldest of the three
GitHubbers, and had accumulated a proportionally larger amount of debt and
monthly expenditure. I was used to my six digit lifestyle. Further confounding
the issue was the imminent return of my wife, Theresa, from her PhD fieldwork in
Costa Rica. I would soon be transitioning from make-believe bachelor back to
married man.</p>

<p>To muddy the waters of decision even more, the Microsoft employment offer was
juicy. Salary + $300k over three years juicy. That’s enough money to make
anybody think twice about anything. So I was faced with this: a safe job with
lots of guaranteed money as a Microsoft man –or– a risky job with
unknown amounts of money as an entrepreneur. I knew things with the other GitHub
guys would become extremely strained if I stayed on at Powerset much longer.
Having saved up some money and become freelancers some time ago, they had both
started dedicating full time effort to GitHub. It was do or die time. Either
pick GitHub and go for it, or make the safe choice and quit GitHub to make
wheelbarrows full of cash at Microsoft.</p>

<p>If you want a recipe for restless sleep, I can give you one. Add one part “what
will my wife think” with 3,000 parts Benjamin Franklin; stir in a “beer anytime
you damn well please” and top with a chance at financial independence.</p>

<p>I’ve become pretty good at giving my employers the bad news that I’m leaving the
company to go do something cooler. I broke the news to my boss at Powerset on
the day the employment offer was due. I told him I was quitting to go work full
time on GitHub. Like any great boss, he was bummed, but understanding. He didn’t
try to tempt me with a bigger bonus or anything. I think deep down he knew I was
going to leave. I may have even received a larger incentive to stay than others,
on account of my being a flight risk. Those Microsoft managers are crafty, I
tell you. They’ve got retention bonuses down to a science. Well, except when you
throw an entrepreneur, the singularity of the business world, into the mix.
Everything goes wacky when you’ve got one of those around.</p>

<p>In the end, just as Indiana Jones could never turn down the opportunity to
search for the Holy Grail, I could no less turn down the chance to work for
myself on something I truly love, no matter how safe the alternative might be.
When I’m old and dying, I plan to look back on my life and say “wow, that was an
adventure,” not “wow, I sure felt safe.”</p>


guid: http://tom.preston-werner.com/2008/10/27/looking-back-on-selling-gravatar-to-automattic
title: Looking back on Selling Gravatar to Automattic
url: http://tom.preston-werner.com/2008/10/27/looking-back-on-selling-gravatar-to-automattic.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="looking-back-on-selling-gravatar-to-automattic">Looking back on Selling Gravatar to Automattic</h1>

<p class="meta">23 Oct 2008 - San Francisco</p>

<p>For an entrepreneur, the line between horrible mistake and runaway success can
be so thin that even Kate Moss would be envious. I lived with
<a href="http://gravatar.com/">Gravatar</a> for nearly four years before that line even
became thick enough to measure.</p>

<p>As it’s become one of my favorite parables, I’ll save the details of how I came
up with the idea for Gravatar for a future post. What’s important to know is
that the idea was spawned not from a business perspective, but from a desperate
desire to create something new in the world of blogging.</p>

<p>Spin the clock back four years and you’ll find me sitting at my Windows desktop
machine in my underwear with a box of Life cereal to my left and a day old Coke
to my right. Since I’d been laid off from my job as a Java developer some months
earlier, I’d decided to take the entrepreneurial plunge doing what I knew best:
web design. Working in a cone of isolation, I’d become accustomed to waking up
late, swinging my legs over the right side of the bed, and in one fluid movement
sliding over to the ratty chair I stole from my old college dorm room. I’d spend
most of the day working on client projects in ColdFusion or PHP. It was hard
work and could become a bit tiresome. I needed an outlet, something that didn’t
have a suit on the other end of a telephone telling me how blue was the wrong
color and things would be so much better if only the photo had a slightly bigger
border. Gravatar would become that outlet for me.</p>

<p>I was really big into web standards at the time, having recently read Zeldman’s
seminal work, and became a true believer. Eric Meyers, Dan Cederholm, and Jon
Hicks became like gods to me. I worked very hard at making relevant and witty
comments around the right kinds of blogs. Being a part of that movement became a
significant goal for me. My Movable Type weblog rarely went more than two days
without a post on design or standards.</p>

<p>Two weeks after I had the idea for Gravatar the first version was written and
deployed. Every request hit the database and dynamically generated a properly
sized gravatar via PHP’s gd2 api. Premature optimization and all that, right?
The first thing I did after getting the system to a workable state was email all
the bloggers I looked up to (and that had no idea who I was). Blog comments at
the time were a pretty dreary affair and I guess Cederholm was intrigued enough
by my idea that he linked to it in a sidebar micro-post on simplebits.com.</p>

<p>That single mention kicked off a slow but steady trickle of interest in the
system. A few blogs here and there installed the plugin and the world started
seeing avatars that mysteriously followed them around. At the same time, just as
people must have thought Cheez Whiz was a stupid idea when it first came out,
some bloggers started railing against Gravatar, calling it frivolous,
inefficient, and “an abomination.” This was my first nibble at the smorgasbord
of what was to become the “horrible mistake” aspect of Gravatar.</p>

<p>Due to the inherently self-advertising nature of gravatars (the “what the hell
is that and how do I get one?” brand of advertising), Gravatar adoption
increased at a rapid rate. Having crafted the idea for Gravatar without any
semblance of business model or growth projection or build-out strategy, things
took a rather dramatic dive away from “runaway success” as my server (yes,
singular) buckled under the pressure of tens of requests per second! As it turns
out, regenerating a gravatar on every request is not very CPU efficient.
Gravatars worldwide suddenly turned into little red Xs. Then, in what has become
known as the Twitter Effect, a barrage of emails hit me complaining about how
the free service on which they had come to depend was down, and how this would
adversely impact my well-being.</p>

<p>I fixed the code. Gravatar came back online with caching. All the while I’d had
the bright idea that gravatars would be rated for content, MPAA-style. Because
users clearly were not fit to rate their own images, I was manually rating 400
or more avatars each day. If I missed a day, I’d have damn near a thousand
waiting for me the next day. In addition to the angry mob, I was very fortunate
to have an amazingly supportive group of users that volunteered to help me rate
images. I owe them my sanity, and it freed up enough time for me to work on the
next iteration of the site.</p>

<p>G2, as I called it, would be written in Rails and use lighttpd plus a convoluted
directory structure of symlinks to enable me to pre-render every gravatar (1x1
up to 80x80) and serve only static images. I did this to avoid having to rent or
buy the kind of hardware necessary to hook up a properly scaled system. Up until
the end I ran Gravatar on a maximum of two rented commodity servers that set me
back a mere $300/month, a pittance for the kind of traffic I was serving. I say
it was a pittance, but that’s not really true. Donations didn’t even come close
to covering that cost.</p>

<p>At some point early in the development of G2, Toni Schneider, the CEO of
Automattic (the company behind WordPress.com and Akismet) contacted me after
hearing my <a href="http://blog.gravatar.com/2007/10/18/automattic-gravatar/">interview about the future of
Gravatar</a> on the
WordPress podcast. This was exciting news! How perfect a fit would it be for
Gravatar to be bought by Automattic? I was already planning a trip up to San
Francisco to meet with the Powerset guys, so the timing worked out perfectly for
me to meet in person with Toni. I ended up having lunch with Toni and Matt
Mullenweg at 21st Ammendment on 2nd Street. It was a bit intimidating to come to
the mecca of tech startups to meet with such huge players in the blogging
community. Turned out that both Matt and Toni are great guys and so we had
drinks for about two hours, talking about my ideas for Gravatar and how we might
be able to work together. Everything seemed great—I was jazzed and they
seemed excited—but a few weeks later Toni let me know that the timing was
wrong and they couldn’t make a play at that time. He suggested I proceed with G2
and they’d proceed with their own avatar system. I was pretty bummed about the
outcome, but I took their advice and kept going.</p>

<p>A few weeks before G2 was finished, the site imploded in a big way. One machine.
Hundreds of requests per second. That poor CPU must have thought it was the End
Times. Instead of wasting time on getting the existing system back up, I put on
my headphones, turned it up to eleven, and got back to work on G2.</p>

<p>I’m not sure if you’ve ever had to work on a project while your users are
publicly skewering you on your blog for allowing the service to go down, but
it’s as close to depression as I’ve ever come. If I’d had less pride, I would
have popped everyone a huge middle finger and let the service die, but instead I
waded through the hundreds of comments and deleted the ones with threats,
hatred, or my favorite, the words “fuck you” repeated 600 times. It wasn’t fair,
I told myself, that I should be sitting here with high blood pressure trying to
raise Gravatar from the dead while the unappreciative masses do what they do
best on the internet. The only thing that kept me going was never being able to
tell which side of the mistake/success boundary I was sitting on. It was hard to
think of the situation as anything but a huge failure, but the shitstorm the
downtime was causing indicated that people found the service valuable. I want to
say that Twitter went through the same thing, but they suffered their downtime
with millions of dollars in the bank. The only thing I had was a full time job
unrelated to Gravatar and a credit card that reminded me every month of my bad
judgment.</p>

<p>Finally G2 was done, deployed, and fully operational. I have no idea how many
users I lost due to the several weeks of downtime, but I don’t think it was very
many. There seems to be a corollary to the Twitter Effect that I’d call the
Forgiveness Effect. It dictates that if a user enjoys a free service and that
service is currently up, all past atrocities will be easily and quickly
forgiven. With the site running again, things looked to be shifting back towards
a success.</p>

<p>Things were not all rainbows and unicorns though. My Rube Goldbergian
architecture had a few quirks that needed to be dealt with. The site still had
some elusive bugs from the overly-rapid development cycle. And just like new
lanes on freeways always fill up immediately, the two new servers I was running
started causing expensive bandwidth overages. I had taken a job at Powerset at
this point and the combined pressures of these two commitments started to weigh
me down. Once again I started feeling like all the effort I put into Gravatar
was for nothing. Like I would never benefit from any of it.</p>

<p>In a last ditch effort to save Gravatar from final doom, I emailed Toni and
pitched him again on Gravatar. I figured it was a long shot, but what the hell,
couldn’t hurt. Things must have changed in the prior 6 months because Toni was
very receptive to the idea. We met again, at 21st Ammendment, and hashed out a
tentative deal over drinks. I’d never sold anything like this before, so my
technique was probably very amateurish. I’m almost certain I could have gotten a
better deal out of it, but I had the smell of desperation about me and I really
did want to see Gravatar end up in Automattic’s hands.</p>

<p>Four days later, Automattic made their official offer. On September 21st, 2007
we inked the deal and Gravatar became both the first company that I ever sold
and the first company that Automattic ever acquired.</p>

<p>I am quite satisfied with the sale to Automattic. Some will say that I should
have pursued VC funding. Indeed, I was contacted by several firms but never
travelled very far down that road. I always felt like Gravatar was a feature,
and I wasn’t comfortable building a company on such a tiny foundation.
Reinforcing this decision, no viable business model ever coalesced during the
time I was building the site. It was also made clear by Toni that Automattic
would maintain Gravatar as a separate brand and continue its evolution (instead
of just absorbing it into WordPress). This appealed to my ego. Most companies
kill or maim everything they acquire, but here was a chance for Gravatar to
carry forward with all of Automattic’s resources behind it (instead of two
measly servers). Part of me just wanted to see what Gravatar could become with
time, money, and man-power moving it forward.</p>

<p>Things always seem so clear in retrospect. But it was pride and persistence that
kept me in the game long enough to have anything to look back on at all. While
the line between horrible mistake and runaway success may be difficult to see,
you can still find it if you look hard enough.</p>


guid: http://tom.preston-werner.com/2008/11/03/how-to-meet-your-next-cofounder
title: How to Meet Your Next Cofounder
url: http://tom.preston-werner.com/2008/11/03/how-to-meet-your-next-cofounder.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="how-to-meet-your-next-cofounder">How to Meet Your Next Cofounder</h1>

<p class="meta">3 Nov 2008 - San Francisco</p>

<p>Over the last few months I’ve seen a number of people looking for cofounders on
Hacker News or via their own personal blogs. I think this is, at best, a highly
inefficient way to find a cofounder and, at worst, a way to fool yourself into
finding the <em>wrong</em> cofounder. In any case, it’s a naive approach to finding the
person that will need to stand by your side in the coming storm that we call
“running a startup.”</p>

<p>Don’t get me wrong, the internet is an amazing tool for meeting people. The
wider the net you cast, the more likely you are to find the perfect match. But
the internet has its limitations. I’ve had internet friends that were engaging,
witty, and brilliant online, but in person felt awkward and boring. Conversely,
I know people that are volatile and inflammatory online, but present an attitude
of friendliness and caring in person. This phenomenon makes it difficult to
gauge an individual’s personality from online interaction alone.</p>

<p>A far better use of the internet is to find groups of people that share your
interests. Track down the local users group for your language or technology of
choice. The simple fact that members of these groups take time out of their day
to show up means that they’re more motivated and driven than the average person.
Even if it’s a bit of a commute to get to the meetings, start showing up
regularly. Prepare a few presentations on topics that you’re passionate about.
Bonus points if you present on ideas related to your potential startup. Don’t
worry about revealing your game-changing secrets; stealth mode is bullshit. Talk
to everyone. Steer the conversation toward your interests and if someone there
is excited about the same things, it will be clear.</p>

<p>It may take weeks or months, but in a good group you’ll find a handful of people
that you really like. If at all possible, go out drinking with these people
after the meetups. This is one of the easiest ways to go from “acquaintance” to
“friend” and gives you free license to bring up your craziest of ideas without
sounding like too much of a nutjob.</p>

<p>Of the people that you like, several may make excellent candidates for
cofounders. Do a little research on these individuals. What does their code look
like? Have they done much open source? Do they demonstrate an entrepreneurial
spirit? Can they stick with a single project for a long time? Have they been
loyal to their friends and companies in the past? A good cofounder should be
someone with whom you feel privileged to work. And they should feel privileged
to work with you. The two of you should be on very solid ground before you begin
your startup adventure, because once you do, the impact of every argument is
going to feel like it’s been multiplied by a thousand.</p>

<p>This all sounds like a lot of hard work. Maybe you’re wondering if it would be
better to just go solo. I did that with Gravatar, and, in retrospect, it’s
painfully obvious that I made a lot of stupid mistakes. When it’s just you and
your thoughts it becomes too easy to pick the first thing that pops into your
head. We’re programmed to think all of our ideas are good, but reality tells a
different story. Truly good decisions are forged from the furnace of argument,
not plucked like daisies from the pasture of a peaceful mind. A good cofounder
tells you when your ideas are half-baked and ensures that your good ideas
actually get implemented.</p>

<p>The second biggest danger with going solo is the loss of motivation. Solipsism
might make you feel important at first, but the constant lack of feedback and
the absence of support during tough times can easily lead to a premature end to
your adventure. Cofounders are like workout buddies. Just when you think there’s
no possible way you can do another rep, there they are, rooting you on toward an
achievement that wouldn’t be possible without them.</p>

<p>Your choice of cofounder will affect everything you do in your startup. They’ll
share every defeat with you and celebrate every success. They’ll help you
understand your own ideas better by offering a different perspective. They’ll be
the single most important decision you make during the tenure of your startup,
so choose wisely and with extreme care.</p>


guid: http://tom.preston-werner.com/2008/11/17/blogging-like-a-hacker
title: Blogging Like a Hacker
url: http://tom.preston-werner.com/2008/11/17/blogging-like-a-hacker.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="blogging-like-a-hacker">Blogging Like a Hacker</h1>

<p class="meta">17 Nov 2008 - San Francisco</p>

<p>Back in 2000, when I thought I was going to be a professional writer, I spent
hours a day on LiveJournal doing writing practice with other aspiring poets and
authors. Since then I’ve blogged at three different domains about web standards,
print design, photography, Flash, illustration, information architecture,
ColdFusion, package management, PHP, CSS, advertising, Ruby, Rails, and Erlang.</p>

<p>I love writing. I get a kick out of sharing my thoughts with others. The act of
transforming ideas into words is an amazingly efficient way to solidify and
refine your thoughts about a given topic. But as much as I enjoy blogging, I
seem to be stuck in a cycle of quitting and starting over. Before starting the
current iteration, I resolved to do some introspection to determine the factors
that were leading to this destructive pattern.</p>

<p>I already knew a lot about what I <em>didn’t</em> want. I was tired of complicated
blogging engines like WordPress and Mephisto. I wanted to write great posts, not
style a zillion template pages, moderate comments all day long, and constantly
lag behind the latest software release. Something like Posterous looked
attractive, but I wanted to style my blog, and it needed to be hosted at the
domain of my choosing. For the same reason, other hosted sites (wordpress.com,
blogger.com) were disqualified. There are a few people directly using GitHub as
a blog (which is very cool), but that’s a bit too much of an impedance mismatch
for my tastes.</p>

<p>On Sunday, October 19th, I sat down in my San Francisco apartment with a glass
of apple cider and a clear mind. After a period of reflection, I had an idea.
While I’m not specifically trained as an author of prose, I <em>am</em> trained as an
author of code. What would happen if I approached blogging from a software
development perspective? What would that look like?</p>

<p>First, all my writing would be stored in a Git repository. This would ensure
that I could try out different ideas and explore a variety of posts all from the
comfort of my preferred editor and the command line. I’d be able to publish a
post via a simple deploy script or post-commit hook. Complexity would be kept to
an absolute minimum, so a static site would be preferable to a dynamic site that
required ongoing maintenance. My blog would need to be easily customizable;
coming from a graphic design background means I’ll always be tweaking the site’s
appearance and layout.</p>

<p>Over the last month I’ve brought these concepts to fruition and I’m pleased to
announce <a href="http://github.com/mojombo/jekyll">Jekyll</a>. Jekyll is a simple, blog
aware, static site generator. It takes a template directory (representing the
raw form of a website), runs it through Textile and Liquid converters, and spits
out a complete, static website suitable for serving with Apache or your favorite
web server. If you’re reading this on the website
(http://tom.preston-werner.com), you’re seeing a Jekyll generated blog!</p>

<p>To understand how this all works, open up my <a href="http://github.com/mojombo/tpw">TPW</a>
repo in a new browser window. I’ll be referencing the code there.</p>

<p>Take a look at
<a href="http://github.com/mojombo/tpw/tree/master/index.html">index.html</a>. This file
represents the homepage of the site. At the top of the file is a chunk of YAML
that contains metadata about the file. This data tells Jekyll what layout to
give the file, what the page’s title should be, etc. In this case, I specify
that the “default” template should be used. You can find the layout files in the
<a href="http://github.com/mojombo/tpw/tree/master/_layouts">_layouts</a> directory. If you
open
<a href="http://github.com/mojombo/tpw/tree/master/_layouts/default.html">default.html</a>
you can see that the homepage is constructed by wrapping index.html with this
layout.</p>

<p>You’ll also notice Liquid templating code in these files.
<a href="http://www.liquidmarkup.org/">Liquid</a> is a simple, extensible templating
language that makes it easy to embed data in your templates. For my homepage I
wanted to have a list of all my blog posts. Jekyll hands me a Hash containing
various data about my site. A reverse chronological list of all my blog posts
can be found in <code>site.posts</code>. Each post, in turn, contains various
fields such as <code>title</code> and <code>date</code>.</p>

<p>Jekyll gets the list of blog posts by parsing the files in the
<a href="http://github.com/mojombo/tpw/tree/master/_posts">_posts</a> directory. Each post’s
filename contains the publishing date and slug (what shows up in the URL) that
the final HTML file should have. Open up the file corresponding to this blog
post:
<a href="http://github.com/mojombo/tpw/tree/master/_posts/2008-11-17-blogging-like-a-hacker.textile">2008-11-17-blogging-like-a-hacker.textile</a>.
GitHub renders textile files by default, so to better understand the file, click
on the
<a href="http://github.com/mojombo/tpw/tree/master/_posts/2008-11-17-blogging-like-a-hacker.textile?raw=true">raw</a>
view to see the original file. Here I’ve specified the <code>post</code> layout.
If you look at that file you’ll see an example of a nested layout. Layouts can
contain other layouts allowing you a great deal of flexibility in how pages are
assembled. In my case I use a nested layout in order to show related posts for
each blog entry. The YAML also specifies the post’s title which is then embedded
in the post’s body via Liquid.</p>

<p>Posts are handled in a special way by Jekyll. The date you specify in the
filename is used to construct the URL in the generated site. This post, for
instance, ends up at
<code>http://tom.preston-werner.com/2008/11/17/blogging-like-a-hacker.html</code>.</p>

<p>Files that do not reside in directories prefixed with an underscore are mirrored
into a corresponding directory structure in the generated site. If a file does
not have a YAML preface, it is not run through the Liquid interpreter. Binary
files are copied over unmodified.</p>

<p>In order to convert your raw site into the finished version, you simply run:</p>

<pre class="terminal"><code>$ jekyll /path/to/raw/site
/path/to/place/generated/site</code></pre>

<p>Jekyll is still a very young project. I’ve only developed the exact
functionality that I’ve needed. As time goes on I’d like to see the project
mature and support additional features. If you end up using Jekyll for your own
blog, drop me a line and let me know what you’d like to see in future versions.
Better yet, fork the project over at GitHub and hack in the features yourself!</p>

<p>I’ve been living with Jekyll for just over a month now. I love it. Driving the
development of Jekyll based on the needs of my blog has been very rewarding. I
can edit my posts in TextMate, giving me automatic and competent spell checking.
I have immediate and first class access to the CSS and page templates.
Everything is backed up on GitHub. I feel a lightness now when I’m writing a
post. The system is simple enough that I can keep the entire conversion process
in my head. The distance from my brain to my blog has shrunk, and, in the end, I
think that will make me a better author.</p>


guid: http://tom.preston-werner.com/2009/05/19/the-git-parable
title: The Git Parable
url: http://tom.preston-werner.com/2009/05/19/the-git-parable.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="the-git-parable">The Git Parable</h1>

<p class="meta">19 May 2009 - San Francisco</p>

<p>Git is a simple, but extremely powerful system. Most people try to teach Git by
demonstrating a few dozen commands and then yelling “tadaaaaa.” I believe this
method is flawed. Such a treatment may leave you with the ability to use Git to
perform simple tasks, but the Git commands will still feel like magical
incantations. Doing anything out of the ordinary will be terrifying. Until you
understand the concepts upon which Git is built, you’ll feel like a stranger in
a foreign land.</p>

<p>The following parable will take you on a journey through the creation of a
Git-like system from the ground up. Understanding the concepts presented here
will be the most valuable thing you can do to prepare yourself to harness the
full power of Git. The concepts themselves are quite simple, but allow for an
amazing wealth of functionality to spring into existence. Read this parable all
the way through and you should have very little trouble mastering the various
Git commands and wielding the awesome power that Git makes available to you.</p>

<h2 id="the-parable">The Parable</h2>

<p>Imagine that you have a computer that has nothing on it but a text editor and a
few file system commands. Now imagine that you have decided to write a large
software program on this system. Because you’re a responsible software
developer, you decide that you need to invent some sort of method for keeping
track of versions of your software so that you can retrieve code that you
previously changed or deleted. What follows is a story about how you might
design one such version control system (VCS) and the reasoning behind those
design choices.</p>

<h2 id="snapshots">Snapshots</h2>

<p>Alfred is a friend of yours that works down at the mall as a photographer in one
of those “Special Moments” photo boutiques. All day long he takes photos of
little kids posing awkwardly in front of jungle or ocean backdrops. During one
of your frequent lunches at the pretzel stand, Alfred tells you a story about a
woman named Hazel who brings her daughter in for a portrait every year on the
same day. “She brings the photos from all the past years with her,” Alfred tells
you. “She likes to remember what her daughter was like at each different stage,
as if the snapshots really let her move back and forth in time to those saved
memories.”</p>

<p>Like some sort of formulaic plot device, Alfred’s innocent statement acts as a
catalyst for you to see the ideal solution to your version control dilemma.
Snapshots, like save points in a video game, are really what you care about when
you need to interact with a VCS. What if you could take snapshots of your
codebase at any time and resurrect that code on demand? Alfred reads the dawning
realization spreading across your face and knows you’re about to leave him
without another word to go back and implement whatever genius idea he just
caused you to have. You do not disappoint him.</p>

<p>You start your project in a directory named <code>working</code>. As you code,
you try to write one feature at a time. When you complete a self-contained
portion of a feature, you make sure that all your files are saved and then make
a copy of the entire working directory, giving it the name
<code>snapshot-0</code>. After you perform this copy operation, you make sure to
never again change the code files in the new directory. After the next chunk of
work, you perform another copy, only this time the new directory gets the name
<code>snapshot-1</code>, and so on.</p>

<p>To make it easy to remember what changes you made in each snapshot, you add a
special file named <code>message</code> to each snapshot directory that contains
a summary of the work that you did and the date of completion. By printing the
contents of each message, it becomes easy to find a specific change that you
made in the past, in case you need to resurrect some old code.</p>

<h2 id="branches">Branches</h2>

<p>After a bit of time on the project, a candidate for release begins to emerge.
Late nights at the keyboard finally yield <code>snapshot-99</code>, the nascent
form of what will become Release Version 1.0. It comes to pass that this
snapshot is packaged and distributed to the eagerly awaiting masses. Stoked by
excellent response to your software, you push forward, determined to make the
next version an even bigger success.</p>

<p>Your VCS has so far been a faithful companion. Old versions of your code are
there when you need them and can be accessed with ease. But not long after the
release, bug reports start to come in. Nobody’s perfect, you reassure yourself,
and <code>snapshot-99</code> is readily retrievable, glad to be brought back to
life for the purposes of applying bug fixes.</p>

<p>Since the release, you’ve created 10 new snapshots. This new work must not be
included in the 1.0.1 bug fix version you now need to create. To solve this, you
copy <code>snapshot-99</code> to <code>working</code> so that your working
directory is at exactly the point where Version 1.0 was released. A few swift
lines of code and the bug is fixed in the working directory.</p>

<p>It is here that a problem becomes apparent. The VCS deals very well with linear
development, but for the first time ever, you need to create a new snapshot that
is not a direct descendent of the preceding snapshot. If you create a
<code>snapshot-110</code> (remember that you created 10 snapshots since the
release), then you’ll be interrupting the linear flow and will have no way of
determining the ancestry of any given snapshot. Clearly, you need something more
powerful than a linear system.</p>

<p>Studies show that even short exposures to nature can help recharge the mind’s
creative potential. You’ve been sitting behind the artificially polarized light
of your monitor for days on end. A walk through the woods in the brisk Autumn
air will do you some good and with any luck, will help you arrive at an ideal
solution to your problem.</p>

<p>The great oaks that line the trail have always appealed to you. They seem to
stand stark and proud against the perfectly blue sky. Half the ruddy leaves have
departed from their branches, leaving an intricate pattern of branches in their
wake. Fixating on one of the thousands of branch tips you idly try to follow it
back to the solitary trunk. This organically produced structure allows for such
great complexity, but the rules for finding your way back to the trunk are so
simple, and perfect for keeping track of multiple lines of development! It turns
out that what they say about nature and creativity are true.</p>

<p>By looking at your code history as a tree, solving the problem of ancestry
becomes trivial. All you need to do is include the name of the parent snapshot
in the <code>message</code> file you write for each snapshot. Adding just a
single upstream pointer will enable you to easily and accurately trace the
history of any given snapshot all the way back to the root.</p>

<h2 id="branch-names">Branch Names</h2>

<p>Your code history is now a tree. Instead of having a single latest snapshot, you
have two: one for each branch. With a linear system, your sequential numbering
system let you easily identify the latest snapshot. Now, that ability is lost.</p>

<p>Creating new development branches has become so simple that you’ll want to take
advantage of it all the time. You’ll be creating branches for fixes to old
releases, for experiments that may not pan out; indeed it becomes possible to
create a new branch for every feature you begin!</p>

<p>But like everything good in life, there is a price to be paid. Each time you
create a new snapshot, you must remember that the new snapshot becomes the
latest on its branch. Without this information, switching to a new branch would
become a laborious process indeed.</p>

<p>Every time you create a new branch you probably give it a name in your head.
“This will be the Version 1.0 Maintenance Branch,” you might say. Perhaps you
refer to the former linear branch as the “master” branch.</p>

<p>Think about this a little further, though. From the perspective of a tree, what
does it mean to name a branch? Naming every snapshot that appears in the history
of a branch would do the trick, but requires the storage of a potentially large
amount of data. Additionally, it still wouldn’t help you efficiently locate the
latest snapshot on a branch.</p>

<p>The least amount of information necessary to identify a branch is the location
of the latest snapshot on that branch. If you need to know the list of snapshots
that are part of the branch you can easily trace the parentage.</p>

<p>Storing the branch names is trivial. In a file named <code>branches</code>,
stored outside of any specific snapshot, you simply list the name/snapshot pairs
that represent the tips of branches. To switch to a named branch you need only
look up the snapshot for the corresponding name from this file.</p>

<p>Because you’re only storing the latest snapshot on each branch, creating a new
snapshot now contains an additional step. If the new snapshot is being created
as part of a branch, the <code>branches</code> file must be updated so that the
name of the branch becomes associated with the new snapshot. A small price to
pay for the benefit.</p>

<h2 id="tags">Tags</h2>

<p>After using branches for a while you notice that they can serve two purposes.
First, they can act as movable pointers to snapshots so that you can keep track
of the branch tips. Second, they can be pointed at a single snapshot and never
move.</p>

<p>The first use case allows you to keep track of ongoing development, things like
“Release Maintenance”. The second case is useful for labeling points of
interest, like “Version 1.0” and “Version 1.0.1”.</p>

<p>Mixing both of these uses into a single file feels messy. Both types are
pointers to snapshots, but one moves and one doesn’t. For the sake of clarity
and elegance, you decide to create another file called <code>tags</code> to
contain pointers of the second type.</p>

<p>Keeping these two inherently different pointers in separate files will help you
from accidentally treating a branch as a tag or vice versa.</p>

<h2 id="distributed">Distributed</h2>

<p>Working on your own gets pretty lonely. Wouldn’t it be nice if you could invite
a friend to work on your project with you? Well, you’re in luck. Your friend Zoe
has a computer setup just like yours and wants to help with the project. Because
you’ve created such a great version control system, you tell her all about it
and send her a copy of all your snapshots, branches, and tags so she can enjoy
the same benefits of the code history.</p>

<p>It’s great to have Zoe on the team but she has a habit of taking long trips to
far away places without internet access. As soon as she has the source code, she
catches a flight to Patagonia and you don’t hear from her for a week. In the
meantime you both code up a storm. When she finally gets back, you discover a
critical flaw in your VCS. Because you’ve both been using the same numbering
system, you each have directories named ‘snapshot-114’, ‘snapshot-115’, and so
on, but with different contents!</p>

<p>To make matters worse, you don’t even know who authored the changes in those new
snapshots. Together, you devise a plan for dealing with these problems. First,
snapshot messages will henceforth contain author name and email. Second,
snapshots will no longer be named with simple numbers. Instead, you’ll use the
contents of the message file to produce a hash. This hash will be guaranteed to
be unique to the snapshot since no two messages will ever have the same date,
message, parent, and author. To make sure everything goes smoothly, you both
agree to use the SHA1 hash algorithm that takes the contents of a file and
produces a 40 character hexadecimal string. You both update your histories with
the new technique and instead of clashing ‘snapshot-114’ directories, you now
have distinct directories named ‘8ba3441b6b89cad23387ee875f2ae55069291f4b’ and
‘db9ecb5b5a6294a8733503ab57577db96ff2249e’.</p>

<p>With the updated naming scheme, it becomes trivial for you to fetch all the new
snapshots from Zoe’s computer and place them next to your existing snapshots.
Because every snapshot specifies its parent, and identical messages (and
therefore identical snapshots) have identical names no matter where they are
created, the history of the codebase can still be drawn as a tree. Only now, the
tree is comprised of snapshots authored by both Zoe and you.</p>

<p>This point is important enough to warrant repeating. A snapshot is identified by
a SHA1 that uniquely identifies it (and its parent). These snapshots can be
created and moved around between computers without losing their identity or
where they belong in the history tree of a project. What’s more, snapshots can
be shared or kept private as you see fit. If you have some experimental
snapshots that you want to keep to yourself, you can do so quite easily. Just
don’t make them available to Zoe!</p>

<h2 id="offline">Offline</h2>

<p>Zoe’s travel habits cause her to spend countless hours on airplanes and boats.
Most of the places she visits have no readily available internet access. At the
end of the day, she spends more time offline than online.</p>

<p>It’s no surprise, then, that Zoe raves about your VCS. All of the day to day
operations that she needs to do can be done locally. The only time she needs a
network connection is when she’s ready to share her snapshots with you.</p>

<h2 id="merges">Merges</h2>

<p>Before Zoe left on her trip, you had asked her to start working off of the
branch named ‘math’ and to implement a function that generated prime numbers.
Meanwhile, you were also developing off of the ‘math’ branch, only you were
writing a function to generate magic numbers. Now that Zoe has returned, you are
faced with the task of merging these two separate branches of development into a
single snapshot. Since you both worked on separate tasks, the merge is simple.
While constructing the snapshot message for the merge, you realize that this
snapshot is special. Instead of just a single parent, this merge snapshot has
two parents! The first parent is your latest on the ‘math’ branch and the second
parent is Zoe’s latest on her ‘math’ branch. The merge snapshot doesn’t contain
any changes beyond those necessary to merge the two disparate parents into a
single codebase.</p>

<p>Once you complete the merge, Zoe fetches all the snapshots that you have that
she does not, which include your development on the ‘math’ branch and your merge
snapshot. Once she does this, both of your histories match exactly!</p>

<h2 id="rewriting-history">Rewriting History</h2>

<p>Like many software developers you have a compulsion to keep your code clean and
very well organized. This carries over into a desire to keep your code history
well groomed. Last night you came home after having a few too many pints of
Guinness at the local brewpub and started coding, producing a handful of
snapshots along the way. This morning, a review of the code you wrote last night
makes you cringe a little bit. The code is good overall, but you made a lot of
mistakes early on that you corrected in later snapshots.</p>

<p>Let’s say the branch on which you did your drunken development is called ‘drunk’
and you made three snapshots after you got home from the bar. If the name
‘drunk’ points at the latest snapshot on that branch, then you can use a useful
notation to refer to the parent of that snapshot. The notation ‘drunk^’ means
the parent of the snapshot pointed to by the branch name ‘drunk’. Similarly
‘drunk^^’ means the grandparent of the ‘drunk’ snapshot. So the three snapshots
in chronological order are ‘drunk^^’, ‘drunk^’, and ‘drunk’.</p>

<p>You’d really like those three lousy snapshots to be two clean snapshots. One
that changes an existing function, and one that adds a new file. To accomplish
this revision of history you copy ‘drunk’ to ‘working’ and delete the file that
is new in the series. Now ‘working’ represents the correct modifications to the
existing function. You create a new snapshot from ‘working’ and write the
message to be appropriate to the changes. For the parent you specify the SHA1 of
the ‘drunk^^^’ snapshot, essentially creating a new branch off of the same
snapshot as last night. Now you can copy ‘drunk’ to ‘working’ and roll a
snapshot with the new file addition. As the parent you specify that snapshot you
created just before this one.</p>

<p>As the last step, you change the branch name ‘drunk’ to point to the last
snapshot you just made.</p>

<p>The history of the ‘drunk’ branch now represents a nicer version of what you did
last night. The other snapshots that you’ve replaced are no longer needed so you
can delete them or just leave them around for posterity. No branch names are
currently pointing at them so it will be hard to find them later on, but if you
don’t delete them, they’ll stick around.</p>

<h2 id="staging-area">Staging Area</h2>

<p>As much as you try to keep your new modifications related to a single feature or
logical chunk, you sometimes get sidetracked and start hacking on something
totally unrelated. Only half-way into this do you realize that your working
directory now contains what should really be separated as two discrete
snapshots.</p>

<p>To help you with this annoying situation, the concept of a staging directory is
useful. This area acts as an intermediate step between your working directory
and a final snapshot. Each time you finish a snapshot, you also copy that to a
<code>staging</code> directory. Now, every time you finish an edit to a new
file, create a new file, or remove a file, you can decide whether that change
should be part of your next snapshot. If it belongs, you mimic the change inside
<code>staging</code>. If it doesn’t, you can leave it in <code>working</code>
and make it part of a later snapshot. From now on, snapshots are created
directly from the staging directory.</p>

<p>This separation of coding and preparing the stage makes it easy to specify what
is and is not included in the next snapshot. You no longer have to worry too
much about making an accidental, unrelated change in your working directory.</p>

<p>You have to be a bit careful, though. Consider a file named <code>README</code>.
You make an edit to this file and then mimic that in <code>staging</code>. You
go on about your business, editing other files. After a bit, you make another
change to <code>README</code>. Now you have made two changes to that file, but
only one is in the staging area! Were you to create a snapshot now, your second
change would be absent.</p>

<p>The lesson is this: every new edit must be added to the staging area if it is to
be part of the next snapshot.</p>

<h2 id="diffs">Diffs</h2>

<p>With a working directory, a staging area, and loads of snapshots laying around,
it starts to get confusing as to what the specific code changes are between
these directories. A snapshot message only gives you a summary of what changed,
not exactly what lines were changed between two files.</p>

<p>Using a diffing algorithm, you can implement a small program that shows you the
differences in two codebases. As you develop and copy things from your working
directory to the staging area, you’ll want to easily see what is different
between the two, so that you can determine what else needs to be staged. It’s
also important to see how the staging area is different from the last snapshot,
since these changes are what will become part of the next snapshot you produce.</p>

<p>There are many other diffs you might want to see. The differences between a
specific snapshot and its parent would show you the “changeset” that was
introduced by that snapshot. The diff between two branches would be helpful for
making sure your development doesn’t wander too far away from the mainline.</p>

<h2 id="eliminating-duplication">Eliminating Duplication</h2>

<p>After a few more trips to Namibia, Istanbul, and Galapagos, Zoe starts to
complain that her hard drive is filling up with hundreds of nearly identical
copies of the software. You too have been feeling like all the file duplication
is wasteful. After a bit of thinking, you come up with something very clever.</p>

<p>You remember that the SHA1 hash produces a short string that is unique for a
given file contents. Starting with the very first snapshot in the project
history, you start a conversion process. First, you create a directory named
<code>objects</code> outside of the code history. Next, you find the most deeply
nested directory in the snapshot. Additionally, you open up a temporary file for
writing. For each file in this directory you perform three steps. Step 1:
Calculate the SHA1 of the contents. Step 2: Add an entry into the temp file that
contains the word ‘blob’ (binary large object), the SHA1 from the first step,
and the filename. Step 3: Copy the file to the objects directory and rename it
to the SHA1 from step 1. Once finished with all the files, find the SHA1 of the
temp file contents and use that to name the temp file, also placing it in the
objects directory.</p>

<p>If at any time the objects directory already contains a file with a given name,
then you have already stored that file’s contents and there is no need to do so
again.</p>

<p>Now, move up one directory and start over. Only this time, when you get to the
entry for the directory that you just processed, enter the word ‘tree’, the SHA1
of the temp file from last time, and the directory’s name into the new temp
file. In this fashion you can build up a tree of directory object files that
contain the SHA1s and names of the files and directory objects that they
contain.</p>

<p>Once this has been accomplished for every directory and file in the snapshot,
you have a single root directory object file and its corresponding SHA1. Since
nothing contains the root directory, you must record the root tree’s SHA1
somewhere. An ideal place to store it is in the snapshot message file. This way,
the uniqueness of the SHA1 of the message also depends on the entire contents of
the snapshot, and you can guarantee with absolute certainty that two identical
snapshot message SHA1s contain the same files!</p>

<p>It’s also convenient to create an object from the snapshot message in the same
way that you do for blobs and trees. Since you’re maintaining a list of branch
and tag names that point to message SHA1s you don’t have to worry about losing
track of which snapshots are important to you.</p>

<p>With all of this information stored in the objects directory, you can safely
delete the snapshot directory that you used as the source of this operation. If
you want to reconstitute the snapshot at a later date it’s simply a matter of
following the SHA1 of the root tree stored in the message file and extracting
each tree and blob into their corresponding directory and file.</p>

<p>For a single snapshot, this transformation process doesn’t get you much. You’ve
basically just converted one filesystem into another and created a lot of work
in the process. The real benefits of this system arise from reuse of trees and
blobs across snapshots. Imagine two sequential snapshots in which only a single
file in the root directory has changed. If the snapshots both contain 10
directories and 100 files, the transformation process will create 10 trees and
100 blobs from the first snapshot but only one new blob and one new tree from
the second snapshot!</p>

<p>By converting every snapshot directory in the old system to object files in the
new system, you can drastically reduce the number of files that are stored on
disk. Now, instead of storing perhaps 50 identical copies of a rarely changed
file, you only need to keep one.</p>

<h2 id="compressing-blobs">Compressing Blobs</h2>

<p>Eliminating blob and tree duplication significantly reduces the total storage
size of your project history, but that’s not the only thing you can do to save
space. Source code is just text. Text can be very efficiently compressed using
something like the LZW or DEFLATE compression algorithms. If you compress every
blob before computing its SHA1 and saving it to disk you can reduce the total
storage size of the project history by another very admirable quantity.</p>

<h2 id="the-true-git">The True Git</h2>

<p>The VCS you have constructed is now a reasonable facsimile of Git. The main
difference is that Git gives you very nice command lines tools to handle such
things as creating new snapshots and switching to old ones (Git uses the term
“commit” instead of “snapshot”), tracing history, keeping branch tips
up-to-date, fetching changes from other people, merging and diffing branches,
and hundreds of other common (and not-so-common tasks).</p>

<p>As you continue to learn Git, keep this parable in mind. Git is really very
simple underneath, and it is this simplicity that makes it so flexible and
powerful. One last thing before you run off to learn all the Git commands:
remember that it is almost impossible to lose work that has been committed. Even
when you delete a branch, all that’s really happened is that the pointer to that
commit has been removed. All of the snapshots are still in the objects
directory, you just need to dig up the commit SHA. In these cases, look up
<code>git reflog</code>. It contains a history of what each branch pointed to
and in times of crisis, it will save the day.</p>

<p>Here are some resources that you should follow as your next step. Now, go, and
become a Git master!</p>

<ul>
  <li><a href="http://learn.github.com/">Learn Git</a></li>
  <li><a href="http://book.git-scm.com/">Git Community Book</a></li>
  <li><a href="http://www-cs-students.stanford.edu/~blynn/gitmagic/">Git Magic</a></li>
</ul>

<p>–</p>

<p><a href="http://news.ycombinator.com/item?id=615308">Discuss this post on Hacker News</a></p>

<p><a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/us/"><img alt="Creative Commons License" style="border-width:0" src="http://i.creativecommons.org/l/by-nc-sa/3.0/us/80x15.png" /></a></p>


guid: http://tom.preston-werner.com/2010/05/11/tomdoc-reasonable-ruby-documentation
title: TomDoc - Reasonable Ruby Documentation
url: http://tom.preston-werner.com/2010/05/11/tomdoc-reasonable-ruby-documentation.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="tomdoc---reasonable-ruby-documentation">TomDoc - Reasonable Ruby Documentation</h1>

<p class="meta">11 May 2016 - San Francisco</p>

<p><a href="http://rdoc.rubyforge.org">RDoc</a> is an abomination. It’s ugly to read in plain
text, requires the use of the inane :nodoc: tag to prevent private method
documentation from showing up in final rendering, and does nothing to encourage
complete or unambiguous documentation of classes, methods, or parameters.
<a href="http://yardoc.org">YARD</a> is much better but goes too far in the other direction
(and still doesn’t look good in plain text). Providing an explicit way to
specify parameters and types is great, but having to remember a bunch of strict
tag names in order to be compliant is not a good way to encourage coders to
write documentation. And again we see a <code class="highlighter-rouge">@private</code> tag that’s necessary to hide
docs from the final render.</p>

<p>Three years ago, after suffering with these existing documentation formats for
far too long, I started using my own documentation format. It looked a bit like
RDoc but had a set of conventions for specifying parameters, return values, and
the expected types. It used plain language and full sentences so that a human
could read and understand it without having to parse machine-oriented tags or
crufty markup. I called this format TomDoc, because if Linus can name stuff
after himself, then why can’t I?</p>

<p>After years in the making, TomDoc is finally a well specified documentation
format. You can find the full spec at <a href="http://tomdoc.org">http://tomdoc.org</a>.</p>

<p>But enough talk. Here’s a sample of what a TomDoc’d method might look like:</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Public: Duplicate some text an abitrary number of times.</span>
<span class="c1">#</span>
<span class="c1"># text  - The String to be duplicated.</span>
<span class="c1"># count - The Integer number of times to duplicate the text.</span>
<span class="c1">#</span>
<span class="c1"># Examples</span>
<span class="c1">#</span>
<span class="c1">#   multiplex('Tom', 4)</span>
<span class="c1">#   # =&gt; 'TomTomTomTom'</span>
<span class="c1">#</span>
<span class="c1"># Returns the duplicated String.</span>
<span class="k">def</span> <span class="nf">multiplex</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">count</span><span class="p">)</span>
  <span class="n">text</span> <span class="o">*</span> <span class="n">count</span>
<span class="k">end</span>
</code></pre></div></div>

<p>At first glance you’ll notice a few things. First, and most important, is that
the documentation looks nice in plain text. When I’m working on a project, I
need to be able to scan and read method documentation quickly. Littering the
docs with tags and markup (especially HTML markup) is not acceptable. Code
documentation should be optimized for human consumption. Second, all parameters
and return values, and their expected types are specified. Types are generally
denoted by class name. Because Ruby is so flexible, you are not constrained by a
rigid type declaration syntax and are free to explain precisely how the expected
types may vary under different circumstances. Finally, the basic layout is
designed to be easy to remember. Once you commit a few simple conventions to
memory, writing documentation becomes second nature, with all of the tricky
decision making already done for you.</p>

<p>Today’s Ruby libraries suffer deeply from haphazard versioning schemes. Even
RubyGems itself does not follow a sane or predictable versioning pattern. This
lack of discipline stems from the absence of well defined Public APIs. TomDoc
attempts to solve this problem by making it simple to define an unambiguous
Public API for your library. Instead of assuming that all classes and methods
are intended for public consumption, TomDoc makes the Public API opt-in. To
denote that something is public, all you have to do is preface the main
description with “Public:”. By forcing you to explicitly state that a class or
method is intended for public consumption, a deliberate and thoughtful Public
API is automatically constructed that can inform disciplined version changes
according to the tenets of <a href="http://semver.org">Semantic Versioning</a>. In
addition, the prominent display of “Public” in a method description ensures that
developers are made aware of the sensitive nature of the method and do not
carelessly change the signature of something in the Public API.</p>

<p>Once a Public API has been established, some very exciting things become
possible. We’re currently working on a processing tool that will render TomDoc
into various forms (terminal, HTML, etc). If you run this tool on a library,
you’ll get a printout of the Public API documentation. You can publish this
online so that others have easy access to it. When you roll a new version of the
library, you can run the tool again, giving it a prior version as a base, and
have it automatically display only the methods that have changed. This diff will
be extremely useful for users while they upgrade to the new version (or so they
can evaluate whether an upgrade is warranted)!</p>

<p>While I’ve been using various nascent forms of TomDoc for several years, we’re
just now starting to adopt it for everything we do at GitHub. Now that I’ve
formalized the spec it will be easy for the entire team to write compliant
TomDoc. The goal is to have every class, method, and accessor of every GitHub
library documented. In the future, once we have proper tooling, we’d even like
to create a unit test that will fail if anything is missing documentation.</p>

<p>TomDoc is still a rough specification so I’m initially releasing it as 0.9.0.
Over the coming months I’ll make any necessary changes to address user concerns
and release a 1.0.0 version once things have stabilized. If you’d like to
suggest changes, please open an issue on the <a href="http://github.com/mojombo/tomdoc">TomDoc GitHub
repository</a>.</p>


guid: http://tom.preston-werner.com/2010/08/23/readme-driven-development
title: Readme Driven Development
url: http://tom.preston-werner.com/2010/08/23/readme-driven-development.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="readme-driven-development">Readme Driven Development</h1>

<p class="meta">23 August 2010 - San Francisco</p>

<p>I hear a lot of talk these days about TDD and BDD and Extreme Programming and SCRUM and stand up meetings and all kinds of methodologies and techniques for developing better software, but it’s all irrelevant unless the software we’re building meets the needs of those that are using it. Let me put that another way. A perfect implementation of the wrong specification is worthless. By the same principle a beautifully crafted library with no documentation is also damn near worthless. If your software solves the wrong problem or nobody can figure out how to use it, there’s something very bad going on.</p>

<p>Fine. So how do we solve this problem? It’s easier than you think, and it’s important enough to warrant its very own paragraph.</p>

<p>Write your Readme first.</p>

<p>First. As in, before you write any code or tests or behaviors or stories or ANYTHING. I know, I know, we’re programmers, dammit, not tech writers! But that’s where you’re wrong. Writing a Readme is absolutely essential to writing good software. Until you’ve written about your software, you have no idea what you’ll be coding. Between The Great Backlash Against Waterfall Design and The Supreme Acceptance of Agile Development, something was lost. Don’t get me wrong, waterfall design takes things way too far. Huge systems specified in minute detail end up being the WRONG systems specified in minute detail. We were right to strike it down. But what took its place is too far in the other direction. Now we have projects with short, badly written, or entirely missing documentation. Some projects don’t even have a Readme!</p>

<p>This is not acceptable. There must be some middle ground between reams of technical specifications and no specifications at all. And in fact there is. That middle ground is the humble Readme.</p>

<p>It’s important to distinguish Readme Driven Development from Documentation Driven Development. RDD could be considered a subset or limited version of DDD. By restricting your design documentation to a single file that is intended to be read as an introduction to your software, RDD keeps you safe from DDD-turned-waterfall syndrome by punishing you for lengthy or overprecise specification. At the same time, it rewards you for keeping libraries small and modularized. These simple reinforcements go a long way towards driving your project in the right direction without a lot of process to ensure you do the right thing.</p>

<p>By writing your Readme first you give yourself some pretty significant advantages:</p>

<ul>
  <li>
    <p>Most importantly, you’re giving yourself a chance to think through the project without the overhead of having to change code every time you change your mind about how something should be organized or what should be included in the Public API. Remember that feeling when you first started writing automated code tests and realized that you caught all kinds of errors that would have otherwise snuck into your codebase? That’s the exact same feeling you’ll have if you write the Readme for your project before you write the actual code.</p>
  </li>
  <li>
    <p>As a byproduct of writing a Readme in order to know what you need to implement, you’ll have a very nice piece of documentation sitting in front of you. You’ll also find that it’s much easier to write this document at the beginning of the project when your excitement and motivation are at their highest. Retroactively writing a Readme is an absolute drag, and you’re sure to miss all kinds of important details when you do so.</p>
  </li>
  <li>
    <p>If you’re working with a team of developers you get even more mileage out of your Readme. If everyone else on the team has access to this information before you’ve completed the project, then they can confidently start work on other projects that will interface with your code. Without any sort of defined interface, you have to code in serial or face reimplementing large portions of code.</p>
  </li>
  <li>
    <p>It’s a lot simpler to have a discussion based on something written down. It’s easy to talk endlessly and in circles about a problem if nothing is ever put to text. The simple act of writing down a proposed solution means everyone has a concrete idea that can be argued about and iterated upon.</p>
  </li>
</ul>

<p>Consider the process of writing the Readme for your project as the true act of creation. This is where all your brilliant ideas should be expressed. This document should stand on its own as a testament to your creativity and expressiveness. The Readme should be the single most important document in your codebase; writing it first is the proper thing to do.</p>

<p>–</p>

<p><a href="http://news.ycombinator.com/item?id=1627246">Discuss this post on Hacker News</a></p>


guid: http://tom.preston-werner.com/2010/10/18/optimize-for-happiness
title: Optimize for Happiness
url: http://tom.preston-werner.com/2010/10/18/optimize-for-happiness.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="optimize-for-happiness">Optimize for Happiness</h1>

<p class="meta">18 October 2010 - San Francisco</p>

<p>Two days ago I had the pleasure of speaking at Startup School, a yearly conference on entrepreneurism put on by the great folks at Y Combinator. Never before have I see such a high concentration of smart ambitious people in one place.</p>

<p>You can watch the recording of my thirty minute slot on Youtube:</p>

<center><iframe width="560" height="315" src="https://www.youtube.com/embed/T4VtBcmbbSs" frameborder="0" allowfullscreen=""></iframe></center>

<p>Since I only had about 25 minutes for the talk and 5 minutes for questions, I wanted to expand upon and clarify some of the ideas I introduced during the talk and then make myself available for additional questions. So today (Monday, 18 October 2010) I’ll be answering any questions you have via Hacker News:</p>

<center><b><a href="http://news.ycombinator.com/item?id=1804443">Ask me a question on HN!</a></b></center>

<p>The very first commit to GitHub was made exactly three years ago tomorrow. In that time our team of thirteen has signed up over 420,000 developers and now hosts 1.3 million Git repositories, making us the largest code host on the planet. And we’ve done all of this without ever taking a dime of funding from outside the company. In fact, even within the company we only invested a few thousand dollars out of our own pockets during the first months to cover legal fees.</p>

<p>During the presentation I talk about a choice between optimizing for happiness and optimizing for money. When I say “optimizing for money” I mean following the traditional venture capital route of raising a ton of money to stash in your bank account and going for a huge exit. The unfortunate reality of this approach is that for aspiring entrepreneurs that are not well connected to the VC world, it can take an extraordinary amount of time and effort to raise that money. Even if you are able to raise capital, you are suddenly responsible to your investors and will need to align your interests with theirs.</p>

<p>In a world dominated by news about Facebook, Apple, Google, YouTube, Zappos, and other companies heavily funded by venture capital, it’s easy to forget that you can still build a highly profitable business with significant impact on a global market without having to first spend three months on Sand Hill Road asking for permission to build your product.</p>

<p>The infrastructure components necessary to run an internet business are finally cheap enough that you can get started without a huge up-front investment. In the months that you would traditionally spend in glass-walled conference rooms you can now build a sophisticated prototype of your product and start getting users signed up and engaging you with useful feedback.</p>

<p>This is what I mean by optimizing for happiness: I’m a hacker; I’m happy when I’m building things of value, not when I’m writing a business plan filled with make believe numbers.</p>

<p>When Chris and I started GitHub, I was working full time at Powerset and Chris was doing consulting work and plugging away on a product of his own. GitHub became the leisure activity that I worked on when I got home from the office. I could craft it however I pleased, and there was nobody telling me what to do. This feeling of control and ownership of something you own is intoxicating.</p>

<p>Within three months we had a simple product and moved into private beta. In six months we launched to the public and started charging for private plans. We’ve been profitable every month since public launch except for one (in which we hired two new employees at once). We did this by making a paycheck via other means until GitHub was generating enough revenue to support us full time at about 2/3 of what we were accustomed to making. We then raised our salaries over the next months when we hit specific revenue goals that allowed us to remain profitable. This happened about one year after inception.</p>

<p>A side effect of bootstrapping a sustainable company is what I like to call <b>infinite runway.</b> This is another element of optimizing for happiness. With venture backed endeavors you generally find that during the first several years the numbers in your bank account are perpetually decreasing, giving your company an expiration date. Your VCs have encouraged you to grow fast and spend hard, which makes perfect sense for them, but not necessary for you. Not if you’re trying to optimize for happiness.</p>

<p>VCs want to see quick success or quick failure. They are optimizing for money. There’s nothing wrong with that as long as you want the same things they do. But if you’re like me, then you care more about building a kickass product than you do about having a ten figure exit. If that’s true, then maybe you should be optimizing for happiness. One way to do this is by bootstrapping a sustainable business with infinite runway. When there are fewer potentially catastrophic events on the horizon, you’ll find yourself smiling a lot more often.</p>

<p>The ironic thing about bootstrapping and venture capital is that once you demonstrate some success, investors will come to YOU. When this happens you will be in a much better place to make a more reasoned choice about taking on additional capital and all the complexities that come with it. Talking to VCs with some leverage in your back pocket is an entirely different game from throwing yourself in front of a conference table full of general partners and trying to persuade them that you’re worth their time and money. Power is happiness.</p>

<p>There are other really great things you can do when you optimize for happiness. You can throw away things like financial projections, hard deadlines, ineffective executives that make investors feel safe, and everything that hinders your employees from building amazing products.</p>

<p>At GitHub we don’t have meetings. We don’t have set work hours or even work days. We don’t keep track of vacation or sick days. We don’t have managers or an org chart. We don’t have a dress code. We don’t have expense account audits or an HR department.</p>

<p>We pay our employees well and give them the tools they need to do their jobs as efficiently as possible. We let them decide what they want to work on and what features are best for the customers. We pay for them to attend any conference at which they’ve gotten a speaking slot. If it’s in a foreign country, we pay for another employee to accompany them because traveling alone sucks. We show them the profit and loss statements every month. We expect them to be responsible.</p>

<p>We make decisions based on the merits of the arguments, not on who is making them. We strive every day to be better than we were the day before.</p>

<p>We hold our board meetings in bars.</p>

<p>We do all this because we’re optimizing for happiness, and because there’s nobody to tell us that we can’t.</p>

<center><b><a href="http://news.ycombinator.com/item?id=1804443">Ask me a question on HN!</a></b></center>


guid: http://tom.preston-werner.com/2010/12/11/designer-architect-developer
title: Designer, Architect, Developer
url: http://tom.preston-werner.com/2010/12/11/designer-architect-developer.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="designer-architect-developer">Designer, Architect, Developer</h1>

<p class="meta">11 Dec 2010 - San Francisco</p>

<p>Over the last six years I’ve bootstrapped three successful enterprises (Cube6 Media, Gravatar, and GitHub) and failed to gain traction with a handful of others. After a lot of thought and reflections on these experiences, I’ve identified three major skills that should be present in order to best build a successful web application. These roles can be loosely defined as the Designer, the Architect, and the Developer.</p>

<p>In college I spent a lot of time in the campus dark room dipping rolls of film and sheets of paper into various chemical baths beneath a dim red light. The most interesting part, though, was mounting the negative into the projector and exposing the photo paper. Every time I turned on the bright light of the projector I was reminded of a saying that has stuck with me ever since: “A photograph is nothing more than an image created by light.” Think about that for a second. The only way the photograph, and hence, the viewer, interact with the original subject is via the light that was captured. None of the fancy flashes, soft boxes, bounces, umbrellas, or backdrops mean a thing if the light they produce or redirect is in the wrong place. If the light is bad, the photograph is bad.</p>

<p>I think the same concept holds true for web applications. Adapting the saying for our own situation, I would say: “A web application is nothing more than an experience created by design.” Users can’t see what technology you use or whether you follow an agile development process or not. All they experience is what’s on the screen. It can’t be confusing, it can’t look amateur, and it can’t have spelling errors. If the UX is bad, the web application is bad. It’s that simple.</p>

<p>The way you get good UX is by having a good designer. Someone on the team must be skilled not only in making things pretty, but in making them usable as well. Without a good UX/visual design, you may as well not even bother. It’s impossible to stress how important this is.</p>

<p>Design comes first. It defines what you will build. Once you have an idea of what you’re creating, you need to figure out how to make it happen. That’s where the Architect comes in.</p>

<p>With the recent explosion of open source solutions to common problems like databases, web frameworks, job processors, messaging systems, etc, you need a team member that has a broad understanding of the technology landscape. The choices you make early on will impact your company for many years, and the wrong choices can spell disaster. The role of the Architect is to choose the best tools for the job, and to decide when new tools need to be created.</p>

<p>The Architect must also be ready to scale any piece of the site when you start attracting users. There’s a fine line between premature optimization and crumbling under the wave of thousands of new signups. A good architect will always be one step ahead of the curve, laying the groundwork for future scaling needs just before they are needed.</p>

<p>Design and architecture dictate what you build and how you build it, but without someone to do the construction, you’re dead in the water. The role of the Developer is to turn the wishes of the Designer into reality while staying within the constraints that the Architect has put forth. In addition, the Developer has to ensure that the codebase remains healthy and protect against technical debt. Sloppy development up front means a huge amount of wasted effort later on.</p>

<p>The three roles of the Designer, the Architect, and the Developer may reside in a single person, but it’s much more common to see groups of two or three people satisfy all these skills. In fact, the best founding teams are those where everyone fills some combination of roles. This fosters an environment of friendly argument that leads to better decisions.</p>

<p>But whatever you do, make sure your team fills all of these roles. Once you do, executing on your idea should come easily!</p>


guid: http://tom.preston-werner.com/2011/03/29/ten-lessons-from-githubs-first-year
title: Ten Lessons from GitHub's First Year
url: http://tom.preston-werner.com/2011/03/29/ten-lessons-from-githubs-first-year.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="ten-lessons-from-githubs-first-year">Ten Lessons from GitHub’s First Year</h1>

<p class="meta">29 Mar 2011 / 29 Dec 2008 - San Francisco</p>

<div class="note">
This post was written in late December of 2008, more than two years
ago. It has stayed in my drafts folder since then, waiting for the last 2% to be
written. Why I never published it is beyond my reckoning, but it serves as a
great reminder of how I perceived the world back then. In the time since I wrote
this we've grown from four people to twenty-six, settled into an office,
installed a kegerator, and still never taken outside funding. In some ways,
things have changed a great deal, but in the most important ways, things are
still exactly the same. Realizing this puts a big smile on my face.
</div>

<p>The end of the year is a great time to sit down with a glass of your favorite
beverage, dim the lights, snuggle up next to the fire and think about what
you’ve learned over the past twelve months.</p>

<p>For me, 2008 was the year that I helped design, develop, and launch GitHub.
Creating a new startup is an intense learning experience. Through screwups and
triumphs, I have learned some valuable lessons this year. Here’s a few of them.</p>

<h2 id="start-early">Start Early</h2>

<p>When Chris and I started working on GitHub in late 2007, Git was largely unknown
as a version control system. Sure, Linux kernel hackers had been using it since
day one, but outside of that small microcosm, it was rare to come across a
developer that was using it on a day-to-day basis. I was first introduced to Git
by Dave Fayram, a good friend and former coworker during my days at Powerset.
Dave is the quintessential early adopter and, as far as I can calculate, patient
zero for the spread of Git adoption in the Ruby community and beyond.</p>

<p>Back then, the Git landscape was pretty barren. Git had only recently become
usable by normal people with the 1.5 release. As for Git hosting, there was
really only <a href="http://repo.or.cz/">repo.or.cz</a>, which felt to me very limited,
clumsy, and poorly designed. There were no commercial Git hosting options
whatsoever. Despite this, people were starting to talk about Git at the Ruby
meetups. About how awesome it was. But something was amiss. Git was supposed to
be this amazing way to work on code in a distributed way, but what was the
mechanism to securely share private code? Your only option was to setup user
accounts on Unix machines and use that as an ad-hoc solution. Not ideal.</p>

<p>And so GitHub was born. But it was born into a world where there was no existing
market for paid Git hosting. We would be <em>creating</em> the market. I vividly
remember telling people, “I don’t expect GitHub to succeed right away. Git
adoption will take a while, but we’ll be ready when it happens.” Neither Chris
nor I were in any particular hurry for this to happen. I was working full time
at Powerset, and he was making good money as a Rails consultant. By choosing to
build early on top of a nascent technology, we were able to construct a startup
with basically no overhead, no competition, and in our free time.</p>

<h2 id="adapt-to-your-customers">Adapt to Your Customers</h2>

<p>Here’s a seemingly paradoxical piece of advice for you: Listen to your
customers, but don’t let them tell you what to do. Let me explain. Consider a
feature request such as “GitHub should let me FTP up a documentation site for my
project.” What this customer is really trying to say is “I want a simple way to
publish content related to my project,” but they’re used to what’s already out
there, and so they pose the request in terms that are familiar to them. We could
have implemented some horrible FTP based solution as requested, but we looked
deeper into the underlying question and now we allow you to publish content by
simply pushing a Git repository to your account. This meets requirements of both
functionality <em>and</em> elegance.</p>

<p>Another company that understands this concept at a fundamental level is Apple.
I’m sure plenty of people asked Apple to make a phone but Steve Jobs and his
posse looked beneath the request and figured out what people really wanted: a
nice looking, simple to use, and easy to sync mobile device that kicked some
serious ass. And that’s the secret. Don’t give your customers what they ask for;
give them what they <em>want</em>.</p>

<h2 id="have-fun">Have Fun</h2>

<p>I went to college at a little school in California called Harvey Mudd. Yeah, I
know you haven’t heard of it, but if you remember those US News &amp; World Report
“Best Colleges” books that you obsessed over in highschool (ok, maybe you
didn’t, but I did), Harvey Mudd was generally ranked as the engineering school
with the greatest number of hours of homework per night. Yes, more than MIT, and
yes, more than Caltech. It turned out to be true, as far as I can tell. I have
fond memories of freaking out about ridiculously complex spring/mass/damper
systems and figuring the magnetic flux of a wire wrapped around a cylinder in a
double helix. We studied hard–very hard. But we played hard too. It was the
only thing that could possibly keep us sane.</p>

<p>Working on a startup is like that. It feels a bit like college. You’re working
on insanely hard projects, but you’re doing it with your best friends in the
world and you’re having a great time (usually). In both environments, you have
to goof off a lot in order to balance things out. Burnout is a real and
dangerous phenomenon. Fostering a playful and creative environment is critical
to maintaining both your personal health, and the health (and idea output) of
the company.</p>

<h2 id="pay-attention-to-twitter">Pay attention to Twitter</h2>

<p>I’ve found Twitter to be an extremely valuable resource for instant feedback. If
the site is slow for some reason, Twitter will tell me so. If the site is
unreachable for people in a certain country (I’m looking at you China), I’ll
find out via Twitter. If that new feature we just released is really awesome,
I’ll get a nice ego boost by scanning the Twitter search.</p>

<p>People have a tendency to turn to Twitter to bitch about all the little bugs
they see on your website, usually appended with the very tiresome “FAIL”. These
are irksome to read, but added together are worth noticing. Often times these
innocent tweets will inform a decision about whether an esoteric bug is worth
adding to the short list. We also created a GitHub account on Twitter that our
support guy uses to respond to negative tweets. Offering this level of customer
service almost always turns a disgruntled customer into a happy one.</p>

<p>If you have an iPhone, I heartily recommend the
<a href="http://fanzter.com/products/download/summizer">Summizer</a> app from Fanzter, Inc.
It makes searching, viewing, and responding to tweets a cinch.</p>

<h2 id="deploy-at-will">Deploy at Will!</h2>

<p>At the first RailsConf I had the pleasure of hearing Martin Fowler deliver an
amazing keynote. He made some apt metaphors regarding agile development that I
will now paraphrase and mangle.</p>

<p>Imagine you’re tasked with building a computer controlled gun that can
accurately hit a target about 50 meters distant. That is the only requirement.
One way to do this is to build a complex machine that measures every possible
variable (wind, elevation, temperature, etc.) before the shot and then takes aim
and shoots. Another approach is to build a simple machine that fires rapidly and
can detect where each shot hits. It then uses this information to adjust the aim
of the next shot, quickly homing in on the target a little at a time.</p>

<p>The difference between these two approaches is to realize that bullets are
cheap. By the time the former group has perfected their wind detection
instrument, you’ll have finished your simple weapon and already hit the target.</p>

<p>In the world of web development, the target is your ideal offering, the bullets
are your site deploys, and your customers provide the feedback mechanism. The
first year of a web offering is a magical one. Your customers are most likely
early adopters and love to see new features roll out every few weeks. If this
results in a little bit of downtime, they’ll easily forgive you, as long as
those features are sweet. In the early days of GitHub, we’d deploy up to ten
times in one afternoon, always inching closer to that target.</p>

<p>Make good use of that first year, because once the big important customers start
rolling in, you have to be a lot more careful about hitting one of them with a
stray bullet. Later in the game, downtime and botched deploys are money lost and
you have to rely more on building instruments to predict where you should aim.</p>

<h2 id="you-dont-need-an-office">You Don’t Need an Office</h2>

<p>All four fulltime GitHub employees work in the San Francisco area, and yet we
have no office. But we’re not totally virtual either. In fact, a couple times a
week you’ll find us at a cafe in North Beach, huddled around a square table that
was made by nailing 2x4s to an ancient fold-out bulletin board. It’s no Google
campus, but the rent is a hell of a lot cheaper and the drinks are just as good!</p>

<p>This is not to say that we haven’t looked at a few places to call home. Hell, we
almost leased an old bar. But in the end there’s no hurry to settle down. We’re
going to wait until we find the perfect office. Until then, we can invest the
savings back into the company, or into our pockets. For now, I like my couch and
the cafe just fine.</p>

<p>Of course, none of this would be possible without 37signals’
<a href="http://www.campfirenow.com/">Campfire</a> web-based chat and the
very-difficult-to-find-but-totally-amazing
<a href="http://productblog.37signals.com/products/2008/10/propane-takes-c.html">Propane</a>
OSX desktop app container that doubles the awesome. Both highly recommended.</p>

<h2 id="hire-through-open-source">Hire Through Open Source</h2>

<p>Beyond the three cofounders of GitHub, we’ve hired one full time developer
(Scott Chacon) and one part time support specialist (Tekkub).</p>

<p>We hired Tekkub because he was one of the earliest GitHub users and actively
maintains more than 75 projects (WoW addons mostly) on GitHub and was very
active in sending us feedback in the early days. He would even help people out
in the IRC channel, simply because he enjoyed doing so.</p>

<p>I met Scott at one of the San Francisco Ruby meetups where he was presenting on
one of his myriad Git-centric projects. Scott had been working with Git long
before anyone else in the room. He was also working on a pure Ruby
implementation of Git at the same time I was working on my fork/exec based Git
bindings. It was clear to me then that depending on how things went down, he
could become either a powerful ally or a dangerous foe. Luckily, we all went
drinking afterwards and we became friends. Not long after, Scott started
consulting for us and wrote the entire backend for what you now know of as
<a href="http://gist.github.com/">Gist</a>. We knew then that we would do whatever it took
to hire Scott full time. There would be no need for an interview or references.
We already knew everything we needed to know in order to make him an offer
without the slightest reservation.</p>

<p>The lesson here is that it’s far easier and less risky to hire based on relevant
past performance than it is to hire based on projected future performance.
There’s a corollary that also comes into play: if you’re looking to work for a
startup (or anyone for that matter), contribute to the community that surrounds
it. Use your time and your code to prove that you’re the best one for the job.</p>

<h2 id="trust-your-team">Trust your Team</h2>

<p>There’s nothing I hate more than micromanagers. When I was doing graphic design
consulting 5 years ago I had a client that was very near the Platonic form of a
micromanager. He insisted that I travel to his office where I would sit in the
back room at an old Mac and design labels and catalogs and touch up photographs
of swimwear models (that part was not so bad!). While I did these tasks he would
hover over me and bark instructions. “Too red! Can you make that text smaller?
Get rid of those blemishes right there!” It drove me absolutely batty.</p>

<p>This client could have just as easily given me the task at the beginning of the
day, gone and run the business, and come back in 6 hours whereupon I would have
created better designs twice as fast as if he were treating me like a robot that
converted his speech into Photoshop manipulations. By treating me this way, he
was marginalizing my design skills and wasting both money and talent.</p>

<p>Micromanagement is symptomatic of a lack of trust. The remedy for this ailment
is to hire experts and then trust their judgment. In a startup, you can
drastically reduce momentum by applying micromanagement, or you can boost
momentum by giving trust. It’s pretty amazing what can happen when a group of
talented people who trust each other get together and decide to make something
awesome.</p>

<h2 id="you-dont-need-venture-capital">You Don’t Need Venture Capital</h2>

<p>A lot has been written recently about how the venture capital world is changing.
I don’t pretend to be an expert on the subject, but I’ve learned enough to say
that a web startup like ours doesn’t need any outside money to succeed. I know
this because we haven’t taken a single dime from investors. We bootstrapped the
company on a few thousand dollars and became profitable the day we opened to the
public and started charging for subscriptions.</p>

<p>In the end, every startup is different, and the only person that can decide if
outside money makes sense is you. There are a million things that could drive
you to seek and accept investment, but you should make sure that doing so is in
your best interest, because it’s quite possible that you don’t <em>need</em> to do so.
One of the reasons I left my last job was so that I could say “the buck stops
here.” If we’d taken money, I would no longer be able to say that.</p>

<h2 id="open-source-whatever-you-can">Open Source Whatever You Can</h2>

<p>In order for GitHub to talk to Git repositories, I created the first ever Ruby
Git bindings. Eventually, this library become quite complete and we were faced
with a choice: Do we open source it or keep it to ourselves? Both approaches
have benefits and drawbacks. Keeping it private means that the hurdle for
competing Ruby-based Git hosting sites would be higher, giving us an advantage.
But open sourcing it would mean that</p>

<p><b>NOTE: This is where the post ended and remained frozen in carbonite until
today. I intend to write a follow up post on our open source philosophy at
GitHub in the near future. I’m sure the suspense is killing you!</b></p>

<p><a href="http://news.ycombinator.com/item?id=2384320">Discuss this post on Hacker News</a></p>


guid: http://tom.preston-werner.com/2011/05/03/rejected-bio-from-the-setup
title: Rejected Bio from The Setup
url: http://tom.preston-werner.com/2011/05/03/rejected-bio-from-the-setup.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="rejected-bio-from-the-setup">Rejected Bio from The Setup</h1>

<p class="meta">03 May 2011 - San Francisco</p>

<p>Yesterday, <a href="http://tom.preston-werner.usesthis.com/">the autobiographical post I wrote for The
Setup</a> went live. I wrote that post
over a year ago and then entered into an epic battle with
<a href="http://twitter.com/#!/waferbaby">@waferbaby</a> about the length of my “Who are
you, and what do you do?” section. He said it was too long. I said it could
not be shortened. And so the post sat for a year, collecting dust, neither of
us prepared to back down.</p>

<p>About a month ago I decided that it was foolish to let the words I had written rot on my hard drive and so I did the only thing I knew how to do: overreact. So I cut the original nine-hundred words of my bio down to fourteen words and resubmitted it to Daniel. Those are the words you see in the post now.</p>

<p>For your pleasure, here is the original bio in its full, unabridged glory.</p>

<hr />

<p>My name is Tom Preston-Werner. I find that the hyphenated last name
makes me sound distinguished and worth listening to. I grew up three
decades ago in a small city in Iowa along the Mississippi, which means
I shucked a lot of corn and know exactly how many mosquitos will land
on your arm should you hold it still for ten minutes at dusk on the
muggiest day of the summer. As an aspiring theoretical particle
physicist, I worked my way through entire shelves of scientific
literature from the public library, desperately wanting to understand
the bewildering mathematics that littered the pages like so many
leaves on the bottom of that morning’s cup of green tea. I searched in
vain for instructors or classmates that could provide me with the
insight necessary to comprehend the true meaning of Heisenberg’s
Uncertainty Principle, but all I found were underpaid math teachers
and disillusioned “students” in search of their next smoke break.
After obsessing over US News’ Best Colleges reports for months I
finally chose and was accepted to Harvey Mudd, a tiny engineering
school in California famous for assigning the greatest number of hours
of homework per night. This sounded just perfect to me. Finally a
place I could bring up the EPR Paradox and not be immediately
stigmatized as “that science weirdo with the hilariously thick glasses
and unfortunate hairdo.”</p>

<p>Mudd did not disappoint. But now I had the opposite problem. In order
to properly understand particle physics, you must have a deep and
profound love of math. You have to be so comfortable with abstract
concepts that even Picasso would be jealous. Ironically, in order to
grasp the fundamental reality of our universe, you must forget about
the “reality” of everyday life and start living in a world comprised
of eigenvectors, Hilbert spaces, and Planck’s constant. This was a
leap I could not make. I like math, but I’m too easily distracted by
macroscopic reality to make it my profession.</p>

<p>Once I accepted that I would never spend late nights poring over
bubble chamber printouts at Fermilab, it became obvious that I was
destined to enter computer science. I started programming in BASIC on
a TRS-80 that my dad bought from Radio Shack when I was 8 years old.
Since then, I’d learned to love the discipline and creativity involved
in making a machine do my bidding. It was like having a super-obedient
but annoyingly logical little brother. He’ll do anything you want as
long as you tell him in precise and unambiguous language. The best
thing is, the feedback is immediate. In physics, it can take twenty
years to prove that a single esoteric particle even exists. When
you’re writing a program that displays the number of electrons in each
of the shells around the nucleus of every element, the feedback is
immediate and intoxicating. With just a few keystrokes, the world is
changed forever. Try to get that kind of rush even once in a lifetime
as a theoretical particle physicist. I dare you.</p>

<p>In 1999, after two years of college, I dropped out of Harvey Mudd to
join a startup with some friends that were graduating. It was the end
of the first dot-com bubble and I thought I could strike it rich,
right then and there. Sadly, like so many startups of the day, we
never accomplished what we envisioned and I ended up bouncing between
jobs and consulting gigs for six years until I found myself in San
Francisco. If Harvey Mudd was my mecca for physicists, then San
Francisco was my mecca for programmers. Where else can you be grabbing
lunch at a taqueria and overhear a group at the next table discussing
the finer points of optimizing C code to run on an embedded processor?</p>

<p>I moved to San Francisco to take a job as a Ruby developer with a
Wikipedia search engine called Powerset. I also began attending Ruby
meetups and drinking with local software developers. There are a lot
of talented people in the Bay Area and I wanted to meet them all.
Within the Ruby community, a distributed version control system called
Git was starting to get some attention. It was a really cool way of
working with other people on code, but there wasn’t an easy way to get
up and running with a group of developers. Along with cofounders Chris
Wanstrath and PJ Hyett (who I met at the Ruby meetups) I started a
company called GitHub that would address this problem and make it dead
simple to share Git repositories and collaborate on code with other
developers.</p>

<p>At first, we worked on GitHub on the side, putting in time on evenings
and weekends. After six months we launched the site to the public and
started charging. Not long after that, Powerset was acquired by
Microsoft and I was faced with a choice: stay on as a Microsoft
employee with a big retention bonus and give up GitHub or turn down
the Microsoft money and quit Powerset to work on GitHub full-time. You
can read more about this saga in my blog post entitled <a href="http://tom.preston-werner.com/2008/10/18/how-i-turned-down-300k.html">How I Turned
Down $300,000 from Microsoft to go Full-Time on
GitHub</a>.
I think I made the right decision.</p>

<p>Today GitHub has twenty-nine employees and more than 730,000 users with over
2,000,000 repositories. We’re growing fast, and I’m having the time of
my life!</p>


guid: http://tom.preston-werner.com/2011/11/22/open-source-everything
title: Open Source (Almost) Everything
url: http://tom.preston-werner.com/2011/11/22/open-source-everything.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="open-source-almost-everything">Open Source (Almost) Everything</h1>

<p class="meta">22 Nov 2011 - San Francisco</p>

<p>When Chris and I first started working on GitHub in late 2007, we split the work into two parts. Chris worked on the Rails app and I worked on Grit, the first ever Git bindings for Ruby. After six months of development, Grit had become complete enough to power GitHub during our public launch of the site and we were faced with an interesting question:</p>

<p>Should we open source Grit or keep it proprietary?</p>

<p>Keeping it private would provide a higher hurdle for competing Ruby-based Git hosting sites, giving us an advantage. Open sourcing it would mean thousands of people worldwide could use it to build interesting Git tools, creating an even more vibrant Git ecosystem.</p>

<p>After a small amount of debate we decided to open source Grit. I don’t recall the specifics of the conversation but that decision nearly four years ago has led to what I think is one of our most important core values: open source (almost) everything.</p>

<h2 id="why-is-it-awesome-to-open-source-almost-everything">Why is it awesome to open source (almost) everything?</h2>

<p>If you do it right, open sourcing code is <strong>great advertising</strong> for you and your company. At GitHub we like to talk publicly about libraries and systems we’ve written that are still closed but destined to become open source. This technique has several advantages. It helps determine what to open source and how much care we should put into a launch. We recently open sourced Hubot, our chat bot, to widespread delight. Within two days it had 500 watchers on GitHub and 409 upvotes on Hacker News. This translates into goodwill for GitHub and more superfans than ever before.</p>

<p>If your code is popular enough to attract outside contributions, you will have created a <strong>force multiplier</strong> that helps you get more work done faster and cheaper. More users means more use cases being explored which means more robust code. Our very own <a href="https://github.com/defunkt/resque">resque</a> has been improved by 115 different individuals outside the company, with hundreds more providing 3rd-party plugins that extend resque’s functionality. Every bug fix and feature that you merge is time saved and customer frustration avoided.</p>

<p>Smart people like to hang out with other smart people. Smart developers like to hang out with smart code. When you open source useful code, you <strong>attract talent</strong>. Every time a talented developer cracks open the code to one of your projects, you win. I’ve had many great conversations at tech conferences about my open source code. Some of these encounters have led to ideas that directly resulted in better solutions to problems I was having with my projects. In an industry with such a huge range of creativity and productivity between developers, the right eyeballs on your code can make a big difference.</p>

<p>If you’re hiring, <strong>the best technical interview possible</strong> is the one you don’t have to do because the candidate is already kicking ass on one of your open source projects. Once technical excellence has been established in this way, all that remains is to verify cultural fit and convince that person to come work for you. If they’re passionate about the open source code they’ve been writing, and you’re the kind of company that cares about well-crafted code (which clearly you are), that should be simple! We hired <a href="https://github.com/tanoku">Vicent Martí</a> after we saw him doing stellar work on <a href="https://github.com/libgit2/libgit2">libgit2</a>, a project we’re spearheading at GitHub to extract core Git functionality into a standalone C library. No technical interview was necessary, Vicent had already proven his skills via open source.</p>

<p>Once you’ve hired all those great people through their contributions, dedication to open source code is an amazingly effective way to <strong>retain that talent</strong>. Let’s face it, great developers can take their pick of jobs right now. These same developers know the value of coding in the open and will want to build up a portfolio of projects they can show off to their friends and potential future employers. That’s right, a paradox! In order to keep a killer developer happy, you have to help them become more attractive to other employers. But that’s ok, because that’s exactly the kind of developer you want to have working for you. So relax and let them work on open source or they’ll go somewhere else where they can.</p>

<p>When I start a new project, I assume it will eventually be open sourced (even if it’s unlikely). This mindset leads to <strong>effortless modularization</strong>. If you think about how other people outside your company might use your code, you become much less likely to bake in proprietary configuration details or tightly coupled interfaces. This, in turn, leads to cleaner, more maintainable code. Even internal code should pretend to be open source code.</p>

<p>Have you ever written an amazing library or tool at one job and then left to join another company only to rewrite that code or remain miserable in its absence? I have, and it sucks. By getting code out in the public we can drastically <strong>reduce duplication of effort</strong>. Less duplication means more work towards things that matter.</p>

<p>Lastly, <strong>it’s the right thing to do</strong>. It’s almost impossible to do anything these days without directly or indirectly executing huge amounts of open source code. If you use the internet, you’re using open source. That code represents millions of man-hours of time that has been spent and then given away so that everyone may benefit. We all enjoy the benefits of open source software, and I believe we are all morally obligated to give back to that community. If software is an ocean, then open source is the rising tide that raises all ships.</p>

<h2 id="ok-then-what-shouldnt-i-open-source">Ok, then what shouldn’t I open source?</h2>

<p>That’s easy. Don’t open source anything that represents core business value.</p>

<p>Here are some examples of what we don’t open source and why:</p>

<ul>
  <li>Core GitHub Rails app (easier to sell when closed)</li>
  <li>The Jobs Sinatra app (specially crafted integration with github.com)</li>
</ul>

<p>Here are some examples of things we do open source and why:</p>

<ul>
  <li>Grit (general purpose Git bindings, useful for building many tools)</li>
  <li>Ernie (general purpose BERT-RPC server)</li>
  <li>Resque (general purpose job processing)</li>
  <li>Jekyll (general purpose static site generator)</li>
  <li>Gollum (general purpose wiki app)</li>
  <li>Hubot (general purpose chat bot)</li>
  <li>Charlock_Holmes (general purpose character encoding detection)</li>
  <li>Albino (general purpose syntax highlighting)</li>
  <li>Linguist (general purpose filetype detection)</li>
</ul>

<p>Notice that everything we keep closed has specific business value that could be compromised by giving it away to our competitors. Everything we open is a general purpose tool that can be used by all kinds of people and companies to build all kinds of things.</p>

<h2 id="what-is-the-one-true-license">What is the One True License?</h2>

<p>I prefer the MIT license and almost everything we open source at GitHub carries this license. I love this license for several reasons:</p>

<ul>
  <li>
    <p>It’s short. Anyone can read this license and understand exactly what it means without wasting a bunch of money consulting high-octane lawyers.</p>
  </li>
  <li>
    <p>Enough protection is offered to be relatively sure you won’t sue me if something goes wrong when you use my code.</p>
  </li>
  <li>
    <p>Everyone understands the legal implications of the MIT license. Weird licenses like the WTFPL and the Beer license pretend to be the “ultimate in free licenses” but utterly fail at this goal. These fringe licenses are too vague and unenforceable to be acceptable for use in some companies. On the other side, the GPL is too restrictive and dogmatic to be usable in many cases. I want everyone to benefit from my code. Everyone. That’s what Open should mean, and that’s what Free should mean.</p>
  </li>
</ul>

<h2 id="rad-how-do-i-get-started">Rad, how do I get started?</h2>

<p>Easy, just flip that switch on your GitHub repository from private to public and tell the world about your software via your blog, Twitter, Hacker News, and over beers at your local pub. Then sit back, relax, and enjoy being part of something big.</p>

<p><a href="http://news.ycombinator.com/item?id=3267432">Discuss this post on Hacker News</a></p>


guid: http://tom.preston-werner.com/2014/04/21/farewell-github-hello-immersive-computing
title: Farewell GitHub, Hello Immersive Computing
url: http://tom.preston-werner.com/2014/04/21/farewell-github-hello-immersive-computing.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="farewell-github-hello-immersive-computing">Farewell GitHub, Hello Immersive Computing</h1>

<p class="meta">21 Apr 2014 - New York City</p>

<p>Today is my last day at GitHub. Recent events have given me a lot of time to
reflect on what’s important to me, and I’ve decided to switch gears and focus on
building something from scratch again. Since visiting the Oculus VR team at
their office three months ago, I’ve come to believe that immersive computing
(aka virtual reality) is poised to rival the personal computer, the web, social
networking, and mobile devices in its impact. While the timing is more abrupt
than I had intended, with everything that’s happened, I think now is the right
time to do this, and I’d like to explain why.</p>

<p>First, I want to address the serious accusations that were made against me and
my family over the past month. With every decision I made at GitHub and in every
interaction I had with employees, I tried to treat people better than they
expected and to resolve conflict with empathy. Despite that, I’ve made mistakes,
and I am deeply sorry to anyone who was hurt by those mistakes. It devastates me
to know that I missed the mark, and I will strive to do better, every day.</p>

<p>That said, I want to be very clear about one thing: neither my wife, Theresa,
nor I have ever engaged in gender-based harassment or discrimination. The
results of GitHub’s independent investigation unequivocally confirm this and we
are prepared to fight any further false claims on this matter to the full extent
of the law. I believe in diversity and equality for all people in all
professions, especially the tech sector. It’s immensely important to me and I
will continue to do my very best to further that belief.</p>

<p>Unfortunately, the investigation and all the attention surrounding it have me
concerned that remaining at GitHub would be a distraction for both me and the
company. I’m incredibly proud of what I’ve helped build at GitHub and I don’t
want the events of the past month to jeopardize that. I care too much about the
company and the people here to let that happen. The GitHub team is incredibly
strong, with fierce vision, and I have no doubt they will continue to
revolutionize software development for decades to come. Founding and building
GitHub has been the greatest adventure of my life. I’ve been so lucky to be on
this journey with such amazing, helpful, talented, and real people. I’m going to
miss working with such a great team, but I’m also insanely excited about the
future.</p>

<p>Since the early days of GitHub, I’ve wanted to create a different kind of
business. One that was <a href="http://tom.preston-werner.com/2010/10/18/optimize-for-happiness.html">Optimized for
Happiness</a>
and built atop a <a href="https://www.youtube.com/watch?v=i0FzgvYxdvk">Framework of
Happiness</a>. One where great people
could work on hard problems together to create unbelievably good products. I
believe I was able to achieve a great deal of success with that model at GitHub,
even if things didn’t always go perfectly according to plan. All of this has
been a tremendous learning experience for me.</p>

<p>Last January I stepped down as CEO and handed that role over to cofounder Chris
Wanstrath so I could focus on future-facing R&amp;D projects with small teams. This
kind of rapid, team-based innovation is what I live for. During my time away
from GitHub I started experimenting with Go, OpenGL, and Unity with an eye
towards the software side of immersive computing. It felt really good to get
back into a code editor and challenge the deeply logical and analytical part of
my brain. I’ve enjoyed the challenges of learning how to lead a company with
hundreds of people, but it’s very hard for me to deny the allure of coding a
system that could once again change the course of history.</p>

<p>I’m telling you this because I think stealth mode is bullshit and if you feel
the same way I do about immersive computing then I want to talk with you about
it. For the next few months I’m going to be living in Manhattan. My wife,
Theresa, is currently participating in Techstars NYC as their very first
nonprofit. Her startup, <a href="https://omakasecharity.org/">The Omakase Charity</a>,
helps donors learn about and support nonprofits that are changing the world with
technology. She’s one of the strongest and most thoughtful women I know, and I’m
hoping to help her succeed with her mission while I’m here.</p>

<p>Thank you to everyone that reached out to me over the last month, including the
generous team at Andreessen Horowitz. Your support has made a huge difference
and I’m truly excited for what’s next.</p>


guid: http://tom.preston-werner.com/2015/06/19/replicated
title: Replicated - An Easier Path from SaaS to Enterprise
url: http://tom.preston-werner.com/2015/06/19/replicated.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="replicated---an-easier-path-from-saas-to-enterprise">Replicated - An Easier Path from SaaS to Enterprise</h1>

<p class="meta">19 Jun 2015 - San Francisco</p>

<p>Over the last year I’ve had a chance to learn a lot more about early stage
funding and made angel investments in a handful of startups. So far I’ve
restricted my involvement to companies with ideas in which I have significant
domain knowledge. I also insist on founders with relentless product focus, a
fierce desire to help their customers be more awesome, and excellent
communication skills. I recently met just such a company, with just such
founders, tackling a problem that has personally caused me much pain.</p>

<p>The company is <a href="http://replicated.com">Replicated</a>, and founders Grant Miller
and Marc Campbell are making it easier to roll out an on-prem Enterprise
offering based on an existing cloud-based SaaS product.</p>

<p>At GitHub, we burned through <strong>a lot</strong> of developer cycles building our own
installer (several times), securing the installation environment, coding an
automated licensing management system, integrating single sign-on services
(LDAP, Active Directory, CAS, etc, etc), building out a searchable audit system,
supporting customer-reviewable support bundles (logs and other diagnostic
output), allowing numerous backup strategies, and countless other
Enterprise-specific features that were killing our Enterprise deals. All of this
on top of hiring and building out the necessary sales, support, and accounting
teams to create a smooth Enterprise experience for our customers.</p>

<p>Replicated provides common Enterprise functionality (much of what I mentioned
above, and all of it eventually) that you can wrap around your SaaS product,
resulting in a first-class on-prem product in a fraction of the time. Beyond
just technology, Replicated will help you understand your Enterprise customers
through documentation on best practices and insight into the requirements and
reasons that large companies desire the features they do. Until you can
empathise with your customer (which is very hard to do as a fast-moving SaaS
startup), you’ll never build the best product possible.</p>

<p>Getting into the Enterprise market will always be hard. But by reducing the
technology burden, Replicated plans to erase much of the pain so you can focus
on the other human-centric tasks. Not only am I an investor in Replicated, I
believe in their mission and their founders so much that I’ve joined as an
advisor. I understand what the uphill slog of the SaaS to Enterprise climb feels
like, and I’m going to do my best to ensure you don’t have to suffer it as much
as I did.</p>

<p>I’m also pleased to announce that Travis CI is now shipping their Enterprise
product using Replicated. To see what the installation process is like, <a href="https://www.youtube.com/watch?v=ViN-qkcovL0&amp;feature=youtu.be">watch
Grant install Travis CI
Enterprise</a> on a
fresh server in about seven minutes. For a deeper dive, Travis CI has also
published a blog post covering some of their process in <a href="http://blog.travis-ci.com/2015-06-19-how-we-improved-travis-ci-installation/">getting their
Enterprise installer ready using
Replicated</a>.</p>

<p>In the coming weeks, you’ll start to see other well-known startups launching (or
re-launching) Enterprise versions of their SaaS software on top of Replicated.
If you’re looking to do the same, and want to save yourself a lot of heartache,
email <a href="mailto:contact@replicated.com">contact@replicated.com</a>, and start
focusing on what matters the most: your unique and kickass product.</p>


guid: http://tom.preston-werner.com/2016/11/10/snyk
title: Snyk - Automatically Scan and Fix Ruby and Nodejs Vulnerabilities
url: http://tom.preston-werner.com/2016/11/10/snyk.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="snyk---automatically-scan-and-fix-ruby-and-nodejs-vulnerabilities">Snyk - Automatically Scan and Fix Ruby and Nodejs Vulnerabilities</h1>

<p class="meta">10 Nov 2016 - San Francisco</p>

<p>This is a story about a company called
<a href="https://snyk.io/blog/welcome-ruby-users/">Snyk</a> (pronounced “sneak”), their
founder Guy Podjarny, my decision to become one of their advisors, and how they
are going to help save you from malevolent agents trying to steal your digital
stuff.</p>

<p>If you’re anything like me, you’re simultaneously terrified and in awe of the
increasing commonality of large corporate security breaches. Even big names like
Ebay, Home Depot, Anthem, JP Morgan Chase, Target, LinkedIn, Dropbox, and Yahoo
are falling victim to sophisticated attacks. If you spend even a few minutes
looking into it, you’ll be shocked at how frequently these breaches are
happening now. The fine folks at Information is Beautiful have an excellent
interactive visualization of the <a href="http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/">World’s Biggest Data
Breaches</a>
over the last twelve years, in case you want to read all the gory details and
never get a restful night of sleep ever again:</p>

<p><a href="http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/">
  <img src="/images/posts/2016-11-10/breaches.png" />
</a></p>

<p>I’ve used a fair number of emotionally charged words above that might be
triggering your FUD detectors right about now. But be advised: it’s not paranoia
when they really are out to get you. If recent, extremely high profile (and
subsequently weaponized) breaches like those of the Clinton Campaign and the DNC
aren’t enough to make you want to air gap your entire life, then I envy your
steely-eyed mettle and implore you to teach me your meditation techniques.</p>

<p>The fact is, security is hard. And it’s getting harder every day. To win, you
have to get it right every single time. To lose (and lose big), you only have to
screw it up once.</p>

<p>During my years at GitHub, I spent a lot of time assembling a dedicated security
team, managing security audits and penetration tests, and working to establish a
culture of security awareness amongst our development team. All of this is
challenging and expensive, especially for a young company. Even worse, it’s the
kind of investment that’s totally invisible when it’s working, making it hard to
sustain until that crucial and terrible moment you end up on the front page of
Hacker News as the latest victim.</p>

<p>A year ago I was contemplating this, especially the difficult proposition of
having developers, furious at work on new features, constantly maintain
awareness of security vulnerabilities they might be inadvertently weaving into
the product. Web application developers are generally not security experts, and
though I would love to live in a world where that wasn’t true, it’s just not a
realistic expectation. Meanwhile, modern development means an increasing
reliance on 3rd party code. Even a small Rails app will probably have 300 or
more gem dependencies after a few months of development. It’s even more in the
nodejs world. This level of modularization and code reuse, driven by the
explosion of high quality open source over the last decade, is amazing and I
absolutely love it, but it comes at a security expense.</p>

<p>Open source projects are not known for their excellent security records.
Vulnerabilities like <a href="http://heartbleed.com/">Heartbleed</a> and
<a href="https://blog.cloudflare.com/inside-shellshock/">Shellshock</a> painfully
demonstrate the idea that “given enough eyeballs, all bugs are shallow” is
completely false. In fact, due to a flaw in YAML, Rails had a <a href="http://blog.codeclimate.com/blog/2013/01/10/rails-remote-code-execution-vulnerability-explained/">pretty extreme
remote code execution
vulnerability</a>
for years. If you were running any version of Rails prior to the fix, you were
vulnerable. This stuff is real, and as responsible developers, we need to be
more proactive about it.</p>

<p>Luckily, at the time I was pondering these matters, I ran into Guy Podjarny. As
a former cofounder of Blaze.io and then CTO of Web Experience at Akamai (which
acquired Blaze.io), Guy intimately understands the impact of security on today’s
web developers. He was working on an automated tool to scan and fix security
vulnerabilities in 3rd party dependencies. I was intrigued. They already had a
way to scan nodejs projects and look for known security vulnerabilities in the
dependency tree and automatically upgrade or patch affected libraries. I thought
this was pretty cool, but it was his vision for what automated security tooling
could be that sold me on him and his company. I can’t talk much about that
now, but just know that what Snyk is today is just the tip of what will
become an intelligent and proactive bodyguard for your entire codebase.</p>

<p>A few months ago, Snyk released GitHub integration to make it fantastically
simple to hook up your repos to Snyk and, my favorite feature: the ability to
monitor your repo for future vulnerabilities and then <strong>automatically submit a
pull request</strong> with the suggested package upgrade or hotfix patch (nodejs only for
now).</p>

<p>Today, <a href="https://snyk.io/blog/welcome-ruby-users/">Snyk announced support for
Ruby</a>. Take a look at that blog post,
it does an awesome job of explaining how simple it is to set up and what the
generated pull requests look like. It’s totally free for open source projects,
and extremely cheap insurance for your important projects.</p>

<p>Make no mistake, 3rd party code is a clear and present danger to your business.
If you don’t know if you’re vulnerable, then you must assume that you are and
take steps to protect yourself. Snyk makes it easy.</p>


guid: http://tom.preston-werner.com/2020/03/04/joining-the-netlify-board
title: Joining the Netlify Board to Help Shape the Future of the JAMstack
url: http://tom.preston-werner.com/2020/03/04/joining-the-netlify-board.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="joining-the-netlify-board-to-help-shape-the-future-of-the-jamstack">Joining the Netlify Board to Help Shape the Future of the JAMstack</h1>

<p class="meta">4 Mar 2020 - Cordon, France</p>

<p>In late September, 2015, I got an email out of the blue from <a href="https://twitter.com/biilmann">Mathias Biilmann</a>. He told me about his brand new 2-person startup called <a href="https://netlify.com/">Netlify</a> and asked if I’d fancy getting together to chat about it. You see, they were making it really easy to deploy static sites like Jekyll and thought I might be interested in learning more. It sounded cool, and it’s hard for me to turn down anything Jekyll-related, so I agreed to meet.</p>

<p>On October 1st we met up at a bar in the Dogpatch. I sat down with Matt and his cofounder <a href="https://twitter.com/Chr_Bach">Christian Bach</a> for several hours over some craft beer and they showed me how they could already build and deploy Jekyll sites (and a few other static site generators) to a global CDN with the push of a button. It was like GitHub Pages, but souped up and turbocharged with global scale production deployments in mind from day one.</p>

<p>That existing technology (and the smooth developer experience it offered) was already very impressive, but what really got my attention was their vision around a new term they wanted to coin: the <a href="https://jamstack.org/">JAMstack</a> (the JAM is for <strong>J</strong>avaScript, <strong>A</strong>PIs, &amp; <strong>M</strong>arkup).</p>

<p>At the time, we all called Jekyll and friends “static site generators”, even though many of them were starting to do really interesting things with JavaScript and 3rd party APIs to handle the dynamic bits like comments, text search, eCommerce, and more. The problem was not that this description of the generators was inaccurate, but that it failed to capture the capabilities and imaginative possibilities of this new architecture, namely, pre-rendering content into CDN-deliverable static pages and then layering interactivity on top with JavaScript and 3rd party APIs.</p>

<p>In the same way that the term “AJAX” allowed us to take discussions of what was possible on the web to new conceptual heights, Chris and Matt argued that developers and businesses could communicate more effectively about the potential of “JavaScript and API enhanced static markup sites you can deploy with a <code class="highlighter-rouge">git push</code>” if we had a catchy name to capture that development methodology and deployment flow.</p>

<p>I thought they were right, loved the named, and advised they start pushing it. While it was impossible to predict whether it would catch on or not (only the developer community gets to decide that), I had a strong inclination that the market was ready for it, and there was a high probability that it would be embraced.</p>

<p>In February of 2016 I participated in Netlify’s seed round, and we kept the conversations and craft beer flowing through the next four years and two rounds of funding.</p>

<p>In that time, Netlify has become the darling of the JAMstack universe by focusing on developer experience and relentlessly pushing the envelope of what you can do with JAMstack, and how easily you can do it. Early on they changed the game with <a href="https://www.netlify.com/blog/2016/07/20/introducing-deploy-previews-in-netlify/?utm_campaign=devex&amp;utm_medium=swyx-git&amp;utm_source=netliblog#main">Deploy Previews</a> so you can see a preview version of your entire site for every pull request you make. <a href="https://www.netlify.com/blog/2016/01/15/free-ssl-on-custom-domains/#main">They integrated Let’s Encrypt</a> so everyone can get SSL for free with zero hassle. They built <a href="https://www.netlify.com/blog/2017/09/19/form-handling-with-the-jamstack-and-netlify/#main">Netlify Forms</a> so you can submit simple web forms to their servers and see all the results. They launched <a href="https://www.netlify.com/blog/2017/09/07/introducing-built-in-identity-service-to-streamline-user-management/#main">Netlify Identity</a> to handle authentication and user administration. And then they launched <a href="https://www.netlify.com/blog/2018/03/20/netlifys-aws-lambda-functions-bring-the-backend-to-your-frontend-workflow/#main">Netlify Functions</a>—allowing you to write AWS Lambda functions in your git repo and have them deployed to AWS with that same single <code class="highlighter-rouge">git push</code>—and this time it not only revolutionized the ecosystem, it changed something inside of <strong>me</strong>.</p>

<p>For many years I’ve been looking for something that makes it possible to author a complex web application and <code class="highlighter-rouge">git push</code> deploy it to a scalable “delivery/compute/storage” layer, but with minimal lock-in and awesome global performance. AWS and other cloud providers have tons of building blocks for this, but I’ve never come across an end-to-end solution that felt right to me.</p>

<p>When I first heard about Netlify Functions, something clicked. What if you could author your web app in two parts: a JavaScript frontend client written in something like React that could be delivered statically via Netlify’s CDN, and the business logic layer that could run on AWS Lambda deployed via Netlify Functions? Both can be run on the edge and auto-scale. Choice of storage layer still depends quite a bit on the read/write characteristics of your app, but a variety of auto-scaling solutions are already on the market, with many more to come. All of this could be deployed with a simple <code class="highlighter-rouge">git push</code> to GitHub, with Netlify grabbing the repo, building it, and distributing it to CDN and AWS Lambda. HOLY SMOKES this is getting exciting!</p>

<p>Suffice to say, I started badgering Matt and Chris with my ideas to take the JAMstack to the next conceptual level. Beyond pre-rendered pages with layered interactivity, the JAMstack could just as easily encompass this broader scope of full-stack web app development and deployment.</p>

<p>I’ve been so into this idea (and all the rest of the potential of the JAMstack) that I started talking with the Netlify founders about joining the board to help in a more official capacity. I’m pleased to say that as of the closing of their just announced <a href="https://www.netlify.com/blog/2020/03/04/netlify-secures-series-c-funding-to-push-forward-our-vision-for-the-web/">$53 million Series C funding</a> (in which I also participated via <a href="https://prestonwernerventures.com/">Preston-Werner Ventures</a>), I have joined the Netlify board and will be doing my best to keep pushing the JAMstack envelope in ways that will surprise and delight you.</p>

<p>If you get as excited about this stuff as I do, come help us build Netlify into a developer happiness juggernaut. There is so much territory to explore, and we’re going to need a creative and diverse group of individuals to make it happen. <a href="https://www.netlify.com/careers/">Check out what we’re hiring for!</a></p>

<p>Oh, and speaking of that end-to-end web app development and deployment flow I mentioned…I never found a startup or open source project doing it the way I thought it should be done, so I’m building it. It’s a full-stack edge-ready web application framework designed to deploy to Netlify, and I’ll be announcing it on March 10th. Follow me on Twitter <a href="https://twitter.com/mojombo">@mojombo</a> to make sure you don’t miss it!</p>


guid: http://tom.preston-werner.com/2020/03/23/giving-1-million-toward-sf-covid-19-response
title: We Are Giving $1M Toward San Francisco COVID-19 Response
url: http://tom.preston-werner.com/2020/03/23/giving-1-million-toward-sf-covid-19-response.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="we-are-giving-1m-toward-san-francisco-covid-19-response">We Are Giving $1M Toward San Francisco COVID-19 Response</h1>

<p class="meta">23 Mar 2020 - San Francisco</p>

<p>COVID-19 represents an unprecedented threat to the health and livelihood of people everywhere, and as we wait for the the US government to act on this emergency, my wife, Theresa, and I have decided to put one million dollars toward our local community’s efforts to prepare for and mitigate the effects of the disease.</p>

<p>We have split the funds equally amoung four organizations, focusing on the well-being of frontline medical workers and food security for vulnerable populations here in the city. The programs we chose are:</p>

<p>UCSF Health - We asked the leadership of the UCSF hospital system what they needed most to help their frontline workers and they told us their most urgent current need is in providing hotel rooms for their medical staff that are worried about going home and infecting their families. We hope this money can provide that form of safety and comfort during these difficult times. <a href="https://giving.ucsf.edu/coronavirus">Give to UCSF</a>.</p>

<p>Zuckerberg SF General Hospital - We talked with the CEO of SF General and will be supporting their efforts in acquiring personal protective equipment (PPE) for their medical staff. We’re also in touch with Ryan Peterson, CEO of Flexport, who is helping coordinate logistics around the purchase and shipping of PPE to SF hospitals in order to help make this happen as quickly as possible. <a href="https://sfghf.org/">Give to SF General</a>.</p>

<p>YMCA - As the parents of three young children, we greatly empathize with the challenges that parents face during this crisis. This is especially true for those critical healthcare workers who we rely on to care for us. With that in mind, we have partnered with the YMCA to provide free childcare (including for ages 0-5 years of age) to frontline workers here in SF, regardless of where they live. <a href="https://www.ymcasf.org/covid-19">Give to YMCA</a>.</p>

<p>Give2SF Fund - In addition to frontline medical workers, we wanted to support food security for some of the most vulnerable populations in SF. Mayor London Breed and the Office of the Mayor have set up the Give2SF Fund to streamline donations for several programs, one of which will provide grocery store gift cards to at-risk residents of SF, including undocumented and mixed status households, low-income households with a
pregnant woman or infant, and older adults and persons who have underlying health conditions who are quarantined and/or who have lost in-home support. <a href="https://give2sf.org/">Give2SF</a>.</p>

<p>These efforts are all local to our home town, but the coronavirus is global. Through <a href="https://prestonwernerventures.com/">Preston-Werner Ventures</a> we are digging into the most effective ways to help abroad as well. Most of our philanthropic dollars go to international causes, especially around community health workers in Africa and Asia, so we hope to leverage that expertise in this crisis.</p>

<p>If you’ve found that the US economy has worked in your favor over the last several years or decades and are able to contribute funds of any amount to these organizations or others, here in SF or anywhere else in the world, I implore you to do so now. Be decisive. Preparation will make a difference in saving lives and protecting those that are risking theirs in our hospitals and other essential workplaces.</p>

<p>Together, we are strong.</p>


guid: http://tom.preston-werner.com/2020/06/15/committing-250k-to-racial-justice-efforts
title: Committing $250k this Year to Racial Justice Efforts
url: http://tom.preston-werner.com/2020/06/15/committing-250k-to-racial-justice-efforts.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="committing-250k-this-year-to-racial-justice-efforts">Committing $250k this Year to Racial Justice Efforts</h1>

<p class="meta">15 June 2020 - San Francisco</p>

<p>On February 23rd, Ahmaud Arbery was murdered by armed white men while jogging near Brunswick, Georgia. On March 13th, Breonna Taylor was murdered by Louisville Metro Police in her own home. On May 25th, George Floyd was murdered by Minneapolis Police officers. These are just three of the names on the list of thousands of Black men and women who lost their lives to the systemic racism that is woven into the fabric of this country.</p>

<p>It will take a monumental anti-racist movement to change the underlying conditions that allow racism in the United States to flourish. As part of that movement, <strong>Theresa and I are committing $250,000 this year</strong> to projects that fight systemic racism. Here’s how we intend the funds to be dispersed:</p>

<p>$50,000 toward efforts to put an end to police brutality and the criminalization system.</p>

<p>$50,000 to support Black political candidates and candidates who are most likely to push forward anti-racist policy change.</p>

<p>$50,000 for Black education and business initiatives.</p>

<p>$100,000 to long-term strategic efforts to create jobs and invest in Black communities.</p>

<p>We will issue this financial support over the next few months. When we finalize commitments to specific projects, we will announce them on the Preston-Werner Ventures Blog.</p>

<p>This is the first of many steps we will be taking in our formalized approach to funding racial justice efforts. We look forward to working with Black leaders and community members to dismantle the racist structures that pervade our society. If you find yourself in a position of privilege, I hope you too will join the movement with your time, money, and effort.</p>


guid: http://tom.preston-werner.com/2022/04/04/redwood-v1-and-funding
title: Announcing RedwoodJS 1.0 and $1M Funding
url: http://tom.preston-werner.com/2022/04/04/redwood-v1-and-funding.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="announcing-redwoodjs-10-and-1m-funding">Announcing RedwoodJS 1.0 and $1M Funding</h1>

<p class="meta">4 Apr 2022 - Bay Area, CA</p>

<p>Three years ago, I had an idea for a new web app framework. Two years ago, we released <a href="https://redwoodjs.com/">RedwoodJS</a> v0.1 to the world. Today, more than 300 talented individuals have lent their ideas and time in crafting our documentation, design, community, marketing, and code. During the journey, we found our mission: to help more startups explore more territory, more quickly.</p>

<p>Over the last six months alone, startups using RedwoodJS have raised over $19M USD in funding. They are the brave, early adopters that helped shape what Redwood has become. Now it’s time to serve everyone. As of this moment, we fully endorse Redwood for production use across a wide range of applications, from simple solo side projects to funded startups.</p>

<p>To celebrate this milestone, I have two big announcements for you.</p>

<p><strong>#1 - Today, we release RedwoodJS 1.0.</strong></p>

<p>As with all things that start with the number one, this is the first major version of many to come. There is so much more that we can do to help startups bring their multi-client apps to fruition. That is our guiding star. To reach our goals, we need to accelerate. Which brings me to the second announcement.</p>

<p><strong>#2 - Today, I am committing $1,000,000 to RedwoodJS development over the next year.</strong></p>

<p>In the past, we’ve talked about whether we should turn RedwoodJS into a company and raise money for it. We certainly could; other players in the space have done so. But we always ask ourselves if that’s what would best serve the Redwood community. When you take venture funding, you sign up for a rocket ship ride that will either take you to the moon or to crash-land painfully back on earth. Those are the only two choices. And both rides tend to require heavy extraction of value from the customer.</p>

<p>Right now, we prefer a third choice: build a sustainable open source project that evolves and grows naturally, without artificial growth hormones. For a difficult-to-directly-monetize framework project like Redwood, it will take us time to find the right model of sustainability that aligns with the community. That’s hard to do without some money, though.</p>

<p>Fortunately, I find myself in a position to solve this problem in an unusual way. I love open source. I love working on RedwoodJS. I love bootstrapping. So my million dollar spend on Redwood development comes with no strings attached, except that we continue to focus on building the best app framework for startups.</p>

<p>I hope you enjoy the Redwood 1.0 release as much as we do. We will be celebrating all week with events centered on developers, founders, partners, and more, all culminating in <strong>another</strong> big announcement on Thursday, April 7th. <a href="https://v1launchweek.redwoodjs.com/">RSVP now to get reminders!</a></p>

<p>If you want to learn more, skim through the newly updated <a href="https://redwoodjs.com/">Redwood Website</a> and spend some time doing the comprehensive <a href="https://redwoodjs.com/docs/tutorial/foreword">Redwood Tutorial</a>.</p>

<p>I hope to see you in the community!</p>

<p><a href="https://community.redwoodjs.com/t/1m-redwoodjs-funding/2960">Continue the conversation on the RedwoodJS Discourse Forum.</a></p>


guid: http://tom.preston-werner.com/2022/04/07/the-redwood-startup-fund
title: Introducing the Redwood Startup Fund
url: http://tom.preston-werner.com/2022/04/07/the-redwood-startup-fund.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="introducing-the-redwood-startup-fund">Introducing the Redwood Startup Fund</h1>

<p class="meta">7 Apr 2022 - Bay Area, CA</p>

<p>Today I’m pleased to announce the Redwood Startup Fund: a $1M fund that will invest $25k–$50k in startups that use <a href="https://redwoodjs.com/">RedwoodJS</a> as a primary component in their stack.</p>

<p>I have several goals in mind with this fund:</p>

<ol>
  <li><strong>Help more startups explore more territory, more quickly.</strong> This is our mission at RedwoodJS. Until now we’ve provided software, documentation, community, connections, advice, and support, but we’ve had to stop short of providing money. Now, for the most promising people and ideas, we can go all the way.</li>
  <li><strong>Increase the diversity of founders in the Redwood community.</strong> Over the last six months, Redwood-based startups have raised more than $19M, but the demographics of those founders skew heavily white and male. I’d like to create more opportunities for Black devs, women devs, and other minoritized devs within our broader community to travel down the startup path.</li>
  <li><strong>Encourage climate-focused software.</strong> Climate change is <em>the</em> existential crisis of our time. We need more ideas and progress in every sector, including software-based approaches to preventing emissions and sequestering carbon.</li>
</ol>

<p>The fund will be part of my angel investing practice at <a href="https://prestonwernerventures.com/">Preston-Werner Ventures</a>, where I’ve made early investments in companies like Stripe, Netlify, Snyk, Prisma, Supabase, PlanetScale, Gitpod, Railway, Snaplet, StackBlitz, FaunaDB, GraphCDN, Retool, and many other dev tools. I also invest heavily in climate technology companies like Beta (electric airplanes), Mote Hydrogen (green hydrogen &amp; permanent carbon sequestration), and Prometheus Fuels (air to carbon-neutral fuel). Though this is a limited set of sectors, the Redwood Startup Fund is intended to invest broadly across industries and I will be engaging others in helping me evaluate potential startups.</p>

<p>I anticipate deploying the fund’s capital over the next 12 months. This will mean a fairly rapid pace of investment (2–4 investments per month) and I’m hoping that some of these checks will go to individuals that would otherwise not have any means to even try their hand at doing a startup. If I am first money in, even before incorporation, I am totally fine with that, and hope to help folks reach the stage where they can raise a pre-seed. Is there such a thing as pre-pre-seed? Maybe we can call it the “existential round”.</p>

<p>I don’t have all the mechanics figured out yet; how to take applications, or what terms on an existential round might be (though, as an entrepreneur myself, I intend them to founder friendly), but hope to finalize that within a month. If you’re interested in knowing more, sign up for <a href="http://redwoodstartupfund.com/">the Redwood Startup Fund mailing list</a>.</p>

<p>For now, dream big and try out the <a href="https://redwoodjs.com/docs/tutorial/chapter1/prerequisites">RedwoodJS Tutorial</a>. I hope to talk with you about your Redwood-based startup soon!</p>

<p>–</p>

<p><a href="https://news.ycombinator.com/item?id=30948063">Discuss this post on Hacker News</a></p>


guid: http://tom.preston-werner.com/2022/05/23/major-version-numbers-are-not-sacred
title: Major Version Numbers are Not Sacred
url: http://tom.preston-werner.com/2022/05/23/major-version-numbers-are-not-sacred.html
feedurl: https://feeds.feedburner.com/tom-preston-werner
content_mime_type: text/html
content:
<h1 id="major-version-numbers-are-not-sacred">Major Version Numbers are Not Sacred</h1>

<p class="meta">23 May 2022 - Positano, Italy</p>

<p>Ten years ago, I sat down and wrote the first version of what became the <a href="https://semver.org/">Semantic Versioning</a> (SemVer) spec. I was tired of everyone using version numbers in whatever way they wanted and knew we could do better if we all agreed on what each part of a version number meant and how those numbers should change as the project’s code changed.</p>

<p>It worked.</p>

<p>Today, SemVer is one of the world’s most well-known and heavily adopted versioning schemes. Several prominent package ecosystems (like Node’s npm and Rust’s Cargo) built-in SemVer’s concepts from day one. npm alone has more than 1.8 million packages, nearly all of which are <a href="https://docs.npmjs.com/about-semantic-versioning">versioned with SemVer</a>.</p>

<p>While it’s unrealistic to expect every package to perfectly adhere to SemVer all the time (mistakes do happen), I’ve seen in practice that the vast majority of package developers are quite careful about version number changes and try to follow SemVer in good faith. This can be hard, though, especially when it comes to breaking changes.</p>

<p>In the time before Semantic Versioning, major version number changes often signified large shifts in the architecture, approach, or broad capabilities of the underlying software. In that world, the minor versions would contain breaking changes, especially for products with large API surface areas. Bumping the major version often corresponded to a marketing push to communicate those improvements to the world. This produced a natural outcome: increasing the major version number was a big deal.</p>

<p>Today, in SemVer ecosystems, it’s <strong>still</strong> a big deal. And I think that’s a problem.</p>

<p>Our collective hesitance to bump the major version of a package is so strong that we sometimes concoct elaborate justifications as to why a breaking change can be included in a minor version release.</p>

<p>“Not many people are using that feature yet; it’ll be fine.”</p>

<p>Or,</p>

<p>“It was clearly intended to be an experimental feature, so we should be able to retroactively mark it as such and then break it in minor, right? Right? RIGHT????”.</p>

<p>Or plainly,</p>

<p>“There isn’t enough to warrant a major version bump; we’re not going to release a major version for some tiny breaking change.”</p>

<p>Either SemVer means something, or it doesn’t. I choose to believe that it does. I’ve seen the benefits to huge package ecosystems and upgrade processes. If we want to live in a world where SemVer improves our collective experience as much as possible, it means we need to believe something new:</p>

<p><strong>Major version numbers are not sacred.</strong></p>

<p>In practice, this means two things:</p>

<ol>
  <li>When you need to release a breaking change—<strong>any</strong> breaking change—you do it in a major version. Period. No excuses.</li>
  <li>You need to find a different way to communicate more significant project-wide changes and improvements. Major version numbers can’t do double duty as both breaking-change indicator <strong>and</strong> marketing hook.</li>
</ol>

<p>The first one should be easy. You just have to internalize that major version numbers are not sacred, you’re not going to run out of them, and it’s your duty as a responsible release manager to always indicate that a breaking change is included within. Of course, just because numbers are free doesn’t mean it’s wise to break your API on every release. Each breaking change means manual code modifications for some users of your package. Break the API too frequently, and you’ll risk fatiguing your users by forcing them to dig through your release notes (you do have release notes, right?) and working through the breaking changes to see if any of them are relevant and need action.</p>

<p>The second one is harder. There have been various proposals over the years, a common one being that SemVer should add a 4th part, perhaps called “epoch,” that is the property of the marketing department. So you’d see <code class="highlighter-rouge">version 2.3.1.0</code> instead of just <code class="highlighter-rouge">version 3.1.0</code>, and the <code class="highlighter-rouge">2</code> indicates that this is the 2nd epochal version corresponding to some marketing event or a larger shift in the library. It’s not a horrible idea, but it does increase the visual complexity of the version number quite a bit and would be challenging to get adopted throughout the various SemVer package ecosystems.</p>

<p>Another idea (and one that doesn’t require SemVer itself to change) is to use a code name or marketing name to associate with a range of major versions. When the name changes, <strong>that</strong> is the marketing event. For instance, Ubuntu is famous for its wacky animal code names like Trusty Tahr and Bionic Beaver. With a bit of extra discipline, it’s not hard to imagine starting with the letter A and naming each “epoch” with a name starting with the next letter in the alphabet, giving users some sense of how many big iterations the software has been through.</p>

<p>These points are not simply an academic exercise for me. In fact, this post is in part an explanation for why we’re about to release <a href="https://redwoodjs.com/">RedwoodJS</a> 2.0 only a bit over a month after releasing 1.0.</p>

<p>Weird, right? See how ingrained it has become to think that major versions should only happen once a year?</p>

<p>So why do we need a 2.0 so soon?</p>

<ol>
  <li>Being a full-stack web framework, RedwoodJS has an extensive API surface area. There is simply a lot of functionality being worked on.</li>
  <li>We need to move quickly and iterate to stay relevant. The web framework space is a constantly moving target, and success will require rapidly improving capabilities and developer experience.</li>
  <li>Not all changes can maintain backward compatibility without incurring unacceptable tech debt or complexity increases in the codebase. There are real development, maintenance, and velocity costs associated with supporting both the old and new ways of doing something. We need to be able to make the call to move forward with new ideas without having to incur these costs or wait many months until a marketing-approved major release can be rolled. The specific breaking change that is prompting 2.0 is <a href="https://github.com/redwoodjs/redwood/pull/5500">this set of changes to the Baremetal deploy strategy</a>.</li>
  <li>We are unwilling to compromise the trust of our users by finding some excuse for shipping breaking changes in a minor release. If you know we’re serious about marking breaking changes with a major release, you can spend less time verifying that our minor releases are safe upgrades.</li>
</ol>

<p>Ok, even if all that’s true, how are we going to reduce the impact of releasing breaking changes so often (perhaps monthly)?</p>

<ol>
  <li>Every minor or major release gets a serious write-up of everything that changed and how to adapt your code to any breaking changes. Even during the pre-1.0 days of RedwoodJS, our team was obsessive about top-notch, comprehensive release notes, and we’ve gotten very good at making them as useful as possible. See the <a href="https://github.com/redwoodjs/redwood/releases/tag/v1.0.0-rc.final.1">RedwoodJS v1.0.0 release notes</a> for an example of what I mean.</li>
  <li>On top of the release notes, whenever possible, we provide executable code mods to automatically upgrade your code to work with the new release! This is a non-trivial undertaking, but by taking the pain out of upgrades, we believe we can do more breaking changes with less impact, leading to a better framework and improved DX more quickly. The Go programming language famously provided code mods to reduce the impact of their frequent breaking changes during the initial development of the language, giving them free license to make sweeping changes without having to support every syntactical mistake they ever made.</li>
  <li>I actually prefer more frequent major versions with smaller sets of breaking changes. There is a real danger in saving up all your significant changes and releasing them in one whopping major version once a year. In the early days of GitHub, when we would need to upgrade Ruby on Rails to the latest major version, it would often take <strong>weeks or months</strong> to accomplish, even with a dedicated developer on the task. Once, because of how extensive the changes needed to be, the upgrade branch started to deviate so much from the main branch that it became too unwieldy to merge, and we had to throw it away and start over with a different approach. Spreading upgrade work out throughout the year would have saved us a lot of time and anguish.</li>
</ol>

<p>In an effort to drink my own philosophical champagne, I am excited to very soon deliver to you <strong>RedwoodJS 2.0</strong>, part of the <strong>Arapaho Forest</strong> epoch. Yep, you guessed it, each epochal version of Redwood will be named after a <a href="https://en.wikipedia.org/wiki/List_of_national_forests_of_the_United_States">US national forest</a>, and when we announce a new epochal version (that starts with a B), you can be sure you’ll know about it. In the meantime, you’ll see a number of major versions of Redwood (all with the Arapaho Forest epoch name) without a ton of fanfare surrounding the releases.</p>

<p>I want to live in a world where every breaking change comes gift-wrapped in a major release. I want the promise of SemVer to be fully realized. I think we can get there by rejecting the tyranny of sacred major version numbers. If you feel the same, I hope you’ll join me in embracing this philosophy.</p>

<p><a href="https://news.ycombinator.com/item?id=31483695">Discuss this post on Hacker News</a></p>


guid: 0a687f2bf4a6881e1d397f5a267372bc82a12d96
title: Who is listening on my ports
url: https://unixsheikh.com/tutorials/who-is-listening-on-my-ports.html
feedurl: https://unixsheikh.com/feed.rss
content:
This is a mini tutorial in how to figure out what applications are listening on your ports on GNU/Linux, OpenBSD, and FreeBSD.


guid: f1659a1d0922496637ec4ecc6c2a3bd7dd0ec203
title: Revision control
url: https://unixsheikh.com/articles/revision-control.html
feedurl: https://unixsheikh.com/feed.rss
content:
In connection with some corrupted files in CVS I decided that it was time to take a look on some other revision control systems. This article is a summary of my three day "journey" into the land of revision controlling.


guid: f38853a8f3e51ab42e23c439e45ac35837dae5d8
title: Choosing a good book on programming
url: https://unixsheikh.com/articles/choosing-a-good-book-on-programming.html
feedurl: https://unixsheikh.com/feed.rss
content:
This is a small article on how to choose a good book on programming. The same method can be applied to other areas as well. It should be noted that the described method is my preferred method, it is not a universal method.


guid: 75ac53c340d88d6b61d992f029a73bda0c9e1cad
title: Programming in C, Cpp, and Ada
url: https://unixsheikh.com/articles/programming-in-c-cpp-and-ada.html
feedurl: https://unixsheikh.com/feed.rss
content:
One day back in 2007 I was dribbling down some notes about programming languages and compilers with a special focus on the use of C, C++ and Ada. These are some of my dribbles.


guid: 13ec6bc9cdbe77a3a431a20f81331d720c2e59ad
title: Pointers in C
url: https://unixsheikh.com/tutorials/pointers-in-c.html
feedurl: https://unixsheikh.com/feed.rss
content:
This is tutorial on how pointers work in the C programming language. The tutorial is addressed at beginners. Rather than focusing a lot on the more technical aspects of pointers, and using complex examples, the tutorial focuses on a deeper understanding of the subject so that the issue is understood firmly. The tutorial assumes prior knowledge of C, such as assigning values to variables, compiling, printing to the screen and using comments, but it is not necessary.


guid: ca7c9eddeb4e75fa3f705ce37bca818559c26a73
title: Speed down your CPU
url: https://unixsheikh.com/tutorials/speed-down-your-cpu.html
feedurl: https://unixsheikh.com/feed.rss
content:
Getting the CPU to slow down using a lower frequency when not used can save you power and noise. Note, this is automatically setup on laptops on Debian stable aka (currently) Lenny during installation.


guid: 2b8ff311902d6cc51381af9e289ee6fab9bddc52
title: Choosing a new backup solution, Duplicity, rdiff-backup or Rsnapshot
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/tutorials/choosing-a-new-backup-solution-duplicity-rdiff-backup-or-rsnapshot.html

content:
Currently I am using Mercurial revision control system to version control my programming files and written articles etc. Other important files I manually tar up and encrypt and upload to my backup server using sftp from OpenSSH. I decided that it was time to take a look at a better way to deal with my personal backups.


guid: ca68a5fd080200863238910c6225ecb24973e413
title: Using EncFS to encrypt some of your files
url: https://unixsheikh.com/tutorials/using-encfs-to-encrypt-some-of-your-files.html
feedurl: https://unixsheikh.com/feed.rss
content:
EncFS provides an encrypted filesystem in user-space. It runs without any special permissions and uses the FUSE library and Linux kernel module to provide the filesystem interface.


guid: 483b1abc62d198d46ae410389377b239b1696dd6
title: Ubuntu as in Debian?
url: https://unixsheikh.com/articles/ubuntu-as-in-debian.html
feedurl: https://unixsheikh.com/feed.rss
content:
Ubuntu GNU/Linux has grown to become the most famous Linux distribution in the world. It is the Linux distribution on DistroWatch that currently has the greatest hit rate, and even though it is based completely upon Debian GNU/Linux, it has almost twice as many hits. This articles explains what Ubuntu is and what relation it has got with Debian.


guid: 5b6e7c7d7d10c92dfc6463911c44ac5afbbb0811
title: Some notes on Arch Linux
url: https://unixsheikh.com/articles/some-notes-on-arch-linux.html
feedurl: https://unixsheikh.com/feed.rss
content:
I have been using GNU/Linux, FreeBSD and OpenBSD exclusively since about 1999 and I spend time with Linux and BSD every single day both at home and at work. From time to time I still like to test some of the other Linux distributions to see whats new in the world of Linux. This time I decided to test Arch Linux. Arch Linux is really great, but it is not for people wanting a stable desktop or server system.


guid: 4669397e849fa275fceb45b8520cc94c7fc028ed
title: PHP Security
url: https://unixsheikh.com/articles/php-security.html
feedurl: https://unixsheikh.com/feed.rss
content:
PHP web applications are one of the most commonly attacked pieces of software on the Internet today. Anyone who has looked at their web server logs can attest to the frequency of probes for vulnerable PHP applications. PHP's easy learning curve has lead to its popularity and breadth of applications, but not without some hard learned lessons on the way. This document serves as a reminder of some of the important security related issues when programming in PHP. The paper is not a security manual. The paper is just a collection of notes. If you are writing PHP applications I strongly suggest that you research the subject in depth.


guid: c82c9503839fe8d0701a1aaeeeaa74b08fd99c9c
title: Remove -- MARK -- from the message log
url: https://unixsheikh.com/tutorials/remove-mark-from-the-message-log.html
feedurl: https://unixsheikh.com/feed.rss
content:
This small tutorial describes how you can get rid of the `-- MARK --` message on Debian GNU/Linux in the message log.


guid: 27fdecd4ef8b2f272185d47f828ec2531e4f54f6
title: Encrypting and decrypting files using GnuPG, OpenSSL or Mcrypt
url: https://unixsheikh.com/tutorials/encrypting-and-decrypting-files-using-gnupg-openssl-or-mcrypt.html
feedurl: https://unixsheikh.com/feed.rss
content:
How to encrypt and decrypt a file with GnuPG, OpenSSL or LibreSSL, or Mcrypt using the Advanced Encryption Standard and a password phrase.


guid: 717d60b86858d668f9ed1df15c2c9505d2b85e6a
title: Why I do not like derived Linux distributions
url: https://unixsheikh.com/articles/why-i-do-not-like-derived-linux-distributions.html
feedurl: https://unixsheikh.com/feed.rss
content:
In the world of Linux (also called GNU/Linux) there currently exist over 600 distributions, approximately 300 of those are in active development, constantly being revised and improved, and while most of them look great, only a few of them are actually worth paying attention to IMHO.


guid: 6a66c2f64909802a6539ce4b4dd657f4d82b4222
title: PHP include path problems
url: https://unixsheikh.com/tutorials/php-include-path-problems.html
feedurl: https://unixsheikh.com/feed.rss
content:
Working with include paths in PHP can sometimes provide quite a challenge. When everything resides in the document root there are no problems, but once the files are moved into one or more subdirectories difficulty may arise. This article uses Unix paths, not Microsoft Windows paths.


guid: 9b370d10c675851540ebe7018a60ba03996fb513
title: Some personal notes on revision control and backup
url: https://unixsheikh.com/articles/some-personal-notes-on-revision-control-and-backup.html
feedurl: https://unixsheikh.com/feed.rss
content:
This document is based upon some simple notes to myself. I have actually re-writtin this document a couple of times - each time I changed my directory structure, or considered another way to implement some kind of backup solution. Backup may be as simple as a copy or archive-file of a tree of files, or may take advantage of recording small differences since a previous backup, to make efficient use of storage space. This document is about my personal preferences with regard to backup solutions, revision control system, and directory structure.


guid: 9ee37a63bd715b93e52621edad2ced827583c406
title: Mercurial branching, main and stable, a possible workflow
url: https://unixsheikh.com/tutorials/mercurial-branching-main-and-stable-a-possible-workflow.html
feedurl: https://unixsheikh.com/feed.rss
content:
Software projects, whether small or big, often needs to go trough periodic official releases. Often you need to release a stable branch of a project while at the same time development continues on the main branch. This small tutorial is about how this can be done using the Mercurial revision control system.


guid: a33d2160a1d8f5c8295b9d2c6a817f543292f803
title: How to convert a man page into PS, PDF or DVI
url: https://unixsheikh.com/tutorials/how-to-convert-a-man-page-into-ps-pdf-or-dvi.html
feedurl: https://unixsheikh.com/feed.rss
content:
Man pages (short for "manual pages") are the extensive documentation that comes preinstalled with almost all substantial Unix-like operating systems. The Unix command used to display them is `man`. Each page is a self-contained document. This how-to is about how to convert a man page into a PS or PDF document.


guid: 51e8fbfdfdd7af899d7232f1c30a1194e5615bc9
title: umount - device is busy
url: https://unixsheikh.com/tutorials/umount-device-is-busy.html
feedurl: https://unixsheikh.com/feed.rss
content:
Have you ever tried to unmount a partition only to get the message "device is busy"? Well, there's a reason for that and here's what you can do.


guid: 456aad76f803463fe65988fdac7402ed170538cb
title: Git bare vs. non-bare repositories
url: https://unixsheikh.com/tutorials/git-bare-vs-non-bare-repositories.html
feedurl: https://unixsheikh.com/feed.rss
content:
The Git revision control system has something called a "bare" and a "non-bare" repository. This tutorial deals with the issue and also compares the Git design to the design of Mercurial and Bazaar.


guid: 735355885d500f86d6c2e3bf97ed4aba8d655ebc
title: Code Navigation, Design Patterns, and Too Many Functions
url: https://unixsheikh.com/articles/code-navigation-design-patterns-and-too-many-functions.html
feedurl: https://unixsheikh.com/feed.rss
content:
When going into extremes in using a specific programming style or a specific design pattern the result is always bad code - even if it looks nice.


guid: 35d0373d3b1461ccc3ac3bad8ddb02e7faf3544a
title: Code comments are good right? Well, not if there are too many!
url: https://unixsheikh.com/articles/too-many-comments.html
feedurl: https://unixsheikh.com/feed.rss
content:
Too much commenting makes the code difficult to navigate. Experienced programmers try to create code that is self-explaining. Good code is almost always self-explaining.


guid: e61563a4b549446a0efc9e9a86165aabfcda25d3
title: The Book "Pro PHP Refactoring"
url: https://unixsheikh.com/articles/the-book-pro-php-refactoring.html
feedurl: https://unixsheikh.com/feed.rss
content:
I just finished reading the book Pro PHP Refactoring by Franvesco Trucchia and Jacopo Romei. The book itself should be refactored - it's not pro at all. The book is an attempt to take the guidelines from the famous "Refactoring, Improving the Design of Existing Code" by Martin Fowler and turn them into PHP refactoring principles.


guid: b6bea31f0512f35b3fab3cdf8b6a397199e9ca3a
title: Benchmark 1 - Dynamic pages vs. static pages vs. Memcached
url: https://unixsheikh.com/articles/benchmark-1-dynamic-pages-vs-static-pages-vs-memcached.html
feedurl: https://unixsheikh.com/feed.rss
content:
This is a benchmark test of the loading speed of dynamic pages vs. static pages vs. pages stored in Memchached using PHP. The test was performed on an Intel P4 1.7 Ghz with 1036028 kB of memory over a privat LAN. The server runs MySQL 5.0.51a and PHP 5.3.3 on a standard Debian installation. Between each change of settings all related processes was restarted.


guid: 6279d5b3be9b9cc0ffdd8da1b2c07da9b11a7baf
title: Benchmark 2 - MySQL Cache
url: https://unixsheikh.com/articles/benchmark-2-mysql-cache.html
feedurl: https://unixsheikh.com/feed.rss
content:
This is a benchmark test of the performance of MySQL Cache. The test was performed on an Intel P4 1.7 Ghz with 1036028 kB of memory over a private LAN. The server runs MySQL 5.0.51a and PHP 5.3.3 on a standard Debian installation. Between each change of settings all related processes was restarted. Since Debian Squeeze the MySQL Cache is enabled by default.


guid: 1e7f002657c1f97cf3c4a4fa00862b3359e850d3
title: Benchmark 3 - The Alternative PHP Cache (APC)
url: https://unixsheikh.com/articles/benchmark-3-the-alternative-php-cache-apc.html
feedurl: https://unixsheikh.com/feed.rss
content:
This is a benchmark test of the performance of PHP with and without APC (Alternative PHP Cache). The test was performed on an Intel P4 1.7 Ghz with 1036028 kB of memory over a private LAN. The server runs PHP 5.3.3 on a standard Debian installation. Between each change of settings all related processes was restarted.


guid: f6638462258b932a39ab657a6ec7740f068497fb
title: GnuPG Tutorial
url: https://unixsheikh.com/tutorials/gnupg-tutorial.html
feedurl: https://unixsheikh.com/feed.rss
content:
This is a minor tutorial in the usage of GnuPG to encrypt, decrypt, and sign emails and files. The examples provided in this tutorial has been tested on Debian GNU/Linux, Devuan GNU/Linux, Arch Linux and Ubuntu GNU/Linux.


guid: b117366f52d83e8acd9beda55e8256d51dfaebda
title: How to set your hostname and domain name correctly
url: https://unixsheikh.com/tutorials/how-to-set-your-hostname-and-domain-name-correctly.html
feedurl: https://unixsheikh.com/feed.rss
content:
This tutorial describes how to set your hostname, domain name and fully qualified domain name properly. Both a mail server and a web server needs to run with the right hostname and domain name.


guid: 72b73532356387729423e092a7fcca07a4a7e0c7
title: An alternative way to play YouTube videos without using Flash
url: https://unixsheikh.com/tutorials/an-alternative-way-to-play-youtube-videos-without-using-flash.html
feedurl: https://unixsheikh.com/feed.rss
content:
This tutorial was written when it was only possible to view videos on YouTube via Flash, however the tutorial is still useful if you need to download videos from YouTube, or some other video service, in an easily manner and watch them on your favorite video player.


guid: a4453886c99101c7b978771aebad80ce2f75f8d0
title: PSR-0 is not a solution it is a bypass!
url: https://unixsheikh.com/articles/psr-0-is-not-a-solution-it-is-a-bypass.html
feedurl: https://unixsheikh.com/feed.rss
content:
This is a small article that takes an objective look at the PSR-0 autoloader "standard" implemented by PHP-FIG.


guid: 82c553a9439c31a9338dac33ba6c6ed4cd5a83a4
title: Some notes on Arch Linux - 2
url: https://unixsheikh.com/articles/some-notes-on-arch-linux-2.html
feedurl: https://unixsheikh.com/feed.rss
content:
I have previously written a small article called "Some notes on Arch Linux", first in 2008, then revisited in 2009, approximately a year later. At the time of this writing (2013) I haven't touched Arch Linux in four years. I then decided to re-test it and write some notes. In the previous notes I compared Arch Linux to Debian GNU/Linux (and to some extend FreeBSD and OpenBSD), I will again compare it to Debian GNU/Linux, but will not mention the BSD flavors. In these notes I will not address the problems about Arch Linux mirrors not being in sync. The problems however remains.


guid: fa5017c5d8f19ee0a91e12ecb234be863087254a
title: Design Patterns Can Be Very Evil
url: https://unixsheikh.com/articles/design-patterns-can-be-very-evil.html
feedurl: https://unixsheikh.com/feed.rss
content:
"Design Patterns - Elements of Reusable Object Oriented Software", is a software engineering book which describes recurring solutions to common problems in software design. The book has been very influential to the field of software engineering and it is viewed upon as an important source for object-oriented design theory and practice. The authors are often referred to as the Gang of Four (GoF).


guid: 82692122a2aa3137d3ac2df83f97413c35984e54
title: No your API isn't REST, but don't worry, it really shouldn't be!
url: https://unixsheikh.com/articles/no-your-api-isnt-rest.html
feedurl: https://unixsheikh.com/feed.rss
content:
In 2008 Roy T. Fielding expressed his frustration with people calling their HTTP-based interfaces for REST API's when in fact they aren't. Today in 2015 nothing has changed, people are still calling their HTTP-based interfaces a REST API, even though they are not. But that's not the worst part of all of this, the worst part is that no HTTP driven API should try to be REST in the first place!


guid: c715f739683343fb8fb4a6423933f17452a26c2d
title: To framework or not to framework?
url: https://unixsheikh.com/articles/to-framework-or-not-to-framework.html
feedurl: https://unixsheikh.com/feed.rss
content:
What level of complexity do you want to be at? Are you really thinking about and understanding what you're doing or are you just following "trends" and always using a framework?!


guid: dec6cc2c8bbaf93593e8c49e6ec6508accfcde4b
title: Composer and Packagist - a bad implementation!
url: https://unixsheikh.com/articles/composer-packagist-a-bad-implementation.html
feedurl: https://unixsheikh.com/feed.rss
content:
On many websites, in a huge amount of Github projects, and in several books, like the book 'Modern PHP, New Features and Good Practices' by John Lockhart, so-called 'modern PHP' is made synonymous to using Composer and Packagist in combination with the usage of the recommendations provided by the PHP-FIG. Lots of people follow these advices almost blindly, but the reality is that this is NOT good practice from a security stand point.


guid: 67336240cc36aea98bd694d3b128c6127c271032
title: Bummer PHP
url: https://unixsheikh.com/articles/bummer-php.html
feedurl: https://unixsheikh.com/feed.rss
content:
Since the release of the PHP programming language, PHP has been both loved and hated by many. In this article we'll take a look at one of the reasons why PHP really is such "a bummer".


guid: 3fe95007a067e35a8f91e3529887e5c1f58efa53
title: Programming languages and programming paradigms
url: https://unixsheikh.com/articles/programming-languages-and-programming-paradigms.html
feedurl: https://unixsheikh.com/feed.rss
content:
The Internet is filled with articles, blog posts, and forum debates about which programming language or which programming paradigm that is better than another. Python vs. Java vs. PHP vs. Ruby. Procedural vs. object oriented, object oriented vs functional, etc. In this article we'll take a much deeper look at programming languages and programming paradigms and try to understand these concepts from a different angle.


guid: 44af4117c46f1e97a4a44350300c04a32647fc44
title: How to split your DNS requests when using a VPN
url: https://unixsheikh.com/tutorials/how-to-split-your-dns-requests-when-using-vpn.html
feedurl: https://unixsheikh.com/feed.rss
content:
This tutorial describes how you can split your DNS requests between a local DNS resolver and remote DNS resolvers in order to prevent a DNS leak when you use a VPN connection.


guid: 473f591356d478c14786bbec6e5d0825b3701a25
title: No your PHP framework isn't MVC, but don't worry, it really shouldn't be!
url: https://unixsheikh.com/articles/no-your-web-application-isnt-mvc.html
feedurl: https://unixsheikh.com/feed.rss
content:
Most of the popular PHP frameworks postulate that they implement the model-view-controller (MVC) pattern, but it's just not true.


guid: edc2870752ce9a6da15f647812fb881e93499ef8
title: Real full disk encryption using GRUB on Arch Linux for BIOS and UEFI
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/tutorials/real-full-disk-encryption-using-grub-on-arch-linux-for-bios-and-uefi.html

content:
In this tutorial we're gonna take a look at setting up full disk encryption on Arch Linux using GRUB for both a BIOS/MBR based setup and a UEFI based setup. While the choice to install in UEFI mode is encouraging, early vendor UEFI implementations carry more bugs than their BIOS/MBR counterparts. You are advised to do a search relating to your particular motherboard model before proceeding. Contrary to "modern" advice I haven't personally found any compelling reason to use UEFI.


guid: 755e7898d3ffe0bff014771e710ecc269ab59d06
title: Real full disk encryption using GRUB on Debian GNU/Linux for BIOS
url: https://unixsheikh.com/tutorials/real-full-disk-encryption-using-grub-on-debian-linux-for-bios.html
feedurl: https://unixsheikh.com/feed.rss
content:
In this tutorial we're gonna take a look at setting up full disk encryption on a BIOS MBR based system using GRUB on Debian GNU/Linux - the KISS way.


guid: 15e996db274bb0bd9a95e1b205df60fbf6e7cdb9
title: The typical discussions about BSD vs Linux
url: https://unixsheikh.com/articles/the-typical-discussions-about-bsd-vs-linux.html
feedurl: https://unixsheikh.com/feed.rss
content:
In this article we're going to take a little look at the typical discussions about the differences between popular operating systems such as BSD and Linux.


guid: 54acbc6d70678d457f6e03c24160cfff2f14ca4e
title: Make the current commit the initial commit in a Git repository
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/tutorials/make-the-current-commit-the-initial-commit-in-a-git-repository.html

content:
In this tutorial we're gonna take a look at how you can make the current commit in your Git repository the initial and only commit. This is useful if you're using Git as a backup solution and you want to clean out old files. It also works on GitHub.


guid: 2a75bf07f4f2b8b9b5b40950609920a9841d4de3
title: Is Google evil?
url: https://unixsheikh.com/articles/is-google-evil.html
feedurl: https://unixsheikh.com/feed.rss
content:
In this article we're going to take a hard look at the question about Google's "evilness" and the general condition of the Internet.


guid: 40a584fe615a5d4b8768ed58234dd0b476d3b174
title: Stay away from CloudFlare
url: https://unixsheikh.com/articles/stay-away-from-cloudflare.html
feedurl: https://unixsheikh.com/feed.rss
content:
CloudFlare has become one of the fastest growing DNS and CDN hosters on the Internet delivering some of the best DNS speeds worldwide, but if you value customers or visitors to your website you maybe need reconsider using CloudFlare.


guid: ad7155a202ddf03a10c44be1f169145b0935aee9
title: Arch Linux mail server tutorial - part 1 - What is email?
url: https://unixsheikh.com/tutorials/arch-linux-mail-server-tutorial-part-1-what-is-email.html
feedurl: https://unixsheikh.com/feed.rss
content:
In this tutorial we're going to setup Arch Linux as a mail server for virtual users using openSMTPD and Dovecot. The setup I am going to use is a simple setup for home usage or office usage without webmail or database management of virtual users (though you can easily add that). Instead the users are managed using a simple flat file and they connect to the server using an email client. The setup is very easy to manage and it is very well suited for home usage, if you want to manage your own email, and perhaps friends and family email as well.


guid: 4ba1cbc772d617c9f856ecc7752bb6f06c285276
title: Arch Linux mail server tutorial - part 2 - OpenSMTPD, Dovecot, DKIMproxy, and Let's Encrypt
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/tutorials/arch-linux-mail-server-tutorial-part-2-opensmtpd-dovecot-dkimproxy-and-lets-encrypt.html

content:
This is the second part of the mail server tutorial in which we'll install and setup OpenSMTPD, Dovecot, DKIMproxy, and Let's Encrypt.


guid: f63318d7d14c54dcb55cd0e18d8d7bf4bd31daae
title: Arch Linux mail server tutorial - part 3 - Get DNS right, it's important!
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/tutorials/arch-linux-mail-server-tutorial-part-3-get-dns-right-it-is-important.html

content:
In this part of the tutorial we're going to take a look at the important parts of setting up DNS in order to run a mail server.


guid: 4edd08dd394f3bc36aaf4675e9fa498dc39d7e5e
title: OpenBSD is fantastic
url: https://unixsheikh.com/articles/openbsd-is-fantastic.html
feedurl: https://unixsheikh.com/feed.rss
content:
I have been using OpenBSD, a FREE, multi-platform 4.4BSD-based Unix-like operating system, both professionally and privately since about 2004, and today I'm going to share some of my experiences.


guid: 11b4727efe86c711565a91b79ebe6f3763acb469
title: Package management in OpenBSD
url: https://unixsheikh.com/tutorials/package-management-in-openbsd.html
feedurl: https://unixsheikh.com/feed.rss
content:
If you have experience running any of the popular Linux distributions like Debian GNU/Linux, Arch Linux, Fedora, OpenSUSE, etc., and are trying out OpenBSD, one of the easiest things to get confused about is the package management, it's a bit different on OpenBSD.


guid: eb5fe1c90d797243efce9ca1b8107caffb1cb9a5
title: Real full disk encryption using GRUB on Void Linux for BIOS
url: https://unixsheikh.com/tutorials/real-full-disk-encryption-using-grub-on-void-linux-for-bios.html
feedurl: https://unixsheikh.com/feed.rss
content:
In this tutorial we're going to take a look at setting up full disk encryption on the Void Linux base system using GRUB for a BIOS/MBR based setup. While the choice to install in UEFI mode is encouraging, vendor UEFI implementations still carry more bugs than their BIOS/MBR counterparts. Contrary to "modern" advice I still haven't found any compelling reason to use UEFI yet.


guid: 47a5e9475a8ba3bef4ba7268e7579951d8eb94d6
title: Void Linux - a great and unique Linux distribution
url: https://unixsheikh.com/articles/void-linux-a-great-and-unique-linux-distribution.html
feedurl: https://unixsheikh.com/feed.rss
content:
Void Linux is a great rolling release and independent Linux distribution that uses the X Binary Package System (XBPS) package manager, which was designed and implemented from scratch, and the runit init system. Excluding binary kernel blobs, a base install is composed entirely of free software, but users can access an official non-free repository to install proprietary software.


guid: d5121282edb7616364cbed7b927e86c9d7b0d7e1
title: The real motivation behind systemd
url: https://unixsheikh.com/articles/the-real-motivation-behind-systemd.html
feedurl: https://unixsheikh.com/feed.rss
content:
In this article we're going to take a close look at the real motivation behind the development of systemd, and we're going to take a look at some of the future perspectives for GNU/Linux as an operating system.


guid: 6728b15e885932e094977284a914b44f273024ce
title: Reviving Cubox with Arch Linux ARM
url: https://unixsheikh.com/tutorials/reviving-cubox-with-arch-linux-arm.html
feedurl: https://unixsheikh.com/feed.rss
content:
I decided it was time to try and revive my old Cubox that has been lying around unused for a very long time. Back then it was running a custom Armbian (Debian for ARM) release from SolidRun, but SolidRun have long since abandoned the old Cubox and deleted all content from their servers. In this tutorial I'll share how I got the Cubox up and running again with the latest Linux kernel. It is currently working as a small backup server running two USB harddrives in a Btrfs mirror.


guid: c32a4e81ad4b808ae3f1f2936fa616b5d7d0ae43
title: Installing Arch Linux with ZFS on a USB stick
url: https://unixsheikh.com/tutorials/installing-arch-linux-with-zfs-on-a-usb-stick.html
feedurl: https://unixsheikh.com/feed.rss
content:
In this small tutorial we're going to install Arch Linux with ZFS on a USB stick. This can be useful if you're running a homelab NAS and you want to boot of the USB stick and only use your drives for ZFS. It can also be used as a diagnostic or administration tool for dealing with ZFS.


guid: 2bbe66193b378e0c197cac1d1e64bc4f45eca7ff
title: Battle testing ZFS, Btrfs and mdadm+dm-integrity
url: https://unixsheikh.com/articles/battle-testing-zfs-btrfs-and-mdadm-dm.html
feedurl: https://unixsheikh.com/feed.rss
content:
In this article I share the results of a home-lab experiment in which I threw some different problems at ZFS, Btrfs and mdadm+dm-integrity in a RAID-5 setup.


guid: 6e226642422d84f57ca4c55aa8c74e3f18acf1c7
title: When abstraction, design patterns, and principles leads to spaghetti code
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/articles/when-abstraction-design-patterns-and-principles-leads-to-spaghetti-code.html

content:
If you are thinking about abstraction, design patterns, and programming principles while you are programming, then you are not really programming. Rather you are wasting time fantasizing, not solving problems. The end goal is always to have good working code, not code that has been bent, twisted, and shoved into formations and structures in order to fulfill the vanity of someone living in a fairy tail.


guid: 3f98ae85a6112b3396aa3535436876082ed02ca5
title: Stop pushing JavaScript!
url: https://unixsheikh.com/articles/stop-pushing-javascript.html
feedurl: https://unixsheikh.com/feed.rss
content:
Frontend developers are pushing JavaScript as though the web could not function without it. The fact is that JavaScript is one of the top reasons for security breaches on the client computer and client mobile phones, it is a plague that turns privacy into a public exhibition, and it has paralyzed the industry in such a way that you can hardly find a website that doesn't display "the white screen of death" if JavaScript has been disabled in the browser.


guid: c92d990ff86ca1feb22df6429bd3c4fa6d66efdf
title: Go (Golang) - Logging that matters
url: https://unixsheikh.com/articles/go-logging-that-matters.html
feedurl: https://unixsheikh.com/feed.rss
content:
Some people complain about the logging capabilities of the Go standard library log package, but I not only find it adequate, I really love it.


guid: 48d586aaa2d3fa4a2fab4bd1168bc2057cd4e2b1
title: Go (Golang) - Errors and panics
url: https://unixsheikh.com/articles/go-errors-and-panics.html
feedurl: https://unixsheikh.com/feed.rss
content:
In Go you have to deal with both errors and panics and while some people consider it one of the worst things about Go, I think it demonstrates exactly how well designed and carefully crafted Go is.


guid: 6ead943c53c28db11412d6a6d57f057e613a3f2e
title: Go (Golang) - understanding the object oriented features with structs, methods, and interfaces
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/articles/go-understanding-the-object-oriented-features-with-structs-methods-and-interfaces.html

content:
Object-oriented programming is all about organizing of code. It is an extension of procedural programming, and it is about avoiding a global scope and about sharing resources between functions. Let's take a look at some of the object-oriented features of Go.


guid: 55c951b5da8e931087ed08440d1796a8e52e5c18
title: My favorite note-taking method
url: https://unixsheikh.com/articles/my-favorite-note-taking-method.html
feedurl: https://unixsheikh.com/feed.rss
content:
On Hacker News, Reddit, and elsewhere, people often ask about favorite note-taking methods. In this post I share mine.


guid: 5ec82c1e8e56788e679da1f5658bbe0d68fbc379
title: When the modern approach is nothing but hype
url: https://unixsheikh.com/articles/when-the-modern-approach-is-nothing-but-hype.html
feedurl: https://unixsheikh.com/feed.rss
content:
Usually we regard the phrase "the modern approach" as something that denote improvement over time, something that has changed to the better. However, the phrase "the modern approach" are often abused and we must be careful to separate fact from hype.


guid: ea004ba2cce51c64bfdda053e1c65f98760efe1f
title: Job interviews in web development gone haywire
url: https://unixsheikh.com/articles/job-interviews-in-web-development-gone-haywire.html
feedurl: https://unixsheikh.com/feed.rss
content:
A bad trend is becoming a standard at more and more tech companies when interviewing candidates for job positions as more and more people report ridiculous code testing and absurd tech questions.


guid: df51e25cfaf57555bd6a92743c2db08e7725ce0c
title: Why you should migrate everything from Linux to BSD
url: https://unixsheikh.com/articles/why-you-should-migrate-everything-from-linux-to-bsd.html
feedurl: https://unixsheikh.com/feed.rss
content:
As an operating system GNU/Linux has become a mess because of the fragmented nature of the project, the bloatware in the kernel, but mainly because of the manipulation by corporate interests. There exist several technical reasons for when a migration from GNU/Linux to BSD make sense, but this article isn't about that, it's an "analyzes" of the current status in Linux-land, and it is an opinionated rant, more than anything else.


guid: adb76c53d72051ba5792bd94aa24531559db9359
title: FreeBSD is an amazing operating system
url: https://unixsheikh.com/articles/freebsd-is-an-amazing-operating-system.html
feedurl: https://unixsheikh.com/feed.rss
content:
This is a small article about some of my adventures with FreeBSD and why I believe it is an amazing operating system.


guid: 64c3acc25bae28fd731d95519274070d9f45d9fe
title: Why you should migrate everything from Linux to BSD - part 2
url: https://unixsheikh.com/articles/why-you-should-migrate-everything-from-linux-to-bsd-part-2.html
feedurl: https://unixsheikh.com/feed.rss
content:
After I posted the first part of this article it got posted on Hacker News, Reddit and Lobster, and someone made a response. This second part of the article is my response to that.


guid: 3e933553fbd411eff5db5c00a788b1d37497b37e
title: How to setup FreeBSD with a riced desktop - part 1 - Basic setup
url: https://unixsheikh.com/tutorials/how-to-setup-freebsd-with-a-riced-desktop-part-1-basic-setup.html
feedurl: https://unixsheikh.com/feed.rss
content:
In this three part tutorial I am going to show you how you can setup FreeBSD with a riced Xfce or i3 desktop.


guid: a023dd1289b6e54b0db05e507f2b7f022d9a1a3b
title: How to setup FreeBSD with a riced desktop - part 2 - Xfce
url: https://unixsheikh.com/tutorials/how-to-setup-freebsd-with-a-riced-desktop-part-2-xfce.html
feedurl: https://unixsheikh.com/feed.rss
content:
This is the second part of the tutorial. In this part we're going to install Xfce and rice it a bit. You can look further down in the tutorial at the screenshots in order to determine if you like the results.


guid: dda51ae0357c83e0af2ea7e6f4b772e0b22ef3d5
title: How to setup FreeBSD with a riced desktop - part 3 - i3
url: https://unixsheikh.com/tutorials/how-to-setup-freebsd-with-a-riced-desktop-part-3-i3.html
feedurl: https://unixsheikh.com/feed.rss
content:
In this part we're going to install i3 and rice it bit. You can look further down in the tutorial at the screenshots in order to determine if you like the results.


guid: 158b060616b56d441e42cc1b4a240460a0722d5e
title: Samba and NFS performance - ZFS on Linux vs ZFS on FreeBSD
url: https://unixsheikh.com/articles/samba-and-nfs-performance-zfs-on-linux-vs-zfs-on-freebsd.html
feedurl: https://unixsheikh.com/feed.rss
content:
I am currently in the process of migrating some ZFS storage servers from GNU/Linux Debian to FreeBSD and decided to do some simple performance tests.


guid: ff35d3e12c44f0312bb1901d1bae202179a79fc1
title: systemd isn't safe to run anywhere
url: https://unixsheikh.com/articles/systemd-isnt-safe-to-run-anywhere.html
feedurl: https://unixsheikh.com/feed.rss
content:
Since the release of systemd in 2010 the project has been getting a continuous and steady stream of new features and added capabilities. With a code count of more than 1.3 million lines of code, where Lennart Poettering has just added yet another 20.000 new lines of code with the merge of his personal systemd-homed git tree into systemd, and with a continuous open issue counter at about 1.400 issues, where new issues and bugs keep popping up, systemd should be considered experimental and not safe to run anywhere.


guid: 48db1f155486889023b50900258bf3cc4351fc0b
title: Choosing between OpenBSD and FreeBSD
url: https://unixsheikh.com/articles/choosing-between-openbsd-and-freebsd.html
feedurl: https://unixsheikh.com/feed.rss
content:
One of the questions I frequently run into is how to choose between OpenBSD and FreeBSD. In this small article I'll try to answer that question.


guid: a4371eba63f8ad3b1a45cf7bf6b421d30b19a318
title: Some of the great GNU/Linux distributions
url: https://unixsheikh.com/articles/some-of-the-great-gnu-linux-distributions.html
feedurl: https://unixsheikh.com/feed.rss
content:
In my previous article titled "Why you should migrate everything from Linux to BSD" I stated that "BSD is the place to be". But this doesn't mean that there isn't any good Linux distributions left. There do exist some really good Linux distributions out there where the developers and maintainers have removed all systemd dependencies, and where people are also aware of the status of the kernel development, and Red Hats major corporate influence. Let's take a look at some of these great Linux distributions.


guid: f1f51f720270c79a9be116f611a9e3094b6cd9eb
title: Migrating ZFS from Linux to FreeBSD
url: https://unixsheikh.com/tutorials/migrating-zfs-from-linux-to-freebsd.html
feedurl: https://unixsheikh.com/feed.rss
content:
ZFS on Linux might get a lot of the latest features, and with a distribution like Arch Linux you have the bleeding edge, but it makes great sense to migrate everything ZFS related to FreeBSD. On FreeBSD ZFS is a first class citizen. This means that you don't have to worry about hostile kernel commits that suddenly breaks ZFS, or kernel modules that has to be re-compiled every time the kernel is updated. Being a first class citizen also means that the entire operating system is tailored to work really well with ZFS. The installer makes it really easy to get ZFS on root, with support for all the different possible configurations, and all the relevant tools know about ZFS, even the 'top' commando shows the memory usage of the ZFS ARC!


guid: 76e7554b1bcc13c57e646b8baec41689c5017a9b
title: How you can manage the i3 window manager on multiple computers
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/tutorials/how-you-can-manage-the-i3-window-manager-on-multiple-computers.html

content:
In this tutorial I'll show you a simple way you can manage the configuration of i3 across multiple computers with different setups on each computer.


guid: 74b261095be31fa8899e574e36e8ae1569a206a1
title: Turning suckless into suckmore
url: https://unixsheikh.com/articles/turning-suckless-into-suckmore.html
feedurl: https://unixsheikh.com/feed.rss
content:
The suckless philosophy is all about keeping things simple, minimal and usable, but some people seem to have misunderstood it completely.


guid: e1981cbdd589c201aaeb293a683dc8511e82be32
title: Technical reasons to choose FreeBSD over GNU/Linux
url: https://unixsheikh.com/articles/technical-reasons-to-choose-freebsd-over-linux.html
feedurl: https://unixsheikh.com/feed.rss
content:
In my previous article "Why you should migrate everything from Linux to BSD" part 1 and part 2 I addressed some of the "political issues" with GNU/Linux. And while I believe these issues are important and reasons to consider migrating from GNU/Linux to BSD, there also exist some important technical benefits to consider.


guid: d04060974fa02d857456cb2498bb5dc6f7c28053
title: No, you cannot have a third of my life!
url: https://unixsheikh.com/articles/no-you-cannot-have-a-third-of-my-life.html
feedurl: https://unixsheikh.com/feed.rss
content:
You want a dedicated employee? You want a hard working employee? You want an honest employee? You want an employee who will give you the very best he can while he is working? You've got it! What you cannot have is a third of my life! Life is too short and too precious!


guid: 80891f3fe618ee3692eeffbe571057dd2450ae6c
title: Come full circle - back to HTML
url: https://unixsheikh.com/articles/come-full-circle-back-to-html.html
feedurl: https://unixsheikh.com/feed.rss
content:
From time to time people email me asking about what kind of work flow I use for this blog. In this article I'll elaborate on the process.


guid: 9cb36ab94da227493ac3a53d4506520858d6029e
title: Choose your browser carefully
url: https://unixsheikh.com/articles/choose-your-browser-carefully.html
feedurl: https://unixsheikh.com/feed.rss
content:
Privacy on the Internet is important because privacy risks range from the gathering of statistics on users to more malicious acts such as the spreading of spyware and the exploitation of various forms of bugs (software faults). Many companies, such as Google, track which websites people visit and then use the information, for instance by sending advertising based on one's web browsing history. Sometimes prices on products are changed on the same website, depending on tracking information, and two people may view the exact same product on the exact same website yet be presented with very different prices.


guid: 448eb1210e3d084507d84d80e738a7076afce30a
title: Important open source projects should not use GitHub
url: https://unixsheikh.com/articles/important-open-source-projects-should-not-use-github.html
feedurl: https://unixsheikh.com/feed.rss
content:
Thousands of the worlds best open source projects are still hosting their code repositories on GitHub. Since Microsoft has purchased GitHub this has become a serious problem.


guid: ed256fd12151dd3ea016e0a8df93eadf0764e633
title: The terminal, the console and the shell - what are they?
url: https://unixsheikh.com/articles/the-terminal-the-console-and-the-shell-what-are-they.html
feedurl: https://unixsheikh.com/feed.rss
content:
The other day, as I was going through some of my old notes, I stumbled upon something I had written about the console, the terminal and the shell on Unix-like operating systems. I have decided to rewrite these notes in order to share them here on my website. So without further ado we will now stroll down memory lane and take a quick look at the origins of the Unix terminal and shell. And I will also give my advice to new users on Linux or BSD regarding the choice of terminal emulator and shell.


guid: ce3025a9771d7f058994c23323dc3baee5fd4b05
title: Future prediction: The so-called modern web will die soon
url: https://unixsheikh.com/articles/future-prediction-the-so-called-modern-web-will-die-soon.html
feedurl: https://unixsheikh.com/feed.rss
content:
For many years I have been advocating passionately against the so-called "modern web" as it is really nothing more than a poor excuse for not using your brain and instead following the hype and trends of stuffing web applications with unneeded JavaScript and slow and crappy frameworks. But now the time has come when soon the "modern web" will finally die.


guid: 18f448a0a488d0f4519f7b9203d2c1f9d90ff235
title: The problems with the GPL
url: https://unixsheikh.com/articles/the-problems-with-the-gpl.html
feedurl: https://unixsheikh.com/feed.rss
content:
In my previous article "Why you should migrate everything from Linux to BSD", in the License problems section, I very briefly addressed one of the problems with the GPL license. I have since received a few emails asking for elaboration on the subject, which is what I will try to do in this article.


guid: 0d3db9dae2d3063f9d207857c216f61f2123ddcb
title: Real full disk encryption using GRUB on Artix Linux for BIOS and UEFI
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/tutorials/real-full-disk-encryption-using-grub-on-artix-linux-for-bios-and-uefi.html

content:
In this tutorial we're going to take a look at setting up full disk encryption on the Artix Linux base system using GRUB for both a BIOS/MBR based setup and a UEFI based setup. While the choice to install in UEFI mode is encouraging, vendor UEFI implementations still carry more bugs than their BIOS/MBR counterparts. You are advised to do a search relating to your particular motherboard model before proceeding. Contrary to "modern" advice I still haven't found any compelling reason to use UEFI yet.


guid: b52842d55ae26fcc916edaee8a1c4f10452a5758
title: Are you trusting open source blindly? Then you're in for a world of hurt!
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/articles/are-you-trusting-open-source-blindly-then-you-are-in-for-a-world-of-hurt.html

content:
So, you normally do pip install foo, or composer install foo, or npm install foo, or perhaps go get foo, and you never read the source code of the package you just pulled down? Well guess what, that's one (almost) sure way to blow up your project!


guid: cd57624e1635044317a4bb23db698f1eb9fc059a
title: This is how you keep your dignity and not sell your soul
url: https://unixsheikh.com/articles/this-is-how-you-keep-your-dignity-and-not-sell-your-soul.html
feedurl: https://unixsheikh.com/feed.rss
content:
In 2020 Laurent Bercot, the developer of the s6 init tools, was contacted by a recruiter at Google. He wrote an answer which he made public. That answer deserves to be spread far and wide. I present the answer here as a direct copy from the original.


guid: a2d23ba6ede57b6157023aafb534c7338fc6d03d
title: Code has to be simplified until it becomes absolutely correct
url: https://unixsheikh.com/articles/code-has-to-be-simplified-until-it-becomes-absolutely-correct.html
feedurl: https://unixsheikh.com/feed.rss
content:
systemd is an example of the extreme manifestation of low quality software in the area of system programming. This is because complexity is a very important factor that controls both the creation and defeat of errors, including security related errors.


guid: c315ef6856ed03ba03953de72d5d9f816c1f0737
title: JavaScript malware infested nightmare
url: https://unixsheikh.com/articles/javascript-malware-infested-nightmare.html
feedurl: https://unixsheikh.com/feed.rss
content:
Are you a JavaScript framework or library fan-boy? One of those frontend developers who cannot figure out how to build a website without completely smothering and drowning everything in JavaScript? Well, if you are, let me share a JavaScript nightmare with you!


guid: 31186260dee9827dd385b47ccf0446af3d11a8aa
title: No more object oriented programming with PHP
url: https://unixsheikh.com/articles/no-more-object-oriented-programming-with-php.html
feedurl: https://unixsheikh.com/feed.rss
content:
It has happened again! Another company has contacted me because they had a product developed which has now turned into a huge beast of complexity. Even basic functionality has stopped working and nobody wants to have anything to do with it. It has devastating consequences for the company because they are loosing customers every single day. I have come to the same conclusion, which I have reached in several other similar situations, the codebase is one big pile of complex object oriented mess.


guid: cbfe68b48ce05ec74d4bb0c4388d8c12e518a84d
title: How to test procedural functions with PHPUnit
url: https://unixsheikh.com/tutorials/how-to-test-procedural-functions-with-phpunit.html
feedurl: https://unixsheikh.com/feed.rss
content:
Some people argue that you cannot test procedural PHP code, but that is not true. In this small tutorial I am going to show you how you can use PHPUnit to test your procedural functions.


guid: e7ab3a00c529b556608cb99498d384687f1093b6
title: Procedural programming in a nutshell
url: https://unixsheikh.com/articles/procedural-programming-in-a-nutshell.html
feedurl: https://unixsheikh.com/feed.rss
content:
In my last article No more object oriented programming with PHP I addressed some of the real-life problems that object oriented theory and implementation creates. In this article we'll take a look at some of the benefits of procedural programming and what you need to do, to do it in a proper manner, whether it's PHP, C, or some other programming language.


guid: 6e504a91d4fd975de2db6a058418808cff63b00f
title: How do I know if I really need object oriented programming?
url: https://unixsheikh.com/articles/how-do-i-know-if-i-really-need-object-oriented-programming.html
feedurl: https://unixsheikh.com/feed.rss
content:
It's easy to determine if you really need the object oriented paradigm, you just need to look for the things that you can only do with object oriented programming and then ask yourself whether you really need that. It's called "The Three Pillars of Object-Oriented Programming". If you don't use all three at the same time, then you're not doing object oriented programming and you don't need it.


guid: 35114452ccc3ceeaccee2b6cc368eb26ac8d77ae
title: Stop being frantic about code testing
url: https://unixsheikh.com/articles/stop-being-frantic-about-code-testing.html
feedurl: https://unixsheikh.com/feed.rss
content:
Code testing is not only great, it is a must as you cannot develop software without testing your code. However, there is something which I prefer to call "frantic code testing", which is what needs to be avoided.


guid: ce87c6155bdab9da28bd30841d40f01e0bbed898
title: Straight forward problem solving mentality
url: https://unixsheikh.com/articles/straight-forward-problem-solving-mentality.html
feedurl: https://unixsheikh.com/feed.rss
content:
In my last couple of articles I have been talking about the problems with object oriented programming theory and implementation, and I have been talking about the problems with frantic focusing on software testing. In this article I'm almost not going to say anything, I'll just recommend a really good talk for you :)


guid: 12208f316c1ac7908f6d9ce7b58e7b83d4de6dfd
title: Don't be afraid to code it yourself
url: https://unixsheikh.com/articles/dont-be-afraid-to-code-it-yourself.html
feedurl: https://unixsheikh.com/feed.rss
content:
Have you ever thought about developing some type of application from the grounds up and people are telling you that you shouldn't do it, and that it is difficult, and that you shouldn't reinvent the wheel, etc.? Guess what, what you shouldn't do is to listen to those people!


guid: b6b505e74dc4d0206beeaa7f4787d51892825e16
title: SQLite the only database you will ever need in most cases
url: https://unixsheikh.com/articles/sqlite-the-only-database-you-will-ever-need-in-most-cases.html
feedurl: https://unixsheikh.com/feed.rss
content:
The name SQLite is a nice name, but the "lite" part is misleading, it sounds like it is only useful for tiny things - which is very wrong. SQLite should be named AwesomeSQL, because that is what it is. SQLite is probably the only database you will ever need in most cases.


guid: 207ae93951cd3334f9946d14a27ad24f869f9707
title: Stop submitting to social conformity and use your brain instead
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/articles/stop-submitting-to-social-conformity-and-use-your-brain-instead.html

content:
Whether we deal with technology, religion, science, or any other subject, nothing has ever been more harmful to the human race, both generally and individually, than people submitting to social conformity rather than use the gift of their intellect!


guid: e56fd5d64d3262e07a37c89815a5764ff73ed100
title: Using GitHub Copilot will turn you into a bad programmer
url: https://unixsheikh.com/articles/using-github-copilot-will-turn-you-into-a-bad-programmer.html
feedurl: https://unixsheikh.com/feed.rss
content:
I have already written about how extremely disappointing it is to watch people cheer in ignorance at having their FOSS work and time exploited by one of the largest and crookiest companies in the world, but you also need to understand something else, if you're into programming, using GitHub Copilot will eventually turn you into a bad programmer!


guid: 429f18ee8425ec94e641ec88879d08b66eadde3c
title: Open source projects need to migrate their code away from github NOW!
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/articles/open-source-projects-need-to-migrate-their-code-away-from-github-now.html

content:
If you're writing any kind of open source code and storing it on GitHub, you need to move it away from the platform NOW!


guid: baa0a160f948ff5e37921250df9281231c1f54f0
title: How to do encrypted incremental backups and snapshots with GNU Tar and GnuPG
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/tutorials/how-to-do-encrypted-incremental-backups-and-snapshots-with-gnu-tar-and-gnupg.html

content:
There exist a bunch of really cool open source backup tools such as Borg and Restic, but did you know that you can use GNU Tar and GnuPG to create encrypted incremental snapshots?


guid: badabf0df641ca33005dff4e7d9a95116a5925e4
title: How I store my files and why you should not rely on fancy tools for backup
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/articles/how-i-store-my-files-and-why-you-should-not-rely-on-fancy-tools-for-backup.html

content:
Have you ever lost important data? I have. I learned doing backups the hard way after I lost a entire book I had just finished writing!


guid: a06c963f4bf83333e49fc4092aa532524f908a35
title: A simple Git workflow - using main as the development branch
url: https://unixsheikh.com/tutorials/a-simple-git-workflow-using-main-as-the-development-branch.html
feedurl: https://unixsheikh.com/feed.rss
content:
One of the main benefits of Git is that it is an extremely flexible set of tools with which you can build many development models and branching strategies from. In this tutorial I'll show you my favorite Git workflow that works very well both as a solo developer and in a team with many developers.


guid: 06c7b11db81edc0909a97496a854803ef608d96a
title: Use the right tool for the job
url: https://unixsheikh.com/articles/use-the-right-tool-for-the-job.html
feedurl: https://unixsheikh.com/feed.rss
content:
I am sure you have heard it many times, "use the right tool for the job", yet at the same time we're often inclined toward a specific operating system or a specific piece of software. And while it's quite normal to have a favorite operating system or a favorite piece of software, it can be very counter productive to only use a single solution for every problem.


guid: 07414e401dc9e8e8fdab210c338230067b2a8659
title: There is no such thing as privacy on the Internet
url: https://unixsheikh.com/articles/there-is-no-such-thing-as-privacy-on-the-internet.html
feedurl: https://unixsheikh.com/feed.rss
content:
As soon as you step outside your house or apartment, you're located in the "public arena" where there is no privacy. I don't mean juridically speaking, but practically, as everyone can see you. Now, why would you think it is any different with the Internet!?


guid: f40406cf3112dabbd2f0772f5b3606a7698f0f6a
title: How to write software that will keep working for decades without problems
url: https://unixsheikh.com/articles/how-to-write-software-that-will-keep-working-for-decades.html
feedurl: https://unixsheikh.com/feed.rss
content:
I think that most people in the software industry have been in the situation of running a legacy system that everyone dreads because it was developed on a platform that was outdated a long time ago. In this article I will address the issue of how you develop software that keeps running without fear of breakage even when you upgrade to the latest version of the platform that the system is running on.


guid: 176ae236ad5bd19a36c821768f87483e72ad24ce
title: Using a framework can make you stupid!
url: https://unixsheikh.com/articles/using-a-framework-can-make-you-stupid.html
feedurl: https://unixsheikh.com/feed.rss
content:
By the word "stupid" I mean "a lack of knowledge", not stupid in the literal sense of the word, so please don't take any offense.


guid: b9d587ee1e34ed8ea19062a37a7caa94aa1c3775
title: So-called modern web developers are the culprits
url: https://unixsheikh.com/articles/so-called-modern-web-developers-are-the-culprits.html
feedurl: https://unixsheikh.com/feed.rss
content:
2022 needs to be the year when we seriously begin to get rid of bloatware, hype and trends in web development. It needs to be the year when we focus on performance and on utilizing the technology in harmony with the intend behind it. I am not saying that we should do all web development in assembly or C, but we need to stop the JavaScript, Ruby on Rails, Python with Django and PHP framework madness.


guid: 088346182073b33845891d2ad58026e54f593d9f
title: No, your website is not a web app even if you call it so
url: https://unixsheikh.com/articles/no-your-website-is-not-a-webapp-even-if-you-call-it-so.html
feedurl: https://unixsheikh.com/feed.rss
content:
As a response to my article, So-called modern web developers are the culprits, someone stated that my advice was outdated because today we make web apps, not websites. I will address that issue in this article.


guid: ece40412833d76d7d72b76adf359dc0f138c6190
title: My 70 year old mother has been using Linux on the desktop for the past 21 years
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/articles/my-70-year-old-mother-has-been-using-linux-on-the-desktop-for-the-past-21-years.html

content:
My mother turns 70, February this year, and she has been using Linux on the desktop for the past 21 years. My mother-in-law is 65, she has been using Linux on the desktop since 2015.


guid: c4956518d1f9fba6da87e5d0e583322c15b23c04
title: Is the madness ever going to end?
url: https://unixsheikh.com/articles/is-the-madness-ever-going-to-end.html
feedurl: https://unixsheikh.com/feed.rss
content:
There is something seriously wrong with the IT industry. It's so bad that I haven't managed to find a single industry with the same massive amount of stupidity, with the exception of perhaps the fashion industry. It's like the IT industry has been paralyzed in some strange kind of mass hypnosis.


guid: a9b7c8085771aa2010ed40ab01ce49700314013d
title: A warning to business owners and managers, you are a big part of the problem!
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/articles/a-warning-to-business-owners-and-managers-you-are-a-big-part-of-the-problem.html

content:
In my last couple of articles, mainly So-called modern web developers are the culprits and Is the madness ever going to end? I have written about some of the major problems with so-called modern web development and I have addressed the issues to the developers themselves, which is where I think most of the blame lies. However, in some cases the business owners and managers are the true culprits. They are the major driving force behind the bad decisions, so if you're a business owner or a manager looking for the quick fix, I advice you to read this article carefully.


guid: 8529730894c01f664de1f62e72f289b04fa6910d
title: Questions about which operating system is more secure or fastest makes no sense
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/articles/questions-about-which-operating-system-is-more-secure-or-fastest-makes-no-sense.html

content:
Reddit, misc. forums and mailing lists are filled with questions and debates about whether operating system A is more secure than operating system B, or whether operating system A is faster than operating system B. Such questions and debates are not only completely useless, but they are also somewhat childish.


guid: 41e966c6392bb5021547e3cee9dc444741f27774
title: How security is handled by package maintainers
url: https://unixsheikh.com/articles/how-security-is-handled-by-package-maintainers.html
feedurl: https://unixsheikh.com/feed.rss
content:
A lot of myth and misunderstanding exist regarding this issue and it is easy to get confused because it varies from project to project. This post is not about base systems or base packages, rather it's about third party packages.


guid: 35e7f07268fa63b243ef8b969a5e32aec76ac7f9
title: The delusions of debian
url: https://unixsheikh.com/articles/the-delusions-of-debian.html
feedurl: https://unixsheikh.com/feed.rss
content:
Debian was my favorite Linux distribution for servers where Linux where required to run for many years, and I think I have been running it in production since about 1998 up until the time about when the systemd conflict arose. Since that time I have noticed a rapid decline in many areas of Debian. This has continued up until today and Debian is no longer what Debian used to be.


guid: 65692f22c85be087bd30340fa49569d388b8d2d2
title: Linux distribution long term support might not be what you think it is
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/articles/linux-distribution-long-term-support-might-not-be-what-you-think-it-is.html

content:
Since I wrote The delusions of Debian, I have received a lot of email and questions. I have therefore decided to do a small follow-up about the true meaning of Long Term Support (LTS) in these projects.


guid: 677338e8f5317af6de5962dedf35dd1baa7f5592
title: Don't use Reddit for Linux or BSD related questions
url: https://unixsheikh.com/articles/dont-use-reddit-for-linux-or-bsd-related-questions.html
feedurl: https://unixsheikh.com/feed.rss
content:
If you're looking for answers related to Linux or BSD (or anything for that matter), Reddit is most likely not the right place to search for them.


guid: 3cb208d5758be2eca85ce7f336082175a48f849f
title: The flaws of distro hopping and asking other people about their OS of choice
feedurl: https://unixsheikh.com/feed.rss
url:
https://unixsheikh.com/articles/the-flaws-of-distro-hopping-and-asking-other-people-about-their-os-of-choice.html

content:
Distro hopping is a term that refers to the activity of computer users constantly installing and trying out different Linux distributions and/or BSD variants, either on bare metal or in a virtual environment, having a real difficult time choosing what to use. Some people believe that distro hopping is a result of boredom, or because of a lack of satisfaction with current choice, or that it's simply a matter of trying out multiple options, but that is not the case. Rather, it's a kind of psychological problem that arise because of a wrong approach being used in the process of determination.


guid: https://www.alexblackie.com/articles/devops-in-the-closet/
title: DevOps in the Closet
author: Alex Blackie
url: https://www.alexblackie.com/articles/devops-in-the-closet/
feedurl: https://www.alexblackie.com/feed.xml
content_mime_type: text/html
content:
 		<p>
  I love hardware. I love the raw power you can get from bare metal, and the
  ownership and control one can have over it. Even though my needs are modest,
  and could easily be satisfied by a couple mid-range VPS instances, there's no
  fun in that! Recently I moved to a hybrid setup -- a 12U rack of machines in
  my apartment connected to a load balancer running on a cloud provider.
</p>

<p>
  For a while I rented colocation space from the wonderful people at Hurricane
  Electric -- I highly recommend checking them out if you need colocation, IP
  transit, or anything of the sort. However, in my efforts to move all my data
  and property within Canadian borders, I had to take my children back into my
  home.
</p>

<p>
  I am fortunate enough to have a symmetrical gigabit fibre connection to my
  apartment, and am doubly fortunate to have the resources and ability to run
  it <a href="/articles/fibe">entirely on my own hardware</a> &mdash; this
  means I have ample bandwidth to spare, and with a newfound pile of servers at
  my disposal, what better than to run production traffic from my closet?
</p>

<h2>Overcoming NAT</h2>

<p>
  The first problem here, of course, is addressing. Consumer ISPs being what
  they are, my residential internet plan has only IPv4, and only one address.
  Not only that, but an address that is different every time the PPPoE tunnel
  renegotiates. None of these attributes are particularly helpful for hosting.
</p>

<p>
  But then, an idea: what if I used a VPN <em>as they are intended to be
  used</em>? Everyone loves to use VPNs these days just to tunnel their traffic
  through someone else's connection, but really that's nothing more than an
  abuse of a very powerful tool. The point of a Virtual Private Network server
  is, well, to provide a virtual, private network.
</p>

<p>
  <strong>So I set up a VPS running OpenVPN</strong>. Every machine in my
  closet has an OpenVPN client running on it, connected to this server (I refer
  to this machine as "the hub"). All machines connect to this VPN and get a
  static IP assigned to them in the VPN's private subnet. Now, any machine
  connected to The Hub can seamlessly communicate with any other, as if they
  were sitting next to each other on a switch. <strong>This</strong> is what
  VPN's are for.
</p>

<p>
  The astute among you may have already realised: there's nothing about this
  setup that restricts it to servers in my closet. Indeed, since the OpenVPN
  Clients authenticate using certificates, I could simply copy the certs to a
  server somewhere entirely different, and it would connect and be available on
  the same IP. This means with no deployments or config changes I can fail-over
  machines to ones in entirely different regions, behind entirely different
  networks -- even a hidden machine in my parents' basement -- and it would be
  available just the same as everything else, and on the same IP.
</p>

<p>
  <strong>The one downside</strong> with this setup is the OpenVPN "hub" server
  is a single point of failure. If the VPN goes down, so does every machine
  connected to it. I'm sure there are ways to work around this and deploy a HA
  VPN, but for my purposes I'm fine with the risk of failure in the name of
  simplicity.
</p>

<h2>Serving the Web from my closet</h2>

<p>
  So thus far I have a bunch of servers connected to a VPN and it's all peachy,
  but this still doesn't solve the original problem: they're still not
  publicly-accessible. A private IP, even if connected to a public server, is
  still private. <strong>Enter, HAProxy.</strong>
</p>

<p>
  Playing off of the fact any machine can connect to the VPN, I have a second
  VPS beside The Hub running HAProxy. This machine acts as my "edge" (the
  machine that accepts all incoming traffic) and has IPv6, backups, a fast
  datacenter-grade network connection, etc. This machine is also connected to
  the VPN, just as the rest of the servers, and HAProxy's backends point to the
  static IPs on the VPN for the respective services.
</p>

<p>
  This allows me to still support IPv6 and still have the edge of my network in
  a reliable place, but host the bulk of my data and harness the power of raw
  hardware at the same time -- without having to shell out hundreds of dollars
  per month to pay for colocation.
</p>

<p>
  Additionally, this allows me to build in high-availability completely
  transparently -- I could have HAProxy load balance between two servers: one
  in my closet, and one on a VPS provider (for example), and then even if my
  power goes out or I accidentally unplug everything while cleaning traffic
  would be uninterrupted. This also means that under heavy traffic, I can
  quickly scale-out instances on a cloud provider to handle it, while still
  keeping my affordable baseline of dedicated hardware.
</p>

<h2>Simplified management</h2>

<p>
  One other benefit of this setup is <strong>my laptop has an IP on The Hub
    too</strong>. With one click I can have my laptop directly attached to the
  production VPN, and have full access to every machine in my arsenal, no
  matter where it is or what firewalls it may be behind.
</p>

<p>
  This means I don't have to expose SSH anywhere except on the private VPN
  network (as I can run Ansible anywhere in the world over the VPN), and don't
  need to worry about which servers are on which providers to make sure I have
  the right VPNs enabled -- it's only one VPN everywhere.
</p>

<h2>Utilization of&hellip; unusual hardware</h2>

<figure>
  <img src="unconventional-desires.jpg" alt="A top-down shot of a rack shelf with some consumer-grade computers" width="960" height="342">
</figure>

<p>
  In addition to my pile of Xeon-based production-grade server hardware, I also
  had a pile of unused consumer trash boxes that needed something to do.  Under
  this new setup, I now am running several "internal" or "secondary" servers on
  a few perhaps-not-production-grade computers: a Gigabyte Brix and a Zotac
  ZBox, as well as a couple old laptops.
</p>

<p>
  These are obviously not machines I would have put in a datacenter, but are
  still useful nonetheless, and for non-critical "tier two" services (like CI
  builders, yum repo mirrors, or internal services) they serve their role
  perfectly and compliment the "shoestring budget" of the entire setup.
</p>

<p>
  This has made me feel much better about the previously-useless pile of old
  laptops and dust-covered relics I had been hiding away in a box. This project
  has allowed me to give them a new life and purpose.
</p>

<h2>But Don't Do This.</h2>

<p>
  This setup works great for me, because I run a handful of shitty websites
  only for myself, and don't get any significant amount of traffic.
  <strong>Please do not use this article as a guide. Do not deploy this
  ever.</strong> If any of my projects ever came close to hosting anything that
  mattered or held customer data I would immediately move everything to
  colocation and implement full high-availability practices.
</p>

<p>
  This was a fun project, and since I pay out-of-pocket for all this it made a
  significant budgeting difference. If you're in a similar boat, perhaps it
  could work for you, but absolutely do not think this is an acceptible
  production network for anything that remotely matters.
</p>

<h2>What would it take&hellip;</h2>

<p>
  But <em>what if</em> you did in fact want to make this setup more legitimate?
  What if we wanted to deploy a proper on-premises "hybrid cloud" solution?
  Perhaps surprisingly, it's not that far off from what we already have.
</p>

<ol>
  <li>
    <p><strong>Router-level tunneling</strong>.</p>
    <p>
      Switching the individually-connected servers to a router-level network
      bridge would probably be a better permanent solution. This would mean
      running a single tunnel between the on-prem VLAN and the remote cloud
      private network, and continuing to utilize the existing subnets of both.
      This would mean every machine in each subnet would be automatically
      visible to the whole network, as the tunnel is bridging <em>networks</em>
      instead of <em>machines</em>.
    </p>
    <p>
      However, this would impose more conditions on each on-prem network: each
      network being connected would have to be specifically designed to be
      connected. This would lose the current benefit of being able to connect
      any machine anywhere without anything more than running OpenVPN.
    </p>
    <p>
      This also would mean using a simple VPS for this would probably not work,
      as the networking options provided by most VPS providers are limited at
      best. You would need something more akin to an AWS VPC or Azure Virtual
      Network.
    </p>
  </li>
  <li>
    <p><strong>HA OpenVPN</strong>.</p>
    <p>
      The OpenVPN "Hub" server is a single point of failure in this design. For
      more legitimate hybrid cloud deployments, you'd want to ensure high-availability
      of the OpenVPN hub so that a machine failure or region outage wouldn't
      take down the entirety of your infrastructure.
    </p>
    <p>
      Unfortunately I don't really know the extent of OpenVPN's HA capabilities,
      as it's never been something I've needed; the first thought I have is CARP
      to enable machine-level redundancy in case of hardware failure. This isn't
      "globally-redundant", but it is a good first step. Chances are, if you're
      running on-prem, an outage of that premises would be catastrophic anyway.
    </p>
  </li>
</ol>


guid: https://www.alexblackie.com/articles/static-site-bash/
title: Generating Static Sites with Make and Bash
author: Alex Blackie
url: https://www.alexblackie.com/articles/static-site-bash/
feedurl: https://www.alexblackie.com/feed.xml
content_mime_type: text/html
content:
 		<p>
	When you think of <code>bash</code>, you may not consider it suitable for
	building websites. For the most part, you are probably correct. But I
	wanted to try anyway. I don't buy into most trends in the frontend web
	development community, so literally none of the "modern" static site
	generators appeal to me. All of them feel overcomplicated, too niche, or
	too javascript-y for my needs, which involve <em>just rendering a
	website</em>.
</p>

<p>
	Inspired by
	<a href="https://www.johnhawthorn.com/2018/01/this-website-is-a-makefile/">John Hawthorn's makefile website workflow</a>,
	I decided that was almost exactly what I wanted. However, I didn't really
	want to depend on anything outside of standard unix tools (so no Ruby).
	This is where bash comes in.
</p>

<h2>"uhhh"</h2>

<p>
	Look, hear me out. There are some reasons I actually ended up liking this:
</p>

<ul>
	<li>All the normal benefits of building static sites</li>
	<li>Really fast (sub-second full site builds on my laptop)</li>
	<li>No dependencies outside of GNU/POSIX tools</li>
	<li>It's kind of fun and weird</li>
	<li>i don't need to install 2500 npm packages and learn a cool new templating language that will be obsolete and unmaintained in 4 months</li>
</ul>

<p>
	Ultimately, this was just a fun project to make my personal site a little
	more interesting. This really isn't something that is overly practical.
	Nothing wrong with a little fun now and then, though!
</p>

<h2>The result</h2>

<p>
	Before we get in the weeds, let's see what the end-goal for project
	structure is. I've removed a few things that are in my own site, just to
	keep this example simple.
</p>

<div class="code">
	<div class="code-title">$ tree -F --dirsfirst</div>
<pre><code>.
├── _build/
│   └── ...
├── bin/
│   └── render*
├── layouts/
│   └── site.html
├── pages/
│   ├── some-page/
│   │   ├── index.html
│   │   └── index.meta
│   ├── index.html
│   └── index.meta
├── static/
│   ├── _/
│   │   └── site.css
│   └── favicon.ico
├── Makefile
└── helpers.sh</code></pre>
</div>

<p>
	Basically, we have a directory for pages (<code>./pages</code>) and then one directory for each page, with a <code>.meta</code> file for metadata (title, variables, etc) and a <code>.html</code> file for the content itself.
</p>

<h2>Setting up <code>make</code></h2>

<p>
	Originally, I had started with pure bash -- just a single bash script that
	handled the entire pipeline from finding the files, to parsing them, to
	writing to a file. This actually worked out great, but was missing some
	niceties like incremental builds, and meant a lot of it was sort of
	"intertwined" where rendering a page had weird knowledge about where that
	page should be written to disk.
</p>

<p>
	So ultimately I decided simplifying the shell script to just render a page
	to <code>stdout</code> and let good ol&rsquo; <code>make</code> handle the
	reading and writing seemed like a more robust option.
</p>

<p>
	Luckily, because we're just running one command and writing that to disk,
	our Makefile can be pretty simple:
</p>

<div class="code">
	<div class="code-title">Makefile</div>
<pre class="chroma"><span class="c"># Find all html and meta files in the pages directory
</span><span class="c"></span><span class="nv">pageSources</span><span class="o">=</span><span class="k">$(</span>shell find pages -type f -name <span class="s1">&#39;*.html&#39;</span> -o -name <span class="s1">&#39;*.meta&#39;</span><span class="k">)</span>

<span class="c"># Map all the html files in the list of sources to the target path in _build
</span><span class="c"></span><span class="nv">pageTargets</span><span class="o">=</span><span class="k">$(</span>pageSources:pages/%.html<span class="o">=</span>_build/%.html<span class="k">)</span>

<span class="nf">all</span><span class="o">:</span> <span class="k">$(</span><span class="nv">pageTargets</span><span class="k">)</span>

<span class="nf">_build/%.html</span><span class="o">:</span> <span class="n">pages</span>/%.<span class="n">html</span> <span class="n">pages</span>/%.<span class="n">meta</span>
	@# Silently <span class="o">(</span>@<span class="o">)</span> make all directories leading up to our target file <span class="o">(</span><span class="nv">$@</span><span class="o">)</span>
	@mkdir -p <span class="k">$(</span>dir <span class="nv">$@</span><span class="k">)</span>

	@# Run our render script, giving the <span class="nb">source</span> file name <span class="o">(</span>$&lt;<span class="o">)</span>, substring-
	@# replacing <span class="s2">&#34;html&#34;</span> with <span class="s2">&#34;meta&#34;</span> <span class="o">(</span>as our script expects to be given
	@# the meta file<span class="o">)</span>. Write the output to the target filename <span class="o">(</span><span class="nv">$@</span><span class="o">)</span>
	bin/render <span class="k">$(</span>&lt;:html<span class="o">=</span>meta<span class="k">)</span> <span class="p">&amp;</span>gt<span class="p">;</span> <span class="nv">$@</span>

<span class="nf">clean</span><span class="o">:</span>
	rm -rf _build/*
</pre>
</div>

<p>
	All this does is find all <code>*.meta</code> and <code>*.html</code>
	files, and tell <code>make</code> that those should turn into a single HTML
	file in <code>_build</code>, by running the <code>bin/render</code> script.
	Of course, a lot of this is terse or a bit weird to read if you're
	unfamiliar with Makefiles, but in the end it is rather simple, and provides
	some pretty significant power for not a lot of code.
</p>

<h2>Proof of concept: straight renders</h2>

<p>
	To prove that this might actually work, the first thing to get going is a
	just rendering the page HTML itself, with basic variable substitution for
	things like page title, etc., but without a layout or any fancy stuff.
</p>

<p>
	The general idea here is to parse our template files inside a Here
	Document, which will give them full access to shell functions, local
	variables, and any command they want, really. This is obviously a huge
	security risk, but there should never be untrusted code being executed in
	this context, so the realistic risk is pretty minimal.
</p>

<div class="code">
	<div class="code-title">bin/render</div>
<pre class="chroma"><span class="cp">#!/usr/bin/env bash
</span><span class="cp"></span>
<span class="nb">set</span> -euf -o pipefail

render<span class="o">()</span> <span class="o">{</span>
	<span class="nv">title</span><span class="o">=</span><span class="s2">&#34;My web page&#34;</span>
	<span class="nb">echo</span> <span class="s2">&#34;</span><span class="k">$(</span><span class="nb">eval</span> <span class="s2">&#34;cat &lt;&lt;EOF
</span><span class="s2">		</span><span class="k">$(</span>&lt;<span class="si">${</span><span class="nv">1</span><span class="p">/.meta/.html</span><span class="si">}</span><span class="k">)</span><span class="s2">
</span><span class="s2">EOF&#34;</span><span class="k">)</span><span class="s2">&#34;</span>
<span class="o">}</span>

render <span class="nv">$@</span>
</pre>
</div>

<p>
	In essence, what we're doing here is reading an HTML file in as a string,
	which is interpolated into a heredoc, which is piped into <code>cat</code>,
	which is in a string, which we are evaluating as if it were a script
	itself. This means that the normal variable scope and expansion applies to
	our template.
</p>

<p>
	The <code>index.html</code> source file, for reference:
</p>

<div class="code">
	<div class="code-title">pages/index.html</div>
<pre class="chroma"><span class="p">&lt;</span><span class="nt">h1</span><span class="p">&gt;</span>$title<span class="p">&lt;/</span><span class="nt">h1</span><span class="p">&gt;</span>
</pre>
</div>

<p>
	And that's honestly about it. Everything else from here is just doing this
	multiple times, or adding some sugar or abstractions. Speaking of, let's
	make this nicer.
</p>

<h2>Adding a layout</h2>

<p>
	Currently we're just copying one file around, which isn't really helpful
	for a static site builder! What really made this "click" was getting a
	layout template wrapping the pages -- the most basic feature of any static
	site generator.
</p>

<p>
	So let's do that. All this is really doing is assigning the page to a
	variable, then rendering the layout as if it was the page.
</p>

<div class="code">
	<div class="code-title">bin/render</div>
<pre><code>#!/usr/bin/env bash

set -euf -o pipefail

render() {
	title="My web page"
<span class="code-rem">	echo "$(eval "cat &lt;&lt;EOF</span>
<span class="code-add">	content="$(eval "cat &lt;&lt;EOF</span>
		$(&lt;${1/.meta/.html})
EOF")"

<span class="code-add">	echo "$(eval "cat &lt;&lt;-EOF</span>
<span class="code-add">		$(&lt;layouts/site.html)</span>
<span class="code-add">	EOF")"</span>
}

render $@
</code></pre></div>

<p>
	In our layout, we can then just use <code>$content</code> where we want to
	render the page content.
</p>

<div class="code">
	<div class="code-title">layouts/site.html</div>
<pre><code>&lt;!doctype html&gt;
&lt;title&gt;$title&lt;/title&gt;
<span class="code-hl">$content</span></code></pre>
</div>

<p>
	We're getting pretty close to something useful. But hard-coding all our
	metadata in the build script doesn't scale. Let's fix that.
</p>


<h2>Metadata is just scripts</h2>

<p>
	Previously I had mentioned <code>.meta</code> files, but up until now we
	haven't used them.  Since our pages are being rendered within the context
	of the bash script, we can source other scripts in dynamically. So, our
	<code>.meta</code> files are just little bash scripts that are evaluated
	right before render. Thus, they could contain variables, functions, calls
	to external APIs or programs&hellip;
</p>

<p>
	Here's the <code>.meta</code> file for our page:
</p>

<div class="code">
	<div class="code-title">page/index.meta</div>
<pre class="chroma"><span class="nv">title</span><span class="o">=</span><span class="s2">&#34;My web page&#34;</span>
<span class="nv">lastUpdatedAt</span><span class="o">=</span><span class="s2">&#34;2019-06-13&#34;</span>
</pre>
</div>

<p>
	And in the build script, all we need to do is source our <code>.meta</code>
	file dynamically, as we do with the HTML template. Since our render script
	is being given the meta file as the first argument, we can just source
	that:
</p>

<div class="code">
	<div class="code-title">bin/render</div>
<pre><code>&hellip;
render() {
<span class="code-rem">	title="My web page"</span>
<span class="code-add">	. "$1"</span>
	content=$(eval "cat &lt;&lt;EOF
		$(&lt;${1/.meta/.html})
EOF")
&hellip;</code></pre>
</div>

<p>
	Another "lightbulb" moment is when you start to add shell functions. Want
	to format a date? Good news! Basically every computer probably already has
	<code>date(1)</code>.
</p>

<p>
	We can just add more functions to the <code>bin/render</code> file, or if
	we have a lot we can source a second file into it... It's just a shell
	script, so I mean you can do whatever you want.
</p>

<div class="code">
	<div class="code-title">bin/render</div>
<pre><code>#!/usr/bin/env bash

set -euf -o pipefail

<span class="code-add">formatDate() {</span>
<span class="code-add">	echo $(date -d "$1" +"%B %-d %Y")</span>
<span class="code-add">}</span>

render() {
&hellip;</code></pre>
</div>

<div class="code">
	<div class="code-title">pages/index.html</div>
<pre class="chroma"><span class="p">&lt;</span><span class="nt">h1</span><span class="p">&gt;</span>$title<span class="p">&lt;/</span><span class="nt">h1</span><span class="p">&gt;</span>
Last updated: $(formatDate $lastUpdatedAt)
</pre>
</div>

<p>
	That's it. Now we can use ISO-8601 dates in our metadata, but still display
	"human" dates on the site itself.
</p>

<p>
	You can even combine it with Here Documents to make more complex render
	helper functions. I do this on my site to dynamically discover blog
	articles from the filesystem, parse and sort them based on their date in
	metadata, and then render a provided template string with the new local
	variables. It's a bit convoluted and gross to use as a direct example, so
	here's a simplified one:
</p>

<div class="code">
	<div class="code-title">build.sh</div>
	<pre class="chroma">...

renderItems<span class="o">()</span> <span class="o">{</span>
	<span class="nv">items</span><span class="o">=(</span>one two three<span class="o">)</span>
	<span class="nv">template</span><span class="o">=</span><span class="s2">&#34;</span><span class="k">$(while</span> <span class="nb">read</span> template <span class="p">;</span> <span class="k">do</span> <span class="nb">echo</span> <span class="nv">$template</span> <span class="p">;</span> <span class="k">done)</span><span class="s2">&#34;</span>

	<span class="k">for</span> item in <span class="si">${</span><span class="nv">items</span><span class="p">[*]</span><span class="si">}</span> <span class="p">;</span> <span class="k">do</span>
		<span class="nb">eval</span> <span class="s2">&#34;cat &lt;&lt;-EOFOR
</span><span class="s2">			</span><span class="nv">$template</span><span class="s2">
</span><span class="s2">		EOFOR&#34;</span>
	<span class="k">done</span>
<span class="o">}</span>

...
</pre>
</code></pre>
</div>

<div class="code">
	<div class="code-title">pages/index.html</div>
<pre class="chroma">...

&lt;ul&gt;
<span class="k">$(</span>renderItems <span class="s">&lt;&lt;-EOTEMPLATE
</span><span class="s">	&lt;li&gt;$item&lt;/li&gt;
</span><span class="s">EOTEMPLATE</span>
<span class="k">)</span>
&lt;/ul&gt;
</pre>
</div>

<p>
	At this point, that's basically 90% of a static site generator. You got
	layouts, templates, and metadata variables with support for helper
	functions, extenal "plugins" (normal CLI tools)&hellip; What more do you
	need?
</p>

<h2>Assets</h2>

<p>
	Every site will have some CSS, JS, images, and whatever else. You could
	call out to some preprocessor or crazy thing if you really wanted, but for
	me I am happy with raw CSS and JS, so here is my asset pipeline:
</p>

<div class="code">
	<div class="code-title">Makefile</div>
<pre><code>pageSources=$(shell find pages -type f -name '*.html' -o -name '*.meta')
pageTargets=$(pageSources:pages/%.html=_build/%.html)

<span class="code-add">staticSources=$(shell find static -type f)</span>
<span class="code-add">staticTargets=$(staticSources:static/%=_build/%)</span>

<span class="code-rem">all: $(pageTargets)</span>
<span class="code-add">all: $(pageTargets) $(staticTargets)</span>

&hellip;

<span class="code-add">_build/%: static/%</span>
<span class="code-add">	@mkdir -p $(dir $@)</span>
<span class="code-add">	cp $< $@</span>

&hellip;
</code></pre>
</div>

<p>
	🖌👌✨
</p>

<h2>What's the catch?</h2>

<p>
	While the core idea proved to work out great, it was not without rough
	edges. In the end, these all proved fine to work around or put up with, but
	they exist nonetheless.
</p>

<ul>
	<li><strong>Escaping normal content</strong> &mdash; since every page is parsed as a string, you need to be careful what that string contains. For example, "<code>I paid $50</code>" needs to be "<code>I paid \$50</code>" to avoid <code>$5</code> being parsed as a variable.</li>
		<li><strong>Shell scripts kind of suck</strong> &mdash; shell scripting is not really a pleasant environment. Decades of weird decisions, cumbersome syntax... While shell scripts get you a powerful and highly extensible environment, that comes at a cost of some readability and "niceness" that you might expect from something like ERB or Handlebars, for example.</li>
		<li><strong>Cross-OS compatibility is a nightmare</strong> &mdash; you'd think POSIX meant everything behaves the same but HA-HA NOPE good luck maintaining weird <code>if</code> statements to try and make sure you pass different flags to GNU vs. BSD <code>date</code> or <code>find</code> to support both platforms.</li>
</ul>

<h2>Also I left out a bunch of other stuff</h2>

<p>
	I have left out a significant amount of work and numerous features that are
	implemented in the "real" version of this script that powers my site. You
	are free to
	<a href="https://github.com/alexblackie/watashi">peruse my website's source code</a>,
	if you are curious. This post was intended not to be a guide or tutorial
	but just a neat showcase of a semi-crazy idea.
</p>

<p>
	In brief, some features that were pretty easy to add that my script ended
	up supporting:

	<ul>
		<li>custom layouts per page</li>
		<li>nested layouts for articles</li>
		<li>xml pages and layouts (rss feed)</li>
		<li>dynamically building list of articles from filesystem</li>
	</ul>
</p>

<hr>

<p>
	This was a fun project for me, and the end result solved a real need.
	Hopefully it was as interesting to you as it was me!
</p>


guid: https://www.alexblackie.com/articles/trying-to-be-worse/
title: Build Worse Software
author: Alex Blackie
url: https://www.alexblackie.com/articles/trying-to-be-worse/
feedurl: https://www.alexblackie.com/feed.xml
content_mime_type: text/html
content:
 		<p>
	<big>
		One of my first software projects -- probably the first project
		I did that wasn't just building a website theme -- was building
		my own CMS. The year was 2010, I was a middle school student
		doing web design on the side, and I was tired of
		WordPress.
	</big>
</p>

<p>
	I am reminded of this project from time to time; usually I remember how
	much I enjoyed building it. I used this custom CMS for some local business
	websites during my consulting days. The project was just a pile of
	procedural PHP scripts, with HTML and PHP just smashed together. MySQL was
	the database of choice, with unprepared SQL strings littered everywhere.
</p>

<p>
	Somewhere along the way since then, I learned the "right" way to build
	things: carefully, with lots of planning, tests-first, deployed in a
	scalable way, stateless if possible&hellip; the list goes on. And with that
	knowledge, I lost the ability to just <em>build something</em>. My first
	question wasn't "What features should it have?", but rather "What tools and
	architecture should I choose?".
</p>

<p>
	In my path to becoming a "professional programmer" <mark>I lost the most
		important goal of writing software: to solve problems</mark>. I became
	enamoured with the tools, processes, and dogmatic "correctness" of various
	communities that I blinded myself to what I was actually trying to
	accomplish.
</p>

<p>
	My projects languished in various states of incompleteness as every time I
	would try and make progress, I would be hindered by a crippling narrative
	that I needed to do it <em>the right way</em>. I was paralyzed by my own
	irrational standards.
</p>

<p>
	<mark>My solution to this was to build worse software.</mark> All the
	barriers of code quality, tests, flawless architecture -- they're are all
	constructs that affect <em>production</em> software; but they do not affect
	software that's just getting started. I had to unlearn my impulses and
	force myself to take naïve risks and shortcuts to getting a working
	solution. Visible progress is the finest motivator.
</p>

<p>
	Since then I've made an effort to purposefully write bad software first,
	and good software later. By prototyping a messy, mostly-working version of
	something, you explore the solution organically and have a starting point
	for improvement.
</p>

<p>
	This practice of care-free prototyping is what has allowed me to build
	<a href="https://www.zaiko.xyz">Zaiko</a>, my inventory management
	software. I needed to track server parts as I built and shipped some
	machines to a datacenter, maybe 4 or 5 years ago. None of the existing
	solutions seemed particularly nice to use, so I started building my own.
</p>

<p>
	I could have spent decades planning, scheming, and architecting a
	textbook-worthy application that followed all the rules; instead, <mark>I
		slapped together a Rails application that worked "enough"</mark> and
	solved my problem.  Five years and one rewrite in Elixir later, I
	understand so much more about what I want from the product and have slowly
	built up a production-ready SaaS application that is ready to scale and
	extend, and has pretty decent test coverage.
</p>

<p>
	Was that Rails prototype a waste of time, since I rewrote it? Absolutely
	not. It let me quickly explore features and ideas, and got me using my own
	product as soon as possible. Would I have felt comfortable opening it up to
	the world? Also absolutely not. The goal isn't to build worse software and
	call it done; the goal is to build worse software so you get the context
	you need to evolve it into good software over time.
</p>

<p>
	If I had to sum up the lesson I learned in a sentence, it'd be&hellip; Time
	heals all software? You have to build the village before the roads?
	Something like that, anyway; I'm still working on it.
</p>


<hr>

<p>
	<strong>Addendum:</strong> A while back I found a snapshot of the code for
	my first CMS archived on Google Code (RIP). I managed to export the SVN
	repo into Git and have
	<a href="https://github.com/alexblackie/x96-panel">archived the project on Github</a>
	for those who are curious to see some godawful PHP.  Only took a few
	seconds to find an SQL injection vulnerability. Oh, how times change.
</p>


guid: https://www.alexblackie.com/articles/nextcloud-on-k8s/
title: Stateless Nextcloud on Kubernetes
author: Alex Blackie
url: https://www.alexblackie.com/articles/nextcloud-on-k8s/
feedurl: https://www.alexblackie.com/feed.xml
content_mime_type: text/html
content:
 		<p>
	<big>
		Since early 2018, I have used <a href="https://nextcloud.com">Nextcloud</a>
		in a production capacity, providing domestic, privacy-respecting
		cloud file storage for myself and my family. Over the years, I
		have spent a considerable amount of time finding a way to run it
		safely, ensuring that decades of documents, photos, and other
		critical data do not go missing.
	</big>
</p>

<div class="note">
	<div class="note--body">
		<strong>Update 2021-08:</strong> Since writing thist post I have moved
		back to a more "traditional" hosting method involving physical hardware
		in a datacentre and lots of SSDs. I enjoyed the challenge of making
		Nextcloud "cloud-native" but ultimately I think it's more trouble than
		it's worth. I am very happy with the simplicity of just running it on
		normal bare hardware and don't intend on going back to a
		Kubernetes-based deployment any time soon.
	</div>
</div>

<p>
	In recent years, Kubernetes has powered my production infrastructure, and
	so finding a way to force Nextcloud into the stateless container model has
	been a goal of mine for a while, as that would provide the desired
	reliability and horizontal scaling capabilities.
</p>

<p>
	Like many legacy-ridden PHP applications, Nextcloud is heavily filesystem
	driven, and demands a mutable filesystem to operate in any capacity. This
	filesystem dependence was a large problem in the migration to Kubernetes,
	but with a bit of late-night research, a few experiments, and a couple
	shell scripts, I am finally content with the redundancy and elasticity of
	my current solution.
</p>

<p>
	As of now, I have achieved all the goals I set as a baseline for me to
	trust Nextcloud:

	<ul>
		<li>
			<mark>All user content is stored in object storage</mark>,
			removing "storage limit anxiety" and offering unmatched data
			resiliency compared to a single disk or an array of physical
			drives.
		</li>
		<li>
			<mark>Pods can be scaled</mark> with haste to meet demand during
			heavy usage periods, and utilise existing autoscaling concepts like
			HPA's.
		</li>
		<li>
			<mark>Containers are stateless</mark>, and do not share a
			filesystem with each other or the user data, meaning rollbacks and
			scaling carry minimal risk.
		</li>
	</ul>
</p>

<p>
	Over the past year I've slowly worked toward these goals, introducing one
	at a time to ensure stability.
</p>

<h2>Stage One: Object Storage</h2>

<p>
	Before even considering moving to Kubernetes, I had to first get object
	storage working. I had previously experimented with the "External Storages
	[sic]" support, which seemed to work well but wasn't as transparent as I
	wanted.
</p>

<p>
	Luckily, there is
	<a href="https://docs.nextcloud.com/server/15/admin_manual/configuration_files/primary_storage.html">official documentation</a>
	outlining how to set this up. It's quite simple, and just requires a few
	lines in <code>config.php</code>:
</p>

<div class="code">
	<div class="code-title">config.php</div>
	<pre class="chroma"><span class="cp">&lt;?php</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
  <span class="s1">&#39;objectstore&#39;</span> <span class="o">=&gt;</span> <span class="k">array</span> <span class="p">(</span>
    <span class="s1">&#39;class&#39;</span> <span class="o">=&gt;</span> <span class="s1">&#39;OC\\Files\\ObjectStore\\S3&#39;</span><span class="p">,</span>
    <span class="s1">&#39;arguments&#39;</span> <span class="o">=&gt;</span> <span class="k">array</span> <span class="p">(</span>
      <span class="s1">&#39;bucket&#39;</span> <span class="o">=&gt;</span> <span class="s1">&#39;my-nextcloud-data-bucket&#39;</span><span class="p">,</span>
      <span class="s1">&#39;autocreate&#39;</span> <span class="o">=&gt;</span> <span class="k">false</span><span class="p">,</span>
      <span class="s1">&#39;use_path_style&#39;</span> <span class="o">=&gt;</span> <span class="k">true</span><span class="p">,</span>
      <span class="s1">&#39;port&#39;</span> <span class="o">=&gt;</span> <span class="mi">9000</span><span class="p">,</span>
      <span class="s1">&#39;key&#39;</span> <span class="o">=&gt;</span> <span class="s1">&#39;AKIAxxx&#39;</span><span class="p">,</span>
      <span class="s1">&#39;secret&#39;</span> <span class="o">=&gt;</span> <span class="s1">&#39;asd+fx/xxxx=&#39;</span><span class="p">,</span>
      <span class="s1">&#39;hostname&#39;</span> <span class="o">=&gt;</span> <span class="s1">&#39;nxc-minio&#39;</span><span class="p">,</span>
      <span class="s1">&#39;use_ssl&#39;</span> <span class="o">=&gt;</span> <span class="k">false</span><span class="p">,</span>
      <span class="s1">&#39;region&#39;</span> <span class="o">=&gt;</span> <span class="s1">&#39;ca-central-1&#39;</span><span class="p">,</span>
    <span class="p">),</span>
  <span class="p">),</span>
<span class="p">[</span><span class="o">...</span><span class="p">]</span>
</pre>
</div>

<p>
	Astute readers may notice I am not pointing to S3 directly, but more on
	that later.
</p>

<p>
	The only thing I was unsure about was migrating existing data to a new
	primary backend, which I'm still not sure is possible. Nextcloud stores
	files in object storage as flat data-only blobs, so it's not just as simple
	as copying some files around.
</p>

<p>
	Ultimately, I ended up simply having a downtime event and manually
	re-syncing all the files for each user. This was still early on in our
	usage of Nextcloud--only dozens of gigabytes at most--so it wasn't a huge
	deal.
</p>

<p>
	With this, I now had a single server running Nextcloud, but with the
	benefits of more resilient storage, and an unlimited amount of it. I ran
	things this way for a while, but eventually the perceived risk of that
	single server wore me down. It was time to scale.
</p>

<h2>Stage Two: Kubernetes</h2>

<p>
	Obviously with Nextcloud being primarily filesystem-driven, just putting it
	on Kubernetes or using their official Docker images didn't really help
	much as we would still be bound to a single PVC containing the webroot.
</p>

<p>
	There were maybe a dozen different things I tried, from Azure
	Storage's CIFS integration in AKS, to running my own NFS server, and
	eventually settling on what I am currently running: <mark>webroot
	snapshots</mark>.
</p>

<p>
	Everything starts with a custom Docker image, which is mostly just a normal
	container with Apache HTTPD and PHP, running on an Alpine Linux base image.
	It was incredibly simple to get working, and contains a few dozen PHP
	extensions to maximize compatibility. I settled on this more generic
	approach because a lot of the world runs on PHP and while I try my best to
	avoid it, I may need to run some other PHP software in the future, and this
	leaves a door open.
</p>

<p>
	It's also probably worth noting this container currently sits at just over
	200MB, which is 73% smaller than the official Nextcloud images.
</p>

<p>
	<strong>Update:</strong> I have open-sourced this container image, if
	others want to use it directly or as a reference. You can find it on Github
	at <a href="https://github.com/alexblackie/prison">alexblackie/prison</a>.
</p>

<p>
	Inside of this PHP container I then place a couple of shell scripts: a
	container entrypoint, and one which generates a tarball of the webroot and
	stores it in object storage.
</p>

<p>
	This means that the container entrypoint fetches the webroot, extracts it,
	and only then starts HTTPD. By late-starting HTTPD synchronously, we can
	rely on Kubernetes' <code>readinessProbe</code>s to achieve zero-downtime
	rollouts. That entrypoint looks something like this:
</p>

<div class="code">
	<div class="code-title">start.sh</div>
	<pre class="chroma"><span class="cp">#!/bin/sh
</span><span class="cp"></span><span class="nb">set</span> -e

<span class="k">if</span> <span class="o">[</span> <span class="s2">&#34;</span><span class="nv">$WEBROOT_S3_URL</span><span class="s2">&#34;</span> <span class="o">=</span> <span class="s2">&#34;&#34;</span> <span class="o">]</span> <span class="p">;</span> <span class="k">then</span>
	<span class="c1"># [snip] some error message printing</span>
	<span class="nb">exit</span> <span class="m">1</span>
<span class="k">fi</span>

<span class="nb">echo</span> <span class="s2">&#34;Extracting webroot...&#34;</span>
aws s3 cp <span class="s2">&#34;</span><span class="nv">$WEBROOT_S3_URL</span><span class="s2">&#34;</span> /tmp/webroot.tar.gz
tar -xzf /tmp/webroot.tar.gz -C /srv/www
rm /tmp/webroot.tar.gz
chown -R apache:www-data /srv/www

<span class="nb">echo</span> <span class="s2">&#34;Starting httpd...&#34;</span>
/usr/sbin/httpd -D FOREGROUND -f /etc/apache2/httpd.conf
</pre>
</div>

<p>
	With this, we need only set <code>WEBROOT_S3_URL</code> in the container
	environment to the full URL to the webroot tarball on S3. Then on boot it
	fetches and extracts that tarball, ensures that HTTPD can access it, and
	finally starts HTTPD.
</p>

<p>
	This is paired with another script for updating the tarball in S3 with what
	is currently in the webroot:
</p>

<div class="code">
	<div class="code-title">update-source.sh</div>
<pre class="chroma"><span class="cp">#!/bin/sh
</span><span class="cp"></span><span class="nb">set</span> -e

<span class="nb">cd</span> /srv/www
	<span class="nb">echo</span> <span class="s2">&#34;Building source tarball...&#34;</span>
	tar -czf /tmp/webroot.tar.gz ./
	aws s3 cp /tmp/webroot.tar.gz <span class="s2">&#34;</span><span class="nv">$WEBROOT_S3_URL</span><span class="s2">&#34;</span>
<span class="nb">cd</span> -
</pre>
</div>

<p>
	This script requires twofold:

	<ol>
		<li>You must make whatever changes you need on a single pod only, which
			if what you're doing involves load balanced UI means you
			either need more complex dynamic routing configuration, or
			accept the possibility of degraded performance as you scale
			replicas to 1.</li>
		<li>You must <code>exec</code> into the pod yourself, and manually
			execute this script.</li>
	</ol>
</p>

<p>
	Updating is still a bit of a pain point, because I still just scale
	everything to 1 replica to make sure my traffic hits the same pod
	every time. I am planning on investigating some sort of "sticky
	routing" with cookies or alternative hostnames so that I can target
	a specific pod without compromising the performance of the entire
	service. As of right now, that's still a future improvement,
	however.
</p>

<p>
	Regarding point #2 above, I am additionally considering building a
	small Nextcloud plugin which would let me trigger the update script
	from the Admin UI, thereby eliminating the need to manually
	<code>exec</code> into a pod. Again, as of right now, this is only a
	theoretical future endeavour.
</p>

<p>
	These very simple little scripts are what made all the difference for me,
	and were the final piece of the puzzle that allowed me to deploy Nextcloud
	with confidence.
</p>

<h2>Stage Three: Minio Gateway</h2>

<p>
	The move from the official Nextcloud Docker images to my own Alpine-based
	image provided a noticable improvement in performance. Intoxicated by this
	unexpected gain, I then looked at other improvements that could be made.
	One such change, which I am now running in production, is
	<mark>Minio Gateway</mark>.
</p>

<p>
	<a href="https://docs.min.io/docs/minio-gateway-for-s3.html">Minio Gateway for S3</a>
	can be configured to <mark>store local caches of objects</mark>,
	which can dramatically increase performance on heavy workloads such
	as Nextcloud. It also can lead to lower transfer costs, as hot
	objects served from the cache don't have to be repeatedly downloaded
	from S3, which can become expensive.
</p>

<p>
	Currently, Minio Gateway is running alongside Nextcloud in the same
	Kubernetes namespace, and is only available internally. Right now, I
	am taking advantage of the somewhat-generous temporary disk provided
	on the Kubernetes nodes I am running, and am mounting a temporary
	<code>emptyDir</code> into Minio's container. Since this data is
	ephemeral and its performance is not critical to running the app,
	the potential limits of temporary storage and the potential loss of
	the cache is unimportant. This gives me approximately 70GB of space
	to work with, which is plenty.
</p>

<p>
	Minio Gateway has a few other features that I have not yet
	experimented with, but plan to in the future. Namely, it can provide
	S3-compliant access to non-S3 object storage such as GCS or Azure
	Storage -- this means I am finally decoupled from AWS and can mirror
	or lift-and-shift my Nextcloud data to any object storage service
	supported by Minio Gateway, and have to change very little on the
	Nextcloud side.
</p>

<h2>... Profit?</h2>

<p>
	Now that everything is working and has been running smoothly for a
	while, I have been reaping the benefits. <mark>Nextcloud has been
	operating faster than it has ever been</mark>, and my trust in it has
	strengthened greatly.
</p>

<p>
	This new "decoupled" infrastructure also has helped to simplify and
	hone my backup strategies and disaster recovery scenarios, as all
	important data -- including configuration and the webroot -- can
	just be cloned from object storage to an off-site location.
</p>

<p>
	This also means I have reached the point where <mark>I can redeploy the
	service to any Kubernetes cluster in seconds</mark>, and restore the data
	to any object storage backend supported by Minio Gateway (I plan to
	document my DR strategies in a future article).
</p>

<p>
	I have always hesitated to commit fully to Nextcloud, simply because
	of how difficult it is to scale long-term without a multi-million
	dollar NetApp SAN. This collection of Docker containers, shell
	scripts, and object storage buckets now has now solved my concerns
	around scaling and performance, and I have never felt more confident
	about its resiliency.
</p>

<p>
	That said, I still have at least four backups at all time, in three
	different locations, and in two different formats. Just in case. I
	am confident, not foolish.
</p>


guid: https://www.alexblackie.com/articles/better-saas-through-productisation/
title: Resiliency in SaaS codebases through productisation
author: Alex Blackie
url: https://www.alexblackie.com/articles/better-saas-through-productisation/
feedurl: https://www.alexblackie.com/feed.xml
content_mime_type: text/html
content:
 		<section id="intro">
	<p><big>
		In the world of private codebases, and especially SaaS, you can easily
		fall into the trap of thinking, "well, it's OK because no one is going
		to see this." A mindset like that can wreck a codebase, and can fuel
		micro-decisions that lead to painful long-term mistakes.
	</big></p>

	<p>
		First, regarding "productisation," <a href="https://en.wiktionary.org/w/index.php?title=productization&oldid=51827427">Wiktionary</a> defines it as
	</p>

	<blockquote>
		The act of modifying something, such as a concept or a tool internal to
		an organization, to make it suitable as a commercial product.
	</blockquote>

	<p>
		To put it another way, I find it very helpful to frame things from the
		perspective of "what if someone else had to run this software?" This is
		obviously something that open source projects have to think about quite
		a bit, so you could also frame it as "what if this was open source?"
	</p>

	<p>
		There are many facets to this, but I want to dive into just two:
		<mark>reproducibility</mark> and <mark>portability</mark>. These
		concepts frequently come up in conversations around package managers
		and software releases, but they are also quite pertinent to SaaS.
	</p>
</section>

<section id="reproducibility">
	<h2>Reproducibility: I should be able to run the thing.</h2>

	<aside class="aside">
		<p>
			Note: this is not referring to "reproducibility" in terms of
			package integrity, à la <a href="https://reproducible-builds.org/">reproducible builds</a>.
			I love that project and its efforts, but I am talking about
			something else here.
		</p>
	</aside>

	<p>
		A while back, I was charged with spinning up a validation environment
		for a SaaS product to provide a "safe" place for a QA team to mess
		around with upcoming features. The goal was to have this sit somewhere
		in between our "staging" environment, which was often quite buggy and
		more for internal developers, and production, which we needed to keep
		as stable as possible for customers.
	</p>

	<p>
		"Simple enough." I said. Surely, we already have a couple other
		environments, we can just add one more! The problem was that no one had
		spun up a new environment for <em>several years</em>, and everyone
		involved in doing so had long since left the company.
	</p>

	<p>
		In the end, the new "QA" environment never fully worked. It
		<em>mostly</em> worked, but several components would fail to integrate
		or would mysteriously break in ways no one really understood. We tried
		to copy production as close as we could, and still we failed to
		replicate it.
	</p>

	<p>
		This is what I mean by reproducibility: <mark>anyone should be able to
			run the software you make</mark>. A production environment
		shouldn't be an elusive, one-off occurence; you should be able to
		recreate production, and the steps should be obvious.
	</p>

	<p>
		This naturally leads into discussions about "DevOps" and
		"Infrastructure as Code," both of which are definitely a key part of
		the solution to this. Being able to understand and replicate
		infrastructure based on playbooks or manifests solves a huge part of
		the problem. But it's not the whole story.
	</p>

	<p>
		Even with a fully-automated cloud, there will always be some manual
		work, keys to rotate, service integrations to set up, etc. These
		service dependencies have to be documented and fail safely.
	</p>

	<p>
		In a SaaS world, it's incredibly easy to add a new service
		dependency--for this case, let's use a hypothetical geocoding API--add
		the keys to production, and never think about it again. By applying a
		product focus, it becomes clearer that this dependency should be
		optional, should be documented, and should fail gracefully.
	</p>

	<p>
		These sorts of third-party dependencies hold their own configuration,
		and often that configuration is difficult to export or automate. Do
		certain options need to be enabled for your application? Do certain
		addons need to be active? These are all questions that should have
		documented and discoverable answers -- in code or otherwise.
	</p>

	<p>
		Additionally, external service dependencies should be resilient against
		missing configuration. Configuration should be validated and warnings
		should exist; and if a service is not critical to the operation of the
		application, it shouldn't block the rest of the app just because it's
		unconfigured.
	</p>

	<p>
		This level of "configuration resiliency" helps during development, as devs
		don't need to get 1000 test API keys just to boot a console; and,
		additionally, helps you build better software overall. By baking in
		resiliency, you make your app more flexible in the face of failure,
		where you <em>know</em> that you can disable many parts of it and it
		won't break the entire experience.
	</p>

	<p>
		In essence, <mark>you should be the downstream and the upstream</mark>;
		you should both develop the software, but also treat yourself as a
		consumer of it. Don't take shortcuts to production just because you own
		the whole pipeline.
	</p>

	<p>
		This makes it easier to deploy multiple versions of your service
		-- whether that means on-premises for a large enterprise customer, or
		in a different geographic region for compliance purposes. Ensuring your
		codebase is resilient and reproducible could be a business-value
		feature!
	</p>
</section>

<section id="portability">
	<h2>Portability: hedge against your integrations.</h2>

	<p>
		Being able to replicate, say, a bunch of AWS service configuration is
		powerful, but it still leaves you tightly-coupled to AWS. That doesn't
		help much if we want to replicate production in an on-premises
		environment, or even locally on a laptop.
	</p>

	<p>
		Just as building software to be resilient against missing or invalid
		dependencies, it can be even more advantageous to take it a step
		further, and make your software resilient <em>across vendors</em>.
	</p>

	<p>
		This is not a new concept; really, all this means is <mark>build the
			right abstractions</mark>. Don't leak third-party libraries around
		your codebase, and build your interfaces with interchangeable backend
		implementations.
	</p>

	<p>
		For example, integrating object storage is a very common thing to
		sprinkle across your codebase. Building a proper entrypoint and
		abstraction around it will allow you to swap out the backing
		implementation without having to modify your application.
	</p>

	<p>
		In the object storage example, something like Rails'
		<code>ActiveStorage</code> is a great example of this abstraction.
		Depending on the environment and requirements, you can swap out backing
		storage, be it on-disk, to cloud, etc. Your code calls a standard API,
		and doesn't need to change to support a variety of implementations.
	</p>

	<p>
		These sort of abstractions can facilitate portability, allowing you to
		run your service almost anywhere, with any technology you can build an
		implementation for. Does an enterprise customer need to store file
		uploads on a SAN instead of object storage? Is AWS down and you need to
		migrate production workloads to a DR site on a different provider? All
		of this can be yours, for the low-low price of strong abstractions.
	</p>

	<p>
		However, as with everything, <mark>you need to ensure you don't go too
			far</mark>; you must weigh your dependencies versus any downsides
		of abstraction.
	</p>

	<p>
		For example, heavy abstractions on top of a database (such as
		PostgreSQL) can add more trouble than they're worth, and have
		performance impacts compared to a more optimized, direct
		implementation. For these, the focus should shift from abstractions, to
		choosing the right APIs for your application.
	</p>
</section>

<section id="standardapis">
	<h2>Portability, the sequel: standards are good, actually</h2>

	<p>
		For cases where making your own standard interface for dependencies is
		not worth it or not possible, using standardized APIs is the next best
		thing.
	</p>

	<p>
		Take the aforementioned database example; there can be great benefits
		in performance to optimise your usage of a database to take advantage
		of a specific implementation. What matters in this case is the
		technology you choose.
	</p>

	<aside class="aside">
		<p>
			I make a lot of comparisons to fancy cloud database offerings in
			this section, but perhaps it's worth disclaiming that I have not
			actually used any of these first hand. I'm a Postgres man, forever
			and always.
		</p>
	</aside>

	<p>
		You can still achieve portability and resiliency through non-abstracted
		choices, as long as those choices are either open or standardized. In
		the database example, you can spin up a PostgreSQL server anywhere,
		even on your own hardware. There isn't much that is stopping you from
		moving providers or running in other environments.
	</p>

	<p>
		For an opposite example: Google Spanner on GCP, or AWS's DynamoDB.
		While both are very powerful technologies, they are unique to their
		respective clouds and use non-standard APIs. You cannot move to another
		cloud without significantly changing your application.
	</p>

	<p>
		To be clear, this isn't a dig on "the cloud" -- this is a dig on vendor
		lock-in. Where Google's Spanner invented their own interface, Microsoft
		Azure's CosmosDB gives the consumer the option of various different
		standard APIs, including SQL- or MongoDB-compatible endpoints. This is
		a great win by combining a standard API (one that means a developer can
		run a different MongoDB-like service locally) with the power of an
		integrated and globally-scalable cloud solution in production.
	</p>
</section>

<section id="conclusion">
	<h2>Productisation doesn't mean open source</h2>

	<p>
		While I think it helps with the thought process to ask "what if this
		was open source?", that doesn't mean you have to become open source to
		get the benefits of that mindset. In my experience, just being aware of
		the codebase you are building, and keeping future generations in mind,
		is enough to hopefully put you on the right paths.
	</p>

	<p>
		Hopefully product-focused thinking can help you clean up your codebases
		as well.
	</p>
</section>


guid: https://www.alexblackie.com/articles/simple-config-uuids-ecto/
title: Simple configuration for UUID primary keys in Ecto
author: Alex Blackie
url: https://www.alexblackie.com/articles/simple-config-uuids-ecto/
feedurl: https://www.alexblackie.com/feed.xml
content_mime_type: text/html
content:
 		<p>
	Using UUID's as primary keys can have many benefits, and if you're using
	PostgreSQL you have the advantage of solid performance and efficient
	storage of those UUIDs leading to good scalability even compared to
	<code>bigint</code>.
</p>

<p>
	With <a href="https://github.com/elixir-ecto/ecto/">Ecto</a>, using UUID
	primary keys is very easy, however most guides out there are still written
	for Ecto 1.x or 2.x -- in modern times using UUIDs is <em>even easier</em>
	with Ecto 3+.
</p>

<p>
	In your <code>config.exs</code>, simply configure your repo to use the
	<code>:binary_id</code> by default for primary keys and foreign keys:
</p>

<pre class="chroma"><span class="n">config</span> <span class="ss">:my_app</span><span class="p">,</span> <span class="nc">MyApp.Repo</span><span class="p">,</span>
  <span class="ss">migration_primary_key</span><span class="p">:</span> <span class="p">[</span><span class="ss">name</span><span class="p">:</span> <span class="ss">:id</span><span class="p">,</span> <span class="ss">type</span><span class="p">:</span> <span class="ss">:binary_id</span><span class="p">],</span>
  <span class="ss">migration_foreign_key</span><span class="p">:</span> <span class="p">[</span><span class="ss">column</span><span class="p">:</span> <span class="ss">:id</span><span class="p">,</span> <span class="ss">type</span><span class="p">:</span> <span class="ss">:binary_id</span><span class="p">]</span>
</pre>

<p>
	You don't have to modify your migrations at all; Ecto will use UUIDs for
	keys unless you tell it otherwise, just as it was previously with
	<code>bigint</code> keys.
</p>

<p>
	The only other required step is to annotate your schemas so they use
	<code>:binary_id</code> as well. You can do this on each individually, but
	I generally prefer to do this in a behaviour that wraps
	<code>Ecto.Schema</code>. For example,
</p>

<pre class="chroma"><span class="n">defmodule</span> <span class="nc">MyApp.Schema</span> <span class="n">do</span>
  <span class="na">@moduledoc</span> <span class="sh">&#34;&#34;&#34;
</span><span class="sh">  Behaviour to replace `Ecto.Schema` to set some defaults across all schemas in
</span><span class="sh">  the codebase -- namely, UUID primary keys.
</span><span class="sh">  &#34;&#34;&#34;</span>

  <span class="n">defmacro</span> <span class="n">__using__</span><span class="p">(</span><span class="n">_opts</span><span class="p">)</span> <span class="n">do</span>
    <span class="n">quote</span> <span class="n">do</span>
      <span class="n">use</span> <span class="nc">Ecto.Schema</span>

      <span class="na">@primary_key</span> <span class="p">{</span><span class="ss">:id</span><span class="p">,</span> <span class="ss">:binary_id</span><span class="p">,</span> <span class="ss">autogenerate</span><span class="p">:</span> <span class="n">true</span><span class="p">}</span>
      <span class="na">@foreign_key_type</span> <span class="ss">:binary_id</span>
    <span class="n">end</span>
  <span class="n">end</span>
<span class="n">end</span>
</pre>

<p>
	Then you can use this in your schemas:
</p>

<pre class="chroma"><span class="n">defmodule</span> <span class="nc">MyApp.User</span> <span class="n">do</span>
  <span class="n">use</span> <span class="nc">MyApp.Schema</span>

  <span class="n">...</span>
<span class="n">end</span>
</pre>

<p>
	And that's all there is to it! I hope this helped you avoid setting
	<code>primary_key: false</code> in every migration you write.
</p>


guid: https://www.alexblackie.com/articles/continuous-delivery-aks-acr-azure-terraform/
title: Continuous Delivery to AKS with a Service Principal, Securely
author: Alex Blackie
url: https://www.alexblackie.com/articles/continuous-delivery-aks-acr-azure-terraform/
feedurl: https://www.alexblackie.com/feed.xml
content_mime_type: text/html
content:
 		<p>
	Azure has some great features, and leads the industry in a few different
	ways, especially around region support and disaster recovery. But, as with
	everything, all these great features comes tradeoffs. Azure Active
	Directory (AAD) is one of those tradeoffs.
</p>

<p>
	Our story begins with configuring continuous delivery. It was all going
	smoothly until we had to authenticate with Azure Container Registry (ACR)
	and Azure Kubernetes Service (AKS). What, in AWS, would have just been an
	IAM role, in Azure is a full "Enterprise Application" within AAD.
</p>

<p>
	This "Enterprise Application" is a full-blown (private) Azure Marketplace
	OAuth2 integration. You can take this thing as far as building an entire
	product on top of Azure. However, in our case, we just want to push some
	Docker containers from CI.
</p>

<p>
	To make things worse, finding examples or documentation on building
	restrictive application roles can be often fruitless. Almost every blog
	post, documentation page, or StackOverflow answer just had the bare example
	using the Azure CLI:
</p>

<pre><code>$ az ad sp create-for-rbac --subscription &lt;id&gt;</code></pre>

<p>
	This command will automatically create an Enterprise Application and a
	Service Principal behind the scenes, and then spit out the credentials for
	it right away. Easy, right?
</p>

<p>
	Well, by default it will be scoped to the entire subscription, which means
	it will have full, unfettered access to every resource in your
	subscription. It's basically like deploying a web app using
	<code>root</code>. If these credentials leak, it could bring down your
	entire stack, or possibly even your entire company.
</p>

<p>
	You can lock it down a bit using <code>--scope</code>, to scope the access
	down to even just a single resource. This is great, but not quite far
	enough. It's significantly better than access to the entire account, but
	unfettered access to a production Kubernetes cluster is also rather
	undesirable.
</p>

<p>
	We need to go deeper.
</p>

<h3> Terraforming AAD</h3>

<p>
	Luckily, there is an official <code>hashicorp/azuread</code> provider for
	Terraform! This, along with some help from the usual
	<code>hashicorp/azurerm</code> provider, will allow us to entirely
	automate, audit, and track changes to our AAD service principals.
</p>

<p>
	The first step is to get the Application and Service Principal created.
	That is fairly straightforward:
</p>

<pre class="chroma"><span class="k">resource</span> <span class="s2">&#34;azuread_application&#34; &#34;appcd&#34;</span> {
<span class="n">  display_name</span> <span class="o">=</span> <span class="s2">&#34;MyAppCD${terraform.workspace}&#34;</span>
}

<span class="k">resource</span> <span class="s2">&#34;azuread_service_principal&#34; &#34;appcd&#34;</span> {
<span class="n">  application_id</span> <span class="o">=</span> <span class="k">azuread_application</span><span class="p">.</span><span class="k">appcd</span><span class="p">.</span><span class="k">application_id</span>
}

<span class="k">resource</span> <span class="s2">&#34;azuread_application_password&#34; &#34;appcd&#34;</span> {
<span class="n">  application_object_id</span> <span class="o">=</span> <span class="k">azuread_application</span><span class="p">.</span><span class="k">appcd</span><span class="p">.</span><span class="k">object_id</span>
<span class="n">  end_date_relative</span>     <span class="o">=</span> <span class="s2">&#34;8766h&#34;</span><span class="c1"> # 1.00068 years
</span><span class="c1"></span>}
</pre>

<p>
	Nothing should be too surprising here. The "application password" is
	equivalent to the "client secret". It will be generated for us, and we'll
	use an <code>output</code> later to get access to its value.
</p>

<p>
	The fun begins when we start to define some custom roles. This is why we're
	here in the first place: we want to apply only the most minimal set of
	permissions to allow CI to roll out containers and update our Kubernetes
	deployments.
</p>

<p>
	Let's start with Azure Container Registry:
</p>

<pre class="chroma"><span class="c1"># Similar to the built-in &#34;AcrPush&#34;, but also allows queuing remote builds.
</span><span class="c1"></span><span class="k">resource</span> <span class="s2">&#34;azurerm_role_definition&#34; &#34;acrBuild&#34;</span> {
<span class="n">  name</span>  <span class="o">=</span> <span class="s2">&#34;MyAppACRBuild${terraform.workspace}&#34;</span>
<span class="n">  scope</span> <span class="o">=</span> <span class="k">azurerm_resource_group</span><span class="p">.</span><span class="k">main</span><span class="p">.</span><span class="k">id</span>

  <span class="k">permissions</span> {
<span class="n">    actions</span> <span class="o">=</span> <span class="p">[</span>
      <span class="s2">&#34;Microsoft.ContainerRegistry/registries/read&#34;</span><span class="p">,</span>
      <span class="s2">&#34;Microsoft.ContainerRegistry/registries/pull/read&#34;</span><span class="p">,</span>
      <span class="s2">&#34;Microsoft.ContainerRegistry/registries/push/write&#34;</span><span class="p">,</span>
      <span class="s2">&#34;Microsoft.ContainerRegistry/registries/scheduleRun/action&#34;</span><span class="p">,</span>
      <span class="s2">&#34;Microsoft.ContainerRegistry/registries/runs/*&#34;</span><span class="p">,</span>
      <span class="s2">&#34;Microsoft.ContainerRegistry/registries/listBuildSourceUploadUrl/action&#34;</span><span class="p">,</span>
    <span class="p">]</span>

<span class="n">    not_actions</span> <span class="o">=</span> <span class="p">[]</span>
  }
}
</pre>

<p>
	We rely quite heavily on ACR's "runs", which are remote-builds of
	containers within ACR's infrastructure. This is much safer than building
	them on CI as it can be done entirely within Azure instead of on some
	shared CI platform. It also means we don't have to worry about pushing
	images around or dealing with registry credentials in CI.
</p>

<p>
	This role is a bit "loose" in that it also allows traditional push/pull
	access, but this should be fine, as it's not opening up any more
	capability. If we need to, for whatever reason, switch to local container
	builds in CI, I don't want to have to bother with changing the role just
	for that.
</p>

<p>
	Once we have our containers, we need to access AKS. This is where the
	custom roles really shine. Before, we would have had to have given full
	cluster access just so CI could update some image tags. Now, we can lock
	things down rather tightly:
</p>

<pre class="chroma"><span class="k">resource</span> <span class="s2">&#34;azurerm_role_definition&#34; &#34;aksDeploy&#34;</span> {
<span class="n">  name</span>  <span class="o">=</span> <span class="s2">&#34;MyAppAKSDeploy${terraform.workspace}&#34;</span>
<span class="n">  scope</span> <span class="o">=</span> <span class="k">azurerm_resource_group</span><span class="p">.</span><span class="k">main</span><span class="p">.</span><span class="k">id</span>

  <span class="k">permissions</span> {
<span class="n">    actions</span> <span class="o">=</span> <span class="p">[</span>
      <span class="s2">&#34;Microsoft.ContainerService/managedClusters/read&#34;</span><span class="p">,</span>
      <span class="s2">&#34;Microsoft.ContainerService/managedClusters/accessProfiles/listCredential/action&#34;</span><span class="p">,</span>
    <span class="p">]</span>

<span class="n">    data_actions</span> <span class="o">=</span> <span class="p">[</span>
      <span class="s2">&#34;Microsoft.ContainerService/managedClusters/apps/deployments/read&#34;</span><span class="p">,</span>
      <span class="s2">&#34;Microsoft.ContainerService/managedClusters/apps/deployments/write&#34;</span><span class="p">,</span>
    <span class="p">]</span>

<span class="n">    not_actions</span> <span class="o">=</span> <span class="p">[]</span>
  }
}
</pre>

<p>
	This policy allows our Service Principal to <strong>only</strong> get their
	credentials, and read and write to deployments. They can't delete anything,
	can't create new resources, only updates and only to deployments.
</p>

<p>
	This is still "open" in that we aren't specifying <em>which
		deployments</em> it should update, but this is still much better than
	anything else I've seen so far. There is likely a way for us to take this
	further, but for now this is sufficient.
</p>

<p>
	Which leads us to the assignments. The Role Assignments are where we can
	lock things down even further by only scoping these permissions down to
	specific resources.
</p>

<pre class="chroma"><span class="k">resource</span> <span class="s2">&#34;azurerm_role_assignment&#34; &#34;appcdAKS&#34;</span> {
<span class="n">  scope</span>              <span class="o">=</span> <span class="k">azurerm_kubernetes_cluster</span><span class="p">.</span><span class="k">main</span><span class="p">.</span><span class="k">id</span>
<span class="n">  role_definition_id</span> <span class="o">=</span> <span class="k">azurerm_role_definition</span><span class="p">.</span><span class="k">aksDeploy</span><span class="p">.</span><span class="k">role_definition_resource_id</span>
<span class="n">  principal_id</span>       <span class="o">=</span> <span class="k">azuread_service_principal</span><span class="p">.</span><span class="k">appcd</span><span class="p">.</span><span class="k">id</span>
}

<span class="k">resource</span> <span class="s2">&#34;azurerm_role_assignment&#34; &#34;appcdACR&#34;</span> {
<span class="n">  scope</span>              <span class="o">=</span> <span class="k">azurerm_container_registry</span><span class="p">.</span><span class="k">main</span><span class="p">.</span><span class="k">id</span>
<span class="n">  role_definition_id</span> <span class="o">=</span> <span class="k">azurerm_role_definition</span><span class="p">.</span><span class="k">acrBuild</span><span class="p">.</span><span class="k">role_definition_resource_id</span>
<span class="n">  principal_id</span>       <span class="o">=</span> <span class="k">azuread_service_principal</span><span class="p">.</span><span class="k">appcd</span><span class="p">.</span><span class="k">id</span>
}
</pre>

<p>
	With this, we should have everything we need. We can then assemble an
	output with the equivalent JSON blob that the <code>az</code> command line
	spits out:
</p>

<pre class="chroma"><span class="k">data</span> <span class="s2">&#34;azurerm_subscription&#34; &#34;primary&#34;</span> { }

<span class="k">output</span> <span class="s2">&#34;appcd_service_principal&#34;</span> {
<span class="n">  value</span> <span class="o">=</span> <span class="err">&lt;&lt;</span><span class="k">EOF</span>
{
  <span class="s2">&#34;clientId&#34;: &#34;${azuread_application.appcd.application_id}&#34;</span><span class="p">,</span>
  <span class="s2">&#34;clientSecret&#34;: &#34;${azuread_application_password.appcd.value}&#34;</span><span class="p">,</span>
  <span class="s2">&#34;subscriptionId&#34;: &#34;${data.azurerm_subscription.primary.subscription_id}&#34;</span><span class="p">,</span>
  <span class="s2">&#34;tenantId&#34;: &#34;${data.azurerm_subscription.primary.tenant_id}&#34;</span><span class="p">,</span>
  <span class="s2">&#34;activeDirectoryEndpointUrl&#34;: &#34;https://login.microsoftonline.com&#34;</span><span class="p">,</span>
  <span class="s2">&#34;resourceManagerEndpointUrl&#34;: &#34;https://management.azure.com/&#34;</span><span class="p">,</span>
  <span class="s2">&#34;activeDirectoryGraphResourceId&#34;: &#34;https://graph.windows.net/&#34;</span><span class="p">,</span>
  <span class="s2">&#34;sqlManagementEndpointUrl&#34;: &#34;https://management.core.windows.net:8443/&#34;</span><span class="p">,</span>
  <span class="s2">&#34;galleryEndpointUrl&#34;: &#34;https://gallery.azure.com/&#34;</span><span class="p">,</span>
  <span class="s2">&#34;managementEndpointUrl&#34;: &#34;https://management.core.windows.net/&#34;</span>
}
  <span class="k">EOF</span>
}
</pre>

<p>
	This JSON blob will be output at the end of <code>terraform apply</code>
	with all the right values populated. Take it, put it in a secret, and then
	you can use it to authenticate with Azure from CI and deploy what you need.
</p>

<h3>Where to go from here</h3>

<p>
	Azure AD is a complex and often unforgiving beast. But given a lot of time
	and learning, it can be honed into a very powerful tool that makes you and
	your infrastructure safer.
</p>

<p>
	Getting this far took many hours of trial-and-error and research. It was
	not a particularly pleasant process. However, the result was worth it, and
	I'm pretty happy with how it turned out.
</p>

<p>
	Figuring out which actions you need is by far the worst part of this process.
	Perhaps the best resource is <a href="https://docs.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations">this enormous list of every <code>action</code> in Azure</a>.
	It's a lot to search through, but you can usually figure out which ones you
	want.
</p>

<p>
	And, if it comes down to it, the errors returned by Azure when you are
	missing a permission often include the exact action required so you can
	just copy-and-paste that into your role and try again.
</p>

<p>
	Hopefully you were able to take something from this article, and here's to
	more secure deployments.
</p>


guid: https://www.alexblackie.com/articles/automate-commit-coauthors-pairing/
title: Automate commit co-authors when pairing
author: Alex Blackie
url: https://www.alexblackie.com/articles/automate-commit-coauthors-pairing/
feedurl: https://www.alexblackie.com/feed.xml
content_mime_type: text/html
content:
 		<p>
	Pairing is a fantastic way to quickly bounce ideas around, get help with
	something, or even as a tool for mentoring and growth. In recent times,
	most Git-based tools have adopted the <code>Co-Authored-By</code> trailer
	to identify when there are multiple authors to a commit. This is great, but
	often a pain to type out manually, especially if there are many commits or
	multiple people involved.
</p>

<p>
	Asking around at work, there were a few things people had &mdash; everything
	from vim macros, to custom shell functions &mdash; but nothing quite like what I
	wanted. So I wrote a script to do it for me.
</p>

<p>
	It begins with <code>git-pair</code> (to be used as <code>git pair</code>).
	This script enumerates all the commit authors in the current repo, offers
	them to you <a href="https://github.com/jhawthorn/fzy">with a fuzzy-finder</a>,
	and then appends those people as co-authors on every commit until you run a
	companion script to end the session.
</p>

<div class="code">
	<div class="code-title">~/bin/git-pair</div>
<pre class="chroma"><span class="cp">#!/usr/bin/env bash
</span><span class="cp"></span>
<span class="nb">set</span> -uf -o pipefail

<span class="nv">TEMPLATE_PATH</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$HOME</span><span class="s2">/.config/git/template&#34;</span>

<span class="nv">TRAILERS</span><span class="o">=</span><span class="s2">&#34;&#34;</span>

<span class="k">while</span> <span class="nb">true</span> <span class="p">;</span> <span class="k">do</span>
	<span class="nv">pick</span><span class="o">=</span><span class="s2">&#34;</span><span class="k">$(</span><span class="nb">echo</span> -e <span class="s2">&#34;</span><span class="k">$(</span>git shortlog -e -s <span class="p">|</span> cut -f2<span class="k">)</span><span class="s2">&#34;</span> <span class="p">|</span> fzy<span class="k">)</span><span class="s2">&#34;</span>

	<span class="k">if</span> <span class="o">[[</span> <span class="s2">&#34;</span><span class="nv">$pick</span><span class="s2">&#34;</span> !<span class="o">=</span> <span class="s2">&#34;&#34;</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
		<span class="nv">TRAILERS</span><span class="o">+=</span><span class="s2">&#34;Co-authored-by: </span><span class="nv">$pick</span><span class="s2">\n&#34;</span>
		<span class="nb">echo</span> <span class="s2">&#34;Added co-author: </span><span class="nv">$pick</span><span class="s2">&#34;</span>
	<span class="k">else</span>
		<span class="nb">break</span>
	<span class="k">fi</span>
<span class="k">done</span>

<span class="k">if</span> <span class="o">[[</span> <span class="s2">&#34;</span><span class="nv">$TRAILERS</span><span class="s2">&#34;</span> !<span class="o">=</span> <span class="s2">&#34;&#34;</span> <span class="o">]]</span> <span class="p">;</span> <span class="k">then</span>
	<span class="nb">echo</span> -e <span class="s2">&#34;\n&#34;</span> &gt; <span class="nv">$TEMPLATE_PATH</span>
	<span class="nb">echo</span> -e <span class="s2">&#34;</span><span class="nv">$TRAILERS</span><span class="s2">&#34;</span> &gt;&gt; <span class="nv">$TEMPLATE_PATH</span>

	<span class="nb">echo</span> <span class="s2">&#34;🧑‍💻Pairing mode enabled. Run &#39;git unpair&#39; to end session.&#34;</span>
<span class="k">fi</span>
</pre>

</div>

<p>
	This uses Git's "commit template" feature to hack in the trailers for us
	automatically on each commit. This requires you to have a bit of Git
	configuration to point it at a template file:
</p>

<div class="code">
	<div class="code-title">~/.config/git/config</div>
<pre class="chroma"><span class="k">[commit]</span>
<span class="na">template</span> <span class="o">=</span> <span class="s">~/.config/git/template</span>
</pre>
</div>

<p>
	Unfortunately, as of 2.32 at least, template files <em>need to exist</em>
	at all times, even if they are empty. Make sure to touch it before
	proceeding to keep Git happy.
</p>

<p>
	As you may have noticed, we also will want <code>git-unpair</code> to end a
	pairing session. This script is much simpler, however, as "ending a
	session" really just means "emptying the commit template."
</p>

<div class="code">
	<div class="code-title">~/bin/git-unpair</div>
<pre class="chroma"><span class="cp">#!/usr/bin/env bash
</span><span class="cp"></span>
<span class="nb">set</span> -euf -o pipefail

<span class="nb">echo</span> &gt; <span class="s2">&#34;</span><span class="nv">$HOME</span><span class="s2">/.config/git/template&#34;</span>
<span class="nb">echo</span> <span class="s2">&#34;👋 Disabled pairing mode.&#34;</span>
</pre>
</div>

<p>
	One quick other feature, however: what if you're pairing with someone who
	doesn't have commits in the current repo? Well, we can supplement the
	commit author list with a static list of "known colleagues" so that even in
	a blank git repo we have a list of people to choose from.
</p>

<div class="code">
	<div class="code-title">~/bin/git-pair</div>
<pre class="chroma">TEMPLATE_PATH=&#34;$HOME/.config/git/template&#34;

<span class="gi">+if [[ -e &#34;$HOME/.colleagues.txt&#34; ]] ; then
</span><span class="gi">+	COLLEAGUES=&#34;$(cat $HOME/.colleagues.txt)&#34;
</span><span class="gi">+else
</span><span class="gi">+	COLLEAGUES=&#34;&#34;
</span><span class="gi">+fi
</span><span class="gi"></span>
TRAILERS=&#34;&#34;

while true ; do
<span class="gd">-	pick=&#34;$(echo -e &#34;$(git shortlog -e -s | cut -f2)&#34; | fzy)&#34;
</span><span class="gd"></span><span class="gi">+	pick=&#34;$(echo -e &#34;$COLLEAGUES\n$(git shortlog -e -s | cut -f2)&#34; | uniq | fzy)&#34;
</span><span class="gi"></span>
	if [[ &#34;$pick&#34; != &#34;&#34; ]]; then
</pre>
	</pre>
</div>

<p>
	Then you can automate or manually maintain <code>~/.colleagues.txt</code>,
	which should be a plain text list of <code>Name &lt;email&gt;</code> lines.
	We pass everything through <code>uniq</code> as well, to make sure any
	people in both lists only get one entry.
</p>

<p>
	With all of that in place, you can now pair! It should be as easy as:

	<ol>
		<li><code>git pair</code></li>
		<li>Fuzzy-select people (hit <code>^D</code> when done)</li>
		<li>Make some commits</li>
		<li><code>git unpair</code></li>
	</ol>
</p>

<p>
	Hopefully this makes your pairing sessions easier, with significantly less
	awkward typing of trailers after every commit. The full source of the script, and all my other scripts and configuration, are <a href="https://github.com/alexblackie/dotfiles/tree/e1c1136e8efb278d0a5a8493bf16376cb15e328a/configs/bin">in my dotfiles repository</a>.
</p>

<p>
	Happy pairing 🧑‍💻✨
</p>


guid: https://www.alexblackie.com/articles/email-authenticity-dkim-spf-dmarc/
title: Email Authenticity 101: DKIM, DMARC, and SPF
author: Alex Blackie
url: https://www.alexblackie.com/articles/email-authenticity-dkim-spf-dmarc/
feedurl: https://www.alexblackie.com/feed.xml
content_mime_type: text/html
content:
 		<div class="note">
	<p class="note--body">
		<strong>Prefer to listen?</strong> I was a guest on
		<a href="https://packetpushers.net/podcast/heavy-networking-602-all-about-spf-dkim-dmarc-email-security/">Heavy Networking Episode 602</a>
		and covered all the details in this post!
	</p>
</div>

<p>
	<big>
		Password resets, two factor codes, business secrets, private
		conversations&hellip; Email is at the centre of most of life and
		business, and so we must ensure it is trustworthy and authentic.
	</big>
</p>

<p>
	If you use email with your own domain, a lot of the burden of authenticity
	has suddenly shifted from your service provider to you. This guide will
	hopefully give you the information and practices you need to keep your
	domain's email authentic and less vulnerable to spoofing.
</p>

<p>
	We'll cover the three major components of modern email domain security:
	<strong>DKIM</strong> for signing, <strong>SPF</strong> for sender
	verification, and <strong>DMARC</strong> for stricter enforcement of the
	other two. It is assumed the reader has a basic understanding of DNS and
	has experience using email with their own domain.
</p>

<h2>1. SPF</h2>

<p>
	SPF, or Sender Policy Framework, is one of the most basic email
	verification technologies, and is the easiest and more common protection.
	Often service providers will give you the DNS record contents you need to
	simply copy-paste during setup.
</p>

<p>
	It takes the form of a DNS <code>TXT</code> record on whatever domain you
	are sending email from. It looks something like this:
</p>

<pre><code>"v=spf1 include:spf.messagingengine.com -all"</code></pre>


<aside class="aside">
	<p>
		<strong>Note</strong>:
		Keep in mind that SPF only supports 10 recursive lookups
		(<code>a</code>, <code>mx</code>, <code>ptr</code>,
		<code>exists</code>, or <code>include</code> are recursive). However,
		if you need more than this, it's probably a symptom of much worse IT
		bloat.
	</p>
</aside>

<p>
	At its core, SPF is just a list of IP addresses that are authorized to send
	email from your domain. This can be of a few different forms:

	<ul>
		<li>Most commonly, <code>include:</code> which does a recursive lookup
			to include all the IPs from a different hostname.
		<li>Also common is <code>ip4:</code> and <code>ip6:</code> for
			referencing IP addresses directly.
		<li>There are a few others (<code>a</code>, <code>mx</code>,
			<code>ptr</code>, <code>exists</code>), but they are not generally
			used in normal circumstances so I'm ignoring them for now.
		<li>Finally we have the special <code>all</code> mechanism, which is a
			wildcard catch-all that matches, well, all IP addresses. This is
			primarily used to blocklist everything by default, which we'll
			explain more below.
	</ul>
</p>

<p>
	For our example above, <code>spf.messagingengine.com</code> has a couple
	dozen <code>A</code> records on it which are included for this SPF policy
	via the <code>include:</code> mechanism.
</p>

<p>
	Other than the IPs and included hostnames, we have a
	<mark>qualifier</mark>, which is <a href="https://datatracker.ietf.org/doc/html/rfc7208#section-4.6.2">one of a few symbols</a>
	that prefix a mechanism.
</p>

<p>
	Each symbol recommends a different policy to a mail server that tells it
	what to do if it receives a message from your domain from that IP. By
	default, with no symbol, it is considered equivalent to <code>+</code>,
	which is a "pass."
</p>

<p>
	In our example, we have two mechanisms:
	<ol>
		<li><code>include:spf.messagingengine.com</code>, which includes all
			the IPs for Fastmail, and has no qualifier, making it an implicit
			"pass all messages from these IPs."
		<li><code>-all</code>, which is a fallback with the "reject" qualifier,
			instructing the receiver to "fail all messages from any other IP."
	</ol>
</p>

<p>
	And that is all there is to it. SPF is a very simple tool, but provides the
	very base level of email verification ("what IPs are allowed to send my
	email") necessary to do basic spam filtering. Even just setting up SPF
	alone should help significantly with your delivery success.
</p>

<h2>2. DKIM</h2>

<p>
	DKIM, or Domain Keys Identified Mail, is another security mechanism that
	uses asymmetric keys to cryptographically verify the server sending email
	for your domain is authorized to do so. With DKIM configured, the server
	receiving your mail can look up the public key in DNS and validate the
	email was legitimately sent from your domain.
</p>

<p>
	DKIM protects against IP's changing hands, or large service providers
	sharing IP space between customers. If you say, "Google IPs can send my
	email", what's stopping someone else from spoofing email from your domain
	and sending it from their own Google account? Since those IPs are shared,
	it will still pass SPF checks, but it <em>will not</em> pass DKIM.
</p>

<p>
	There are two main pieces to DKIM: a DNS record with the public key, and a
	header added to every sent email with cryptographic signatures and details
	on how to find the aforementioned DNS record.
</p>

<p>
	The DNS record is just a normal <code>TXT</code> record, but under a special name. The generic format is:
</p>

<pre>&lt;selector&gt;._domainkey.&lt;domain&gt;</pre>

<p>
	The "selector" is generally set by your email service provider and will be
	provided to you when you enable DKIM with them. Some providers, like
	Fastmail and Microsoft 365, even provide multiple selectors for you to set.
	For example, for Google it's just one, "<code>google</code>":
</p>

<pre>google._domainkey.example.com</pre>

<p>
	Lately these domain key records have been moving to <code>CNAME</code>'s
	to a service provider domain to make configuration easier, and allow the
	service provider to rotate keys without making you update your DNS. For
	example, both Fastmail and Microsoft 365 have switched to
	<code>CNAME</code>, though Google is still providing keys directly.
</p>

<p>
	These selectors and DNS records don't mean much unless the receiving server
	knows how to find them, though. Which brings us to the DKIM headers.
</p>

<p>
	DKIM headers are added to every email you send, and contain <a href="https://datatracker.ietf.org/doc/html/rfc6376#section-3.5">a few important details</a>.
	The two major ones we're interested in here are:

	<ol>
		<li><code><strong>d=&lt;domain&gt;</strong></code> lists the sender's domain name for verification.</li>
		<li><code><strong>s=&lt;selector&gt;</strong></code> is the "selector", matching the DNS subdomain.</li>
	</ol>
</p>

<p>
	With these two pieces of metadata, the receiving server can rebuild the
	subdomain containing the DKIM key and resolve it. Taking this key, they can
	then cryptographically verify the DKIM signature and validate if the
	message is authentic or not.
</p>

<p>
	DKIM is a much stronger signal than SPF for detecting spam, as there is
	actual maths involved and not just an IP list. If all you do is
	configure SPF and DKIM, you're in a pretty good place, however we can take
	it a step further.
</p>

<h2>3. DMARC</h2>

<p>
	You may have noticed that DKIM only applies if there is a header in the
	message. This means that illegitimate messages will not have the header,
	and thus no DKIM validation will happen. This results in the DKIM
	validation being "neutral" instead of a "failure", as it was simply
	omitted.
</p>

<p>
	Adding a DMARC policy allows us to

	<ul>
		<li>enforce SPF and DKIM checks on <em>all</em> emails claiming to be from our domain;</li>
		<li>give hints to the receiving server on how to handle failed checks;</li>
		<li>provide a reporting address so we can receive reports about these checks in the wild from email providers.</li>
	</ul>
</p>

<p>
	A DMARC record is the same format as the other two, and is also pretty
	simple. Here's an example of a very basic, permissive policy:
</p>

<pre>v=DMARC1; p=none; adkim=r; aspf=r;</pre>

<p>
	Please don't actually use this policy, it won't do anything for you. But it
	gives us a place to start explaining all the bits.
</p>

<ul>
	<li><strong><code>p=none</code></strong> sets our "global policy" for handling emails that fail authentication checks. This can be <code>none</code>, <code>quarantine</code>, or <code>reject</code>.
	<li><strong><code>adkim=r</code></strong> sets our policy for enforcing DKIM checks. This can be <code>r</code> ("relaxed") or <code>s</code> ("strict").
	<li><strong><code>aspf=r</code></strong> sets our policy for enforcing SPF checks. This takes the same values as <code>adkim</code>.
</ul>

<p>
	Before we get to locking things down, we need to talk about reporting. One
	of the most important things to do is figure out how you're going to
	monitor your DMARC reports. There are many services that do this for you; I
	use <a href="https://dmarc.postmarkapp.com">Postmark's free DMARC reports</a>
	which sends you weekly digests, but I have used other paid tools in the past
	and would recommend going for one of those if you are a business and need
	more advanced insights.
</p>

<p>
	Then you can set a permissive policy with one extra value:
	<code>rua</code>, which provides an email address for the providers
	receiving your email to send reports back to you. For example:
</p>

<pre>v=DMARC1; p=none; adkim=r; aspf=r; rua=mailto:re+someidentifier@dmarc.postmarkapp.com;</pre>

<p>
	This policy does not enforce any checks, but does set a report address.
	This will give you insight into where your mail is currently coming from.
	You don't want to start out of the gate with a locked-down policy and
	accidentally start blocking legitimate email from a source you forgot
	about.
</p>

<p>
	Once you are confident you know all the legitimate email sources, you can
	start tightening the policy. A good next step would be to set the policy to
	quarantine, to hint that failures should be considered spammy.
</p>

<pre>v=DMARC1; p=<mark>quarantine</mark>; adkim=r; aspf=r; rua=mailto:re+someidentifier@dmarc.postmarkapp.com;</pre>

<p>
	Let that sink in, and keep and eye on your reports for a bit. If you are
	confident it all looks good, you can move to marking SPF and DKIM
	validation as "strict" to spam-ify failing emails with more prejudice:
</p>

<pre>v=DMARC1; p=quarantine; <mark>adkim=s</mark>; <mark>aspf=s</mark>; rua=mailto:re+someidentifier@dmarc.postmarkapp.com;</pre>

<p>
	At this point, you have a very strong policy, and this is a perfectly fine
	place to leave it. If you are particularly paranoid, you can move up to
	<code>p=reject</code> which will instruct the receiving servers to throw
	out all email that fails authenticity checks, keeping it out of the
	recipient's mailbox entirely. This can be a bit dangerous, as there is no
	recourse if legitimate email accidentally fails a check, it's just gone
	forever, but is technically the most secure policy.
</p>

<p>
	DMARC is the most powerful piece of modern email security, and its
	reporting can be incredibly insightful into what spam is out there
	pretending to be you. If you take anything from this guide, I hope it is
	that you should take the time and care to set a strict DMARC policy.
</p>

<h2>Bonus: domains without email</h2>

<p>
	SPF and DMARC don't just apply to domains with email configured. A good
	practice is to configure highly-restrictive blocking policies for any
	domains you have that do not send email. This will ensure anyone trying to
	spoof email as that domain gets blocked immediately.
</p>

<p>
	For all of my non-email domains I set this "block all" SPF record: </p>

<pre>v=spf1 -all</pre>


<p>
	And I set a DMARC policy that hard-rejects all email that fails SPF or DKIM
	(which will be everything):
</p>

<pre>v=DMARC1; p=reject; adkim=s; aspf=s;</pre>


<p>
	Just these two empty-looking records are enough to prevent any spam being
	sent in your name from a domain you didn't even expect to send email.
</p>


guid: https://www.alexblackie.com/articles/neovim-fzy/
title: Finding files in Neovim with fzy
author: Alex Blackie
url: https://www.alexblackie.com/articles/neovim-fzy/
feedurl: https://www.alexblackie.com/feed.xml
content_mime_type: text/html
content:
 		<p>
	<big>
		I'm a big fan of <a href="https://github.com/jhawthorn/fzy">fzy</a> for
		quickly finding things in a list. I used to use it as my fuzzy file
		finder in vim, but lost that functionality when I moved to neovim a
		while back. It's time to finally fix that.
	</big>
</p>

<p>
	In vim, we're able to use the <code>system</code> function to run a command
	synchronously and capture its output. Since neovim focuses on
	asynchronicity, this behaviour is no longer supported. While this seemed
	like a blocker at the time, it turns out to not be a significant problem.
</p>

<p>
	With neovim, we can create a buffer, start an embedded terminal in it, and
	then read the contents of that buffer when the terminal exits. Since fzy
	prints the file you selected as its only output, we can easily use the
	neovim buffer API's <code>nvim_buf_get_lines</code> to read the lines in
	the buffer.
</p>

<p>
	The neovim support for Lua configuration has really provided a comfortable
	platform for me to build upon. Part of the reason for my sharing this
	integration was because of how pleased I was with the brevity and clarity
	of the final solution.
</p>

<p>
	Here is the entirety of what it takes:
</p>

<pre class="chroma"><span class="kr">function</span> <span class="nc">_G</span><span class="p">.</span><span class="nf">FzyFiles</span><span class="p">(</span><span class="n">vim_command</span><span class="p">)</span>
	<span class="kd">local</span> <span class="n">terminal_command</span> <span class="o">=</span> <span class="s1">&#39;fd -t f -H | fzy&#39;</span>

	<span class="n">CreateFloatingWindow</span><span class="p">({</span> <span class="n">height</span> <span class="o">=</span> <span class="mi">10</span> <span class="p">})</span>

	<span class="kd">local</span> <span class="n">opts</span> <span class="o">=</span> <span class="p">{</span>
		<span class="n">on_exit</span> <span class="o">=</span> <span class="kr">function</span><span class="p">()</span>
			<span class="kd">local</span> <span class="n">filename</span> <span class="o">=</span> <span class="n">table.concat</span><span class="p">(</span><span class="n">vim.api</span><span class="p">.</span><span class="n">nvim_buf_get_lines</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="kc">false</span><span class="p">),</span> <span class="s2">&#34; &#34;</span><span class="p">)</span>
			<span class="n">vim.cmd</span><span class="p">(</span><span class="s1">&#39;bdelete!&#39;</span><span class="p">)</span>
			<span class="n">vim.fn</span><span class="p">.</span><span class="n">execute</span><span class="p">(</span><span class="n">vim_command</span> <span class="o">..</span> <span class="s1">&#39; &#39;</span> <span class="o">..</span> <span class="n">filename</span><span class="p">)</span>
		<span class="kr">end</span>
	<span class="p">}</span>
	<span class="n">vim.fn</span><span class="p">.</span><span class="n">termopen</span><span class="p">(</span><span class="n">terminal_command</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span>
	<span class="n">vim.cmd</span><span class="p">(</span><span class="s1">&#39;startinsert&#39;</span><span class="p">)</span>
<span class="kr">end</span>
</pre>

<p>
	There's not much to it! The magic is really in the <code>on_exit</code>
	callback function, where we use the neovim API
	<code>nvim_buf_get_lines</code> function to return the lines of output from
	the terminal buffer, which will be the file path printed by
	<code>fzy</code> when it terminates.
</p>

<p>
	Then we delete the terminal buffer, and execute a
	<code>vim_command</code> (<code>:e</code>, <code>:vs</code>, etc.) with the
	file path we got.
</p>

<p>
	<code>CreateFloatingWindow</code> is a little helper function I have to
	make popping up a floating window easier. It's a little dense but pretty
	simple (most of the complexity is just calculating how to centre it):
</p>

<pre class="chroma"><span class="kr">function</span> <span class="nf">CreateFloatingWindow</span><span class="p">(</span><span class="n">overrides</span><span class="p">)</span>
	<span class="c1">-- Window maths lovingly stolen from:</span>
	<span class="c1">--   https://www.2n.pl/blog/how-to-make-ui-for-neovim-plugins-in-lua</span>
	<span class="kd">local</span> <span class="n">width</span> <span class="o">=</span> <span class="n">vim.api</span><span class="p">.</span><span class="n">nvim_get_option</span><span class="p">(</span><span class="s1">&#39;columns&#39;</span><span class="p">)</span>
	<span class="kd">local</span> <span class="n">height</span> <span class="o">=</span> <span class="n">vim.api</span><span class="p">.</span><span class="n">nvim_get_option</span><span class="p">(</span><span class="s1">&#39;lines&#39;</span><span class="p">)</span>
	<span class="kd">local</span> <span class="n">win_height</span> <span class="o">=</span> <span class="n">math.ceil</span><span class="p">(</span><span class="n">height</span> <span class="o">*</span> <span class="mf">0.8</span> <span class="o">-</span> <span class="mi">4</span><span class="p">)</span>
	<span class="kd">local</span> <span class="n">win_width</span> <span class="o">=</span> <span class="n">math.ceil</span><span class="p">(</span><span class="n">width</span> <span class="o">*</span> <span class="mf">0.8</span><span class="p">)</span>
	<span class="kd">local</span> <span class="n">col</span> <span class="o">=</span> <span class="n">math.ceil</span><span class="p">((</span><span class="n">width</span> <span class="o">-</span> <span class="n">win_width</span><span class="p">)</span> <span class="o">/</span> <span class="mi">2</span><span class="p">)</span>
	<span class="kd">local</span> <span class="n">buf</span> <span class="o">=</span> <span class="n">vim.api</span><span class="p">.</span><span class="n">nvim_create_buf</span><span class="p">(</span><span class="kc">false</span><span class="p">,</span> <span class="kc">true</span><span class="p">)</span>

	<span class="kd">local</span> <span class="n">opts</span> <span class="o">=</span> <span class="p">{</span>
		<span class="n">relative</span> <span class="o">=</span> <span class="s1">&#39;editor&#39;</span><span class="p">,</span>
		<span class="n">width</span> <span class="o">=</span> <span class="n">win_width</span><span class="p">,</span>
		<span class="n">height</span> <span class="o">=</span> <span class="n">win_height</span><span class="p">,</span>
		<span class="n">col</span> <span class="o">=</span> <span class="n">col</span><span class="p">,</span>
		<span class="n">row</span> <span class="o">=</span> <span class="mi">2</span><span class="p">,</span>
		<span class="n">border</span> <span class="o">=</span> <span class="s1">&#39;rounded&#39;</span><span class="p">,</span>
	<span class="p">}</span>
	<span class="kr">for</span> <span class="n">k</span><span class="p">,</span><span class="n">v</span> <span class="kr">in</span> <span class="n">pairs</span><span class="p">(</span><span class="n">overrides</span><span class="p">)</span> <span class="kr">do</span> <span class="n">opts</span><span class="p">[</span><span class="n">k</span><span class="p">]</span> <span class="o">=</span> <span class="n">v</span> <span class="kr">end</span>

	<span class="kr">return</span> <span class="n">vim.api</span><span class="p">.</span><span class="n">nvim_open_win</span><span class="p">(</span><span class="n">buf</span><span class="p">,</span> <span class="kc">true</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span>
<span class="kr">end</span>
</pre>

<p>
	Then, all we need to do is bind <code>v:lua.FzyFiles(':e')</code> to a key,
	and we're done! We now have <code>fzy</code> integrated into neovim.
</p>


guid: https://jvns.ca/blog/2022/01/11/how-to-find-a-domain-s-authoritative-nameserver/
title: How to find a domain's authoritative nameservers
url: https://jvns.ca/blog/2022/01/11/how-to-find-a-domain-s-authoritative-nameserver/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Here&rsquo;s a very quick &ldquo;how to&rdquo; post on how to find your domain&rsquo;s authoritative
nameserver.</p>

<p>I&rsquo;m writing this because if you made a DNS update and it didn&rsquo;t work, there are 2 options:</p>

<ol>
<li>Your authoritative nameserver doesn&rsquo;t have the correct record</li>
<li>Your authoritative nameserver <strong>does</strong> have the correct record, but an old record is cached and you need to wait for the cache to expire</li>
</ol>

<p>To be able to tell which one is happening (do you need to make a change, or do
you just need to wait?), you need to be able to find your domain&rsquo;s
authoritative nameserver and query it to see what records it has.</p>

<p>But when I looked up &ldquo;how to find a domain&rsquo;s authoritative nameserver&rdquo; to see
what advice was out there, I found a lot of different methods being mentioned,
some of which can give you the wrong answer.</p>

<p>So let&rsquo;s walk through a way to find your domain&rsquo;s authoritative nameservers
that&rsquo;s guaranteed to always give you the correct answer. I&rsquo;ll also explain why
some of the other methods aren&rsquo;t always accurate.</p>

<h3 id="first-an-easy-but-less-accurate-way">first, an easy but less accurate way</h3>

<p>If you definitely haven&rsquo;t updated your authoritative DNS server in the last
week or so, a very easy way to find it is to run <code>dig +short ns DOMAIN</code></p>

<pre><code>$ dig +short ns jvns.ca
art.ns.cloudflare.com.
roxy.ns.cloudflare.com.
</code></pre>

<p>In this case, we get the correct answer. Great!</p>

<p>But if you <em>have</em> updated your authoritative DNS server in the last few days
(maybe because you just registered the domain!), that can give you an
inaccurate answer. So here&rsquo;s the slightly more complicated way that&rsquo;s
guaranteed to always give you the correct answer.</p>

<h3 id="step-1-query-a-root-nameserver">step 1: query a root nameserver</h3>

<p>We&rsquo;re going to look up the authoritative nameserver for <code>jvns.ca</code> in this example.</p>

<p>No matter what domain we&rsquo;re looking up, we need to start with the root
nameservers. <code>h.root-servers.net</code> is one of the <a href="https://www.iana.org/domains/root/servers">13 DNS root nameservers</a>, and <code>dig @h.root-servers.net</code> means &ldquo;send the query to <code>h.root-servers.net</code>&rdquo;.</p>

<pre><code>$ dig @h.root-servers.net jvns.ca
;; Got answer:
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 42165
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 9
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;jvns.ca.			IN	A

;; AUTHORITY SECTION: &lt;------------ this is the section we're interested in
ca.			172800	IN	NS	c.ca-servers.ca. &lt;------- we'll use this record
ca.			172800	IN	NS	j.ca-servers.ca.
ca.			172800	IN	NS	x.ca-servers.ca.
ca.			172800	IN	NS	any.ca-servers.ca.

;; ADDITIONAL SECTION:
c.ca-servers.ca.	172800	IN	A	185.159.196.2
j.ca-servers.ca.	172800	IN	A	198.182.167.1
x.ca-servers.ca.	172800	IN	A	199.253.250.68
any.ca-servers.ca.	172800	IN	A	199.4.144.2
c.ca-servers.ca.	172800	IN	AAAA	2620:10a:8053::2
j.ca-servers.ca.	172800	IN	AAAA	2001:500:83::1
x.ca-servers.ca.	172800	IN	AAAA	2620:10a:80ba::68
any.ca-servers.ca.	172800	IN	AAAA	2001:500:a7::2

;; Query time: 96 msec
;; SERVER: 198.97.190.53#53(198.97.190.53)
;; WHEN: Tue Jan 11 08:30:57 EST 2022
;; MSG SIZE  rcvd: 289
</code></pre>

<p>The answer we&rsquo;re looking for is this line in the &ldquo;AUTHORITY SECTION&rdquo;:</p>

<pre><code>ca.			172800	IN	NS	c.ca-servers.ca.
</code></pre>

<p>It doesn&rsquo;t matter which line in this section you pick, you can use any of them. I just picked the first one.</p>

<p>This tells us the server we need to talk to in step 2: <code>c.ca-servers.ca.</code></p>

<h3 id="step-2-query-the-ca-nameservers">step 2: query the .ca nameservers</h3>

<p>Now we run <code>dig @c.ca-servers.ca jvns.ca</code></p>

<pre><code>$ dig @c.ca-servers.ca jvns.ca
;; Got answer:
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 24920
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;jvns.ca.			IN	A

;; AUTHORITY SECTION: &lt;------------ this is the section we're interested in
jvns.ca.		86400	IN	NS	art.ns.cloudflare.com. &lt;---- we'll use this record
jvns.ca.		86400	IN	NS	roxy.ns.cloudflare.com.

;; Query time: 26 msec
;; SERVER: 185.159.196.2#53(185.159.196.2)
;; WHEN: Tue Jan 11 08:32:44 EST 2022
;; MSG SIZE  rcvd: 90
</code></pre>

<p>Same as last time: the answer we&rsquo;re looking for is this line in the &ldquo;AUTHORITY
SECTION&rdquo;:</p>

<pre><code>jvns.ca.		86400	IN	NS	art.ns.cloudflare.com.
</code></pre>

<p>Again, it doesn&rsquo;t matter which line in this section you pick, you can use any
of them. I just picked the first one.</p>

<h3 id="success-we-know-the-authoritative-nameserver">success! we know the authoritative nameserver!</h3>

<p>The authoritative nameserver for <code>jvns.ca</code> is <code>art.ns.cloudflare.com.</code>. Now you
can now query <code>art.ns.cloudflare.com.</code> directly to see what DNS records it has
for <code>jvns.ca</code>.</p>

<pre><code>$ dig @art.ns.cloudflare.com. jvns.ca
jvns.ca.		292	IN	A	172.64.80.1
</code></pre>

<p>Nice, it worked.</p>

<h3 id="this-is-exactly-what-s-happening-behind-the-scenes-when-you-make-a-dns-query">this is exactly what&rsquo;s happening behind the scenes when you make a DNS query</h3>

<p>The reason I like this method is that it mimics what&rsquo;s happening behind the
scenes when you make a DNS query. When Google&rsquo;s DNS resolver <code>8.8.8.8.</code> looks
up <code>jvns.ca</code>, the server it queries to to get <code>jvns.ca</code>&rsquo;s authoritative nameserver is
<code>c.ca-servers.net</code> (or one of the other options, like <code>j.ca-servers.ca.</code> or <code>x.ca-servers.ca.</code>)</p>

<p>Because this method uses the exact same information source as a real DNS query,
you&rsquo;re guaranteed to get a correct answer every time.</p>

<p>Often in practice I skip step 1 because I remember that the answer for <code>.ca</code>
domains is <code>c.ca-servers.net</code>, so I can skip straight to step 2.</p>

<h3 id="this-is-useful-to-do-when-you-re-updating-your-nameservers">this is useful to do when you&rsquo;re updating your nameservers</h3>

<p>When I update my nameservers with my domain registrar, they don&rsquo;t actually update the
authoritative nameserver right away. It takes a while, maybe an hour. So I like
to go through these steps to check if my registrar has actually updated my
authoritative nameserver yet.</p>

<h3 id="other-ways-to-get-a-domain-s-authoritative-nameserver">other ways to get a domain&rsquo;s authoritative nameserver</h3>

<p>Here are a few other ways you can get the authoritative nameserver for a domain
and why I didn&rsquo;t recommend them as the main method.</p>

<p><strong>dig +trace jvns.ca</strong></p>

<p>This does the exact same thing so it will always give you the right answer, but
the output is a bit confusing to read so I&rsquo;m a bit more hesitant to recommend
it.</p>

<p><strong>dig ns jvns.ca</strong></p>

<p>This will usually give you the right answer, but there are 2 reasons it might
be wrong:</p>

<ol>
<li>You might get an old cached record</li>
<li>The NS record you get doesn&rsquo;t come from the same place as it does when we do
the method described in this post. In this example, instead of getting a NS
record from <code>c.ca-servers.net</code>, <code>dig ns jvns.ca</code> will give you an NS record
from <code>art.ns.cloudflare.com</code>.  In practice usually these are the exact same
thing, but in some weird edge cases they might not be.</li>
</ol>

<p><strong>dig soa jvns.ca</strong></p>

<p>You can also find nameservers in the SOA record!</p>

<pre><code>$ dig SOA jvns.ca
jvns.ca.   3600    IN    SOA    art.ns.cloudflare.com. dns.cloudflare.com. 2267173366 10000 2400 604800 3600
                                ^^^^^^^^^^^^^^^^^^^^^
                                    here it is
</code></pre>

<p>This will usually give the right answer, there are 2 reasons it might be wrong, similarly to the NS record:</p>

<ol>
<li>This response comes from your authoritative nameserver. So if you&rsquo;re in the
middle of updating your nameserver, you might get the wrong answer because
your DNS resolver sent the request to the old nameserver.</li>
<li>Your authoritative nameserver could be returning a SOA record which doesn&rsquo;t
have the correct nameserver for some reason</li>
</ol>

<p><strong>whois jvns.ca</strong></p>

<p>This will usually give you the right answer, but it might be an old cached version.</p>

<p>Here&rsquo;s what this looks like on my machine for this example: (it gives us the right answer)</p>

<pre><code>$ whois jvns.ca | grep 'Name Server'
Name Server: art.ns.cloudflare.com
Name Server: roxy.ns.cloudflare.com
</code></pre>

<h3 id="that-s-all">that&rsquo;s all!</h3>

<p>I hope this helps some of you debug your DNS issues!</p>


guid: https://jvns.ca/blog/2022/01/15/some-ways-dns-can-break/
title: Some ways DNS can break
url: https://jvns.ca/blog/2022/01/15/some-ways-dns-can-break/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>When I first learned about it, DNS didn&rsquo;t seem like it should be THAT
complicated. Like, there are DNS records, they&rsquo;re stored on a server, what&rsquo;s
the big deal?</p>

<p>But with DNS, reading about how it works in a textbook doesn&rsquo;t prepare you for
the sheer volume of different ways DNS can break your system in practice. It&rsquo;s
not just caching problems!</p>

<p>So I <a href="https://twitter.com/b0rk/status/1481265429897261058">asked people on Twitter</a> for
example of DNS problems they&rsquo;ve run into, especially DNS problems that <strong>didn&rsquo;t
initially appear to be DNS problems</strong>. (the popular &ldquo;it&rsquo;s always DNS&rdquo; meme)</p>

<p>I&rsquo;m not going to discuss how to solve or avoid any of these problems in this
post, but I&rsquo;ve linked to webpages discussing the problem where I could find
them.</p>

<h3 id="problem-slow-network-requests">problem: slow network requests</h3>

<p>Your network requests are a little bit slower than expected, and it&rsquo;s actually
because your DNS resolver is slow for some reason. This might be because the
resolver is under a lot of load, or it has a memory leak, or something else.</p>

<p>I&rsquo;ve run into this before with my router&rsquo;s DNS forwarder &ndash; all of my DNS
requests were slow, and I restarted my router and that fixed the problem.</p>

<h3 id="problem-dns-timeouts">problem: DNS timeouts</h3>

<p>A couple of people mentioned network requests that were taking 2+ seconds or 30
seconds because of DNS queries that were timing out. This is sort of the same
as &ldquo;slow requests&rdquo;, but it&rsquo;s worse because queries can take several seconds to
time out.</p>

<p>Sophie Haskins has a great blog post <a href="https://blog.sophaskins.net/blog/misadventures-with-kube-dns/">Misadventures with Kube DNS</a> about DNS
timeouts with Kubernetes.</p>

<h3 id="problem-ndots">problem: ndots</h3>

<p>A few people mentioned a specific issue where Kubernetes sets <code>ndots:5</code> in its <code>/etc/resolv.conf</code></p>

<p>Here&rsquo;s an example /etc/resolv.conf from <a href="https://pracucci.com/kubernetes-dns-resolution-ndots-options-and-why-it-may-affect-application-performances.html">Kubernetes pods /etc/resolv.conf ndots:5 option and why it may negatively affect your application performances</a>.</p>

<pre><code>nameserver 100.64.0.10
search namespace.svc.cluster.local svc.cluster.local cluster.local eu-west-1.compute.internal
options ndots:5
</code></pre>

<p>My understanding is that if this is your <code>/etc/resolv.conf</code> and you look up
<code>google.com</code>, your application will call the C <code>getaddrinfo</code> function, and
<code>getaddrinfo</code> will:</p>

<ol>
<li>look up <code>google.com.namespace.svc.cluster.local.</code></li>
<li>look up <code>google.com.svc.cluster.local.</code></li>
<li>look up <code>google.com.cluster.local.</code></li>
<li>look up <code>google.com.eu-west-1.compute.internal.</code></li>
<li>look up <code>google.com.</code></li>
</ol>

<p>Basically it checks if <code>google.com</code> is actually a subdomain of everything on the <code>search</code> line.</p>

<p>So every time you make a DNS query, you need to wait for 4 DNS queries to fail
before you can get to the actual real DNS query that succeeds.</p>

<h3 id="problem-it-s-hard-to-tell-what-dns-resolver-s-your-system-is-using">problem: it&rsquo;s hard to tell what DNS resolver(s) your system is using</h3>

<p>This isn&rsquo;t a bug by itself, but when you run into a problem with DNS, often
it&rsquo;s related in some way to your DNS resolver. I don&rsquo;t know of any foolproof
way to tell what DNS resolver is being used.</p>

<p>A few things I know:</p>

<ul>
<li>on Linux, I think that most things use /etc/resolv.conf to choose a DNS
resolver. There are definitely exceptions though, for example your browser
might ignore /etc/resolv.conf and use a different DNS-over-HTTPS service
instead.</li>
<li>if you&rsquo;re using UDP DNS, you can use <code>sudo tcpdump port 53</code>  to see where DNS
requests are being sent. This doesn&rsquo;t work if you&rsquo;re using DNS over HTTPS or
DNS over TLS though.</li>
</ul>

<p>I also vaguely remember it being even more confusing on MacOS than on Linux,
though I don&rsquo;t know why.</p>

<h3 id="problem-dns-servers-that-return-nxdomain-instead-of-noerror">problem: DNS servers that return NXDOMAIN instead of NOERROR</h3>

<p>Here&rsquo;s a problem that I ran into once, where nginx couldn&rsquo;t resolve a domain.</p>

<ul>
<li>I set up nginx to use a specific DNS server to resolve DNS queries</li>
<li>when visiting the domain, nginx made 2 queries, one for an <code>A</code> record, and one for an <code>AAAA</code> record</li>
<li>the DNS server returned a <code>NXDOMAIN</code> reply for the <code>A</code> query</li>
<li>nginx decided &ldquo;ok, that domain doesn&rsquo;t exist&rdquo;, and gave up</li>
<li>the DNS server returned a successful reply for the <code>AAAA</code> query</li>
<li>nginx ignored the <code>AAAA</code> record because it had already given up</li>
</ul>

<p>The problem was that the DNS server should have returned <code>NOERROR</code> &ndash; that
domain <em>did</em> exist, it was just that there weren&rsquo;t any <code>A</code> records for it. I
reported the bug, they fixed it, and that fixed the problem.</p>

<p>I&rsquo;ve implemented this bug myself too, so I understand why it happens &ndash; it&rsquo;s
easy to think &ldquo;there aren&rsquo;t any records for this query, I should return an
<code>NXDOMAIN</code> error&rdquo;.</p>

<h3 id="problem-negative-dns-caching">problem: negative DNS caching</h3>

<p>If you visit a domain before creating a DNS record for it, the <strong>absence</strong> of
the record will be cached. This is very surprising the first time your run into
it &ndash; I only learned about this last year!</p>

<p>The TTL for cache entry is the TTL of the domain&rsquo;s SOA record &ndash; for example
for <code>jvns.ca</code>, it&rsquo;s an hour.</p>

<h3 id="problem-nginx-caching-dns-records-forever">problem: nginx caching DNS records forever</h3>

<p>If you put this in your nginx config:</p>

<pre><code>location / {
    proxy_pass https://some.domain.com;
}
</code></pre>

<p>then nginx will resolve <code>some.domain.com</code> once on startup and never again. This
is especially dangerous if the IP address for <code>some.domain.com</code>  changes
infrequently, because it might keep happily working for months and then
suddenly break at 2am one day.</p>

<p>There are pretty well-known ways to fix this and this post isn&rsquo;t about nginx so
I won&rsquo;t get into it, but it&rsquo;s surprising the first time you run into it.</p>

<p>Here&rsquo;s a <a href="https://medium.com/driven-by-code/dynamic-dns-resolution-in-nginx-22133c22e3ab">blog post</a> with a story of how this happened to someone with an AWS load balancer.</p>

<h3 id="problem-java-caching-dns-records-forever">problem: Java caching DNS records forever</h3>

<p>Same thing, but for Java: <a href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html">Apparently</a>
depending on how you configure Java, &ldquo;the JVM default TTL [might be] set so
that it will never refresh DNS entries until the JVM is restarted.&rdquo;</p>

<p>I haven&rsquo;t run into this myself but I asked a friend about it who writes more
Java than me and they told me that it&rsquo;s happened to them.</p>

<p>Of course, literally any software could have this problem of caching DNS
records forever, but the main cases I&rsquo;ve heard of in practice are nginx and
Java.</p>

<h3 id="problem-that-entry-in-etc-hosts-you-forgot-about">problem: that entry in /etc/hosts you forgot about</h3>

<p>Another variant on caching issues: entries in <code>/etc/hosts</code> that override your
usual DNS settings!</p>

<p>This is extra confusing because <code>dig</code> ignores <code>/etc/hosts</code>, so everything SEEMS
like it should be fine (&rdquo;<code>dig whatever.com</code> is working!&ldquo;).</p>

<h3 id="problem-your-email-isn-t-being-sent-is-going-to-spam">problem: your email isn&rsquo;t being sent / is going to spam</h3>

<p>The way email is sent and validated is through DNS (MX records, SPF records,
DKIM records), so a lot of email problems are DNS problems.</p>

<h3 id="problem-internationalized-domain-names-don-t-work">problem: internationalized domain names don&rsquo;t work</h3>

<p>You can register domain names with non-ASCII characters or emoji like <a href="https://💩.la/">https://💩.la</a>.</p>

<p>The way this works with DNS is that <code>💩.la</code> gets translated into <code>xn--ls8h.la</code> with an encoding called &ldquo;punycode&rdquo;.</p>

<p>But even though there&rsquo;s a clear standard for how they should work with DNS, a lot of software doesn&rsquo;t handle internationalized domain names well!
There&rsquo;s a fun story about this in Julian Squires&rsquo; great talk <a href="https://www.youtube.com/watch?v=UE-fJjMasec">The emoji that Killed Chrome!!</a>.</p>

<h3 id="problem-tcp-dns-is-blocked-by-a-firewall">problem: TCP DNS is blocked by a firewall</h3>

<p>A couple of people mentioned that some firewalls allow UDP port 53 but not TCP
port 53. But large DNS queries need to use TCP port 53, so this can cause weird
intermittent problems that are hard to debug.</p>

<h3 id="problem-musl-doesn-t-support-tcp-dns">problem: musl doesn&rsquo;t support TCP DNS</h3>

<p>A lot of applications use libc&rsquo;s <code>getaddrinfo</code> to make DNS queries. musl is an
alternative to <code>glibc</code> that&rsquo;s used in Alpine Docker container which doesn&rsquo;t
support TCP DNS. This can cause problems if you make DNS queries where the
response would be too big to fit inside a regular DNS UDP packet (512 bytes).</p>

<p>I&rsquo;m still a bit fuzzy on this so I might have it wrong, but my understanding of how this can break is:</p>

<ol>
<li>musl&rsquo;s getaddrinfo makes a DNS query</li>
<li>the DNS server notices that the response is too big to fit in a single DNS response packet</li>
<li>the DNS server returns an <strong>empty</strong> truncated response, expecting that the client will retry by making a TCP DNS query</li>
<li><code>musl</code> does not support TCP so it does not retry</li>
</ol>

<p>A blog post about this: <a href="https://christoph.luppri.ch/fixing-dns-resolution-for-ruby-on-alpine-linux">DNS resolution issue in Alpine Linux</a></p>

<h3 id="problem-round-robin-dns-doesn-t-work-with-getaddrinfo">problem: round robin DNS doesn&rsquo;t work with <code>getaddrinfo</code></h3>

<p>One way you could approach load balancing is to use &ldquo;round robin DNS&rdquo;. The idea
is that every time you make a DNS query, you get a different IP address.
Apparently this works if you use <code>gethostbyname</code> to make DNS queries, but it
does not work if you use <code>getaddrinfo</code> because <code>getaddrinfo</code> sorts the IP
responses it receives.</p>

<p>So you could run into an upsetting problem if you switch from <code>gethostbyname</code> to <code>getaddrinfo</code> behind the scenes without realising that this will break your DNS load balancing.</p>

<p>This is especially insidious because you might not realize that you&rsquo;re
switching to <code>gethostbyname</code> to <code>getaddrinfo</code> at all &ndash; if you&rsquo;re not writing a
C program, those functions calls are hidden inside some library. So it could be
part of a seemingly innocuous upgrade.</p>

<p>Here are a couple of pages discussing this:</p>

<ul>
<li><a href="https://groups.google.com/g/consul-tool/c/AGgPjrrkw3g">getaddrinfo breaks round robin DNS</a></li>
<li><a href="https://daniel.haxx.se/blog/2012/01/03/getaddrinfo-with-round-robin-dns-and-happy-eyeballs/">getaddrinfo with round robin DNS and happy eyeballs</a></li>
</ul>

<h3 id="problem-a-race-condition-when-starting-a-service">problem: a race condition when starting a service</h3>

<p>A problem someone <a href="https://mobile.twitter.com/omatskiv/status/1481305175440646148">mentioned</a>
with Kubernetes DNS: they had 2 containers which started simultaneously and
immediately tried to resolve each other. But the DNS lookup failed because the
Kubernetes DNS change hadn&rsquo;t happened yet, and then the failure was cached so
it kept failing.</p>

<h3 id="that-s-all">that&rsquo;s all!</h3>

<p>I&rsquo;ve definitely missed some important DNS problems here, so I&rsquo;d love to hear
what I&rsquo;ve missed. I&rsquo;d also love links to blog posts that write up examples of
these problems &ndash; I think it&rsquo;s really useful to see how the problem
specifically manifests in practice and how people debugged it.</p>


guid: https://jvns.ca/blog/2022/01/24/hosting-my-static-sites-with-nginx/
title: Hosting my static sites with nginx
url: https://jvns.ca/blog/2022/01/24/hosting-my-static-sites-with-nginx/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Hello! Recently I&rsquo;ve been thinking about putting my static sites on servers
that I run myself instead of using managed services like Netlify or GitHub
Pages.</p>

<p>Originally I thought that running my own servers would require a lot of
maintenance and be a huge pain, but I was chatting with Wesley about what kind
of maintainance <a href="https://blog.wesleyac.com/posts/how-i-run-my-servers">their servers</a> require, and
they convinced me that it might not be that bad.</p>

<p>So I decided to try out moving all my static sites to a $5/month server to see
what it was like.</p>

<p>Everything in here is pretty standard but I wanted to write down what I did
anyway because there are a surprising number of decisions and I like to see
what choices other people make.</p>

<h3 id="the-constraint-only-static-sites">the constraint: only static sites</h3>

<p>To keep things simple, I decided that this server would only run <code>nginx</code> and
only serve static sites. I have about 10 static sites right now, mostly projects for <a href="https://wizardzines.com">wizard zines</a>.</p>

<p>I decided to use a $5/month DigitalOcean droplet, which should very easily be
able to handle my existing traffic (about 3 requests per second and 100GB
of bandwidth per month). Right now it&rsquo;s using about 1% of its CPU. I picked
DigitalOcean because it was what I&rsquo;ve used before.</p>

<p>Also all the sites were already behind a CDN so they&rsquo;re still behind the same
CDN.</p>

<h3 id="step-1-get-a-clean-git-repo-for-each-build">step 1: get a clean Git repo for each build</h3>

<p>This was the most interesting problem so let&rsquo;s talk about it first!</p>

<p>Building the static sites might seem pretty easy &ndash; each one of them already
has a working build script.</p>

<p>But I have pretty bad hygiene around files on my laptop &ndash; often I have a bunch of
uncommitted files that I don&rsquo;t want to go onto the live site. So I wanted to
start every build with a clean Git repo. I also wanted this to be <em>fast</em> &ndash; I&rsquo;m
impatient so I wanted to be able to build and deploy most of my sites in less than 10
seconds.</p>

<p>I handled this by hacking together a tiny build system called
<a href="https://github.com/jvns/tinybuild/">tinybuild</a>. It&rsquo;s
basically a 4-line bash script, but with extra some command line arguments and
error checking. Here are the 4 lines of bash:</p>

<pre><code>docker build - -t tinybuild &lt; Dockerfile
CONTAINER_ID=$(docker run -v &quot;$PWD&quot;:/src -v &quot;./deploy:/artifact&quot; -d -t tinybuild /bin/bash)
docker exec $CONTAINER_ID bash -c &quot;git clone /src /build &amp;&amp; cd /build &amp;&amp; bash /src/scripts/build.sh&quot;
docker exec $CONTAINER_ID bash -c &quot;mv /build/public/* /artifact&quot;
</code></pre>

<p>These 4 lines:</p>

<ol>
<li>Build a Dockerfile with all the dependencies for that build</li>
<li>Clone my repo into <code>/build</code> in the container, so that I always start with a clean Git repo</li>
<li>Run the build script (<code>/src/scripts/build.sh</code>)</li>
<li>Copy the build artifacts into <code>./deploy</code> in the local directory</li>
</ol>

<p>Then once I have <code>./deploy</code>, I can rsync the result onto the server</p>

<p>It&rsquo;s fast because:</p>

<ul>
<li>the <code>docker build -</code> means I don&rsquo;t send any state from the repository to
the Docker daemon. This matters because one of my repos is 1GB (it has a lot
of PDFs in it) and sending all that to the Docker daemon takes forever</li>
<li>the <code>git clone</code> is from the local filesystem and I have a SSD so it&rsquo;s fast even for a 1GB repo</li>
<li>most of the build scripts just run <code>hugo</code> or <code>cat</code> so they&rsquo;re fast. The <code>npm</code> build scripts take maybe 30 seconds.</li>
</ul>

<h3 id="apparently-local-git-clones-make-hard-links">apparently local git clones make hard links</h3>

<p>A tiny interesting fact: I tried to do <code>git clone --depth 1</code> to speed up my git
clone, but git gave me this warning:</p>

<pre><code>warning: --depth is ignored in local clones; use file:// instead.
</code></pre>

<p>I think what&rsquo;s going on here is that git makes hard links of all the objects to
make a local clone (which is a lot faster than copying). So I guess with the
hard links approach <code>--depth 1</code> doesn&rsquo;t make sense for some reason? And
<code>file://</code> forces git to copy all objects instead, which is actually slower.</p>

<h3 id="bonus-now-my-builds-are-faster-than-they-used-to-be">bonus: now my builds are faster than they used to be!</h3>

<p>One nice thing about this is that my build/deploy time is less than it was on
Netlify. For <code>jvns.ca</code> it&rsquo;s about 7 seconds to build and deploy the site
instead of about a minute previously.</p>

<h3 id="running-the-builds-on-my-laptop-seems-nice">running the builds on my laptop seems nice</h3>

<p>I&rsquo;m the only person who develops all of my sites, so doing all the builds in a
Docker container on my computer seems to make sense. My computer is pretty fast
and all the files are already right there! No giant downloads! And doing it in
a Docker container keeps the build isolated.</p>

<h3 id="example-build-scripts">example build scripts</h3>

<p>Here are the build scripts for this blog (<code>jvns.ca</code>).</p>

<p><strong>Dockerfile</strong></p>

<pre><code>FROM ubuntu:20.04

RUN apt-get update &amp;&amp; apt-get install -y git
RUN apt-get install -y wget python2
RUN wget https://github.com/gohugoio/hugo/releases/download/v0.40.1/hugo_0.40.1_Linux-64bit.tar.gz
RUN wget https://github.com/sass/dart-sass/releases/download/1.49.0/dart-sass-1.49.0-linux-x64.tar.gz
RUN tar -xf dart-sass-1.49.0-linux-x64.tar.gz
RUN tar -xf hugo_0.40.1_Linux-64bit.tar.gz
RUN mv hugo /usr/bin/hugo
RUN mv dart-sass/sass /usr/bin/sass
</code></pre>

<p><strong>build-docker.sh</strong>:</p>

<pre><code>set -eu
scripts/parse_titles.py
sass sass/:static/stylesheets/
hugo
</code></pre>

<p><strong>deploy.sh</strong>:</p>

<pre><code>set -eu
tinybuild -s scripts/build-docker.sh \
          -l &quot;$PWD/deploy&quot; \
          -c /build/public

rsync-showdiff ./deploy/ root@staticsites:/var/www/jvns.ca
rm -rf ./deploy
</code></pre>

<h3 id="step-2-get-rsync-to-just-show-me-which-files-it-updated">step 2: get rsync to just show me which files it updated</h3>

<p>When I started using rsync to sync the files, it would list every single file
instead of just files that had changed. I think this was because I was
generating new files for every build, so the timestamps were always newer than
the files on the server.</p>

<p>I did a bunch of Googling and figured out this incantation to get rsync to just
show me files that were updated;</p>

<pre><code>rsync -avc --out-format='%n' &quot;$@&quot; | grep --line-buffered -v '/$'
</code></pre>

<p>I put that in a script called <code>rsync-showdiff</code> so I could reuse it. There might
be a better way, but this seems to work.</p>

<h3 id="step-3-configuration-management">step 3: configuration management</h3>

<p>All I needed to do to set up the server was:</p>

<ul>
<li>install nginx</li>
<li>create directories in /var/www for each site, like <code>/var/www/jvns.ca</code></li>
<li>create an nginx configuration for each site, like <code>/etc/nginx/sites-enabled/jvns.ca.conf</code></li>
<li>deploy the files (with my deploy script above)</li>
</ul>

<p>I wanted to use some kind of configuration management to do this because that&rsquo;s how I&rsquo;m
used to managing servers. I&rsquo;ve used Puppet a lot in the past at work, but I don&rsquo;t
really <em>like</em> using Puppet. So I decided to use Ansible even though I&rsquo;d never
used it before because it seemed simpler than using Puppet. Here&rsquo;s <a href="https://gist.github.com/jvns/06754e9e65b49dd461fefa071dd4aace">my current Ansible configuration</a>,
minus some of the templates it depends on.</p>

<p>I didn&rsquo;t use any Ansible plugins because I wanted to maximize the probability
that I would actually be able to run this thing in 3 years.</p>

<p>The most complicated thing in there is probably the <code>reload nginx</code> handler,
which makes sure that the configuration is still valid after I make an nginx
configuration update.</p>

<h3 id="step-4-replace-a-lambda-function">step 4: replace a lambda function</h3>

<p>I was using one Netlify lambda function to calculate purchasing power parity
(&ldquo;PPP&rdquo;) for countries that have a weaker currency relative to the US on <a href="https://wizardzines.com">https://wizardzines.com</a>. Basically it
gets your country using IP geolocation and then returns a discount code if
you&rsquo;re in a country that has a discount code. (like 70% off for India, for
example). So I needed to replace it.</p>

<p>I handled this by rewriting the (very small) program in Go, copying the
static binary to the server, and adding a <code>proxy_pass</code> for that site.</p>

<p>The program just looks up the country code from the <a href="https://support.cloudflare.com/hc/en-us/articles/200168236-Configuring-Cloudflare-IP-Geolocation">geolocation HTTP header</a>
in a hashmap, so it doesn&rsquo;t seem like it should cause maintenance problems.</p>

<h3 id="a-very-simple-nginx-config">a very simple nginx config</h3>

<p>I used the same nginx config file for templates for almost all my sites:</p>

<pre><code>server {
	listen 80;
	listen [::]:80;

	root /var/www/{{item.dir}};
	index index.html index.htm;
	server_name {{item.server}};

    location / {
        # First attempt to serve request as file, then
        # as directory, then fall back to displaying a 404.
        try_files $uri $uri/ =404;
    }
}
</code></pre>

<p>The <code>{{item.dir}}</code> is an Ansible thing.</p>

<p>I also added support for custom 404 pages (<code>error_page /404.html</code>) in the main <code>nginx.conf</code>.</p>

<p>I&rsquo;ll probably add TLS support with certbot later. My CDN handles TLS to the
client, I just need to make the connection between the CDN and the origin
server use TLS</p>

<p>Also I don&rsquo;t know if there are problems with using such a simple nginx config.
Maybe I&rsquo;ll learn about them!</p>

<h3 id="bonus-i-can-find-404s-more-easily">bonus: I can find 404s more easily</h3>

<p>Another nice bonus of this setup is that it&rsquo;s easier to see what&rsquo;s happening
with my site &ndash; I can just look at the nginx logs!</p>

<p>I ran <code>grep 404 /var/log/nginx/access.log</code> to figure out if I&rsquo;d broken
anything during the migration, and I actually ended up finding a lot of
links that had been broken for many years, but that I&rsquo;d just never noticed.</p>

<p>Netlify&rsquo;s analytics has a &ldquo;Top resources not found&rdquo; that shows you the most
common 404s, but I don&rsquo;t think there&rsquo;s any way to see <em>all</em> 404s.</p>

<h3 id="a-small-factor-costs">a small factor: costs</h3>

<p>Part of my motivation for this switch was &ndash; I was getting close to the Netlify
free tier&rsquo;s bandwidth limit (100GB/month), and Netlify charges $20/100GB for
additional bandwidth. Digital Ocean charges $1/100GB for additional bandwidth
(20x less), and my droplet comes with 1TB of bandwidth. So the bandwidth
pricing feels a lot more reasonable to me.</p>

<h3 id="we-ll-see-how-it-goes">we&rsquo;ll see how it goes!</h3>

<p>All my static sites are running on my own server now. I don&rsquo;t really know what
this will be like to maintain, we&rsquo;ll see how it goes &ndash; maybe I&rsquo;ll like it!
maybe I&rsquo;ll hate it! I definitely like the faster build times and that I can
easily look at my nginx logs.</p>


guid: https://jvns.ca/blog/2022/01/29/reasons-for-servers-to-support-ipv6/
title: Reasons for servers to support IPv6
url: https://jvns.ca/blog/2022/01/29/reasons-for-servers-to-support-ipv6/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>I&rsquo;ve been having a hard time understanding IPv6. On one hand, the basics initially seem pretty
straightforward (there aren&rsquo;t enough IPv4 addresses for all the devices on the
internet, so people invented IPv6! There are enough IPv6 addresses for
everyone!)</p>

<p>But when I try to actually understand it, I run into a lot of questions. One
question is: <code>twitter.com</code> does not support IPv6. Presumably it can&rsquo;t be causing
them THAT many issues to not support it. So why <em>do</em> websites support IPv6?</p>

<p>I asked people on Twitter <a href="https://twitter.com/b0rk/status/1487156306884636672">why their servers support IPv6</a>
and I got a lot of great answers, which I&rsquo;ll summarize here. These all come
with the disclaimer that I have basically 0 experience with IPv6 so I can&rsquo;t evaluate these reasons very well.</p>

<p>First though, I want to explain why it&rsquo;s possible for <code>twitter.com</code> to not
support IPv6 because I didn&rsquo;t understand that initially.</p>

<h3 id="how-can-you-tell-twitter-com-doesn-t-support-ipv6">how can you tell <code>twitter.com</code> doesn&rsquo;t support IPv6?</h3>

<p>You can tell they don&rsquo;t support IPv6 is because if you look up their AAAA
record (which contains their IPv6 address), there isn&rsquo;t one. Some other big
sites like <code>github.com</code> and <code>stripe.com</code> also don&rsquo;t support IPv6.</p>

<pre><code>$ dig AAAA twitter.com
(empty response)
$ dig AAAA github.com
(empty response)
$ dig AAAA stripe.com
(empty response)
</code></pre>

<h3 id="why-does-twitter-com-still-work-for-ipv6-users">why does <code>twitter.com</code> still work for IPv6 users?</h3>

<p>I found this really confusing, because I&rsquo;ve always heard that lots of internet
users are forced to use IPv6 because we&rsquo;ve run out of IPv4 addresses. But if
that&rsquo;s true, how could twitter.com continue to work for those people without
IPv6 support? Here&rsquo;s what I learned from the Twitter thread yesterday.</p>

<p>There are two kinds of internet service providers (ISPs):</p>

<ol>
<li>ISPs who own enough IPv4 address for all of their customers</li>
<li>ISPs who don&rsquo;t</li>
</ol>

<p>My ISP is in category 1 &ndash; my computer gets its own IPv4 address, and actually
my ISP doesn&rsquo;t even support IPv6 at all.</p>

<p>But lots of ISPs (especially outside of North America) are in category 2: they
don&rsquo;t have enough IPv4 addresses for all their customers. Those ISPs handle the problem by:</p>

<ul>
<li>giving all of their customers a unique IPv6 address, so they can access IPv6 sites directly</li>
<li>making large groups of their customers <em>share</em> IPv4 addresses. This can either be with CGNAT (&rdquo;<a href="https://en.wikipedia.org/wiki/Carrier-grade_NAT">carrier-grade NAT</a>&rdquo;) or &ldquo;464XLAT&rdquo; or maybe something else.</li>
</ul>

<p>All ISPs need <em>some</em> IPv4 addresses, otherwise it would be impossible for their
customers to access IPv4-only sites like twitter.com.</p>

<h3 id="what-are-the-reasons-to-support-ipv6">what are the reasons to support IPv6?</h3>

<p>Now we&rsquo;ve explained why it&rsquo;s possible to <em>not</em> support IPv6. So why support it?
There were a lot of reasons.</p>

<h3 id="reason-cgnat-is-a-bottleneck">reason: CGNAT is a bottleneck</h3>

<p>The argument that was most compelling to me was: CGNAT (carrier-grade NAT) is a
bottleneck and it causes performance issues, and it&rsquo;s going to continue to get
worse over time as access to IPv4 addresses becomes more and more restricted.</p>

<p>Someone also mentioned that because CGNAT is a bottleneck, it&rsquo;s an attractive
DDoS target because you can ruin lots of people&rsquo;s internet experience just by
attacking 1 server.</p>

<p>Servers supporting IPv6 reduces the need for CGNAT (IPv6 users can just connect
directly!) which makes the internet work better for everyone.</p>

<p>I thought this argument was interesting because it&rsquo;s a &ldquo;public commons&rdquo; /
community argument &ndash; it&rsquo;s less that supporting IPv6 will make your site
specifically work better, and more that if <em>almost everyone</em> supports IPv6 then
it&rsquo;ll make the experience of the internet better for everyone, especially in
countries where people don&rsquo;t have easy access to IPv4 addresses.</p>

<p>I don&rsquo;t actually know how much of an issue this is in practice.</p>

<p>There were lots of more selfish arguments to use IPv6 too though, so let&rsquo;s get
into those.</p>

<h3 id="reason-so-ipv6-only-servers-can-access-your-site">reason: so IPv6-only servers can access your site</h3>

<p>I said before that most IPv6 users still have access to IPv4 through some kind
of NAT. But apparently that&rsquo;s not true for everyone &ndash; some people mentioned
that they run some servers which only have IPv6 addresses and which aren&rsquo;t
behind any kind of NAT. So those servers are actually totally unable to access
IPv4-only sites.</p>

<p>I imagine that those servers aren&rsquo;t connecting to arbitrary machines that much
&ndash; maybe they only need to connect to a few hosts with IPv6 support.</p>

<p>But it makes sense to me that a machine should be able to access my site even
if it doesn&rsquo;t have an IPv4 address.</p>

<h3 id="reason-better-performance">reason: better performance</h3>

<p>For users who are using both IPv4 and IPv6 (with a dedicated IPv6 address and a
shared IPv4 address), apparently IPv6 is often faster because it doesn&rsquo;t need
to go through an extra translation layer.</p>

<p>So supporting IPv6 can make the site faster for users sometimes.</p>

<p>In practice clients use an algorithm called &ldquo;Happy Eyeballs&rdquo; which tries to
figure out whether IPv4 or IPv6 will be faster and then uses whichever seems
faster.</p>

<p>Some other performance benefits people mentioned:</p>

<ul>
<li>maybe sometimes using IPv6 can get you a SEO boost because of the better performance.</li>
<li>maybe using IPv6 causes you to go through better (faster) network hardware because it&rsquo;s a newer protocol</li>
</ul>

<h3 id="reason-resilience-against-ipv4-internet-outages">reason: resilience against IPv4 internet outages</h3>

<p>One person said that they&rsquo;ve run into issues where there was an internet outage
that only affected IPv4 traffic, because of accidental BGP poisoining.</p>

<p>So supporting IPv6 means that their site can still stay partially online during
those outages.</p>

<h3 id="reason-to-avoid-nat-issues-with-home-servers">reason: to avoid NAT issues with home servers</h3>

<p>A few people mentioned that it&rsquo;s much easier to use IPv6 with home servers &ndash;
instead of having to do port forwarding through your router, you can just give
every server a unique IPv6 address and then access it directly.</p>

<p>Of course, for this to work the client needs to have IPv6 support, but more and
more clients these days have IPv6 support too.</p>

<h3 id="reason-to-learn-about-ipv6">reason: to learn about IPv6</h3>

<p>One person said they work in security and in security it&rsquo;s very important to
understand how internet protocols work (attackers are using internet
protocols!). So running an IPv6 server helps them learn how it works.</p>

<h3 id="reason-to-push-ipv6-forward-ipv4-is-legacy">reason: to push IPv6 forward / IPv4 is &ldquo;legacy&rdquo;</h3>

<p>A couple of people said that they support IPv6 because it&rsquo;s the current
standard, and so they want to contribute to the success of IPv6 by supporting
it.</p>

<p>A lot of people also said that they support IPv6 because they think sites that only
support IPv4 are &ldquo;behind&rdquo; or &ldquo;legacy&rdquo;.</p>

<h3 id="reason-it-s-easy">reason: it&rsquo;s easy</h3>

<p>I got a bunch of answers along the lines of &ldquo;it&rsquo;s easy, why not&rdquo;. Obviously
adding IPv6 support is not easy in all situations, but a couple of reasons it
might be easy in some cases:</p>

<ul>
<li>you automatically got an IPv6 address from your hosting company, so all you need to do is add an <code>AAAA</code> record pointing to that address</li>
<li>your site is behind a CDN that supports IPv6, so you don&rsquo;t need to do anything extra</li>
</ul>

<h3 id="reason-safer-networking-experimentation">reason: safer networking experimentation</h3>

<p>Because the address space is so big, if you want to try something out you can
just grab an IPv6 subnet, try out some things in it, and then literally never
use that subnet again.</p>

<h3 id="reason-to-run-your-own-autonomous-system-as">reason: to run your own autonomous system (AS)</h3>

<p>A few people said they were running their own autonomous system (I talked about what an AS is a bit in this <a href="https://jvns.ca/blog/2021/10/05/tools-to-look-at-bgp-routes/">BGP post</a>). IPv4 addresses are too expensive so they bought IPv6 addresses for their AS instead.</p>

<h3 id="reason-security-by-obscurity">reason: security by obscurity</h3>

<p>If your server <em>only</em> has a public IPv6 address, attackers can&rsquo;t easily find it
by scanning the whole internet. The IPv6 address space is too big to scan!</p>

<p>Obviously this shouldn&rsquo;t be your only security measure, but it seems like a
nice bonus &ndash; any time I run an IPv4 public server I&rsquo;m always a tiny bit
surprised by how it&rsquo;s constantly being scanned for vulnerabilities (like old versions of WordPress, etc).</p>

<h3 id="very-silly-reason-you-can-put-easter-eggs-in-your-ipv6-address">very silly reason: you can put easter eggs in your IPv6 address</h3>

<p>IPv6 addresses have a lot of extra bits in them that you can do frivolous
things with. For example one of Facebook&rsquo;s IPv6 addresses is
&ldquo;2a03:2880:f10e:83:face:b00c:0:25de&rdquo; (it has <code>face:b00c</code> in it).</p>

<h3 id="there-are-more-reasons-than-i-thought">there are more reasons than I thought</h3>

<p>That&rsquo;s all I&rsquo;ve learned about the &ldquo;why support IPv6?&rdquo; question so far.</p>

<p>I came away from this conversation more motivated to support IPv6 on my
(very small) servers than I had been before. But that&rsquo;s because I think
supporting IPv6 will require very little effort for me. (right now I&rsquo;m using a
CDN that supports IPv6 so it comes basically for free)</p>

<p>I know very little about IPv6 still but my impression is that IPv6 support
often isn&rsquo;t zero-effort and actually can be a lot of work. For example, I have
no idea how much work it would actually be for Twitter to add IPv6 support on
their edge servers.</p>

<h3 id="supporting-ipv6-can-also-cause-problems">supporting IPv6 can also cause problems</h3>

<p>A friend who runs a large service told me that their service has tried to add
IPv6 support multiple times over the last 7 years, but each time it&rsquo;s caused
them problems. What happened to them was:</p>

<ul>
<li>they advertised an AAAA record</li>
<li>users would get the AAAA record and try to connect to them over IPv6</li>
<li>some network equipment in the user&rsquo;s ISP/internal network somewhere was broken, so the IPv6 connection failed</li>
<li>as a result those users were unable to use their service</li>
</ul>

<p>I thought it was interesting and surprising that supporting IPv6 can actually
in some cases make things <em>worse</em> for people on dual stack (IPv4 + IPv6)
networks.</p>

<h3 id="some-more-ipv6-questions">some more IPv6 questions</h3>

<p>Here are some more IPv6 questions I have that maybe I&rsquo;ll explore later:</p>

<ul>
<li>what are the <em>disadvantages</em> to supporting IPv6? what goes somehow wrong? (here&rsquo;s one <a href="https://support.fastly.com/hc/en-us/community/posts/360040169531-I-often-can-t-access-Fastly-servers-using-HTTPS-IPv6-RST-packets-received">example of an IPv6 problem</a> someone linked me to, for example)</li>
<li>what are the incentives for ISPs that own enough IPv4 addresses for their customers to support IPv6? (another way of asking: is it likely that my ISP will move to supporting IPv6 in the next few years? or are they just not incentivized to do it so it&rsquo;s unlikely?)</li>
<li><a href="https://docs.digitalocean.com/products/networking/floating-ips/">digital ocean</a> seems to only support IPv4 floating IPs, not IPv6 floating IPs. Why not? Shouldn&rsquo;t it be
<em>easier</em> to give out IPv6 floating IPs since there are more of them?</li>
<li>when I try to ping an IPv6 address (like example.com&rsquo;s IP <code>2606:2800:220:1:248:1893:25c8:1946</code> for example) I get the error <code>ping: connect: Network is unreachable</code>. Why? (answer: it&rsquo;s because my ISP doesn&rsquo;t support IPv6 so my computer doesn&rsquo;t have a public IPv6 address)</li>
</ul>

<p>This <a href="https://tailscale.com/kb/1134/ipv6-faq/">IPv4 vs IPv6 article from Tailscale</a> looks interesting and answers some of these questions.</p>


guid: https://jvns.ca/blog/2022/02/01/a-dns-resolver-in-80-lines-of-go/
title: A toy DNS resolver
url: https://jvns.ca/blog/2022/02/01/a-dns-resolver-in-80-lines-of-go/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Hello! I wrote a comic last week called &ldquo;life of a DNS query&rdquo; that explains how
DNS resolvers work.</p>

<p>In this post, I want to explain how DNS resolvers work in a different way &ndash;
with a short Go program that does the same thing described in the comic.
The main function (<code>resolve</code>) is actually just 20 lines, including comments.</p>

<p>I usually find it easier to understand things work when they come in the form
of programs that I can run and modify and poke at, so hopefully this program
will be helpful to some of you.</p>

<p>The program is here: <a href="https://github.com/jvns/tiny-resolver/blob/main/resolve.go">https://github.com/jvns/tiny-resolver/blob/main/resolve.go</a></p>

<h3 id="what-s-a-dns-resolver">what&rsquo;s a DNS resolver?</h3>

<p>When your browser needs to make a DNS query, it asks a <strong>DNS resolvers</strong>. When
they start, DNS resolvers don&rsquo;t know any DNS records (except the IP addresses
of the root nameservers). But they <em>do</em> know how to find DNS records for you.</p>

<p>Here&rsquo;s the &ldquo;life of a DNS query&rdquo; comic, which explains how DNS resolvers find DNS records for you.</p>

<p><a href="https://wizardzines.com/comics/life-of-a-dns-query/">
<img src="https://wizardzines.com/comics/life-of-a-dns-query/life-of-a-dns-query.png">
</a></p>

<h3 id="we-ll-use-a-library-for-parsing-dns-packets">we&rsquo;ll use a library for parsing DNS packets.</h3>

<p>I&rsquo;m not going to write this completely from scratch &ndash; I think parsing DNS
packets is really interesting, but it&rsquo;s definitely more than 80 lines of
code, and I find that it kind of distracts from the algorithm.</p>

<p>I really recommend writing a toy DNS resolver that actually does the parsing of
DNS packets if you want to learn about binary protocols though, it&rsquo;s really fun
and it&rsquo;s a totally doable to get something basic working in a weekend.</p>

<p>So I&rsquo;ve used <a href="https://github.com/miekg/dns">https://github.com/miekg/dns</a> for creating and parsing the DNS packets.</p>

<h3 id="dns-responses-contain-4-sections">DNS responses contain 4 sections</h3>

<p>You might think of DNS queries as just being a question and an answer (&ldquo;what&rsquo;s
the IP for <code>example.com</code>? it&rsquo;s <code>93.184.216.34</code>!). But actually DNS responses
contain 4 sections, and we need to use all 4 sections to write our DNS
resolver. So let&rsquo;s explain what they are.</p>

<p>Here&rsquo;s the <code>Msg</code> struct from the <code>miekg/dns</code> library, which lists the sections.</p>

<pre><code>type Msg struct {
        MsgHdr
        Compress bool       `json:&quot;-&quot;` // If true, the message will be compressed when converted to wire format.
        Question []Question // Holds the RR(s) of the question section.
        Answer   []RR       // Holds the RR(s) of the answer section.
        Ns       []RR       // Holds the RR(s) of the authority section.
        Extra    []RR       // Holds the RR(s) of the additional section.
}
</code></pre>

<p><strong>Section 1: Question</strong>. This is the section you use when you&rsquo;re creating a
query. There&rsquo;s not much to it &ndash; it just has a query name (like <code>jvns.ca.</code>), a
type (like <code>A</code>, but encoded as an integer), and a class (which is always the
same these days, &ldquo;internet&rdquo;).</p>

<p>Here&rsquo;s what the Question struct <code>miekg/dns</code> looks like:</p>

<pre><code>type Question struct {
        Name   string `dns:&quot;cdomain-name&quot;` // &quot;cdomain-name&quot; specifies encoding (and may be compressed)
        Qtype  uint16
        Qclass uint16
}
</code></pre>

<p><strong>Section 2: Answer</strong>. When you make a request like this:</p>

<pre><code>$ dig +short google.com
93.184.216.34
</code></pre>

<p>the IP address <code>93.184.216.34</code> comes from the <strong>Answer</strong> section.</p>

<p>The Answer, Authority, and Additional sections all contain <strong>DNS records</strong>.
Different types of records have different formats, but they all contain a <strong>name</strong>, <strong>type</strong>, <strong>class</strong>, and <strong>TTL</strong></p>

<p>Here&rsquo;s what the shared header looks like in <code>miekg/dns</code>:</p>

<pre><code>type RR_Header struct {
        Name     string `dns:&quot;cdomain-name&quot;`
        Rrtype   uint16
        Class    uint16
        Ttl      uint32
        Rdlength uint16 // Length of data after header.
}
</code></pre>

<p>&ldquo;RR&rdquo; stands for &ldquo;Resource Record&rdquo;.</p>

<p><strong>Section 3: Authority</strong>. When a nameserver redirects you to another server
(&ldquo;ask <code>a.iana-servers.net</code> instead!&ldquo;), this is the section it uses. <code>miekg/dns</code>
calls this section <code>Ns</code> instead of <code>Authority</code>, I guess because it contains
<code>NS</code> records.</p>

<p>Here&rsquo;s an example of an record in the Authority section of a DNS response.</p>

<pre><code>$ dig +noall +authority @h.root-servers.net example.com 
com.			172800	IN	NS	a.gtld-servers.net.
com.			172800	IN	NS	b.gtld-servers.net.
</code></pre>

<p>The Authority section can also contain SOA records but that&rsquo;s not relevant to
this post so I&rsquo;m not going to talk about that.</p>

<p><strong>Section 4: Additional</strong>. This is where &ldquo;glue records&rdquo; live. What&rsquo;s a glue
record? Well, basically when a nameserver redirects you to another server,
often it&rsquo;ll include the IP address of that server as well.</p>

<p>Here are the glue records from the same query above.</p>

<pre><code>$ dig +noall +additional @h.root-servers.net example.com 
a.gtld-servers.net.	172800	IN	A	192.5.6.30
b.gtld-servers.net.	172800	IN	A	192.33.14.30
</code></pre>

<p>There are other things in the Additional section as well, not just glue
records, but they&rsquo;re not relevant to this blog post so I&rsquo;m not going to talk
about them.</p>

<h3 id="the-basic-resolve-function-is-pretty-short">the basic <code>resolve</code> function is pretty short</h3>

<p>Now that we&rsquo;ve talked about the different sections in a DNS response, I can explain the resolver code.</p>

<p>Let&rsquo;s jump into the main function for resolving a name to an IP address.</p>

<p><code>name</code> here is a domain name, like <code>example.com.</code>`</p>

<pre><code>func resolve(name string) net.IP {
   // We always start with a root nameserver
   nameserver := net.ParseIP(&quot;198.41.0.4&quot;)
   for {
      reply := dnsQuery(name, nameserver)
      if ip := getAnswer(reply); ip != nil { // look in the &quot;Answer&quot; section
         // Best case: we get an answer to our query  and we're done
         return ip
      } else if nsIP := getGlue(reply); nsIP != nil { // look in the &quot;Additional&quot; section
            // Second best: we get a &quot;glue record&quot; with the *IP address* of
            // another nameserver to query 
         nameserver = nsIP
      } else if domain := getNS(reply); domain != &quot;&quot; { // look in the &quot;Authority&quot; section
            // Third best: we get the *domain name* of another nameserver to
            // query, which we can look up the IP for
         nameserver = resolve(domain)
      } else {
         // If there's no A record we just panic, this is not a very good
         // resolver :)
         panic(&quot;something went wrong&quot;)
      }
   }
}
</code></pre>

<p>Here&rsquo;s what that <code>resolve</code> function is doing:
1. We start with the root nameserver
2. Then we do a loop:
    a. Query the nameserver and parse the response
    a. Look in the &ldquo;Answer&rdquo; section for a response. If we find one, we&rsquo;re done
    a. Look in the &ldquo;Additional&rdquo; section for a glue record. If we find one, use that as the nameserver for the next query
    a. Look in the &ldquo;Authority&rdquo; section for a nameserver domain. If we find one, look up its IP and then use that IP as the nameserver for the next query</p>

<p>That&rsquo;s basically the whole program. There are a few helper functions to get
records out of the DNS response and to make DNS queries but I don&rsquo;t think
they&rsquo;re that interesting so I won&rsquo;t explain them.</p>

<h3 id="the-output">the output</h3>

<p>The resolver prints out all DNS queries it made, and the record it used to figure out what query to make it next.</p>

<p>It prints out <code>dig -r @SERVER DOMAIN</code> for each query even though it&rsquo;s not
actually using <code>dig</code> to make the query because I liked being able to run
the same query myself from the command line to see the response myself, for
debugging purposes.</p>

<p><code>-r</code> just means &ldquo;ignore what&rsquo;s in <code>.digrc</code>&rdquo;, it&rsquo;s there because I have some
options in my <code>.digrc</code>  (<code>+noall +answer</code>) that I wanted to disable when
debugging.</p>

<p>Let&rsquo;s look at 3 examples of the output.</p>

<h3 id="example-1-jvns-ca">example 1: jvns.ca</h3>

<pre><code>$ go run resolve.go jvns.ca.
dig -r @198.41.0.4 jvns.ca.
   any.ca-servers.ca.	172800	IN	A	199.4.144.2
dig -r @199.4.144.2 jvns.ca.
   jvns.ca.	86400	IN	NS	art.ns.cloudflare.com.
dig -r @198.41.0.4 art.ns.cloudflare.com.
   a.gtld-servers.net.	172800	IN	A	192.5.6.30
dig -r @192.5.6.30 art.ns.cloudflare.com.
   ns3.cloudflare.com.	172800	IN	A	162.159.0.33
dig -r @162.159.0.33 art.ns.cloudflare.com.
   art.ns.cloudflare.com.	900	IN	A	173.245.59.102
dig -r @173.245.59.102 jvns.ca.
   jvns.ca.	256	IN	A	172.64.80.1
</code></pre>

<p>We can see it had to make 6 DNS queries, 3 to look up <code>jvns.ca</code> and 3 to look up <code>jvns.ca</code>&rsquo;s nameserver, <code>art.ns.cloudflare.com</code></p>

<h3 id="example-2-archive-org">example 2: archive.org</h3>

<pre><code>$ go run resolve.go archive.org.
dig -r @198.41.0.4 archive.org.
   a0.org.afilias-nst.info.	172800	IN	A	199.19.56.1
dig -r @199.19.56.1 archive.org.
   ns1.archive.org.	86400	IN	A	208.70.31.236
dig -r @208.70.31.236 archive.org.
   archive.org.	300	IN	A	207.241.224.2
Result: 207.241.224.2
</code></pre>

<p>This one only had to make 3 DNS queries. This is because there was a glue
record available for archive.org&rsquo;s nameserver (<code>ns1.archive.org.</code>).</p>

<h3 id="example-3-www-maths-ox-ac-uk">example 3: www.maths.ox.ac.uk</h3>

<p>One last example: let&rsquo;s look up <code>www.maths.ox.ac.uk</code>. There&rsquo;s a reason for this one, I promise!</p>

<pre><code>dig -r @198.41.0.4 www.maths.ox.ac.uk.
   dns1.nic.uk.	172800	IN	A	213.248.216.1
dig -r @213.248.216.1 www.maths.ox.ac.uk.
   ac.uk.	172800	IN	NS	ns0.ja.net.
dig -r @198.41.0.4 ns0.ja.net.
   e.gtld-servers.net.	172800	IN	A	192.12.94.30
dig -r @192.12.94.30 ns0.ja.net.
   ns0.ja.net.	172800	IN	A	128.86.1.20
dig -r @128.86.1.20 ns0.ja.net.
   ns0.ja.net.	86400	IN	A	128.86.1.20
dig -r @128.86.1.20 www.maths.ox.ac.uk.
   ns2.ja.net.	86400	IN	A	193.63.105.17
dig -r @193.63.105.17 www.maths.ox.ac.uk.
   www.maths.ox.ac.uk.	300	IN	A	129.67.184.128
Result: 129.67.184.128
</code></pre>

<p>This makes <strong>7</strong> DNS queries, which is more than <code>jvns.ca</code>, which only needed
6. Why does it make 7 DNS queries instead of 6?</p>

<p>Well, it&rsquo;s because there are 4 nameservers involved in resolving <code>www.maths.ox.ac.uk</code> instead of 3. They are:</p>

<ul>
<li>the <code>.</code> nameserver</li>
<li>the <code>uk.</code> nameserver</li>
<li>the <code>ac.uk.</code> nameserver</li>
<li>the <code>ox.ac.uk.</code> nameserver</li>
</ul>

<p>You could even imagine there being a 5th one (a <code>maths.ox.ac.uk.</code> nameserver), but there isn&rsquo;t in this case.</p>

<p>jvns.ca only involves 3 nameservers:</p>

<ul>
<li>the <code>.</code> nameserver</li>
<li>the <code>ca.</code> nameserver</li>
<li>the <code>jvns.ca.</code> nameserver</li>
</ul>

<h3 id="real-dns-resolvers-actually-make-more-queries-than-this">real DNS resolvers actually make more queries than this</h3>

<p>When my resolver resolves <code>reddit.com.</code>, it only makes 3 DNS queries.</p>

<pre><code>$ go run resolve.go reddit.com.
dig -r @198.41.0.4 reddit.com.
   e.gtld-servers.net.	172800	IN	A	192.12.94.30
dig -r @192.12.94.30 reddit.com.
   ns-378.awsdns-47.com.	172800	IN	A	205.251.193.122
dig -r @205.251.193.122 reddit.com.
   reddit.com.	300	IN	A	151.101.129.140
Result: 151.101.129.140
</code></pre>

<p>But when <code>unbound</code> (the actual DNS resolver that I have running on my laptop)
resolves reddit.com, it makes more DNS queries. I captured them with <code>tcpdump</code>
to see what they were.</p>

<p>This <code>tcpdump</code> output might be a little illegible because well, that&rsquo;s how
tcpdump is, but hopefully it makes some sense.</p>

<p>Unbound skips the first step, because it has the address of the <code>com.</code>
nameserver cached.  Then the next 2 queries <code>unbound</code> makes are exactly the
same as my tiny Go resolver, except that it sends its first query to
<code>k.gtld-servers.net</code> instead of <code>e.gtld-servers.net</code>:</p>

<pre><code>12:38:35.479222 wlp3s0 Out IP pomegranate.19946 &gt; k.gtld-servers.net.domain: 51686% [1au] A? reddit.com. (39)
12:38:35.757033 wlp3s0 Out IP pomegranate.29111 &gt; ns-378.awsdns-47.com.domain: 8859% [1au] A? reddit.com. (39)
</code></pre>

<p>But then it keeps making DNS queries, even after it&rsquo;s done resolving <code>reddit.com</code>:</p>

<pre><code>12:38:35.757033 wlp3s0 Out IP pomegranate.29111 &gt; ns-378.awsdns-47.com.domain: 8859% [1au] A? reddit.com. (39)
12:38:35.757396 wlp3s0 Out IP pomegranate.31913 &gt; ns-1775.awsdns-29.co.uk.domain: 54236% [1au] A? ns-378.awsdns-47.com. (49)
12:38:35.757761 wlp3s0 Out IP pomegranate.62059 &gt; g.gtld-servers.net.domain: 28793% [1au] A? awsdns-05.net. (42)
12:38:35.757955 wlp3s0 Out IP pomegranate.34743 &gt; b0.org.afilias-nst.org.domain: 24975% [1au] A? awsdns-00.org. (42)
12:38:35.758051 wlp3s0 Out IP pomegranate.8977 &gt; a0.org.afilias-nst.info.domain: 53387% [1au] A? awsdns-00.org. (42)
12:38:35.758285 wlp3s0 Out IP pomegranate.11376 &gt; j.gtld-servers.net.domain: 41181% [1au] A? awsdns-05.net. (42)
12:38:35.775497 wlp3s0 In  IP ns-378.awsdns-47.com.domain &gt; pomegranate.29111: 8859*-$ 4/4/1 A 151.101.1.140, A 151.101.129.140, A 151.101.65.140, A 151.101.193.140 (240)
12:38:35.775948 lo    In  IP localhost.domain &gt; localhost.34429: 4033 4/0/1 A 151.101.1.140, A 151.101.129.140, A 151.101.65.140, A 151.101.193.140 (103)
# now it's done -- it returned its DNS response!
# but it keeps making queries about reddit.com's nameservers...
12:38:35.843811 wlp3s0 Out IP pomegranate.44738 &gt; ns-706.awsdns-24.net.domain: 14817% [1au] A? ns-1029.awsdns-00.org. (50)
12:38:35.845563 wlp3s0 Out IP pomegranate.55655 &gt; ns-1027.awsdns-00.org.domain: 3120% [1au] A? ns-1029.awsdns-00.org. (50)
12:38:36.017618 wlp3s0 Out IP pomegranate.53397 &gt; ns-775.awsdns-32.net.domain: 32671% [1au] A? ns-557.awsdns-05.net. (49)
12:38:36.045151 wlp3s0 Out IP pomegranate.40525 &gt; ns-454.awsdns-56.com.domain: 20823% [1au] A? ns-557.awsdns-05.net. (49)
</code></pre>

<p>So that&rsquo;s kind of interesting. I guess it makes sense that unbound would want
to cache more nameserver addresses in case it needs them in the future. Or
maybe that&rsquo;s what the DNS specification says to do?</p>

<h3 id="is-this-a-recursive-program">is this a &ldquo;recursive&rdquo; program?</h3>

<p>DNS resolvers are often called &ldquo;recursive nameservers&rdquo;. I&rsquo;ve stopped using that
terminology myself in explanations, but as far as I can tell, this is because
the <code>resolve</code> function is often a recursive function.</p>

<p>And the <code>resolve</code> function I wrote is definitely recursive! But I ran this
program on 500 different domains, and these are the number of times it
recursed:</p>

<ol>
<li>Sometimes 0 times (the function never calls itself)</li>
<li>Sometimes 1 time (the function calls itself once, to look up the IP address of one nameserver)</li>
<li>Very rarely 2 times (like for example to resolve <code>abc.net.au.</code> right now it needs to look up <code>r.au.</code>, then <code>eur2.akam.net.</code> then <code>abc.net.au.</code>)</li>
<li>So far, never 3 times</li>
</ol>

<p>Maybe there&rsquo;s a domain that this function would recurse more than 2 times on, but I don&rsquo;t know.</p>

<p>You definitely <em>could</em> write this program in a way that recurses more, by
replacing the loop with more recursion. And then it would recurse 3 or 6 or 7
or 9 times, depending on the domain. But to me the loop feels easier to read so
I wrote it with a loop instead.</p>

<h3 id="a-bash-version-of-this-resolver">a bash version of this resolver</h3>

<p>I wanted to see if it was possible to write a DNS resolver in 10-15 lines of bash, similarly to <a href="https://gist.github.com/jvns/ea2e4d572b4e2285148b8e87f70eed73">this short &ldquo;run a container&rdquo; script</a></p>

<p>The program I came up with was kind of too long in the end (it&rsquo;s about 36 lines), but here it is anyway. It uses the exact same algorithm as the Go program.</p>

<p><a href="https://github.com/jvns/tiny-resolver/blob/main/resolver.sh">https://github.com/jvns/tiny-resolver/blob/main/resolver.sh</a></p>

<p>The bash version is even more janky and uses <code>grep</code> in very questionable ways
but it did resolve every domain I tried which is cool.</p>

<p>It actually helped me write the Go resolver (which I actually started back in
November but got stuck on) because bash&rsquo;s limitations forced me to simplify the
design and simplifying it fixed a bug I was running into.</p>

<h3 id="how-is-this-different-from-a-real-dns-resolver">how is this different from a &ldquo;real&rdquo; DNS resolver?</h3>

<p>Obviously this is only 80 lines so there are a lot of differences between this
an a &ldquo;real&rdquo; DNS resolver. Here are a few:</p>

<ul>
<li>it only handles A records, not other record types</li>
<li>specifically it doesn&rsquo;t handle CNAME records (though you can easily add CNAME support with just <a href="https://github.com/jvns/tiny-resolver/commit/8a2dada63ec214ecf01046e3f57eb5406706b302">another 12 lines of code</a>)</li>
<li>it always only returns one A record even if there are more</li>
<li>it has absolutely no ability to handle errors like &ldquo;there were no A records&rdquo; (the Go program just panics)</li>
<li>the way it handles the glue records is a bit sketchy, probably it should
check that they match the nameservers in the &ldquo;Authority&rdquo; section or
something.  It seems to work though.</li>
<li>DNS resolvers are usually servers, this is a command line program</li>
<li>it doesn&rsquo;t validate DNSSEC or whatever</li>
<li>it doesn&rsquo;t do caching</li>
<li>it doesn&rsquo;t try a different nameserver if one of the domain&rsquo;s nameservers isn&rsquo;t working and times out the DNS query</li>
<li>like we mentioned above, unbound seems to look up the addresses of all the nameservers for a domain</li>
<li>probably there are other bugs and ways it violates the DNS spec that I don&rsquo;t know about</li>
</ul>

<h3 id="tiny-versions-of-real-programs-are-fun">tiny versions of real programs are fun</h3>

<p>As usual I always learn something from writing tiny versions of real programs.
I&rsquo;ve written this program before but I think this version is better than the
first version I wrote.</p>

<p>In 2020 I ran a 2-day workshop with my friend Allison called &ldquo;Domain Name
Saturday&rdquo; where all the participants wrote DNS resolvers. Basically the idea
was that you implement the algorithm described in this post, as well as the
binary parsing pieces that the <code>miekg/dns</code> library handles here. At some point
I want to write up that workshop so that other people could run it, because it
was really fun.</p>

<p>One question I still have is &ndash; are there domains where the <code>resolve</code> function
would recurse 3 times or more on? Obviously you could manufacture such a domain
by making it intentionally have to go through a bunch of hoops, but.. do they
exist in the real world?</p>


guid: https://jvns.ca/blog/2022/02/14/some-dns-terminology/
title: The multiple meanings of "nameserver" and "DNS resolver"
url: https://jvns.ca/blog/2022/02/14/some-dns-terminology/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>I&rsquo;m working on a zine about DNS right now, so I&rsquo;ve been thinking about DNS
terminology a lot more than a normal person. Here&rsquo;s something slightly
confusing I&rsquo;ve noticed about DNS terminology!</p>

<p>Two of the most common DNS server terms (&ldquo;nameserver&rdquo; and &ldquo;DNS resolver&rdquo;)
have different meanings depending on the situation.</p>

<p>Now this isn&rsquo;t a problem if you already understand how DNS works &ndash; I can
easily figure out what type of &ldquo;nameserver&rdquo; is being discussed based on
context.</p>

<p>But it can be a problem if you&rsquo;re trying to learn how DNS works and you don&rsquo;t
realize that those words might refer to different things depending on the
context &ndash; it&rsquo;s confusing! So I&rsquo;m going to explain the different possible
meanings and how to figure out which meaning is intended.</p>

<h3 id="the-2-meanings-of-nameserver">the 2 meanings of &ldquo;nameserver&rdquo;</h3>

<p>There are 2 types of nameservers, and which one the term &ldquo;nameserver&rdquo; means
depends on the context.</p>

<p><strong>Meaning 1: &ldquo;authoritative&rdquo; nameservers</strong></p>

<p>When you update the DNS records for a domain, those records are stored on a
server called an <strong>authoritative nameserver</strong>.</p>

<p>This is what &ldquo;nameserver&rdquo; means in the context of a <strong>specific domain</strong>. Here are a few examples:</p>

<ul>
<li>&ldquo;Connect a domain you already own to Wix by changing its <em>name servers</em>.&rdquo;</li>
<li>&ldquo;Almost all domains rely on multiple <em>nameservers</em> to increase reliability:
if one nameserver goes down or is unavailable, DNS queries can go to
another one.&rdquo;</li>
<li>&ldquo;You can update the <em>nameserver</em> records yourself by following the steps
your domain registrar may provide in the help content at their website&rdquo;</li>
</ul>

<p><strong>Meaning 2. &ldquo;recursive&rdquo; nameservers, also known as &ldquo;DNS resolvers&rdquo;</strong></p>

<p>These servers cache DNS records. Your browser doesn&rsquo;t make a request to an
authoritative nameserver directly. Instead it makes a request to a DNS resolver (aka recursive nameserver)
which figures out what the right authoritative nameserver to talk to is, gets
the record, and caches the result.</p>

<p>This is what &ldquo;nameserver&rdquo; means in the context of <strong>you browsing the internet</strong>.
(&ldquo;your computer&rsquo;s nameservers&rdquo;). Here are a few examples:</p>

<ul>
<li>&ldquo;Changing <em>nameservers</em> can be a pain on some devices and require multiple
clicks through a user interface. On Windows 10, for example&hellip;&rdquo;</li>
<li>&ldquo;Are your DNS <em>nameservers</em> impeding your Internet experience? NEW RELEASE
adds nameservers  1.1.1.1,  1.0.0.1  and  9.9.9.9&rdquo;</li>
<li>&ldquo;Configure your network settings to use the IP addresses 8.8.8.8 and 8.8.4.4 as your DNS servers&rdquo;</li>
</ul>

<p>I prefer to use the term &ldquo;DNS resolver&rdquo; even though it has 2 meanings because
it&rsquo;s much more commonly used than &ldquo;recursive nameserver&rdquo;.</p>

<h3 id="meanings-of-dns-resolver">meanings of &ldquo;DNS resolver&rdquo;</h3>

<p>A DNS resolver can either be a library or a server. (I&rsquo;m sorry, I know I said
that a DNS resolver is a server earlier. But sometimes it&rsquo;s a library.)</p>

<p><strong>Meaning 1a: &ldquo;stub resolver&rdquo; (library version)</strong></p>

<p>A &ldquo;stub resolver&rdquo; is something (it can be either a library or a DNS server)
which doesn&rsquo;t know how to resolve DNS names itself, it&rsquo;s just in charge of
forwarding DNS queries to the &ldquo;real&rdquo; DNS resolver. Let&rsquo;s talk about stub resolvers that are libraries first.</p>

<p>For example, the <code>getaddrinfo</code> function from libc doesn&rsquo;t know how to look up
DNS records itself, it just knows to look in <code>/etc/resolv.conf</code> and forward the
query to whatever DNS server(s) it finds there.</p>

<p>How you can tell if this is what&rsquo;s meant: if it&rsquo;s part of your computer&rsquo;s
operating system and/or if it&rsquo;s a library, it&rsquo;s a stub resolver.</p>

<p>Examples of this meaning of &ldquo;DNS resolver&rdquo;:</p>

<ul>
<li>&ldquo;The <em>resolver</em> is a set of routines in the C library that provide access to
the Internet Domain Name System (DNS)&rdquo;</li>
<li>&ldquo;These are the DNS servers used to resolve web addresses. You can list up to
three, and the <em>resolver</em> tries each of them, one by one, until it finds one
that works.&rdquo;</li>
<li>&ldquo;If the command succeeds, you will receive the following message &ldquo;Successfully flushed the <em>DNS Resolver</em> Cache.&ldquo;&rdquo;</li>
</ul>

<p><strong>Meaning 1b: &ldquo;stub resolver&rdquo; (server version)</strong></p>

<p>Stub resolvers aren&rsquo;t always libraries though, like <code>systemd-resolved</code> and
<code>dnsmasq</code> are stub resolvers but they&rsquo;re servers. Your router might be running
<code>dnsmasq</code>.</p>

<p>This is also known as a &ldquo;DNS forwarder&rdquo;.</p>

<p>How you can tell if this is what&rsquo;s meant: if your router is running it or it&rsquo;s
part of your OS, it&rsquo;s probably a stub resolver.</p>

<p><strong>Meaning 2: a recursive nameserver (a server)</strong></p>

<p>A &ldquo;recursive nameserver&rdquo; (like we talked about before) is a server that knows
how to find the authoritative nameservers for a domain. This is the kind of DNS resolver I was talking about in this <a href="https://jvns.ca/blog/2022/02/01/a-dns-resolver-in-80-lines-of-go/">A toy DNS resolver</a>
post a couple of weeks ago (though mine wasn&rsquo;t a server).</p>

<p>How to tell if this is what&rsquo;s meant: if it&rsquo;s <code>unbound</code>, <code>bind</code>, 8.8.8.8,
1.1.1.1, or run by your ISP, then it&rsquo;s a recursive nameserver.</p>

<p>Examples of this meaning of &ldquo;DNS resolver&rdquo;:</p>

<ul>
<li>&ldquo;The DNS Resolver in pfSense® software utilizes unbound, which is a
validating, recursive, caching <em>DNS resolver</em>&hellip;&rdquo;</li>
<li>&ldquo;We invite you to try Google Public DNS as your primary or secondary <em>DNS resolver</em>&hellip;&rdquo;</li>
<li>&ldquo;I work for a reasonably large mobile service provider and we are in the process of implementing our own <em>DNS resolver</em>&hellip;&rdquo;</li>
</ul>

<h3 id="the-most-popular-dns-server-words">the most popular DNS server words</h3>

<p>I also did a quick unscientific survey of which terms to refer to DNS servers
were the most common by counting Google results. Here&rsquo;s what I found:</p>

<ul>
<li>dns server: 8,000,000</li>
<li>nameserver: 4,200,000</li>
<li>dns resolver: 933,000</li>
<li>public DNS server: 204,000</li>
<li>root nameserver: 42,000</li>
<li>recursive resolver: 38,500</li>
<li>stub resolver: 26,100</li>
<li>authoritative nameserver: 17,000</li>
<li>dns resolution service: 9,450</li>
<li>TLD nameserver: 7,500</li>
<li>dns recursor: 5,300</li>
<li>recursive nameserver: 5,060</li>
</ul>

<p>Basically what this tells me is that by a pretty big margin, the most popular
words used when talking about DNS serves are &ldquo;nameserver&rdquo;, and &ldquo;DNS resolver&rdquo;.</p>

<p>The more specific terms like &ldquo;recursive nameserver&rdquo;, &ldquo;authoritative
nameserver&rdquo;, and &ldquo;stub resolver&rdquo; are much less common.</p>

<h3 id="that-s-all">that&rsquo;s all!</h3>

<p>I hope this helps some folks understand what these words mean! The terminology
is a bit messier than I&rsquo;d like, but it seems better to me to explain it than to
use less-ambiguous language that isn&rsquo;t as commonly used in practice.</p>


guid: https://jvns.ca/blog/2022/02/20/things-that-used-to-be-hard-and-are-now-easy/
title: Things that used to be hard and are now easy
url: https://jvns.ca/blog/2022/02/20/things-that-used-to-be-hard-and-are-now-easy/
feedurl: https://jvns.ca/atom.xml
content_mime_type: text/html
content:
<p>Hello! I was talking to some friends the other day about the types of
conference talks we enjoyed.</p>

<p>One category we came up with was &ldquo;you know this thing that used to be super
hard? Turns out now it&rsquo;s WAY EASIER and maybe you can do it now!&ldquo;.</p>

<p>So I asked on Twitter about <a href="https://twitter.com/b0rk/status/1494301227533967369?s=20&amp;t=M_RYvWzEOymZ--YjE9jvpQ">programming things that used to be hard and are now easy</a></p>

<p>Here are some of the answers I got. Not all of them are equally &ldquo;easy&rdquo;, but I
found reading the list really fun and it gave me some ideas for things to
learn. Maybe it&rsquo;ll give you some ideas too.</p>

<ul>
<li>SSL certificates, with Let&rsquo;s Encrypt</li>
<li>Concurrency, with async/await (in several languages)</li>
<li>Centering in CSS, with flexbox/grid</li>
<li>Building fast programs, with Go</li>
<li>Image recognition, with <a href="https://simonwillison.net/2018/Oct/29/transfer-learning/">transfer learning</a> (someone pointed out that the joke in <a href="https://xkcd.com/1425/">this XKCD</a> doesn&rsquo;t make sense anymore)</li>
<li>Building cross-platform GUIs, with Electron</li>
<li>VPNs, with Wireguard</li>
<li>Running your own code inside the Linux kernel, with eBPF</li>
<li>Cross-compilation (Go and Rust ship with cross-compilation support out of the box)</li>
<li>Configuring cloud infrastructure, with Terraform</li>
<li>Setting up a dev environment, with Docker</li>
<li>Sharing memory safely with threads, with Rust</li>
</ul>

<p>Things that involve hosted services:</p>

<ul>
<li>CI/CD, with GitHub Actions/CircleCI/GitLab etc</li>
<li>Making useful websites by only writing frontend code, with a variety of &ldquo;serverless&rdquo; backend services</li>
<li>Training neural networks, with Colab</li>
<li>Deploying a website to a server, with Netlify/Heroku etc</li>
<li>Running a database, with hosted services like RDS</li>
<li>Realtime web applications, with Firebase</li>
<li>Image recognition, with hosted ML services like Teachable Machine</li>
</ul>

<p>Things that I haven&rsquo;t done myself but that sound cool:</p>

<ul>
<li>Cryptography, with opinionated crypto primitives like libsodium</li>
<li>Live updates to web pages pushed by the web server, with LiveView/Hotwire</li>
<li>Embedded programming, with MicroPython</li>
<li>Building videogames, with Roblox / Unity</li>
<li>Writing code that runs on GPU in the browser (maybe with Unity?)</li>
<li>Building IDE tooling with LSP (the language server protocol)</li>
<li>Interactive theorem provers (not sure with what)</li>
<li>NLP, with HuggingFace</li>
<li>Parsing, with PEG or parser combinator libraries</li>
<li>ESP microcontrollers</li>
<li>Batch data processing, with Spark</li>
</ul>

<p>Language specific things people mentioned:</p>

<ul>
<li>Rust, with non-lexical lifetimes</li>
<li>IE support for CSS/JS</li>
</ul>

<h3 id="what-else">what else?</h3>

<p>I&rsquo;d love more examples of things that have become easier over the years.</p>


